How to convert request to dictionary in Python?

The

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

Learning how to send HTTP requests allows you to query websites and API's for data or even a webpage so you can scrape data out with something like Beatifulsoup.

Installation

To install the requests module, first make sure you have pip setup. If not, follow my tutorial on setting up pip. After making sure pip is working, execute

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

Making a Simple Request

Just like your browser does, you can request a web page using Python. Get the URL of a website and use the following code to download the content of the page.

import requests

r = requests.get('https://nitratine.net/blog/')
print (r.text)

When executing this, a lot of text should be printed. This is the content returned by the site requested, in the case of using the URL demonstrated above, the content will be HTML that renders my blog feed.

In this example, I have called

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

When a browser calls a web page, it initially requests the URL being visited and then calls other URLs that are referenced in the HTML document to render the page with the CSS, JS, image and other files required.

Attributes You Should Know About

The

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

• import requests
  
  request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
  data = request.json()
  print('Title: ' + data['title'])
  

  8: The raw content of the request

• import requests
  
  request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
  data = request.json()
  print('Title: ' + data['title'])
  

  6: The text content of the request

• import requests
  
  file_destination = 'nitratine-logo.png'
  url = "https://nitratine.net/static/img/logo.png"
  response = requests.get(url)
  if response.status_code == 200:
      with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
          f.write(response.content)
  

  0: The status code of the response; e.g. 200 OK, 404 Not Found, 418 Short and Stout...

• import requests
  
  file_destination = 'nitratine-logo.png'
  url = "https://nitratine.net/static/img/logo.png"
  response = requests.get(url)
  if response.status_code == 200:
      with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
          f.write(response.content)
  

  1: The headers of the response

• import requests
  
  file_destination = 'nitratine-logo.png'
  url = "https://nitratine.net/static/img/logo.png"
  response = requests.get(url)
  if response.status_code == 200:
      with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
          f.write(response.content)
  

  2: The cookies returned in the response. You can access cookie value like you would use a dict:

  import requests
  
  file_destination = 'nitratine-logo.png'
  url = "https://nitratine.net/static/img/logo.png"
  response = requests.get(url)
  if response.status_code == 200:
      with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
          f.write(response.content)
  

  3.

To get the value of these attributes, use them as I did in the previous code example.

.json()

If a site returns a JSON response, you can call

import requests

file_destination = 'nitratine-logo.png'
url = "https://nitratine.net/static/img/logo.png"
response = requests.get(url)
if response.status_code == 200:
    with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
        f.write(response.content)

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

This saves the effort on deserializing the .text value using the .

Downloading an Image and Other Files

Just like downloading a web page, you can also download other files like images or videos. Simply find the URL of the image/other file (make sure it is the URL of the file, not the URL of the page it's on) and use it like before; but this time put the content into a file.

import requests

file_destination = 'nitratine-logo.png'
url = "https://nitratine.net/static/img/logo.png"
response = requests.get(url)
if response.status_code == 200:
    with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
        f.write(response.content)

If you expect the file to be large (or just want to use a stream), you can use a stream to write to the output file as you receive the data. This means as the data comes in, it is written to the output file so the data does not have to sit in memory.

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

These examples have been modified from stackoverflow.com/a/13137873

Different Methods

When making a request to a URL/URI, different 'methods' can be used. These tell the server what sort of action you want to perform. HTTP defines actions like GET, POST, PUT, DELETE and many others.

To use these different methods, simply replace the

import requests

file_destination = 'nitratine-logo.png'
url = "https://nitratine.net/static/img/logo.png"
response = requests.get(url)
if response.status_code == 200:
    with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
        f.write(response.content)

import requests

file_destination = 'nitratine-logo.png'
url = "https://nitratine.net/static/img/logo.png"
response = requests.get(url)
if response.status_code == 200:
    with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
        f.write(response.content)

import requests

file_destination = 'nitratine-logo.png'
url = "https://nitratine.net/static/img/logo.png"
response = requests.get(url)
if response.status_code == 200:
    with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
        f.write(response.content)

import requests

file_destination = 'nitratine-logo.png'
url = "https://nitratine.net/static/img/logo.png"
response = requests.get(url)
if response.status_code == 200:
    with open(file_destination, 'wb') as f: # Make sure to use wb are we are writing bytes
        f.write(response.content)

import requests

r = requests.delete('https://jsonplaceholder.typicode.com/posts/1')
assert r.status_code == 200 # Check for HTTP 200 (OK)

Parameters in URLs

Instead of constructing a URL to add parameters to it, the requests module offers a method to add them automatically for you. Simply create your key-value pairs in a Python dictionary and then pass this to the

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

parameters = { 'key1' : 'value1', 'key2' : 'value2' }
r = requests.get('https://httpbin.org/get', params=parameters)
print (r.text)

To show that the parameters were added to the url, we can verify it using a PreparedRequest.

import requests

parameters = { 'key1' : 'value1', 'key2' : 'value2' }
req = requests.Request('GET','https://httpbin.org/get', params=parameters)
prepared = req.prepare() # Get a PreparedRequest object
print (prepared.url)

The output of the above script is "https://httpbin.org/get?key1=value1&key2=value2" as expected.

Sending Data in HTTP Body

When making calls to endpoints that require data in the body, we can use form-encoded, JSON or raw bodies.

Form Encoded Body

In cases where you want to send some form-encoded data (like a HTML form would submit), we can pass key-value pairs as we used above to the

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

body_data = { 'key1' : 'value1', 'key2' : 'value2' }
r = requests.post('https://example.com/create-something', data=body_data)

JSON Body

Today, JSON bodies are becoming more popular over form-encoded key-value pairs; this is due to how much more compact JSON is over XML (XML and JSON are typically used to carry larger payloads). Fortunately, JSON bodies are as easy to create in the request module as form-encoded are.

Instead of passing your dictionary to the

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

body_data = { 'key1' : 'value1', 'key2' : 'value2' }
r = requests.post('https://example.com/create-something', json=body_data)

Raw Body

In the cases where you want to specify exactly what is in the body manually, simply provide the string to the

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

body_data = "This is my body"
r = requests.post('https://example.com/create-something', data=body_data)

Custom headers

Previously I discussed that the

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

file_destination = 'multi-clipboard.gif'
url = 'https://nitratine.net/posts/multi-clipboard/multi-clipboard.gif'
r = requests.get(url, stream=True)
if r.status_code == 200:
    with open(file_destination, 'wb') as f:
        for chunk in r:
            # print('Writing chunk') # Uncomment this to show that the file is being written chunk-by-chunk when parts of the data is received
            f.write(chunk) # Write each chunk received to a file

import requests

r = requests.delete('https://jsonplaceholder.typicode.com/posts/1')
assert r.status_code == 200 # Check for HTTP 200 (OK)

import requests

request = requests.get('https://jsonplaceholder.typicode.com/todos/1')
data = request.json()
print('Title: ' + data['title'])

This is Only The Basics

Please know that these are only the basics of the requests library and there is a lot more to offer from this library; this tutorial however, should help you understand what you need to do to request and send data.

If you want to read up more on this library, Google is full of answers but the docs at requests.readthedocs.io is great for reference material.

I Don't Have s Server, How Can I Test These?

webhook.site is an amazing place to test requests. Simply going to the page will create a new webhook for you to use - copy the unique URL at the top of the page (not the URL address bar). Now whenever you make a request to that URL, you will be able to see the content of the request you made.

How do you turn something into a dictionary in Python?

How to convert a JSON to dictionary in Python?

How to convert a string into dictionary in Python?

How do I extract data from a Python request?