Iterating through Multiple URL CSV Strings in Python

Question

I want to learn how to get CSV files from URLs.

While I can make the code below work by hard coding the name of the CSV string variable, I want to learn how to iterate through many CSV strings.

import csv
import requests

CSV_URL_1 = 'https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_1994-2003_CDC_NCHS.csv'
CSV_URL_2 = 'https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_2000-2014_SSA.csv'

csv_list = []

for i in range(1,3):
    concat = "CSV_URL_" + str(i)
    csv_list.append(concat)

with requests.Session() as s:
    csv_list_dict = {}
    for i in csv_list:
        download = s.get(i) #This part is the problem
        decoded_content = download.content.decode('utf-8')
        cr= csv.reader(decoded_content.splitlines(), delimiter=',')
        my_list = list(cr)
        csv_list_dict[i]=my_list

csv_list_dict

In case it's not clear, I want the "i" in download = s.get(i) to become "CSV_URL_1" on the first iteration and "CSV_URL_2" on the second (I can copy the code twice and hard code these values to get the correct result), but I can't figure out how to make this iteration work. Instead, I get a missing schema error.

What am I doing wrong?


Show source
| url   | python   | csv   2017-08-08 23:08 1 Answers

Answers to Iterating through Multiple URL CSV Strings in Python ( 1 )

  1. 2017-08-08 23:08

    When you do this:

    concat = "CSV_URL_" + str(i)
    csv_list.append(concat)
    

    You are putting the strings "CSV_URL_1" and "CSV_URL_2" in csv_list.

    But the first time your code does this:

    download = s.get(i)
    

    you are clearly expecting this to mean

    download = s.get('https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_1994-2003_CDC_NCHS.csv')
    

    but it doesn't. It means

    download = s.get("CSV_URL_1")
    

    Now you can see why you are getting a missing schema error. There is no https:// in the URL. Your code is computing a variable name and then trying to use that name as if it were a variable.

    Do this instead:

    CSV_URL = ['https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_1994-2003_CDC_NCHS.csv', 'https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_2000-2014_SSA.csv']
    
    for i in range(2):
        download = s.get(CSV_URL[i]) 
    

Leave a reply to - Iterating through Multiple URL CSV Strings in Python

◀ Go back