Iterating through Multiple URL CSV Strings in Python


I want to learn how to get CSV files from URLs.

While I can make the code below work by hard coding the name of the CSV string variable, I want to learn how to iterate through many CSV strings.

import csv
import requests

CSV_URL_1 = ''
CSV_URL_2 = ''

csv_list = []

for i in range(1,3):
    concat = "CSV_URL_" + str(i)

with requests.Session() as s:
    csv_list_dict = {}
    for i in csv_list:
        download = s.get(i) #This part is the problem
        decoded_content = download.content.decode('utf-8')
        cr= csv.reader(decoded_content.splitlines(), delimiter=',')
        my_list = list(cr)


In case it's not clear, I want the "i" in download = s.get(i) to become "CSV_URL_1" on the first iteration and "CSV_URL_2" on the second (I can copy the code twice and hard code these values to get the correct result), but I can't figure out how to make this iteration work. Instead, I get a missing schema error.

What am I doing wrong?

Show source
| url   | python   | csv   2017-08-08 23:08 1 Answers

Answers to Iterating through Multiple URL CSV Strings in Python ( 1 )

  1. 2017-08-08 23:08

    When you do this:

    concat = "CSV_URL_" + str(i)

    You are putting the strings "CSV_URL_1" and "CSV_URL_2" in csv_list.

    But the first time your code does this:

    download = s.get(i)

    you are clearly expecting this to mean

    download = s.get('')

    but it doesn't. It means

    download = s.get("CSV_URL_1")

    Now you can see why you are getting a missing schema error. There is no https:// in the URL. Your code is computing a variable name and then trying to use that name as if it were a variable.

    Do this instead:

    CSV_URL = ['', '']
    for i in range(2):
        download = s.get(CSV_URL[i]) 

Leave a reply to - Iterating through Multiple URL CSV Strings in Python

◀ Go back