No Response For Xhr Request In Python With Requests.get()

I want to scrape german poll data from a server. Here, I search for an examplary street, straße 'Judengasse'. I have been trying to reproduce this. Unfortunately, the link from th

Solution 1:

After some research I finally managed to get a 200 response from this server.

Firstly, requests.get in this case should be replace by requests.post, since you want to replicate an HTTP POST request, according to the info you got from Chrome's dev mode, "General" section.

Secondly, from the headers we can see that the data is sent as being of type "multipart/form-data" request. As far as I could understand, this is a type of request that is used to send files instead of regular data (more about this type of request here).

So, I converted the string sent through the POST request to binary (this is achieved by prepending b) and passed it to the files parameter of the request. For some reason, this parameter requires a tuple (a, b) inside a set {c}, hence the {(None, data)}.

I also passed the street name as a parameter to data, so it's easier to manipulate it.

I got this working code (I'm using my browser's request):

import requests

url = 'https://online-service2.nuernberg.de/Finder/action/getItems'

street = b'Judengasse'

data = b'-----------------------------15242581323522\r\n' \
       b'Content-Disposition: form-data; name=\"action\"\r\n\r\n' \
       b'\"action/getItems\"\r\n-----------------------------15242581323522\r\n' \
       b'Content-Disposition: form-data; name="data"\r\n\r\n' \
       b'{\"finder\":\"Wahlraumfinder\",\"strasse\":\"%s\",\"hausnummer\":\"0\"}\r\n' \
       b'-----------------------------15242581323522--' % street

headers = {"Host": "online-service2.nuernberg.de",
            "User-Agent": "Mozilla/5.0 (WindowsNT10.0; Win64; x64; rv:70.0) Gecko/20100101Firefox/70.0",
            "Accept": "*/*",
            "Accept-Language": "en-US,en;q=0.5",
            "Accept-Encoding": "gzip, deflate, br",
            "X-Requested-With": "XMLHttpRequest",
            "Content-Type": "multipart/form-data; boundary=---------------------------15242581323522",
            "Content-Length": "321",
            "Origin": "https://online-service2.nuernberg.de","DNT": "1",
            "Connection": "keep-alive",
            "Referer": "https://online-service2.nuernberg.de/Finder/?Wahlraumfinder",
           }


multipart_data = {(None, data,)}
response = requests.post(url, files=multipart_data, headers=headers)

print(response.text)

I got this raw response:

{"id":"8c4f7a57-1bd6-423a-8ab8-e1e40e1e3852","items":[{"zeilenbeschriftung":"Wahl-/Stimmbezirk","linkAdr":null,"mapUrl":"http://online-service.nuernberg.de/Themenstadtplan/sta_gebietsgli
ederungen.aspx?p_urlvislayer=Stimmbezirke&XKoord=4433503.05&YKoord=5480253.301&Zaehler=1&Textzusatz=Judengasse+0&z_XKoord=4433670.0&z_YKoord=5480347.0&z_Zaehler=1&z_Textzusatz=Wahllokal%
20Willst%E4tt.-Gym.%2C+Innerer+Laufer+Platz+11","items":["0652","Judengasse, Neue Gasse","Willstätt.-Gym., Innerer Laufer Platz 11","Zi. 101 ,1. OG",null]},{"zeilenbeschriftung":"Stimmkr
eis Landtagswahl","linkAdr":null,"mapUrl":"http://online-service.nuernberg.de/Themenstadtplan/sta_gebietsgliederungen.aspx?p_urlvislayer=Stimmkreis_LTW&XKoord=4433503.05&YKoord=5480253.3
01&Zaehler=1&Textzusatz=Judengasse+0&p_scale=100000","items":["501","Nürnberg-Nord"]},{"zeilenbeschriftung":"Wahlkreis Bundestagswahl","linkAdr":null,"mapUrl":"http://online-service.nuer
nberg.de/Themenstadtplan/sta_gebietsgliederungen.aspx?p_urlvislayer=Wahlkreis_BTW&XKoord=4433503.05&YKoord=5480253.301&Zaehler=1&Textzusatz=Judengasse+0&p_scale=150000","items":["244","N
ürnberg-Nord"]}],"status":200}

which you can easily parse to get the result you expect:

print(response.json()["items"][0]["items"])

yilding...

['0652', 'Judengasse, Neue Gasse', 'Willstätt.-Gym., Innerer Laufer Platz 11', 'Zi. 101 ,1. OG', None]

Hope it helps.

Regards