How To Scrape Data After Clicking Button

I am trying to scrape data from website with beautiful soup, but to scrape all content, I have to click button t

Solution 1:

You can use the same API endpoint the page does which returns all the info in json form. Set a records return count higher than the total expected number. I show parsing out the album titles/urls from the json. You can explore response here. You can find this endpoint in the browser network tab when refreshing the url you supplied.

import requests

data = {"fan_id":1812622,"older_than_token":"1557167238:2897209009:a::","count":1000}
r = requests.post('https://bandcamp.com/api/fancollection/1/wishlist_items', json = data).json()
details = [(item['album_title'], item['item_url']) for item in r['items']]
print(details)