Skip to content Skip to sidebar Skip to footer

Python Beautiful Soup Only Scraping The Lower Part Of The Page

I'm trying to pull product info from a rather large apparel website page, but soup only appears to scrape a lower half of the html document, at an arbitrary cutoff, so the data tha

Solution 1:

As stated in one of the comments the HTML you are trying to get is added with JavaScript running on the browser.

I recommend you this package Requests-HTML created by the author of very popular requests.

from requests_html import HTMLSession

session = HTMLSession()
r = session.get('https://www.pullandbear.com/rs/man/sale-c1030036006.html')
r.html.render()

print(r.html.html)

Post a Comment for "Python Beautiful Soup Only Scraping The Lower Part Of The Page"