How To Loop Through 100 Url And Extract Info From Each One With Selenium

I have this list, what is the best to go to extract a piece of info from each one and store this info into another list consider the wanted info hello world def

Solution 1:

Since you are dealing with a single host. so you have to maintain the session object firstly so you will avoid to be blocked or flagged by almost of sites firewalls as DDOS-Attack, where you actually maintain the same TCP socket stream without keep open/close/reopen the socket.

After that you can loop over your desired pagination parameter and extract the title.

Below is an example for that.

import requests
from bs4 import BeautifulSoup


defmain(url):
    with requests.Session() as req:
        for page inrange(1, 11):
            r = req.get(url.format(page))
            soup = BeautifulSoup(r.content, 'html.parser')
            print(soup.title.text)


main("https://www.example.com/page={}")