Beautifulsoup: Scraping Different Data Sets Having Same Set Of Attributes In The Source Code

I'm using the BeautifulSoup module for scraping the total number of followers and total number of tweets from a Twitter account. However, when I tried inspecting the elements of th

Solution 1:

In this case, one way to achieve it, is to check that data-is-compact="true" only appears twice for each piece of data you want to extract, and also you know that tweets is first and followers second, so you can have a list with those titles in same order and use a zip to join them in a tuple to print both at same time, like:

import urllib2
from bs4 import BeautifulSoup

profile = ['Tweets', 'Followers']

link = "https://twitter.com/iamjericho"
r = urllib2.urlopen(link)
src = r.read()
res = BeautifulSoup(src)
followers = ''for p, d inzip(profile, res.find_all('span', { 'data-is-compact': "true"})):
    print p, d.text

It yields:

Tweets 21,8K                                                                                                                                                                                                                                                                   
Followers 2,47M