The Python library BeautifulSoup is an incredible tool for pulling out information from a webpage. You can use it not only to extract tables and lists but you can also use to to pull out very specific elements like a paragraph with a green font color. To briefly illustrate this functionality and in honor of the upcoming World Cup we will use BeautifulSoup on world soccer rankings.

Working with BeautifulSoup

If you’re reading this post I’ll assume you know how to install a Python library (hint easy_install beautifulsoup from the command line). For this example we also use the library urllib2 to help us open a URL.

To start, of course, you’ll want to import the two libraries:

from BeautifulSoup import BeautifulSoup
import urllib2

With the two libraries installed you can now open the URL and use BeautifulSoup to read the web page. Given that the World Cup is coming up we decided to apply this example to the FIFA rankings listed on the ESPN FC web page. We're using Mexico as the example (although we'd like to see them move deep into the tournament we're not hopeful).

Here is what the page on ESPN's soccer site looks like (you can find the main page

In order to open and read the page using BeautifulSoup (and urllib2) you would use the following code.

url= 'http://www.espnfc.com/spi/rankings/_/view/fifa/teamId/203/mexico?cc=5901'
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())

If you were to print out soup you could see the entire webpage. Although it looks like a simple, long text string there is a lot more too it.

Let's use a simple example. If you were to look at the underlying code (or at soup) you would see a div called rank-box which has the FIFA rank. Here is what this looks like in the Chrome developer tools console:

To grab this number is incredibly simple. You 'find' the div by class, you identify the h6 element child and you get the contents. All in one line of code:

rank = soup.find("div", {"class": "rank-box"}).h6.contents

It's as easy as this. In our case we want the team name, the rank and the rating. Based on an inspection of the web page DOM we determined that these can be extracted using the following code:

rank = soup.find("div", {"class": "rank-box"}).h6.contents
teaminfo = soup.find("div", {"class": "team-info"})
name = teaminfo.h4.contents
rating = teaminfo.ul.p.span.contents

We can then loop through the teams and grab the data. We were particularly interested in the difference between the FIFA rank and the rank determined by ESPN's Soccer Power Index. Perhaps an average of these two would be a good indicator of success in the tournament. By default the table is sorted by this average. Take a look at the final table below (all columns are sortable and the table is not limited to World Cup teams).

The difference in FIFA and SPI is particularly amazing for Portugal. Who will the US face -- 3rd ranked Portugal or 14th ranked Portugal?

`