How to Download Multiple Records using Python

I'm pleased to announce my latest endeavour to teach historians how to program computers in aid of their research has been published and is freely available on the Programming Historian 2 website.

Full Lesson: Downloading Multiple Records Using Query Strings

The hands-on lesson builds upon some introductory Python tutorials also available on the site and is intended to teach practicing historians how to use Python to discriminately download a copy of sources found in scholarly databases to their own computer for further processing. Thanks to Fred Gibbs, Miriam Posner, and Sara Palmer for their assistance working the lesson into shape. The abstract is available below:

Downloading Multiple Records Using Query Strings

Downloading a single record from a website is easy, but downloading many records at a time – an increasingly frequent need for a historian – is much more efficient using a programming language such as Python. In this lesson, we will write a program that will download a series of records from the Old Bailey Online using custom search criteria, and save them to a directory on our computer. This process involves interpreting and manipulating URL Query Strings. In this case, the tutorial will seek to download sources that contain references to people of African descent that were published in the Old Bailey Proceedings between 1700 and 1750.

For Whom is this Useful?

Automating the process of downloading records from an online database will be useful for anyone who works with historical sources that are stored online in an orderly and accessible fashion and who wishes to save copies of those sources on their own computer. It is particularly useful for someone who wants to download many specific records, rather than just a handful. If you want to download all or most of the records in a particular database, you may find Ian Milligan’s tutorial on Automated Downloading with WGET more suitable.

The present tutorial will allow you to download discriminately, isolating specific records that meet your needs. Downloading multiple sources automatically saves considerable time. What you do with the downloaded sources depends on your research goals. You may wish to create visualizations or perform various data analysis methods, or simply reformat them to make browsing easier. Or, you may just want to keep a backup copy so you can access them without Internet access