Showing posts with label digital humanities. Show all posts
Showing posts with label digital humanities. Show all posts

Saturday, August 9, 2014

Learning Python with the Programming Historian

For those humanists out there looking to learn Python to aid your research processes, the Programming Historian has a great set of lessons to get you started. The lessons are designed to teach you Python by doing the types of tasks historians might want to do. So instead of learning about managing an inventory of widgets (as is common in intro-to-programming books) you learn how to manage a set of historical sources.

The Programming Historian used to make it more obvious that these lessons were originally written sequentially, so that readers could build upon their skills slowly. It's not quite so obvious anymore because of the new way we've organised our table of contents. But for those of you interested in learning Python, or using it with students, I thought it would be helpful to post their original order here so that you can easily find your way through them.

Happy learning.

Your First Lesson

Introduction to Python

---

You may also like to supplement your learning with other tutorials. I found Mark Lutz, 'Learning Python' (O'Reilly) very useful. My co-editor, Fred Gibbs, is a big fan of the Code Academy. Use whatever combination works for you. Good luck.

Thursday, October 10, 2013

Would you buy a product to support digital humanities?

I'm launching an experiment, and I'd love for you to be involved. My PhD funding has just run out and I've been given a final £350 tuition bill during what's known as my 'writing up' period. In my search for solutions to cover this cost, I've found dozens of small grants that will pay for me to buy train tickets or hotel stays for research trips or conferences. I've found dozens more that will let me buy train tickets or hotel stays for others to come to a conference I'd organize. I can even get money to buy equipment for my research projects. But no one will give me money to pay my rather modest fees.

So I've decided to be creative. Crowdfunding has become rather trendy lately. Sites such as Kickstarter are even being taught as part of digital humanities courses, suggesting those of us in the field need to get out there and convince the public to part with some money in support of the research we do. Shawn Graham at Carleton University is now using this idea to raise money for an Undergraduate Scholarship in digital history, with funds to be matched by his university if he meets a certain threshold, and I wish him the best with what I consider a great initiative. But I know the marketplace can only handle so many campaigns that take the same form. So I've decided to go another route and ask: would you buy a product if you knew the procedes went to support digital humanities? Or more specifically: helped to pay my tuition fees?

So I've teamed up with Cafepress, and designed some digital humanities schwag to tempt you into my experiment. I've focused my product line on three key areas for the digital humanities:

  1. Bags and Electronics - Your electronics never looked so digital humanities
  2. Baby Clothes - Your baby makes digital humanities look good
  3. Mugs and Water Bottles - Support digital humanities while you drink


All of my profits will go directly towards my tuition fees. And in the interest of this experiment, I'll report back on progress at the end of 2013 when these limited edition products will disappear FOREVER! Are baby clothes the key to the future of digital humanities? We'll soon find out.

I thank you most humbly for your support.
Edited note: It has been wisely pointed out to me that not everyone needs baby clothes or more 'stuff'. If you'd like to contribute directly, I've set up a link through Paypal where you can do so. Thanks again.

Monday, September 9, 2013

Digital Humanities Comic 'Big Data + Old History'


You used to submit an abstract to a conference to share your findings. Now you ‘Dance your Thesis’ or compete to convince a world-class cartoonist to animate your research and turn it into a video. The modes of disseminating research have broadened in the past decade, with students in particular being offered a range of new contests designed to get them thinking creatively about engaging the public with academic research.

Jorge Cham, the internationally renowned animator behind ‘PhD Comics’, asked students ‘can you describe your thesis in two minutes?’ Cham then chose the best descriptions and turned them into animated cartoons. I'm very pleased to announce my entry was one of the winners, and the animated video of my thesis has just been released:


My two-minute talk focused on how distant reading has been central to my PhD research. There's only so much detail you can fit into a two minute talk, but I hope has been able to introduce the idea of distant reading to a much wider audience and that some of them might take the step to learn more. It's been a great experience, and I'd like to thank Jorge and his team for creating this opportunity. And since getting selected as one of the winners was partially down to voting from the public, I'd also like to thank everyone who took a moment last year to vote for my entry. The response has been wonderful. So thanks again.

I hope you enjoy the result.

Monday, August 5, 2013

Can We Reconstruct a Text from a Wordcloud?


We’ve all seen Word Clouds. Many of us have even wondered if they’re of any value. I have used word clouds in the past; I find them useful in presentations when I want to highlight the relative importance of certain words over others. For example, I often use this word cloud to the left, to show the most common Irish surnames in the London area during the early 19th century. I hope my listeners will note that Murphy or Sullivan is more common than Burke or Foley, without me having to take the time to explain the connection between word-size and significance.

I’ve also used word clouds in analysis. In a previous post I discussed how I was able to use the below word cloud to show the relative frequency of topics found in the Gentleman’s Magazine between 1800 and 1820, which allowed me to get a pretty good idea of what the gentry and the middle class were interested in during that period.

I think both of those uses for word clouds have been productive. They’ve allowed me to transmit ideas, and formulate my own thoughts on a set of data in an effective manner. But I began to think about other uses, and I began to wonder about the process of getting back to the original data. Word clouds take the individual words (tokens) out of context. As I mentioned in my last post, we think in metaphors, or ideas. Not in words. That means a word cloud reduces a single idea such as “green bowl” into two tokens “green” and “bowl”. It then combines the word “green” into a single graphic based on how often it appears in the text. The program does not take into consideration the fact that “green” as it refers to a bowl is entirely different than Mr. Green or Green Park. An article about Mr. Green’s picnic in Green Park with his favourite green bowl might give you a skewed idea about the importance of the word green, here representing three completely different ideas, and in all three cases simply acting as modifiers to more important concepts (a man, a park, and a bowl).

Just for fun, I decided to do a test. I asked 4 colleagues, all experts on the criminal trial transcripts of the Old Bailey Online, to look at word cloud of a trial. Each person was asked to describe what key information they could tell me about the crime. I was interested in knowing if they could tell me the who, what, when, where, why type details, and if they could reconstruct the basic building blocks from the prevalence of certain keywords. In the spirit of exploration, I played along as well and offered my own interpretation.

The word cloud was created at random by my wife without my knowledge of the trial. All I knew (and all the participants knew) was that the trial took place between 1801 and 1820, and was between 2,000 and 3,000 words long. The word cloud was limited to 75 unique words and common English words were removed. The resultant visualization looks like this:
  
I have colour-coded my assessment (blue) for aspects I got correct and (red) for the bits I got wrong. What struck me immediately was that this was a case involving theft. That’s a safe bet anyways, since about 50% of Old Bailey trials during this period were theft cases. It was the large number of nouns that led me to this conclusion. I know that trial transcripts always list the items that were stolen, and the testimony in the trial almost always discusses the various objects repeatedly as several witnesses are called to give their account of what transpired. In this case, I’d assume there was a large quantity of alcohol that went missing, ranging from red wine, to port, to gin – stored in bottles, measured by gallons, and in at least one case: a cask.

At the time it was stolen the booze was being stored in a cellar before it was transferred to a cart that was being driven by a horsea mare to be specific. Why Restoration actress Nell Gwyn appears in the set of words, I have no idea since she died over a century before this era, unless the name is a coincidence or perhaps refers to the name of a pub that lost its liquor.

There were a large number of people involved with giving evidence against the defendant including Messers Hutt, Wells, Powell, Wood and Bagnigge, as well as possibly a Mr. Limbrick, and definitely someone named Hart – though again that may be the name of the pub. One of those men is likely the watchman and another an officer. Based on what I know about Old Bailey trials, this suggests there were a lot of witnesses, meaning the prosecutor was concerned that his case may not have succeeded.

The alcohol heist took place in the morning (and was perhaps discovered the following night), and the goods were then transferred down either Maiden-lane or City-road. Given the volume of goods stolen and the fact that death appears in the list suggests our defendant was found guilty and sentenced to death.

My conclusion: a pub named either Hart or Nell Gwyn located on City-road or Maiden-lane was robbed of a large volume of alcohol by a solo male defendant, who was found guilty and sentenced to death.

Analysis

You can compare my assessment with the full trial transcript. It turns out I wasn’t that far off. There had been three defendants, but they were found guilty and sentenced to death. It had been a large alcohol theft. I wasn’t able to accurately pick out the fact that Messers Wood, Powell, and Hart were the defendants, meaning the who aspect of the challenge had completely eluded me. I also didn’t recognize Bagnigge Wells, which was the location of the crime, not someone’s name. Nell Gwyn in this case was the pub that had its door pried open to reach the alcohol.

Participant 1

“Powell and Hutt were found guilty of breaking into the wine cellar of the Nell Gywnn inn in Bagnigge Wells (I know about this place) They were accused of stealing two gallons of wine, casks of gin and bottles of port from the cellar  belonging to Mr Hart . A Watchmen Mr Wood on his round at 1 o'clock saw a broken iron (lock)on the wine cellar door  and saw two men drive off in a horse and cart down Maiden lane towards City Road and immediately called for an officer. The men were stopped by an officer who examined the cart and found the cask of wine, gin and port belonging to Mr Hart and arrested them. They received the death penalty.”

Participate 2

“I would guess it involves stealing a hamper of goods including a bottle of wine from the back of a cart.  My suspicion is that the defendants were two women, and that the servant of the person who owned the cart/hamper discovered the theft and was called in evidence, though the actual owner wasn't.  The cart was on its way to or from northwest London to banigge wells, for a social occasion and some visiting, it happened at night (suggesting they were returning home) and there was a runner or 'officer' involved in the arrest.”

Analysis

In these examples, the participants tried to be very specific about the details of the transcript, and in doing so were incorrect more often than not. The basics of the case, including the location of the crime, was however, correct for participant 1. This would suggest that expert readers are able to get some of the basics – though by no means all. However, that expertise does little to bring forth the specifics of the trial.

Participant 3

“This trial seems to have a richer vocabulary than most.  It looks like a theft case from a wine cellar (wine, port, cask, gin, gallons, hampers, bottles, etc. suggest as much), presumably at Bagnigge Wells; actually the prominence of the word cart, and also the word horse, suggest the material might have been in the process or coming or going there, perhaps parked on the City Road (or Maiden Lane). Went suggests action.  Some force was used: broken, crow, saw, which suggests that the cart was broken into.   There is a certain amount of vocabulary indicating how the culprit might have been apprehended: officer, stopped, examined, watchman, observed, charge—this suggests that officers were used to apprehend the defendant. There are some names: Powell, Gwyn, Hart, Mr, Nell, William.  The most frequently mentioned, Hart, is presumably the victim.  Timing is important: o'clock, morning, night—the prominence of night suggests that that is when the crime took place, with the suspect arrested in the morning?  Numbers indicate either the number of items stolen or the time of day (one or two in the morning).

Analysis

This one came out surprisingly accurate. The participant hadn’t recognized Nell Gwyn as the name of the actress. As in my own case, it proved impossible to figure out who was the defendant and who was the victim. However, the details of the crime and the process of apprehending the defendant is almost bang on. This participant didn’t try to reconstruct the narrative in the same way as Participant 1 or 2, and thus avoided many of the pitfalls. However, there has been no guess at the verdict, and while the basics of the trial are here, the richness of what actually is recorded in the transcript is nearly entirely lost.


Participant 4

“Geographical location
Wine Cellar –property crime
Bagnigge Wells – recognise this as the location of a rather seedy spa/pleasure
grounds on the outskirts of Clerkenwell towards Kings Cross.
City Road – not too far away runs from the City to Islington
Maiden Lane – small street that runs parallel to Covent Garden Piazza on the
south-side although there might be other Maiden Lanes
Time –mention of time o’clock and watchman (usually worked only at night)
and night although mentions morning (in the or the next)
Crime probably burglary because of the mention of the relevance of time
together with watchman
Stolen Goods: bottles of red wine, port, cask, gin from wine cellar
Broken into cellar with an iron bar, maybe crow bar
Probably took it away or hid it in a cart, certainly it is central to the plot and
perhaps discovery of the stolen goods. There may well have been a significant
amount of drink gallons, casks and bottles? Hampers – might have been used
to transport/hide the goods
Arrested – yes (examined) and officer (probably from one of the Police Courts
Observed - spotted
Names Hart – think this is a personal name rather than pub sign and possibly
Wood as it’s mentioned frequently but not as much as the things I think were
actually stolen
Gwyn –forename Welsh? Ha – just spotted nell (same size) and I know that
Nell Gywn was supposed to have lived at Banigge Wells. John, William, think
Hutt is a personal name (not a shed)
Death –punishment (so guilty)”

Analysis

This one was particularly interesting, because the participant included their thought process as it related to the various words on the visualization. The fact that it was clearly a theft of goods, and that time was mentioned, lead this person to conclude we were dealing with a burglary, which was correct.  The conclusion that Hart was the name of a person rather than a pub was correct, but equally could have been wrong. While Nell Gwyn may have lived at Bagnigge Wells, that wasn’t really relevant to this case.

Conclusions

Can an expert on a historical source reconstruct the details of that source from a word cloud? It would seem the answer is: sort of. Of the five participants, two (#3 and #4) did an incredibly good job of getting some of the details. These people were able to reason out what the words meant by drawing upon their experience with the ways certain types of words were used in criminal trial transcripts. However, in both cases they were light on the details, and it would appear decided not to guess on elements of the crime that they couldn’t be confident in. That is, they spoke confidently when they were confident but otherwise stayed silent.

I think my analysis fell in the middle. I got lots of bits right, but I was also wrong just about as often. I was disappointed that I couldn’t pick out the roles people played in the trial from the word cloud. There was no way to guess who was the defendant, who was the victim, and without recognizing the names, who were the officers. I wasn’t even able to guess how many defendants were on trial. On the other hand, I did guess the type of crime, the verdict, and a few details and circumstances surrounding the arrest. Having said that, I can’t say I was confident in all of my conclusions and was guessing.

And finally, two of the participants were way off (#1 and #2). These two attempted to reconstruct the narrative of what had happened, providing a level of detail that involved a good deal of guesswork.

What does this mean? I think it shows that when faced with a simplified visualization such as a word cloud, the process of getting back to the original is fraught with a level of guesswork. However, an expert in the source material can, with reasonable accuracy, reconstruct some of the more basic details of what’s going on.

How do we move forward? Well, as I pointed out earlier in this article, I think the secret is in moving away from the idea that tokens transmit ideas. Ideas and metaphors transmit ideas, and it would be far more useful to have an idea or concept cloud than one that focuses on individual tokens. But I also think it’s time that those ideas were linked back to the original data points, so that people interpreting the word clouds can test their assumptions. We are ready to see the distance between the underlying data and the visualization contracted. We’re ready to see the proof embedded in the graph. And I hope we continue to see a development in this trend.

Thanks to my participants, Janice Turner, Bob Shoemaker, Louise Falcini, and Tim Hitchcock.

Thursday, August 1, 2013

Can you explain this graph to me? Peer Reviewing a Visualization

"For sale: Mixing bowl set designed to please a cook".

That opening sentence contains 10 words, or "tokens" as linguists often call them. Yet either in its spoken or written form, it really only transmits 4 ideas, or what I imagine Marc Alexander would call "metaphors", which are concepts that go beyond the words but that express meaning and understanding. They allow us to think in chunks.



What?: For sale
What's for sale?: a mixing bowl set
What's it like?: designed to please
Please whom?: a cook

The same sentence represents an attempt to conjure a very measured set of thoughts in another person. I can't take credit for the sentence, but when the author wrote it down, they hoped that you, dear reader, would understand those 4 ideas in the same way as all the other readers, and as they themselves understood them. It's their attempt to control your mind temporarily by drawing upon your understanding and memories associated with those 4 ideas. We may not get all the details exactly the same. Your mixing bowl set may be blue. Mine is seafoam and has spout on each bowl to make it easier to pour your batter into the baking tin. So we likely havn't had exactly the same understanding of the sentence, but our understandings are almost certainly within the limits of what's acceptable to the author.

If we add 2 more ideas to the end of the sentence we end up with a failed conjuration:

"For sale: Mixing bowl set designed to please a cook with a round bottom for efficient beating".

Because of the misplaced modifier, there are now two ways to understand these ideas. Does the bowl have a round bottom for efficient beating, or should the cook who will enjoy the bowl be so proportioned?

Visualizations can offer the same ambiguity.



Is this an image of a rabbit, or a duck?

In this case, it's both, and it's that very ambiguity that the artist intended us to understand. Not all visualizations are intended to teach us something specific, or to so carefully conjure a series of ideas in our minds. That's wholly too modernist for some. Visualizations can be exploratory, used by researchers to come to a different understanding of their data by slicing it in lots of ways until they see something interesting. Or, as I demonstrated in an earlier post, can be a quick way to get a distant look at a large amount of data by reducing it to something easier to digest. In that sense graphing can aid the discovery process of research even before the conclusions are ready to be shared with the world.

But when it comes to visualizations for academic publication, unintentional ambiguity is something we must strive to avoid. If done well, there should only be one proper way of interpreting the visualization. It's our job to create something that can conjure specific thoughts in the reader's head based on the graph's shape, colour, size, orientation, etc. And it should go without saying that those conjured thoughts should be grounded in rigorous research.

As academics we spend so much time and care on our prose, and even our footnotes. Usually (we hope) that prose comes out lucid and if we're lucky, is enjoyable to read. One of the ways we ensure that is through peer review. The editors help us find people who are willing to take the time to read what we've written and provide constructive feedback upon it.

Yet few of us feel we have the aptitude to offer similar feedback on visualizations. We're not visual artists and so we can be forgiven for using colour in confusing ways, or for thinking a pie chart with 100 categories is a good way to express an idea. As I mentioned previously, I'm quite confident that in the present climate, unique looking or impressive visualizations will slip through peer review unchecked, lest the reviewer's lack of expertise in visualization be exposed by making a comment to the effect of "I don't under stand this graph".

Now, far be it from me to suggest we only use column graphs or line graphs, or that we do X, but not Y. I think it's fantastic that so many people out there are pushing the boundaries of what we can achieve via visualization. The folks at the Guardian Data Blog do great work on bringing data to life, and are a wonderful place for anyone seeking inspiration.

Instead, what I would suggest is that as creators of academic visualizations, we make sure our graphs are reviewed, even if our reviewers cannot or will not do so in the traditional peer review process.

The way I'd propose we do that is to show our friends and colleagues what we've made as often as we can, including during the drafting process. But it's not just about showing them. We have to ask the right questions. Let's use the graph below as a (relatively poor) example of a visualization that we might like to get feedback on. Please note that this is not a graph showing real data about the cost of grain in the 19th century. It's just an example.

Most of us likely want to ask "Do you like my graph?" or "What do you think of this?"

A more productive starting point is probably: Can you explain this graph to me? You aren't going to be there when your reader or viewer is interpreting your graph. The best way to find out what set of ideas are going to form in their mind is to ask them to explain their thought process out loud.

In this case, I had intended to show the seasonal difference in the price of grain in London and Edinburgh over a 20 year period. You may not have picked up on that, which means I need to fix something.

Don't be affraid to ask explicitly: Is there any element of this graph that you do not inherently understand? Make sure they can explain the labels on both axes (if relevant). If they don't know where you're getting those values from you may need to rethink your axis labels. You'd be forgiven for asking what the numbers on the Y-axis represent in the example. I didn't label it, so how could you know?

When you start experimenting with your visualizations, you're bound to come up with ideas you think are clear, but that just don't translate into ideas that your reader can interpret. Looking at the sample graph, I wouldn't fault you for asking what the top and bottom line of the curves represent. They're supposed to be two line graphs: one representing Edinburgh prices, and one representing London. I've shaded in the space between the lines to emphasize the size of the gap. If this is in fact two lines, then which one is Edinburgh? Which one is London? And when they overlap, how do I know which bit corresponds to which line? Do they cross, or merely meet and diverge again? I havn't made the fact that this is a line graph obvious because the lines aren't distinguishable from the shape formed by the colours.

Speaking of colour, you'll want to make sure you havn't come up with a palette that is going to make interpreting your graph difficult for someone with colour blindness. There are many different forms of colour-blindness, so it pays to run a test on your graph. You can do this online by using a "Colour Blindness Simulator" on your finished image.

Sticking with the negatives, ask your tester which element of the graph they like the least. For the sample graph, they may say they don't like the colours, or the font, or the legend. Personally, I think using --------> to represent arrows looks lazy. Everyone will have their own opinions on what's worst about your work. If you know what turns people off you can make visualizations that people like. And if they like the visualization, readers are more likely to engage with its message. With this in mind, go ahead and ask if they like your graph. Or if there are any elements of the graph that they particularly fancy.

Just as with your prose, it may take a few iterations and a number of different opinions from colleagues before a graph says to others what you think it says in your own mind. Just because you submitted a graph with your article and the peer-reviewers didn't comment on it doesn't mean you've done a good job of clearly expressing your ideas visually.

And one last question to ask, just to make sure your readers get the right message and aren't distracted: does the shape of the graph make it look like anything unrelated?

Graphs and visualizations have tremendous potential for expressing ideas in academic research, but it's not a skill we're typically taught in school. Most of us learn on the job, or emulate graphs we saw elsewhere that we found effective. Taking the time to ensure the graphs you create transmit the right ideas to your reader is good scholarship. Knowing the right questions to ask makes it that much easier to reach that result.

Questions to ask about a visualization:
  1. Can you explain this graph to me?
  2. Are there any elements you do not inherently understand?
  3. Can you explain what each axis shows (if applicable)
  4. Will people with colour blindness be able to differentiate your colour palette? (check online)
  5. What do you like least about the graph?
  6. Do you like the graph / a particular element of the graph?
  7. Does the shape of the graph make it look like anything distracting?

Tuesday, July 23, 2013

Students should be empowered, not bullied into open access

'Bully Free Zone' by Eddie-S
The American Historical Association (AHA) has just adopted a resolution in support of recent graduates, encouraging them to feel empowered to keep their dissertations offline while they seek a publisher to turn that dissertation into a scholarly monograph.

Surprise, surprise, open access advocates everywhere have started snivelling.

No! they cry. We shouldn't support a resolution passed in good faith to protect the career progression of new scholars against scholarly presses that are allegedly refusing to accept manuscripts based on openly available dissertations. We should be burning books and the organizations that publish them. Down with books, up with free information on the Internet!

Lovely, but you can't eat free information. Makes a shit shelter as well.

Now, I certainly understand, sympathize, and even agree with the complaints of the open access community. Trevor Owens posted some great suggestions last night for ways to amend the AHA statement into one that recognizes some real flaws in the publication / promotion / tenure model that is over-reliant upon books. I certainly agree with Owens that it makes no sense to leave career progression of historians in the hands of acquisition editors at famous scholarly presses.

I'd also suggest that the AHA's claim that history is a "book" discipline is a bit too narrow. From where I live in London England, hundreds of thousands of people make their living either directly or indirectly off of history. That can be anything from freelance tour guides who offer historic walks through the City, to the cafeteria workers in the museums and historic sites, the actor who draws you into his theatre for a rendition of Richard III or the actress who portrays Elizabeth Woodville in a television series, or even her Majesty the Queen whose very presence and connection to a historic institution draws in millions of tourists every year.

The AHA's perspective is probably flawed in terms of the negative reaction of presses towards open access of dissertations. A yet to be published (and open access) article Do Open Access Electronic Theses and Dissertations Diminish Publishing Opportunities in the Social Sciences and Humanities suggests that the vast majority of publishers are willing to consider submissions based on openly available theses.

With all of this in mind, let's give the open access community what they want: You're right.

But dear God you're obnoxious.

The decision of the AHA to support this measure is nothing but a well-intentioned gesture designed to protect and empower those at the most vulnerable point in their career from a perceived threat. How could anyone could criticize them for that? The AHA and scholarly societies like it are not the enemy, and they don't operate to keep scholarship in the 19th century. They exist to promote the interests of their members, and that's exactly what the AHA has done with this resolution. If you want to change their direction, join them. Run for positions of power within their ranks, and influence the opinions of their membership. The historians who belong to these organizations aren't stupid, so if your ideas are good and your models sound, there's no reason we can't expect gradual change towards open access.

Both scholarly monographs and open access have their merits. We shouldn't be pushing for either / or, just like we havn't driven actors from the stage because we have television. Scholarly monographs are an effective way of preserving historical knowledge; they're in a format that the vast majority of us understand and even appreciate. We don't need to give that up.

And while I can appreciate the advantages of open access, its advocates often ignore the problems of an open access model. We live in a society in which things that have no cost have no perceived value. You wouldn't expect your lawyer to work for free, so why your historian? The scholarly presses defend their (failing) business model because it keeps their friends and family employed, their kids fed, and their bills paid. This isn't just a matter of profits funneling into the pockets of the rich. It's the way people like you and me make modest and honest livings.

If we start giving everything away we're promoting a model in which certain professions operate without the security of a paycheque while others doing important work continue to charge for their services. It's all well and good for open access advocates to tell us the benefits of their model, but until they come up with some solutions for its failings, they won't gain any friends who are sitting on the fence. Especially not if every well-intentioned effort by a scholarly society is met with a hostile barrage on Twitter by an extremist perspective that ignores the fact that we're all on the same team: We love history and we want to spend our careers sharing it with others.

If you want to give your dissertation away online, by all means do so. But it is your dissertation. You should feel equally empowered to bury it in a hole in the back yard, or throw it off a bridge. Anyone who tells you that you're bound by some moral obligation to give it away has a job, or a trust fund, and has no business putting any demands on your labour. Even if your scholarly book never earns you a cent, it's your prerogative to try and flog it any way you like. That doesn't make you a bad person. Neither does withholding your thesis from the Internet if you think that will help your pursuit towards a career that allows you to provide for your family. I hold my right to support my family far above your right to read my ideas for free.

I wholeheartedly want to thank the AHA for standing up for and empowering new scholars. No good deed goes unpunished, but there are many of us out there who appreciate your efforts and look forward to continued progress in what we hope becomes a civil debate and progression towards increased open access.








Saturday, April 20, 2013

Is the Programming Historian 2 a MOOC?

'Evil Robot' by Jennifer Morrow (cc-by)
A few months ago I was asked if the Programming Historian 2 is a MOOC. For the uninitiated, a MOOC is a Massive OpenOnline Course. They’ve been popping up online for the past couple of years, principally at major American universities like MIT and Stanford, claiming to be able to teach thousands or even hundreds of thousands of students at the same time – for free. They’ve so far had mixed results but it seems most people in academia have an opinion on them – either, meh it’s a fad, damn we gotta get one of those at our school, or the robots have come for our jobs! Defend! Defend!

I can’t speak for the other editors of the Programming Historian 2 (PH2). But I can say: No. I don’t think the PH2 is a MOOC.  If you havn’t found us yet, the PH2 is an open access series of tutorials designed to let humanities researchers get their toes wet with computer programming. The lessons involve learning simple programming tasks that are immediately useful to ordinary working humanists. That might be automatically downloading historical recordsfrom the Internet, or analyzing a collection of sources with topic modeling. All of the lessons are online – like a MOOC – and there is no teacher in the room with you – like a MOOC.

So why no MOOC? For me, what sets a MOOC apart from a classroom-based course is a belief that the tutor-tutee relationship can be depersonalized and made redundant. MOOCs replace this relationship with a series of steps. If you learn the steps in the right order and engage actively with the material you learn what you need to know and who needs teacher?

I don’t think that’s what we’re about. Instead, some of the most exciting feedback we’ve got at the PH2 has been from academics who have used the PH2 as a teaching tool in their classroom. Either they’ve assigned lessons for their students to work through, they’ve challenged students to write lessons of their own, or they’ve used the PH2 to teach themselves a skill that they can then pass along to their students.

That’s not to say you can’t use the PH2 to teach yourself some programming if you havn’t got a teacher. It’s to say the PH2 is not the evil robot looking to take your job away. It’s the friendly robot looking to give your teaching toolkit a few more options, and maybe a new skill or two with which to impress your friends and colleagues. Not unlike a book. And Books havn’t put literature professors out of a job, but they have made English lit courses more interesting.

Monday, April 15, 2013

Trust Me: The Old Bailey Online as a model for digitization projects

The Old Bailey Online (OBO) turned 10 years old this week, and to celebrate, Sharon Howard has been encouraging blog posts and tweets from the project's wide network of contributors. I thought I'd add just a few brief thoughts on what I like about the OBO, and why I avoid so many other competing digitization projects. Rather than explain what the OBO is, I thought I'd save time and steal the explanation from their own website:
A fully searchable edition of the largest body of texts detailing the lives of non-elite people ever published, containing 197,745 criminal trials held at London's central criminal court.
The trials run from 1678 to 1914, making it a great resource for social historians or historians of crime. I broadly fit into both of those categories, but what really interests me is knowledge management. I want to know how we can extract useful knowledge from bodies of text far larger than we could ever read in our lifetime. I'm interested in the historical research questions I pursue, but I'm more interested in the processes of understanding and discovery that the pursuing of those questions lets me explore. That is to say: I'm more interested in how we can know something than what we find out. This all means I have slightly different criteria for a good resource than does a typical historian. When I'm planning a project I'm not looking for 'gaps in the literature'. Instead, I'm really only looking for 2 things:
  1. A corpus of downloadable electronic text
  2. A corpus that does not assume I want to read anything
 1) A Corpus of Electronic Text

At the moment my work is almost exclusively based on textual analysis. By that I mean I work with words rather than sounds or images or smells or physical objects. I want to know what human knowledge is contained in the symbols on pages. That means for me the best thing you can give me is a good clean set of electronic text. The Old Bailey Online does this beautifully - better than just about anyone else actually - by providing more than a hundred million words of transcription. Most important: the OBO is entirely downloadable. That means I can put it on my own computer and I can measure it, twist it around, write programs to analyse it, use other people's programs...anything I like. No one is going to threaten to sue me or press criminal charges for downloading the records, And best of all, once I have the records I don't have to read them. Because that's not the focus of what I do.

2) A Corpus That Does Not Assume I want to Read Anything

I'm certainly not one to suggest reading is obsolete, or that historians should stop going to the archives. But I'm always disheartened to see new scholarly - usually commercial - databases come online that only allow reading. I'm talking about the ones that cost an arm and a leg to university libraries, let you keyword search, but then force you to read a scanned copy of the original while hiding the electronic text layer.

I find these projects infuriating, and would rather pretend they don't exist than struggle to find a research question that's appropriate for their limited interface. The thing that bothers me most about these gated resources is that the publishers who create them are implicitly saying: we don't trust you. They don't trust us because the only thing they possess that allows them to sell their product is the electronic text. That's the part of the project that cost the most and took the longest to create. They think if that starts floating around on the Internet they won't be able to make money anymore.

The OBO is different because it's non-commercial. The OBO trusts us and encourages anyone interested to use the records to explore human knowledge in any way they see fit. For some that means sitting down and reading from digital copies of the original source. For others like me, it means downloading the entire corpus and measuring the rates of transcription errors, or of the impact of courtroom reporters on the vocabulary used in the records, or on the pace of migration in eighteenth century London.

The OBO and its team have trusted us. And from that have poured forth far more research about early modern crime in London than anyone ever could have imagined. Perhaps more research than we need. Meanwhile, researchers like myself continue to ignore the large commercial databases who lock up access to their resources, and hope intently that these people will learn from what is still the best online scholarly database I've worked with. We're starting to see steps forward from some (see the Library of Wales' Newspaper Collection for a good example), but overall there's room to improve.

Until we see a shift away from mandated reading, I'll stick to resources like the OBO. So happy birthday to the OBO and cheers to the project team for trusting us. I hope it's paid off.

Sunday, February 10, 2013

Identifying and Fixing Transcription Errors in Large Corpuses

"Underwood 11 Typewriter", by Alex Kerhead.
This is the third post in my series on the Old Bailey Online (OBO) corpus. In previous posts I looked at the impact of courtroom reporters and editors on the vocabulary used in the Old Bailey trial transcripts, and at ways of measuring the diversity of immigration in London between the 1680s and 1830s.

Since I'm dealing with a huge amount of text (51 million words, 100,000 trials), I thought I'd turn my attention to the accuracy of the transcription. For such a large corpus, the OBO is remarkably accurate. The 51 million words in the set of records between 1674 and 1834 were transcribed entirely manually by two independent typists. The transcriptions of each typist was then compared and any discrepancies were corrected by a third person. Since it is unlikely that two independent professional typists would make the same mistakes, this process known as “double rekeying” ensures the accuracy of the finished text.

But typists do make mistakes, as do we all. How often? By my best guess, about once every 4,000 words, or about 15,000-20,000 total transcription errors across 51 million words. How do I know that, and what can we do about it?

Well as you may have read in the previous posts, I ran each unique string of characters in the corpus through a series of four English language dictionaries containing roughly 80,000 words, as well as a list of 60,000 surnames known to be present in the London area by the mid-nineteenth century. Any word in neither of these lists has been put into a third list (which I've called the “unidentified list”). This unidentified list contains 43,000 unique “words” and I believe is the best place to look for transcription errors.

Not all of the words on the unidentified list are in fact errors. Many are archaic verb conjugations or spellings (catched – 1,657 uses or forraign – 1 use), compound words (shopman – 4,036 or watchhouse – 2,661), London place names (Houndsditch – 877), uncommon names that had not been marked up as such during the XML tagging process (Woolnock – 1), Latin words (paena – 1), or abbreviations (knt – 1,921) – short for “knight”, a title used by many gentlemen in the eighteenth century.

On the other hand, many of these words are clearly errors. We see mistyped letters as in “insluence” instead of “influence” or “doughter” instead of “daughter”. We also see transposed letters as in “sivler” instead of “silver”. And there are missing letters: “Wlliam” instead of “William”. Finding the difference between the real words such as “watchhouse” and the errors such as “Wlliam” amongst the 43,000 terms on the unidentified list is the real challenge.

Checking manually is impractical as these terms appear nearly 200,000 times in the corpus. Correcting every single error might not be worth the effort. However, to get an idea for the types of errors we see appearing and in what proportions, I checked every entry on the unidentified list against the image of the original scanned record during a single session of the court: January 1800. The unidentified words fell into the categories seen in Figure 1.

Figure 1: January 1800 Old Bailey Online transcription errors and the type of error.
The most surprising category here for me is the purple section, which showcases three instances that I would have categorized as typos by the transcribers, but which were actually typos in the original source. This compounds the problem because it means we must acknowledge that in some instances the error is not with the OBO team but is in fact reflecting the content of the contemporary document. From the perspective of a person searching for a particular keyword in the database they may be frustrated by the original error. On the other hand, from the perspective of those who want to be true to the original, that mistake should be preserved. I won't weigh in on that particular issue here, but it is something anyone working to correct transcription errors should consider.

With this in mind we can begin to look at the other categories, and by the looks of things approximately 40% of entries can in theory be corrected if we can figure out the intended word. Admittedly, I only looked at a single session of the trials, and this may not be representative - particularly if we consider Early Modern English, which might lead us to believe earlier trials are more likely to have archaic non-standardized spellings. If however the session from 1800 is roughly representative of a typical session then we should expect to find somewhere in the neighbourhood of 15,000-20,000 errors.

What can we do about it?

How can we automatically find and correct those errors? Given the fail-safes put in place by the double rekeying process, it's already incredibly unlikely that we will find typing errors by the transcribers. That means when we do encounter such errors it's likely only going to happen once or twice, meaning most errors are probably words that appear only once or twice in the corpus and that do not appear on either the dictionary list or the surname list.

That's not to say of course that just because a word appears in the dictionary that it is not transcribed incorrectly; however, at this stage it is much easier to identify those errors that are not recognized words. Unfortunately there are over 30,000 unique words on the unidentified list that appear only once, meaning this is still impractical to explore manually. Luckily the double rekeying means that any mistakes are more likely to be a matter of the transcriber interpreting the marks on the page differently than we might have liked them to than it is a case of fat fingers hitting the wrong key.

The early modern “long S” is the perfect such example. In the early modern era, up to about 1820, it was entirely common to find the letter S represented as what we might think looks like a lower-case “f”. This is the “suck” vs “fuck” problem that the Google N-Grams viewer runs into, as a slew of esses are interpreted as efs. When viewing the result one might be tempted to conclude people had quite a potty mouth on them in the early nineteenth century, as can be seen in Figure 2. Though not necessarily an incorrect assumption, it wouldn't be wise to make the assumption on this particular evidence.

Figure 2: Google N-Gram results for "suck" and "fuck" in the early nineteenth century

When we look through many of the words on the unidentified list it becomes clear that the Long S is a substantial problem. We find examples of the following:
  • abufes
  • afcertained
  • assaffin
  • affaulting
  • affize 
Or, the other way around:
  • assair
  • assixed
  • assluent
  • asorethought
  • artisice 
By writing a Python program that changed the letter F to an S and vise versa, I was able to check if making such a change created a word that was in fact an English word. When I did this I was pointed to several thousand possible typos. As I inspected the list further I noticed there were other common errors probably caused by the very high contrast scans of the original documents. These original documents often included missing parts of letters, difficult to read words, or little bits of dirt or smudges that made interpreting the marks more challenging.

Some of the most obvious switches were:
  • F / S 
  • I / L 
  • U / N
  • C / E
  • A / O
  • S / Z
  • V / U 
Why these particular switches appeared again and again I'm not entirely sure. Some of them are easy to understand: the lower-case C and lower-case E are easy to mix up. Especially when a fleck of dirt shows up in just the right spot on the scan. Others are a bit more difficult to explain, as with U and N, which we wouldn't expect an automated optical character recognition program to have trouble with, but which seems to have stumped the human transcribers repeatedly.

By running these seven sets of letters through the program and testing the results against the English dictionaries I was able to come up with 2,780 suggested corrections. If these are all correct, that simple switching would correct 9,503 typos in the OBO corpus. The results of these changes broken down by letter-pair can be seen in Figure 3.

Figure 3: The number of suggested corrections in the OBO corpus by switching letter pair combinations in misspelled words.
I say suggested corrections because in some cases the switch is actually wrong, or may be wrong. The English dictionaries missed "popery", a common term used to refer to Roman Catholics in the eighteenth century and has instead suggested the unlikely "papery" as an alternative. In 86 cases the switching has come up with two possible suggestions, both of which are English words, at least one of which is obviously incorrect. The unidentified word "faucy" could be "saucy" or "fancy". Turns out it's saucy, referring to the behaviour of a Peter Dayley - that naughty boy.

This switcheroo method will not solve all problems. It cannot fix transposed letters, as with sivler and silver; Levenstein distance is likely needed for that. It does nothing for missing letters as in Wlliam. But it does take us well along the path to making some rather dramatic improvements with a very reasonable amount of effort, and I would argue, could be an economical way to improve the accuracy of projects which have already been transcribed but which suffer from accuracy issues. As with all great things in life this algorithm still requires a human's careful eye, but at least it has pointed that eye in the right direction. And when you're looking at 51 million words of text, that's nine-tenths of the battle.

If you're working on a project that could use some accuracy improvements, or have explored other ways of achieving similar results, I'd be very happy to hear from you.

Thursday, January 17, 2013

Measuring the Diversity of Immigration using the Old Bailey Online 1674-1834

"Mother's Wartime Passport -1941" A. Davey
This is the second in my series of posts on the Old Bailey Online (OBO) corpus. I've downloaded all of the trial transcripts from 1674 to 1834 (find out how on the Programming Historian 2), which is about 100,000 trials and 51 million words of text. In the last post I looked at the impact of editors and scribes on the vocabulary in the Old Bailey Proceedings.

This time I thought I'd look at something a little closer to my area of expertise: immigration to London in the Early Modern era. I've used the OBO heavily in my doctoral work on Irish immigrants, but that's been focused exclusively on the years 1801-1820, immediately following the 1801 Union of Irish and British parliaments. I've yet to take a longer look at immigrants across the centuries using the OBO and I thought this would be the perfect opportunity to do so.

This time I'll be looking at the "people words" extracted from the OBO corpus. As I mentioned in the last post they were identified by extracting all of the words that appeared between a set of "persName" tags in the XML version of the transcripts. This gave me just shy of 62,000 unique strings (referred to hereafter as "words") used to represent people. That's nearly half of all unique words in the corpus. Of those 62,000 words, most (55,000) are not found in the four English language dictionaries I used to identify English words. The remaining 7,000 are words such as "green" or "woman" or "the", which are used to refer to people such as "the woman" or "John Green", but which can also be used in other contexts (the woman's green hat). Not all of these words are therefore proper names; instead, they are words that have been marked up by the OBO team as a reference to a person somewhere in the corpus.

In Figure 1 you can see the rate at which these new "person words" appeared in the corpus.

Figure 1: Total number of "person words" found in the OBO corpus to date. [expand +]

Nothing particularly exciting here. It seems like most of the person names that are also English words appear very early on. It also looks like the number of unique words used to describe people increases at a steady pace throughout the long eighteenth century. From a cursory look at the list of names, it seems evident that many of these words are surnames.

Given names (first names) on the other hand, are much less common. That's because most early modern Londoners had pretty common given names (William, John, Elizabeth, or some variation thereof). Silly names for babies are largely an invention of twenty-first century Hollywood actors and football players.

While London is home to hundreds of thousands of people in the eighteenth century, it's safe to say the number of surnames people had in London increased over time as migrants flooded in from across England and beyond carrying new names with them. New surnames therefore have a few ways to end up in the corpus:
  1. An established London family is mentioned in the record for the first time
  2. An immigrant family with a new name arrives in the area and ends up in the record
  3. Someone with a funny accent tries to say their name and it gets spelled phonetically
In the case of #1, it's entirely possible an established family (or anyone with that name) just avoided the Old Bailey for decades on end. I've managed to do so and there's no reason to expect others didn't too. However, common names shared by many people and local to Londoners should eventually show up. In fact, there's a reasonable chance they'll show up very early. We see this is in fact the case, as Smith, Wilson, Brown, White, and Allen all appear for the first time before 1680. "McCaffrey" on the other hand doesn't show up until 1834 and it's safe to say represents a name brought to London by an immigrant (either scenario #2 or #3 above).

New names arriving in the area may not be indicitive of the total number of new people who have migrated to London, but I do believe it reflects the growing diversity of immigrants arriving. Malcolm Smith and Donald MacRaild's article, "The Origins of the Irish in Northern England" shows that at least with Irish surnames, most names can be pinpointed to a particular region in Ireland. This regionality of names was still evident into the middle of the nineteenth century and will be no surprise to any genealogist who has sought out their family's past. We can see direct evidence of this regionality by mapping surnames. Great Britain Family Names allows you to see the distribution of any name in Britain in 1881. In Figure 2 you can see the distribution of "Howard" families, which clearly cluster around a few areas.

Figure 2: The distribution of "Howard" families in 1881 [expand +]
John Mannion agrees with Smith and MacRaild's conclusions about migration. In "Old World Antecedants, New World Adaptations" he argues that people tend to follow migration "channels". That is, someone from their village went before them and came home to say how great it was. Mannion was looking specifically at Irish migrants to Newfoundland and was able to show that villages who had already sent migrants to Newfoundland were vastly more likely to continue to do so than somewhere without the same history. That means the first "McCaffrey" in London was far more adventurous than the 351st. In fact, we could suggest that in many cases the first McCaffrey drew the other 350 over time by breaking the ice. I am interested in why that first McCaffrey decided to come to London, and what factors might have influenced his or her decision to do so.

The reason I think the OBO corpus is a useful set of records for monitoring this growing diversity of migrants is because I'm quite firmly convinced that the Old Bailey is the place one takes people they don't know when they've wronged you. I believe strangers (including immigrants and migrants) were much more likely to be subjected to the official justice system than were people with deep roots in the community.If a stranger wronged you, they had to be caught and punished quickly, or they might disappear. If your long-time neighbour steals your linen tablecloth, you have lots of options for how best to deal with them. You could smack him, you could set the dogs on him, you could knock on the door and demand it back. You had options, and time, because you knew he would be there tomorrow. And the next day. You don’t have that same assurance with someone you've never seen before. And that meant I believe people were more likely to seek a legal response than to choose a community resolution when dealing with strangers or those they do not know very well. In John Beattie's wonderful book, Crime and the Courts in England: 1660-1800, he appears to agree:
In the small-scale society of the village a prosecution may not have been the most effective way to deal with petty violence and theft. Demanding an apology and a promise not to repeat the offense, perhaps with some monetary or other satisfaction, may have been a more natural as well as a more effective response to such an offense, or perhaps simple revenge directly taken (Princeton: Princeton University Press, 1986, p. 8).
If I am correct in my assumption then immigrants are more likely to appear in the Old Bailey records than established members of the community (at least as a defendant), and are even more likely to do so shortly after they arrive in London as opposed to several generations later. That means that there is likely a reasonably strong connection between the date a name first appears in the OBO corpus and the date that name first appeared in the London area. It may not be a precise way to measure the arrival of new names, but I would hasard to say that in most cases it's probably accurate to within a few years or a decade at the most.

Therefore, one way to find new families with few if any connections to the locals arriving in the area is to look for the first time a given surname appears in the records. Considering the nature of the Proceedings, most names that appear in the record likely refer to people in London as opposed to strangers living far away. That's not always going to be true but for the most part it's a reasonable assumption. That means by measuring the rate at which new names appear in the OBO, we should get a reasonable if rough idea of the rate of immigration from distinct family groups over time.

To isolate surnames I've taken all of the 60,000 names present in the London area in the 1841 census and checked them against the "person words". This returned a list of just over 20,000 surnames. That means one third of all unique surnames in London at the end of the period have showed up in the Old Bailey corpus at some point or another. I'm sure I've missed a few, particularly those spelled phonetically, but this is probably a pretty good start.

You can see when each of these names first appears in the corpus in Figure 3.

Figure 3: Number of new surnames in the OBO corpus per year [expand+].
What Figure 3 shows is that the rate of new surnames arriving is actually fairly stable over the course of the eighteenth century. As mentioned in the previous post, the big dip around 1700 is caused by missing data and very short trial accounts, and we might be wise to assume that the entries around 1715 should actually be lower if we had the full set of trials as more names would appear earlier, filling in the gap. The long slow decline therefore over the course of the eighteenth century might actually be better understood as a reasonably flat line hovering around 100 new surnames per year and declining slightly towards 50 or 60. But it turns out that is not the whole story, and the clue to that is the increase in new names in the years immediately following the Napoleonic Wars c. 1815.

After the fall of Napoleon at Waterloo it seems quite clear that there's an influx of new surnames into the London area. I've got a suspicion that the cause of this influx is decommissioned soldiers and sailors who were dumped in London (or found their way there) after the war and got themselves into trouble. War collects soldiers and sailors from far and wide and brings them together. When those soldiers and sailors are no longer needed they're released to go on with their lives.

It would seem that after two decades of war enough people had been uprooted from their native regions by this process for a long enough period that they felt no inclination to go back home. Instead some of them obviously resettled in London or the growing industrial cities in the north, which seemingly offered greater opportunities or were more germane to their skills than the family farm. In fact, many people may have found themselves without a farm to go back to, since the enclosure movement had been consolidating ariable land into much larger units throughout the second half of the eighteenth century, leaving many people landless. That landlessness may have attracted them to the army or navy in the first place, and now with military life behind them they had to find something else to do with themselves. London, it would seem, was it. At least for some.

This trend of more new names after the Napoleonic War doesn't appear to be an isolated incident; it's merely the most obvious case. Instead we see similar patterns in other major wars and domestic conflicts involving the British, the results of which can be seen in Figure 4.

Figure 4: Number of new surnames in the OBO corpus per year, colour coded to highlight periods of war and peace [expand+]
Figure 4 shows the same number of new surnames per year arriving in the OBO corpus, but this time is highlighted to show some of Britain's major wars and domestic conflicts in the latter-half of the eighteenth century. The wars depicted in red are:
  1. The Seven Years War (1756-1763)
  2. The American Revolutionary War (1775-1782)
  3. The French Revolutionary Wars (1793-1802)
  4. The Napoleonic Wars (1803-1815)
While the American Revolution wasn't officially settled until 1783, it was effectively over by the end of 1782. The grey bar between 1802 and 1803 separates visually the two wars with the French.

The black bars represent years in which significant domestic conflicts took place:
  1. The Jacobite Rising of 1745 (1745-1746)
  2. The Gordon Riots (1780)
  3. The Irish Rebellion (1798)
In nearly all cases we see a decrease in the number of new names showing up shortly after a war or domestic conflict erupts. This is most evident for the Jacobite Rising of 1745. The apparent dip in migration at this point makes sense; who wants to move when there's a rebellion going on? This dip is then followed by lower than average numbers of new names until the conflict ends, at which point almost invariably the following years experience an above average result. This is evident both for domestic conflicts as well as international wars. The pattern appears again and again.

The differences between the average number of new surnames per year during war compared to the average in the five years after the end of a war are in fact statistically significant, at least for the American Revolution and the combined French Wars (paried t-test: p = 0.0418, and p = 0.0001 respectively. Significance in this case was p < 0.05). The Seven Years War does not pass the statistical t-test (p = 0.1346), However, I am confident we are seeing evidence of the same trend. While my statistical skills are rather rudimentary, I think it's worth noting that failing a t-test does not mean something is not true. Instead, it suggests the numbers alone cannot support that conclusion beyond all reasonable doubt. For me, the fact that the latter wars are so obviously following this trend strenghtens my confidence in a similar trend for the Seven Years War and we can see this in Figure 5, which shows the average number of new surnames per year during and after the three wars.

Figure 5: The average number of new surnames per year during the wars and in the five years following the wars [enlarge+]

The strength of the correlation between these conflicts and the decrease in new names, followed by an increase in peacetime suggests to me that my original assumptions about newcomers getting in trouble with the law were correct. It also suggests some interesting things about migration patterns in the eighteenth century. That is, people migrated when they felt it was safe. During times of turmoil, they stayed put and waited things out.

There are implicitly two groups of people here, so each requires its own discussion, I think. Firstly there are the sailors and soldiers. The reason we don't see these people arriving in London during wartime is perhaps obvious: they were in the employ of the state, off fighting the enemy. Gathered from across Britain and Ireland, as mentioned above, when they were decommissioned they had the opportunity to move where they liked and it would seem some chose London. This may have disproportionately included sailors who may have hoped to find work in London's booming shipping business.

The second group are families or individuals who have decided for economic reasons to move to London. Since we don't have evidence that someone with that name lived in London previously, many of them are presumably amongst the first of their stock to try their hand at London living. This in itself should not be taken lightly, as moving to early modern London without a social support network was an incredibly lonely and dangerous prospect, which is why so many migrants failed and found themselves in gaol, or starving and desperate, looking for any chance to get away. Sadly, we see many immigrants like Sarah Holmes, who claim that they "have no friend but God" as they throw themselves at the mercy of the courts.

What does all this mean? What can we learn about these arriving surnames? War and domestic conflict are not the only variables at play here, but I think it puts forth a reasonable case for the effects of war and peace on migration patterns of those moving towards London in the long eighteenth century. Returning to the idea mentioned earlier about the first McCaffrey (or the first of any family), it seems that families were only too willing to bide their time during periods of war, waiting instead for peace before making their way to a new life in London. We can see why this strategy might have been appealing. Why take a risk when the country is at war?

Unfortunately it may have been the wrong approach from an economic standpoint. According to Ball and Sunderland's "An Economic History of London, 1800-1914", the gap between real wages and cost of living peaked just after peace was called with France in 1817 (p. 95). That means people were most desperate when the government realised it actually had to start paying for the war it had waged. It seems to me likely that the two trends are actually connected. As new families arrived they may have been desperate for any work, forcing down the price of labour in London as a surplus of workers vied for jobs. It may seem counterintuitive, but these data suggest it's best to move during war rather than after.

Nevertheless, this unlikely set of criminal records has provided, I think, an interesting window into wider migration strategies across the eighteenth century. And it came about not by looking at how many people arrived, but when unique groups of people likely first emerged. London it would seem was home to an increasingly diverse population. That population continues to diversify to this day. And though I'm sure there are some Brits who might see war as a strategy for keeping the net migration in the "tens of thousands", take heed, for when peace comes, so too will the immigrants.