Can We Reconstruct a Text from a Wordcloud?

We’ve all seen Word Clouds. Many of us have even wondered if they’re of any value. I have used word clouds in the past; I find them useful in presentations when I want to highlight the relative importance of certain words over others. For example, I often use this word cloud to the left, to show the most common Irish surnames in the London area during the early 19^th century. I hope my listeners will note that Murphy or Sullivan is more common than Burke or Foley, without me having to take the time to explain the connection between word-size and significance.

I’ve also used word clouds in analysis. In a previous post I discussed how I was able to use the below word cloud to show the relative frequency of topics found in the Gentleman’s Magazine between 1800 and 1820, which allowed me to get a pretty good idea of what the gentry and the middle class were interested in during that period.

I think both of those uses for word clouds have been productive. They’ve allowed me to transmit ideas, and formulate my own thoughts on a set of data in an effective manner. But I began to think about other uses, and I began to wonder about the process of getting back to the original data. Word clouds take the individual words (tokens) out of context. As I mentioned in my last post, we think in metaphors, or ideas. Not in words. That means a word cloud reduces a single idea such as “green bowl” into two tokens “green” and “bowl”. It then combines the word “green” into a single graphic based on how often it appears in the text. The program does not take into consideration the fact that “green” as it refers to a bowl is entirely different than Mr. Green or Green Park. An article about Mr. Green’s picnic in Green Park with his favourite green bowl might give you a skewed idea about the importance of the word green, here representing three completely different ideas, and in all three cases simply acting as modifiers to more important concepts (a man, a park, and a bowl).

Just for fun, I decided to do a test. I asked 4 colleagues, all experts on the criminal trial transcripts of the Old Bailey Online, to look at word cloud of a trial. Each person was asked to describe what key information they could tell me about the crime. I was interested in knowing if they could tell me the who, what, when, where, why type details, and if they could reconstruct the basic building blocks from the prevalence of certain keywords. In the spirit of exploration, I played along as well and offered my own interpretation.

The word cloud was created at random by my wife without my knowledge of the trial. All I knew (and all the participants knew) was that the trial took place between 1801 and 1820, and was between 2,000 and 3,000 words long. The word cloud was limited to 75 unique words and common English words were removed. The resultant visualization looks like this:

  

I have colour-coded my assessment (blue) for aspects I got correct and (red) for the bits I got wrong. What struck me immediately was that this was a case involving theft. That’s a safe bet anyways, since about 50% of Old Bailey trials during this period were theft cases. It was the large number of nouns that led me to this conclusion. I know that trial transcripts always list the items that were stolen, and the testimony in the trial almost always discusses the various objects repeatedly as several witnesses are called to give their account of what transpired. In this case, I’d assume there was a large quantity of alcohol that went missing, ranging from red wine, to port, to gin – stored in bottles, measured by gallons, and in at least one case: a cask.

At the time it was stolen the booze was being stored in a cellar before it was transferred to a cart that was being driven by a horse – a mare to be specific. Why Restoration actress Nell Gwyn appears in the set of words, I have no idea since she died over a century before this era, unless the name is a coincidence or perhaps refers to the name of a pub that lost its liquor.

There were a large number of people involved with giving evidence against the defendant including Messers Hutt, Wells, Powell, Wood and Bagnigge, as well as possibly a Mr. Limbrick, and definitely someone named Hart – though again that may be the name of the pub. One of those men is likely the watchman and another an officer. Based on what I know about Old Bailey trials, this suggests there were a lot of witnesses, meaning the prosecutor was concerned that his case may not have succeeded.

The alcohol heist took place in the morning (and was perhaps discovered the following night), and the goods were then transferred down either Maiden-lane or City-road. Given the volume of goods stolen and the fact that death appears in the list suggests our defendant was found guilty and sentenced to death.

My conclusion: a pub named either Hart or Nell Gwyn located on City-road or Maiden-lane was robbed of a large volume of alcohol by a solo male defendant, who was found guilty and sentenced to death.

Analysis

You can compare my assessment with the full trial transcript. It turns out I wasn’t that far off. There had been three defendants, but they were found guilty and sentenced to death. It had been a large alcohol theft. I wasn’t able to accurately pick out the fact that Messers Wood, Powell, and Hart were the defendants, meaning the who aspect of the challenge had completely eluded me. I also didn’t recognize Bagnigge Wells, which was the location of the crime, not someone’s name. Nell Gwyn in this case was the pub that had its door pried open to reach the alcohol.

Participant 1

“Powell and Hutt were found guilty of breaking into the wine cellar of the Nell Gywnn inn in Bagnigge Wells (I know about this place) They were accused of stealing two gallons of wine, casks of gin and bottles of port from the cellar  belonging to Mr Hart . A Watchmen Mr Wood on his round at 1 o'clock saw a broken iron (lock)on the wine cellar door  and saw two men drive off in a horse and cart down Maiden lane towards City Road and immediately called for an officer. The men were stopped by an officer who examined the cart and found the cask of wine, gin and port belonging to Mr Hart and arrested them. They received the death penalty.”

Participate 2

“I would guess it involves stealing a hamper of goods including a bottle of wine from the back of a cart.  My suspicion is that the defendants were two women, and that the servant of the person who owned the cart/hamper discovered the theft and was called in evidence, though the actual owner wasn't.  The cart was on its way to or from northwest London to banigge wells, for a social occasion and some visiting, it happened at night (suggesting they were returning home) and there was a runner or 'officer' involved in the arrest.”

Analysis

In these examples, the participants tried to be very specific about the details of the transcript, and in doing so were incorrect more often than not. The basics of the case, including the location of the crime, was however, correct for participant 1. This would suggest that expert readers are able to get some of the basics – though by no means all. However, that expertise does little to bring forth the specifics of the trial.

Participant 3

“This trial seems to have a richer vocabulary than most.  It looks like a theft case from a wine cellar (wine, port, cask, gin, gallons, hampers, bottles, etc. suggest as much), presumably at Bagnigge Wells; actually the prominence of the word cart, and also the word horse, suggest the material might have been in the process or coming or going there, perhaps parked on the City Road (or Maiden Lane). Went suggests action.  Some force was used: broken, crow, saw, which suggests that the cart was broken into.   There is a certain amount of vocabulary indicating how the culprit might have been apprehended: officer, stopped, examined, watchman, observed, charge—this suggests that officers were used to apprehend the defendant. There are some names: Powell, Gwyn, Hart, Mr, Nell, William.  The most frequently mentioned, Hart, is presumably the victim.  Timing is important: o'clock, morning, night—the prominence of night suggests that that is when the crime took place, with the suspect arrested in the morning?  Numbers indicate either the number of items stolen or the time of day (one or two in the morning).”

Analysis

This one came out surprisingly accurate. The participant hadn’t recognized Nell Gwyn as the name of the actress. As in my own case, it proved impossible to figure out who was the defendant and who was the victim. However, the details of the crime and the process of apprehending the defendant is almost bang on. This participant didn’t try to reconstruct the narrative in the same way as Participant 1 or 2, and thus avoided many of the pitfalls. However, there has been no guess at the verdict, and while the basics of the trial are here, the richness of what actually is recorded in the transcript is nearly entirely lost.

Participant 4

“Geographical location

Wine Cellar –property crime

Bagnigge Wells – recognise this as the location of a rather seedy spa/pleasure

grounds on the outskirts of Clerkenwell towards Kings Cross.

City Road – not too far away runs from the City to Islington

Maiden Lane – small street that runs parallel to Covent Garden Piazza on the

south-side although there might be other Maiden Lanes

Time –mention of time o’clock and watchman (usually worked only at night)

and night although mentions morning (in the or the next)

Crime probably burglary because of the mention of the relevance of time

together with watchman

Stolen Goods: bottles of red wine, port, cask, gin from wine cellar

Broken into cellar with an iron bar, maybe crow bar

Probably took it away or hid it in a cart, certainly it is central to the plot and

perhaps discovery of the stolen goods. There may well have been a significant

amount of drink gallons, casks and bottles? Hampers – might have been used

to transport/hide the goods

Arrested – yes (examined) and officer (probably from one of the Police Courts

Observed - spotted

Names Hart – think this is a personal name rather than pub sign and possibly

Wood as it’s mentioned frequently but not as much as the things I think were

actually stolen

Gwyn –forename Welsh? Ha – just spotted nell (same size) and I know that

Nell Gywn was supposed to have lived at Banigge Wells. John, William, think

Hutt is a personal name (not a shed)

Death –punishment (so guilty)”

Analysis

This one was particularly interesting, because the participant included their thought process as it related to the various words on the visualization. The fact that it was clearly a theft of goods, and that time was mentioned, lead this person to conclude we were dealing with a burglary, which was correct.  The conclusion that Hart was the name of a person rather than a pub was correct, but equally could have been wrong. While Nell Gwyn may have lived at Bagnigge Wells, that wasn’t really relevant to this case.

Conclusions

Can an expert on a historical source reconstruct the details of that source from a word cloud? It would seem the answer is: sort of. Of the five participants, two (#3 and #4) did an incredibly good job of getting some of the details. These people were able to reason out what the words meant by drawing upon their experience with the ways certain types of words were used in criminal trial transcripts. However, in both cases they were light on the details, and it would appear decided not to guess on elements of the crime that they couldn’t be confident in. That is, they spoke confidently when they were confident but otherwise stayed silent.

I think my analysis fell in the middle. I got lots of bits right, but I was also wrong just about as often. I was disappointed that I couldn’t pick out the roles people played in the trial from the word cloud. There was no way to guess who was the defendant, who was the victim, and without recognizing the names, who were the officers. I wasn’t even able to guess how many defendants were on trial. On the other hand, I did guess the type of crime, the verdict, and a few details and circumstances surrounding the arrest. Having said that, I can’t say I was confident in all of my conclusions and was guessing.

And finally, two of the participants were way off (#1 and #2). These two attempted to reconstruct the narrative of what had happened, providing a level of detail that involved a good deal of guesswork.

What does this mean? I think it shows that when faced with a simplified visualization such as a word cloud, the process of getting back to the original is fraught with a level of guesswork. However, an expert in the source material can, with reasonable accuracy, reconstruct some of the more basic details of what’s going on.

How do we move forward? Well, as I pointed out earlier in this article, I think the secret is in moving away from the idea that tokens transmit ideas. Ideas and metaphors transmit ideas, and it would be far more useful to have an idea or concept cloud than one that focuses on individual tokens. But I also think it’s time that those ideas were linked back to the original data points, so that people interpreting the word clouds can test their assumptions. We are ready to see the distance between the underlying data and the visualization contracted. We’re ready to see the proof embedded in the graph. And I hope we continue to see a development in this trend.

Thanks to my participants, Janice Turner, Bob Shoemaker, Louise Falcini, and Tim Hitchcock.

The Two Data Visualization Skills Historians Lack

Four Stages of Data Visualization, by Tobias Sturt at the Guardian

To create a great data visualization you need four skills. You don't have all of them. That was the message of Tobias Sturt and Adam Frost of the Guardian at a recent masterclass on data-vis held in London. The pair both work for the newspaper's "Digital Agency", a for-hire data visualization consultancy company run by the paper. Frost's role is to work with the data and find the story. Sturt determines the most appropriate chart style and the design that will help the reader interpret and engage with that data. That doesn't mean Frost knows nothing about the strengths and weaknesses of certain types of charts, or that Sturt runs away shrieking when he sees a spreadsheet. It does mean they each bring strengths to the table which allow them to create engaging visualizations that are true to the underlying data. That's what good collaborations achieve and anyone that's seen the outputs of the Guardian's team knows they're an incredibly talented group.

Where do historians fit in? I'd say most of us are like Frost. We can handle our data, be it numbers or words, or images, or material culture. We interpret what we see. And we find the story that adds the context to that data. According to Frost and Sturt, these two steps bring the integrity and meaning to the audience. But when it comes to data, words aren't always the best way to present them, and raw data in tabular form (as we've all seen so many times in journal articles) is what Frost refers to as "clarity without persuasion".

That means we need to find and work with the Tobias Sturts of the world. We need to collaborate with those with an eye for colour and form, who can take numbers and turn them into understanding. Without people like Sturt, the above visualization would be nothing more than it's raw data:

1. Data
2. Story
3. Chart
4. Design

But we get so much more from his visual representation of those four ideas, and few of us have the skills to compete with the creative power of designers. They know things we don't. They know how colours make us feel or what they imply. They know you're more likely to believe a statement written in Baskerville than Comic Sans font. They understand how your eye scans a page, what it's looking for, and how the location of certain elements on the page or the size of those elements change the way we interpret them. They know what we don't.

The question is: where are these people and do they want to work with us?

I'm afraid I'll have to disappoint you and admit: I don't know. Sturt is likely out of the price range for most academic historians. His clients tend to be corporations looking to develop their brands, or large non-profits trying to reach huge audiences. But we all know there are artists out there looking for work. It seems to me the issue may be that we havn't yet realized we need each other, so we havn't yet had to build those relationships. We could say those artists have failed to market themselves to us, but unless we let them know we're interested, we can hardly blame them for ignoring us.

So maybe the best way is to ask. Artists: how do we find you? What should we be looking for in an artist? And what would you look for in us?

Where are the Digital Humanists?

This year has brought a lot of discussion about just what the digital humanities are. Self-described digital humanists have blogged on what they think makes a digital humanist and what doesn't. As with all definitions of identity, everyone has a slightly different opinion.

I won't rehash those discussions here. Instead, what I'd like to do is ask where the digital humanities are by looking at the participants of two major crowd-sourced activities from the past 12 months: the Day in the Life of the Digital Humanities (Day of DH) [March 18, 2011], and Hacking the Academy (Hackacad) [May 21-28, 2010]. If you are unfamiliar with these projects, a brief background will help.

Day of DH asks self-described digital humanists to blog or otherwise record their activities on March 18, 2011. The project is based at the University of Alberta in Edmonton. Participants are pre-screened before being given a blog, though it's not clear who if anyone was rejected. Selection is self-applied and because there is no physical meeeting, participation is open worldwide.

Hackacad was a crowd-sourced book written in a week. The project was initiated by the Center for History & New Media at George Mason University in Virginia. Anyone could contribute a chapter (blog post), as long as it was submitted by the end of the week. Though this was not targetted specifically at Digital Humanists (more technologically literate public historians and researchers, I think), it did attract quite a few DH-type articles. Like Day of DH, there was no requirement to physically be somewhere so it was open to participants worldwide.

Last year, I compiled data on where the Hackacad contributors were writing from. Meg Stewart compiled my data in an easy-to-read map. What I argued at the time, was that contributors were skewed towards those who lived near, or had likely met or worked with the organizers and was not truely representative of the "crowd" of possible contributors. This isn't meant to be a criticism of the organizers who pulled off an impressive feat with their crowdsource experiment. Rather, it shows us that when crowdsourcing anything, one must ask who's in the crowd and who got left out.

So to take a closer look at that, I have made some maps. The first shows where the Hackacad contributors came from, the second shows where the Day of DH contributors live, and the third shows the people who contributed to both projects. (The tags won't lead you to anyone's house; just the city in which their university or institution is located). You can view each map in Google Maps and explore the data by clicking on one of the images below.

Hackacad 2010 Contributors

Day of DH 2011 Contributors

Hackacad 2010 + Day of DH 2011 Contributors

What this shows us in both cases is that the personal and professional networks of the organizers influence who participates. If one only looked at the Hackacad map, one would be forced to conclude that digital humanists - broadly defined - lived almost exclusively in the North Eastern United States. There are pockets of Digital Humanists in southern California, and a few solo scholars plugging away through much of the western world, presumably in great loneliness. As English was the language of the book, the great empty continents are perhaps not too surprising, but they are telling in terms of who belongs to this great academic discussion.

The Day of DH map tells a different story. That is: Digital Humanities is not so focused on the US. There are strong communities in Europe, the United Kingdom, Canada, and Mexico. The trickle of Asian-based DH people suggests there's obviously a lot more, but perhaps few have the command of the English language needed to participate more fully in this English-language discussion.

The third map surprised me. Only a handful of people participated in both events. Though, I did recognize most of the names from either Twitter or active blogs, suggesting that as with most things, there is a small vocal core to every community.

So where are the digital humanities?

Well, it depends. The top cities, according to Hackacad are:

• Fairfax Virginia (project home): 12


• New York: 11


• Boston: 8


• Los Angeles: 8


• . . .


• Edmonton: 0

According to the Day of DH:

• London: 19


• Edmonton (project home): 14


• Oxford: 10


• San Francisco / Bay Area: 10


• Los Angeles: 8


• . . .


• Fairfax VA: 3

What does this all mean? Well I think for one, it shows that in any crowdsourced activity, those close to the organizers will be overrepresented in the results. Edmonton is not - to my knowledge - a Mecca of digital scholarship, though I will say they do some fantastic work there. Instead, Edmonton is where the Day of DH exists out loud. The organizers live and work there and have almost certainly discussed the project with staff and students. This encourages participation in a way that goes outside of the crowdsource and instead is based on face to face interaction.

Overall, I think the maps, if combined, give us a good idea of where the digital humanities are. Certain areas are surely over represented (London, I think fits this category). Some are vastly under-represented (Australia & NZ). Perhaps most interesting is that the top schools in terms of reputation, according to Times Higher Education, often don't appear at all on these maps.

The top 10 schools by reputation and the number of Digital Humanists based on these two events are as follows:

• Harvard: 2


• MIT: 0


• Cambridge: 1


• UCAL - Berkley: 0


• Stanford: 6


• Oxford: 11


• Princeton: 0


• University of Tokyo: 0


• Yale: 0


• California Inistutute of Technology: 0

It would seem that these schools havn't built their reputations on innovative humanities research!

We can also conclude that the self-identified digital humanists are primarily: urban, working in Western countries, at major research focused universities. But hey, we probably already knew that, huh?

#apiworkshop Reflection: free the data

I recently attended Bill Turkel's Workshop on APIs for the Digital Humanities held in Toronto and had the pleasure of coming face to face with many of the people who have created the historical data repositories I have used so enthusiastically.

What I came away with was an even stronger conviction that data in humanities repositories should be completely liberated. By that, I mean given away in their entirety.

The mass digitizations that have occurred recently have provided a great first step for researchers. I no longer need to fly to London and sit in a dark room sifting through boxes to look at much of the material I seek. But, I'm still - generally - unable to do with it what I like.

Many repositories contain scanned documents which have been OCR'd so that they are full text searchable, but that OCR is not shared with the end user, rendering the entire thing useless for those wanting to do a text analysis or mash up the data with another repository's content.

Most databases require the user to trust the site's search mechanisms to query the entire database an return all relevant results. If I'm doing a study, I'd prefer to do so with all the material available. Without access to the entire database on my hard drive, I have no way of verifying that the search has returned what I sought.

Many of those at the workshop who administered repositories were willing and eager to email their data at the drop of a hat, but that is not yet the norm. Most of my past requests for data have been completely ignored. When it comes to scholarly data, possessiveness leads to obscurity.

As humanists become increasingly confident programmers, many will define research projects based on the accessibility of the sources. Those who are giving their data away will end up cited in books and journal articles. Those desperate to maintain control will get passed by. If someone asks you for your data, think of it as a compliment of your work, then say yes.