Can you explain this graph to me? Peer Reviewing a Visualization

"For sale: Mixing bowl set designed to please a cook".

That opening sentence contains 10 words, or "tokens" as linguists often call them. Yet either in its spoken or written form, it really only transmits 4 ideas, or what I imagine Marc Alexander would call "metaphors", which are concepts that go beyond the words but that express meaning and understanding. They allow us to think in chunks.



What?: For sale
What's for sale?: a mixing bowl set
What's it like?: designed to please
Please whom?: a cook

The same sentence represents an attempt to conjure a very measured set of thoughts in another person. I can't take credit for the sentence, but when the author wrote it down, they hoped that you, dear reader, would understand those 4 ideas in the same way as all the other readers, and as they themselves understood them. It's their attempt to control your mind temporarily by drawing upon your understanding and memories associated with those 4 ideas. We may not get all the details exactly the same. Your mixing bowl set may be blue. Mine is seafoam and has spout on each bowl to make it easier to pour your batter into the baking tin. So we likely havn't had exactly the same understanding of the sentence, but our understandings are almost certainly within the limits of what's acceptable to the author.

If we add 2 more ideas to the end of the sentence we end up with a failed conjuration:

"For sale: Mixing bowl set designed to please a cook with a round bottom for efficient beating".

Because of the misplaced modifier, there are now two ways to understand these ideas. Does the bowl have a round bottom for efficient beating, or should the cook who will enjoy the bowl be so proportioned?

Visualizations can offer the same ambiguity.

Is this an image of a rabbit, or a duck?

In this case, it's both, and it's that very ambiguity that the artist intended us to understand. Not all visualizations are intended to teach us something specific, or to so carefully conjure a series of ideas in our minds. That's wholly too modernist for some. Visualizations can be exploratory, used by researchers to come to a different understanding of their data by slicing it in lots of ways until they see something interesting. Or, as I demonstrated in an earlier post, can be a quick way to get a distant look at a large amount of data by reducing it to something easier to digest. In that sense graphing can aid the discovery process of research even before the conclusions are ready to be shared with the world.

But when it comes to visualizations for academic publication, unintentional ambiguity is something we must strive to avoid. If done well, there should only be one proper way of interpreting the visualization. It's our job to create something that can conjure specific thoughts in the reader's head based on the graph's shape, colour, size, orientation, etc. And it should go without saying that those conjured thoughts should be grounded in rigorous research.

As academics we spend so much time and care on our prose, and even our footnotes. Usually (we hope) that prose comes out lucid and if we're lucky, is enjoyable to read. One of the ways we ensure that is through peer review. The editors help us find people who are willing to take the time to read what we've written and provide constructive feedback upon it.

Yet few of us feel we have the aptitude to offer similar feedback on visualizations. We're not visual artists and so we can be forgiven for using colour in confusing ways, or for thinking a pie chart with 100 categories is a good way to express an idea. As I mentioned previously, I'm quite confident that in the present climate, unique looking or impressive visualizations will slip through peer review unchecked, lest the reviewer's lack of expertise in visualization be exposed by making a comment to the effect of "I don't under stand this graph".

Now, far be it from me to suggest we only use column graphs or line graphs, or that we do X, but not Y. I think it's fantastic that so many people out there are pushing the boundaries of what we can achieve via visualization. The folks at the Guardian Data Blog do great work on bringing data to life, and are a wonderful place for anyone seeking inspiration.

Instead, what I would suggest is that as creators of academic visualizations, we make sure our graphs are reviewed, even if our reviewers cannot or will not do so in the traditional peer review process.

The way I'd propose we do that is to show our friends and colleagues what we've made as often as we can, including during the drafting process. But it's not just about showing them. We have to ask the right questions. Let's use the graph below as a (relatively poor) example of a visualization that we might like to get feedback on. Please note that this is not a graph showing real data about the cost of grain in the 19th century. It's just an example.

Most of us likely want to ask "Do you like my graph?" or "What do you think of this?"

A more productive starting point is probably: Can you explain this graph to me? You aren't going to be there when your reader or viewer is interpreting your graph. The best way to find out what set of ideas are going to form in their mind is to ask them to explain their thought process out loud.

In this case, I had intended to show the seasonal difference in the price of grain in London and Edinburgh over a 20 year period. You may not have picked up on that, which means I need to fix something.

Don't be affraid to ask explicitly: Is there any element of this graph that you do not inherently understand? Make sure they can explain the labels on both axes (if relevant). If they don't know where you're getting those values from you may need to rethink your axis labels. You'd be forgiven for asking what the numbers on the Y-axis represent in the example. I didn't label it, so how could you know?

When you start experimenting with your visualizations, you're bound to come up with ideas you think are clear, but that just don't translate into ideas that your reader can interpret. Looking at the sample graph, I wouldn't fault you for asking what the top and bottom line of the curves represent. They're supposed to be two line graphs: one representing Edinburgh prices, and one representing London. I've shaded in the space between the lines to emphasize the size of the gap. If this is in fact two lines, then which one is Edinburgh? Which one is London? And when they overlap, how do I know which bit corresponds to which line? Do they cross, or merely meet and diverge again? I havn't made the fact that this is a line graph obvious because the lines aren't distinguishable from the shape formed by the colours.

Speaking of colour, you'll want to make sure you havn't come up with a palette that is going to make interpreting your graph difficult for someone with colour blindness. There are many different forms of colour-blindness, so it pays to run a test on your graph. You can do this online by using a "Colour Blindness Simulator" on your finished image.

Sticking with the negatives, ask your tester which element of the graph they like the least. For the sample graph, they may say they don't like the colours, or the font, or the legend. Personally, I think using --------> to represent arrows looks lazy. Everyone will have their own opinions on what's worst about your work. If you know what turns people off you can make visualizations that people like. And if they like the visualization, readers are more likely to engage with its message. With this in mind, go ahead and ask if they like your graph. Or if there are any elements of the graph that they particularly fancy.

Just as with your prose, it may take a few iterations and a number of different opinions from colleagues before a graph says to others what you think it says in your own mind. Just because you submitted a graph with your article and the peer-reviewers didn't comment on it doesn't mean you've done a good job of clearly expressing your ideas visually.

And one last question to ask, just to make sure your readers get the right message and aren't distracted: does the shape of the graph make it look like anything unrelated?

Graphs and visualizations have tremendous potential for expressing ideas in academic research, but it's not a skill we're typically taught in school. Most of us learn on the job, or emulate graphs we saw elsewhere that we found effective. Taking the time to ensure the graphs you create transmit the right ideas to your reader is good scholarship. Knowing the right questions to ask makes it that much easier to reach that result.

Questions to ask about a visualization:

1. Can you explain this graph to me?
2. Are there any elements you do not inherently understand?
3. Can you explain what each axis shows (if applicable)
4. Will people with colour blindness be able to differentiate your colour palette? (check online)
5. What do you like least about the graph?
6. Do you like the graph / a particular element of the graph?
7. Does the shape of the graph make it look like anything distracting?

The Role of Blogging in the Academic Feedback Cycle

Feedback Diversity is Good

Last year I delivered a couple of research papers on the history of crime. The first was in October at the Institute of Historical Research or the IHR as it’s known, here in London. The second was in January, on a beach in Belize. I thought I'd talk a little bit today about how those two experiences were different, how they were the same, and what place I think each holds in the future of scholarship.

Now before you start looking for tropical conferences on 18th century crime, I should qualify that the first paper was delivered to a room full of people. The second was posted on my blog while I was on vacation – and yes, sadly, I DID write about 18th century crime while gazing out over the Caribbean Sea. For some people speaking to a room and blogging are probably significantly different activities. But for me they aren’t all that dissimilar. Let me explain why by talking about what I got out of both experiences as well as what went into them.

At the IHR, I presented an hour-long paper based on three chapters from my PhD thesis. It was about two years worth of work that I had condensed down and tried to make engaging for a room full of people. For about two months before I gave the talk I didn’t do much other than scramble to get the research done, create the graphs build the powerpoint presentation, and craft the 8,000 words that I was to deliver. It was an incredible amount of work. I wore a jacket and tie, and I think I might have even gotten a haircut. Good thing because some of the most eminent crime historians in the world happened to be in town and decided to come to my talk. In all, I think there were about 50 historians in the room, most of whom knew far more about crime and the eighteenth century than I do.

The talk was followed by a really engaging discussion – at least from my perspective. I had a number of people offer suggestions for improving my argument, or on sources and archives I should visit. A couple of scholars who also write on similar topics challenged my findings – though were collegial and offered their own suggestions. Afterwards we continued onto the pub and to dinner as a group and over the course of the evening I must have heard ideas, criticisms, and praise from about 25 individuals on what I was doing.

The beach was a very different experience. The paper itself was just shy of 3,000 words, so somewhere in the 20-25 minute range if I had delivered it orally. This time my paper was based on some quick research I’d done just before Christmas. In total I’d invested a little more than a week analyzing the use of language in the Old Bailey Proceedings over a two hundred year period. It was really nothing more than an idea I'd wanted to test out, based on a conversation I'd had at the pub concerning the size of the lexicon over time.

The results I came up with were what you might call half-baked. Not that I’d been lazy, or that I didn’t know what I was talking about, or that the results were wrong. Just that I hadn’t spent weeks or months revising my methodology and my prose as I had at the IHR. Nor did do an in depth literature review. Instead it was more an activity in play. I had some sources, I had an idea, I tried it out, I wrote it up – with a reasonable amount of care – and I posted it to the world, curious to see what it thought.

The world doesn’t scare me, although many postgraduate conferences suggest it should. It’s not uncommon for these postgraduate affairs to advertise the fact that they are collegial, and a safe place to try out ideas. No senior academics are going to be on hand to put you in your place and tell you how wrong you are. I’ve never been one for intellectual safety, so I don’t see putting a half-baked idea before the world as one of risk. Rather it’s one of potential. But it’s also one of uncertainty and often loneliness.

When I posted my paper on the blog, there was no beer and pizza afterwards – though I did have a nice swim. And in the end I got one comment on the post from Ben Schmidt at Harvard who offered a suggestion for improving the methodology and the results.

On the surface it looks like the blog post was significantly less successful, since the number of comments I got were 25 at the physical presentation, and only one on the blog. But I don’t think that’s quite fair, for a couple of reasons.

Firstly, the talk at the IHR was a formal affair presenting years of research, with a moderator that gazes around the room encouraging more questions from the audience. The blog post was a way to test an idea, which is shouted into the great wilderness. That level of anonymity readers of blogs enjoy means there isn’t the same pressure to respond. But just because they don’t respond doesn’t mean they didn’t engage with the content. It’s difficult to know how many people engage with content on the Internet. I know 50 people were in the room for my seminar paper at the IHR, and I didn’t notice anyone sleeping, but even then I can’t be sure who disappeared into the recesses of their mind as I talked away.

My blog however offers statistics, and though I know not everyone who visits a blog post reads it, I do know about 600 people came to take a look. That’s about 12 times more than showed up to hear my seminar, and because a blog post is printed on the Internet rather than delivered orally, vanishing on the wind as it’s spoken, my blog readers could be anywhere in the world, and could even have been sleeping when I delivered it.

But what I think is important is not how many people read the blog post. Rather, it’s the diversity of the people who did so. The seminar at the IHR was attended almost exclusively by specialists in 18th century British history. The blog reaches a much more diverse audience who typically come through one of two channels:

• Twitter
• Digital Humanities Now.

When I post a new blog post I then post a notice on Twitter letting my followers know. If I’m lucky a few people will notice and share it with their followers on Twitter, or will write a response on their own blogs. And if I’m really lucky a group of scholars in Virginia who run a blog called Digital Humanities Now, which post the best blog posts of the day related to digital humanities, will tell their audience about my post, sending even more people. That’s basically what happened in the case of my Belizean blog post. I published it to the blog, told Twitter, was re-tweeted by a few people, and was showcased on the Digital Humanities Now blog.

That meant my audience included a large number of digital humanists who work in a wide range of academic disciplines including linguistics, computational analysis, literary studies, and history. I think it’s fair to say most of that audience doesn’t care about 18th century British history. However, they do share an interest in the methodology I used to work with the sources. One of those digital humanists, Ben Schmidt, posted the comment that helped me refine my methodology and come up with even stronger results.

No one in the IHR seminar was going to give me that type of feedback because that’s not the type of expertise they have. Instead they focused on the details related to the history of crime or on the records they knew of in the archives. So by seeking out a different audience through the blog, I was able to get interdisciplinary feedback on my work.

History seminars are extraordinarily valuable, particularly for early career scholars like myself. The level of intimacy you get in that type of environment is unparalleled. But they’re a bit like poorly designed focus groups. If you want to take the pulse of the nation on welfare reform or Euroskepticism, you don’t want a room full of Horse and Hound subscribers. You need the diversity of a few Daily Mail readers thrown in the mix, who see the world from a slightly different angle.

And I think the blog and twitter provide that diversity for me. In my case, my blog attracts a lot of digital humanists, but blogs aren’t just a way to get feedback from digital humanists. I posted another blog post a few weeks later on the same research material, this time focused on using criminal records to measure immigration. I again got one comment, but this time it was from Tim Hitchcock, a historian of 18th century Britain, who offered a historical interpretation that might explain what I had found. Tim’s expertise with the provenance of the records meant he knew things about the sources I didn’t.

I posted a third blog post again on a slightly different topic, and received different types of comments again from linguists, computer scientists, and Sharon Howard, the project manager from the Old Bailey Online project. With three blog posts and roughly the same number of words as my seminar paper, I’d engaged a number of different types of people from all over the world with very different sets of expertise, and different types of feedback than I could ever expect to get from a room full of crime historians.

Which experience was more valuable? The seminar or the blog posts? For me, I don’t think there’s much that can compare with a room full of world experts devoting their combined experience to listening and critiquing years of your hard work. I also don’t think you can beat the type of connections that can only be made in a face-to-face meeting at the pub, or over pizza with people who share your interests. But I also don’t think we should sniff at a model that allowed me to test 3 ideas in an informal setting, get a broad range of feedback from interdisciplinary experts all over the world, and all without costing anyone a penny.

I’ve taken on board all of the feedback I’ve received from these two papers. My PhD thesis is stronger for having delivered the seminar paper, and I’ve decided to pursue the ideas expressed in my blog more formally as a future research project. So these papers were both valuable in their own right, and I think I’m a better historian for having delivered them.

This is the text of my talk at 'Our Criminal Past: Digitisation, Social Media, and Crime History' held at the London Metropolitan Archives, 17 May 2013. With thanks to Heather Shore for inviting me to speak.

Programming Historian 2 Lessons I'd Like to See

I've been actively part of the Programming Historian 2 team for the past two years and I've been really pleased to see so many people using and learning from the site, including a number of university courses. I learned to write Python code from the original Programming Historian, and I still regularly reference skills and techniques found in the lessons in my day-to-day research.

My role as an editor of the project means I help guide lessons contributed by others through peer review and editing. I'm also always looking around the blogosphere for people working on cool new techniques or writing guides of their own that I think would be useful for practicing historians. For the most part this is a passive process. I sit, I wait, and I watch. But every once in a while I come across something I'd really like to see. So rather than wait, I thought I'd post my personal wish list of Programming Historian 2 lessons I'd like you to write for all of us.

In no particular order:

• How do you turn a spreadsheet into a database and write custom queries? The jump from an Excel spreadsheet which you can see to a MySQL or sqlite3 database that you can't see is not an easy one. A lesson on making this leap would be well received and widely used I would imagine.
• What the heck do you do with topic models? The entire digital humanities world seems fixated on topic models these days. Our most popular lesson by far is a tutorial on Getting Started with Topic Modeling and MALLET. But what are the cool things we can do once we HAVE generated topic models? What can we know? How do we use it responsibly? How do I interpret all these numbers and topics?
• What can we do with our sources once they've been downloaded? I see so many people using programming to curate sources, but far fewer people asking historical questions of their sources using programming. What are some of the ways we can actually answer questions about the past with programming?

I'd be very happy to hear from anyone who'd like to take on these challenges and create a Programming Historian 2 lesson, or from anyone with an idea of their own they think others could benefit from. Check out our submission guidelines and be in touch.

Providing A Familiar Assessment Process for Digital Projects

"Trust Me" by Anderson Mancini

A few weeks ago in my Early Modern British History tutorial the students read a book review that had been printed in a scholarly journal. I asked the students, “Why are we reading this? What’s the point of this piece of writing?”

My cynical self tried not to let on what I was thinking: wasn’t it written so some PhD student could score a free book and put another line on his C.V.? My students thankfully didn’t see it that way.

Historical writing generally comes in one of a handful of tried-and-tested forms. In the course I teach we usually read and discuss articles published in peer reviewed journals. The students are now used to the goals of that particular mode of writing. They know it contains a historian’s contribution to a particular scholarly discussion, which is fairly narrowly defined and tends to correlate rather strongly to the topic we’re discussing that week in lecture.

They understand edited collections contain a series of articles by different authors on a theme defined by the editor. They know books (or monographs as historians like to call them) cover broader topics than a journal article, but they expect to find chapters in those monographs that rival the scope of an article. And for a first year undergraduate, that about covers the known forms of printed historical research.

Scholarly book reviews are generally a fairly new beast for the students. My question about the value of book reviews was of the type I knew I would not have been able to answer as a first year undergraduate, but I’m blessed with very clever students who had all manner of ideas. My favourite comment was that a book review:

provides another perspective on the strengths and weaknesses of the work, which are important to keep in mind when coming to our own conclusions about the author’s arguments.

I believe that idea holds the key for digital humanities scholars, many of whom feel their colleagues have been unwilling to award due credit for digital projects. However, the problem is not necessarily an unwillingness; the problem is that most people aren’t qualified to assess digital humanities work, so they just don’t. Forcing someone to assess something without the skills to do so is not the answer. I think most of us would prefer that a review of our work came from someone with the background to assess it fairly.

Scholarly reviews are a great platform for that assessment process for two reasons. Firstly, they take a form that nearly all humanities scholars already understand. Secondly, they allow the digital humanities community to self-define the major projects so that non-digital colleagues don’t have to.

Familiar Clothes

Like my students, most scholars can visualize where a review fits amongst the various scholarly apparatus floating around in the intellectual space in which we share and critique each other’s work. They understand that a quality review in a reputable journal represents an arms-length critical evaluation of the work by someone with the skills to make the assessment. Reviews certainly are not a perfect model. There is clearly a risk to any creator or team that they will get an unfairly negative treatment by someone who perhaps is not as qualified to write the review as they may think. But this is the same risk authors of monographs face, and to me, the benefits far outweigh the risks.

The world of publishing is changing, but there’s something to be said for making something look familiar. The Journal of Digital Humanities has taken this approach by experimenting with new ways of finding content that is important to the scholarly development of digital humanities – although I imagine that process of identifying content would look rather shocking to a typical historian or English lit. professor. (If you’d like to learn more about that project read their “How to Contribute” page.) However, by wrapping it up in the clothes of a traditional peer reviewed journal, the editors get to experiment, the authors get credit for their contributions, and the readers are drawn to some of the best ideas that appear online. Everyone wins.

For me, the Journal of Digital Humanities has represented a great success as a project that has been able to forge ahead, while acknowledging that sometimes it’s a good idea to change things gradually. In that project, the team decided to make sure they held onto some familiar concepts. They registered an ISSN number (2165-6673). They release the journal in “Volumes” with issue “Numbers” to ensure any citations look like a typical humanist expects them to. Neither of these is technically necessary, but they certainly do not hurt. And by having them those who disapprove of any changes to the mechanisms for scholarly communication have a few less things about which to complain and are instead forced to discuss the actual changes such as the open submission system rather than a lack of an ISSN number.

The Major Contributions

Digital humanities projects – particularly the digital ones – have taken an entirely different path to the familiar clothes approach. They look completely different. Many people still aren’t clear where a scholarly database or a digital tool fits amongst the other scholarly apparatus. Is it akin to a monograph? Is it a chapter in an edited collection? Is it an edited collection? Just how much work was it after all?

By changing the clothes completely digital humanists have made it difficult for non-digital colleagues to assess the work because they don’t necessarily understand how it was constructed or the intellectual considerations that went into building it. I believe we can change that by beginning to write and publish reviews, because within the traditional humanities framework reviews are reserved for major contributions. In historical journals it is generally only books that warrant a review, and I would suggest the book-equivalent amongst digital projects could benefit from the same.

If the community of digital humanists own this process, then it’s the community that gets to decide what the major contributions to the field are, by putting them up for review. And that process means non-digital colleagues not only have an arms-length evaluation of the merits and shortcomings of digital work from a reputable expert, but this review can then become the basis for digital scholars to go to their departments and say: this is what I achieved. And with that form of evidence, I think we’ll be one step closer to slotting digital projects into that mental framework of scholarly contributions.

Let’s Get Started!

Ok, ok. But before we get going, perhaps we should sit back and think for a few moments to ensure the reviews follow that model of familiar clothes. For these reviews to hold weight with non-digital faculty and to play an important role ensuring digital work receives the credit it deserves, reviews need to withstand a certain level of scrutiny. But they cannot merely look like reviews. They have to be rigorous, reliable, and arms length.

The first question then is where should the reviews appear? Many digital humanists would likely suggest blogs are the answer. After all, they are cheap and they are efficient publishing platforms. But I believe that would be an error in judgment. Firstly, they fail the familiar clothes principle. Some academics are still untrusting of blogs and whether we collectively agree with that skepticism or not the goal of reviews is to bridge that gap in trust, not entrench it; therefore we must make reviews as easy to swallow as possible for those we hope to engage.

Secondly, blogs are not arms length because the author controls all levels of the review’s distribution. A few years ago Amazon’s review system showed us why that can be a problem when the wife of an academic wrote poisoned pen reviews of his competitor’s books. If project reviews are to have an important role in assessment and credit then it is important that those reading them can be confident that the review was not written by a friend or colleague (or enemy) of a project leader who may have had an agenda.

For me, the natural home of project reviews is in a scholarly journal. Not that many dedicated scholarly journals out there focus on the digital humanities and from what I can tell only one (Literary and Linguistics Computing) currently publishes reviews - at the moment limited to reviews of books about digital humanities. This suggests to me that there is an opportunity for any of these journals to take the lead on such an initiative and adopt a project reviews section. I would particularly urge the Journal of Digital Humanities to take on this challenge not only as the new kid on the block, but as a group committed to trying new things in publishing and championing those who do things just a little bit differently.

By having reviews centred in a scholarly journal they gain not only a trusted distribution system – again familiar clothes – but they also fall under the control of an impartial editor. This is important for the same reasons that a blog is not the right venue for such work; by putting the system of soliciting reviews in the hands of an arms-length editor, a further check is placed upon the quality of reviews. This will not only reduce the number of friends-helping-friends reviews, but also ensures that the best work gets reviewed, rather than the work of people with large professional and personal networks.

By providing a familiar form of critical assessment for non-digital colleagues, digital humanists can collectively define what makes their work good, innovative, and scholarly. Because it is they who are best positioned to do so. In a few years I hope to be sitting with a new class, asking them: “Why are we reading this digital project review? What’s the point of this piece of writing?”

Then as now, the answer is not because a PhD student wanted to add a line to his C.V. Rather, it is because the digital humanities community needed a mode of assessing their work in a way that reflected the unique challenges and assets of such projects so that their colleagues had the tools available to critique some great if oftentimes overlooked projects.

I’ll even volunteer to write the first review. You know where to find me.

How Best to Approach Academic Journals

Are you a humanist trying to get published in an academic journal? Trying to decide if it's worth the time and effort? Do you know someone who fits this category?

The round table discussion (audio) at the American Society for Environmental History, recorded in Talahassee this past month, offers some excellent tips for grad students and post docs. The round table was made up of editors from several academic journals and offers insights into what they like, what they hate, what to do and what not to do. A must listen for any grad student and a great recommendation for any prof to give to their students. Don't let the environmental history topic scare you away; it doesn't factor into the equation at all. This recording is useful for anyone in the humanities; including profs with poor writing skills.

Open Peer Review

I came across an open peer review request the other day. Ara Basmadjian, a UWO digital history student posted a draft essay proposal on his blog, asking his colleagues for constructive criticism.

Not only is this brave and a nice change from the defensive, secretive nobody look at my paper attitude that so many students have, but, if it works, it might prove to be a good first step into an open peer review system that helps create good, solid scholarship in a timely manner, without the need for the bottleneck of journals. I hope it works.