Providing A Familiar Assessment Process for Digital Projects

"Trust Me" by Anderson Mancini

A few weeks ago in my Early Modern British History tutorial the students read a book review that had been printed in a scholarly journal. I asked the students, “Why are we reading this? What’s the point of this piece of writing?”

My cynical self tried not to let on what I was thinking: wasn’t it written so some PhD student could score a free book and put another line on his C.V.? My students thankfully didn’t see it that way.

Historical writing generally comes in one of a handful of tried-and-tested forms. In the course I teach we usually read and discuss articles published in peer reviewed journals. The students are now used to the goals of that particular mode of writing. They know it contains a historian’s contribution to a particular scholarly discussion, which is fairly narrowly defined and tends to correlate rather strongly to the topic we’re discussing that week in lecture.

They understand edited collections contain a series of articles by different authors on a theme defined by the editor. They know books (or monographs as historians like to call them) cover broader topics than a journal article, but they expect to find chapters in those monographs that rival the scope of an article. And for a first year undergraduate, that about covers the known forms of printed historical research.

Scholarly book reviews are generally a fairly new beast for the students. My question about the value of book reviews was of the type I knew I would not have been able to answer as a first year undergraduate, but I’m blessed with very clever students who had all manner of ideas. My favourite comment was that a book review:

provides another perspective on the strengths and weaknesses of the work, which are important to keep in mind when coming to our own conclusions about the author’s arguments.

I believe that idea holds the key for digital humanities scholars, many of whom feel their colleagues have been unwilling to award due credit for digital projects. However, the problem is not necessarily an unwillingness; the problem is that most people aren’t qualified to assess digital humanities work, so they just don’t. Forcing someone to assess something without the skills to do so is not the answer. I think most of us would prefer that a review of our work came from someone with the background to assess it fairly.

Scholarly reviews are a great platform for that assessment process for two reasons. Firstly, they take a form that nearly all humanities scholars already understand. Secondly, they allow the digital humanities community to self-define the major projects so that non-digital colleagues don’t have to.

Familiar Clothes

Like my students, most scholars can visualize where a review fits amongst the various scholarly apparatus floating around in the intellectual space in which we share and critique each other’s work. They understand that a quality review in a reputable journal represents an arms-length critical evaluation of the work by someone with the skills to make the assessment. Reviews certainly are not a perfect model. There is clearly a risk to any creator or team that they will get an unfairly negative treatment by someone who perhaps is not as qualified to write the review as they may think. But this is the same risk authors of monographs face, and to me, the benefits far outweigh the risks.

The world of publishing is changing, but there’s something to be said for making something look familiar. The Journal of Digital Humanities has taken this approach by experimenting with new ways of finding content that is important to the scholarly development of digital humanities – although I imagine that process of identifying content would look rather shocking to a typical historian or English lit. professor. (If you’d like to learn more about that project read their “How to Contribute” page.) However, by wrapping it up in the clothes of a traditional peer reviewed journal, the editors get to experiment, the authors get credit for their contributions, and the readers are drawn to some of the best ideas that appear online. Everyone wins.

For me, the Journal of Digital Humanities has represented a great success as a project that has been able to forge ahead, while acknowledging that sometimes it’s a good idea to change things gradually. In that project, the team decided to make sure they held onto some familiar concepts. They registered an ISSN number (2165-6673). They release the journal in “Volumes” with issue “Numbers” to ensure any citations look like a typical humanist expects them to. Neither of these is technically necessary, but they certainly do not hurt. And by having them those who disapprove of any changes to the mechanisms for scholarly communication have a few less things about which to complain and are instead forced to discuss the actual changes such as the open submission system rather than a lack of an ISSN number.

The Major Contributions

Digital humanities projects – particularly the digital ones – have taken an entirely different path to the familiar clothes approach. They look completely different. Many people still aren’t clear where a scholarly database or a digital tool fits amongst the other scholarly apparatus. Is it akin to a monograph? Is it a chapter in an edited collection? Is it an edited collection? Just how much work was it after all?

By changing the clothes completely digital humanists have made it difficult for non-digital colleagues to assess the work because they don’t necessarily understand how it was constructed or the intellectual considerations that went into building it. I believe we can change that by beginning to write and publish reviews, because within the traditional humanities framework reviews are reserved for major contributions. In historical journals it is generally only books that warrant a review, and I would suggest the book-equivalent amongst digital projects could benefit from the same.

If the community of digital humanists own this process, then it’s the community that gets to decide what the major contributions to the field are, by putting them up for review. And that process means non-digital colleagues not only have an arms-length evaluation of the merits and shortcomings of digital work from a reputable expert, but this review can then become the basis for digital scholars to go to their departments and say: this is what I achieved. And with that form of evidence, I think we’ll be one step closer to slotting digital projects into that mental framework of scholarly contributions.

Let’s Get Started!

Ok, ok. But before we get going, perhaps we should sit back and think for a few moments to ensure the reviews follow that model of familiar clothes. For these reviews to hold weight with non-digital faculty and to play an important role ensuring digital work receives the credit it deserves, reviews need to withstand a certain level of scrutiny. But they cannot merely look like reviews. They have to be rigorous, reliable, and arms length.

The first question then is where should the reviews appear? Many digital humanists would likely suggest blogs are the answer. After all, they are cheap and they are efficient publishing platforms. But I believe that would be an error in judgment. Firstly, they fail the familiar clothes principle. Some academics are still untrusting of blogs and whether we collectively agree with that skepticism or not the goal of reviews is to bridge that gap in trust, not entrench it; therefore we must make reviews as easy to swallow as possible for those we hope to engage.

Secondly, blogs are not arms length because the author controls all levels of the review’s distribution. A few years ago Amazon’s review system showed us why that can be a problem when the wife of an academic wrote poisoned pen reviews of his competitor’s books. If project reviews are to have an important role in assessment and credit then it is important that those reading them can be confident that the review was not written by a friend or colleague (or enemy) of a project leader who may have had an agenda.

For me, the natural home of project reviews is in a scholarly journal. Not that many dedicated scholarly journals out there focus on the digital humanities and from what I can tell only one (Literary and Linguistics Computing) currently publishes reviews - at the moment limited to reviews of books about digital humanities. This suggests to me that there is an opportunity for any of these journals to take the lead on such an initiative and adopt a project reviews section. I would particularly urge the Journal of Digital Humanities to take on this challenge not only as the new kid on the block, but as a group committed to trying new things in publishing and championing those who do things just a little bit differently.

By having reviews centred in a scholarly journal they gain not only a trusted distribution system – again familiar clothes – but they also fall under the control of an impartial editor. This is important for the same reasons that a blog is not the right venue for such work; by putting the system of soliciting reviews in the hands of an arms-length editor, a further check is placed upon the quality of reviews. This will not only reduce the number of friends-helping-friends reviews, but also ensures that the best work gets reviewed, rather than the work of people with large professional and personal networks.

By providing a familiar form of critical assessment for non-digital colleagues, digital humanists can collectively define what makes their work good, innovative, and scholarly. Because it is they who are best positioned to do so. In a few years I hope to be sitting with a new class, asking them: “Why are we reading this digital project review? What’s the point of this piece of writing?”

Then as now, the answer is not because a PhD student wanted to add a line to his C.V. Rather, it is because the digital humanities community needed a mode of assessing their work in a way that reflected the unique challenges and assets of such projects so that their colleagues had the tools available to critique some great if oftentimes overlooked projects.

I’ll even volunteer to write the first review. You know where to find me.

Citation in Digital Humanities: Is the Old Bailey Online a Film, or a Science Paper?

Recently I was writing a paper for a journal and needed to cite the Old Bailey Online (OBO). Not any particular piece of content contained in the project, but the project itself as an outstanding example of digital humanities work. For those unfamiliar with the venture, it's a database containing 127 million words of historical trial transcripts marked up extensively with XML; still the flagship project of its kind in this author's opinion. I found myself struggling to decide who the authors of the project were; that is, whose names was I bound by "good scholarship" to include in the citation. Who deserved public credit? I happen to meet regularly with one of the project's principle investigators, Tim Hitchcock of the University of Hertfordshire, and raised the issue with him over drinks at the pub - incidently the pub is the most engaging place to discuss topics as dry as citation practices and the discussion becomes increasingly more engaging as the evening progresses. As it happens, the project had over 40 known contributors who actively participated in its creation. His initial response was that the team decided not to include any names when citing the project to avoid leaving people out and focusing credit in the hands of only some of the team members. The resulting citation looks like this:

Old Bailey Proceedings Online. Version 6.0, March 2011. http://www.oldbaileyonline.org/

This is a very noble position for the project leaders to take; however, I do not believe it is the right position. In an effort not to emphasize the contributions of some over that of others, this policy makes most contributors entirely invisible. This is particularly significant for people in the alternative academic (alt-ac) fields whose career progression and in many cases, next meal, depend upon the strength of their portfolios. These people have roles such as project management, database building, and web design, all of which are crucial to ensure the projects themselves are world class. If we adopt the no-names policy across the board, these people will never be cited anywhere, whereas traditional academics may still have books and journal articles on top of their digital project work.

Though we brought our positions much closer together, the issue proved too much for a bottle of wine to solve. We parted ways and Hitchcock took the discussion to H-Albion, a list-serv for historians of Britain and Ireland where many historians and librarians have contributed their opinions. Seth Denbo then brought the discussion to Humanist, another list-serv for digital humanities scholars where a separate conversation has now begun. Rather than contribute to either or both of those conversations, I have decided to address the issue here with the hopes that it can find new contributors who may not otherwise see it in the list-servs.

The most interesting question to arise so far is whether digital humanities projects like the OBO are films or science papers. Not literally of course, but in terms of the model of credit offered to contributors of the finished product. Both films and science journals have developed unique models of credit. In films, the credits run at the end. In science papers, everyone who made a meaningful contribution gets listed as an author and those who made minor contributions get an acknowledgement. I will argue that digital humanists would be doing their field and industry a great service by adopting both models simultaneously. The OBO and projects like it are both films and science papers.

Films

One of the respondants to the list-serv discussion, a retired librarian Malcolm Shifrin, suggested that the point of a citation was to retrieve the source, not to provide credit. In this sense, it does not matter whose names appear in the citation, as long as there is no ambiguity and the item can be identified. However, if that were the case, we could merely cite ISBN numbers, which would drastically cut down on the size of footnotes. Or, in nearly all cases, titles alone would suffice. For example, if I were to task you with finding a copy of the paper: "An alternative definition of the scapular coordinate system for use with RSA" without any further information, I'm entirely confident you would make your way to a paper by my lovely wife, which appears in the Journal of Biomechanics. Citation is not merely about finding an item, it is also about credit; however, as Shifrim points out, it is not crucial that credit appears within a citation. An alternative model is the one used by the film industry in which a portion of the finished product is dedicated to letting everyone know who was involved with its creation.

Most major website projects, including the OBO, already do this. The OBO's "About this Project" page lists 24 of the leading contributors along with their roles and effectively mimics the credits on a film. A listing of this sort is important because it offers an official "in-house" acknowledgement that's difficult to fake without breaking the law and hijacking the website to add your name. This allows everyone to direct future potential employers to evidence of past work that can be independently verified. I would certainly argue that any collaborative digital humanities project should reserve a space on their website for such a page, which has absolutely no cost but can be instrumental to the future career development of your team members. But, I certainly do not think it's enough.

We do not know where the alt-ac world is going, and we would be wise to ensure that as many doors as possible remain open to those people who currently occupy this grey space in academia. Some members may aspire to a future tenure-track position and may find it difficult to convince more conservative senior faculty that film-style credits on a webpage are akin to hits on JSTOR. And because these conservative attitudes change slowly, it would be rash for digital humanists to abandon a well established if perhaps dated model of credit just because we want to rebel in the name of progress. There's a baby in that bathwater.

Science Papers

This is where the model used by the academic science community is particularly helpful. In the humanities, typically if someone got paid to do the work as part of a grant or part-time role, we pretend they didn't exist. The work "was done" rather than "was done by soandso". We don't expect McDonalds to list the names of individual "team members" when they brag about how delicious their french fries are. It doesn't matter who made your fries. They were paid to do so and thereby give up their right to credit.

In the sciences, everyone who makes a meaningful contribution is entitled to a share of the authorship of a paper. Assuming each of the 24 members of the OBO team met those criteria, a citation for the OBO might look like this:

Hitchcock T, Shoemaker R, Emsley C, Howard S, Hardman P, Bayman A, Garrett E, Lewis-Roylance C, Parkinson S, Simmons A, Smithson G, Wilcox N, Wright C, Clayton M, Bankhurst B, Lingwood D, MacKenzie E, Rogers K, McLaughlin J, Henson L, Black J, Newman E, O'Flaherty K, Smithson G. Old Bailey Proceedings Online. Version 6.0, March 2011. http://www.oldbaileyonline.org/

It may be a bit more of an eyefull than the previous example, but at least it's a more accurate reflection of the work people put into the site's creation. The exact criteria for determining a "meaningful contribution" generally rests with the policies of individual journals. A typical example, from the International Committee of Medical Journal Editors requires that each author must have made substantial contributions to all of the following:

1. the conception and design of the study, or acquisition of data, or analysis and interpretation of data
2. drafting the article or revising it critically for important intellectual content
3. final approval of the version to be submitted

Obviously those criteria are designed specifically with a peer-reviewed journal article in mind. However, they can easily be adapted to the needs of a digital humanities web-based project, which typically is split into two parts: the project itself, and the digital infrastructure for allowing the audience to interact with the project. A digital humanities "author" could be someone that must have made substantial contributions to all of the following:

1. the conception and design of the project or website; or acquisition of data or materials; or analysis, transformation and interpretation of data or materials
2. drafting or creating any text, artwork, sound, video, workflow, interface, user experience, or code, that was integral to the success of the project and that would have been substantially different if it had been completed by someone else
3. final approval of the finished product
   

In the case of the OBO, that may eliminate some people from the list of those credited with the project. As I am not one of those people, it is not my place to decide. But it is something I think as a community we should start discussing as soon as project teams are put together. What is the intended output, and how will each person's contribution be credited? It can be an awkward conversation at first, but it's a proactive solution to the elephant in the room for those in the alt-ac community.

Conclusion

The OBO is both a film and a science paper. Project leaders of web-based digital humanities projects would be doing their industry a favour by ensuring projects have both a page of film-style credits which outline contributors and their roles, as well as a science-style listing of substantial contributors or authors that are prominently displayed for anyone wishing to cite.

This two-pronged approach can only serve to help digital humanities to find its place within the academic world. It's the model that keeps the most doors open for those alt-ac members of our project teams who are unsure of which path their career will take in the future. It acknowledges the tremendous teamwork that goes into producing world class digital humanities work, setting them apart from single-authored papers. And it doesn't misrepresent or misconstrue the purposes of either model of credit. The citation may not mean much to a tenured professor, but it can help launch the career of someone in the alt-ac world. And so, the citation may be a bit clunkier if we use the science model, but at least it's an honest reflection.

Photo credit: "Steve Jobs rendered in Applesoft BASIC" by Blake Patterson.