Providing A Familiar Assessment Process for Digital Projects

"Trust Me" by Anderson Mancini

A few weeks ago in my Early Modern British History tutorial the students read a book review that had been printed in a scholarly journal. I asked the students, “Why are we reading this? What’s the point of this piece of writing?”

My cynical self tried not to let on what I was thinking: wasn’t it written so some PhD student could score a free book and put another line on his C.V.? My students thankfully didn’t see it that way.

Historical writing generally comes in one of a handful of tried-and-tested forms. In the course I teach we usually read and discuss articles published in peer reviewed journals. The students are now used to the goals of that particular mode of writing. They know it contains a historian’s contribution to a particular scholarly discussion, which is fairly narrowly defined and tends to correlate rather strongly to the topic we’re discussing that week in lecture.

They understand edited collections contain a series of articles by different authors on a theme defined by the editor. They know books (or monographs as historians like to call them) cover broader topics than a journal article, but they expect to find chapters in those monographs that rival the scope of an article. And for a first year undergraduate, that about covers the known forms of printed historical research.

Scholarly book reviews are generally a fairly new beast for the students. My question about the value of book reviews was of the type I knew I would not have been able to answer as a first year undergraduate, but I’m blessed with very clever students who had all manner of ideas. My favourite comment was that a book review:

provides another perspective on the strengths and weaknesses of the work, which are important to keep in mind when coming to our own conclusions about the author’s arguments.

I believe that idea holds the key for digital humanities scholars, many of whom feel their colleagues have been unwilling to award due credit for digital projects. However, the problem is not necessarily an unwillingness; the problem is that most people aren’t qualified to assess digital humanities work, so they just don’t. Forcing someone to assess something without the skills to do so is not the answer. I think most of us would prefer that a review of our work came from someone with the background to assess it fairly.

Scholarly reviews are a great platform for that assessment process for two reasons. Firstly, they take a form that nearly all humanities scholars already understand. Secondly, they allow the digital humanities community to self-define the major projects so that non-digital colleagues don’t have to.

Familiar Clothes

Like my students, most scholars can visualize where a review fits amongst the various scholarly apparatus floating around in the intellectual space in which we share and critique each other’s work. They understand that a quality review in a reputable journal represents an arms-length critical evaluation of the work by someone with the skills to make the assessment. Reviews certainly are not a perfect model. There is clearly a risk to any creator or team that they will get an unfairly negative treatment by someone who perhaps is not as qualified to write the review as they may think. But this is the same risk authors of monographs face, and to me, the benefits far outweigh the risks.

The world of publishing is changing, but there’s something to be said for making something look familiar. The Journal of Digital Humanities has taken this approach by experimenting with new ways of finding content that is important to the scholarly development of digital humanities – although I imagine that process of identifying content would look rather shocking to a typical historian or English lit. professor. (If you’d like to learn more about that project read their “How to Contribute” page.) However, by wrapping it up in the clothes of a traditional peer reviewed journal, the editors get to experiment, the authors get credit for their contributions, and the readers are drawn to some of the best ideas that appear online. Everyone wins.

For me, the Journal of Digital Humanities has represented a great success as a project that has been able to forge ahead, while acknowledging that sometimes it’s a good idea to change things gradually. In that project, the team decided to make sure they held onto some familiar concepts. They registered an ISSN number (2165-6673). They release the journal in “Volumes” with issue “Numbers” to ensure any citations look like a typical humanist expects them to. Neither of these is technically necessary, but they certainly do not hurt. And by having them those who disapprove of any changes to the mechanisms for scholarly communication have a few less things about which to complain and are instead forced to discuss the actual changes such as the open submission system rather than a lack of an ISSN number.

The Major Contributions

Digital humanities projects – particularly the digital ones – have taken an entirely different path to the familiar clothes approach. They look completely different. Many people still aren’t clear where a scholarly database or a digital tool fits amongst the other scholarly apparatus. Is it akin to a monograph? Is it a chapter in an edited collection? Is it an edited collection? Just how much work was it after all?

By changing the clothes completely digital humanists have made it difficult for non-digital colleagues to assess the work because they don’t necessarily understand how it was constructed or the intellectual considerations that went into building it. I believe we can change that by beginning to write and publish reviews, because within the traditional humanities framework reviews are reserved for major contributions. In historical journals it is generally only books that warrant a review, and I would suggest the book-equivalent amongst digital projects could benefit from the same.

If the community of digital humanists own this process, then it’s the community that gets to decide what the major contributions to the field are, by putting them up for review. And that process means non-digital colleagues not only have an arms-length evaluation of the merits and shortcomings of digital work from a reputable expert, but this review can then become the basis for digital scholars to go to their departments and say: this is what I achieved. And with that form of evidence, I think we’ll be one step closer to slotting digital projects into that mental framework of scholarly contributions.

Let’s Get Started!

Ok, ok. But before we get going, perhaps we should sit back and think for a few moments to ensure the reviews follow that model of familiar clothes. For these reviews to hold weight with non-digital faculty and to play an important role ensuring digital work receives the credit it deserves, reviews need to withstand a certain level of scrutiny. But they cannot merely look like reviews. They have to be rigorous, reliable, and arms length.

The first question then is where should the reviews appear? Many digital humanists would likely suggest blogs are the answer. After all, they are cheap and they are efficient publishing platforms. But I believe that would be an error in judgment. Firstly, they fail the familiar clothes principle. Some academics are still untrusting of blogs and whether we collectively agree with that skepticism or not the goal of reviews is to bridge that gap in trust, not entrench it; therefore we must make reviews as easy to swallow as possible for those we hope to engage.

Secondly, blogs are not arms length because the author controls all levels of the review’s distribution. A few years ago Amazon’s review system showed us why that can be a problem when the wife of an academic wrote poisoned pen reviews of his competitor’s books. If project reviews are to have an important role in assessment and credit then it is important that those reading them can be confident that the review was not written by a friend or colleague (or enemy) of a project leader who may have had an agenda.

For me, the natural home of project reviews is in a scholarly journal. Not that many dedicated scholarly journals out there focus on the digital humanities and from what I can tell only one (Literary and Linguistics Computing) currently publishes reviews - at the moment limited to reviews of books about digital humanities. This suggests to me that there is an opportunity for any of these journals to take the lead on such an initiative and adopt a project reviews section. I would particularly urge the Journal of Digital Humanities to take on this challenge not only as the new kid on the block, but as a group committed to trying new things in publishing and championing those who do things just a little bit differently.

By having reviews centred in a scholarly journal they gain not only a trusted distribution system – again familiar clothes – but they also fall under the control of an impartial editor. This is important for the same reasons that a blog is not the right venue for such work; by putting the system of soliciting reviews in the hands of an arms-length editor, a further check is placed upon the quality of reviews. This will not only reduce the number of friends-helping-friends reviews, but also ensures that the best work gets reviewed, rather than the work of people with large professional and personal networks.

By providing a familiar form of critical assessment for non-digital colleagues, digital humanists can collectively define what makes their work good, innovative, and scholarly. Because it is they who are best positioned to do so. In a few years I hope to be sitting with a new class, asking them: “Why are we reading this digital project review? What’s the point of this piece of writing?”

Then as now, the answer is not because a PhD student wanted to add a line to his C.V. Rather, it is because the digital humanities community needed a mode of assessing their work in a way that reflected the unique challenges and assets of such projects so that their colleagues had the tools available to critique some great if oftentimes overlooked projects.

I’ll even volunteer to write the first review. You know where to find me.