How to Write a Zotero Translator: Wikified

I'm pleased to announce that the book I wrote a little over a year ago: How to Write a Zotero Translator (2009) has been released as a user-editable wiki on the Zotero website, released under an "Attribution-NonCommercial-ShareAlike 3.0 Unported" license.

The new version, How to Write a Zotero Translator ++, contains all the original text, but allows anyone with an understanding of HTML, JavaScript and Zotero translators to update the book. This became necessary as Zotero is an ever-evolving program and in the 17 months since I first released the guide, much has changed.

I encourage anyone with the required skills and energy to help make this resource as useful as possible for as wide an audience as possible.

Thanks to Rintze Zelle and Tom Roche for bringing the need to create a wiki version of the book to my attention.

And I leave with a challenge: take a look at your past works. Would any of them benefit from being released in a similar fashion?

Thomson Reuters harassing Zotero community

I, along with presumably 285 other people who are interested in Zotero's development got this email this afternoon:

Dear Zotero Development Community Members,

First off, please allow me to apologize for clogging your inbox with this unsolicited message, but I hope you'll understand that the severity of the situation requires me to contact you. In its ongoing litigation with George Mason University, Thomson Reuters has demanded that the university produce contact information (name, email, and username) associated with all two hundred eighty-six Zotero SVN/Trac accounts.

We can think of no use Thomson Reuters's counsel would have for this information other than to intimidate and harass you, and we made every effort to avoid turning over this information until compelled. We have requested that the contact information be placed under protective order, which in principle means that only the lawyers involved should have access to the information. Nonetheless, we feel it is our obligation to notify you that we are being forced to release this data. Please note that you are in no way required or requested to keep this disclosure confidential. If you are contacted by Thomson Reuters or their attorneys in connection with this lawsuit, please do let us know.

We deeply apologize for this encroachment on your privacy, and we sincerely hope that it does not dissuade you from remaining active members of the Zotero development community.

How to Write a Zotero Translator

I have put together what I hope is a very comprehensive online guide detailing How to Write a Zotero Translator. This guide is the guide I wished was available for me when I was given the task of writing a Zotero translator for the site http://www.canadiana.org/. The task was part of an internship during my Masters of History at the University of Western Ontario. This internship, during the summer of 2008, was held at the Center for History and New Media — the home of Zotero. When I took on the project, I had no experience with computer programming; when I left, I had created over fifty site-specific translators.

This guide is the product of that brief, yet intensive learning experience.

The goal of this guide is to provide readers enough skills and direction to create a Zotero translator of their own (which I hope they will share). It also gives researchers, libraries, archives and databases a specific resource to make their own translators rather than rely on the overworked Zotero team.

http://niche-canada.org/zotero-guide

Canadianization of Zotero: Results

In June, I sent out a request for suggestions of databases useful to Canadian historians. My goal was to make Zotero - a citation management program designed for use on the internet - friendlier for Canadian history researchers.

A tool like Zotero is only good if it works when you need it to. By extending it's capabilities to include more Canadian content, Canadian researchers now have more incentive to move away form expensive or subscription software that can grab citations from a limited number of databases.

Thank you to everyone who made suggestions and told me about the databases they use regularly.

Zotero now has more translators dedicated to Canadian sites than any other country outside the US.

My focus was on making translators for archives, journal repositories newspapers and university libraries. The results of my Canadianization is as follows:

National Archives and Archival networks:

Archives Canada (archivescanada.ca)
BCAIN
Archives Network of Alberta
Saskatchewan AIN
Manitoba AIN
Archeion
Bibliotheque et Archives Nationales Quebec
PEI AIN

Databases and Repositories:

Artefacts Canada
Archives Canada-France
Canadiana.org
Champlain Society
Civilization.ca
Canadian Letters and Images Project
Glenbow Museum
AdvoCAT - Great Library Catalogue
CARL Harvester
Eighteenth Century Collections Online

Newspapers:

The Globe and Mail
The National Post
The Toronto Star
Le Devoir
The Hamilton Spectator
Winnipeg Free press
All newspapers hosted on Canada.com
All newspapers hosted on Cyberpresse.ca

University Libraries:

UBC Library
UQAM Library

* note: most Canadian university library systems were already supported by Zotero. Zotero now supports 90% of Canadian university libraries.

------

Anyone using Zotero can now automatically grab citation information for anything from fonds, to journal articles to artifacts on these Canadian content sites and dozens of others that are already supported. Go forth and research!

Thank you to the Center for History and New Media for funding this project.

The Canadianization of Zotero

Zotero is a pretty amazing computer program for historical researchers using the internet. What makes it even better are the translators. Translators allow you to click on a little icon in the address bar and automatically save all the bibliographic information found on the website. This is a great time-saver as those of you who use Zotero for this purpose already know.

If you're not familiar with this feature and already have Zotero installed, go to www.amazon.com, search for your favourite book, go to the entry and click on the little blue book icon on the right hand side of your address bar. Then take a look at what got saved in Zotero. Pretty nifty for one click of a mouse.

However, they're not as easy to make as they are to use. The translator for Amazon.com only works for Amazon.com. Each website that is supported - and there are a lot - have a custom-coded translator, specific to that site.

Right now, most of the sites that are supported are American. I say it's time for a change.

So, in the interest of promoting Canadian history research, I'm offering you a chance to get the translator of your dreams, free of charge.

I am taking requests for translators for sites that are used by CANADIANS for research.
These sites can be in English or French (or both), and priority will go to historical databases, and requests made by UWO history professors who gave me good grades.

If you know of a site that fits these criteria that you would like a translator for (or if you operate such a site), please post your request HERE, and include the word "Canada" somewhere in your message.

I will do my best to fulfill all suggestions, provided they are posted prior to July 15, 2008.

To be eligible, the site must contain:

• a large database of records (1000+ entries).
• each entry must have its own page with a stable URL (if you can cut and paste the URL into a blank browser's address bar and it takes you to the entry, then it's stable enough).
  
• Each entry must have a title.
  
• The entries must be searchable via a search box.
  
• I must be able to access the records. (That means if it's password protected, it must either be accessible to me via the library at the University of Western Ontario's subscription, or you must provide me with access.)
• The site cannot be under construction, or planning changes to its structure/design in the near future.
  

Examples include:

• Canadiana.org
• Glenbow Library and Archives
• the Globe and Mail
• CAIN
• BCain
• UWO Library

Please forward this request to any of your colleagues who may find this helpful.

Remember, only until July 15, 2008. After which time I'll be on to other things.

The Importance of MetaData on websites

As Bill Turkel noted in his recent blog post, I've been doing a summer internship, part of which involves making translators for a program called Zotero.

If you're a historian or a history student and you don't know what Zotero is, you should definitely look into it. It allows you to save and collect bibliographical information for just about anything you find on the internet, often with the click of a button.

For example, if you go to a webpage about the United Irishmen, you can use Zotero to save a snapshot of the page (kind of like a bookmark), you can attach notes to it, create tags to help you remember what the page is about, add the author's name, date, publisher...just about whatever you want. You can then export that bibliographic data in proper Chicago/MLA/APA format and save yourself writing it all out.

Some pages are even easier to use. These are pages that Zotero has translators for. On these pages - Amazon.com for example, a little icon will appear in the address bar of the page. If you're looking at the entry for Harry Potter and the Philosopher's Stone, when you click the little icon Zotero automatically saves all the relevant bibliographical information for you. You don't have to type in a thing!
Unfortunately, these translators have to be made one by one. Each and every page on the internet has to have its own translator. Because of this, only the most important historical repositories are currently supported.

You'll find them for websites such as JSTOR, Amazon, even the University of Western Ontario's library page. But there are several important (often Canadian) pages that are not yet working.

How a translator works, is that a JavaScript program is told to check if the webpage you're currently on is one of the webpages that Zotero knows how to find bibliographic data on. This often entails checking the website's address. For example:

If this webpage's address starts with www.canadiana.org then,
I should load the translator for Canadiana.org.

The next part is quite a bit trickier! Zotero is just a program. It doesn't know anything about what it is reading. We have to teach it how to recognize which piece of information on the screen is the title, which tells us the author, etc. And I've noticed two distinct trends: those sites who provide this information in metadata and those who do not.

Metadata, for those who don't know, is helpful, clearly formatted information about your site. Go to any webpage, click on the "View" menu, and select "Page Source." If the website in question has metadata, you'll notice quite near the top several lines of code that read something like this:

This essentially tells us that there are some keywords that you might find helpful in remembering what this website is about. They include "Adam Crymble, history".

We can also tell that the author is "Adam Crymble" and this website was last revised in "spring 2008."

If a webpage contains this data, it makes it MUCH easier for other people to use the data on your page. Zotero can easily be taught to recognize that the words after the meta name=”author” tag should go in the bibliographical field "author." It is also quite easy to tell Zotero that words after the meta name=”keywords” should be separated and made into "tags" which you can then use to organize your work.

However, many...rather, most webpages do not have very good (or any) metadata. In these cases, it requires extensive work to tell Zotero what it is looking at. Rather than simply associating one metatag with one entry in Zotero, the person must analyze your page's HTML code, figure out how your page is structured and write a customized line of code called an XPath that looks something like this:

//div[@id="Content"]/div[@class="NormalRecord"]/table[@class="Bibrec"]/tbody/tr/td[2];

Don't worry, it looks like gobbledigook to me too. Each and every part of data that Zotero wants to collect needs a custom written Xpath like this. This one would find the title of a book in Canadiana.org's repository.

What could have been 3 lines of code had there been Metadata on the page now requires dozens of lines.

None of the three websites I have been working on translators for include metadata. In two of these cases - well known Canadian museums, the websites are almost brand new. They're visually stimulating and engaging. Yet the information is hidden in complex code and confusing paths.

In the 21st century, websites are not merely a static representation of one person's work. Especially those that hold information for others to use, such as libraries, archives and repositories. Designing your webpage to incorporate metadata makes the information you have put out there easier for others to use. It makes people more likely to use it. And it encourages people like those who use Zotero to help your site stand out, with exciting add-ons that are changing the way people do internet research.