Using OpenRefine with Pleiades

This past summer, DC3's Ryan Baumann developed a reconciliation service for Pleiades. He's named it Geocollider. It has two manifestations:

• Upload a CSV file containing placenames and/or longitude/latitude coordinates, set matching parameters, and get back a CSV file of possible matches.
• An online Application Programming Interface (API) compatible with the OpenRefine data-cleaning tool.

The first version is relatively self-documenting. This blog post is about using the second version with OpenRefine.

Reconciliation

I.e., matching (collating, aligning) your placenames against places in Pleiades.

Running OpenRefine against Geocollider for reconciliation purposes is as easy as:

• Download and install OpenRefine.
• Follow the standard OpenRefine instructions for "Reconciliation," but instead of picking the pre-installed "Wikidata Reconciliation Service," select the "Add standard service..." button and enter "http://geocollider-sinatra.herokuapp.com/reconcile" in the service URL dialog, then select the "Add Service" button.

When you've worked through the results of your reconciliation process and selected matches, OpenRefine will have added the corresponding Pleiades place URIs to your dataset. That may be all you want or need (for example, if you're preparing to bring your own dataset into the Pelagios network) ... just export the results and go on with your work. 

But if you'd like to actually get information about the Pleiades places, proceed to the next section.

Augmentation

I.e., pulling data from Pleiades into OpenRefine and selectively parsing it for information to add to your dataset.

Pleiades provides an API for retrieving information about each place resource it contains. One of the data formats this API provides is JSON, which is a format with which OpenRefine is designed to work. The following recipe demonstrates how to use the General Refine Expression Language to extract the "Representative Location" associated with each Pleiades place. 

Caveat: this recipe will not, at present, work with the current Mac OSX release of OpenRefine (2.7), even though it should and hopefully eventually will.  It has not been tested with the current releases for Windows and Linux, but they probably suffer from the same limitations as the OSX release. More information, including a non-trivial technical workaround, may be had from OpenRefine Issue 1265. I will update this blog post if and when a resolution is forthcoming.

1. Create a new column containing Pleiades JSON. 

Assuming your dataset is open in an OpenRefine project and that it contains a column that has been reconciled using Geocollider, select the drop-down menu on that column and choose "Edit column" -> "Add column by fetching URLs ..."

In the dialog box, provide a name for the new column you are about to create. In the "expression" box, enter a GREL expression that retrieves the Pleiades URL from the reconciliation match on each cell and appends the string "/json" to it:

cell.recon.match.id + "/json"

OpenRefine retrieves the JSON for each matched place from Pleiades and inserts it into the appropriate cell in the new column. 

2. Create another new column by parsing the representative longitude out of the JSON.

From the drop-down menu on the column containing JSON, select "Edit column" -> "Add column based on this column..."

In the dialog box, provide a name for the new column. In the expression box, enter a GREL expression that extracts the longitude from the reprPoint object in the JSON:

value.parseJson()['reprPoint'][0]

Note that the reprPoint object contains a two-element list, like:

[ 37.328382, 38.240638 ]

Pleiades follows the GeoJSON specification in using the longitude, latitude ordering of elements in coordinate pairs so, to get the longitude, you use the index (0) for the first element in the list.

3. Create a column for the latitude

Use the method explained in step 2, but select the second list item from reprPoint (index=1).

4. Carry on ...

Your data set in OpenRefine will now look something like this:

Mining AWOL for Identifiers

NB: There is now a follow-up post to this one, in which various bad assumptions made here are addressed: "Mining AWOL more carefully for ISSNs".

In collaboration with Pavan Artri, Dawn Gross, Chuck Jones, Ronak Parpani, and David Ratzan, I'm currently working on a project to port the content of Chuck's Ancient World Online (AWOL) blog to a Zotero library. Funded in part by a grant from the Gladys Krieble Delmas Foundation, the idea is to make the information Chuck gathers available for more structured data needs, like citation generation, creation of library catalog records, and participation in linked data graphs. So far, we have code that successfully parses the Atom XML "backup" file we can get from Blogger and uses the Zotero API to create a Zotero record for each blog post and to populate its title (derived from the title of the post), url (the first link we find in the body of the post), and tags (pulled from the Blogger "labels").

We know that some of the post bodies also contain standard numbers (like ISSNs and ISBNs), but it has been unclear how many of them there are and how regular the structure of text strings in which they appear. Would it be worthwhile to try to mine them out programmatically and insert them into the Zotero records as well? If so, what's our best strategy for capturing them ... i.e., what sort of parenthetical remarks, whitespace, and punctuation might intervene between them and the corresponding values? Time to do some data prospecting ...

We'd previously split the monolithic "backup" XML file into individual XML files, one per post (click at your own risk; there are a lot of files in that github listing and your browser performance in rendering the page and its JavaScript may vary). Rather than writing a script to parse all that stuff just to figure out what's going on, I decided to try my new favorite can-opener, ack (previously installed stresslessly on my Mac with another great tool, the Homebrew package manager).

Time for some fun with regular expressions! I worked on this part iteratively, trying to start out as liberally as possible, thereby letting in a lot of irrelevant stuff so as not to miss anything good. I assumed that we want to catch acronyms, so strings of two or more capital letters, preceded by a word boundary. I didn't want to just use a [A-Z] range, since AWOL indexes multilingual resources, so I had recourse to the Unicode Categories feature that's available in most modern regular expression engines, including recent versions of Perl (on which ack relies). So, I started off with:

\b\p{Lu}\p{Lu}+

After some iteration on the results, I ended up with something more complex, trying to capture anything that fell between the acronym itself and the first subsequent colon, which seemed to be the standard delimiter between the designation+explanation of the type of identifier and the identifying value itself. I figure we'll worry how to parse the value later, once we're sure which identifiers we want to capture. So, here's the regex I ultimately used:

\b\p{Lu}\p{Lu}+[:\s][^\b\p{P}]*[\b\:]

The full ack command looked like this:

ack -oh "\b\p{Lu}\p{Lu}+[:\s][^\b\p{P}]*[\b\:]" post-*.xml > ../awol-acronyms/raw.txt

where the -h option telling ack to "suppress the prefixing of filenames on output when multiple files are searched" and the -o option telling ack to "show only the part of each line matching" my regex pattern (quotes from the ack man page). You can browse the raw results here.

So, how to get this text file into a more analyzable state? First, I thought I'd pull it into my text editor, Sublime, and use its text manipulation functions to filter for unique lines and then sort them. But then, it occurred to me that I really wanted to know frequency of identifier classes across the whole of the blog content, so I turned to OpenRefine.

I followed OR's standard process for importing a text file (being sure to set the right character encoding for the file on which I was working). Then, I used the column edit functionality and the string manipulation functions in the Open Refine Expression Language (abbreviated GREL because it used to be called "Google Refine Expression Language") to clean up the strings (regularizing whitespace, trimming leading and trailing whitespace, converting everything to uppercase, and getting rid of whitespace immediately preceding colons). That part could all have been done in a step outside OR with other tools, but I didn't think about it until I was already there.

Then came the part OR is actually good at, faceting the data (i.e., getting all the unique strings and counts of same). I then used the GREL facetCount() function to get those values into the table itself, followed this recipe to get rid of matching rows in the data, and exported a CSV file of the unique terms and their counts (github's default display for CSV makes our initial column very wide, so you may have to click on the "raw" link to see all the columns of data).

There are some things that need investigating, but what strikes me is that apparently only ISSN is probably worth capturing programmatically. ISSNs appear 44 times in 14 different variations:

ISSN:	17
ISSN paper:	9
ISSN electrònic:	4
ISSN electronic edition:	2
ISSN electrónico:	2
ISSN électronique:	2
ISSN impreso:	2
ISSN Online:	2
ISSN edición electrónica:	1
ISSN format papier:	1
ISSN Print:	1
ISSN print edition:	1
ONLINE ISSN:	1
PRINT ISSN:	1

Compare ISBNs:

ISBN of Second Part:	2
ISBN:	1
ISBN Compiled by:	1

DOIs make only one appearance, and there are no Library of Congress cataloging numbers.

Now to point my collaborators at this blog post and see if they agree with me...

Planet Atlantides grows up and gets its own user-agent string

So, sobered by recent spelunking and bad-bot-chasing in various server logs and convicted by sage advice that ought to be followed by everyone in the UniversalFeedParser documentation, I have customized the bot used on Planet Atlantides for fetching web feeds so it identifies itself unambiguously to the web servers from which it requests those feeds.

Here's the explanatory text I just posted to the Planet Atlantides home page. Please let me know if you have suggestions or critiques.

Feed reading, bots, and user agents

As implied above, Planet Atlantides uses Sam Ruby's "Venus" branch of the Planet "river of news" feed reader. That code is written in the Python language and uses an earlier version of the Universal Feed Reader library for fetching web feeds (RSS and Atom formats). Out of the box, its http requests use the feed parser's default user agent string, so your server logs will only have recorded "UniversalFeedParser/4.2-pre-274-svn +http://feedparser.org/" when our copy of the software pulled your feed in the past. 

Effective 27 February 2014, the Planet Atlantides production version of the code now identifies itself with the following user agent string: "PlanetAtlantidesFeedBot/0.2 +http://planet.atlantides.org/". Production code runs on a machine with the IP address 66.35.62.81, and never runs more than once per hour. Apart for a one-time set of test episodes on 27 February 2014 itself, log entries recording our user agent string and a different IP address represent spoofing by a potential bad actor other than me and my automagical bot. You should nuke them from orbit; it's the only way to be sure. Note that from time-to-time, I may run test code from other IP addresses, but I will in future use the user agent string beginning with "PlanetAtlantidesTestBot" for such runs. You can expect them to be infrequent and irregular.

Please email me if you have any questions about Planet Atlantides, its bot, or these user agent strings. In particular, if you put something like "PlanetAtlantidesBot is messing up my site" in your subject line, I'll look at it and respond as quickly as I can.

Citing Sources in Digital Annotations

I'm collaborating with other folks both in and outside ISAW on a variety of digital scholarly projects in which Linked Open Data is playing a big role. We're using the Resource Description Framework (RDF) to provide descriptive information for, and make cross-project assertions about, a variety of entities of interest and the data associated with them (places, people, themes/subjects, creative works, bibliographic items, and manuscripts and other text-bearing objects). So, for example, I can produce the following assertions in RDF (using the Terse RDF Triple Language, or TuRTLe):

<http://syriaca.org/place/45> a <http://geovocab.org/spatial#Feature> ;
  rdfs:label "Serugh" ;
  rdfs:comment "An ancient city where Jacob of Serugh was bishop."@en ;
  foaf:primaryTopicOf <http://en.wikipedia.org/wiki/Suruç> ;
  owl:sameAs <http://pleiades.stoa.org/places/658405#this> .

This means: 'There's a resource identified with the Uniform Resource Identifier (URI) "http://syriaca.org/place/45" about which the following is asserted:

• it is a "Feature" as defined in the NeoGeo Spatial Ontology;
• the human-readable version of its name is "Serugh";
• a human-readable description (in the English language) of it is "An ancient city where Jacob of Serugh was bishop.";
• it is the primary topic of a document that is identified by the URI "http://en.wikipedia.org/wiki/Suruç"; and
• it is the same resource as that identified by another URI: "http://pleiades.stoa.org/places/658405#this".'

(Folks familiar with what Sean Gillies has done for the Pleiades RDF will recognize my debt to him in the what proceeds.)

But there are plenty of cases in which just issuing a couple of triples to encode an assertion about something isn't sufficient; we need to be able to assign responsibility/origin for those assertions and to link them to supporting argument and evidence (i.e., standard scholarly citation practice). For this purpose, we're very pleased by the Open Annotation Collaboration, whose Open Annotation Data Model was recently updated and expanded in the form of a W3C Community Draft (8 February 2013) (the participants in Pelagios use basic OAC annotations to assert geographic relationships between their data and Pleiades places).


A basic OADM annotation uses a series of RDF triples to link together a "target" (the thing you want to make an assertion about) and a "body" (the content of your assertion). You can think of them as footnotes. The "target" is the range of text after which you put your footnote number (only in OADM you can add a footnote to any real, conceptual, or digital thing you can identify) and the "body" is the content of the footnote itself. The OADM draft formally explains this structure in section 2.1. This lets me add an annotation to the resource from our example above (the ancient city of Serugh) by using the URI "http://syriaca.org/place/45" as the target of an annotation) thus:

<http://syriaca.org/place/45/anno/desc6> a oa:Annotation ;
  oa:hasBody <http://syriaca.org/place/45/anno/desc6/body> ;
  oa:hasTarget <http://syriaca.org/place/45> ;
  oa:motivatedBy oa:describing ;
  oa:annotatedBy <http://syriaca.org/editors.xml#tcarlson> ;
  oa:annotatedAt "2013-04-03T00:00:01Z" ;
  oa:serializedBy <https://github.com/paregorios/srpdemo1/blob/master/xsl/place2ttl.xsl> ;
  oa:serializedAt "2013-04-17T13:35:05.771-05:00" .

<http://syriaca.org/place/45/anno/desc6/body> a cnt:ContentAsText, dctypes:Text ;
  cnt:chars "an ancient town, formerly located near Sarug."@en ;
  dc:format "text/plain" ;

I hope you'll forgive me for not spelling that all out in plain text, as all the syntax and terms are explained in the OADM. What I'm concerned about in this blog post is really what the OADM doesn't explicitly tell me how to do, namely: show that the annotation body is actually a quotation from a published book. The verb oa:annotatedBy lets me indicate that the annotation itself was made (i.e., the footnote was written) by a resource identified by the URI "http://syriaca.org/editors.xml#tcarlson". If I'd given you a few more triples, you could have figured out that that resource is a real person named Thomas Carlson, who is one of the editors working on the Syriac Reference Portal project. But how do I indicate (as he very much wants to do because he's a responsible scholar and has no interest in plagiarizing anyone) that he's deliberately quoting a book called The Scattered Pearls: A History of Syriac Literature and Sciences? Here's what I came up with (using terms from Citation Typing Ontology and the DCMI Metadata Terms):

<http://syriaca.org/place/45/anno/desc7/body> a cnt:ContentAsText, dctypes:Text ;
  cnt:chars "a small town in the Mudar territory, between Ḥarran and Jarabulus. [Modern name, Suruç (tr.)]"@en ;
  dc:format "text/plain" ;
  cito:citesAsSourceDocument <http://www.worldcat.org/oclc/255043315> ;
  dcterms:biblographicCitation  "The Scattered Pearls: A History of Syriac Literature and Sciences, p. 558"@en .

The addition of the triple containing cito:citesAsSourceDocument lets me make a machine-actionable link to the additional structured bibliographic data about the book that's available at Worldcat (but it doesn't say anything about page numbers!). The addition of the triple containing dcterms:bibliographicCitation lets me provide a human-readable citation.

I'd love to have feedback on this approach from folks in the OAC, CITO, DCTERMS, and general linked data communities. Could I do better? Should I do something differently?

---

The SRP team is currently evaluating a sample batch of such annotations, which you're also welcome to view. The RDF can be found here. These files are generated from the TEI XML here using the XSLT here.

Planet Taygete update

I just learned, thanks to a blog post at the old blog, that the website and blog for the Leon Levy Expedition to Ashkelon has moved but not put any standard redirects in place at the old blog or feed. I've updated the subscription list for Taygete Atlantis to point at the new source, where I see there are a large number of posts (since early June) that I had missed.

Semantic Web, Scholarly Resources for Antiquity and the Museum

Our on-going work on geographically functional, cross-resource, machine-actionable citation(!) with the Web continues to get more interesting.

The kickoff was, of course, the joint NEH/JISC grant that is (under the rubric of the Concordia project) funding our look at this in collaboration with the Centre for Computing in the Humanities at King's College, London. Our two workshops (and lots of discussion with other parties in between) have led us through KML, Atom+GeoRSS, citation vocabularies and OAI/ORE on to Cool URIs, Linked Data, CIDOC CRM and more.

Traffic is now building on the Graph of Ancient World Data discussion group (e.g., Sebastian Heath's post on coin hoard data at nomisma.org). Yesterday, Sean Gillies rolled out some changes to the Pleiades interface that provide #this endpoints for Pleiades places, so that Sebastian and others can make explicit reference either to the historical places themselves (non-information resources cited like http://pleiades.stoa.org/places/639166#this) or our descriptions of them on the web (information resources, cited like http://pleiades.stoa.org/places/639166/).

And then this afternoon I came across the latest Talis Semantic Web podcast, featuring Koven Smith on Semantic Web initiatives at the Metropolitan Museum of Art. 38 minutes well-spent. They're thinking about and exploring a number of the approaches and technologies we're interested in, but from a museum perspective. It would be interesting to discuss how these methods could be used to better bridge gaps between museums, field archaeologists, epigraphers, numismatists, papyrologists, prosopographers, historical geographers, librarians, archivists and the rest!

Smithsonian 2.0

Smithsonian 2.0, a "gathering to re-imagine the Smithsonian in the digital age," is going on right now in Washington. You can follow the procedings via:

• The Smithsonian 2.0 Blog
• Tweets with the hashtag #si20
  

Digital Archimedes Palimpsest

This just in:

Ten years ago today, a private American collector purchased the Archimedes Palimpsest. Since that time he has guided and funded the project to conserve, image, and study the manuscript. After ten years of work, involving the expertise and goodwill of an extraordinary number of people working around the world, the Archimedes Palimpsest Project has released its data. It is a historic dataset, revealing new texts from the ancient world. It is an integrated product, weaving registered images in many wavebands of light with XML transcriptions of the Archimedes and Hyperides texts that are spatially mapped to those images. It has pushed boundaries for the imaging of documents, and relied almost exclusively on current international standards. We hope that this dataset will be a persistent digital resource for the decades to come. We also hope it will be helpful as an example for others who are conducting similar work. It published under a Creative Commons 3.0 attribution license, to ensure ease of access and the potential for widespread use. A complete facsimile of the revealed palimpsested texts is available on Googlebooks as "The Archimedes Palimpsest." It is hoped that this is the first of many uses to which the data will be put.

For information on the Archimedes Palimpsest Project, please visit:
www.archimedespalimpsest.org

For the dataset, please visit:
www.archimedespalimpsest.net

We have set up a discussion forum on the Archimedes Palimpsest Project. Any member can invite anybody else to join. If you want to become a member, please email:

wnoel@thewalters.org

I would be grateful if you would circulate this to your friends and colleagues.

Thank you very much
Will Noel
The Walters Art Museum
October 29th, 2008.

I found it a bit tricky to find the Google Books version of this, so here's the link.

The DH Stack(s)

Lots of interesting posts in the last couple of days about Digital Humanities skills, software and cyberinfrastructure initiatives:

• Sean issues some service-oriented caveats for Project Bamboo and thinks about skillsets for DH-ready graduates
• Lisa surveys recent DH job postings, muses on DH 'fieldness' and visits the aspiring digital humanist's skillset too
• Bill T.'s whole blog remains essential reading in this meme-space (e.g., see his recent "Navigating Digital History")
  
• Liza provides an example of TEI+Python+lxml in publishing primary sources online (the same general stack the Duke/Heidelberg/King's/Columbia/UNC/NYU team has been using in the rework of the Duke Databank of Documentary Papyri and Heidelberger Gesamtverzeichniss -- coming soon)
  

Why email a newsletter but not post it?

I have to confess publicly, at the risk of being thought rude, that I was dumbfounded to read Polly Low's comment this morning:

The absence of the latest newsletter from the BES website is deliberate — only [listserv] subscribers get the cutting-edge news!

Why on earth would a professional academic organization with a web presence and a mission statement thereon that contains the following words limit themselves in this way?:

to promote the study of inscriptions, texts and historical documents ... disseminating news of the latest developments in epigraphic studies, in Britain and around the world

Or is the real issue that "subscribers" = "members" and timely access to the newsletter is seen by the Society as an exclusive benefit of membership?

Database Normalization and the Historian

Over at the UVA Library's Scholar's Lab Blog, Jean Bauer has a useful post ("Normality: For or Against") in which she considers the process of database normalization, its value in the context of particular historical research tasks, and the interesting problems that arise when you consider publishing such a database -- designed originally to support a particular line of inquiry -- for the use of other scholars.

Natual Language Toolkit (NLTK) penetration?

I'd be interested to know of digital classicists, antiquisters and those inhabiting neighboring nodes who are making use of NLTK and what your impressions of strengths and weaknesses are.

BMCR and SAFE Events Feeds Pulled from Maia

It is with regret that this morning I have pulled the Bryn Mawr Classical Review (BMCR) Most Recent Articles feed from the list of feeds aggregated by Maia Atlantis. I took this step in accordance with my own Atlantis Suppression Policy.

It would appear that every time the BMCR adds a new article, dates on all articles in the feed are updated to present. As of this writing every single entry contains an identical "pubdate" tag with the value "03:49:18, Sunday, 03 August 2008" even though some of the entries have been in the list since it was first deployed a few weeks ago. This is non-standard behavior, and has the effect of pushing all the BMCR entries, in a block, to the top of any feed reader or aggregator, ahead of other content that is actually new. And in most feed readers they will show up highlighted or bolded, to indicate "new content." The appropriate behavior is to adjust dates only on those entries that have been added or substantially changed.

The Saving Antiquities for Everyone (SAFE) Events Feed is also blocked because it is forward-dating announcements of events to the date of the event, rather than the date of the entry. For example, the current feed contains a single entry with the following pubdate: "Thu, 16 Oct 2008 07:00:00 EST". This is the event date, not the publication date of the feed entry. This is also abuse of feed entry date fields and has the effect of causing these entries to linger at the top of the aggregation list for weeks or months until the date of the event passes.

I will be contacting the editors of both resources in the hopes of resolving these technical difficulties so that their content can once again be featured in Maia Atlantis.

Hidden Web: Don't Love It, Leave It

There's been a bit of buzz lately about Google's "failure" to effectively search the "hidden (deep) web". In the discussions I've been seeing, the hidden web is equated with stuff in academic and digital library repositories, i.e., "OAI-based resources" (which I assume to mean OAI/PMH).

I have to say: repositories != hidden web. The hidden web is simply the stuff the search engines don't find. Systems that surface information about their content only through OAI/PMH interfaces might make up a small part of the hidden web because they're not being surfaced to the bots, but frankly the hidden web holds way more stuff than what's in Fedora and DSpace at universities. Just ask Wikipedia.

The assertion that repository content == the hidden web is circular and false rhetoric that obscures the real problem: people are fighting the web instead of working with it. If you fight it, it will ignore you. This sort of thinking also makes hay for enterprises like the Internet Search Environment Number that seem to me to be trying to carve out business models that exploit, perpetuate and promote the cloistering of content and the rationing of information discovery.

Yesterday, Peter Millington posted what's effectively the antidote on the JISC-REPOSITORIES list (cross-posted to other lists). I reproduce it here in full because it's good advice not just for repositories but for anybody who is putting complex collections of content on the web and wants that content to be discoverable and useful:

Ways to snatch defeat from the jaws of victory
Peter Millington
SHERPA Technical Development Officer
University of Nottingham

You may have set up your repository and filled it with interesting papers, but it is still possible to screw things up technically so that search engines and harvesters cannot index your material. Here are seven common gotchas spotted by SHERPA:

1. Require all visitors to have a username and password
2. Do not have a 'Browse' interface with hyperlinks between pages
3. Set a 'robots.txt' file and/or use 'robots' meta tags in HTML headers that prevent search engine crawling
4. Restrict access to embargoed and/or other (selected) full texts
5. Accept poor quality or restrictive PDF files
6. Hide your OAI Base URL
7. Have awkward URLs

Full explanations and some solutions are given at: http://www.sherpa.ac.uk/documents/ways-to-screw-up.html

If you know of any other ways in which things may go awry, please contact us and we will consider adding them to the list.

I'm happy to say: Pleiades gets a clean bill of health if we count nos. 5 and 6 as non-applicable (since we're not a repository per se and we don't have a compelling use case for OAI/PMH or PDF).

Disclaimer: we are exploring the use of OAI/ORE through our Concordia project. One of the things we like most about it is that its primary serialization format is Atom, which is already indexed by the big search engines. With the web.

Kostis Kourelis on GIS and Greek history

Kostis Kourelis has recently posted the third entry (Mapping Eleia) in an interesting series on toponyms, archaeology and GIS. There's no distinctive tag, so I'll explicitly link to the two previous posts as well:

• Mapping Greek Toponyms
• Mapping Greek Landscapes in GIS

He makes some useful observations about the challenges of working with historical GIS, including sourcing spatial data, tracking name changes and appropriations, addressing the shifting locations of particular conceptual places, navigating the influence of politics (both past and present) and handling issues of languages, scripts and transliteration.

Apropos language, script and transliteration there are many things to consider. It's in this domain of course that the simple, flat-file approach to GIS often breaks down, particularly given the problems of change over time and of fragmentary witnesses. For Pleiades -- which was designed with an object-oriented data model -- we record individual name variants, each of which has a language-and-script combination, an "original script" representation (using Unicode) and a transliteration. We can have as many names assigned to a given "place" as we have variants (or theories).

We have our own slightly idiosyncratic transliteration scheme for classical Greek (inherited from the Barrington Atlas; one of the major benefits of Pleiades is the ability to add back the original orthography). We could easily add multiple transliteration schemes (and the corresponding strings generated programmatically from the Unicode Greek). We may well need such a development when we move, as we eventually must, to include historical toponymy in Arabic (where both past and present variation in transliteration schemes renders even recent bibliography a veritable maze).

If you're trying to do this ArcGIS, you'll probably have to set up a relational database or manage a series of joined tables manually.

Barrington Atlas IDs

Update: follow the batlasids tag trail for follow-ups.

Back in February, I blogged about clean URLs and feed aggregation. In March, we learned about the ORE specification for mapping resource aggregations in Atom XML, just as we were gearing up to start work on the Concordia project, with support from the US National Endowment for the Humanities and the UK Joint Information Services Committee.

Our first workshop was held in May. One of the major outcomes was a to-do for me: provide a set of stable identifiers for every citable geographic feature in the Barrington Atlas so collaborators could start publishing resource maps and building interoperation services right away, without waiting for the full build-out of Pleiades content (which will take some time).

The first fruits can be downloaded at: http://atlantides.org/batlas/ . All content under that URL is licensed cc-by. Back versions are in dated subdirectories.

There you'll find XML files for 3 of the Atlas maps (22, 38 and 65). There's only one feature class for which we don't provide IDs: roads. More on why not another time. I'll be adding files for more of the maps as quickly as I can, beginning with Egypt and the north African coast west from the Nile delta to Tripolitania (the Concordia "study area"). Our aim is full coverage for the Atlas within the next few months.

What do you get in the files?

IDs (aka aliases) for every citable geographic feature in the Barrington Atlas. For example:

• BAtlas 65 G2 Ouasada = ouasada-65-g2

If you combine one of these aliases with the "uribase" also listed in the file (http://atlantides.org/batlas/) you get a Uniform Resource Identifier for that feature (this should answer Sebastian Heath's question).

For features with multiple names, we provide multiple aliases to facilitate ease of use for our collaborators. For example, for BAtlas 65 A2 Aphrodisias/Ninoe, any of the following aliases are valid:

• aphrodisias-ninoe-65-a2
• aphrodisias-65-a2
• ninoe-65-a2
  

Features labeled in the Atlas with only a number are also handled. For example, BAtlas 38 C1 no. 9 is glossed in the Map-by-Map Directory with the location description (modern names): "Siret el-Giamel/Gasrin di Beida". So, we produce the following aliases, all valid:

• (9)-38-c1
• (9)-siret-el-giamel-gasrin-di-beida-38-c1
• (9)-siret-el-giamel-38-c1
• (9)-gasrin-di-beida-38-c1

Most unlabeled historical/cultural features also get identifiers. For example:

• Unnamed aqueduct at Laodicea ad Lycum in BAtlas 65 B2 = aqueduct-laodicea-ad-lycum-65-b2
  
• Unnamed bridge at Valerian in BAtlas 22 B5 = bridge-valeriana-22-b5

Unlocated toponyms and false names (appearing only in the Map-by-Map Directory) get treated like this:

• BAtlas 22 unlocated Acrae = acrae-22-unlocated
  
• BAtlas 38 unlocated Ampelos/Ampelontes? = ampelos-ampelontes-38-unlocated = ampelos-38-unlocated = ampelontes-38-unlocated
• BAtlas 65 false name ‘Itoana’ = itoana-65-false
  

The XML files also provide associated lists of geographic names, formatted BAtlas citations and other information useful for searching, indexing and correlating these entries with your own existing datasets. What you don't get is coordinates. That's what the Pleiades legacy data conversion work is for, and it's a slower and more expensive process.

Read on to find out how you can start using these identifiers now, and get links to the corresponding Pleiades data automatically as it comes on line over time.

Why do we need these identifiers?

Separate digital projects would like to be able to refer unambiguously to any ancient Greek or Roman geographic feature using a consistent, machine-actionable scheme. The Barrington Atlas is a stable, published resource that can provide this basis if we construct the corresponding IDs.

Even without coordinates, other projects can begin to interoperate with each other immediately, as long as they have a common scheme of identifiers. After using BAtlas URIs to normalize, control or annotate their geographic description, they can publish services or crosswalks that provide links for the relationships within and between their datasets. For example, for each record in a database of coins you might like links to all the other coins minted by the same city, or to digital versions (in other databases) of papyrus documents and inscriptions found at that site.

Moreover, we would like other projects to start using a consistent identifier scheme now, so that as Pleiades adds content we can build more interoperation around it (e.g., dynamic mapping, coordinate lookup, proximity search across multiple collections). To that end, Pleiades will provide redirects (303 see other) from Barrington Atlas URIs (following the scheme described here) as follows:

• If a corresponding entry exists in Pleiades, the web browser will be redirected to that Pleiades page automatically
  
• If there is not yet a corresponding entry in Pleiades, the web browser will be redirected to an HTML page providing a full human-readable citation of the Atlas, as well as information about this service

So, for example:

• http://atlantides.org/batlas/aphrodisias-ninoe-65-a2 will re-direct to http://pleiades.stoa.org/places/638753
• http://atlantides.org/batlas/vlahii-22-e4 will re-direct to http://atlantides.org/batlas/vlahii-22-e4.html until there is a corresponding Pleiades record

The HTML landing pages for non-Pleiades redirects are not in place yet, but we're working on it. We'll post again when that's working.

Why URIs for a discretely citable feature in a real-world, printed atlas?

I'll let Bizer, Cyganiak and Heath explain the naming of resources with URI references. In the parlance of "Linked Data on the Web," Barrington Atlas features are "non-information resources"; that is, they are non-digital/real-world discrete entities about which web authors and services may want to make assertions or around which to perform operations. What we are doing is creating a stable system for identifying and citing these resources so that those assertions and operations can be automated using standards-compliant web mechanisms and applications. The HTML pages to which web browsers will be automatically redirected constitute "information resources" that describe the "non-information resources" identified by the original URIs.

How

If I get a comment box full of requests for a blow-by-blow description of the algorithm, I'll post something on that. If you're really curious and energetic, have a look at the code. It's intended mostly for short-term, internal use, so it's not marvelously documented. Yes, it's a hack.

One of the big headaches was deciding how to decompose the complex labels into simple, clean ASCII strings that can be legal URL components. Sean blogged about that, and wrote some code to do it, shortly after the workshop.

Credit where credit is due

Sean and I had a lot of help from the workshop participants (Ben Armintor, Gabriel Bodard, Hugh Cayless, Sebastian Heath, Tim Libert, Sebastian Rahtz and Charlotte Roueché) in sorting out what to do here. Older, substantive conversations that informed this process (with these folks and others; notably Rob Chavez, Greg Crane, Ruth Mostern, Dan Pett, Ross Scaife†, Patrick Sims-Williams, Linda Smith and Neel Smith) go back as far as 2000, shortly after the Atlas was published.

Many thanks to all!

Examples in the Wild

Sebastian Rahtz has already mocked up an example service for the Lexicon of Greek Personal Names. It takes a BAtlas alias and returns you all the name records in their system that are associated with the corresponding place. So, for example:

• http://clas-lgpn2.class.ox.ac.uk/batlas/aloros-50-b3

This is just one of several services that LGPN is developing. See the LGPN web services page, as well as the LGPN presentation to the Digital Classicist Seminar in London last month.

Sebastian Heath, for some time, has been incorporating Pleiades identifiers into the database records of the American Numismatic Society. He has blogged about that work in the context of Concordia.

Do you have an application? Let me know!

Glossary functionality

The National Weather Service has added glossary links for technical terms and acronyms in the text products surfaced via their website, e.g. the Huntsville Area Forecast Discussion. The links point to a separate glossary application, which doesn't seem to succeed in looking up the word that's passed by the link.

:(

What I'd really love, even beyond getting this particular app to actually work, is a mouseover kite containing the appropriate gloss. There are all sorts of javascripty ways to do that, but the easiest way (in a modern browser, if the gloss is short) is to simply wrap the technical term or acronym in a span element with a title attribute containing the textual gloss. Fewer clicks, more happiness.

This sort of functionality should be a standard component, IMO, of any technical document (a classics journal article, say) that's posted to the web with any desire to reach students or a non-specialist audience. With a bit of infrastructure work, and maintenance of a glossary, it could be made to happen almost automagically as part of a publishing workflow. I'm sure there are sites where that's already the case.

Clean URLs, Sebastian Heath and feed aggregation

I see that Sebastian has submitted his post on URLs to PDQ. That seems to me to be a good thing, and not just because I'm a clean URL fettishist. Sebastian helped me explain our Pleiades URLs and the thinking behind them -- their sanity is Sean's brainchild -- in a series of posts and comments last fall.

I've been thinking more about URLs lately (which partly prompted yesterday's mouthing off about the TEI website) ... and here's why:

This morning, I gave a talk to a group at the British School in Rome (remotely, alas). The topic was "Atom+GeoRSS for interoperability". I imagined a feed-of-feeds that might aggregate glosses/citations/summaries of content on multiple websites related to the archaeology and epigraphy of Cyrene (this, an example of what we'd like to do for every place cited in Pleiades).

Most feed aggregators now bring feed content together thematically as a consequence of selection and filtering criteria established by the aggregator's editor (consider my Planet Atlantides, or Planet OSGeo or Planet Code4Lib). But we could also bring feed entries together by virtue of the values contained in the href attribute on <link rel="related"> tags (think: "all entries related to Cyrene, if the href value is the Pleiades URL for Cyrene"). Spatial correlation (containment, proximity) could also be a great way to aggregate feed content (see Sean Gillies' Mush demo) if your feeds have coordinates in them (by way of GeoRSS) or if you can geocode on the basis of an asserted link relationship with a reference that has coordinates.

I hacked up a fake, hypothetical result feed:

• nearby/cyrene.atom
  

And a fantasy of one way the content could be exploited in the Pleiades website:

• nearby/cyrene.html

The latter was generated from the former using an xslt stylesheet (atom2nearby.xsl).

In these mockups, I re-imagined the URLs of the resources glossed/described in the various entries (in some cases, I imagined their web resources structure pretty much from scratch!). The consequences for various otherwise innocent websites are as follows:

Inscriptions of Roman Cyrenaica: Cyrene
Base URL (now):
http://ircyr.kcl.ac.uk/
Real URL for collection/search of resources (now):
n/a (in development)
Fantasy URL for collection/search of resources:
http://ircyr.kcl.ac.uk/inscriptions/cyrene
Real URL for basic resource (inscription) now:
n/a
Fantasy URL for basic resource:
http://ircyr.kcl.ac.uk/inscriptions/C61300

Epigraphische Datenbank Heidelberg
Base URL (now):
http://www.uni-heidelberg.de/institute/sonst/adw/edh/Fantasy base URL:
http://edh.uni-heidelberg.de/
Real URL for collection/search of resources (now):
http://edh12.iaw.uni-heidelberg.de/offen/suchen2.html?fuan=Cyrene (hidden by frames and a form)
Fantasy URL for collection/search of resources:
http://edh.uni-heidelberg.de/findspots/cyrene
Real URL for basic resource (inscription) now:
http://edh12.iaw.uni-heidelberg.de/offen/suchen2.html?hdnr=000838 (hidden by frames and a form)
Fantasy URL for basic resource (inscription) now:
http://edh.uni-heidelberg.de/inscriptions/HD000838

Cyrenaica Archaeological Project
Base URL (now):
http://www.cyrenaica.org
Real URL for collection/search of resources (now):
n/a (in development)
Fantasy URL for collection/search of resources (in this case, an index to a reference of features and monuments on the site, as indexed in an earlier publication, Bonacasa 2000):
http://www.cyrenaica.org/bonacasa-2000/
Real URL for basic resource (monument/feature) now:
n/a
Fantasy URL for basic resource:
http://www.cyrenaica.org/bonacasa-2000/fontana-dei-buoi-di-euripilo

American Numismatic Society
Base URL (now):
http://numismatics.org/
Real URL for collection/search of resources (now):
http://publicserver.numismatics.org/collection/accnum/list?field1=mint&field1op=equals&field1kws=Cyrene (hidden by a form)
Fantasy URL for collection/search of resources:
http://numismatics.org/mints/cyrene
Real URL for basic resource (coin) now:
http://numismatics.org/dnid/numismatics.org:1997.9.200
Fantasy URL for basic resource (coin) -- I left Sebastian's DNIDs, but I was tempted to replace them with:
http://numismatics.org/coins/1997.9.200/

So, what principles do I think I've embodied in these fantasy URLs? Simplicity. Intuition. Implementation-hiding. Elimination of redundancy. Linkability and browseability for collections and likely searches (and therefore visibility of database content that would otherwise be hidden from 3rd party search engines).

Your thoughts?

What do URLs "mean": tei-c.org

Does anybody besides me find this bizarre:

http://www.tei-c.org/index.xml

gets you:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

http://www.tei-c.org/index.xml?style=raw

gets you:

<?xml version="1.0" encoding="UTF-8"?>
<?oxygen RNGSchema="http://www.tei-c.org/cms/system/modules/org.tei.www/_common/schemas/teilite.rnc" type="compact"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0" rend="home">

http://www.tei-c.org/index.xml?style=printable

gets you:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

Why not:

http://www.tei-c.org/index.html
http://www.tei-c.org/index.xml
http://www.tei-c.org/index-printable.html

or some such (in which filename extensions actually bear some relationship to what the user gets, rather than the hidden underlying format or application/file hierarchy on the server) -- and thus throughout the whole website.

Semantic Web in Archaeology

There's an interesting discussion on the Semantic Web in Archaeology unfolding on the Antiquist List.