Preserving Accented and Non-Roman Characters in CSV Workflows

Digital work in and around the Humanities often involves moving data from one system or format to another. That data often involves complex textual materials in multiple languages and writing systems. One commonly used format is the "Comma-Separated Values" text file. It's not uncommon to find that characters not used in English get garbled when exported from a spreadsheet program like Microsoft Excel to CSV (or imported from CSV into such a program). What's going on and how do you make it stop?

Why

CSV began life in an era before Unicode and, because of that background, some software assumes that CSV should be encoding using the ASCII text encoding scheme (some older versions of Excel). Some software defaults to using ASCII, but you can override it manually (more recent versions of Excel). Some software tries to guess what encoding to use when reading or writing a given CSV file, but how it guesses may not be foolproof. Some software writes a special code called a Byte-Order Mark (BOM) into the beginning of any CSV file that uses a Unicode-aware encoding (Excel for Mac 2016). Some software doesn't expect a BOM and will fail to read the data correctly even if the encoding (e.g., UTF-8) is otherwise supported.

How to make it stop

The best way to make it stop is to:

1. Make sure that any CSV file you import or export is encoded in UTF-8 without a Byte-Order Mark.
2. Make sure that any software you're using is capable of reading and writing CSV files in UTF-8 without BOM and has been told to do so.

Failing that (i.e., you got the CSV file from someone else), you need to find out what encoding it uses and configure your software to read it properly. But note that if the creator of the CSV file allowed it to be written in ASCII, it can't be repaired. You'll have to get them to re-export properly, or to send you the original file so you can open it in appropriate software and save it more deliberately.

It is happening

A couple of hours ago, I was sitting out on the back deck with my wife and pets, enjoying perfect temperatures, morning birdsong, lavender-scented country air, and a cup of freshly brewed Costa Rican coffee (roasted by the good folks at the Kaffeeklatsch in Huntsville). Idyllic.

I was flipping through the latest news stories, blog posts, and such, brought into my phone by my feed reader (currently Feedly). I was trying to ignore the omnipresent bad news of the world, when this popped up:

Forma[m] Lollianus fecit?!? I'm predisposed by my dissertation experience to trigger on certain Latin and Greek words because of their sometime significance for the study of Roman geography. Forma is of course one of those words, and it does (probably more often than justified) get translated as "map" or "plan." Could this be — admittedly against the odds —an inscription on a map or plan drafted or surveyed by some guy named Lollianus?

If you're me, the possibility warrants a click-through to a corresponding record in the Heidelberg Epigraphic Databank (EDH).

My mappish hopes were quickly dashed, but just as quickly were replaced by interest in a group of inscribed objects I hadn't run across before: mirrors from Roman Dacia bearing makers' inscriptions. "Forma" can mean "mirror"? A quick check of Lewis & Short at Perseus doesn't support that idea, but builds confidence in a better interpretation: "mold, stamp, form". Was this mirror, or some part of it, somehow cast or stamped out? The EDH entry tells me there are 9 identical mirrors extant and that the inscription goes around the "Fassung" (frame?). Yup.

Cool. I learned something today before breakfast. And it's knowledge I can use when I come back to doing more with the geographical/mapping/surveying vocabulary.

And then it hits me: that's not information I went looking for, not a search I initiated. New information of interest was pushed to me because I had previously used a software tool to express interest in a number of information sources including, but not limited to, ancient inscriptions. The software kept an eye on new output from those sources and made it available to me for review and engagement in a mode and at a time and place of my choosing. And because the source data was online, open, and linked in a standard format, I was able to drink coffee and pet my dog on the back deck in Moontown, Alabama while making use of the scholarly work done yesterday(!) by Brigitte Gräf in Heidelberg, Germany.

Isn't this one of the things we've been working toward?

How did that happen?

Sometime earlier this year, Frank Grieshaber in Heidelberg rolled out web page listings and corresponding Atom feeds of recently changed content in the EDH (e.g., latest updates to the inscriptions database). I added them, along with similar data-oriented feeds, to a feed aggregator I dubbed Planet Potamos (with "Potamos" trying lamely to evoke a rushing river of data; the "Planet" acknowledges the feed aggregation software I use). I put the same feed subscriptions into my personal feed reader (I could also have put the Potamos aggregator's feed, but it only updates periodically and I'm an immediacy junkie). I installed and configured my feed reader on every device I use.

The rest is magic. Magic made the old-fashioned way by lots of people in many different places and times developing standards, building software, creating data, doing research, and sharing.

What next?

Well, I hope that Frank and his colleagues in Heidelberg will eventually add thumbnail images (where they have them) to the EDH feeds. I hope that the other epigraphic databases (and indeed all kinds of ancient studies web applications) will set up similar feeds. I hope that we can all start using more linked-data approaches in and alongside such feeds in order to communicate seminal interpretive/discovery facets (like geography, personography, temporality and genre) in machine-actionable ways. I hope the spirit and practice of openness that lubricates and accelerates this sort of synergy continues to grow and flower.

As for me, I'm thinking about how I might set up some kind of filtering mechanism that would highlight or prioritize content in my feed reader that's potentially relevant to my (e.g.) geo/map/survey vocabulary interests. Hmmmmm....

Citing Sources in Digital Annotations

I'm collaborating with other folks both in and outside ISAW on a variety of digital scholarly projects in which Linked Open Data is playing a big role. We're using the Resource Description Framework (RDF) to provide descriptive information for, and make cross-project assertions about, a variety of entities of interest and the data associated with them (places, people, themes/subjects, creative works, bibliographic items, and manuscripts and other text-bearing objects). So, for example, I can produce the following assertions in RDF (using the Terse RDF Triple Language, or TuRTLe):

<http://syriaca.org/place/45> a <http://geovocab.org/spatial#Feature> ;
  rdfs:label "Serugh" ;
  rdfs:comment "An ancient city where Jacob of Serugh was bishop."@en ;
  foaf:primaryTopicOf <http://en.wikipedia.org/wiki/Suruç> ;
  owl:sameAs <http://pleiades.stoa.org/places/658405#this> .

This means: 'There's a resource identified with the Uniform Resource Identifier (URI) "http://syriaca.org/place/45" about which the following is asserted:

• it is a "Feature" as defined in the NeoGeo Spatial Ontology;
• the human-readable version of its name is "Serugh";
• a human-readable description (in the English language) of it is "An ancient city where Jacob of Serugh was bishop.";
• it is the primary topic of a document that is identified by the URI "http://en.wikipedia.org/wiki/Suruç"; and
• it is the same resource as that identified by another URI: "http://pleiades.stoa.org/places/658405#this".'

(Folks familiar with what Sean Gillies has done for the Pleiades RDF will recognize my debt to him in the what proceeds.)

But there are plenty of cases in which just issuing a couple of triples to encode an assertion about something isn't sufficient; we need to be able to assign responsibility/origin for those assertions and to link them to supporting argument and evidence (i.e., standard scholarly citation practice). For this purpose, we're very pleased by the Open Annotation Collaboration, whose Open Annotation Data Model was recently updated and expanded in the form of a W3C Community Draft (8 February 2013) (the participants in Pelagios use basic OAC annotations to assert geographic relationships between their data and Pleiades places).


A basic OADM annotation uses a series of RDF triples to link together a "target" (the thing you want to make an assertion about) and a "body" (the content of your assertion). You can think of them as footnotes. The "target" is the range of text after which you put your footnote number (only in OADM you can add a footnote to any real, conceptual, or digital thing you can identify) and the "body" is the content of the footnote itself. The OADM draft formally explains this structure in section 2.1. This lets me add an annotation to the resource from our example above (the ancient city of Serugh) by using the URI "http://syriaca.org/place/45" as the target of an annotation) thus:

<http://syriaca.org/place/45/anno/desc6> a oa:Annotation ;
  oa:hasBody <http://syriaca.org/place/45/anno/desc6/body> ;
  oa:hasTarget <http://syriaca.org/place/45> ;
  oa:motivatedBy oa:describing ;
  oa:annotatedBy <http://syriaca.org/editors.xml#tcarlson> ;
  oa:annotatedAt "2013-04-03T00:00:01Z" ;
  oa:serializedBy <https://github.com/paregorios/srpdemo1/blob/master/xsl/place2ttl.xsl> ;
  oa:serializedAt "2013-04-17T13:35:05.771-05:00" .

<http://syriaca.org/place/45/anno/desc6/body> a cnt:ContentAsText, dctypes:Text ;
  cnt:chars "an ancient town, formerly located near Sarug."@en ;
  dc:format "text/plain" ;

I hope you'll forgive me for not spelling that all out in plain text, as all the syntax and terms are explained in the OADM. What I'm concerned about in this blog post is really what the OADM doesn't explicitly tell me how to do, namely: show that the annotation body is actually a quotation from a published book. The verb oa:annotatedBy lets me indicate that the annotation itself was made (i.e., the footnote was written) by a resource identified by the URI "http://syriaca.org/editors.xml#tcarlson". If I'd given you a few more triples, you could have figured out that that resource is a real person named Thomas Carlson, who is one of the editors working on the Syriac Reference Portal project. But how do I indicate (as he very much wants to do because he's a responsible scholar and has no interest in plagiarizing anyone) that he's deliberately quoting a book called The Scattered Pearls: A History of Syriac Literature and Sciences? Here's what I came up with (using terms from Citation Typing Ontology and the DCMI Metadata Terms):

<http://syriaca.org/place/45/anno/desc7/body> a cnt:ContentAsText, dctypes:Text ;
  cnt:chars "a small town in the Mudar territory, between Ḥarran and Jarabulus. [Modern name, Suruç (tr.)]"@en ;
  dc:format "text/plain" ;
  cito:citesAsSourceDocument <http://www.worldcat.org/oclc/255043315> ;
  dcterms:biblographicCitation  "The Scattered Pearls: A History of Syriac Literature and Sciences, p. 558"@en .

The addition of the triple containing cito:citesAsSourceDocument lets me make a machine-actionable link to the additional structured bibliographic data about the book that's available at Worldcat (but it doesn't say anything about page numbers!). The addition of the triple containing dcterms:bibliographicCitation lets me provide a human-readable citation.

I'd love to have feedback on this approach from folks in the OAC, CITO, DCTERMS, and general linked data communities. Could I do better? Should I do something differently?

---

The SRP team is currently evaluating a sample batch of such annotations, which you're also welcome to view. The RDF can be found here. These files are generated from the TEI XML here using the XSLT here.

New in Potamos: New Testament Virtual Manuscript Room

I've just added to the Potamos Atlantis feed aggregator the "Recent Additions feed" from the New Testament Virtual Manuscript Room of the Münster Institut für Neutestamentliche Textforschung.

New in Potamos: tDAR

I've just added the update feed from The Digital Archaeological Record (tDar) to the Potamos Atlantis feed aggregator.

New in Potamos

I have added the following data feeds to the Potamos Atlantis feed aggregator:

• American Numismatic Society database (Greek, Roman, and Byzantine departments)
• Online Coins of the Roman Empire (OCRE)
• OpenContext

New feed aggregator for ancient data

I've added a new feed aggregator to the Planet Atlantides constellation. It's called Potamos, and it's dedicated to aggregating feeds of data from web sites about the ancient world (photos, archaeological records, bibliographic records, spatial data, and the like, rather than blog posts, news items, or event announcements).

I've listed the modest number of feeds it's presently pulling together below; if you know of a web resource whose feed (Atom or RSS) you think should be added, please send me a link and I'll have a look.

• Ancient World Image Bank Pool (Flickr) [page | feed]
• Ancient World Open Bibliographies (Zotero) [page | feed]
• Chiron, CC images of the Classical World Pool (Flickr) [page | feed]
• Chronique des fouilles en ligne / Archaeology in Greece Online [page | feed]
• Epigraphische Datenbank Heidelberg: Inschriften: Letzte Änderungen [page | feed]
• Epigraphische Datenbank Heidelberg: Fotos: Letzte Änderungen [page | feed]
• Epigraphische Datenbank Heidelberg: Publikationen: Letzte Änderungen [page | feed]
• Papyri.info: Recent Commits to idp.data:master (Github) [page | feed]
• Pleiades Places Pool (Flickr) [page | feed]
• Pleiades: Recently Modified [page | feed]
• Portable Antiquities Scheme Database: Latest Finds [page | feed]
• Visibile Parlare - Greek inscriptions Pool (Flickr) [page | feed]
• Visible words - Visibile parlare Pool (Flickr) [page | feed]