Using OpenRefine with Pleiades

This past summer, DC3's Ryan Baumann developed a reconciliation service for Pleiades. He's named it Geocollider. It has two manifestations:

• Upload a CSV file containing placenames and/or longitude/latitude coordinates, set matching parameters, and get back a CSV file of possible matches.
• An online Application Programming Interface (API) compatible with the OpenRefine data-cleaning tool.

The first version is relatively self-documenting. This blog post is about using the second version with OpenRefine.

Reconciliation

I.e., matching (collating, aligning) your placenames against places in Pleiades.

Running OpenRefine against Geocollider for reconciliation purposes is as easy as:

• Download and install OpenRefine.
• Follow the standard OpenRefine instructions for "Reconciliation," but instead of picking the pre-installed "Wikidata Reconciliation Service," select the "Add standard service..." button and enter "http://geocollider-sinatra.herokuapp.com/reconcile" in the service URL dialog, then select the "Add Service" button.

When you've worked through the results of your reconciliation process and selected matches, OpenRefine will have added the corresponding Pleiades place URIs to your dataset. That may be all you want or need (for example, if you're preparing to bring your own dataset into the Pelagios network) ... just export the results and go on with your work. 

But if you'd like to actually get information about the Pleiades places, proceed to the next section.

Augmentation

I.e., pulling data from Pleiades into OpenRefine and selectively parsing it for information to add to your dataset.

Pleiades provides an API for retrieving information about each place resource it contains. One of the data formats this API provides is JSON, which is a format with which OpenRefine is designed to work. The following recipe demonstrates how to use the General Refine Expression Language to extract the "Representative Location" associated with each Pleiades place. 

Caveat: this recipe will not, at present, work with the current Mac OSX release of OpenRefine (2.7), even though it should and hopefully eventually will.  It has not been tested with the current releases for Windows and Linux, but they probably suffer from the same limitations as the OSX release. More information, including a non-trivial technical workaround, may be had from OpenRefine Issue 1265. I will update this blog post if and when a resolution is forthcoming.

1. Create a new column containing Pleiades JSON. 

Assuming your dataset is open in an OpenRefine project and that it contains a column that has been reconciled using Geocollider, select the drop-down menu on that column and choose "Edit column" -> "Add column by fetching URLs ..."

In the dialog box, provide a name for the new column you are about to create. In the "expression" box, enter a GREL expression that retrieves the Pleiades URL from the reconciliation match on each cell and appends the string "/json" to it:

cell.recon.match.id + "/json"

OpenRefine retrieves the JSON for each matched place from Pleiades and inserts it into the appropriate cell in the new column. 

2. Create another new column by parsing the representative longitude out of the JSON.

From the drop-down menu on the column containing JSON, select "Edit column" -> "Add column based on this column..."

In the dialog box, provide a name for the new column. In the expression box, enter a GREL expression that extracts the longitude from the reprPoint object in the JSON:

value.parseJson()['reprPoint'][0]

Note that the reprPoint object contains a two-element list, like:

[ 37.328382, 38.240638 ]

Pleiades follows the GeoJSON specification in using the longitude, latitude ordering of elements in coordinate pairs so, to get the longitude, you use the index (0) for the first element in the list.

3. Create a column for the latitude

Use the method explained in step 2, but select the second list item from reprPoint (index=1).

4. Carry on ...

Your data set in OpenRefine will now look something like this:

Scheduled Votes on Bills in Congress, Week of 30 January 2017

I'll try to make these better in future (e.g., with actionable links and headers and such), but right now I figured it was best to paste as plaintext in order to avoid a bunch of interstitial tracking links that IFTTT and gmail inserted.

H.R. 347 - DHS Acquisition Documentation Integrity Act of 2017

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 347 - DHS Acquisition Documentation Integrity Act of 2017

Sponsor
Rep. Bonnie Watson Coleman

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 560 - To amend the Delaware Water Gap National Recreation Area Improvement Act to provide access to certain vehicles serving residents of municipalities adjacent to the Delaware Water Gap National Recreation Area, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 560 - To amend the Delaware Water Gap National Recreation Area Improvement Act to provide access to certain vehicles serving residents of municipalities adjacent to the Delaware Water Gap National Recreation Area, and for other purposes.

Sponsor
Rep. Tom Marino

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 58 - First Responder Identification of Emergency Needs in Disaster Situations

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 58 - First Responder Identification of Emergency Needs in Disaster Situations

Sponsor
Rep. Sheila Jackson Lee

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 276 - A bill to amend title 49, United States Code, to ensure reliable air service in American Samoa.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 276 - A bill to amend title 49, United States Code, to ensure reliable air service in American Samoa.

Sponsor
Del. Aumua Amata

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 366 - DHS SAVE Act

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 366 - DHS SAVE Act

Sponsor
Rep. Scott Perry

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 437 - Medical Preparedness Allowable Use Act

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 437 - Medical Preparedness Allowable Use Act

Sponsor
Rep. Gus Bilirakis

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 505 - Border Security Technology Accountability Act of 2017

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 505 - Border Security Technology Accountability Act of 2017

Sponsor
Rep. Martha McSally

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 526 - Counterterrorism Advisory Board Act of 2017

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 526 - Counterterrorism Advisory Board Act of 2017

Sponsor
Rep. John Katko

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 549 - Transit Security Grant Program Flexibility Act

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 549 - Transit Security Grant Program Flexibility Act

Sponsor
Rep. Daniel Donovan

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 584 - Cyber Preparedness Act of 2017

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 584 - Cyber Preparedness Act of 2017

Sponsor
Rep. Daniel Donovan

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 612 - United States-Israel Cybersecurity Cooperation Enhancement Act of 2017

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 612 - United States-Israel Cybersecurity Cooperation Enhancement Act of 2017

Sponsor
Rep. Jim Langevin

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 642 - To amend the Homeland Security Act of 2002 to enhance the partnership between the Department of Homeland Security and the National Network of Fusion Centers, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 642 - To amend the Homeland Security Act of 2002 to enhance the partnership between the Department of Homeland Security and the National Network of Fusion Centers, and for other purposes.

Sponsor
Rep. Lou Barletta

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 655 - To amend the Homeland Security Act of 2002 to establish the Securing the Cities program to enhance the ability of the United States to detect and prevent terrorist attacks and other high consequence events utilizing nuclear or other rad...

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 655 - To amend the Homeland Security Act of 2002 to establish the Securing the Cities program to enhance the ability of the United States to detect and prevent terrorist attacks and other high consequence events utilizing nuclear or other radiological materials that pose a high risk to homeland security in high-risk urban areas, and for other purposes.

Sponsor
Rep. Daniel Donovan

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 665 - To modernize and enhance airport perimeter and access control security by requiring updated risk assessments and the development of security strategies, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 665 - To modernize and enhance airport perimeter and access control security by requiring updated risk assessments and the development of security strategies, and for other purposes.

Sponsor
Rep. William Keating

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 666 - To amend the Homeland Security Act of 2002 to establish the Insider Threat Program, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 666 - To amend the Homeland Security Act of 2002 to establish the Insider Threat Program, and for other purposes.

Sponsor
Rep. Pete King

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 677 - To amend the Homeland Security Act of 2002 to establish chemical, biological, radiological, and nuclear intelligence and information sharing functions of the Office of Intelligence and Analysis of the Department of Homeland Security and...

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 677 - To amend the Homeland Security Act of 2002 to establish chemical, biological, radiological, and nuclear intelligence and information sharing functions of the Office of Intelligence and Analysis of the Department of Homeland Security and to require dissemination of information analyzed by the Department to entities with responsibilities relating to homeland security, and for other purposes.

Sponsor
Rep. Martha McSally

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 678 - To require an assessment of fusion center personnel needs, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 678 - To require an assessment of fusion center personnel needs, and for other purposes.

Sponsor
Rep. Martha McSally

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 687 - To amend the Homeland Security Act of 2002 to establish a process to review applications for certain grants to purchase equipment or systems that do not meet or exceed any applicable national voluntary consensus standards, and for other...

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 687 - To amend the Homeland Security Act of 2002 to establish a process to review applications for certain grants to purchase equipment or systems that do not meet or exceed any applicable national voluntary consensus standards, and for other purposes.

Sponsor
Rep. Donald Payne

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 690 - To amend the Homeland Security Act of 2002 to enhance certain duties of the Domestic Nuclear Detection Office, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 690 - To amend the Homeland Security Act of 2002 to enhance certain duties of the Domestic Nuclear Detection Office, and for other purposes.

Sponsor
Rep. Cedric Richmond

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

H.R. 697 - To amend the Homeland Security Act of 2002 to improve the management and administration of the security clearance processes throughout the Department of Homeland Security, and for other purposes.

The House of Representatives is scheduled to vote the week of January 30th, 2017 on H.R. 697 - To amend the Homeland Security Act of 2002 to improve the management and administration of the security clearance processes throughout the Department of Homeland Security, and for other purposes.

Sponsor
Rep. Bennie Thompson

via Sunlight Foundation http://ift.tt/2jJ4Mmz For more information on a bill or resolution: http://ift.tt/1c1EyNR To contact your Representative: http://ift.tt/P3cvWa To contact your Senator: http://ift.tt/y4GfE9 January 30, 2017 at 05:13AM
via ProPublica http://docs.house.gov/floor/Default.aspx?date=2017-01-30

On Hidden Figures

This weekend I finally got a chance to see Hidden Figures. The large theater was two-thirds full or so, mostly white (as one might expect in South Huntsville), mostly women and girls (you missed out, dudes, it was great). 

There was a profound mix of audience laughter and silence throughout. At the end of the historical/photographic epilogue, there was a moment of silence into which a female voice clearly whispered: "That was amazing!" Then LOUD APPLAUSE. 

That's the world I thought I was living in. I want it back. For real.

Let's go get it.

Paying for the wall

It's my opinion that the Trump team never imagined that Mexico would "pay for the wall." What's always been meant is "I will make those Mexicans pay for the wall. I will make them pay."

So, the plan is surely trade sanctions and more. The real point of the whole thing will never be cost recovery (nor true border security). It's ritualized combat display: the leader publicly acting out his supposed manly vigor and strength of will by exercising the organs of state symbolically to humiliate, impoverish, emasculate, feminize, and punish a straw-man external enemy erected for the purpose. He will try to supplant the reality of the Mexican people's complex humanity, society, and shared-with-us economy with a fictionally puny, dirty, criminal, and base caricature. He will ostentatiously pummel that piñata in order to give the masses of his supporters a vicarious victory and a sham sense of security, a warm spatter of fake enemy blood, while his regime, less watched, loots our inheritance and undermines our liberties, doubling down on the very sorts of injustices that underlie the rage against government and establishment that helped put him in office in the first place.