Aurlog » libraries

Cataloguing coding

Orangeaurochs — Sun, 08 Jan 2012 16:50:00 +0000

I am not a trained programmer, coding is not part of my job description, and I have little direct access to cataloguing and metadata databases at work outside of normal catalogue editing and talking to the systems team, but I thought it might be worth making the point of how useful programming can be in all sorts of little ways. Of course, the most useful way is in gaining an awareness of how computers work, appreciating why some things might be more tricky than others for the systems team to implement, seeing why MARC21 is a bastard to do anything with even if editing it in a cataloguing module is not really that bad, and how the new world of FRDABRDF is going to be glued together. However, some more practical examples that I managed to cobble together include:

Customizing Classification Web with Greasemonkey. This is a couple of short scripts using Javascript, which is what the default Codeacademy lessons use. Javascript is designed for browers and is a good one to start with as you can do something powerful very quickly with a short script or even a couple of lines (think of all the 90s image rollovers). It’s also easy to have a go if you don’t have your own server, or even if you’re confined to your own PC.
Aleph-formatted country and language codes. I wrote a small PHP script to read the XML files for the MARC21 language and country codes and convert them into an up to date list of preferred codes in a format that Aleph can read, basically a text file which needs line breaks and spaces in the right places. It is easy to tweak or run again in the event of any minor changes. I don’t have this publicly available anywhere though. PHP is not the most elegant language but is relatively easy to dip into if you ever want to go beyond Javascript and do more fancy things, although it can be harder to get access to a server running PHP.
MARC21 .mrc file viewer. I occasionally need to quickly look at raw .mrc files to assess their quality and to figure out what batch changes we want to make before importing them into our catalogue. This is an attempt to create something that I could copy and paste snippets of .mrc files into for a quick look. It is written in PHP and is still under construction. There are other better tools for doing much the same thing to be honest, but coding this myself has had the advantages of forcing me to see how a MARC21 file is put together and realising how fiddly it can be. Try this with an .mrc which has some large 520 or 505 fields in it (there are some zipped ones here, to pick at random) and watch the indicators mysteriously degrade thereafter. I will get to the bottom of this…

The following examples are less useful for my own practical purposes but have been invaluable for learning about metadata and cataloguing, in particular, RDF/linked data. I was very interested in LD when I first heard about it. Being able to actually try something out with it (even if the results are not mind-blowing) rather than just read about it, has been very useful. Both are written in PHP and further details are available from the links:

Lodopac: Linked Open Data OPAC. My entry for the UK Discovery Developer Competition.
In Our Time booklist. A forerunner to Lodopac, written in response to an idea raised by psychemedia for the Open Bibliographic Data Challenge: the BBC “In Our Time” Reading List. Sadly it no longer works as the BNB service it relies upon was taken down.

Nothing to do with cataloguing, but what I am most proud of is this, written in Javascript: Cowthello. Let me know if you beat it.

Lodopac example searches

Orangeaurochs — Thu, 15 Sep 2011 09:03:18 +0000

Yay, my entry for the Discovery & DevCSI Developers Competition- Lodopac- was awarded a commendation for its use of the Cambridge University Library (CUL) dataset. During the judging I was asked for searches which were known to work well- the timeout issues I discussed under Limitations being not insignificant, especially with author or title searches. I submitted a version of the following brief general notes which I hope are helpful to anyone else who wants to play:

The British National Bibliography (BNB) server is generally more responsive than the Cambridge University Library one; title seems to work better than author. The following are hopefully useful examples useful:

author=”fisher” seems to work (at least in BNB) so long as the number of hits to return is small (e.g. 5).
title=”fisher” works well for both.
ISBN=”0709039425″ should very quickly give you a single hit in both BNB and CUL.

I would really like to try and think of ways of improving free text regular expression search times for things like author and title in Sparql* although I doubt there is one that doesn’t rely on the configuration, processing power, or indexing of the server being searched.

* thinking aloud, some ideas might include: downloading a larger imprecise set for further local searching (e.g. for an author/title search downloading the title matches and searching the authors locally: although this would also be slow, it would get round the timeout at least); forcing a look-up in a controlled vocab first in order to get an exact string match (esp for authors, although even if this is possible, this forces a user to do more work, which isn’t the point); local indexing of the triple store (this is probably the best way but I’m not sure how to go about it, whether I really have the server capabilities to do it, and can be committed to the updating required).

Sparql recipes for bibliographic data

Orangeaurochs — Fri, 05 Aug 2011 14:06:00 +0000

One of the difficulties in searching RDF data is knowing what the data looks like. For instance, finding a book by its title means knowing something about what how a dataset has recorded the relationship between a book and its title. There is no real standard for publishing MARC/AACR2-style bibliographic data as RDF: it seems libraries publishing RDF are approaching this largely individually, although they are using many of the same vocabularies, dc, bibo, etc. This was one reason why I wanted to create Lodopac: to present some kind of interface so that searchers didn’t need to know these different models but could start to explore them. Below are the Sparql recipes for the different search criteria I used for the BNB and the Cambridge University Library datasets, so they can be compared, re-used, or corrected. All examples use prefixes, which are defined anew in each example. The examples are of course fragments and don’t have all the necessary SELECT and WHERE clauses.

By the way, for an excellent Sparql tutorial with ample opportunity to play as you go along, do have a look at the Cambridge University Library’s SPARQL tutorial. It also gives clues to the way their data is structured. Of use for the BNB is their data model (PDF), which is not nearly as scary as it looks at first, and incredibly helpful.

Author keyword search

This would be relatively straightforward-the unavoidable regular expression being the main complication- but for the fact that the traditional author/editor/etc of bibliographic records can be found in dc:creator as well as dc:contributor which necessitates a UNION. The BNB used foaf:name:

PREFIX dc:
PREFIX foaf:

{?book dc:creator ?author} UNION {?book dc:contributor ?author} .
?author foaf:name ?name .
FILTER regex(?name, “author”, “i”) .

Cambridge uses much the same recipe except that it uses rdfs:label instead of foaf:name:

PREFIX dc:
PREFIX rdfs:

{?book dc:creator ?author} UNION {?book dc:contributor ?author} .
?author rdfs:label ?name .
FILTER regex(?name, “author”, “i”) .

Title keyword searches

This is more straightforward and is in fact the same for both the BNB and Cambridge University Library:

PREFIX dc:

?book dc:title ?title .
FILTER regex(?title, “title”, “i”) .

Date of publication (year)

I imagined this one being simple and for Cambridge University Library it is. However the BNB took some unravelling as they have modelled publication as an event related to a book. The various elements of publication are then related to the event. So, for the BNB we have this:

PREFIX bibliographic:
PREFIX event:
PREFIX rdfs:

?book bibliographic:publication ?pub .
?pub event:time ?year .
?year rdfs:label “date” .

By contrast, Cambridge University Library has it in one line:

PREFIX dc:

?book dc:created “date” .

ISBN

As an identifier, ISBN is relatively straightforward in both models, although care must be taken with the BNB as 10 and 13 digit ISBNs are treated as separate properties and the following assumes that the search will cover both:

PREFIX bibo:

{?book bibo:isbn10 “isbn”} UNION {?book bibo:isbn13 “isbn”} .

For Cambridge University Library, also using the bibo ontology, this is:

PREFIX bibo:

?book bibo:isbn “isbn” .

Conclusion

I didn’t set up to provide ground-breaking conclusions. However, it is remarkable how different data models can be formulated for modelling the same type of data by similar organisations. The real question is whether this is a good, a bad thing, or doesn’t really mattter. Will it need to be standardised? My understanding of how this works is probably not. I think the days of monolithic library standards are probably now gone. I wonder, for instance, if there ever will be a single MARC22 (or whatever you like to call it) and doubt RDA will ever completely replace AACR2 in the way we imagine. What will emerge I suspect will be a soup of various standards and data models, some of which will be more prevalent. One thing I picked up from various linked data talks is that information has frequently been published then re-used in ways that the issuers never imagined; if that is the case, the precise modelling and format is probably not as important as the fact that it is of good quality and intelligently put together. The BNB and Cambridge University Library models are clearly quite different but quite capable of being mapped and used despite this.

If there are any other bibliographic data Sparql endpoints, I would like to include them in a future version of the Lodopac search. Do let me know if you come across them.

More mundanely, do say if there are errors in my Sparql recipes or if there are ways they could be done more efficiently.

Lodopac : simple Linked Open Data OPAC

Orangeaurochs — Sun, 31 Jul 2011 20:34:25 +0000

Lodopac is my entry to the UK Discovery Developer Competition. Aside from obvious mocking of the name, comments on Lodopac are very welcome. If anyone installs it locally, I’d also be very interested to know.

About Lodopac

Lodopac is a simple linked open data opac using Sparql to search remote bibliographic RDF data. By default it is set up to search the BNB and Cambridge University Library datasets, but is designed to allow easy setup of additional datasets with Sparql endpoints (see Installation, source code, and configuration below). It was written in response to the UK Discovery Developer Competition.

The purpose of Lodopac is to provide a simple standard OPAC-style interface to perform searches of various bibliographic RDF datasets without having to know how to formulate a Sparql query and without having to know the structure of the database. I hope it is especially useful for people wanting to get a grip on how bibliographic RDF is put together, what it looks like, and what a Sparql query looks like. For example, an author search is possible without knowing about dc:creator and dc:contributor, or how these need to be linked together in a Sparql search. Similarly, a searcher wouldn’t need to worry about how to construct date searches in different datasets. For the BNB and CUL, these are very different (three lines in the BNB, one for CUL), but in Lodopac, there is only search box to search both. Lodopac displays the Sparql query it constructed to perform the search, as well as the combined RDF for all results found in XML, JSON, N3, and TTL.

How to search Lodopac

Select one or more of the available datasets using the checkboxes.

Author and Title searches are free text phrase searches. In other words, a string you search for will match with any exact match, including spacing and punctuation, and in the middle of words. E.g. searching for “shake” will match “Shakespeare”, “milkshakes”, and “More hits that you can shake a stick at”. Searching is case insensitive. The following punctuation is removed from searches: \”‘<>$^%.

You are strongly advised to keep author and title searches simple: e.g. one word of a title or a surname only.

ISBN searches 10 or 13 digit ISBNs. Any dashes or other non-digit characters are stripped from the search.

Date search will accept a year.

N.B. Keep searches as simple as possible, especially with author and title searches, to avoid them timing out. ISBN and date searches are generally quicker.

Limitations

A bad workman blames his tools and I’m no exception. The greatest limitation is the time taken by Sparql endpoints to perform a Sparql query, especially one that involves a regular expression, such as the Author and Title searches. What is needed is some more robust indexing or some cheat like Virtuoso’s unorthodox bif:contains, which the old version of the linked data BNB used. I touched on this in a blog post about the In Our Time Booklist script I wrote (see section 6).

The load and current capacity on the Sparql endpoints at the time the query is made is another important factor. A search which times out one minute can work fine the next.

The search options are obviously limited but do I hope represent the most common methods of searching normal library catalogues aside from, of course, a general keywords search. The manipulation of results is also rather sparse but allows click through to full data associated with a book, the structure and contents of which can be more fully explored. The aggregation of RDF data in various formats is I hope useful illustratively as well as having potential for further manipulation.

Installation, source code, and configuration

The source code for Lodopac is available as a zip file, which contains all the necessary PHP, Javascript, and CSS files. In addition, you will need to install ARC2, which makes the Sparql queries and manipulates the resultant RDF. Edit the first line of lodopac.php so that it points at your local installation of ARC2.

The programme is basically one long script- there is only one page- but is split for convenience of editing. The key file is lodopac.php which includes the other files as it goes along. The main core of the script which builds the queries and does the searching is all in lodopac.php.

I have attempted to make the script as easily configurable as possible so that additional Sparql endpoints can be added. There are probably more components hard-coded into the script that I have overlooked, but all the setup for the endpoints is in the file setup_endpoints.php. The first part of this file is a list of necessary prefixes that are needed for any possible queries from any of the endpoints and, although not ideal, all these prefixes are sent with any Sparql query. Following that and the declaration of an array of the endpoints, each endpoint has a dedicated block with the information added to a hash. To add another endpoint, duplicate a block and configure the search recipes as appropriate. The keys marked “brief_” are used to fetch information for the brief results display. I have conspicuously chickened out of providing an author and the attendant main entry and multiple author headaches involved.

Customizing Classification Web with Greasemonkey

Orangeaurochs — Fri, 24 Jun 2011 10:51:39 +0000

Classification Web is ace, but there are a couple of things about the interface that annoy me and, in one colleague’s case, seriously put him off using it, in particular:

The opening of a new tab/window when you click on the MARC view for a subject or name.
The confusing menu. We don’t use LCC or DDC, and the browse options don’t really add much, so we only really need two options: Search LC Subject Headings and Search LC Name Headings.

I managed to work out a simple way of modifying how Classification Web works on Firefox using the Greasemonkey add-on and a couple of simple scripts, all of which is quick and easy to install:

Install Greasemonkey: https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/
Make sure the monkey in the bottom-right corner is happy and colourful. Click on it if not.
If you want to prevent the MARC view opening a new window, install the classweb_no_new_ window script by going to http://www.aurochs.org/zlib/js/userjs/classweb_no_new_window.user.js then
Click on the Install button
If you want to reduce the main menu, install classweb_prune_menu script by going to http://www.aurochs.org/zlib/js/userjs/classweb_prune_menu.user.js then
Click on Install button
Reload/refresh Classweb if it’s still open and it should work.

If you want to turn Greasemonkey off altogether, click on the monkey so he’s sad and grey. If you want to stop individual scripts, right click on the monkey, click on Manage User Scripts, select a script from the list, and un-tick the Enabled box in the lower left corner.

These instructions were tested on Firefox 3.5.3 although I imagine they would be fine on any recent version of Firefox. I would be interested to hear anything confirming or undermining that assertion.

If you’re happy to play around, these scripts can be further altered. In particular, you can choose which menu items appear in the pruned menu script:

Right click on the monkey
Click on Manage User Scripts
Select classweb_prune_menu from the list
Click on Edit (you will probably have to select a text editor at this point)
Edit the list of pages under the line var menu_items_to_keep = Array (. Enter each page you want to appear on the menu on a separate line in quotes, with a comma at the end of each line except the last line. The menu item must appear exactly as it does on the Classification Web menu, including capitals. E.g., the default set up looks like this:
```
var menu_items_to_keep = Array ( // end each line with a comma except the last line
  "Search LC Subject Headings",
  "Search LC Name Headings"
);
```
Save the file, and reload Classification Web.

If anyone else finds this useful or can think of more customizations let me know.

RDA as a closed standard

Orangeaurochs — Wed, 23 Mar 2011 12:16:32 +0000

Resource Description and Access (RDA), the new bibliographic standard to replace AACR2, was released in 2010 on the web as a closed standard sitting behind a paywall. This really worries me. I strongly believe it should be an open standard.

What do I mean by closed?
By closed I basically mean that you have to pay or subscribe to access it. In many ways, this is not dissimilar to AARC2. For decades, libraries (and individuals) paid for various editions of AACR2, which has always been primarily a print product, as well as for various updates when it changed to looseleaf format. Recently it has also been available on the web via Cataloger’s Desktop. RDA is primarily a web product via the RDA Toolkit, although a concession was eventually made to release it in print as well.

An open standard would be one that, according to Wikipedia, “is publicly available” even if it “has various rights to use associated with it”. This would be one which any cataloguer, librarian, or crucially, non-librarian, could see and benefit from. A practical definition for the purposes of this post would be a standard I could go and look at right now without subscription. I can if I wish apply the standard without hindrance, I can assess it with ease, and, ideally, build on the standard without restriction. RDA is not open, although to be fair, a part of RDA has been released openly, the element set and vocabularies.

Other open standards
Open (publicly available) standards are quite common. Some well known-examples:

Data: XML, Dublin Core, RDF, MODS, MADS, MARC21, ISBD
Documents: HTML, PDF, ODF,
Images: PNG

The following are open although I’d have a lawyer to hand if you wanted to do anything with them:

Documents: DOC,
Images: JPG, GIF

Some closed ones for comparision:

Data: AACR2, RDA

These are of course more subjective lists than they look, but you get the idea. The closed list was actually bigger until upon examination I found that JPG, GIF, and even MS Office standards are publicly readable even if I’m not sure what more you could legally do with them. I’d be happy to add more to the closed list to balance things out a little.

Why is RDA not open?
Money. This is a delicate matter that I don’t want to delve into too much although it is obviously central to the openness of the standard. It’s also hard to talk about without appearing to make wild assertions, and I hope I haven’t been unfair. I’ve heard Alan Danskin of the JSC explaining that they’d thought about releasing RDA openly but that they had to cover costs. I’m not exactly sure what the costs of production were, although presumably included expenses and staff costs, and production of the product itself. The last is I think unfortunate as I would like to have seen a far simpler publication of RDA without all the bells and whistles, login barriers, and the need to learn a new interface as well as a new standard. Compare with the HTML4 standard which is a set of simple HTML documents with normal links. I don’t need to learn how to use that. Or, come to that, the MARC21 site. I wonder how much of the fee goes towards setting up and maintaining the RDA Toolkit platform.

With my tin foil hat on, I also wonder how much the fee is needed to resume revenue to the co-publishers since AACR2 has been in unrevised abeyance.

Why does it matter?
It matters because RDA (and with it all the high quality traditional cataloguing techniques) will not be widely used without being open. I think you can divide the potential RDA userbase as follows:

Libraries with enough money to switch to RDA
Libraries without enough money to switch to RDA
Non-libraries dealing with metadata

Those in group 1 will buy RDA, but some libraries- Group 2- will not see the benefit for the costs of conversion and training, let alone the costs of subscription. For ‘traditional’ cataloguing to thrive, therefore, we need to involve Group 3. However, those in Group 3 will not be able to even have a look at RDA to see if it meets their needs. I think RDA will be lucky to retain the same user base as AARC2, let alone break into new areas and influence the way other metadata is carried out. Those in the metadata community who, I suspect, have already been put off by AACR2, are unlikely to even try looking at RDA if it involves forking out a subscription.

I recently sat in a room with about 15 or so people mostly involved in metadata for institutional repositories and the like. During some discussions they flagged up two problems they were having: establishing a consistent form of name, and a standard set of data elements. I asked myself, would I recommend RDA to them to help solve these problems? Even if I thought it met their needs, could they even have a look to see if it did? No. They will either come up with their own solution or look elsewhere for it, which is already what they have been doing. I can’t see us taking more people with us, just a proportion of the people already using AACR2.

Openness also matters because haven’t a closed standard doesn’t reflect terrribly well on librarianship in general. I have a friend in IT who Laughed Out Loud* when I said the new library metadata standard was behind a paywall. In the new world of openness where even Microsoft loosely adhere to web standards, traditionally closed governments are leading the charge to release more data, and the world has been transformed by the the open standards of the web, are we to follow The Times behind a paywall? Personally I feel libraries, librarians, and library data should be at the forefront of openness, not grudgingly following behind or not following at all.

What could be done?
This is the nub of the matter. I’m no marketing expert and maybe I’m naive and there is nothing that could be done. However, working on the assumption that all that needs to be done is to break even and pay the costs of production for RDA, I would suggest the following ideas for a start:

Make a flashy web product anyway and charge lots more for it. Many more well-off libraries would pay for a product like the Toolkit if it’s good enough.
There is a need for a more accessible version or versions of RDA, e.g. just for books or in a more convenient format like, say, the Chan books on LCSH or the green editions of AACR2. The co-publishers could fulfill this need which I imagine would be easily done by re-using the data they already have.
Explanatory books. There are a number of these on the market or on the way already. The co-publishers could publish an official companion.
Consultancy and training. There is going to be a big demand for this soon in any case.
Involve more organisations in the drafting and publishing of RDA to share the costs, e.g. publishers, LMS vendors, commercial metadata suppliers, other metadata initiatives. I think it would be a positive and pragmatic move to have these parties on board anyway. They would be more likely to use the high quality standard produced and we would be more likely to be using metadata that meets all our purposes.

See Also
I notice a post covering some of these issues by carolslib from a few days ago. From the Catalogs of Babes also has a similar post, RDA: why it won’t work, from a few weeks ago which much more succinctly makes some of the same points:

Many librarians are balking at the cost of implementing RDA, I think rightfully so, although not for the same reasons. I’m not bitching about it because it’s unaffordable for smaller libraries, or because it’s a subscription rather than a one-time printed book cost (although I think those are valid points). I’m bitching because putting a dollar amount on something, now matter how low it is, will stop people from using something, especially if there’s a free alternative. In this case, I see the free alternative as ‘ignoring rules altogether and/or making you your own standards.’ Requiring a price makes adhering to standards–a key value-added service of libraries and librarians–inaccessible. Which is pretty ironic, considering that libraries are supposed to be all about access. We’re all proactive about offering access to our patrons, but we can’t extend that same philosophy to ourselves, to help us do a better job??

*** He literally LOL’d, although no ROFLing took place admittedly.

In Our Time booklist

Orangeaurochs — Wed, 16 Feb 2011 16:17:15 +0000

I have written a script which takes an unstructured reading list on the BBC’s In Our Time website, searches the British National Bibliography (BNB) using bibliographica for the books on the list, and returns structured metadata for the records it found.

This script was written in response to an idea raised by psychemedia for the Open Bibliographic Data Challenge: the BBC “In Our Time” Reading List:

The BBC “In Our Time” radio programme publishes suggested recommending reading in the programme data in an unstructured and citation style way: author, title (publisher, year), with what looks to be conventional character string separators between references (at least on the pages I looked at).

The idea is to extract and link suggested readings for the In Our time programmes to open, structured bibliographic data. This would make the In Our Time archive even more useful as an informal (open-ish) educational resource, especially as academic libraries start to release data relating to books used on courses. (So for example, this approach might help provide a link from a course to a relevant In Our Time broadcast via a common book.)

I was drawn to this idea as I like the idea of turning unstructured data into structured data: I have for example had some previous fun converting HTML pages into RSS feeds (e.g. CILIP Lisjobnet, Big Brother). I think something similar for any reading list (e.g. a Word document produced by a lecturer) would be an interesting idea.

The programme is written in PHP and is designed to be fired from a Javascript bookmarklet from a page on the In Our Time site, or by appending the In Our Time URL to the end of the URL for this page: http://www.aurochs.org/inourtime_booklist/inourtime_booklist_v1.php?. For example, to use it on the page for The Mexican Revolution (which I used a lot in testing), add the URL http://www.bbc.co.uk/programmes/b00xhz8d to produce http://www.aurochs.org/inourtime_booklist/inourtime_booklist_v1.php?http://www.bbc.co.uk/programmes/b00xhz8d.

The script follows the following steps:

Set up ARC2 to enable SPARQL searching and RDF processing
Extract Further Reading section
Separate out Raw Data for each book
Determine pattern used in citation then extract Basic Data, e.g. author, title, article, publication, using regular expressions
Further refine elements to make searching easier, i.e. one surname for author, only title proper for titles
Construct a SPARQL Query using author surname and title regular expressions pre-filtered for speed by a significant word using bif:contains
Filter Hits by date of publication
Obtain and display metadata from BNB

More details of these steps are below:

1. Set up ARC2 to enable SPARQL searching and RDF processing

ARC2 is a simple-to-use system for using RDF and SPARQL in PHP. I had previously played with it here when experimenting with creating my own RDF. The Sandy site uses SPARQL to populate the See Also sections.

2. Extract Further Reading section

A simple regular expression identifies the div in the HTML code that contains the reading list, which enables the next stage of the script to look for individual books.

3. Separate out Raw Data for each book

Another regular expression pulls out the paragraphs containing books and puts them in an array. You can see this by viewing the Raw Data.

4. Determine pattern used in citation then extract Basic Data, e.g. author, title, article, publication using regular expressions

As the In Our Time site does not use a single standard form of citation, the script has to try and determine which of several possible patterns a citation is using with regular expressions and extract the correct bits of data. This only works as well as it is possible to identify all the patterns, which effectively means looking at as many In Our Time pages as possible. This is one area that would certainly reward more work. It also points out how difficult it would be to extrapolate this into a script that could read any citations. The In Our Time booklist currently uses five citations each identified with a number (1, 2, 3, 4, 15, 5). If you look at the Book Data for a particular book you will see the citation style number given. The regular expressions capture author, title, and publication.

The author information in citations on In Our Time is unpredictable. Sometimes surnames are first, sometimes they are last. The citation patterns take care of this if possible and try to extract one significant name.

5. Further refine elements to make searching easier, i.e. one surname for author, only title proper for titles.

The script removes things like “(ed.)” from the author, which would obviously throw off a catalogue search. Subtitles- everything after and including semi-colons- are also removed from titles to lessen any chance of variation and lost matches.

6. Construct a SPARQL Query using author surname and title regular expressions pre-filtered for speed by a significant word using bif:contains

Constructing the SPARQL query was the most tricky part. Ignoring the various standard prefixes pilfered from the standard example, the most important part is the title search. There are three unsatisfactory options:

Match the extracted title directly to a dc:title. This doesn’t work as the cited title is unlikely to be exactly the same in all matters of words included, spacing, punctuation, etc.
Use bif:contains for keyword searching as used in the BNB SPARQL example. This is certainly quick, but has a number of drawbacks: it can only be used once for a single keyword (any one of the two significant words in The history of Mexico, for example, will produce a huge number of hits). It is also not standard SPARQL. I was happy to overlook this if it worked, but ARC2 didn’t like it at all until I worked out it has to be used in angle brackets e.g. ?title “Mexico”.
Use regular expressions (e.g. FILTER regex (?title, “The History of Mexico”, “i”)) for keyword searching. This is extremely powerful: you can easily construct searches but it is so slow as to routinely time out, so rendering it effectively useless.

The In Our Time script uses a combination of the last two techniques to get a result. First, it finds a significant word in the title, ideally the first four letter word after the first word (i.e. to avoid “The”, “A”, “That”, etc.) or, failing that, the first word. The SPARQL query then uses bif:contains to search for that word. The query then does a regular expression filter on the whole title. I don’t know if this is how SPARQL endpoints would generally work, but the BNB appears to only look for regular expression matches on the records already filtered by the bif:contains. In any case, it works.

In addition, the script also uses a regular expression to search by the author’s surname. It doesn’t search by date as the date of publication on BNB (dc:issued) is not in a standard format (e.g. “1994-01-01 00:00:00″, “2005 printing”, “c2006″). It is also not keyword searchable. You can see all the author-title hits with links to BNB records by viewing Hits.

7. Filter Hits by date of publication

You can, however, retrieve the date from the BNB and process it afterwards, which is what the script does. It finds the four digit year and compares this to the four digit year it found on the In Our Time site. You can see all the author-title-date hits with links to BNB records by viewing Date Hits. Perhaps rather arbitrarily, the first book in the resulting array is selected as the result.

8. Obtain and display metadata from BNB

When the search took place, the matching title, author (only one), and date is obtained from BNB. This title and author are displayed, as are the stripped down year of publication, and a link to the full BNB record. For records that returned no hits on the BNB, the Basic Data is simply regurgitated.

The script also downloads the full combined RDF for all the hits is displayed at the bottom of the page, viewable in a several formats.

Further work

I think a lot more work could be done on this given time, both to improve it and to extend it. In no particular order:

Make it more pretty. It is currently designed to look merely acceptable while I concentrated on functionality. I have also tried to show much of my working, which a finished version would obviously hide.
Refinement of the detection of citation style. This is probably the most critical improvement, and ultimately decides if this approach would be useful outside of In Our Time on other reading lists. There are more patterns that need to be added, especially for older pages.
Further preparation of data for searching. Currently, for example, a book on the Mexico reading list doesn’t return any hits because of the exclamation mark in “Zapata!”. This could be stripped, and there are lots of similar refinements no doubt.
More interesting/useful output. The script’s outputs are currently quite raw or basic as I concentrated on the mechanisms for pulling information from free text for automatic catalogue searching. It might be useful to output proper standalone RDF files, references in standard reference formats (e.g. Harvard) in HTML or text files, files in standard reference management formats, perhaps even MARC, and so on. Some of these would perhaps be fairly straightforward.
Links to catalogues or online bookshops so you could borrow or buy the books from the reading list based on ISBNs taken from the BNB record.
Searching more catalogues. If a search fails on the BNB, the script could search other open catalogues, e.g. the Cambridge catalogue.
Greasemonkey script or plugin so that a button appears next to the Further Reading section when you view an In Our Time page. This could even appear next to individual books. Ideally (pie in the sky) such a plugin would have a stab at finding books on any web page.
Other ways of firing the script not requiring manual addition to the URL or use of a bookmarklet, e.g. a searchbox of some kind (either accepting a URL as input or keywords for the titles of broadcasts).

Please do leave comments or questions.

ISKO-UK linked data day

Orangeaurochs — Fri, 10 Dec 2010 14:32:55 +0000

On 14 September I went to the ISKO-UK one day conference on Linked Data: the Future of Knowledge Organisation on the Web. For me, this followed on from a previous Talis session on Linked Data and Libraries I attended at the British Library in June, which I found really very interesting and informative.

The ISKO conference was a lot broader in scope- it was noticed by several speakers discussing the BBC’s use of linked data that there were 22 attendees from the BBC- and included talks about local and national government, business, libraries, as well as the Ordnance Survey. The following is a brief and personal overview, pausing in more detail over aspects that interested me more. It assumes a passing acquaintance with linked data and basic RDF.

Professor Nigel Shadbolt from the University of Southampton, a colleague of Tim Berners-Lee at Southampton as well as in advising the British Government developing the data.gov.uk site, opened with a talk about Government Linked Data: A Tipping Point for the Semantic Web. There were two interesting points from this (there were many, but you know what I mean). First was the speedy and incredible effects of openly releasing government data. Professor Shadbolt used the analogy of the mapping by John Snow of the 1854 cholera epidemic which identified a pump as the source and led to the realisation that water carried cholera. He mentioned the release of government bike accident data that was little used by the government but which was taken up and used by coders within days to produce maps of accident hotspots in London and guides to avoiding them.
The second point was the notion of the “tipping point” for the semantic web and linked data referred to in the talk’s title. Several speakers and audience members referred to the similar idea of the “killer implementation”, a killer app for the semantic web that would push it into the mainstream. The sheer quantity of data and use it is quickly put to, often beyond the imagination of those who created and initially stored it, was quite compelling. Richard Wallis made a similar point when discussing the relative position of the semantic web compared to the old-fashioned web in the 1990s. He noted that it is now becoming popular to the extent that is nearly impossible to realistically list semantic web sites and predicts that it will explode in the next year or so. Common to both Nigel Shadbolt’s and Richard Wallis’s talks was a feeling almost of evangelism: Richard Wallis explicitly refers to himself as a technology evangelist; Nigel Shadbolt referred to open government data as “a gift”. Despite being relatively long in the tooth, RDF, linked data, and all that have not yet taken off and both seemed keen to push it: when people see the benefits, it won’t fail to take off. There were interesting dissenting voices to this. Martin Hepp, who had spent over eight years coming up with the commercial GoodRelations ontology, was strongly of the opinion that it is not enough to merely convince people of the social or governmental benefits, but rather the linked data community should demonstrate that it can directly help commerce and make money. The fact that GoodRelations apparently accounts for 16% of all RDF triples in existence and is being used by corporations such as BestBuy and O’Reilly (IT publishers) seems to point to a different potential tipping point. Interestingly, Andreas Blumauer in a later talk said that SKOS (an RDF schema to be discussed in the next paragraph) could introduce Web 2.0 mechanisms to the ‘web of data’”. Perhaps, then, SKOS is the killer app for linked data (rather than government data or commercial data as suggested elsewhere), although Andreas Blumauer also agreed with Martin Hepp in saying that “If enterprises are not involved, there is no future for linked data”. In my own ignorant judgement, I would suggest government data is probably a more likely tipping point for linked data, closely followed by Martin Hepp’s commercial data. It is government data that is making people aware of linked data, and especially open data, in the first place. This is more likely to recruit and enthuse. I think the commercial data will be the one that provides the jobs based on the foregoing: it may change the web more profoundly but in ways fewer people will even notice. I suppose it all depends on how you define tipping points or killer apps, which I don’t intend to think about for much longer.

The second talk, and the start of a common theme, was about SKOS and linked data, by Antoine Isaac. This was probably the most relevant talk for librarians and was for me a simple introduction to SKOS, which seems to be an increasingly common acronym. SKOS stands for Simple Knowledge Organisation System and is designed for representing (simply) things like thesauruses* and classification schemes, based around concepts. These concepts have defined properties such as preferred name (“skos:prefLabel”), non-preferred term (“skos:altLabel”), narrower term (“skos:narrower”), broader term (“skos:broader”), and related term (“skos:related”). The example I’ve been aware of for some time is the representation of Library of Congress Subject Headings (LCSH) in SKOS, where all the SKOS ideas I’ve just mentioned will be recognisable to a lot of librarians. In the LCSH red books, for example, preferred terms are in bold, non-preferred terms not in bold preceded by UF, and the relationships between concepts is represented by the abbreviations NT, BT, and RT. In SKOS, concepts and labels are more clearly distinct. An example of SKOS using abbreviated linked data might be (stolen and adapted from the W3C SKOS primer):

ex:animals rdf:type skos:Concept;
skos:prefLabel “animals”;
skos:altLabel “creatures”;
skos:narrower ex:mammals.

This means that ex:animals is a SKOS concept; that the preferred term for ex:animals is “animals”; a non-preferred term is “creatures”; and, that a narrower concept is ex:mammals. In a mock LCSH setting this might look something like this:

Animals
UF Creatures
NT Mammals

In the LCSH example, however, the distinction between concepts and terms is lost. One aspect of SKOS that Antoine Isaac spent some time on is the idea of equivalent concepts, especially across languages. In RDF you can bind terms to languages using an @ sign, something like this:

ex:animals rdf:type skos:Concept;
skos:prefLabel “animals”@en;
skos:prefLabel “animaux”@fr.

However, you can also link concepts more directly using skos:exactMatch, skos:closeMatch, skos:broadMatch, skos:narrowMatch, and relatedMatch to link thesauruses and schemes together. These are admittedly a bit nebulous. He mentioned work that had been done on linking LCSH to the French Rameau and from there to the German subject thesaurus SWD. For example:

Go to http://lcsubjects.org/subjects/sh85005249 which is the LCSH linked data page for “Animals”. (You can view the raw SKOS RDF using the links at the top right, although sadly not in n3 or turtle format which I have used above). At the bottom of the page there are links to “similar concepts” in other vocabularies, in this case Rameau.
Go the the first one, http://stitch.cs.vu.nl/vocabularies/rameau/ark:/12148/cb119328694, and you see the Rameau linked data page for “Animaux”.

In the LCSH RDF you can pick out the following RDF/XML triples:

Animals
Beasts

which is basically saying the same as (clipping the URIs for the sake of clarity):

lcsh:sh85005249#concept rdf:type skos:Concept;
skos:prefLabel “Animals”@en;
skos:altLabel “Beasts”@en;
skos:narrower lcsh:sh95005559#concept;
skos:closeMatch rameau:cb119328694.

Not too far from the first example I gave, with the addition of a mapping to a totally different scheme. Or in mock red book format again but with unrepresentable information missing:

Animals
UF Beasts
NT Food animals

Oh that some mapping like this were available to link LCSH and MeSH…!

Several other talks touched on SKOS, such is its impact on knowledge management. Andreas Blumauer talked about it in demonstrating a service provided by punkt. netServices, called PoolParty.** I don’t want to go into depth about it, but it seemed to offer a very quick and easy way to manage a thesaurus of terms without having to deal directly with SKOS or RDF. During the talk, Andraeas Blumauer briefly showed us an ontology based around breweries, then asked for suggestions for local breweries. Consequently, he added information for Fullers and published it right away. To see linked data actually being created and published (if not hand-crafted) was certainly unusual and refreshing. Most of what I’ve read and seen has talked about converting large amounts of data from other sources, such as MARC records, EAD records, Excel files, Access databases, or Wikipedia. I’ve had a go at hand-coding RDF myself, which I intend to write about if/when I ever get this post finished.

I don’t want to go into detail too much about it***, but another SKOS-related talk was the final one from Bernard Vatant who drew on his experience in a multi-national situation in Europe to promote the need for systems such as SKOS to deal more rigorously with terms, as opposed to concepts. Although SKOS would appear to be about terms, in many ways it is not clear on many matters of context. For instance, using skos:altLabel “Beasts” for the concept of Animals as in the examples given above gives no real idea of what the context of the term is. Here is a theoretical made-up example of some potential altLabels for the concept of Animals which I think makes some of the right points:

Animal (a singular)
Beasts (synonym)
Animaux (French term)
Animalia (scientific taxonomic term)

These could all be UF or altLabels but using UF or altLabel gives no idea about the relationship between the terms, and why one term is a non-preferred term. He gave another instance of where this might be important in a multinational and multilingual context, where the rather blunt instrument of adding @en or @fr is not enough, when a term is different in Belgian, French, or Canadian varieties of French. This has obvious parallels in English, where we often bemoan the use of American terms in LCSH. Whether embedded in LCSH or as a separate list, it might be possible to better tailor the catalogue for local conditions if non-preferred terms were given some context. Perhaps “Cellular telephones” could be chosen by a computer to be displayed in a US context, but “mobile telephones” could be chosen in a UK context if the context of those terms were known and specified in the thesaurus.

Moving away from SKOS, Andy Powell talked about the Dublin Core Metadata Initiative (DCMI). I’ll admit I’ve always been slightly confused as to what the precise purpose of Dublic Core (DC) is and how one should go about using it. Andy Powell’s talk explained a lot of this confusion by detailing how much DC had changed and reshaped itself over the years. To be honest, in many ways I found it surprising how it is still active and relevant given the summary I heard. The most interesting part of his talk for me was his description of the mistakes or limitations of the DCMI caused by its library heritage. Another confession- my notes here are awful- but the most important point that stuck out for me was the library use of a record-centric approach, e.g.:

each book needs a self-contained record
this record has all the details of the author, title, etc.
this record is used to ship the record from A to B (e.g. from bibliographic utility to library catalogue),
this record also tracks the provenance of the data within the record, such as within the 040 field: it all moves together as one unit.

Contrast this with the sematic web approach where data is carried in triples. A ‘record’, such as an RDF file, might only contain a sameAs triple which relates data about a thing to a data store elsewhere; many triples from multiple sources could be merged together and information about a thing could be enriched or added to. This kind of merging is not particularly easy or encouraged by MARC records (although the RLUK database does something similar and quite tortuously when it deduplicates records). There’s a useful summary of all this at all things cataloged which opens thus:

Despite recent efforts by libraries to release metadata as linked data, library records are still perceived as monolithic entities by many librarians. In order to open library data up to the web and to other communities, though, records should be seen as collections of chunks of data that can be separated / parsed out and modeled. Granted, the way we catalog at the moment makes us hold on to the idea of a “record” because this is how current systems represent metadata to us both on the back- and front-end. However with a bit of abstraction we can see that a library record is essentially nothing but a set of pieces of data.

One problem with the linked data approach though is the issue of provenance which was referred to above as one of the roles the MARC record undertakes (ask OCLC, e.g. http://community.oclc.org/metalogue/archives/2008/11/notes-on-oclcs-updated-record.html). If you take a triple out of its original context or host, how can you tell who created the triple? Is it important? Richard Wallis always makes the point that triples are merely statements: like other web content they are not necessarily true at all. Some uneasiness on the trustworthiness or quality of data turned up at various points during the day. I think it is an interesting issue, not that I know what the answer is, especially when current cataloguing practices largely rely on double checking work that has already been done by other institutions because that work cannot really be trusted. There are other issues and possible solutions that are a little outside my comfort zone at the moment, including excellent buzzwords like named graphs and bounded graphs.

Andy Powell also mentioned, among other things:

the “broad semantics” or “fuzzy buckets” of DC which derive in large part from the library catalogue card, where, for instance, “title” or “creator” can mean all sorts of imprecise things;
flat world modelling where two records are needed to describe say, a painting and a digital image of the painting. This sounds to me like the kind of thing RDA/FRBR is attempting (awkwardly in my view) to deal with.
the use of strings instead of things, such as describing an author as “Shakespeare, William” rather than . This mirrors one of the bizarre features of library catalogues where authorities matching is generally done by matching exact strings of characters rather than identifiers. See Karen Coyle for an overview of the headaches involved.

There were three other talks which I don’t propose to go into in much detail. I’ve touched on Richard Wallis’s excellent (and enthusiastic) introduction to the whole idea of linked data and RDF, a version of which I found dangerously intriguing at a previous event given by Talis. He talked about, among other things, the use of the BBC in using linked data to power its wildlife pages (including drawing content from Wikipedia) and World Cup 2010 site; in fact, how linked data is making the BBC look at the whole way it thinks about data.

His other big message for me was to compare the take-up of the web to where the current take of linked data was in order to suggest that we are on the cusp of something big: see above for my discussion of the tipping point.

* I don’t like self-conscious classical plurals where I can help it, not that there’s anything wrong with them as such.
** I can’t help but find this name a little odd, if not actually quite camp. I expect there’s some pun or reference I’m not getting. Anyway. Incidentally, finding information about PoolParty from the punkt. website is not easy, which I find hard to understand given that it is a company wanting to promote its products; and, more specifically, it is a knowledge management (and therefore also information retrieval) company.
*** Partly because I don’t think I could do it justice, partly also because it was the most intellectual talk and took place at the end of the day.

CILIP (not) on Newsnight

Orangeaurochs — Fri, 11 Jun 2010 11:28:14 +0000

I understand that CILIP was actually offered a chance to talk on Newsnight the other day. I’ve always been of the opinion that CILIP should make more effort to push itself into the media, to publicise and stand-up for the profession. The other day the BBC actually asked CILIP onto a programme, and a significant and highly influential one at that, and they didn’t have anyone available. I expect there are real and genuine reasons why there wasn’t anyone- apparently there were people willing to speak by phone or video link- but I think the fact that CILIP was not prepared for such an occasion, even when libraries were unusually in the news that day, is very symbolic of CILIP’s attitude and institutional inability to promote the profession and to deal with modern media. This is especially surprising given all the recent soul-searching within CILIP and the recognised urgent need for the library profession to defend itself as it’s a soft target in any upcoming cuts. Any robust response to the now infamous KPMG report (see p. 19) seems to have consisted of unofficial bloggers (e.g.) and bloggers from CILIP Update (e.g.), all largely preaching to the choir. I hope they were also pushing a press release down the BBC’s throat, although I haven’t heard of one.

Update: I should point out that, according to the thewikiman, CILIP apparently asked to be on Newsnight in the first place:

I now understand that in fact CILIP contacted Newsnight in the first place – although they ultimately couldn’t make it happen. For me, even though the end result is the same, there’s a huge difference between those two scenarios – in one, CILIP is shirking a fabulous opportunity, but in the other (in actuality) they were tryng to MAKE an opportunity offf their own proactiveness… Which is great. Shame, obviously, that it didn’t work out, and I stand by the idea that hevean and earth should be moved in such a scenario, but the most important thing to take from this new understanding of the events is that it shows CILIP is already moving in the right direction.

I have to agree with that.

Phil Bradley on CILIP in 2020

Orangeaurochs — Thu, 20 May 2010 15:28:43 +0000

Phil Bradley has written a long “a stream of thought” post on how he would like to see CILIP looking in ten years’ time. I’m not sure how much I agree with a lot of it, but it is interesting and very positive nonetheless. What struck me (perhaps because I do agree with them) are the following couple of points:

I want information professionals to be able to look at what CILIP does and say to their employers – this is what my professional organization is doing – why can’t I do it as well?

I don’t think it is CILIP’s job to just lead on technology (later on he gives an example of CILIP having something like an iPad that members could have a chance to play with) or web design as this is getting beyond the remit of librarianship. Although those areas are vital I think one of CILIP’s weaknesses in fact is that it is in many respects a vanilla professional institute which, in moving away from some core of specific skills, is leaving us with nothing special to sell. For instance, trying to take ownership of the word Information rather than the word Library is dangerous as there are others, particularly computer scientists, who already own much of that ground, and have broken much of it too. Perhaps this is what Phil Bradley is driving at when he says,

I want librarians, backed by the professional body, to be the ones telling the technical staff what they should be doing, not the other way around.

However, I think that he talking about the role of librarians within an organisation rather than the acquirement of real technical skills that could increase our ability to adapt and increase our services.

Anyway, I do think it is important that CILIP leads the way as an example to its own community as well as a something to be pointed out to others as Phil Bradley suggests, something it certainly hasn’t done in the past.

I heartily agree with the following sentiment:

I want to see CILIP mentioned in the press and the media every single time there’s reference to a library, for good or ill. CILIP needs to be the organisation that’s pulled onto programmes to talk on behalf of the profession.

I think this is a must. Other issue-specific organisations are on the telly or pop up in newspapers quite often. I think if CILIP proactively offered its services and made a hue and cry on an issue, programmes like BBC Breakfast would probably listen. Incidentally, this is one area where I think changing the name from Library Association to CILIP was catastrophic: lots of people outside the profession knew the Library Association and its name is fairly self-explanatory; I don’t think the same could be said of CILIP, and I expect many people would still be mystified when the acronym is expanded.

In one other point I don’t think Phil Bradley goes far enough:

I want CILIP to continue to run courses, and I want those courses to be held, not just in London, but at your desk, with webinars, conference calling/training and so on. Why should it be necessary for me to come to London in order to sit in and watch a council meeting? Why can’t I do it at my desk?

I want those courses to be overhauled and more specific. In particular the monlithic MA needs to be ditched, a series of specific short courses needs to be introduced, and the CILIP courses on offer need to go beyond “An introduction to…”.