|
|
This is a web version for reference of a docx file originally produced for the Mashcat 2012 session I did called How Big Is My Book. Resurrected to form the Manual for Meret, a regular expressions tutorial based on Marcedit examples.
Literals
Characters as you type them. E.g. i will look for a letter “i”. ii will look for two letter “i”s in a row. Eldorado will look for the exact string “Eldorado”, and 1234s will look for “1234s”.
Types of Character
There are a number of ways of looking for specific types of character:
. looks for any single character. It could be a letter, number, punctuation or anything.
[] looks for any one of the characters in square brackets, so m[ae]rc will match “marc” and “merc”. You can also specify ranges, e.g. [a-z] will find any letter from “a” to “z”, so [a-d]ad will match “aad”, “bad”, “cad”, and “dad”. Putting a ^ after the [ will look for any character that isn’t in the square brackets: u[^ks]marc will not match “ukmarc” or “usmarc” but will match “unmarc”.
\d a digit, same as [0-9]. Like all the following, counts as one character although written as two.
\D not a digit, same as [^0-9].
\w alphanumeric, including underscore, same as [A-Za-z0-9_]
\W non-alphanumeric, same as [^A-Za-z0-9_]
\s whitespace characters, e.g. spaces, tabs
\S non-whitespace characters
\b word boundary: the beginning or end of words (i.e. strings of alphanumeric characters), or the beginning or end of strings.
\ is also used before a special character so you can search on it. E.g, searching on . will look for any character and will match “.”, “d”, or “5”. To look for a full-stop, put \ in front: \..
Starts and Ends
^ matches the start of any string. So, in “marc must die” ^marc will match “marc” but ^must will match nothing.
$ matches the end of any string. So, in “marc must die” die$ will match “die” but must$ will match nothing.
Numbers of Characters
* matches the preceding element zero or more times, e.g. catalogu*ing will match “cataloging”, “cataloguing”, as well as “cataloguuing” and “cataloguuuuuuuuuuing”.
? matches the preceding element zero or one times, e.g. catalogu?ing will match “cataloging” and “cataloguing” but not “cataloguuing”. See also ? below.
+ matches the preceding element one or more times, e.g. catalogu+ing will match “cataloguing”, “cataloguuing”, and “cataloguuuuuuuuuuing”, but not “cataloging”.
{n} matches the preceding element exactly n times, e.g. catalogu{10}ing will match “cataloguuuuuuuuuuing” but not “cataloging”, “cataloguing”, or “cataloguuing”.
{m,n} matches the preceding element at least m times and no more than n times.
? also has a special meaning to restrict matches of multiple characters, e.g. looking for catalog.*ing in “cataloguing is ace. I love cataloguing” will greedily find “cataloguing is ace. I love cataloguing” as the .* matches both “uing is ace. I love catalogu” and “u”. Amending the regular expression to catalog.*?ing will find only “cataloguing”.
Grouping
() groups characters together. This has a variety of uses. The group can be used a single character, e.g. (meta)* looks for the string “meta” zero or more times. It can also be used for capturing smaller parts of the expression for later use, e.g. catalog(.*) will match anything starting “catalog” but will also store what comes afterwards as $1.
| [pipe] allows alternatives either side of it, e.g. marc|rdf will match “marc” or “rdf”. Smaller alternatives can be matched with brackets, e.g. (uk|us)marc will match “ukmarc” or “usmarc” (and if there is a match will store “uk” or “us” as $1).
Regular Expressions in Javascript
To get matches, use string.match(//). The regular expression goes between the forward slashes. Put a g after the second slash to search for all matches, rather than just the first one. Put an i after the second slash to do a case-insensitive search. String.match returns an array of matches, or null if it finds nothing.
var hits = “team”.match(/i/g);
hits is null as there is no “i” in “team”.
var text = “Fox in socks in box on Knox”;
var hits = text.match(/\w*ox\b/g);
hits is an array of three elements, all a series of words ending in “ox”: [“Fox”, “box”, “Knox”].
To search and replace within string, use string.replace(//, ””). The regular expression goes between the forward slashes. The g and i work in the same way. The string to replace matches with goes after the comma. You can insert subexpressions captured with round brackets by using $1 for the first, $2 for the second, and so on (see Grouping above and the example below). String.replace returns the string with replacements made:
To search and replace within string, use string.replace(//, ””). The regular expression goes between the forward slashes. The g and i work in the same way. The string to replace matches with goes after the comma. You can insert subexpressions captured with round brackets by using $1 for the first, $2 for the second, and so on (see Grouping above and the example below). String.replace returns the string with replacements made:
var text = “I love MARC. I think MARC is the future.”;
text = text.replace(/MARC/g, ”linked data”);
text is now “I love linked data. I think linked data is the future.”
var text = “UKMARC is better than USMARC”;
text=text.replace(/(.*?MARC) is better than (.*?MARC)/gi, “$2 is better than $1”);
Now, “USMARC is better than UKMARC”. Run the replacement again, and history is reset.
Examples
ISBN (from Thingology blog) ([0-9]{9}[0-9X]|(978|979)[0-9]{10})
UK Postcode (from Wikipedia) (GIR 0AA|[A-PR-UWYZ]([0-9][0-9A-HJKPS-UW]?|[A-HK-Y][0-9][0-9ABEHMNPRV-Y]?) [0-9][ABD-HJLNP-UW-Z]{2})
Below are some articles I wrote for Catalogue & Index, the periodical of the CILIP Cataloguing and Indexing Group (CIG). They are all now freely available as PDFs in UCL’s repository. The first three were written following CIG”s Linked Data: What Cataloguers Needs to Know event in 2013, and appeared in issue 174 (2014). I have also included links to the Slideshare versions of presentations on which they were based:
Introduction to Linked Data: article | presentation
What’s Wrong With MARC: article | presentation
BIBFRAME: article | presentation
The fourth is from issue 172 (2013):
Outsourcing Authority Control at UCL: article
Erratum: it has been pointed out to me on Slideshare that the example on slide 23 of the Introduction to Linked Data is in fact RDF/JSON rather than JSON-LD. This was corrected in the article.
I have finally completed a multiple record MARC Record Viewer. This has been rather long in the making but is essentially a quick and practical tool for looking at and assessing MARC records without having to load them into specialist software like MARCEdit or an LMS. It is essentially the same as the viewer built for my Codecademy project except that:
- It reads multiple records in one file, rather than just one, and provides a count.
- It has an input box so the records don’t have to be hard-coded into the script.
Some example .mrc records of varying lengths can be found here.
It is written in client-side Javascript, so you can view source and see how it works, copy it, and do what you like with it (although I would love to know if you do so). I quite defiantly haven’t used JQuery for this, which would probably have made the whole thing a bit easier; instead it uses proper old skool DOM scripting. It uses a minimal amount of CSS, in two files: a generic one, and one that roughly mimics how MARC records look in an Aleph editing screen. It should be fairly trivial to change this file to suit other purposes.
Thank you to those who have already have a shufti at earlier versions of this, especially on different browsers, and provided feedback! Please do let me know if you have any comments on this, suggestions for improvements, or if you come across errors. I have some ideas for improvements, mainly for making user input easier, and offering different formatting of results. I hope to start using JQuery for these too, and perhaps a later conversion of the whole thing would be in order.
For a Dev8d session I did with Owen Stephens in February I presented data for a single book and followed how it had changed as standards changed, trying above to explain to non-cataloguers why catalogue records look and work the way they do. At least one person found it useful. I am now drafting an internal session at work on the future of cataloguing and am planning to take a similar approach to briefly explain how we got to AARC2 and MARC21, and where we are heading. I took the example I used at Dev8d and hand-crafted some RDA examples, obtained a raw .mrc MARC21 file, and used the RDF from Worldcat to come up with a linked data example.
I have tried to avoid notes on the examples themselves. However, do note the following: the examples only generally use the same simple set of data elements, basically the bits you might find on a basic catalogue card: no subjects, few notes, etc.; the book is quite old so there is no ISBN anyway. The original index card is from our digitised card catalogue. The linked data example was compiled by copying the RDFa from the Worldcat page for the book; this was then put into this RDFa viewer (suggested by Manu Sporny) to extract the raw RDF/Turtle; I manually hacked this further to replace full URIs with prefixes as much as possible in an attempt to make it more readable (I suspect this is where some errors may have crept in). The example itself is of course a conversion from an AARC2/MARC21 record. C.M. Berners-Lee is Tim’s dad.
Feel free to use this and to point out mistakes. I would particularly welcome anyone spotting anything amiss in the RDA and linked data, where I am sure I have mangled the punctuation in both.
Harvard Citation
Berners-Lee, C.M. (ed.) 1965, Models For Decision: a Conference under the Auspices of the United Kingdom Automation Council Organised by the British Computer Society and the Operational Research Society, English Universities Press, London.
Pre-AACR2 on Index Card
BERNERS-LEE, C.M., [ed.].
Models for decision; a conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society.
London, 1965.
x, 149p. illus. 22cm.
AACR2 on Index Card
Models for decision : a conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society / edited by C.M. Berners-Lee. -- London : English Universities Press, 1965.
x, 149 p. : ill. ; 23 cm.
Includes bibliographical references.
- Berners-Lee, C. M.
AACR2 in MARC21 (raw .mrc)
00788nam a2200181 a 4500001002700000005001700027008004100044024001500085245021000100260004900310300003200359504004100391650003300432700002300465710003900488710003000527710004900557_UCL01000000000000000477125_20061112120300.0_850710s1965 enka b 000 0 eng _8 _ax280050495_00_aModels for decision :_ba conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society /_cedited by C.M. Berners-Lee._ _aLondon :_bEnglish Universities Press,_c1965._ _ax, 149 p. :_bill. ;_c23 cm._ _aIncludes bibliographical references._ 0_aDecision making_vCongresses._1 _aBerners-Lee, C. M._2 _aUnited Kingdom Automation Council._2 _aBritish Computer Society._2 _aOperational Research Society (Great Britain)__
AACR2 in MARC21
| 245 |
00 |
$a |
Models for decision : |
|
|
$b |
a conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society / |
|
|
$c |
edited by C.M. Berners-Lee. |
| 260 |
__ |
$a |
London : |
|
|
$b |
English Universities Press, |
|
|
$c |
1965. |
| 300 |
__ |
$a |
x, 149 p. : |
|
|
$b |
ill. ; |
|
|
$c |
23 cm. |
| 504 |
__ |
$a |
Includes bibliographical references. |
| 700 |
1_ |
$a |
Berners-Lee, C. M. |
RDA
| Title proper |
Models for decision |
| Other title information |
a conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society |
| Statement of responsibility relating to title proper |
edited by C.M. Berners-Lee |
| Place of publication |
London |
| Publisher’s name |
The English Universities Press Limited |
| Date of publication |
1965 |
| Copyright date |
©1965 |
| Media type |
unmediated |
| Carrier type |
volume |
| Extent |
x, 149 pages |
| Dimensions |
23 cm |
| Content type |
text |
| Illustrative content |
Illustrations |
| Supplementary content |
Includes bibliographical references. |
| Contributor |
Berners-Lee, C. M. |
| Relationship designator |
editor of compilation |
RDA in MARC21
| 245 |
00 |
$a |
Models for decision : |
|
|
$b |
a conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society / |
|
|
$c |
edited by C.M. Berners-Lee. |
| 264 |
_1 |
$a |
London : |
|
|
$b |
The English Universities Press Limited, |
|
|
$c |
1965. |
| 264 |
_4 |
$c |
©1965 |
| 300 |
__ |
$a |
x, 149 pages : |
|
|
$b |
illustrations ; |
|
|
$c |
23 cm. |
| 336 |
__ |
$a |
text |
|
|
$2 |
rdacontent |
| 337 |
__ |
$a |
unmediated |
|
|
$2 |
rdamedia |
| 338 |
__ |
$a |
volume |
|
|
$2 |
rdacarrier |
| 504 |
__ |
$a |
Includes bibliographical references. |
| 700 |
1_ |
$a |
Berners-Lee, C. M., |
|
|
|
editor of compilation. |
Linked data
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix schema: <http://schema.org/> .
@prefix worldcat: <http://www.worldcat.org/oclc/> .
@prefix library: <http://purl.org/library/> .
@prefix viaf: <http://viaf.org/viaf/> .
@prefix lc_authorities: <http://id.loc.gov/authorities/names/> .
@prefix mads: <http://www.loc.gov/mads/rdf/v1#> .
worldcat:221944758
rdf:type schema:Book;
library:oclcnum "221944758";
schema:name "Models for decision : a conference under the auspices of the United Kingdom Automation Council organised by the British Computer Society and the Operational Research Society";
library:placeOfPublication _:1;
schema:publisher _:4 .
schema:datePublished "[1965]";
schema:numberOfPages "149";
schema:contributor viaf:149407214;
schema:contributor viaf:130073090;
schema:contributor viaf:137135158;
schema:contributor viaf:36887201;
_:1
rdf:type schema:Place;
schema:name "London :" .
_:4
rdf:type schema:Organization;
schema:name "English Universities Press" .
viaf:149407214
rdf:type schema:Organization;
madsrdf:isIdentifiedByAuthority lc_authorities:n79056431;
schema:name "British Computer Society." .
viaf:130073090
rdf:type schema:Organization;
madsrdf:isIdentifiedByAuthority lc_authorities:n85076053;
schema:name "Operational Research Society." .
viaf:137135158
rdf:type schema:Organization;
madsrdf:isIdentifiedByAuthority lc_authorities:n79063901;
schema:name "Institution of Electrical Engineers." .
viaf:36887201
rdf:type schema:Person;
schema:name "Berners-Lee, C. M." .
I have created a Codecademy project (with a lot of help in corrections and improvements from Esther Arens!) that builds a short script to read a raw MARC record and display it in a more readable format. Try it here: http://www.codecademy.com/courses/marc-viewer/.
This is not by no means the last word in reading a MARC file and is basically a walk through of one way to do it. There are other ways and better ways that use more advanced coding, or allow more sophisticated re-use of the bits and pieces that are pulled out of the MARC record. There are also entire programmes and programming utilities designed to do this kind of thing and to manipulate MARC records, not least library management systems and things like MARCEdit. Moreover, there are limitations in formatting on the Codecademy platform that can easily be overcome by adapting the script to be run directly in an HTML file (I have done a direct simple adaptation without any further elaboration (view the HTML source to see the code and the alterations)).
I hope, if nothing else, that it gives cataloguing coders an idea of what a MARC21 record looks like under the hood and helps clarify the cataloguer’s opinions as to whether MARC must or mustn’t die. (HINT: it must).
Please see the following notes below before proceeding:
1. This project was designed for someone who has done the first few weeks of the Code Year course. By necessity it introduces some new things and an attempt has been made to explain them and encourage the cataloguer to enter the actual lines of Javascript that make up the programme. In any case, the Hints always contain the correct code needed to proceed.
2. Output will often consist of many lines, so sometimes you will have to scroll up in the console to see what has happened.
3. Some lines (including line 1!) will always produce errors, although the script will still run. This will is because MARC uses BAD and DANGEROUS characters. BAD and DANGEROUS characters are of course common in the world of cataloguing (mentioning no names…).
There are many ways this could be improved or extended if you wanted the challenge, e.g.:
- Take the HTML version and use more HTML and CSS to make it clearer and prettier (e.g. more spacing, colour, bolding of codes). Try making it look like a specific LMS editing screen.
- Make it capture the elements in more detail and in a more re-usable way. For instance, try making each field an object with tags, indicators, subfields, etc. as properties. This would enable more interesting things, such as…
- A simple OPAC or even a card index display.
- Adapt it to read MARC files with more than one record. This isn’t as hard as it sounds, in that each record ends with a specific terminator (see the guide to record structure below).
For full technical details of how a MARC21 bibliographic record is put together, see the MARC 21 Specifications for Record Structure, Character Sets, and Exchange Media Record Structure. For details of the contents and use of MARC21 fields, see LC’s MARC Standards page. For a HTML version of the completed MARC Viewer script, see my adaptation.
The Code Year programme is part of Codecademy, an online set of programming lessons. Cataloguers interested in learning to programme will find the independent CatCode Wiki useful for extra information, advice, and support. See also the #catcode hashtag on Twitter.
Do let me know if you come across any problems with it or have any comments on the project.
Thank you again to Esther for her help.
https://twitter.com/#!/EstherArensEstter
Yay, my entry for the Discovery & DevCSI Developers Competition– Lodopac– was awarded a commendation for its use of the Cambridge University Library (CUL) dataset. During the judging I was asked for searches which were known to work well- the timeout issues I discussed under Limitations being not insignificant, especially with author or title searches. I submitted a version of the following brief general notes which I hope are helpful to anyone else who wants to play:
The British National Bibliography (BNB) server is generally more responsive than the Cambridge University Library one; title seems to work better than author. The following are hopefully useful examples useful:
I would really like to try and think of ways of improving free text regular expression search times for things like author and title in Sparql* although I doubt there is one that doesn’t rely on the configuration, processing power, or indexing of the server being searched.
* thinking aloud, some ideas might include: downloading a larger imprecise set for further local searching (e.g. for an author/title search downloading the title matches and searching the authors locally: although this would also be slow, it would get round the timeout at least); forcing a look-up in a controlled vocab first in order to get an exact string match (esp for authors, although even if this is possible, this forces a user to do more work, which isn’t the point); local indexing of the triple store (this is probably the best way but I’m not sure how to go about it, whether I really have the server capabilities to do it, and can be committed to the updating required).
One of the difficulties in searching RDF data is knowing what the data looks like. For instance, finding a book by its title means knowing something about what how a dataset has recorded the relationship between a book and its title. There is no real standard for publishing MARC/AACR2-style bibliographic data as RDF: it seems libraries publishing RDF are approaching this largely individually, although they are using many of the same vocabularies, dc, bibo, etc. This was one reason why I wanted to create Lodopac: to present some kind of interface so that searchers didn’t need to know these different models but could start to explore them. Below are the Sparql recipes for the different search criteria I used for the BNB and the Cambridge University Library datasets, so they can be compared, re-used, or corrected. All examples use prefixes, which are defined anew in each example. The examples are of course fragments and don’t have all the necessary SELECT and WHERE clauses.
By the way, for an excellent Sparql tutorial with ample opportunity to play as you go along, do have a look at the Cambridge University Library’s SPARQL tutorial. It also gives clues to the way their data is structured. Of use for the BNB is their data model (PDF), which is not nearly as scary as it looks at first, and incredibly helpful.
Author keyword search
This would be relatively straightforward-the unavoidable regular expression being the main complication- but for the fact that the traditional author/editor/etc of bibliographic records can be found in dc:creator as well as dc:contributor which necessitates a UNION. The BNB used foaf:name:
PREFIX dc: <http://purl.org/dc/terms/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT ?book
WHERE {
{?book dc:creator ?author} UNION {?book dc:contributor ?author} .
?author foaf:name ?name .
FILTER regex(?name, “author”, “i”) .
}
Cambridge uses much the same recipe except that it uses rdfs:label instead of foaf:name:
PREFIX dc: <http://purl.org/dc/terms/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?book
WHERE {
{?book dc:creator ?author} UNION {?book dc:contributor ?author} .
?author rdfs:label ?name .
FILTER regex(?name, “author”, “i”) .
}
Title keyword searches
This is more straightforward and is in fact the same for both the BNB and Cambridge University Library:
PREFIX dc: <http://purl.org/dc/terms/>
SELECT ?book
WHERE {
?book dc:title ?title .
FILTER regex(?title, “title”, “i”) .
}
Date of publication (year)
I imagined this one being simple and for Cambridge University Library it is. However the BNB took some unravelling as they have modelled publication as an event related to a book. The various elements of publication are then related to the event. So, for the BNB we have this:
PREFIX bibliographic: <http://data.bl.uk/schema/bibliographic#>
PREFIX event: <http://purl.org/NET/c4dm/event.owl#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?book
WHERE {
?book bibliographic:publication ?pub .
?pub event:time ?year .
?year rdfs:label “date” .
}
By contrast, Cambridge University Library has it in one line:
PREFIX dc: <http://purl.org/dc/terms/>
SELECT ?book
WHERE {
?book dc:created “date” .
}
ISBN
As an identifier, ISBN is relatively straightforward in both models, although care must be taken with the BNB as 10 and 13 digit ISBNs are treated as separate properties and the following assumes that the search will cover both:
PREFIX bibo: <http://purl.org/ontology/bibo/>
SELECT ?book
WHERE {
{?book bibo:isbn10 “isbn”} UNION {?book bibo:isbn13 “isbn”} .
}
For Cambridge University Library, also using the bibo ontology, this is:
PREFIX bibo: <http://purl.org/ontology/bibo/>
SELECT ?book
WHERE {
?book bibo:isbn “isbn” .
}
Conclusion
I didn’t set up to provide ground-breaking conclusions. However, it is remarkable how different data models can be formulated for modelling the same type of data by similar organisations. The real question is whether this is a good, a bad thing, or doesn’t really mattter. Will it need to be standardised? My understanding of how this works is probably not. I think the days of monolithic library standards are probably now gone. I wonder, for instance, if there ever will be a single MARC22 (or whatever you like to call it) and doubt RDA will ever completely replace AACR2 in the way we imagine. What will emerge I suspect will be a soup of various standards and data models, some of which will be more prevalent. One thing I picked up from various linked data talks is that information has frequently been published then re-used in ways that the issuers never imagined; if that is the case, the precise modelling and format is probably not as important as the fact that it is of good quality and intelligently put together. The BNB and Cambridge University Library models are clearly quite different but quite capable of being mapped and used despite this.
If there are any other bibliographic data Sparql endpoints, I would like to include them in a future version of the Lodopac search. Do let me know if you come across them.
More mundanely, do say if there are errors in my Sparql recipes or if there are ways they could be done more efficiently.
Lodopac is my entry to the UK Discovery Developer Competition. Aside from obvious mocking of the name, comments on Lodopac are very welcome. If anyone installs it locally, I’d also be very interested to know.
About Lodopac
Lodopac is a simple linked open data opac using Sparql to search remote bibliographic RDF data. By default it is set up to search the BNB and Cambridge University Library datasets, but is designed to allow easy setup of additional datasets with Sparql endpoints (see Installation, source code, and configuration below). It was written in response to the UK Discovery Developer Competition.
The purpose of Lodopac is to provide a simple standard OPAC-style interface to perform searches of various bibliographic RDF datasets without having to know how to formulate a Sparql query and without having to know the structure of the database. I hope it is especially useful for people wanting to get a grip on how bibliographic RDF is put together, what it looks like, and what a Sparql query looks like. For example, an author search is possible without knowing about dc:creator and dc:contributor, or how these need to be linked together in a Sparql search. Similarly, a searcher wouldn’t need to worry about how to construct date searches in different datasets. For the BNB and CUL, these are very different (three lines in the BNB, one for CUL), but in Lodopac, there is only search box to search both. Lodopac displays the Sparql query it constructed to perform the search, as well as the combined RDF for all results found in XML, JSON, N3, and TTL.
How to search Lodopac
Select one or more of the available datasets using the checkboxes.
Author and Title searches are free text phrase searches. In other words, a string you search for will match with any exact match, including spacing and punctuation, and in the middle of words. E.g. searching for “shake” will match “Shakespeare”, “milkshakes”, and “More hits that you can shake a stick at”. Searching is case insensitive. The following punctuation is removed from searches: \”‘<>$^%.
You are strongly advised to keep author and title searches simple: e.g. one word of a title or a surname only.
ISBN searches 10 or 13 digit ISBNs. Any dashes or other non-digit characters are stripped from the search.
Date search will accept a year.
N.B. Keep searches as simple as possible, especially with author and title searches, to avoid them timing out. ISBN and date searches are generally quicker.
Limitations
A bad workman blames his tools and I’m no exception. The greatest limitation is the time taken by Sparql endpoints to perform a Sparql query, especially one that involves a regular expression, such as the Author and Title searches. What is needed is some more robust indexing or some cheat like Virtuoso’s unorthodox bif:contains, which the old version of the linked data BNB used. I touched on this in a blog post about the In Our Time Booklist script I wrote (see section 6).
The load and current capacity on the Sparql endpoints at the time the query is made is another important factor. A search which times out one minute can work fine the next.
The search options are obviously limited but do I hope represent the most common methods of searching normal library catalogues aside from, of course, a general keywords search. The manipulation of results is also rather sparse but allows click through to full data associated with a book, the structure and contents of which can be more fully explored. The aggregation of RDF data in various formats is I hope useful illustratively as well as having potential for further manipulation.
Installation, source code, and configuration
The source code for Lodopac is available as a zip file, which contains all the necessary PHP, Javascript, and CSS files. In addition, you will need to install ARC2, which makes the Sparql queries and manipulates the resultant RDF. Edit the first line of lodopac.php so that it points at your local installation of ARC2.
The programme is basically one long script- there is only one page- but is split for convenience of editing. The key file is lodopac.php which includes the other files as it goes along. The main core of the script which builds the queries and does the searching is all in lodopac.php.
I have attempted to make the script as easily configurable as possible so that additional Sparql endpoints can be added. There are probably more components hard-coded into the script that I have overlooked, but all the setup for the endpoints is in the file setup_endpoints.php. The first part of this file is a list of necessary prefixes that are needed for any possible queries from any of the endpoints and, although not ideal, all these prefixes are sent with any Sparql query. Following that and the declaration of an array of the endpoints, each endpoint has a dedicated block with the information added to a hash. To add another endpoint, duplicate a block and configure the search recipes as appropriate. The keys marked “brief_” are used to fetch information for the brief results display. I have conspicuously chickened out of providing an author and the attendant main entry and multiple author headaches involved.
Classification Web is ace, but there are a couple of things about the interface that annoy me and, in one colleague’s case, seriously put him off using it, in particular:
- The opening of a new tab/window when you click on the MARC view for a subject or name.
- The confusing menu. We don’t use LCC or DDC, and the browse options don’t really add much, so we only really need two options: Search LC Subject Headings and Search LC Name Headings.
I managed to work out a simple way of modifying how Classification Web works on Firefox using the Greasemonkey add-on and a couple of simple scripts, all of which is quick and easy to install:
- Install Greasemonkey: https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/
- Make sure the monkey in the bottom-right corner is happy and colourful. Click on it if not.
- If you want to prevent the MARC view opening a new window, install the classweb_no_new_ window script by going to http://www.aurochs.org/zlib/js/userjs/classweb_no_new_window.user.js then
- Click on the Install button
- If you want to reduce the main menu, install classweb_prune_menu script by going to http://www.aurochs.org/zlib/js/userjs/classweb_prune_menu.user.js then
- Click on Install button
- Reload/refresh Classweb if it’s still open and it should work.
If you want to turn Greasemonkey off altogether, click on the monkey so he’s sad and grey. If you want to stop individual scripts, right click on the monkey, click on Manage User Scripts, select a script from the list, and un-tick the Enabled box in the lower left corner.
These instructions were tested on Firefox 3.5.3 although I imagine they would be fine on any recent version of Firefox. I would be interested to hear anything confirming or undermining that assertion.
If you’re happy to play around, these scripts can be further altered. In particular, you can choose which menu items appear in the pruned menu script:
- Right click on the monkey
- Click on Manage User Scripts
- Select classweb_prune_menu from the list
- Click on Edit (you will probably have to select a text editor at this point)
- Edit the list of pages under the line var menu_items_to_keep = Array (. Enter each page you want to appear on the menu on a separate line in quotes, with a comma at the end of each line except the last line. The menu item must appear exactly as it does on the Classification Web menu, including capitals. E.g., the default set up looks like this:
var menu_items_to_keep = Array ( // end each line with a comma except the last line
"Search LC Subject Headings",
"Search LC Name Headings"
);
- Save the file, and reload Classification Web.
If anyone else finds this useful or can think of more customizations let me know.
I have written a script which takes an unstructured reading list on the BBC’s In Our Time website, searches the British National Bibliography (BNB) using bibliographica for the books on the list, and returns structured metadata for the records it found.
This script was written in response to an idea raised by psychemedia for the Open Bibliographic Data Challenge: the BBC “In Our Time” Reading List:
The BBC “In Our Time” radio programme publishes suggested recommending reading in the programme data in an unstructured and citation style way: author, title (publisher, year), with what looks to be conventional character string separators between references (at least on the pages I looked at).
The idea is to extract and link suggested readings for the In Our time programmes to open, structured bibliographic data. This would make the In Our Time archive even more useful as an informal (open-ish) educational resource, especially as academic libraries start to release data relating to books used on courses. (So for example, this approach might help provide a link from a course to a relevant In Our Time broadcast via a common book.)
I was drawn to this idea as I like the idea of turning unstructured data into structured data: I have for example had some previous fun converting HTML pages into RSS feeds (e.g. CILIP Lisjobnet, Big Brother). I think something similar for any reading list (e.g. a Word document produced by a lecturer) would be an interesting idea.
The programme is written in PHP and is designed to be fired from a Javascript bookmarklet from a page on the In Our Time site, or by appending the In Our Time URL to the end of the URL for this page: http://www.aurochs.org/inourtime_booklist/inourtime_booklist_v1.php?. For example, to use it on the page for The Mexican Revolution (which I used a lot in testing), add the URL http://www.bbc.co.uk/programmes/b00xhz8d to produce http://www.aurochs.org/inourtime_booklist/inourtime_booklist_v1.php?http://www.bbc.co.uk/programmes/b00xhz8d.
The script follows the following steps:
- Set up ARC2 to enable SPARQL searching and RDF processing
- Extract Further Reading section
- Separate out Raw Data for each book
- Determine pattern used in citation then extract Basic Data, e.g. author, title, article, publication, using regular expressions
- Further refine elements to make searching easier, i.e. one surname for author, only title proper for titles
- Construct a SPARQL Query using author surname and title regular expressions pre-filtered for speed by a significant word using bif:contains
- Filter Hits by date of publication
- Obtain and display metadata from BNB
More details of these steps are below:
1. Set up ARC2 to enable SPARQL searching and RDF processing
ARC2 is a simple-to-use system for using RDF and SPARQL in PHP. I had previously played with it here when experimenting with creating my own RDF. The Sandy site uses SPARQL to populate the See Also sections.
2. Extract Further Reading section
A simple regular expression identifies the div in the HTML code that contains the reading list, which enables the next stage of the script to look for individual books.
3. Separate out Raw Data for each book
Another regular expression pulls out the paragraphs containing books and puts them in an array. You can see this by viewing the Raw Data.
4. Determine pattern used in citation then extract Basic Data, e.g. author, title, article, publication using regular expressions
As the In Our Time site does not use a single standard form of citation, the script has to try and determine which of several possible patterns a citation is using with regular expressions and extract the correct bits of data. This only works as well as it is possible to identify all the patterns, which effectively means looking at as many In Our Time pages as possible. This is one area that would certainly reward more work. It also points out how difficult it would be to extrapolate this into a script that could read any citations. The In Our Time booklist currently uses five citations each identified with a number (1, 2, 3, 4, 15, 5). If you look at the Book Data for a particular book you will see the citation style number given. The regular expressions capture author, title, and publication.
The author information in citations on In Our Time is unpredictable. Sometimes surnames are first, sometimes they are last. The citation patterns take care of this if possible and try to extract one significant name.
5. Further refine elements to make searching easier, i.e. one surname for author, only title proper for titles.
The script removes things like “(ed.)” from the author, which would obviously throw off a catalogue search. Subtitles- everything after and including semi-colons- are also removed from titles to lessen any chance of variation and lost matches.
6. Construct a SPARQL Query using author surname and title regular expressions pre-filtered for speed by a significant word using bif:contains
Constructing the SPARQL query was the most tricky part. Ignoring the various standard prefixes pilfered from the standard example, the most important part is the title search. There are three unsatisfactory options:
- Match the extracted title directly to a dc:title. This doesn’t work as the cited title is unlikely to be exactly the same in all matters of words included, spacing, punctuation, etc.
- Use bif:contains for keyword searching as used in the BNB SPARQL example. This is certainly quick, but has a number of drawbacks: it can only be used once for a single keyword (any one of the two significant words in The history of Mexico, for example, will produce a huge number of hits). It is also not standard SPARQL. I was happy to overlook this if it worked, but ARC2 didn’t like it at all until I worked out it has to be used in angle brackets e.g. ?title <bif:contains> “Mexico”.
- Use regular expressions (e.g. FILTER regex (?title, “The History of Mexico”, “i”)) for keyword searching. This is extremely powerful: you can easily construct searches but it is so slow as to routinely time out, so rendering it effectively useless.
The In Our Time script uses a combination of the last two techniques to get a result. First, it finds a significant word in the title, ideally the first four letter word after the first word (i.e. to avoid “The”, “A”, “That”, etc.) or, failing that, the first word. The SPARQL query then uses bif:contains to search for that word. The query then does a regular expression filter on the whole title. I don’t know if this is how SPARQL endpoints would generally work, but the BNB appears to only look for regular expression matches on the records already filtered by the bif:contains. In any case, it works.
In addition, the script also uses a regular expression to search by the author’s surname. It doesn’t search by date as the date of publication on BNB (dc:issued) is not in a standard format (e.g. “1994-01-01 00:00:00″, “2005 printing”, “c2006″). It is also not keyword searchable. You can see all the author-title hits with links to BNB records by viewing Hits.
7. Filter Hits by date of publication
You can, however, retrieve the date from the BNB and process it afterwards, which is what the script does. It finds the four digit year and compares this to the four digit year it found on the In Our Time site. You can see all the author-title-date hits with links to BNB records by viewing Date Hits. Perhaps rather arbitrarily, the first book in the resulting array is selected as the result.
8. Obtain and display metadata from BNB
When the search took place, the matching title, author (only one), and date is obtained from BNB. This title and author are displayed, as are the stripped down year of publication, and a link to the full BNB record. For records that returned no hits on the BNB, the Basic Data is simply regurgitated.
The script also downloads the full combined RDF for all the hits is displayed at the bottom of the page, viewable in a several formats.
Further work
I think a lot more work could be done on this given time, both to improve it and to extend it. In no particular order:
- Make it more pretty. It is currently designed to look merely acceptable while I concentrated on functionality. I have also tried to show much of my working, which a finished version would obviously hide.
- Refinement of the detection of citation style. This is probably the most critical improvement, and ultimately decides if this approach would be useful outside of In Our Time on other reading lists. There are more patterns that need to be added, especially for older pages.
- Further preparation of data for searching. Currently, for example, a book on the Mexico reading list doesn’t return any hits because of the exclamation mark in “Zapata!”. This could be stripped, and there are lots of similar refinements no doubt.
- More interesting/useful output. The script’s outputs are currently quite raw or basic as I concentrated on the mechanisms for pulling information from free text for automatic catalogue searching. It might be useful to output proper standalone RDF files, references in standard reference formats (e.g. Harvard) in HTML or text files, files in standard reference management formats, perhaps even MARC, and so on. Some of these would perhaps be fairly straightforward.
- Links to catalogues or online bookshops so you could borrow or buy the books from the reading list based on ISBNs taken from the BNB record.
- Searching more catalogues. If a search fails on the BNB, the script could search other open catalogues, e.g. the Cambridge catalogue.
- Greasemonkey script or plugin so that a button appears next to the Further Reading section when you view an In Our Time page. This could even appear next to individual books. Ideally (pie in the sky) such a plugin would have a stab at finding books on any web page.
- Other ways of firing the script not requiring manual addition to the URL or use of a bookmarklet, e.g. a searchbox of some kind (either accepting a URL as input or keywords for the titles of broadcasts).
Please do leave comments or questions.
|
