Code moved to GitHub

I've finally transferred all code output from last years' project to GitHub. The Perl based analysis and RDF conversion tools, as well as the PHP site source are all available as single repository.

https://github.com/edchamberlain/COMET

Back with more data!

As announced by Jim Michalko on the OCLC research blog, we've launched another dataset!

Its yet more bib data, this time comprising over 600,000 records originating from Worldcat as RDF triples. We've also loaded most of this into our triplestore. OCLC have enhanced this data with links to the FAST and VIAF authority services.

Even better, the previous two datasets we released have also been enhanced with the same links. There are still some things that could be better, especially our vocab choices around VIAF expression, but the data is there.

This data is licensed under an ODC-by Attribution License and is one of the first to make use of OCLC's newly updated community norms (details here), their preference for licensing Worldcat data for reuse. http://www.blogger.com/img/blank.gif

This is slightly in contrast to the pain free PDDL we've managed to provide so far, but we and OCLC are interested to see what users will make of this. The attribution is handled at a dataset level and should be relatively easy to implement and maintain

Dealing with attribution stacking was a major problem we encountered with COMET. That was partly due to Marc21s' inability to manage multiple record identifiers well and necessitated complex decision making regarding record ownership. Hopefully, the clear attribution policy set out here should be much easier to handle than the 'hobo stew' we encountered in our catalogue (as Jim puts it)!

I'd like to thank various folk at OCLC (especially our lead contact Eric Childress) for their support and patience over the past few months whilst we worked through a number of technical, administrative and legal points. They were voluntary partners on COMET but have given us a lot of time and assistance.

Next up, (when I find the time), will be enhanced links to Library of Congress subject headings and the recently released Name Authority File for everything in out triplestore.

Final post

Its time for the final project post for COMET. Here we summarize major outputs over the past six months, cover what we've gained in terms of skills and the most significant lessons we've learnt. We will also take a look at what could be done to follow up on the project.

COMET was perhaps overly ambitious for a six month project, but we've made some firm progress in a number of areas relating to libraries and their open distribution of data.

---

Major outputs
- Document on data ownership analysis - A document describing the major sources of data in the Cambridge University Library catalogue, and making some comments about ownership and licensing around re-use.

- Workflow proposal and tool for record segmentation by vendor code, based on the above work. Suggests a methodology for sorting records when a vendor specifically requests a license other than PDDL.

- Marc21 to RDF triples conversion utility - A standalone tool designed to get
data out of the dead Marc21 format and into something better quickly. It features extensive CSV based customization, see the readme file for more details. Our digital metadata specialist Huw Jones was largely responsible for making this happen.

- data.lib.cam.ac.uk - Our first run at a library-centric open data service. It includes:

• 2.2 million linked RDF bibliographic records licensed under a PDDL with more to come
• A SPARQL endpoint provides access to some of the above data
• Supporting documentation, including an FAQ and SPARQL tutorial, aimed at a 'first rung on the ladder' for RDF and SPARQL novices.

- Application framework code for the above - a PHP / MYSQL application framework to store and deliver RDF data in a variety of formats, with a flexibile SPARQL endpoint. Also includes an experimental library of congress subject headings enrichment utility. A 'getting started' document covers installation and data loading. (For a flashier alternative, take a look at the Open Biblio suite.)

- An interesting sideline into the world of microdata and search engines

- Talks and presentations on Open Bibliographic Data at Birmingham and Manchester

---

Next steps
Publishing 'more open/linked data' would be useful, but data alone will not solve the challenge to improve resource discovery in the UK cultural sector. Here are some thoughts on what could come next. This is quite an eclectic list of ideas and musings on next steps that the Discovery programme could take, with some deeper focus around RDF:

1) Useable services for a wider audience
Open bibliographic data is one thing, but a certain level of skill and understanding is required to fully appreciate it, a criticism of the wider open data movement. To spread the word and enthuse a wider audience beyond 'data geeks', it would be great to see working services built around a framework of Open Data, or at least some impressive tech demos (Its worth mentioning Bibliographica here, which is already a great step in this direction ...)

2) RDF
If RDF is to continue in use as a mechanism for publishing open bibliographic data, its application needs further thought and development. Here are four suggestions:

2.1) Move beyond pure bibliography into holdings data.
In library systems and services, the real interactions that matter to library users are focused around library holdings. This data could potentially be published openly, and modeled in RDF. Links could be established to activity data to provide a framework for user driven discovery services.

2.2) 'Enliven' linked RDF data.
Like most open bib data, we've published a static dump of our catalogue at one time. It would be great to see pipes and processes in place to reflect changes and possibly track provenance. This is not as simple as it sounds, do we provide regular full updates or track incremental changes?

2.3) Better ways to get to RDF.
RDF data is valuable in its own right, but arguably needs easier access methods than SPARQL. Combining RDF data with better indexing and REST API technologies would be useful in widening its access and making it a more 'developer friendly' format. Thankfully, many RDF based tools offer this functionality, including the Talis platform. The Neo4J graph database technology also looks promising.

2.4) Recommendations for RDF vocabularies and linking targets for linked bibliographic data.
I think this needs to happen soonish, otherwise we will still be producing different attempts at the same thing over and over again. It does not need to be complete or final, but a useful set of starting places and guidelines for bibliographic RDF is required. The Discovery program is well placed to provide these recommendations to the UK sector. That would be a great start internationally. Then we can just get on with producing it and improving it :)

3) Cloud based platforms and services for publishing bibliographic data
COMET has shown that this is not yet as easy or cheap as it could be. With library systems teams and infrastructure often overstretched, taking on new publishing practices that do not have an obvious immediate in-house benefits is a hard sell.

To make it more palatable, better mechanisms for sharing are needed. The Extensible Catalog toolkit already provides a great set of tools for doing this with OAI-PMH. Imagine a similar but cloud based data distribution service whereby all a library has to do is (S)FTP a dump of its catalogue once a week. This is transformed on the fly into a variety of formats (RDF, XML, JSON etc.) for simple re-use, with licenses automatically applied depending on set criteria.

4) Microdata, Microformats and sitemaps
This is how Google and Bing want to index sites, and thus how web based data sharing and discovery largely happens outside of academia and libraries. The rest of the Internet gets by on these technologies, could they be applied to the aims of the Discovery programme? What are the challenges standing in the way, how do they compare to current approaches? We've made some first steps into this area by using schema.org microdata in a standard library catalogue interface.

---

Evidence of reuse
We were late in the day releasing our data, so re-use has so far been limited. We've been trying to consume it ourselves in development and our colleagues at OCLC and the British Library have provided useful feedback. We are glad to see it included in the recent developer competition. We've pledged to support our data outputs for a year, so will respond actively to any feedback from consumers over that time.

---

Skills
This project entailed a large amount of 'stepping up', not least on my part. Other than the odd Talis presentation, I had only a conceptual understanding of RDF. Now I've helped write tools to create it and worked with RDF stores and application frameworks. The time out to gain this skillset has been invaluable for me. The book, 'Programming for the Semantic Web' has saved my sanity on a number of occasions.

In terms of embedding this knowledge, our SPARQL workshop is designed to provide the first rung on the ladder for librarians and developers interested in RDF.

Despite this, we've suffered by doing everything in house, and the steep learning curve around RDF has meant that progress is not always as it could have been. Our current datastore is holding over 30 million triples, and we've still not been able to load all of our data output. This has hit the limits of ARC2/MYSQL and we will need a more robust back-end if we are to progress further.

Our RDF vocab choice is also a bit of a shot in the dark in places, and there are things with our data structure that could do with improvement.

If we are to continue to work with RDF data, we would like to bring in external assistance on scaling and development, as well as RDF vocab and modelling.

---

Most significant lessons
To finish, here are some random reflective thoughts ...

1) Don't aim for 100% accuracy in publishing data. In six months, with 2.2 million records that were written over 20 years in a variety of environments, this was never going to happen. I would hope that at least 80% of our data is fit for purpose. This is 80% more open data than we had six months ago.

2) Ask others. There are strong communities built around both open and linked data. Often, its the same people. They can be intimidating, but are useful. With hindsight, we should have asked for community guidance on issues of vocab and technology choice earlier.

3) Its an oldy, but please use Open Source and don't code from scratch if at all possible. The ARC2 framework may have its limitations with our scale of data, but it allowed a workable data site hosting 25k records to be assembled in two days. God bless Github.

Where exactly DOES a record come from?

Early on in the COMET project, Hugh Taylor assembled a complex document attempting to describe problems inherent in understanding the origin of a Marc-encoded bibliographic record. It also included a thorough analysis of Cambridge University Library data and the number and nature of vendor codes contained there-in.

We've updated this document with information on the various contracts and agreements associated with each vendor code to reflect the final work. The next problem was how to make sense of it all.

A major sticking point is related to Marc21 and its usage.

In our Marc data, we have four separate fields (015, 038, 994 ,035) that could indicate ownership, some of which may be repeated multiple times in a record. There is to my knowledge no mechanism in Marc21 or AACR2 to indicate which field and thus which vendor code takes precedence over others, (although cataloguers have some 'community knowledge' in this area).

Furthermore, many vendors change code and field used. Most rely on prefixes. Some are simply unhelpful strings of numbers.

In terms of practicalities, we need to ensure that:

1) Records from vendors who explicitly and contractually prohibit re-sharing in any format are excluded, this includes most ebook and ejournal records. Otherwise, there is no good reason not to share a record, although its origin may have an impact on license choice

2) In our case, records from OCLC are segmented due to a need to publish data from that vendor under an attribution license

3) Data from vendors who prefer non-marc output to be shared openly, but want Marc21 output restricted are segmented (RLUK and the BNB in our case) so these records will need to be split off

4) Data produced in house (usually that with no identifier) can be segmented for clarification

5) Everything else from smaller/ specialist record vendors is segmented together with a view to publishing openly

We've had to make some decisions over which field and vendor takes precedence based largely on this order of importance. To do this, we came up with a rough decision tree regarding record ownership:



The above JPG is also available as a scaled vector graphics file created in MS Visio.


One of my final tasks on COMET was to take this decision tree and turn it into a script to export record data for our final exercises in data publishing. I've also released a Perl script as output for COMET on out code page. (A warning / apology, this script is as ugly as the situation it attempts to resolve. It was pulled together at the last minute and could really do with a rethink.)

In the case of both the script and chart, the situation relates to Cambridge's specific and current situation., but should hopefully be useful for those wishing to replicate this activity.

As a personal opinion, I see this confusion regarding ownership as a key barrier that prevents libraries from openly sharing their data.

Furthermore, it is important that we do NOT see a repeat of this problem with the next set of record container and delivery standards.

Its my worry that stacking attribution statements in records at the bibliographic level could lead to similar problems down the road. Attribution at a data-set level, with some indication of the relationship between a record and a data-set seems more practical.

A standardization of practice across the library community with regards to licensing could help ease this pain in the future.

Because we always need more standards.

More data and status of future updates

I'm pleased to announce the next set of data release at data.lib.cam.ac.uk - 1,741,080 records of linked RDF distributed under a Public Domain Data License.

• This data is from two major UK record suppliers, RLUK and the British Library BNB. Both have indicated to us that they have no problem with records being redistributed as RDF, but would rather we did not redistribute Marc21. See a blog post on licensing for explanations as to why.
  

• This dataset is bulk download only for now - we hope to have it in our triple-store in the future

Although major development work on the COMET project has ended, we will continue to open-up as much of our catalogue as possible, publishing and updating datasets over the next year using the open source tools we've developed. In particular, we hope to carry data from OCLC under an Open Data Commons attribution license and enrich all datasets with FAST and VIAF links. We've been experimenting with these for the past month and results are promising.

In a follow-up post, I'll explain how we decided which data could be shared.

Friday update ...

A few things on a Friday.

Firstly, I've written a small piece on getthedata.org about data.lib.cam.ac.uk and the various mechanisms for querying and retrieving data, also mirrored on our FAQ. May be of interest to those in the Discovery developer competition.

Development work is all but done, so we've also published the application framework code behind data.lib.cam.ac.uk on the code page. This PHP based site provides a lightweight approach to RDF publishing and makes a great starting / exploration point for libraries wanting to publish data as a RDF.
More details and a read-me are available on the code page. As with all our output, its provided 'as-is' under a GPL, but we welcome feedback.

As the COMET deadline approaches next week, we are still working to release as much data as we can. Sadly, we are still waiting on final confirmation from some external bodies. As such we will work to publish and republish data using existing tools throughout the following year, as we can.

Cost benefits

The JISC has asked us to blog on cost benefits of providing open data. I'll give a rough indication costs based on time spent and an idea of what I think the benefits may be with an indication of how the two weigh up.

Costs:

1) Marc21 Data 'ownership' analysis - ( 5 days staff time at SP64)
Mapping and conversion of bibliographic information. An experimental and iterative process.

2) Marc21 to RDF data conversion - ( x2 developers at SP53)
Again, this has been drawn out through experimental work. Several methods and iterations were tried. Those aiming to repeat this may not incur the same cost.

3) Web infrastructure development and record curation- (x2 developers at SP53)
A lightweight approach to development was taken using existing application frameworks. Time was also spent understanding underlying principles of RDF stores and associated best practice for linked data. Several iterative loads of data were undertaken in parallel with Marc to RDF conversion.

4) Hosting and sustainability costs - costs tbc
COMET's web infrastructure makes use of existing VM and MYSQL infrastructure at CARET, so additional infrastructure costs were negligible and hard to determine. We've promised to keep the service running for a year.

5) Other stuff
Project management etc.

External benefits:

• Substantial contribution to Open Bibliography - Open data is arguably a good thing, and whilst it has flaws, ours is hopefully useful enough to be useful to others in its own right

• Clarification on licensing agreements with record vendors - Much headway has been made into this issue by the COMET project, with some clarification on licensing preferences for RDF data from three major UK record Vendors, OCLC, RLUK and the British Library. Down the line, we hope that these organizations will formalize their agreements with us so that others can benefit, which will hopefully help in publishing more data

• Advice on how to analyise records to determine 'ownership' and lightweight (Perl, PHP, MYSQL based) tools to create and publish RDF linked data from Marc21

• Experiments with FAST and VIAF - Two potentially useful data sources

In house benefits:

• Community interaction - There is strong interest in Open Bibliography an its benefits. The University Library has also benefited greatly from its interaction with the open and linked data communities, in its work with OCLC and with others through the JISC Discovery program

• In house skills - We've gained vital in-house understanding of the design and publication of RDF. We've developed basic training materials around SPARQL for non-developers, which could play off down the line

Summary:
External benefits clearly outweigh internal benefits, although as external benefits affect the whole library community, they also benefit us!

Whats' clear is that Open Data is not free data, at least not to us. We could have simply dumped our Marc21 or Dublin core XML and have been done with it, and for many that would have sufficed.

Instead, combining our wish to publish more Open Data with a need to learn about Linked Data (and thus lashing two fast bandwagons nicely together) has pushed the costs far higher.

However, by publishing linked data we've hopefully made our output more useful to a wider community than library metadata specialists, and in that sense added value.

More data being published means greater community feedback to draw upon, which should result in lower costs for those repeating this exercise.

It may indeed be several development cycles before we or others fully reap the benefits of this work. Alternatively, things could move in a different direction, with RDF based linked data falling by the wayside in favour of more accessible mechanisms for record sharing, in which case, our work could be useful in avoiding mistakes.

Project update and following in our footsteps

As the COMET project comes to a close, we are working through the final piece of ownership analysis to identify more data for RDF conversion and publication.

We've loaded sample records with FAST and VIAF and are in discussion with OCLC about the best way to model them.

In the interim, we've been asked to briefly blog about helping others to 'follow in our footsteps'. We ourselves were very much following the work done by the Open Bibliography project, even if we had a slightly different focus and toolset. There was a reason for this. One of the aims of COMET, at least in my mind was to see how easy it would be for an average library systems team to attempt the impressive work seen on projects such as Open Bibliography, work done by those who already had considerable experience of linked data and open licensing.

Here are few tips based on our experiences.

1) Be aware of your licensing. Whilst there is no good reason not to share data, some vendors have explictily prohibited it. We hope to have a better summary of our work examining out contracts up soon, but the main thing to look for is in explicit contractual agreements from vendors that prohibit re-sharing.

Otherwise, you then have to choose an appropriate license. We've ended up 'chunking' our data so that in the public domain stuff can can be PDDL will be. Otherwise, some form of attribution license would be required.

Thankfully, few other libraries should have as complex collections of data as Cambridge, with most relying on one or two vendors.

2) Think about the backend and issues of scaling before you start. We approached COMET with an exploratory hat on, the world of triplestores and SPARQL was new and we were not sure how much data we would be able to publish. The ARC2 datastore we eventual chose was great to develop with, but ultimately unable to adequately store our entire data output. For libraries with smaller datasets, (under half a million records or 16 million triples), its well worth a look. ( At least we are in good company with this, I've noticed that the DBpedia backend does not provide access to everything... )

3) Take a look at our tools. - We have an Perl MArc21 to RDF generation utility ready to go. We chose Perl as it is often used by systems librarians to 'munge' and export data. Our mapping is customisable, and the baseline triples it produces easy to load. We've based a lot of the final output on work done by the British Library in modelling the British National Bibliography.

4) RDF vocab modelling is itself something of a burden, you can give it a lot of thought and concern, try numerous different schemas and still not be sure as to the usefulness of your output. Our advice is focus on useful elements such as subject entries and identifiers. Be careful with the structure, too many links and nodes can lead to data that is 'linked to death'.

Don't expect to get it right first time.

And now for something completely different ...

Well, fairly different. Its not record licensing or RDF vocabs, but has relevance to the wider aims of the discovery programme. Prompted by a recent tweet, I'm blogging about search engine exposure in an academic library context.

Google, Yahoo and Bing have recently clubbed together to create schema.org, a set of data standards for websites to provider richer information for search engines, its a very lightweight 'semantic' approach. Many, including Eric Hellman at ALA have said the libraries should be on board with this, and that Search Engine Optimisation should be one of our aims. Some have taken issue with this approach, but his ideas seems a far cry from the traditional closed library OPAC.

For me at least, his timing was right. As well as working on COMET to produce open, linked library metadata I've been quietly experimenting with Search Engine Optimisation (SEO) on the side.

The main problem lies in crawling. Since Google dropped OAI-PMH support a few years back, they now only accept sitemaps, large XML files of persistent URLS which their robots can crawl over.

Many library catalogues do not even have persistent URL's, let alone sitemaps and many lack the means to develop their own.

Thanks however to some nifty 'unsupported' features in Aquabrowser, our current discovery service for library materials, I've been able to generate sitemaps for each record in the system, which have persistent URLs as standard. I understand that VuFind, an open source equivalent, can also do this.

With help from colleagues at Harvard and Chicago, I've also customised the full record display to work with two microformat standards, COINs (Z39.88) and schema.org microdata. This allowed me to get around some inadequacies in the catalogue page design, in allowing titles to be read over the page title.

Good news, right?

Yes and no. Since April, about 180,000 record page have been indexed by Google. Sounds promising, except we submitted sitemaps of over 5 million URLS. By the time they finish, we may have well moved from Aquabrowser to a new platform. We can of course throttle up the crawling by Google, but need to watch its effect on our live service, even with some fairly beefy servers.

A few thoughts:

• This was not easy and a bit experimental, but undoubtedly a useful exercise and a nice comparison to our work on COMET. I can publish 1.3 million fairly rich records as RDF in a week, but no search engine right now would want to touch them. Outside of the linked and open data communities, few would take notice and it will probably not get extra people through my doors

• Realistically however, search engine exposure will bring few extra people to Cambridge libraries, unless we can get record pages linked to in a useful manner and drive results rankings up to the front page. One rough search for an indexed record 'the cat sacase library' gets me Open Library and Worldcat as top hits, but no Cambridge :(

• Schema.org is aimed at e-commerce and improving services like Google shopping. Metadata choices are limited to author/title, description, identifers, and availability. Seems fair enough, given that Google is an advertising company, but where does academic research or even basic library use actually fit in? Its designed to be extendible. Could an 'academic web crawler' make better use of the tags? What about the clever bods at Wolfram Alpha or True Knowledge? (They are also welcome to some RDF ...)

• Few other libraries have even bothered with search engine exposure and optimisation, mainly due to problems with integrated library systems (Huddersfield and Lincoln being two known exceptions). Their reasons are practical, one rampant crawler could bring down both back and front office systems and few systems support permanent URLs. Sadly, this trend may not be reversing (Aquabrowser being an exception). Services like Summon, EBSCO discovery and Primo central are not search engine friendly, being large closed indexes themselves. Permanent URLs for records may not be a given. Summon even does away with a full page per record, ironically because people don't 'expect that from a search engine'...

• Will schema.org really take off? I am getting the feeling that I've been here before. I remember being told in training sessions many years back to 'always tag my URL' and include meta tags in web page headers. As a young, budding Librarian, this sounded great. I was very disappointed to later learn that most engines ignored them, as they were a great way of breaking ranking systems. How will this 'system gaming' be avoided with schema.org and other microdata formats?

So to summarise, right now I can expose a little data to a lot of people and hope they see it amongst a lot of other data, or expose a lot of data to a little set of people, who just might do something great with it. Meanwhile, those that use our library will probably still know to use the catalogue.

You can try our Google customised search for http://search.lib.cam.ac.uk here. I'd be interested to see what people think.

Two more updates ...

After this weeks' launch of data.lib.cam.ac.uk, its good to follow up with some more updates.

First up is a SPARQL workshop, a three-part tutorial on RDF and the SPARQL query language aimed at I.T. workers with little to no knowledge of the semantic web, technically minded librarians, web designers and those with an interest in metadata and its (re)use.

One of the primary (and justifiable) criticisms of RDF is the high entry barrier. Much of the literature assumes a high level of technical and semantic web knowledge.

In an attempt to 'help others follow in our footsteps', I've tried to represent the learning done by myself using SPARQL to query our dataset. This may not actually lower the entry barrier, but will hopeful provide those with an interest in RDF with a base-line starting place.

Secondly, we are beginning to better link our Linked Data!

we've made some experimental gains in URI enrichment, supplementing our graphs for catalogue subject entries with links the the Library of Congress vocabularies.

See these examples:

http://data.lib.cam.ac.uk/id/entry/cambrdgedb_c1574b4e36a34f04bda61b3ea57b2379

http://data.lib.cam.ac.uk/id/entry/cambrdgedb_2ca5328ca9bebe20f37a7718d5e1f67b

http://data.lib.cam.ac.uk/id/entry/cambrdgedb_2883408d7b714bb6423d5c1ebcb40a48

As our labels are made up of a number of Library of Congress subject components, (subject, geographic, chronological etc) we are taking the inital main entry and representing it with a 'skos:broader' vocab. We would love some feedback on this approach, which is little more than a starting point. As we are using http requests against the id.loc.gov service, we are also running into scaling issues with our 600,000 + subject entries.

Enrichment is being done directly in our RDF store, so for now this is not being reflected in our bulk data downloads for now.

data.lib.cam.ac.uk now live

Good news everyone! Our open data service, and the major output from the COMET project is now live. You can find it along with our first dataset of over a million records at data.lib.cam.ac.uk. We've been running in stealth mode for a while, but are happy to announce the live service today.

This dataset also part of the first Discovery developer competition. Its our first 'in-house' attempt at producing Linked Data and we welcome feedback on it. See our FAQ for more information.

As we wrap COMET up over the coming month, we will have additional outcomes and datasets available at data.lib.cam.ac.uk.

In related news, the Open Bibliography project blog has published its end of project post. Its an excellent read highlighting some great achievements and provides strong example-led arguments for the value of Open Bibliography.

On licensing ...

Background

Licensing of bibliographic metadata is far too complex a subject.
One of the major aims of COMET has been to see how easy it is to identify records from major record vendors in the UK HE environment and address issues and concerns around data reuse. This work is still on-going, but its high time we got a post out on the subject, explaining where things are at.

Like most university libraries, Cambridge University Library relies heavily on external record vendors to meet its cataloging needs and keep up to speed with a high intake of material. Much of this data has its potential reuse and republication covered by an explicit contractual agreement. At the same time, we understand and support the need to produce Open Data as a platform for a better set of services for Higher Education.

State of play

Through the Comet Project we have been investigating our data for traces of 'ownership' and have been examining contracts. We've contacted the major record providers and some have indicated a preference for certain types of licenses in data re-publishing.
As an example, the British Library have published the British National bibliography as RDF formatted data under a PDDL and are happy for others holding BNB data in their catalogues to do the same,(although there is not yet any formal announcement to this effect!).

OCLC, perhaps the biggest record supplier have recently expressed a preference for ODC-By attribution licensing. We are one of a number of libraries working with OCLC to investigate the practicalities around this.

We in turn produce a substantial amount of data in-house, and would still like to publish this under a Public Domain Data License. Identifying this data was actually more difficult than it should be, we ourselves insert no 'made in Cambridge' label on our records, so we had to identify this set via a process of elimination.

Given this disparity between approaches to licensing, we will be aiming to produce several different datasets under established Open Data Commons licenses.

In terms of URI structure and vocab choice, they will be identical, but each whole set will be represented by a separate graph in our RDF datastore itself linked to the appropriate license information. For data produced under anything other than a PDDL, license information will also be made explicitly obvious to those downloading in bulk.

A final solution?

This area is still in flux. We feel that although licenses may vary, there should be no barrier to publishing data for others to reuse. We hope that over time, the library community will work to a set of established practices and community norms over data publishing. This work represents one of the first steps taken in this area.

Public Domain Data Licensing is an obvious ideal and one which we prefer, but adopting a pragmatic approach now can get more useful data out in the wild quickly. Whilst stepping back from PDDL or CC0 is next to impossible, adopting a slightly less open standard as an initial position which can be rethought downstream may be more palatable. Just steer clear of non-commercial licenses for data!

Marc21 - another reason for deviation

Whilst there is strong interest in and backing for Open Bibliographic Data within the international HE Library community, there have been concerns raised about its impact on organizations that rely on commercial Marc21 record supply to maintain and develop services.
We recognize that partner institutions have valid commercial interests in this and benefit ourselves from such services. As such, we are only releasing Marc21 that we can claim total ownership of. Other data is being released as RDF only. We believe our RDF output is sufficiently altered to make cross-walking it back to useful Marc21 next to impossible.

This may not be an approach suited to everyones' tastes, but it is pragmatic. To put this in perspective, how many open data consumers really care about Marc21? Its a format that really deserves to die and is irrelevant to the wider conversation.

Some of this post has been distilled down into a forthcoming F.A.Q for data.lib.cam.acuk.

Metadata and standards - URI construction

We are continuing to finalize our RDf conversion and work through linking to OCLC resources. As we are also finalising the datasets we can make available under a permissive useful license, we are currently working of some random samples of catalogue data.

One issue worth highlighting at this stage is that of URI construction. URI's for records and other important entities described in a catalogue are a key component of linked data. We are taking a standards based approach to URI construction, trying to follow guidelines set out by the cabinet office for UK public sector (pdf link).

Our record URI string is quite simple:

http://data.lib.cam.ac.uk/id/entry/cul_comet_pddl_4589705

the /id/entry/ denotes that the uri relates to an identifier for either a catalogue entry or entity described in our dataset. The following identifier string is a mixture of a string of characters for the dataset (which we may remove) and the catalogue records' identifier, already used in persistent URL's for our catalogue interface.

One issue we've not tackled is human readable unique identifiers for creators. The guild portion at the end constructed from a string of characters (say the 100$a in a Marc record) being stripped of punctuation (where errors tend to occur) and run through an MD5 checksum.


http://data.lib.cam.ac.uk/id/entity/cul_comet_pddl_0a72dd0c8fe090f78970db02b336900f

Human readable URI's would be nice, but some attempt at keeping this unique is probably better. If the Library of Congress were to follow suit on their excellent subject work and publish their name authority file as linked data, we could utilize and guids used there. Hopefully, we will be able to provide links to relevant VIAFF (Virtual International Authority File) entries for authors, where they can be matched by OCLC.

I'll follow this up shortly with a post about how we are ensuring the data behind a URI is easily referenced by both humans and machines.

Small (but fiddly) win for URI's ...

Work on RDF conversion goes on. In addition to eventual complete dumps of data, we've also started putting together the pieces for our application to support RDF queries via SPARQL and HTTPD.

We are using the apache extension mod_rewrite to turn human readable uris like the below ...

http://data.lib.cam.ac.uk/id/entry/cul_comet_pddl_4589705

into those easily parsed by the web application dishing out the record content:

http://data.lib.cam.ac.uk/record.php?uri=http://data.lib.cam.ac.uk/id/entry/cul_comet_pddl_4589705&format=html

Its also considered best practice with linked data to provide dish up records in the format required by the requesting agent in their httpd request. This practice is referred to as 'cool uri's'. As an example, if I want to view 'http://data.lib.cam.ac.uk/id/entry/cul_comet_pddl_4589705' in a browser, when the standard http request accepts content returned as 'text/html', then they should see html in their browser.

Conversely, if they want to see rdf+xml content, they make request it via a script or command line, e.g:

curl -H "Accept: application/rdf+xml" http://data.lib.cam.ac.uk/id/entry/cul_comet_pddl_4589705

They should not have to add any kind of file extension (.rdf) to the request uri, although its also nice to support this.

We can handle this within the web application framework, which will involve monitoring requests and parsing incoming uri strings for file extensions, but that will add precious lines of code. Much easier to let the web server take over, which is where mod_rewite again comes in. It allows you to specify a set of rules that monitor for file extensions and accepted content types and rework uri's so a web application can dish out the required format

Sadly, we can't escape regular expressions, and mod_rewite relies heavily on them. Logging is also essential for de-bugging. Here is our htaccess ruleset, with each rule commented. We are still not supporting all the formats available for RDF distribution, but sticking to xml, json, baseline triples and turtle.

Mod_rewite or equivalent tools are a vital part of semantic web infrastructure, and whilst fiddly, a little knowledge can go a long way. Here are three great tutorials:

• Beginners guide to mod_rewrite


• Patrick Altofts' Ultimate guide


• mod_rewrite and RDF

Quick project update ...

(Extracted from previous post for the sake of brevity)


In terms of project progress, we have a workable and easily customizable (CSV configurable) Marc21 to RDF-triples export script nearing completion. We hope to be able to share this towards the end of the project. Getting a suitable triple-store and associated software frameworks in place for a 'data.lib.cam.ac.uk' domain will be the next focus of our technical work. Work also continues on identifying records for sharing and internal discussions on licensing issues within the project, the main barrier to eventual data-release. We've also been in contact with Eric Childress and his colleagues over at OCLC in enhancing our data with identifiers for the FAST and VIAFF services.

Presentation at the 'Open Data Open Doors' event and other musings ...

Yesterday in Manchester, I was asked to give a brief presentation on our reasons for pursuing Open Data. Beyond the "its a good thing to do" arguments, ( which better people than myself have been able to better put across), I wanted to try and give an impression of the 'internal' reasons for doing so. In particular, I was keen to draw on real world examples, including the work done by Rufus Pollock in estimating the size and growth of the printed public domain from CUL's bibliographic data. I also touched on my own personal ideas for resource discovery services at a national level, and how libraries could be providing developer orientated services.

Open (linked) bibliographic data

There were many useful discussions on both Open and Linked data that day, with Paul Walk at UKOLN on-hand to remind us that we should not jump at lumping the two together, and that each was itself something of a band-wagon.

A real cost-driven business case for opening up data was mentioned, which gave me cause for concern. Benefits of publishing data will only be fully realized when developers get to produce useful outputs, which could take several cycles to emerge. Identifying successes outside of cultural heritage was seen as a good way to sidestep this.

Much concern was given to licensing, and the library-centric issues of record-ownership again came to the fore. For many years and in pre-Internet eco-systems, Libraries and Librarians have benefited greatly from shared practices and resources in cataloguing. It would be a real shame to let the technical and legal frameworks developed to support previous generations of activity get in the way of finding better ways to share data between and beyond the library community.

The linked data approach of the Comet project was compared to the formidable API-orientated work on Jerome, taking place over at Lincoln. There was some discussion over the relative merits of each approach.

My personal take right now is both have fairly separate use cases, and that publishing large amounts of data as RDF (or in 'community' formats such MODs, Marc21 etc.) will be more useful for aggregation services than straight API provision, but that any eventual shared-data-service should itself expose data in API's of the highest quality. Thus the work done by Jerome will be of great importance to the RDTF no matter which way things move. Lincoln are also themselves gaining an excellent platform for future service development.

As I argued in my talk, Linked Data still has a high entry-bar, and many developers are much happier with a simple bit of Json over XML/RDF. RDF may not be the easiest means for aggregation (OAI-PMH works for me) but its arguably a great tool for sharing library data beyond the library community in bulk. Apart from anything, self-describing data means we don't have to explain Marc21 to people with useful things to do.

A third alternative, not currently being investigated by the RDTF (to my knowledge) would be crawler exposure of existing catalogues with RDFa or some kind of useful microformat in place.

I raised a point which no-one seemed able to answer, what types of license are applicable to feeds of data, i.e. an JSON or XML API such as those we provide at www.lib.cam.ac.uk/api, or even an Atom/RSS feed?
Would Creative Commons licenses suffice, or do they need the data specific Open Data Commons licenses? If anything, they are more of a service than a resource. How can we imply complete openness (or otherwise) in easily understandable terms?

The inevitable metadata post

Insipred by a recent post on the LOCAH (Linked Open Copac Archives Hub) blog I'd like to begin to tackle the prickly subject of RDF vocabularies. At the recent Infrastructure for Resource Discovery kick off meeting in Birmingham, a choice of vocabularies for linked data was seen as one of the biggest issues facing projects. Alongside licensing issues for bibliographic data, its taking up a lot of time at this early planning end of the COMET project.

Archives Hub have a specialised form of EAD to convert into RDF, and have been putting a serious amount of work into modeling their data as well as identifying useful existing vocabularies to use. When we approached COMET, we hoped to avoid any such modelling of bibliographic data to RDF and instead make use of existing work, part of our greater project philosophy of aiming to minimise any 'coding from scratch'.

After all, plenty of previous attempts to model bibliographic metadata have been made, it should be simple enough not to reinvent the wheel?

Rather frustratingly for a standards loving Librarian such as myself, there is no accepted single set of vocabularies in place to publish library bibliographic metadata. Whilst the W3C incubator group is closely examining the issue, actual output and recommendations seems a while off. If this is symptomatic of anything, its that publishing linked data is still an exception for libraries rather than standard practice.

When scoping our bid, we initally looked at the SIMILE project, specifically its MARC-21->MODS->RDF XSLT based conversion tool. Initial tests proved promising, and MODS is sufficiently rich enough a standard not to loose 'data richness' although we would still have to do some work to create URI's for entities described by the data.

However, outside of this project, there has been little take-up of this concept, (and we could not see much movement around SIMILE). Indeed, very few library metadata standards have been directly expressed in RDF, although the Library of Congress are looking at MADS.

At the same time, we want our data to be as easily reusable possible. Whilst this is an inherent feature with RDF due to its self describing nature, we felt that using popular vocabularies helps to minimise effort and makes our data more easily readable by those more familiar with existing linked data practices than library standards.

Looking at the Open Bibliography project, their choice of vocabularies for the OpenBiblio software underpinning their Bibliographica service is comprised of several generally used vocabularies, most notably Bibo, the Bibliographic Ontology for additional bibliographic elements, including some elements of FRBRisation. It also includes Dublin core for general descriptive terms, FOAF for people and (Will Waites, a developer at Open Knowledge Foundation has gone into some further detail on their development email list).

The initial knee-jerk concern for any meta-data fixated librarian with a MARC-21 to Dublin-Core conversion is that of 'data loss'. As an example, the many MARC21 fields for author or creator (100, 110, 700, 710 etc) are generally flattened to DC:creator. Given the use case of development within a linked data environment, one has to sit back and question the value of having several different types of author/creator or indeed the myriad of additional alternative and uniform title fields that MARC21 based data may present. Following accepted practice from outside of the library sphere could be more useful in this context.

As such, we are currently aiming to generally adopt the set of vocabularies used by Open Bibliography, with some modifications over. Bibo in particular looks to be good vocab to use, with the growing support, Dortmund University Library has recently adopted it for its open metadata. Sadly, they've no MARC-21 conversion script to hand. Our options right now are to adopt the already outdated MARC import script that is part of Bibliographica / OpenBiblio or create our own.

In following posts, we will discuss URI naming conventions for RDF graphs and accepted practice for bibliographic entities.

"Ownership" of MARC 21 records - please comment

The case of for open bibliographic data has been well made. The Open Knowledge Foundations' Bibliographic Working Group has established a set of Open Bibliographic Principles. The JISC Resource Discovery Taskforce has itself produced a comprehensive Open Bibliographic Data Guide, examining reasons for publishing through use cases and the wider context of open data elsewhere within the UK.

With such principles' and use-cases firmly established, one barrier to publishing open data lies in establishing the 'ownership' of a record, ensuring that as far as a library is aware, no existing license agreements with record vendors are breached.

COMET's in ital document on "Ownership" of MARC-21 records is designed to help identify where MARC-21 encoded metadata originates from and assist in establishing its provenance.

The documentation and underlying investigation was performed by Hugh Taylor, Head of Collection Description and Development at Cambridge University Library. Hugh is as familiar as anyone with the vast and varied dataset at the University Library. Given the size and scope of our data, the issues and examples raised will hopefully be of use to anyone else considering publishing of Open Data.

This guide is something of a work in progress, which we will revisit as COMET progresses. Next up is a brief summary of relevant licenses, aiming to provide an overview of what is allowed and not allowed with the array of data we have.

We would welcome feedback in the comments below.

Welcome

Welcome to the COMET (Cambridge Open Metadata) project blog. COMET is a JISC funded collaboration between Cambridge University Library and CARET, University of Cambridge. It is funded under the JISC Infrastructure for Resource Discovery programme.

COMET will release a large sub-set of bibliographic data from Cambridge University Library catalogues as open (under a Public Domain Dedication License) metadata. It will also explore and test a number of technologies and methodologies for publishing XML/RDF.

COMET aims to build upon the successes of previous work in this area.

The library has previously contributed a dataset of 132,130 bibliographic records to the JISC-funded Open Bibliography project led by the Unilever Centre for Molecular Science Informatics at the University of Cambridge, in partnership with the Open Knowledge Foundation and the International Union of Crystallography.

This collaboration began to develop our understanding of the intellectual property and technical issues relating to the exposure of bibliographic data and potential value in linking the data.

COMET will have a particular focus on library-catalogue derived bibliographic data, aiming to provide the University of Cambridge and wider academic community with a readily accessable RDF store for bibliographic data. The development and installation work behind this will be documented in such a way as to be repeatable by others.

We will also investigate and document the availability of metadata for the library’s collections which can be released openly in machine-readable formats and the barriers which prevent other data from being exposed in this way.

The project will also explore the value of a linked approach to enrichment of records using services provided by OCLC to assign FAST (Faceted Application of Subject Terminology) and VIAF (Virtual International Authority File) headings to the metadata, allowing the development of innovative services for information retrieval and resource discovery.

You can find detailed information regrarding COMET on our about page, including full aims, objectives and expected ouputs, as well as a project plan.