Thanks for the memories

When Kevin Humphreys and I wrote the first version of GATE back in the 1990s we used Tcl/Tk, a nice clean scripting language with an extensible C API underneath. One of the innovative things that Tcl provided was a dynamic loading mechanism, and I used it to allow CREOLE plugins to be reloaded at run-time. A year or two after we'd released the system I could often be heard cursing my stupidity — the reloading system worked well when it was configured, but getting it to run cross-platform at user sites with diverse collections of underlying shared C libraries was a huge pain in the bum.

Fast forward 15 years or so and the class loader code that I put into GATE version 2 (the first Java version) also has some pain associated with it, and it is a real pleasure to see this post with all its carefull study and presentation. Even better, a new chunk of code to take away one of the gotchas with classloading and memory consumption in long-running server processes. Nice one Mark!

One other thing springs to mind — the design choices that we took for GATE 2 (around the turn of the millenium, with a first release in 2002) turned out to be pretty good, by and large (more luck than judgement on my part, of course). GATE has mushroomed orders of magnitude beyond our original plan in the intervening period, but despite a few creaking joints it still holds its own. That's a credit to several of the long-term inmates of the GATE team, and also to Java (and its later offshoots like Spring, Groovy and Grails). It's easy to get blinded by the Next Big Thing in computing, but if you stand on solid foundations (and keep working on reusability and refactoring) you can have your cake and eat it!

(And sorry for the cheesy title.)

Permalink.

Open Data at the National Archives

The GATE team, Ontotext and SSL have won a contract to help open up the UK National Archive's records of .gov.uk websites (going back through 1997 and comprising some 340 million pages).

I've been quite ignorant about this stuff until recently, and it has been a pleasure to discover that the archives and related organisations are actively pursuing the vision of open data and open knowledge. This project has taken a big step forward in the UK recently, with government funding allocated to publishing more and more material on data.gov.uk in open and accessible forms. The battle is by no means over, but I'm really looking forward to contributing in a small way to this work, and, hopefully, showing how GATE can help improve access to large volumes of government data.

We're going to use GATE and Ontotext's open data systems (which hold the largest subsets of Linked Open Data currently available with full reasoning capabilities) to:

1. import/store/index structured data in a scaleable semantic repository
   • data relevant for the web archive
   • in an easy to manipulate form
   • using linked data principles
   • in the range of 10s of billions of facts
2. make links from web archive documents into the structured data
   • over 00s of millions of documents and terabytes of plain text
3. allow browsing/search/navigation
   • from the document space into the structured data space via semantic annotation and vice versa
   • via a SPARQL endpoint
   • as linguistic annotation structures
   • as fulltext
4. create scenarios with usage examples, stored queries
5. show TNA how to DiY more scenarios

Quoting from the proposal,

"Sophisticated and complex semantics can transform this archive... but the real trick is to show how simple and straightfoward mechanisms (delivered on modest and fixed budgets) can add value and increase usage in the short and medium terms. ... We aim to bring new methods of search, navigation and information modelling to TNA and in doing so make the web archive a more valuable and popular resource.

Our experience is that facetted and conceptual search over spaces such as concept hierarchies, specialist terminologies, geography or time can substantialy increase the access routes into textual data and increase usage accordingly."

Text processing technology is inherently inaccurate (think of how often you mis-hear or mis-understand part of a conversation, and then multiply that by the number of times you've seen a computer do something stupid!); what can we do to make this type of access trustworthy?

"Any archive of government publications is an inherently a tool of democracy, and any technology that we apply to such a tool must consider issues relating to reliability of the information that users will be lead to as a result, for example:

• what is the provenance of the information used for structured search and navigation? have there been commercial interests involved? have those interests skewed the distribution of data, and if so how can we make this explicit to the user?
• what is the quality of the annotation? these methods are often less accurate than human performance, and again we must make such inaccuracy a matter of obvious record lest we negatively influence the fidelity of our navigational idioms

Therefore we will take pains to measure accuracy and record provenance, and make these explicit for all new mechanisms added to the archive."

So open science (and our open source implementations of measurement tools in GATE) will contribute to open data and open government.

More open stuff.

Permalink.

Cloud Computing, GATE and Text Processing

When a new thing comes along in computing the first thing that happens is that a small and exclusive set of nerds like me get all excited. If the excitement seems likely to relate to the real world in any fashion that might actually generate someone somewhere some money (or can be spun as something that might do so) then the next thing that happens is that the marketing departments of 1001 IT corporations jump in with both feet and start generating acre after acre of turgid prose about how their aged and creaking product line is actually a prime example of Phenomenon X, the Bright New Thing of Computing.

So it has been with software "in the cloud", which is, it turns out, actually quite a good idea in various ways (setting it apart from most new trends in IT). What does the Cloud Computing commonly refer to (now that the sound and fury of the marketing teams has had a chance to settle a little)? Three things:

• software as a service (SaaS), for example Google Docs or SalesForce.com
• platform as service (PaaS), for example Google App Engine
• infrastructure as service (IaaS), for example Amazon Web Services and most famously their Elastic Compute Cloud (EC2 -- which probably did most to popularise the term in the recent period)

These three now consitute the new wave: they are one of the main tracks that Google is betting on (SaaS and PaaS), what Amazon continues to succeed with (IaaS), and the grist for a hundred new startup mills (from specific applications like searching US campus sites to infrastructural help for cloud developers).

What does it have to do with GATE? IaaS is particularly well-suited to hosting text processing, which is typically bursty in its computational cost and therefore ill-suited to fixed infrastructure. SaaS is great for the provision of large web applications that are complex to install and maintain (like GATE Teamware). Hopefully this and other cloud offerings will be available on GATECloud.net in the not too distant... so watch this space!

Permalink.

Certifiable GATE gurus wanted.

In my previous post I described how we came to start taking our user community more seriously again; in the first part of 2010 the effect of this turn has been that the world and her dog seem to be beating a path to our door with requests for technical support, training, bespoke development and/or access to our latest prototypes. In fact it is proving difficult to keep up with demand, so: if you're a GATE expert how about getting certified and taking on some of the work with us? If you have a good knowledge of one or more part of GATE (and/or related application domains), please get in touch. (We promise not to tell anyone that you're certifiable :-) .) Permalink.

I love GATE users (though I couldn't eat a whole one).

Users. A bit of a nuisance. They insist on asking questions, testing limits, finding bugs. Around 5 years ago, after something like a decade of giving away software, the GATE team felt very like our old systems administrator, who had a habit of saying "the only secure network is one without any computers attached": we knew that our user community was a good idea in principle, but we really rather wished they'd all leave us alone. In fact we did our best to discourage GATE users: we stopped doing regular releases, we ignored the mailing list, and if we could have figured out how to take the thing out in the woods and bury it under a tree we probably would have.

We failed: GATE refused to die, people obstinately continued to use it, and, as we used it ourselves for all sorts of projects, more and more features were added, quality and functionality improved, and every time we decided it was all over someone would turn up with a pile of cash and a novel problem. So we conceded defeat and resolved to succeed. I think.

This is all a long-winded way of explaining our shift in emphasis over the past year or so: we are introverts no longer, but happy and well-adjusted user-friendly liveware. Text processing for ever! Forwards to world domination comrades! Oops, wrong blog.

So now we're back to actively supporting our users and growing our community. We've upgraded the documentation, we're running regular training weeks and developer sprints, and we've built up several new products and services around the core GATE code to cater for more of the cases we've seen of people trying to deploy text processing over the years (15 of which, incredibly, have passed under the bridge since we first set metaphorical pen to digital paper for GATE version 0.1). We've also revamped the website and no longer look like something that might have been produced at CERN circa 1995.

So far the response has been quite astonishingly positive... so perhaps users aren't such a bad thing after all.

Permalink.