Skip to content

open data, APIs, and data formats - lots of small pieces and a platform

When I see a couple of related posts within a few days, and it coincides with some things I’ve been thinking, maybe it’s time to post something on the matter.

One of the trends we’ve seen over the last 20 years or so, which seems to be accelerating, is the idea of “openness”. Open Source software, open document formats and APIs, and a consequence of particularly the latter, open data and knowledge itself.

Of course, there is a yang to this yin. Established players in particular aren’t necessarily thrilled with open source, open document formats, open APIs, and open knowledge. That makes sense. Big established software companies and publishers, to name a couple of players, stand to lose an enormous amount of market share, and perhaps as importantly, will need potentially to go through extraordinary internal upheaval to adapt to an environment where all software must be interoperable though published standard document formats, and where online walled gardens of data and services are replaced by much more distributed and interoperable systems.

If we look to how networks are implemented, there are two basic models, which new generations of network tend to oscillate between.
At one extreme, the intelligence of the network is concentrated at the center of the network, with the devices at the edge of the network being “dumb”.
At the other extreme, the network intelligence is in the devices at the edge of the network, and the network itself is dumb. This latter model is in the ascendance at present.

It seems that this model, which is reasonably well understood in network theory, applies at least by analogy to web content and services. The traditional model, of concentrated services and data, exemplified in services like massive job boards such as monster.com, represents “data at the center”, while new, albeit somewhat experimental services like edgeio (the name tends to give it away) represents the “data at the edges” concept.

At the end of last year I “predicted” that one of this year’s big trends would be a transition to “data at the edges”, but that was a typically end of year glib prediction. In reality, I think we are seeing the beginning of what may be a paradigmatic shift, with all the pain these tend to entail, toward distributed knowledge and data. Of course, this may fizzle out in no time at all, because it does present some serious technical, and user oriented challenges.

In the last week or so, three articles which touch on this issue in one way or another have been published, all well worth reading.

Chris Messina considers Gdata, particularly its API, which appears to be Atom with some extensions, and Blogger’s adoption of this Gdata API, Tim O’Rielly takes up the issue in reference to Chris’s article, making the observation that “a platform beats an application” (which the history of the web demonstrates pretty well - it’s seen off eWorld, MSN, Compuserve, AOL - by which I mean all the proprietary web competitors that emerged in the early 1990s, some of which have morphed into ISPs or portals) while today, over at the Open Knowledge Foundation blog Rufus Pollock observes “open apis don’t equal open knowledge”. Rufus refers to a recent article by Kragen Sitaker, on the implications of software as a service for users, also well worth reading.

There is a lot to be played out here - in theory and in practice. From a business point of view, a technical point of view, and a users point of view. But I think its one of the really interesting things going on with the web, despite not having an interesting acronym, and despite being largely invisible to users.

Microformats come into this by providing a simple interoperability layer for all kinds of data, some of which we already have microformats for (like contact details) but a lot of which we don’t yet. That’s the microformats way, develop the formats as and when they are needed.
The web is itself a platform, and if Tim O’Reilly is right (which he often is), web based platforms, small pieces loosely joined will beat out monolithic apps.

I’m certainly hoping he’s right.

Post a Comment

Your email is never published nor shared. Required fields are marked *