Alex Barnett blog : XML

Why OData Matters (IMHO)

alexbarnett — Fri, 19 Mar 2010 15:24:00 GMT

Earlier this week I was in the MIX10 crowd as Douglas Purdy announced the Open Data Protocol (it was a great presentation - summarized here).

I want to share with you why I think OData could be a very big deal…But before we go there...let's start with the basics...

What is OData? Where Did OData Come From?

To understand the history of how OData came to be, you need to understand how project "Astoria" came to be...I won't go over that again as this is already pretty well documented. ~~Astoria~~ OData has come a long way since.

In a nutshell: Today, OData builds on a few conventions, popularized by AtomPub (see OData AtomPub Format), to using REST-based data services. These services allow resources, identified using Uniform Resource Identifiers (URIs) and defined in an abstract data model (see OData URI Conventions, to be read and edited by web clients using simple HTTP messages (see OData Operations).

An Open Data Protocol for the Web

OData offers a standardized way for programmable data to be made available across the web and in turn allowing "consumers" of that data to rely on a set of conventions to be followed that in turn allows many interesting things to happen if widely adopted...

...And to this end:

As announced, OData has been released by Microsoft under the Open Specification Promise (OSP) "to allow anyone to freely interoperate with OData implementations" . Since then, the W3C has invited the OData team to Bring OData to a W3C Incubator (I haven't seen a public response yet but I urge the team to do so.).
OData is not a Microsoft-only thing and it won’t succeed if it is. The originating philosophy was about transparency in the design process, with an Open end-point as the goal - not a .NET lock-in play (“agree on standards and compete on implementation”). This approach has already yielded an initial set of clients, servers, services, and tools. Today, a number of OData SDKs and libraries are available for .NET, Java, PHP, iPhone (Objective-C) and more – and there’ll be more coming.
There is a small but growing ecosystem of non-Microsoft "producers" and "consumers" - (where OData "producers" include Netflix’s catalog as OData and the VanGuide (a social map of Vancouver Open Data) and Public data from the city of Edmonton as OData) and OData “consumers” - standard web browsers, RIA "data explorers" – such as Sesame OData Browser, and the client libraries mentioned above – p.s. somebody build a javascript-only data browser please!)

Where can OData take us?

The clue is in the OData icon (next to the RSS feed icon. Can you see the similarities?

The big idea here is that in the same way we have the "RSS" feed icon, we'll get used to seeing the "OData" icon on commercial and non-commercial websites everywhere (especially for government-related data). So in the same way you know today that the RSS icon means "get an XML feed for this content", the "OData" icon means "get this web data" - you'll know (and your client will know) what to expect in terms of how to read in, and navigate through and query structured web data sets - and in many cases write against them - using a common syntax.

Q: Right, But So What?

A1: Open Government OData. From Open Government Data Principles

The Internet is the public space of the modern world, and through it governments now have the opportunity to better understand the needs of their citizens and citizens may participate more fully in their government. Information becomes more valuable as it is shared, less valuable as it is hoarded. Open data promotes increased civil discourse, improved public welfare, and a more efficient use of public resources.

That’s great, but it needs to be practicable…And number 5 of the 8 Principles of Open Government Data sensibly states that the data should be (via David Eaves):

5. Machine processable

Data is reasonably structured to allow automated processing.

It'll be down to each government agency (and local government) as to how they decide to implement this principle, but wouldn't it be great if they agreed to a standard (and a powerfully simple, web-oriented one at that)? This is what Jon Udell concluded here:

"The open data movement, in government and elsewhere, aims to help people engage with and participate in processes represented by the data. When you publish data in a fully articulated way, you build a framework for engagement, a trellis for participation. This is a huge opportunity, and it’s what most excites me about OData"

A2: To ODatarize your data is to RESTify your data.

As more data-oriented web APIs come online, each team responsible for the design of each web API is confronted by the same kinds of questions, and each team answers these in their own particular way. Increasingly, “RESTful” is a design goal of web APIs. Great…but what does that mean? How do you expose the data, the relationships between the entities inside the model, and what should the querying syntax look like? Unfortunately, there are as many answers to these questions as there are RESTful web APIs. And there needn’t be.

For to ODatarize your data is to RESTify your data. Do spend the time at the value layer - figure out the way your developers / consumers want to see the data and expose it that way. Do make it easy for devs / consumers to learn / navigate about the data and use it. Do not make them learn about the unique idiosyncrasies you’ve built into your API (or those that leak out of your originating store) :-)

From a developer’s standpoint, OData is ultimately about productivity.

For the OData “Production Developer”: Point at your data store – define your entity model and map it to the data model you already have (so your developers consume / program against the data that makes most sense to them – effectively ORM’ing) and expose as an OData service, inheriting: all the REST characteristics; entity relationship self-discovery; and querying goodness.
For the OData “Consuming Developer”: If you know the web API is OData…great! Pick up a client library, get to the API end-point (data.foo.org/blah.svc). Point and Shoot: Traverse the data model, query it (and bookmark as needed – it’s a URI)…play!

(see links at the bottom of this post to technical content that provides details on all this)

A3: Since the announcement, I’ve seen quite a bit of excitement around the web (especially Twitter) by developers who see the potential here…there is plenty of experimentation going on. At Intuit, my team is also experimenting with ODatarizing some of our data services, exploring how it might be applied across a number of our cloud based data services. And when our team’s architect Tweets that “Looks like #odata is going to be a good fit for our data services”, I know there’s something interesting going on here…

So I encourage you to find out more about OData and get involved.

More Resources

An easy way to set up an OData feed from your SQL Azure database - Stephen Forte
OData interoperability with .NET, Java, PHP, iPhone and more - MSFT Interop team
Microsoft, Bring OData to a W3C Incubator - W3C blog -
Silverlight 4 + RIA Services - Ready for Business: Exposing OData Services-4-ria-services-ready-for-business-exposing-odata-services.aspx - Brad Abrams (MSFT)
Open Data for the Open Web - Douglas Purdy (MSFT)
Producing and consuming OData feeds: An end-to-end example - Jon Udell (MSFT)
OData for collaborative sense-making - Jon Udell
The three laws of open government data – David Eaves
8 Open Government Data Principles
Open Data Protocol (OData) with Pablo Castro - Hanselminutes podcast interview with Pablo Castro (MSFT – Architect for OData)
OData, a RESTful contender for your API - Andy Thompson
Announcing Sesame Data Browser - Fabrice Marguerie
Retrieving Data with the OData SDK for PHP - Brian Swann
Got SQL Azure? Then you've got OData - OData blog
Implementing only certain aspects of OData-only-certain-aspects-of-odata.aspx - Pablo Castro
Every SharePoint 2010 server is a Data Services server - Pablo Castro
Microsoft Creates New OData.org Website for Open Data Protocol – CMS Wire
Microsoft delivers updates on OData, Houston, Dallas - ZDNet
Timeline of Project Astoria - Pablo Castro
How ADO.NET Data Services came to be (formerly known as Project Astoria) - Alex Barnett
previous Astoria / OData posts - Alex Barnett

How ADO.NET Data Services came to be (formerly known as Project Astoria)

alexbarnett — Thu, 21 Aug 2008 02:16:00 GMT

Pablo Castro has recounted some of his timelined memories about how "Project Astoria" evolved from a lunch time conversation to bits in .NET 3.5 SP1 and Visual Studio 2008 SP1 now known as ADO.NET Data Services Framework). Nice write up.

Three memories of my own to add to the story:

1. I was reading up on the whole REST thing in the summer of 2006 - its origins, philosophy and design patterns. I knew there was something interesting going on and some potential dots to join, but I wasn't sure which dots...So I collated and circulated a bunch of research / links to the team, then blogged the links (I liked How I explained REST to my wife. More recently see Explaining REST to Damien Katz). I got a few proverbial (and some literal) blank stares as I shared my enthusiasm for REST, asking how we could apply the ideas to the various projects we were working on. It was Pablo, and (as Pablo attests) Britt Johnston (now a PUM for SQL Business) who were able to develop the initial conceptual leaps into something more concrete like a Think Week Paper and a prototype demo.

2. When it came to brainstorming the code name, the team agreed on a "cloud" theme. A number of proposals were floated around along with their rationales, including "cumulus" and "cirrus". We were then advised that city and town code names were legal-safe. So there we were, struggling to agree on some city or town name we all liked (or at least not hate nor be confused by..."how about Nameless?"...), and then Mike Pizzo's proposal came in: "Astoria - hey, it's the cloudiest city in the USA!" (at least it was in 2006). Sold.

3. I think my favorite memory of all is the reaction Gary Flake provided (of Microsoft's Live Labs) to the prototype Pablo demo'd at one of the pitch meetings: "As God himself would have designed it!" Dr Flake exclaimed..."Cool", I thought to myself - "but does that mean no REST for the wicked?"

Astoria at MIX08 (REST in Vegas)

alexbarnett — Wed, 05 Mar 2008 02:11:00 GMT

As much as I'd love to make it to MIX08 this week, time will not allow me...But if I were, then I'd be going to the following three sessions related to the Project Formerly Known as Astoria:

Wed, March 5th - RESTful Data Services with the ADO.NET Data Services Framework
Fri, March 7th - Accessing Windows Live Services via AtomPub
Fri, March 7th - Building RESTful Real World Applications with the ADO.NET Data Services Framework

Social Clouds, XML 10 Years Old, and Honourable Mentions

alexbarnett — Tue, 12 Feb 2008 13:51:00 GMT

The Social Cloud

Kevin Marks is a software engineer at Google, was principal engineer for Technorati and one of the founders of Microformats. In this video Kevin talks about the big picture re: the phenomenon of online social networks in a presentation called The Social Cloud. Great backgrounder to the topic. More Lift videos here.

My Open ID?

Chris Brogen asked Question about OpenID:

"I’ve chosen to use the Wordpress.com installation of OpenID. I tied it to my Wordpress.com account and have so far used it in only two places. I’m thinking that every time I offer up an OpenID, I’ll point to that one. So far so good, right? ( To get up to speed on OpenID, go here).

What happens if Wordpress.com folds? What happens if they change their mind and start charging me, or I leave them for someone else, or whatever?"

Good question, to which you'll find multiple useful answers provided in the post's comments.

Semantic web enablement

The XML spec first public draft was November '96, the final release as a 1.0 was Feb 1998. XML was ten years old Feb 10, 2008. Tim Bray provides a history of the people involved and the events leading up to the birth of XML in his XML People.

Semantic news discovery

Silobreaker "provides relevance by looking at the data it finds like a person does. It recognises people, companies, topics, places and keywords; understands how they relate to each other in the news flow, and puts them in context for the user."

I need to play more to find out how useable / useful this service is, but I like the idea.

Honourable mention

RIA Weekly #06 - What’s Behind Code-Behind, JavaFX with Adobe tools, Microsoft/Yahoo!, and other acquisitions. The Redmonk podcast with Michael Coté and Ryan Stewart (Adobe). Topics include Kevin Lynch as new Adobe CTO, JavaFX vs. Silverlight vs. Air, code-behind annoyance, Google's Android, and the Oracle / BEA deal. I get an honourable mention on the show.

Random but good

Visual Music Instruments - via Kevin Kelly. No manual required, but it would probably help.

The Web Standards Fluster Cuck

alexbarnett — Tue, 14 Aug 2007 00:58:00 GMT

Clucking bell, Molly Holzshlag really has kicked the web standards beehive with a blog post expressing her great discontent with the W3C and WaSP.

Ridiculously Inadequate Backgrounder

Now, before you head off and read the post and the 60+ comments, here's a bit of background on why I find this post of interest (and rather depressing):

I've been following Molly's work for a while now. She first came on to my radar when after providing an update on the progress made between the Microsoft IE, VS and .NET teams and the Web Standards Project (WaSP). That was in 2005. Then in January 2007, I noted Molly's announcement that she had left WASP to join the IE team on a contract basis to work on standards and interoperability issues. I was pleased to see the IE team was making a real effort.

Entirely seperately, but not entirely, in October of 2006 Tim Berners-Lee called for the reinvention of HTML. His call to action caused a bit of a hoo-ha at the time. What's that got to do with Molly? Well, as noted, some of the reactions to TBL's post varied from skepticism, to 'About time!' - and here's the connection with Molly's latest post - to what role the WHATWG will play in what presumably could be a competing effort to the HTML 5 (or XHTML5) spec in progress at the time. However, I was pleased to hear TBL's public calling for progress and hoped we might see some of this progrss after HTML's 8-year stagnation. Then in July 2007, we had the news that HTML5 was being considered by the W3C.

Confused? You should be.

So after my ridiculously inadequate backgrounder, you can now go ahead and read Molly's post, along with the contributions be the cast of characters (the commenters), some of whom are affiliated with various competing factions wrestling with the future of web standards and HTML, who somehow manage to converge the various threads (now including a Fear of Air, the Semantic Web, microformats, Silverlight, XML, community, accessibility, transparency and who-knows-what-else) into what looks like a complete political mess (read: fluster cuck).

Yes, it is depressing,, but such is the business of web standards agreement. A messy business indeed...There's even a YouTube video covering the drama - HTML5 trailer - Find your Hero.

Thanks to Thomas Vander Wal for the link to Molly's post.

I'll be at MIX07

alexbarnett — Sat, 28 Apr 2007 13:34:00 GMT

I'll be heading down to MIX07 tomorrow. Let me know if you want to hook up (ping me at )

There will be a torrent of Microsoft product announcements over the next few days - Sam Sethi has made a number of predictions on what some of these will be.

One of the sessions I'll be looking forward to is Pablo Castro's talk that Sam picked up on (I used to work with Pablo in Microsoft's Data Programmability team). If you are going to MIX07, DO NOT miss this:

Accessing Data Services in the Cloud
Speaker(s): Pablo Castro - Microsoft

Come learn about new Microsoft technologies that enable you to make your data available over the Web through a simple REST interface and using open formats such as plan XML, JSON or even RDF. We also discuss the underlying entity framework that makes it easy to model, publish, and program against your data over the Web.

JSON - are you ready?

alexbarnett — Fri, 16 Mar 2007 16:39:00 GMT

Assaf's post cracked me up this morning - JSON Assessment Questionnaire: Are you ready to make the transition?.

Here's a sample:

The problem with JSON is … (item: true)

[ ] I never used it, so I wouldn’t know.
[ ] That’s why I never use it.
[ ] No angle brackets.
[ ] Too simple.
[ ] Not in the J2EE BluePrints Solutions Catalog.

MSXML4 will be kill bit-ed in IE

alexbarnett — Tue, 13 Mar 2007 14:19:00 GMT

The Microsoft XML team has announced MSXML4 will be kill bit-ed in IE in the October - December 2007 timeframe.

The team is also *strongly recommending* developers who use MSXML to program with MSXML6 and upgrade apps using older versions to MSXML6.

Good background reading on the topic is this October '06 post by Adam Wiener explaining the versioning history of MSXML and best practices.

Good news for XQuery and XSLT fans

alexbarnett — Wed, 24 Jan 2007 20:18:00 GMT

Good news for XQuery and XSLT fans - the W3C has announced the W3C Recommendations for XQuery 1.0 as an XML-aware syntax for querying collections of structured and semi-structured data both locally and over the Web and XSLT Version 2.0, a specification for transforming data model instances (XML and non-XML) into other documents. The third spec that's made Recommendation status is XPath 2.0, an expression syntax for referring to parts of XML documents.

Some of the big players behind getting these standards ratified (IBM, Oracle, Microsoft) are quoted on this testimonials page.

From the official press release:

"These new Web Standards will play a significant role in enterprise computing by connecting databases with the Web. XQuery allows data mining of everything from memos and Web service messages to multi-terabyte relational databases. XSLT 2.0 adds significant new functionality to the already widely deployed XSLT 1.0, which enables the transformation and styled presentation of XML documents. Both specifications rely on XPath 2.0, also significantly enriched from its previous version.
..."This is a red-letter day for XSLT users," said Michael Kay, editor of the XSLT 2.0 specification, "both for those who have been waiting patiently for this Recommendation to appear before they could use the new features, and for those who have taken a gamble by deploying the new technology before its final stamp of approval. Our biggest achievement, in my view, has been to deliver a huge step forward in functionality and developer productivity, while also retaining a very high level of backwards compatibility, thereby keeping transition costs to the minimum."

But not everyone agrees XQuery is slam dunk. Ex Microsoftee 'Derek' in this November post on hearing the news of the proposed recommendations, wrote at the time:

"I just noticed, XSLT 2.0, XML Query and XPath 2.0 Are Proposed Recommendations. Since leaving MS, I've stopped tracking these. It baffles me that XQuery is only just now 'shipping'. This + the-mess-that-is-XSD spell the doom of XML. This isn't just doom, like the Outlook spell-checker would try an correct the DOM to; this is real doom. These standards to too complicated and too late
...I honestly think we would be better without XQuery. Let the vendors think for themselves and see what customers actually use. XQuery is a standard looking for a use, which is backward and guaranteed to produce a problematic result.
XSLT/XPath 2.0 is a harder one... There are a couple things that XSLT 2.0 adds that were desperately needed vs XSLT 1.0. But I've managed a team implementing a commercial quality XSLT 1.0 implementation and that was a huge amount of work. XSLT 2.0 is at least 4x as much work. That is terrifying. Why not just 'fix' XSLT 1.0? It would be dramatically less work, and provide 80% of the gains, at 10% the cost."

Generally however, the reaction I've seen to the news seems fairly positive. Kimbro Staken never thought he'd see the day. Jonathan Robbie of DataDirect seems chuffed (mind you, his cmapny has been a member of XML Query WG). Alex Miller seems happy too (he's at BEA).

Bob Beauchemin, SQL Server MVP is speculating on the impact this news will have on the Microsoft SQL Server future product line:

"Although SQL Server 2005's XML data type doesn't exactly follow the XQuery 1.0/XPath 2.0 Data Model, rumor has it that the next version of the ISO/ANSI SQL spec (SQL2007?) may have some something to say about this, as well as something to say about XQuery in general. Right now, the SQL2003 spec doesn't specify a query language. [also a view shared by Matija Lah]
It will also be interesting to see what the SQL Server folks do with regards to updates to support the new specs in the next release, and support of a larger portion of the language constructs."

Given that Michael Rys (who also posted on this news) is a member of the SQL Server product team and has been closely involved in the relevant standards Working Groups, I think Bob's speculation may be justified... :-)

Best practices with MSXML on the browser

alexbarnett — Wed, 25 Oct 2006 21:20:00 GMT

Adam Wiener, Lead Program Manager for Data Programmability / XML Technologies, has worked closely with the IE team in the lead up to the IE7 release. As part of that process, Adam looked at the use of XML in the browser and concluded:

"During this investigation one thing has become immediately obvious – there is a lot of confusion around the versioning story for MSXML and how to instantiate the “right” MSXML object in the browser.

In this post, Adam has provided details on best practices for use of MSXML in the browser and written up a short history of the different versions of MSXML, where they ship, and its long term strategy.

Agile Development at Microsoft

alexbarnett — Tue, 17 Oct 2006 03:40:00 GMT

Chris Lovett, who works in our XML Team (within the Data Programmability team) has been interviewed discussing the process used for the development of XML Notepad (an unsupported tool released by the team last month - you can download it here).

The team used an "Agile Development Process" - what's that?

"it's a highly iterative process. We prototype the idea, show it around, get feedback, lots of hallway discussions, great feedback gets incorporated into the next iteration and so on, until we reach the point that we think we have the right set of features to call it done. So it's a light weight process with a small team that can move fast and respond quickly to feedback."

More details at the InfoQ article (a new site by the way, worth subscribing to if you're into dev stuff).

The Future of XML

admin — Tue, 26 Sep 2006 15:16:00 GMT

Mr Scoble got talking with Sun Microsystem's Tim Bray, a co-creator of the XML spec. I've not listened yet, but this interview sounds too good to miss:

"Even Tim Bray is blown away at the popularity of XML. He was one of the co-creators of the XML spec back in the mid 1990s and in this audio interview he and I talk about the state of XML and get his views on what the future holds for XML and other derivative formats like RSS and Atom."