Alex Barnett blog : ADO.NET

Why OData Matters (IMHO)

alexbarnett — Fri, 19 Mar 2010 15:24:00 GMT

Earlier this week I was in the MIX10 crowd as Douglas Purdy announced the Open Data Protocol (it was a great presentation - summarized here).

I want to share with you why I think OData could be a very big deal…But before we go there...let's start with the basics...

What is OData? Where Did OData Come From?

To understand the history of how OData came to be, you need to understand how project "Astoria" came to be...I won't go over that again as this is already pretty well documented. ~~Astoria~~ OData has come a long way since.

In a nutshell: Today, OData builds on a few conventions, popularized by AtomPub (see OData AtomPub Format), to using REST-based data services. These services allow resources, identified using Uniform Resource Identifiers (URIs) and defined in an abstract data model (see OData URI Conventions, to be read and edited by web clients using simple HTTP messages (see OData Operations).

An Open Data Protocol for the Web

OData offers a standardized way for programmable data to be made available across the web and in turn allowing "consumers" of that data to rely on a set of conventions to be followed that in turn allows many interesting things to happen if widely adopted...

...And to this end:

As announced, OData has been released by Microsoft under the Open Specification Promise (OSP) "to allow anyone to freely interoperate with OData implementations" . Since then, the W3C has invited the OData team to Bring OData to a W3C Incubator (I haven't seen a public response yet but I urge the team to do so.).
OData is not a Microsoft-only thing and it won’t succeed if it is. The originating philosophy was about transparency in the design process, with an Open end-point as the goal - not a .NET lock-in play (“agree on standards and compete on implementation”). This approach has already yielded an initial set of clients, servers, services, and tools. Today, a number of OData SDKs and libraries are available for .NET, Java, PHP, iPhone (Objective-C) and more – and there’ll be more coming.
There is a small but growing ecosystem of non-Microsoft "producers" and "consumers" - (where OData "producers" include Netflix’s catalog as OData and the VanGuide (a social map of Vancouver Open Data) and Public data from the city of Edmonton as OData) and OData “consumers” - standard web browsers, RIA "data explorers" – such as Sesame OData Browser, and the client libraries mentioned above – p.s. somebody build a javascript-only data browser please!)

Where can OData take us?

The clue is in the OData icon (next to the RSS feed icon. Can you see the similarities?

The big idea here is that in the same way we have the "RSS" feed icon, we'll get used to seeing the "OData" icon on commercial and non-commercial websites everywhere (especially for government-related data). So in the same way you know today that the RSS icon means "get an XML feed for this content", the "OData" icon means "get this web data" - you'll know (and your client will know) what to expect in terms of how to read in, and navigate through and query structured web data sets - and in many cases write against them - using a common syntax.

Q: Right, But So What?

A1: Open Government OData. From Open Government Data Principles

The Internet is the public space of the modern world, and through it governments now have the opportunity to better understand the needs of their citizens and citizens may participate more fully in their government. Information becomes more valuable as it is shared, less valuable as it is hoarded. Open data promotes increased civil discourse, improved public welfare, and a more efficient use of public resources.

That’s great, but it needs to be practicable…And number 5 of the 8 Principles of Open Government Data sensibly states that the data should be (via David Eaves):

5. Machine processable

Data is reasonably structured to allow automated processing.

It'll be down to each government agency (and local government) as to how they decide to implement this principle, but wouldn't it be great if they agreed to a standard (and a powerfully simple, web-oriented one at that)? This is what Jon Udell concluded here:

"The open data movement, in government and elsewhere, aims to help people engage with and participate in processes represented by the data. When you publish data in a fully articulated way, you build a framework for engagement, a trellis for participation. This is a huge opportunity, and it’s what most excites me about OData"

A2: To ODatarize your data is to RESTify your data.

As more data-oriented web APIs come online, each team responsible for the design of each web API is confronted by the same kinds of questions, and each team answers these in their own particular way. Increasingly, “RESTful” is a design goal of web APIs. Great…but what does that mean? How do you expose the data, the relationships between the entities inside the model, and what should the querying syntax look like? Unfortunately, there are as many answers to these questions as there are RESTful web APIs. And there needn’t be.

For to ODatarize your data is to RESTify your data. Do spend the time at the value layer - figure out the way your developers / consumers want to see the data and expose it that way. Do make it easy for devs / consumers to learn / navigate about the data and use it. Do not make them learn about the unique idiosyncrasies you’ve built into your API (or those that leak out of your originating store) :-)

From a developer’s standpoint, OData is ultimately about productivity.

For the OData “Production Developer”: Point at your data store – define your entity model and map it to the data model you already have (so your developers consume / program against the data that makes most sense to them – effectively ORM’ing) and expose as an OData service, inheriting: all the REST characteristics; entity relationship self-discovery; and querying goodness.
For the OData “Consuming Developer”: If you know the web API is OData…great! Pick up a client library, get to the API end-point (data.foo.org/blah.svc). Point and Shoot: Traverse the data model, query it (and bookmark as needed – it’s a URI)…play!

(see links at the bottom of this post to technical content that provides details on all this)

A3: Since the announcement, I’ve seen quite a bit of excitement around the web (especially Twitter) by developers who see the potential here…there is plenty of experimentation going on. At Intuit, my team is also experimenting with ODatarizing some of our data services, exploring how it might be applied across a number of our cloud based data services. And when our team’s architect Tweets that “Looks like #odata is going to be a good fit for our data services”, I know there’s something interesting going on here…

So I encourage you to find out more about OData and get involved.

More Resources

An easy way to set up an OData feed from your SQL Azure database - Stephen Forte
OData interoperability with .NET, Java, PHP, iPhone and more - MSFT Interop team
Microsoft, Bring OData to a W3C Incubator - W3C blog -
Silverlight 4 + RIA Services - Ready for Business: Exposing OData Services-4-ria-services-ready-for-business-exposing-odata-services.aspx - Brad Abrams (MSFT)
Open Data for the Open Web - Douglas Purdy (MSFT)
Producing and consuming OData feeds: An end-to-end example - Jon Udell (MSFT)
OData for collaborative sense-making - Jon Udell
The three laws of open government data – David Eaves
8 Open Government Data Principles
Open Data Protocol (OData) with Pablo Castro - Hanselminutes podcast interview with Pablo Castro (MSFT – Architect for OData)
OData, a RESTful contender for your API - Andy Thompson
Announcing Sesame Data Browser - Fabrice Marguerie
Retrieving Data with the OData SDK for PHP - Brian Swann
Got SQL Azure? Then you've got OData - OData blog
Implementing only certain aspects of OData-only-certain-aspects-of-odata.aspx - Pablo Castro
Every SharePoint 2010 server is a Data Services server - Pablo Castro
Microsoft Creates New OData.org Website for Open Data Protocol – CMS Wire
Microsoft delivers updates on OData, Houston, Dallas - ZDNet
Timeline of Project Astoria - Pablo Castro
How ADO.NET Data Services came to be (formerly known as Project Astoria) - Alex Barnett
previous Astoria / OData posts - Alex Barnett

How ADO.NET Data Services came to be (formerly known as Project Astoria)

alexbarnett — Thu, 21 Aug 2008 02:16:00 GMT

Pablo Castro has recounted some of his timelined memories about how "Project Astoria" evolved from a lunch time conversation to bits in .NET 3.5 SP1 and Visual Studio 2008 SP1 now known as ADO.NET Data Services Framework). Nice write up.

Three memories of my own to add to the story:

1. I was reading up on the whole REST thing in the summer of 2006 - its origins, philosophy and design patterns. I knew there was something interesting going on and some potential dots to join, but I wasn't sure which dots...So I collated and circulated a bunch of research / links to the team, then blogged the links (I liked How I explained REST to my wife. More recently see Explaining REST to Damien Katz). I got a few proverbial (and some literal) blank stares as I shared my enthusiasm for REST, asking how we could apply the ideas to the various projects we were working on. It was Pablo, and (as Pablo attests) Britt Johnston (now a PUM for SQL Business) who were able to develop the initial conceptual leaps into something more concrete like a Think Week Paper and a prototype demo.

2. When it came to brainstorming the code name, the team agreed on a "cloud" theme. A number of proposals were floated around along with their rationales, including "cumulus" and "cirrus". We were then advised that city and town code names were legal-safe. So there we were, struggling to agree on some city or town name we all liked (or at least not hate nor be confused by..."how about Nameless?"...), and then Mike Pizzo's proposal came in: "Astoria - hey, it's the cloudiest city in the USA!" (at least it was in 2006). Sold.

3. I think my favorite memory of all is the reaction Gary Flake provided (of Microsoft's Live Labs) to the prototype Pablo demo'd at one of the pitch meetings: "As God himself would have designed it!" Dr Flake exclaimed..."Cool", I thought to myself - "but does that mean no REST for the wicked?"

Astoria at MIX08 (REST in Vegas)

alexbarnett — Wed, 05 Mar 2008 02:11:00 GMT

As much as I'd love to make it to MIX08 this week, time will not allow me...But if I were, then I'd be going to the following three sessions related to the Project Formerly Known as Astoria:

Wed, March 5th - RESTful Data Services with the ADO.NET Data Services Framework
Fri, March 7th - Accessing Windows Live Services via AtomPub
Fri, March 7th - Building RESTful Real World Applications with the ADO.NET Data Services Framework

ADO.NET Entity Framework video- 1 year later

alexbarnett — Thu, 30 Aug 2007 14:04:00 GMT

Some old Data Programmability teammates of mine, Britt Johnston, Pablo Castro and Mike Pizzo (well, Mike's not that old), are interviewed by Charles Torre in this Channel 9 video to discuss their progress, challenges and customer feedback in the last year since announcing the first ADO.NET Entity Framework CTP, including some details on project Astoria and LINQ to Entities support, details on the Beta 2 and Tools CTP released earlier this week (the entity model designer is v.cool), plus the future of the Entity Framework.

P.S. Michael - in case you're reading: reconsider the shorts for the next interview :-P

Dr. Peter Chen - the ER Model and ADO.NET Entity Framework - Podcast

alexbarnett — Tue, 20 Feb 2007 21:49:00 GMT

A couple of weeks ago Dr Peter Chen, inventor of the Entity-Relationship model (ER model), visited the Microsoft Campus to provide a lecture sharing his thoughts on the ADO.NET Entity Framework. As we were planning his visit, he was very keen to make sure that his views could be shared with a wider audience beyond Microsoft employees, so we agree to record some sessions while on campus.

His original paper on the Entity-Relationship model (ER model), published in 1976 is one of the most cited papers in the computer software field. I was fortunate enough to have dinner with him, Sam Druker, Jose Blakeley, Britt Johnston, Erik Meijer, Pedro Cellis where we heard how his ideas on the ER model were formulated some 30 years ago. Lot's more fascinating conversation all about this history of databases, and their future too...

The next morning, I picked up Dr Chen from the hotel and drove him to the recording studio to meet with Britt, Jose and Brian Beckman where they got talking for an hour with the microphone on. Here's the podcast, hosted on Channel 9 (.mp3 here) It is superb. I mean it. If you're into 'data', you'll love it.

"Beckman interviews Dr. Chen along with Jose Blakeley, Software Architect, SQL Server, and Britt Johnston, Director of Program Management, Data Programmability. Join as we discuss the ideas behind Dr. Chen's original paper, how these concepts have subsequently influenced the software industry and database technologies, and how the ADO.NET Entity Framework is an execution runtime for the ER model."

A Short History of the Evolution of Microsoft Data Access APIs

alexbarnett — Wed, 06 Dec 2006 02:28:00 GMT

Earlier this year I read Joel Spolsky's 'Joel on Software'. It's a collection of his blog posts bundled up into a real-world book on topics ranging from tips on code development, to advice on managing programmers, to strategies on running a successful small software business. As an ex-Microsoft employee (he was a program manager on the Excel product team), Spolsky's criticisms of Microsoft practices and strategy can be insightful, if not entertaining...

Now, I started reading the book soon after I joined the Data Programmability team to focus on our community efforts, so you could see why the following paragraph from his Fire and Motion blog post / essay made me sit up and pay attention:

"Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET - All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That's probably it, actually.)"

Hey!! He was talking about my team here and the technologies my team is responsible for. And not only that, he was doing so in a well read book (amongst software developers at least), accusing us (at least with some tongue in cheek) of incompetence!!

I asked around our team: why is the history of data access technologies at Microsoft the way it is? How did we get to where we are today? As a newbie to team I wanted to know the history and to somehow get our own historical perspective out there.

So I asked Mike Pizzo, a Microsoft veteran who has been a member of the team responsible for the 'alphabet soup' of data access technologies at Microsoft for the last 15 years (yes, fifteen) if he would blog something on this topic. To my delight, he agreed.

Today, Mike published Part I – The Early Years, the first blog post in a great series of four documenting the history of data access at Microsoft. I think it's a classic. Enjoy.

"Following is a short history of the evolution of Microsoft Data Access APIs in four parts (so far…)
Part I – The Early Years (ODBC, DAO, RDO, ODBCDirect)
Part II – Componentizing Data Access (OLE DB)
Part III – Disconnected Programming in a Managed Environment (ADO.NET)
Part IV – Programming at the Conceptual Model (ADO.NET Entities and LINQ)
My goal is to describe the industry trends, platform shifts, and usage scenarios that shaped our API landscape over the past 15 years (and, in the retelling, perhaps receive some absolution for my sins…)"

Live Search OPML Generator

alexbarnett — Thu, 12 Oct 2006 05:06:00 GMT

Andy Edmonds has posted about the all new Live Search OPML Generator...

"What does it do?: This app runs your query against the Live Search API with the feed: syntax, returning RSS feeds, and assembles an OPML file. You'll get the chance to choose from the results before generating the subscription list."

As an example, here's the search results on the query "ADO.NET". Some great feeds come back: The ADO.NET team blog feed, David Hayden's blog posts as a feed with category 'ADO.NET', Sahil Malik's feed that includes a number of ADO.NET posts, the ADO.NET Technology Preview MSDN Forum feed and plenty more.

What's great is that once you've decided which feeds you want to subscribe to, you select those and an OPML file is generated for you to wack into your feedreader. Alternatively, you could select all, then add the OPML file to your reader and then delete those that you don't want to keep. Or just add them as feeds to read within the Live.com reader.

Another nice thing Andy has pointed out is the potential to create Reading Lists from a search query.

Astrophysicist-turned-Programming-Language-Designer

admin — Sun, 03 Sep 2006 09:27:00 GMT

In my first week with the Data Programmability team, my new manager suggested I set up a bunch of introductory meetings with some of the team members. One of the very first of these meetings was with Software Architect Brian Beckman. I spent an hour with him and was totally blown away.

This is Brian, holding a copy of the 'Towards 2020" report published by Microsoft Research that he contributed to this year:

As I was sitting and listening to Brian's stream of consciousness I felt like I was watching a live 3D version of a Channel 9 video. At the end of our chat I asked him if he wouldn't mind me asking Charles Torre of Channel 9 to interview him. I really wanted others to have the chance to hear him talk. He happily agreed and the end result is now up for downloading on Channel 9. It's a large download but worth it - you'll be thoroughly entertained.

Brian is a cosmologist / astrophysicist / quantum physicist / academic-turned-programming language designer (!). His bookshelf is one of the more unusual you'll see around campus in terms of the mix (and is the only person I know other than me to own a Steve Wolfram book (I have A New Kind of Science - an extraordinary work).

He spent his early career at NASA's Jet Propulsion Lab (JPL) researching nutty stuff such as the Time Warp Operating System and virtual time theory, distributed computing - running the Game of Life and virtual billiards on parallel processors, and developing War Game simulators for government (a little bit like the Game of Life, but more of a Game of Death involving large-scale nuclear warhead deployments and massive Soviet troop and tank maneuver scenarios: think of it as Sims Massively-Accelerated-Anti-Evolution Edition)...

Brian joined the Microsoft Research team years ago (before it was known as MSR) and today leads the our research and product incubation efforts in the Data Programmability team. He loves his functional programming languages: Haskell (Erik Meijer is one of Haskell's designers and is also in Brian's team) and Scheme. And he loves his Monads: Erik and Brian jointly wrote 'The Return of the Monoids' (not a science-fiction horror script, but research paper). As an example of what Brian does today, he was involved in the development of the Entity Data Model (EDM) into ADO.NET. This technology recently saw the light of day in ADO.NET Entity Framework CTP), with Brian taking on the role of translating the theoretical and mathematical models of EDM developed by the research team into coded concepts the product teams could interpret and then develop the technology for ADO.NET, SQL Server, and Visual Studio. In our first meeting we spent some time discussing the dynamic languages Ruby, Python and Perl - the great news for me was that he was very familiar with these and is actually quite a big fan of them although his day-to-day attention is focused on VB.NET as you can see from the video.

Mr Beckman collects calculators. He showed me one I've never see before - the CURTA Calculator. (The story behind its invention is as amazing as the device itself.)

He also show me this:

...which he has programmed to print out these:

His desktop scientific print calculator simply crunches out prime numbers, all day long, every day and has done for years.

Each new prime takes a little longer to calculate than the previous prime. Currently it takes about 6 hours to work out the next prime number. Earlier this year our team ran an internal competition - ADO.NET Idol: it's simple: the coolest app developed by a member the DP / SQL team using the ADO.NET vNext / EDM stack wins. Brian won it by developing an full-on emulator of the print calculator, GUI and all, programming each of the calculator's functions into a virtual ADO.NET vNext calculator to calculate and print out the prime numbers (at a considerably faster rate than the real calculator of course...). For fun.

Anyway, along with Brian and Erik and loads of others on the team, I've always plenty to learn each day around here...