Notes2Self.net

Stephen McGibbon's Web Journal
Voices for Innovation is a global community working together to shape the technology policy debate. We want every voice heard. Join now and make a difference.

Recent comments

Reading

Cutting corners - the realpolitik of ODF standardisation?

I made a comment at an event in Brussels and quite a few people have asked for more information so I thought I'd post here.

I mentioned that in my opinion, Sun were completely aware that ODF wasn't sufficiently defined to support spreadsheet interoperability as long ago as February 2005 and that the realpolitik inside OASIS was to take advantage of the EU IDA's request to standardise by rushing to be first despite knowing the ODF specification was deficient in at least this area.

So, here's how I reached this opinion. As I said, I'm delighted to be corrected if this proves to be factually inaccurate. This isn't intended to be FUD, I am only quoting OASIS' own mailing list.

Back in early February 2005 James Clark made a comment to the OASIS OpenDocument technical Committee about the lack of interoperability for spreadsheet documents (my emphasis)

I really hope I'm missing something, because, frankly, I'm speechless.  You cannot be serious. You have virtually zero interoperability for spreadsheet documents. OpenDocument has the potential to be extraodinarily valuable and important standard. I urge you not to throw away a huge part of that potential by leaving such a gaping hole in your specification.

Claus Agerskov further commented that this provided a means of creating lock-in (my emphasis)

"OpenDocument doesn't specify the formulars used in spreadsheets so every spreadsheet vendor can implement formulars in their own way without being an open standard. This way a vendor can create lock-in to their spreadsheets"

SUN's Chair of the Technical Committee, Michael Bauer, responded that

"There are from our point of view also no interoperability issues, because the namespace prefix mechanism we have specified unambiguously specifies what syntax and semantics are used for a formula".

which David Wheeler interpreted as saying 

"Every implementation must reverse engineer all other implementations' namespaces (they're not in the spec, so everyone's free to invent their own private incompatible namespaces). Then, every implementation must implement all the syntax and semantics of all other implementations' namespaces for formulas, if they wish to achive interoperability. And oh, by the way, your implementation might not implement the namespace for the document you're trying to load, so you may lose all the formulas."

David made some proposals which I believe have now led to the OASIS Open Document -Formula subcommittee, but at the time the OASIS Technical Committee answered David saying amongst other things that: (my emphasis)

"... please keep in mind that we have to fit proposed solutions into the politic of work that has already been done.  A politic that represents years of work that is just now on it's way to ratification at OASIS, and beyond to ISO.  Keep in mind also that the ISO certification comes at the request of the European Union. Time is of the essence. Ratification perhaps trumps perfection. At least for the moment. We are very much aware that whatever we leave outside the specification remains open (or not) and exposed to ambiguities and custom implementations, all of which have proved to be so problematic in the past."

Remember this was back in February 2005. In September 2005 Newsforge carried an artlicle euphamistically titled "OpenDocument office suites lack formula compatibility" but which did a good job of demonstrating that OpenDocument represented a step backwards from the status-quo and included some screen-shots to illustrate the problem, concluding

Sooner or later, a solution to the formula incompatibility problem will be found. Ideally, someone will solve it sooner, and without any intellectual property problems. The capability to exchange spreadsheets freely is essential for free desktops in the business and education markets, and it should not be limited by artificial restrictions.

Now when Tim Bray (also with Sun) heard of this in October 2005 he wrote

Bad Formula Trouble · I learned, to my dismay, that the ODF specification is silent on spreadsheet formulas, they’re just strings. This is obviously a problem; much discussion on what to do ensued. I lean to the idea, much bally-hooed by Novell, of simply figuring out what Excel does, writing that down, and building it into ODF v.Next. Mind you, anyone who’s really been to the mat with Excel, in terms of Math & Macros, knows that it isn’t a pretty picture, there are real coherency problems. But it’s good enough and the world has learned how to make it work.

I don't know if its correct, but the roadmap on OASIS' web site seems to suggest that this might be fixed in ODF 1.2, but it seems that's not due before October 2007 at the earliest.

Remember all of this was known back in very early 2005, pre-ISO certification. Since ODF became ISO26300 OASIS has been working hard to promote its adoption. A prime tactic has been to create the impression with public policy makers that the longevity of public records is in danger unless they mandate ODF.

Somehow they neglect to mention that the spreadsheets can't interoperate though.

Comments

Brian Jones: Open XML Formats said:

Stephen McGibbon has an interesting blog post based on his observations of the politics behind standardization....
# July 15, 2006 2:48 AM

Wesley Parish said:

If I might make a suggestion?  There are three Free and Open Source spreadsheets that are reasonably widely known and/or used - OpenOffice.org's Calc, Gnumeric, and KOffice's KSpread.  They are all using, or intending to use, ODF as one of their default file formats.

Get the various options tried and tested (and failed, whenever necessary) and then make _that_ the formula format to be standardized - it will by then have had three independent test environments to mess up if there is any problem.  And it should be completed well before  October 2007.

As far as taking MS Excel's formula method?  Well, Microsoft's MS Open XML seems to be a form of documentation of it.  And you'd have these three test environments to work it out in.
# July 17, 2006 2:54 AM

Stephen McGibbon said:

Thanks Wesley, the point of a standard is that I shouldn't have to do this right? Whatever they do will be proprietary and non-standard until it is included as part of ODF in the future, so ODF 1.0 doesn't seem to be sufficient for spreadsheet interoperability today.
My point is that Sun seems (from the mails above) to have been well aware of this, but chose to standardise a deficient specification in order to be the first past the post so to speak. Politics trumped engineering in OASIS, at least as far as ODF was concerned.
# July 17, 2006 8:13 AM

Sinleeh said:

"My point is that Sun seems (from the mails above) to have been well aware of this, but chose to standardise a deficient specification in order to be the first past the post so to speak. Politics trumped engineering in OASIS, at least as far as ODF was concerned."

Dear Mr McGibbon,

Given Microsoft's conspicuous absence in the ODF meetings, how did you arrive at the "first-past-the-post" and "Politics beats engineering" conclusion? Those are pretty strong words and need qualifying.
# July 25, 2006 7:40 PM

Stephen McGibbon said:

Hi Sinleeh, don't you think the text supports the conclusion?
# July 25, 2006 7:53 PM

brent's blog. said:

Today marks a very important day on [nteroperability] for Microsoft. Following a series of events that

# December 8, 2006 6:41 AM

Notes2Self.net said:

according to Sun's Daniel Rentz who posts an update about Sun's " Ongoing Work on Office

# March 4, 2007 9:24 PM

Notes2Self.net said:

I was just looking at Karel De Vriendt's ODEF (Open Document Exchange Formats) Workshop Conclusions

# March 21, 2007 9:26 PM

David Overton's Blog said:

I have been using and referencing standards for a long time. They are great. C# is an ECMA standard for

# June 29, 2007 3:20 PM

Notes2Self.net said:

Jason Matusow and I were sat next to each other at a meeting at the end of last week. I was laughing

# July 24, 2007 6:42 PM

Wouter van Vugt said:

There is a lot of discussion about how Open XML and OpenDocument Format should move into the future,

# July 28, 2007 9:57 PM

Emmanuel Birech said:

ODF 1.0 is completely insufficient for spreadsheet interoperability today and SUN were aware and rushed

# August 16, 2007 2:28 PM

Notes2Self.net said:

Earlier today the Burton Group released a free (registration required) report authored by Guy Creese

# January 12, 2008 1:07 AM

Notes2Self.net said:

Sun announced yesterday that they have sponsored Patrick Durusau 's contract to act as editor of the

# February 14, 2008 6:04 PM