Behind the scenes of the Open Platform’s evolution

When I came to the Guardian two years ago, I brought with me some crazy California talk about open strategies and APIs and platforms. Little did I know the Guardian already understood openness. It’s part of its DNA. It just needed new ways of working in an open world.

Last week, The Guardian’s approach to openness and mutualisation took a giant step forward when we brought the Open Platform out of Beta.

It’s a whole new business model with a new technology infrastructure that is already accelerating our ambitions.

I’ll explain how we got to this point, but let me clarify what we just announced:

  • We’ve implemented a tiered access model that I think is a first in this space. We have a simple way to work with anyone who wants to work with us, from hobbyist to large-scale service provider and everything in between.
  • We’ve created a new type of ad network with 24/7 Real Media’s Open AdStream, one where the ads travel with the content that we make available for partners to reuse.
  • That ad network is going to benefit from another first which is Omniture analytics code that travels with the content, as well.
  • License terms that encourage people to add value are rare. Using many of the open license principles we developed T&Cs that will fuel new business, not stop it.
  • Hosted in the cloud on Amazon EC2 the service scales massively. There are no limits to the number of customers we can serve.
  • The API uses the open source search platform Solr which makes it incredibly fast, robust, and easy for us to iterate quickly.
  • We introduced a new service for building apps on our network called MicroApps. Partners can create pages and fully functional applications on guardian.co.uk.

We’re using all the tools in the Open Platform for many of our own products, including the Guardian iPad app, several digital products and more and more news apps that require quick turn-around times and high performance levels.

Open Platform: Build applications with the GuardianThere’s lots of documentation on the Open Platform web site explaining all this and more, but I figured I could use this space to give a picture of what’s been happening behind the scenes to get to this point.

It’s worth noting that this is far from the full picture of all the amazing stuff that has been happening at the Guardian the past 12 months. These are the things that I’ve had the pleasure of being close to.

Beginning with Beta

First, we launched in Beta last year. We wanted to build some excitement around it via the people who would use it first. So, we unveiled it at a launch event in our building to some of the smartest and most influential London developers and tech press.

We were resolute in our strategy, but when you release something with unknown outcomes and a clear path to chaos people get uneasy. So, we created just large enough hurdles to keep it from exploding but a wide enough berth for those who used it to take it to its extreme and to demonstrate its value.

It worked. Developers got it right away and praised us for it. They immediately started building things using it (see the app gallery). All good signs.

Socializing the message

We ran a Guardian Hack Day and started hosting and sponsoring developer events, including BarCamp, Rewired State, FOWA, dConstruct, djugl, Music Hack Day, ScaleCamp, etc.

Next, we knew the message had to reach their bosses soon, and their bosses’ bosses. So, we aimed right for the top.

Industry events can be useful ways to build relationships, but Internet events have been really lacking meaning. People who care about how the Internet is changing the world and who are also actively making that change happen were the types of people we needed to build a long term dialog with.

So, we came up with a new kind of event: Activate Summit.

The quality of the speakers and attendees at Activate was incredible. Because of those people the event has now turned into something much more amazing than what we initially conceived.

Nick Bostrom’s darkly humorous analysis of the likelihood of human extinction as a result of technology haunts me frequently still, but the event also celebrates some brilliant ways technology is making life better. I think we successfully tapped into some kind of shared consciousness about why people invest their careers into the Internet movement…it’s about making a difference.

Developers, developers, developers!

Gordon Brown was wise in his decision to put Tim Berners-Lee and Nigel Shadbolt on the task of opening government data. But they knew enough to know that they didn’t know how to engage developers. Where did they turn for help? The Guardian!

We couldn’t have been more excited to help them get data.gov.uk out the door successfully. It was core to what we’re about. As Free Our Data champion Charles Arthur joked on the way to the launch presentation, “nice of them to throw a party for me.”

We gave them a platform to launch data.gov.uk in the form of developer outreach, advice, support, event logistics, a nice building, etc., but, again, the people involved made the whole effort much more impressive than any contribution we made to it.

Tom Taylor’s Postcode Paper, for example, was just brilliant on so many levels. The message for why open government data could not have been clearer.

Election data

Then when the UK election started to pick up some momentum, we opened up the Guardian’s deep politics database and gave it a free-to-use API. We knew we couldn’t possibly cover every angle of the election and hoped that others could use the Guardian’s resources to engage voters. We couldn’t have asked for a better example of that then Voter Power.

A range of revenue models

All along there were some interesting things happening more behind the scenes, too.

The commercial team was experimenting with some new types of deals. Our ad network business grew substantially, and we added a Food Ad Network and a Diversity Network to our already successful Green Ad network.

It was clear that there was also room for a new type of ad network, a broader content-targeted ad network. And better yet, if we could learn about what happens with content out across the web then we might have the beginnings of a very intelligent targeting engine, too.

24/7 Real Media’s Open Ad Stream and Omniture were ready to help us make this happen. So, we embedded ads and analytics code with article content in the Content API. We’ve launched with some house ads to test it out, but we’re very excited by the possibilities when the network grows.

The Guardian’s commercial teams, including Matt Gilbert, Steve Wing, Dan Hedley and Torsten de Reise, also worked out a range of different partnerships with several Beta customers including syndication, rev share on paid apps, and rev share on advertising. We’re scaling those models and working out some new ones, as well.

It became obvious to everyone that we were on to something with a ton of potential.


Rewriting the API for scale

Similarly, the technology team was busily rebuilding the Content API the moment we realized how big it needed to be.

In addition to supporting commercial partners, we wanted to use it for our own development. The new API had to scale massively, it had to be fast, it had to be reliable, it had to be easy to use, and it had to be cheap. We used the open source search platform Solr hosted on Amazon’s EC2. API service management was handled by Mashery.

The project has hit the desk of nearly every member of the development team at one point or another. Here are some of the key contributions. Mat Wall architected it. Graham Tackley made Mat’s ideas actually work. Graham and Stephen Wells led the development, while Francis Rhys-Jones and Daithi O’Crualaoich wrote most of the functions and features for it. Martyn Inglis and Grant Klopper handled the ad integration. The wonderful API Explorer was written by Francis, Thibault Sacreste and Ken Lim. Matthew O’Brien wrote the Politics API. The MicroApps framework included all these people plus basically the entire team.

Stephen Dunn and Graham Tackley provided more detail in a presentation to the open source community in Prague at Lucid Imagination’s Solr/Lucene EuroCon event.

The application platform we call MicroApps

Perhaps even more groundbreaking than all this is the MicroApp framework. A newspaper web site that can run 3rd party apps? Yes!

MicroApps makes the relationship between the Guardian and the Internet feel like a two-way, read-write, permeable membrane rather than a broadcast tower. It’s a very tangible technology answer to the openness vision.

You can learn more by reading 2 excellent blog posts about MicroApps. Dan Catt explains how he used MicroApps for Zeitgeist. Since most of the MicroApps that exist today are hosted on Google AppEngine, the Google Code team published Chris Thorpe’s insights about what we’re doing with MicroApps on their blog.

The MicroApps idea was born out of a requirement to release smaller chunks of more independent functionality without affecting the core platform….hence the name “MicroApps”. Like many technology breakthroughs, the thing it was intended to do becomes only a small part of the new world it opens up.

Bringing it all together

At the same time our lead software architect Mat Wall was formulating the MicroApp framework, the strategy for openness was forming our positioning and our approach to platforms:

…to weave the Guardian into the fabric of the Internet; to become ‘of‘ the Web, not just ‘on‘ the Web

The Content API is a great way to Open Out and make the Guardian meaningful in multiple environments. But we also knew that we had to find a way to Open In, or to allow relevant and interesting things going on across the Internet to be integrated sensibly within guardian.co.uk.

Similarly, the commercial team was looking to experiment with several media partners who are all thinking about engagement in new ways. What better way to engage 36M users than to offer fully functional apps directly on our domain?

The strategy, technology and business joined up perfectly. A tiered business model was born.

The model

Simon Willison was championing a lightweight keyless access level from the day we launched the Beta API. We tested keyless access with the Politics API, and we liked it a lot. So, that became the first access tier: Keyless.

We offered full content with embedded ads and analytics code in the next access level. We knew getting API keys was a pain. So, we approved keys automatically on signup. That defined the second tier: Approved.

Lastly, we combined unfettered access to all the content in our platform with the MicroApp framework for building apps on the Guardian network. We made this deep integration level available exclusively for people who will find ways to make money with us. That’s the 3rd tier: Bespoke. It’s essentially the same as working in the building with our dev team.

We weren’t precisely clear on how we’d join these things up when we conceived the model. Not surprisingly, as we’ve seen over and over with this whole effort, our partners are the ones who are turning the ideas into reality. Mashery was already working on API access levels, and suddenly the last of our problems went away.

The tiers gave some tangible structure to our partner strategy. The model felt like it just started to create itself.

Now we have lots of big triangle diagrams (see below) and grids and magic quadrants and things that we can put into presentation slides that help us understand and communicate how the ecosystem works.

Officially opening for business

Given the important commercial positioning now, we decided that the launch event had to focus first and foremost on our media partners. We invited media agencies and clients into our offices. Adam Freeman and Mike Bracken opened the presentation. Matt Gilbert then delivered the announcement and gave David Fisher a chance to walk through a deep dive case study on the Enjoy England campaign.

There was one very interesting twist on the usual launch event idea which was a ‘Developer Challenge’. Several members of the development team spent the next 24 hours answering briefs given to us by the media partners at the event. It was run very much like a typical hack day, but the hacks were inspired by the ideas our partners are thinking about. Developer advocate Michael Brunton-Spall wrote up the results if you want to see what people built.

Here is the presentation we gave at the launch event:


(Had we chosen a day to launch other than the same day that Google threw a press release party I think you’d already know all this.)

Do the right thing

Of all the things that make this initiative as successful as it is, the thing that strikes me most is how engaged and supportive the executive team is. Alan Rusbridger, Carolyn McCall, Tim Brooks, Derek Gannon, Emily Bell, Mike and Adam, to name a few, are enthusiastic sponsors because this is the right thing to do.

They created a healthy environment for this project to exist and let everyone work out what it meant and how to do it together.

Alan articulated what we’re trying do to in the Cudlipp lecture earlier this year. Among other things, Alan’s framework is an understanding that our abilities as a major media brand and those of the people formerly known as the audience are stronger when unified than they are when applied separately.

Most importantly, we can afford to venture into open models like this one because we are owned by the Scott Trust, not an individual or shareholders. The organization wants us to support journalism and a free press.

“The Trust was created in 1936 to safeguard the journalistic freedom and liberal values of the Guardian. Its core purpose is to preserve the financial and editorial independence of the Guardian in perpetuity, while its subsidiary aims are to champion its principles and to promote freedom of the press in the UK and abroad.”

The Open Platform launch was a big day for me and my colleagues. It was a big day for the future of the Guardian. I hope people also see that it was a major milestone toward a brighter future for journalism itself.

Using fantasy football to drive network effects

Network effects accelerate when services are accessible wherever the user is engaged. That leap has been made in many different contexts in online media from advertising (AdSense) to participation and personal publishing (Flickr and Twitter).

More mainstream publishers got close to this when they began publishing RSS feeds, but the effects of the RSS reading experience don’t come back to the publisher and add value to the wider network like they should.

A click back to the article on the source domain does not improve that article for everyone else who reads it, for example.

It may seem difficult to create network effects around content except in the form of things like reader comments and social bookmarking. But now there are some new ways to create network effects in the publishing business.

Most publishers have found some kind of social tool that makes sense as part of what they offer. It may be a forum, a friends network, or in some cases a game or contest. All those things can capture activity and engage the participants from anywhere on the Internet.

We recently launched a new fantasy football application at The Guardian (when I say ‘football’ I mean ‘soccer’), and we immediately began thinking about where else people might enjoy playing the game. The developers and product manager cranked out a very rudimentary iGoogle Gadget version of the app so that you can stay on top of what’s happening in the game directly from your browser start page.

The gadget is not yet fully functional, but when we start reflecting your activity in the game back to you through the gadget then network effects will be possible. I haven’t been a huge fan of most of the social apps out there, but I can definitely see myself using this one a lot.

In many ways, it also makes me a more frequent user of Google than I already was, but that’s a topic for another post.

At this point in the evolution of the Internet, the online product launch checklist probably dictates that a portable version of a service is a minimum requirement, must-have feature. In that model, the domain can serve as a rules engine, storage and a transaction hub, but the activity of an application needs only a lightweight container and an end-user who’s happy with the experience wherever it may exist.

Local community data reporting

EveryBlock has taken a very data intensive look at local news reporting. As founder Adrain Holovaty explains:

“An overall goal of EveryBlock is to point you to news near your block. We’ve been working hard to do a good job of this so far by accumulating public records, cataloging newspaper stories and pulling together various other geographic information from the Web.”

This generally takes the form of raw data points placed on maps. They recently rolled out a variation on the theme by using topic-specific data which adds more context to the local news reporting idea.

“A week or so ago, 15 people were arrested on bribery charges as part of a federal probe into corruption in Chicago city government. We’ve analyzed U.S. Attorney Patrick J. Fitzgerald’s complaint documents and cataloged the specific addresses mentioned within. On the project’s front page, you can view every location we found, along with a relevant excerpt from the complaint. You can sort this data in various ways, including a list and map of all the alleged bribe locations.”

This is the type of value that’s otherwise kind of missing from the experience. Rather than providing a mostly pure research tool, the site now gives some insight and perspective with an editorial view on the data. In this case, the data is telling a story that otherwise might seem a little distant to you until you see how the issue may in fact be a very real one right in your backyard, so to speak.

But it occurred to me that the community is probably even better able to capture and share this level of useful insight. It would be really neat to see EveryBlock open the reporting and mapping process so that anyone who has an interest in exposing the trends in their neighborhood or elsewhere had a platform to do so.

Average payment (€) by Area
Similar to the way Swivel allows you to collect data in spreadsheet form, visualize it and then share it the way Flickr and YouTube allow you to share, EveryBlock could provide an environment for individuals to do the reporting in their neighborhood that matters to them. The wider community could then benefit from the work of a few, and suddenly you have a really powerful local news vehicle.

This isn’t necessarily in contrast to the approach Outside.in has taken by aggregating shared information from around the web, but it certainly puts some structure around it in a way that may be necessary.

Managing a community is a very different problem than aggregating and presenting useful local data. But I wonder if it’s a necessary next step to get both of these fledgling but very forward-thinking local media services closer to critical mass.

The useful convergence of data

I have only one prediction for 2008. I think we’re finally about to see the useful combination of the 4 W’s – Who, What, Where, and When.

Marc Davis has done some interesting research in this area at Yahoo!, and Bradley Horowitz articulated how he sees the future of this space unfolding in a BBC article in June ’07:

“We do a great job as a culture of “when”. Using GMT I can say this particular moment in time and we have a great consensus about what that means…We also do a very good job of “where” – with GPS we have latitude and longitude and can specify a precise location on the planet…The remaining two Ws – we are not doing a great job of.”

I’d argue that the social networks are now really honing in on “who”, and despite having few open standards for “what” data (other than UPC) there is no shortage of “what” data amongst all the “what” providers. Every product vendor has their own version of a product identifier or serial number (such as Amazon’s ASIN, for example).

We’ve seen a lot of online services solving problems in these areas either by isolating specific pieces of data or combining the data in specific ways. But nobody has yet integrated all 4 in a meaningful way.


Jeff Jarvis’ insightful post on social airlines starts to show how these concepts might form in all kinds of markets. When you’re traveling it makes a lot of sense to tap into “who” data to create compelling experiences that will benefit everyone:

  • At the simplest level, we could connect while in the air to set up shared cab rides once we land, saving passengers a fortune.
  • We can ask our fellow passengers who live in or frequently visit a destination for their recommendations for restaurants, things to do, ways to get around.
  • We can play games.
  • What if you chose to fly on one airline vs. another because you knew and liked the people better? What if the airline’s brand became its passengers?
  • Imagine if on this onboard social network, you could find people you want to meet – people in the same business going to the same conference, people of similar interests, future husbands and wives – and you can rendezvous in the lounge.
  • The airline can set up an auction marketplace for at least some of the seats: What’s it worth for you to fly to Berlin next Wednesday?

Carrying the theme to retail markets, you can imagine that you will walk into H&M and discover that one of your first-degree contacts recently bought the same shirt you were about to purchase. You buy a different one instead. Or people who usually buy the same hair conditioner as you at the Walgreen’s you’re in now are switching to a different hair conditioner this month. Though this wouldn’t help someone like me who has no hair to condition.

Similarly, you can imagine that marketing messages could actually become useful in addition to being relevant. If CostCo would tell me which of the products I often buy are on sale as I’m shopping, or which of the products I’m likely to need given what they know about how much I buy of what and when, then my loyalty there is going to shoot through the roof. They may even be able to identify that I’m likely buying milk elsewhere and give me a one-time coupon for CostCo milk.

Bradley sees it playing out on the phone, too:

“On my phone I see prices for a can of soup in my neighbourhood. It resolves not only that particular can of soup but knows who I am, where I am and where I live and helps me make an intelligent decision about whether or not it is a fair price.

It has to be transparent and it has to be easy because I am not going to invest a lot of effort or time to save 13 cents.”

It may be unrealistic to expect that this trend will explode in 2008, but I expect it to at least appear in a number of places and inspire future implementations as a result. What I’m sure we will see in 2008 is dramatic growth in the behind-the-scenes work that will make this happen, such as the development and customization of CRM-like systems.

Lots of companies have danced around these ideas for years, but I think the ideas and the technologies are finally ready to create something real, something very powerful.

Photo: SophieMuc

The Internet’s secret sauce: surfacing coincidence

What is it that makes my favorite online services so compelling? I’m talking about the whole family of services that includes Dopplr, Wesabe, Twitter, Flickr, and del.icio.us among others.

I find it interesting that people don’t generally refer to any of these as “web sites”. They are “services”.

I was fortunate enough to spend some time with Dopplr’s Matt Biddulph and Matt Jones last week while in London where they described the architecture of what they’ve built in terms of connected data keys. The job of Dopplr, Mr. Jones said, was to “surface coincidence”.

I think that term slipped out accidentally, but I love it. What does it mean to “surface coincidence”?

It starts by enabling people to manufacture the circumstances by which coincidence becomes at least meaningful if not actually useful. Or, as Jon Udell put it years ago now when comparing Internet data signals to cellular biology:

“It looks like serendipity, and in a way it is, but it’s manufactured serendipity.”

All these services allow me to manage fragments of my life without requiring burdensome tasks. They all let me take my data wherever I want. They all enhance my data by connecting it to more data. They all make my data relevant in the context of a larger community.

When my life fragments are managed by an intelligent service, then that service can make observations about my data on my behalf.

Dopplr can show me when a distant friend will be near and vice versa. Twitter can show me what my friends are doing right now. Wesabe can show me what others have learned about saving money at the places where I spend my money. Among many other things Flickr can show me how to look differently at the things I see when I take photos. And del.icio.us can show me things that my friends are reading every day.

There are many many behaviors both implicit and explicit that could be managed using this formula or what is starting to look like a successful formula, anyhow. Someone could capture, manage and enhance the things that I find funny, the things I hate, the things at home I’m trying to get rid of, the things I accomplished at work today, the political issues I support, etc.

But just collecting, managing and enhancing my life fragments isn’t enough. And I think what Matt Jones said is a really important part of how you make data come to life.

You can make information accessible and even fun. You can make the vast pool feel manageable and usable. You can make people feel connected.

And when you can create meaning in people’s lives, you create deep loyalty. That loyalty can be the foundation of larger businesses powered by advertising or subscriptions or affiliate networks or whatever.

The result of surfacing coincidence is a meaningful action. And those actions are where business value is created.

Wikipedia defines coincidence as follows:

“Coincidence is the noteworthy alignment of two or more events or circumstances without obvious causal connection.”

This is, of course, similar and related to the definition of serendipity:

“Serendipity is the effect by which one accidentally discovers something fortunate, especially while looking for something else entirely.”

You might say that this is a criteria against which any new online service should be measured. Though it’s probably so core to getting things right that every other consideration in building a new online service needs to support it.

It’s probably THE criteria.

Building community is hard

Jay Rosen has an interesting post on the failure of AssignmentZero, an effort to build a publicly funded crowdsourced news organization.

Among the many lessons, he keeps coming back to motivation and incentive.

“A well managed project correctly estimates what motivates people to join in, what the various rewards are for participants, and where the practical limits of their involvement lie.

…amateur production will never replace the system of paid correspondents. It only springs to life when people are motivated enough to self-assign and follow through.”

The idea wasn’t fundamentally broken, in my mind. Crowdsourced news is very powerful. As Derek Powazek said,

“At its best, crowdsourcing is about expanding the walls of the newsroom to the internet, giving an opportunity to people with real experience to share their expertise. This is a point that’s often lost on people who are just looking to make a quick buck on Web 2.0.”

More than anything else, I suspect that AssignmentZero failed because there weren’t any readers. Motivation wouldn’t have been a problem with a NYTimes-sized audience.

To date, I’ve never seen a better explanation of the motivations in collaborative online experiences than Yochai Benkler’s paper called Coase’s Penguin. One of my favorite excerpts from that is where he warns against paying for contributions from the community:

“An act of love drastically changes meaning when one person offers the other money at its end, and a dinner party guest who will take out a checkbook at the end of dinner instead of bringing flowers or a bottle of wine at the beginning will likely never be invited again.”

There are as many motivations as there are contributors in a shared media project. What holds them together is more art than science. Some of that art includes good timing and luck. But it also requires a unique kind of commitment and salesmanship from the leaders of the project.

I’ve begun to wonder if the tipping point happens when the confluence of the community size, the ROI to the contributors and the depth of the trust relationship with the company or the brand creates more value than the sum of the parts. Maybe the science of collaboration services can be found by quantifying the meaning of the relationships between those elements: size, cost, benefit and trust.

Or it could also be that the secret sauce inside the Craig Newmarks, Stewart Butterfields and Jimmy Waleses of the world is much more complicated and nuanced than anyone realizes.

The business of network effects

The Internet platform business has some unique challenges. It’s very tempting to adopt known models to make sense of it, like the PC business, for example, and think of the Internet platform like an operating system.

The similarities are hard to deny, and who wouldn’t want to control the operating system of the Internet?

In 2005, Jason Kottke proposed a vision for the “WebOS” where users could control their experience with tools that leveraged a combination of local storage and a local server, networked services and rich clients.

“Applications developed for this hypothetical platform have some powerful advantages. Because they run in a Web browser, these applications are cross platform, just like Web apps such as Gmail, Basecamp, and Salesforce.com. You don’t need to be on a specific machine with a specific OS…you just need a browser + local Web server to access your favorite data and apps.”

Prior to that post, Nick Carr offered a view on the role of the browser that surely resonated with the OS perspective for the Internet:

“Forget the traditional user interface. The looming battle in the information technology business is over control of the utility interface…Control over the utility interface will provide an IT vendor with the kind of power that Microsoft has long held through its control of the PC user interface.”

He also responded later to Kottke’s vision saying that the reliance on local web and storage services on a user’s PC may be unnecessary:

“Your personal desktop, residing entirely on a distant server, will be easily accessible from any device wherever you go. Personal computing will have broken free of the personal computer.”

But the client layer is merely a piece of the much larger puzzle, in my opinon.

Dare Obasanjo more recently broke down the different ideas of what “Cloud OS” might mean:

“I think it is a good idea for people to have a clear idea of what they are talking about when they throw around terms like “cloud OS” or “cloud platform” so we don’t end up with another useless term like SOA which means a different thing to each person who talks about it. Below are the three main ideas people often identify as a “Web OS”, “cloud OS” or “cloud platform” and examples of companies executing on that vision.”

He defines them as follows:

  1. WIMP Desktop Environment Implemented as a Rich Internet Application (The YouOS Strategy)
  2. Platform for Building Web-based Applications (The Amazon Strategy)
  3. Web-based Applications and APIs for Integrating with Them (The Google Strategy)

The OS metaphor has lots of powerful implications for business models, as we’ve seen on the PC. The operating system in a PC controls all the connections from the application user experience through the filesystem down through the computer hardware itself out to the interaction with peripheral services. Being the omniscient hub makes the operating system a very effective taxman for every service in the stack. And from there, the revenue streams become very easy to enable and enforce.

But the OS metaphor implies a command-and-control dynamic that doesn’t really work in a global network controlled only by protocols.

Internet software and media businesses don’t have an equivilent choke point. There’s no single processor or function or service that controls the Internet experience. There’s no one technology or one company that owns distribution.

There are lots of stacks that do have choke points on the Internet. And there are choke points that have tremendous value and leverage. Some are built purely and intentionally on top of a distribution point such as the iPod on iTunes, for example.

But no single distribution center touches all the points in any stack. The Internet business is fundamentally made of data vectors, not operational stacks.

Jeremy Zawodny shed light on this concept for me using building construction analogies.

He noted that my building contractor doesn’t exclusively buy Makita or DeWalt or Ryobi tools, though some tools make more sense in bundles. He buys the tool that is best for the job and what he needs.

My contractor doesn’t employ plumbers, roofers and electricians himself. Rather he maintains a network of favorite providers who will serve different needs on different jobs.

He provides value to me as an experienced distribution and aggregation point, but I am not exclusively tied to using him for everything I want to do with my house, either.

Similarly, the Internet market is a network of services. The trick to understanding what the business model looks like is figuring out how to open and connect services in ways that add value to the business.

In a precient viewpoint from 2002 about the Internet platform business, Tim O’Reilly explained why a company that has a large and valuable data store should open it up to the wider network:

“If they don’t ride the horse in the direction it’s going, it will run away from them. The companies that “grasp the nettle firmly” (as my English mother likes to say) will reap the benefits of greater control over their future than those who simply wait for events to overtake them.

There are a number of ways for a company to get benefits out of providing data to remote programmers:

Revenue. The brute force approach imposes costs both on the company whose data is being spidered and on the company doing the spidering. A simple API that makes the operation faster and more efficient is worth money. What’s more, it opens up whole new markets. Amazon-powered library catalogs anyone?

Branding. A company that provides data to remote programmers can request branding as a condition of the service.

Platform lock in. As Microsoft has demonstrated time and time again, a platform strategy beats an application strategy every time. Once you become part of the platform that other applications rely on, you are a key part of the computing infrastructure, and very difficult to dislodge. The companies that knowingly take their data assets and make them indispensable to developers will cement their role as a key part of the computing infrastructure.

Goodwill. Especially in the fast-moving high-tech industry, the “coolness” factor can make a huge difference both in attracting customers and in attracting the best staff.”

That doesn’t clearly translate into traditional business models necessarily, but if you look at key business breakthroughs in the past, the picture today becomes more clear.

  1. The first breakthrough business model was based around page views. The domain created an Apple-like controlled container. Exposure to eyeballs was sold by the thousands per domain. All the software and content was owned and operated by the domain owner, except the user’s browser. All you needed was to get and keep eyeballs on your domain.
  2. The second breakthrough business model emerged out of innovations in distribution. By building a powerful distribution center and direct connections with the user experience, advertising could be sold both where people began their online experiences and at the various independent domain stacks where they landed. Inventory beget spending beget redistribution beget inventory…it started to look a lot like network effects as it matured.
  3. The third breakthrough business model seems to be a riff on its predecessors and looks less and less like an operating system. The next breakthrough is network effects.

Network EffectsNetwork effects happen when the value of the entire network increases with each node added to the network. The telephone is the classic example, where every telephone becomes more valuable with each new phone in the network.

This is in contrast to TVs which don’t care or even notice if more TVs plug in.

Recommendation engines are the ultimate network effect lubricator. The more people shop at Amazon, the better their recommendation engine gets…which, in turn, helps people buy more stuff at Amazon.

Network effects are built around unique and useful nodes with transparent and highly accessible connection points. Social networks are a good example because they use a person’s profile as a node and a person’s email address as a connection point.

Network effects can be built around other things like keyword-tagged URLs (del.icio.us), shared photos (flickr), songs played (last.fm), news items about locations (outside.in).

The contribution of each data point wherever that may happen makes the aggregate pool more valuable. And as long as there are obvious and open ways for those data points to talk to each other and other systems, then network effects are enabled.

Launching successful network effect businesses is no easy task. The value a participant can extract from the network must be higher than the cost of adding a node in the network. The network’s purpose and its output must be indespensible to the node creators.

Massively distributed network effects require some unique characteristics to form. Value not only has to build with each new node, but the value of each node needs to increase as it gets leveraged in other ways in the network.

For example, my email address has become an enabler around the Internet. Every site that requires a login is going to capture my email address. And as I build a relationship with those sites, my email address becomes increasingly important to me. Not only is having an email address adding value to the entire network of email addresses, but the value of my email address increases for me with each service that is able to leverage my investment in my email address.

Then the core services built around my email address start to increase in value, too.

For example, when I turned on my iPhone and discovered that my Yahoo! Address Book was automatically cooked right in without any manual importing, I suddenly realized that my Yahoo! Address Book has been a constant in my life ever since I got my first Yahoo! email address back in the ’90’s. I haven’t kept it current, but it has followed me from job to job in a way that Outlook has never been able to do.

My Yahoo! Address Book is becoming more and more valuable to me. And my iPhone is more compelling because of my investment in my email address and my address book.

Now, if the network was an operating system, there would be taxes to pay. Apple would have to pay a tax for accessing my address book, and I would have to pay a tax to keep my address book at Yahoo!. Nobody wins in that scenario.

User data needs to be open and accessible in meaningful ways, and revenue needs to be built as a result of the effects of having open data rather than as a margin-based cost-control business.

But Dare Obasanjo insightfully exposes the flaw in reducing openness around identity to individual control alone:

“One of the bitter truths about “Web 2.0” is that your data isn’t all that interesting, our data on the other hand is very interesting…A lot of “Web 2.0″ websites provide value to their users via wisdom of the crowds appproaches such as tagging or recommendations which are simply not possible with a single user’s data set or with a small set of users.”

Clearly, one of the most successful revenue-driving opportunities in the networked economy is advertising. It makes sense that it would be since so many of the most powerful network effects are built on people’s profiles and their relationships with other people. No wonder advertisers can’t spend enough money online to reach their targets.

It will be interesting to see how some of the clever startups leveraging network effects such as Wesabe think about advertising.

Wesabe have built network effects around people’s spending behavior. As you track your finances and pull in your personal banking data, Wesabe makes loose connections between your transactions and other people who have made similar transactions. Each new person and each new transaction creates more value in the aggregate pool. You then discover other people who have advice about spending in ways that are highly relevant to you.

I’ve been a fan of Netflix for a long time now, but when Wesabe showed me that lots of Netflix customers were switching to Blockbuster, I had to investigate and before long decided to switch, too. Wesabe knew to advise me based on my purchasing behavior which is a much stronger indicator of my interests than my reading behavior.

Advertisers should be drooling at the prospects of reaching people on Wesabe. No doubt Netflix should encourage their loyal subscribers to use Wesabe, too.

The many explicit clues about my interests I leave around the Internet — my listening behavior at last.fm, my information needs I express in del.icio.us, my address book relationships, my purchasing behavior in Wesabe — are all incredibly fruitful data points that advertisers want access to.

And with managed distribution, a powerful ad platform could form around these explicit behaviors that can be loosely connected everywhere I go.

Netflix could automatically find me while I’m reading a movie review on a friend’s blog or even at The New York Times and offer me a discount to re-subscribe. I’m sure they would love to pay lots of money for an ad that was so precisely targeted.

That blogger and The New York Times would be happy share revenue back to the ad platform provider who enabled such precise targeting that resulted in higher payouts overall.

And I might actually come back to Netflix if I saw that ad. Who knows, I might even start paying more attention to ads if they started to find me rather than interrupt me.

This is why the Internet looks less and less like an operating system to me. Network effects look different to me in the way people participate in them and extract value from them, the way data and technologies connect to them, and the way markets and revenue streams build off of them.

Operating systems are about command-and-control distribution points, whereas network effects are about joining vectors to create leverage.

I know little about the mathematical nuances of chaos theory, but it offers some relevant philosophical approaches to understanding what network effects are about. Wikipedia addresses how chaos theory affects organizational development:

“Most of the focus on chaos theory is primarily rooted in the underlying patterns found in an otherwise chaotic enviornment, more specifically, concepts such as self-organization, bifurcation and self-similarity…

Self-organization, as opposed to natural or social selection, is a dynamic change within the organization where system changes are made by recalculating, re-inventing and modifying its structure in order to adapt, survive, grow and develop. Self-organization is the result of re-invention and creative adaptation due to the introduction of, or being in a constant state of, perturbed equilibrium.”

Yes, my PC is often in a state of ‘perturbed equilibrium’ but not because it wants to be.

Thinking about media as a platform

Back in my InfoWorld days (2004-ish?) I somehow woke up to the idea that media could be a platform.1 Whereas my professional media experience prior to that was all about creating user experiences that resulted in better page views and conversions, something changed in the way I perceived how online media was supposed to work.

I didn’t have language to use for it at the time (still working on it, actually), but I knew it wasn’t inspired by the “openness” and “walled garden” metaphors so much. Neither concept reflected the opportunity for me. Once I saw the opportunity, though, the shift happening in online media seemed much much bigger.

In a presentation at the Bioneers conference back in August 2000 (below), architect William McDonough talked about designing systems that leverage nature’s strengths for mutually beneficial growth rather than for conservation or merely sustainability.

He tells us to design with positive results in mind instead of using less bad materials,

Similarly, the implications around the “openness” and “walled garden” concepts get clouded by the tactical impressions those words draw for someone who has unique assets in the media business.

It’s not about stopping bad behavior or even embracing good behavior. It’s about investing in an architecture that promotes growth for an entire ecosystem. If you do it right, you will watch network effects take hold naturally. And then everyone wins.

When you look around the Internet media landscape today you see a lot of successful companies that either consciously or subconsciously understand how to make media work as a platform. MySpace created a fantastic expression platform, though perhaps unwittingly. Wikipedia evolved quickly into a massive research platform. Flickr and del.icio.us, of course, get the network effects inherent in sharing information…photos and links, respectively. Washingtonpost and BBC Backstage are moving toward national political information platforms. Last.fm is a very succssful music listening platform if not one of the most interesting platforms among them all.

All of these share a common approach. At a simple level, the brand gets stronger the further their data and services reach outside of their domain and into the wider market.

But the most successful media platforms are the ones that give their users the power to impact the experience for themselves and to improve the total experience for everyone as they use it.

My commitment to flickr, del.icio.us and last.fm gets deeper and deeper the more I’m able to apply them in my online lifestyle wherever that may be. We have a tangible relationship. And I have a role in the wider community, even if only a small part, and that community has a role in my experience, too.

The lesson is that it’s not about the destination — it’s about the relationship. Or, if you like the Cluetrain language, it’s about the conversation, though somehow “relationship” seems more meaningful than “conversation” to me. Ask any salesperson whether they’d prefer to have a relationship or a conversation with a potential customer.

Ok, so user engagement can extend outside a domain. Where’s the opportunity in that?

Very few media platforms know how to leverage their relationships to connect buyers and sellers and vice versa. They typically just post banner ads or text links on their sites and hope people click on them. Creating a fluid and active marketplace that can grow is about more than relevant advertising links.

Amazon created an incredibly powerful marketplace platform, but they are essentially just a pure play in this space. They are about buying and selling first and foremost. Relationships on their platforms are transactional.

Media knows how to be more than that.

eBay and Craigslist get closer to colliding the buying/selling marketplace with deeper media experiences. People build relationships in micromarkets, but again it’s all about a handshake and then good riddance on eBay and Craigslist.

Again, media knows how to be more than that.

The big opportunity in my mind is in applying the transactional platform concept within a relationship-building environment.

A more tangible example, please…?


Washingtonpost.com is an interesting case, as they have been more aggressive than most traditional media companies in terms of “openness”. They have data feeds for all of their content. And they have an amazing resource in the U.S. Congress Votes Database, a feed of legislative voting records sliced in several different ways. For example, you can watch what legislation Nancy Pelosi votes on and how she votes.

Unfortunately, everything Washingtonpost.com offers is read-only. You can pull information from Washingtonpost.com, but you can’t contribute to it. You can’t serve the wider Washingtonpost.com community with your additions or edits. You can’t engage with other Washingtonpost.com community members in meaningful ways.

Washingtonpost.com thinks of their relationship with you in a one-to-many way. They are one, and you are one of many.

Instead, they should think of themselves as the government data platform. Every citizen in the US should be able to feed data about their local government into the system, and the wider community should be able to help edit and clean community-contributed data (or UGC for you bizdev folks).

For example, I recently spent some time investigating crime data and how that gets shared or not shared in various local communities. Local citizens could provide a very powerful resource if they were empowered to report crime in meaningful ways on the Internet.

Washingtonpost.com is as well suited as anyone to provide that platform.

Now, imagine the opportunity for Washingtonpost.com if people around the US were reporting, editing and analyzing local crime data from Washingtonpost’s platform. They would become a critical source of national information and news across the country. Washintonpost.com would be well poised to be the primary source of any type of government-related information.

The money would soon follow.

As a result of becoming essential in the ecosystem of local and national citizen data, they would expand their advertising possibilities exponentially. They could create an ad platform (or partner with one) that is tuned particularly for their ecosystem. Then any number of services could start forming around the combination of their data platform and their ad platform.


You can imagine legal services, security, counseling and financing services wanting to reach directly into my local Potrero Hill crimewatch community. The marketplace would probably be very fluid where people are recommending services and providers are helping the community as a whole as a way to build relationships.

Washingtonpost could sit behind all these services, powering the data and taking a cut of all the advertising.

Again, it’s not just about being “open” or taking down the “walled garden”.

The “openness” and “walled garden” concepts which often turn into accusations feel more like objectives than strategic directions. If “openness” was the goal, then offering everything as RSS would be the game.

No, RSS is just step one. The media platform game is much more than that.

It’s about both being a part of the larger Internet ecosystem and understanding how to grow and design a future that benefits lots of different constituents. You can be a source in someone else’s platform, a vehicle within a wider networked platform and a hub at the center of your own ecosystem all at the same time.

I would never claim this stuff is easy, as I certainly failed to make that happen while at InfoWorld. The first place to start, in my opinion, is to stop worrying about “openness” and “walled gardens”. Those are scary ideas that don’t necessarioly inspire people to build or participate in growing ecosystems.

Instead, it’s important to understand “network effects” and “platforms“. Once you understand how media can be a platform, the world of opportunity will hopefully start to look a lot bigger, as big as the Internet itself, if not even bigger than that.

It’s at that point that you may wonder why you would pursue anything else.

1 It shouldn’t be surprising that my thinking changed while surrounded by thinkers like Jon Udell, Steve Gillmor, and Steve Fox to name a few who all waved the web services flag and sang the software-as-a-service song before many of the leading IT efforts at some of the most innovative companies knew how to put those words into coherent sentences. Those concepts can apply to lots of markets, media among them.

Freebase.com is hot

I don’t get a chance to review products often enough these days. But when I heard about Freebase I knew I needed to dive into that one as soon as I was able.


Fortunately, I was invited only yesterday to take a peak. And I’m officially joining the hype wagon on this one.

Someone once described it as Wikipedia for structured data. I think that’s a good way to think about it.

That image leaves out one of the most powerful aspect of the tool, though. The pivot points that are created when a piece of data can be interlinked automatically and dynamically with other pieces of data creates a network of information that is more powerful than an edited page.

The Freebase screencast uses the movie database example to show this. You can dive in and out from actor to film which if you wanted could then carry on to topic to location to government to politician to gossip and on and on and on. And everything is editable.

Now, they didn’t stop at making the ultimate community-driven relational database. They exposed all the data in conveniently shareable formats like JSON. This means that I could build a web site that leverages that data and makes it available to my site visitors. I only need to link back to Freebase.com.

But that’s not all. In combination with the conveniently accessible data, they allow people to submit data to Freebase programmatically through their APIs. They will need to create some licensing controls for this to really work for data owners (NBA stats data and NYSE stock data, for example). But that’s getting easier to solve, and you can see that they are moving in that direction already.

Here’s a brief clip of the screencast which shows some other interesting concepts in action, too:

Suddenly, you can imagine that Freebase becomes a data clearinghouse, a place where people post information perhaps even indirectly through 3rd parties and make money or attract customers as others redistribute your data from the Freebase distribution point. They have a self-contained but infinitely scaleable data ecosystem.

I can imagine people wanting to manage their personal profile in this model and creating friends lists much like the typical social network except that it’s reusable everywhere on the Internet. I can imagine consumer goods producers weaving coupons and deals data with local retailer data and reaching buyers in highly relevant ways we haven’t seen yet.

Freebase feels very disruptive to me. I’m pretty sure that this is one to watch. And I’m not alone…

Michael Arrington: “Freebase looks to be what Google Base is not: open and useful.”

Jon Udell: “Freebase is aptly named, I am drawn like a moth to its flame.”

Tim O’Reilly: “Unlike the W3C approach to the semantic web, which starts with controlled ontologies, Metaweb adopts a folksonomy approach, in which people can add new categories (much like tags), in a messy sprawl of potentially overlapping assertions.”

John Markoff: “On the Web, there are few rules governing how information should be organized. But in the Metaweb database, to be named Freebase, information will be structured to make it possible for software programs to discern relationships and even meaning”

In some ways, it seems like the whole Web 2.0 era was merely an incubation period for breakthroughs like Freebase. Judging by the amount of data already submitted in the alpha phase, I suspect this is going to explode when it officially launches.

How to fix building construction bureaucracy

Sometimes I forget to step outside of our little bubble here and see how people use or in fact don’t use the Internet. When I get that chance I often wonder if anything I’m doing in my career actually matters to anyone.

Usually, however, I’m reminded that even though the Internet isn’t weaved into every aspect of everything, it has great potential in places you might not consider.

For example, I’ve been remodelling my house to make room for a new little roommate due to be delivered in September. I’m trying to do most of the work myself or with help from friends and neighbors. I’m trying to save money, but I also really enjoy it. It’s a fantastic way to reconnect with the things that matter…food, shelter, love and life.

Well, I made the mistake of working without permits fully aware that I probably should have them. It’s my natural inclination to run around bureaucracy whenever possible.


As luck would have it, just as the pile of demolition debris on the sidewalk outside my house was at its worst, a building inspector happened to drive by on his way to another job. He asked to see my permit to which I replied, “The boss isn’t here. Can you come back later?”

The building inspector just laughed. After pleading a bit and failing, I started making calls to get drawings and to sort out the permits.

It was at this moment I realized how much building planning and construction could benefit from the advances made in the Internet market the last few years. The part of construction that people hate most is the one that is perhaps the most important. And it is this part that the Internet is incredibly well-suited to improve.

Admittedly, the permit process was not actually that painful and relatively cheap, too. I have spent in total maybe 1 day dealing with permits and drawings, so far, with a bit more to come, I’m sure.

But the desired effect of permitting jobs is sorely underserved by its process.

At the end of the day what you want is the highest building quality possible. You want builders using proven methods with at least semi-predictable outcomes. You want to make sure nobody gets hurt. And you want incentives for people to share expertise and information.

Rather than be a gatekeeper, the city needs to be an enabler.

One of the brochures I read called “How to Obtain a Permit” includes a whitelist of project types. I’m apparently allowed to put down carpets and hang things on my walls without a permit. Glad to know that.

Strangely, after explaining all the ways the city asserts itself into the process, on the very last page of the brochure it then says, “Remember, we are here to assist you. If you have any questions about your project, please give us a call!” I didn’t meet one person in the 6 queues I waded through the first morning who wanted to help me. They were mostly bored out of their brains.

Instead, the city should be putting that brainpower to work finding ways to lubricate conversation and collaboration around solving building problems. If the building community was in fact a community powered by thoughtful city-employed engineers, then I would be much more interested in working with them. I might even become dependent on them.

For example, if they helped me organize, store, print and even share my plans, then I’d be more than happy to let them keep my most current drawings, the actual plans I’m using to build with. If they could connect me to licensed contractors and certified service providers, I’d gladly give them my budget.

As it stands, my incentive is to avoid them and hide information whenever possible.

Imagine if I was able to submit a simple SketchUp plan to a construction service marketplace. I could then sit back and watch architects and interior designers bid for the planning work. My friends in the network could recommend contractors. Tools and parts suppliers could offer me discounts knowing exactly what I needed for the job. I could rate everything that happens and contribute to the reputation of any node in the ecosystem.

Imagine how much more value would be created in the home buying market if a potential buyer could see all this data on a house that was for sale. I might be able to sell my home for a higher price if my remodel was done using highly reputable providers. There would be a financial incentive for me to document everything and to get the right certifications on the work.

Imagine lenders knowing that I’m an excellent remodeller based on my reputation and sales track record. I might be able to negotiate better terms for a loan or even solicit competing bids for my mortgage on the next house I want to invest in.

At every step in the process, there is a role for the city government to add value and thus become more relevant. Then the more I contribute, the more it knows about what’s happening. The more it knows, the more effective it can be in driving better standards and improving safety and legislating where necessary.

My mind spins at the possibilities in such a world. Of course, when you have a hammer everything looks like a nail. But it seems to me that the building permit and inspection business is broken in exactly the places that the Internet is more than capable of fixing.