Category: web services

The Internet’s secret sauce: surfacing coincidence

What is it that makes my favorite online services so compelling? I’m talking about the whole family of services that includes Dopplr, Wesabe, Twitter, Flickr, and del.icio.us among others.

I find it interesting that people don’t generally refer to any of these as “web sites”. They are “services”.

I was fortunate enough to spend some time with Dopplr’s Matt Biddulph and Matt Jones last week while in London where they described the architecture of what they’ve built in terms of connected data keys. The job of Dopplr, Mr. Jones said, was to “surface coincidence”.

I think that term slipped out accidentally, but I love it. What does it mean to “surface coincidence”?

It starts by enabling people to manufacture the circumstances by which coincidence becomes at least meaningful if not actually useful. Or, as Jon Udell put it years ago now when comparing Internet data signals to cellular biology:

“It looks like serendipity, and in a way it is, but it’s manufactured serendipity.”

All these services allow me to manage fragments of my life without requiring burdensome tasks. They all let me take my data wherever I want. They all enhance my data by connecting it to more data. They all make my data relevant in the context of a larger community.

When my life fragments are managed by an intelligent service, then that service can make observations about my data on my behalf.

Dopplr can show me when a distant friend will be near and vice versa. Twitter can show me what my friends are doing right now. Wesabe can show me what others have learned about saving money at the places where I spend my money. Among many other things Flickr can show me how to look differently at the things I see when I take photos. And del.icio.us can show me things that my friends are reading every day.

There are many many behaviors both implicit and explicit that could be managed using this formula or what is starting to look like a successful formula, anyhow. Someone could capture, manage and enhance the things that I find funny, the things I hate, the things at home I’m trying to get rid of, the things I accomplished at work today, the political issues I support, etc.

But just collecting, managing and enhancing my life fragments isn’t enough. And I think what Matt Jones said is a really important part of how you make data come to life.

You can make information accessible and even fun. You can make the vast pool feel manageable and usable. You can make people feel connected.

And when you can create meaning in people’s lives, you create deep loyalty. That loyalty can be the foundation of larger businesses powered by advertising or subscriptions or affiliate networks or whatever.

The result of surfacing coincidence is a meaningful action. And those actions are where business value is created.

Wikipedia defines coincidence as follows:

“Coincidence is the noteworthy alignment of two or more events or circumstances without obvious causal connection.”

This is, of course, similar and related to the definition of serendipity:

“Serendipity is the effect by which one accidentally discovers something fortunate, especially while looking for something else entirely.”

You might say that this is a criteria against which any new online service should be measured. Though it’s probably so core to getting things right that every other consideration in building a new online service needs to support it.

It’s probably THE criteria.

Data dynamics: How the rules of sharing are changing

Today it’s easy to store and share my pictures, my favorite URLs, my thoughts and lots of other things online. There are a range of data repositories that allow me to do this kind of thing in different ways.

What still needs work is how I give trusted services access to much more private data — things like my current location, my spending behavior, access to my friends and family, etc.

To date, most services follow the premise that the looser the controls, the more fluidly data will travel. And that’s all that mattered when it was still hard to get data flowing.

Data flow is no longer an issue. Perhaps data flow has actually become too easy now. And therein lies the problem.

Clearly, blogging, RSS and feed readers drove a lot of the early thinking about syndication. Blogging enabled people to post content in a publicly accessible data repository somewhere for anyone to pull out without any privacy or permissioning controls. The further your content then syndicated, the better.

Wikis and community sites like Slashdot created a slightly more complex read/write dynamic against the central content repository that lots of people could access together. The permissioning model was essentially hierarchical where controls were kept in the hands of a smaller community.

Then Flickr broke ground with a new approach. They applied a user-centric friends and family relationship model to permissioning access to personal photos. Flickr opened up what was once considered private data and defaulted it to a public read-only permission status. But each individual still has a great deal of control over the data he or she contributes.

Similarly, del.icio.us made it possible to store and publicly address what had previously been private data. The nice twist here was the easy-to-understand URLs that allowed machines to consume, interpret and redistribute data stored in del.icio.us.

Where services like Facebook and Wesabe are now breaking ground again is in identifying a security model around highly sensitive data. Contact lists are very personal, but there aren’t many data sets more personal than my purchases and spending patterns.

Neat things can happen when I give machines access to my data, both the things I explicitly ‘own’ and my implicit behaviors. I want machines to act on my behalf and make my data more useful to me in a range of different contexts.

For example, I like the fact that Facebook slurps up my Twitter activity and shares it with my friends in the Facebook network. I don’t want to change my ‘status’ on every service that shows status messages. Similarly, I like that Last.fm captures my listening behavior from iTunes and then uses that data to give back personal recommendations on a badge posted to my blog.

Allowing machines to automatically act on personal data on my bahalf is the right direction for things to go. But important questions need to be resolved.

For example, what happens to my data in all the places I’ve allowed it to appear when I change it? How do permissions pass from one service to another? How do I guarantee that a permission type I grant in one service means the same thing in another service? How do changes propagate? How does consent get revoked?

And even trickier than all that will be the methods for enforcing protection of privacy and penalties for breaking those permissions.

Until trust is measurable with explicit consentual triggers, loosely coupled networks that act on the data I wish to protect are going to struggle to talk to each other. Standards need to enable common sharing tactics. Responsibility needs to be clearly defined. And policies need to be enforceable.

Empowering a person to invest in storing and sharing the more sensitive data he or she owns is going to require a lot more than traditional read/write controls. But given the pace of change right now I suspect the answers will happen as the people behind these services work things out together before the industry taskforces, legal entities and blogosphere sort it out for them.

Thinking about media as a platform

Back in my InfoWorld days (2004-ish?) I somehow woke up to the idea that media could be a platform.1 Whereas my professional media experience prior to that was all about creating user experiences that resulted in better page views and conversions, something changed in the way I perceived how online media was supposed to work.

I didn’t have language to use for it at the time (still working on it, actually), but I knew it wasn’t inspired by the “openness” and “walled garden” metaphors so much. Neither concept reflected the opportunity for me. Once I saw the opportunity, though, the shift happening in online media seemed much much bigger.

In a presentation at the Bioneers conference back in August 2000 (below), architect William McDonough talked about designing systems that leverage nature’s strengths for mutually beneficial growth rather than for conservation or merely sustainability.

He tells us to design with positive results in mind instead of using less bad materials,

Similarly, the implications around the “openness” and “walled garden” concepts get clouded by the tactical impressions those words draw for someone who has unique assets in the media business.

It’s not about stopping bad behavior or even embracing good behavior. It’s about investing in an architecture that promotes growth for an entire ecosystem. If you do it right, you will watch network effects take hold naturally. And then everyone wins.

When you look around the Internet media landscape today you see a lot of successful companies that either consciously or subconsciously understand how to make media work as a platform. MySpace created a fantastic expression platform, though perhaps unwittingly. Wikipedia evolved quickly into a massive research platform. Flickr and del.icio.us, of course, get the network effects inherent in sharing information…photos and links, respectively. Washingtonpost and BBC Backstage are moving toward national political information platforms. Last.fm is a very succssful music listening platform if not one of the most interesting platforms among them all.

All of these share a common approach. At a simple level, the brand gets stronger the further their data and services reach outside of their domain and into the wider market.

But the most successful media platforms are the ones that give their users the power to impact the experience for themselves and to improve the total experience for everyone as they use it.

My commitment to flickr, del.icio.us and last.fm gets deeper and deeper the more I’m able to apply them in my online lifestyle wherever that may be. We have a tangible relationship. And I have a role in the wider community, even if only a small part, and that community has a role in my experience, too.

The lesson is that it’s not about the destination — it’s about the relationship. Or, if you like the Cluetrain language, it’s about the conversation, though somehow “relationship” seems more meaningful than “conversation” to me. Ask any salesperson whether they’d prefer to have a relationship or a conversation with a potential customer.

Ok, so user engagement can extend outside a domain. Where’s the opportunity in that?

Very few media platforms know how to leverage their relationships to connect buyers and sellers and vice versa. They typically just post banner ads or text links on their sites and hope people click on them. Creating a fluid and active marketplace that can grow is about more than relevant advertising links.

Amazon created an incredibly powerful marketplace platform, but they are essentially just a pure play in this space. They are about buying and selling first and foremost. Relationships on their platforms are transactional.

Media knows how to be more than that.

eBay and Craigslist get closer to colliding the buying/selling marketplace with deeper media experiences. People build relationships in micromarkets, but again it’s all about a handshake and then good riddance on eBay and Craigslist.

Again, media knows how to be more than that.

The big opportunity in my mind is in applying the transactional platform concept within a relationship-building environment.

A more tangible example, please…?


Washingtonpost.com is an interesting case, as they have been more aggressive than most traditional media companies in terms of “openness”. They have data feeds for all of their content. And they have an amazing resource in the U.S. Congress Votes Database, a feed of legislative voting records sliced in several different ways. For example, you can watch what legislation Nancy Pelosi votes on and how she votes.

Unfortunately, everything Washingtonpost.com offers is read-only. You can pull information from Washingtonpost.com, but you can’t contribute to it. You can’t serve the wider Washingtonpost.com community with your additions or edits. You can’t engage with other Washingtonpost.com community members in meaningful ways.

Washingtonpost.com thinks of their relationship with you in a one-to-many way. They are one, and you are one of many.

Instead, they should think of themselves as the government data platform. Every citizen in the US should be able to feed data about their local government into the system, and the wider community should be able to help edit and clean community-contributed data (or UGC for you bizdev folks).

For example, I recently spent some time investigating crime data and how that gets shared or not shared in various local communities. Local citizens could provide a very powerful resource if they were empowered to report crime in meaningful ways on the Internet.

Washingtonpost.com is as well suited as anyone to provide that platform.

Now, imagine the opportunity for Washingtonpost.com if people around the US were reporting, editing and analyzing local crime data from Washingtonpost’s platform. They would become a critical source of national information and news across the country. Washintonpost.com would be well poised to be the primary source of any type of government-related information.

The money would soon follow.

As a result of becoming essential in the ecosystem of local and national citizen data, they would expand their advertising possibilities exponentially. They could create an ad platform (or partner with one) that is tuned particularly for their ecosystem. Then any number of services could start forming around the combination of their data platform and their ad platform.


You can imagine legal services, security, counseling and financing services wanting to reach directly into my local Potrero Hill crimewatch community. The marketplace would probably be very fluid where people are recommending services and providers are helping the community as a whole as a way to build relationships.

Washingtonpost could sit behind all these services, powering the data and taking a cut of all the advertising.

Again, it’s not just about being “open” or taking down the “walled garden”.

The “openness” and “walled garden” concepts which often turn into accusations feel more like objectives than strategic directions. If “openness” was the goal, then offering everything as RSS would be the game.

No, RSS is just step one. The media platform game is much more than that.

It’s about both being a part of the larger Internet ecosystem and understanding how to grow and design a future that benefits lots of different constituents. You can be a source in someone else’s platform, a vehicle within a wider networked platform and a hub at the center of your own ecosystem all at the same time.

I would never claim this stuff is easy, as I certainly failed to make that happen while at InfoWorld. The first place to start, in my opinion, is to stop worrying about “openness” and “walled gardens”. Those are scary ideas that don’t necessarioly inspire people to build or participate in growing ecosystems.

Instead, it’s important to understand “network effects” and “platforms“. Once you understand how media can be a platform, the world of opportunity will hopefully start to look a lot bigger, as big as the Internet itself, if not even bigger than that.

It’s at that point that you may wonder why you would pursue anything else.

1 It shouldn’t be surprising that my thinking changed while surrounded by thinkers like Jon Udell, Steve Gillmor, and Steve Fox to name a few who all waved the web services flag and sang the software-as-a-service song before many of the leading IT efforts at some of the most innovative companies knew how to put those words into coherent sentences. Those concepts can apply to lots of markets, media among them.

Preview of the del.icio.us publisher api

I just posted a short screencast on the YDN blog of the cool new publisher api coming from del.icio.us soon. I’ve also embedded the video below. Lots of interesting possibilities with this new service, for sure.

Embed video:
“>

I know where and when my favorite bands are playing

Last.fm is my favoriate web app. There’s nothing else on the Internet that comes close.

This thought occurred to me a few weeks back, but it wasn’t until I explained last.fm to a group of publishers at the Stanford Publishing on the Web course this week that it became a conscious truth.

I mentioned it in the context of the importance of user data in today’s distributed and networked media environment and contrasted it with Pandora. Pandora’s service is driven by vast meta data about content. It’s a very robust service because of the depth of data they work with. But content data without user data is not necessarily a defensible position anymore.


I also happened to stumble on one of the coolest mashups I’ve seen in a while called All Crazy Style via the Yahoo! Mashup Gallery the day of the presentation which I then added into the talk last minute. All Crazy Style simply pulls my last.fm usage data (with my permission) and matches it against my Upcoming.org location (again, with permission) and then shows me where and when the bands I tend to listen to most are playing in my area.

Wow. Love it.

I didn’t know that RJD2 is playing at The Independent in San Francisco December 3rd…and since none of my friends are fans I never would have found out otherwise. And there’s no way an advertisement for such a small event would make it to me through the media I consume.

Additionally, I probably wouldn’t have gone looking through Upcoming.org to find any of these listings, because I’m lazy. But my implicit behavior provides enough data so that I don’t have to explicitly track down when my favorite bands are playing. It also provides enough data to essentially recommend shows that I might like.

I was already a fan of last.fm but I didn’t realize they opened up their APIs this way. Now I’m never going to leave. In fact, I want every music-playing device I own to include the audioscrobbler tracking tool which tracks my listening behavior. I want it to own all my listening behavior, and I want mashups to pull that data to do interesting things for me.

If only I could take last.fm with me offline somehow.

UPDATE: Businessweek coves last.fm this week:

With 15 million unique users a month, 150,000 band biographies, and an amazing 65 million songs listed in its database, Last.FM has attracted the attention of big money.

I hope that’s true for their sake. This is a startup that deserves a big break. But I hope an acquisition doesn’t ruin the service for me.

Scaffolding web sites with Ruby on Rails

I started messing around with Ruby on Rails for the first time on Sunday. This was after spending all day Saturday tearing down kitchen cupboards, tiled sinks and entire walls for a friend who is remodeling his house, so I got my fill of building last weekend whether real or virtual.


Photo: bruce grant

Trying to figure out how Ruby on Rails worked, I felt like I was remodeling my brain. It was as if I walked into Ikea with just a basic idea of what I wanted my new kitchen to look like and then walked out with design schematics and new appliances an hour later. I suddenly had confidence that I could create a really nice web site with a lot of functionality that was basically inaccessible to me before because of my limited programming background.

The “Ah hah!” moment came for me when I added two words to one of the scripts: “scaffold mydatabase”. When I refreshed my web site, I was adding, editing and deleting data in my database via a web interface. It all automatically just worked. Then literally 15 minutes later I had 2 databases talking to eachother.

It’s mindblowing how much power this environment gives to people who aren’t true coders.

I have a feeling I’ll get stuck and frustrated with what I’m trying to build. But I’m very hopeful Ruby on Rails will get me closer than I could with open source PHP tools. If nothing else, I’ll get a sense for this new trend.

Programming seems to have about a 3 year fashion cycle that also intersects with influxes of new ideas for web applications and a full cycle of students coming out of university. Now we’re at the early stages of a creative explosion on the Internet enabled by things like Rails, open APIs, storage solutions like S3, and JSON. And you can also wrap an idea in any number of different business models in even less time than it takes to build the product itself.

Maybe instead of LAMP (Linux, Apache, MySQL, PHP), we now have RASH (Rails, APIs, S3, Hosted).

There must be similar reactions to breakthroughs in the construction industry when things like cross-linked polyethylene (PEX) hit the market. Of course, construction suffers from bad naming as much as any other trade. Not everything can be as cool as a sawzall or funny pipe.

“Loosley Coupled” does not mean “Easy to build”

The concept of “Loose Coupling” is great on so many levels. I’ve used it to describe different types of things in ideal worlds, but I’m starting to see that there is a lot of gray area there that can be maddening in real worlds.

Here is Wikipedia’s current definition of it:

“Loose coupling describes a resilient relationship between two or more computer systems that are exchanging data. Each end of the transaction make their requirements explicit and make few assumptions about the other end. Loosely Coupled systems are considered useful when either the source or the destination computer systems are subject to frequent changes.”

I’m working with a small team on a really fun web-based product that weaves lots of stuff together. The core app we’re working on has a very powerful layer of intelligence built into it, but it depends on a stack of data sources and rendering environments that are all partially isolated and not necessarily production-ready.

This means that we can’t really test the product end-to-end. It means there are several layers of troubleshooting that get added to each bug no matter how small. It means we have to fake a service layer here and there to emulate behavior.

I’m realizing now that “loosely coupled” means you have to think a lot harder about each move instead of just cranking out everything from scratch the way you want it to work.

Ultimately, the power of our platform services including things like scalability and user data management will accelerate this product’s ability to reach a more profound state of being than it could without loose couplings. But the cost of glueing all the things together in parallel means that we spend hours in meetings and constantly reshuffle our attack plan.

It feels like running through mid-court of a dodgeball game.