Category: yahoo

Openness, evil and reusability

I’ve stopped blogging over the last several weeks as I uprooted my family and moved to London to start my new job. But there have been some interesting things worth tracking recently I thought I might mention.

(Interestingly, Twitter usurped any blogging impulses I’ve had during the transition, but it’s time to get back into the long form dialog a little again now that we’re settling in here.)

First, I’m really pleased to see Yahoo!’s open strategy taking shape with things like SearchMonkey, Glue, and the forward-looking presentations done at Web 2.0 Expo. In my opinion, they are still underestimating the power of what Yahoo! could be doing by opening outwardly more, but the momentum is definitely in the right direction regardless of the distracting M&A discussions.

Second, I love where Umair Haque is going with his ‘Good vs Evil’ strategic thinking stuff. He’s getting into why the costs of evil are starting to outweigh the benefits in a globally networked and highly elastic economic landscape.

“As Starbucks and Wal-Mart are discovering, orthodox strategy was built for an industrial world – an equilibrium world of oligopolies, soulless “product”, and zombified “consumers”. But that’s not today’s world.”

Even better than his post, perhaps, is the comment stream which includes this insight from Mike Bonifer who compares today’s competitive landscape to the art of improvisation:

“What many do-gooders fail to acknowledge is that it is not enough to do good. One must also confront, then work artfully at marginalizing, out-witting, out-designing and out-performing the forces of evil that are afoot in the world. Forces like greed, hate, terror, racism, misunderstanding, obfuscation and fear. Heroism is only as strong as the calumny it overcomes.”

Third, I loved hearing the meaty thinking going on in the heads of Lucas Gonze and Jon Udell talking on IT Conversations. It’s as if they are both articulating Clay Shirky’s cognitive surplus view of the world through a music lens:

“Imagine that we lived in a world where all photography was the kind you see in magazines. In this world all photos are taken by professionals and all the people who got their pictures taken are models at the peak of their career. If you had your picture taken normally, you’d think you were hideously ugly. That is the musical world we grew up in, and it’s bogus. Things don’t have to be that way.”

Jon naturally moved the conversation to the problem of discoverability that has been increasingly difficult to deal with as more and more data builds out across the network. He notes some of the challenges as a consumer of interesting things and as someone who has something interesting to offer. He thinks the answer is a bit higher level than traditional syndication:

“There’s a way of publishing that allows something to flow on the network retaining its full fidelity and usability in other contexts.”

Lastly, the open data services space is getting really really interesting now as context and relevance find their way into the mix. For example, the Dash GPS formally rolled out their open service. And then a Guardian colleague pointed me to the AMEE service (“Avoiding Mass Extinctions Engine”) which finds itself being used on Dopplr and the the current Radiohead tour web site (click on ‘Carbon Calculator’).

There are tons of interesting developments unfolding, and I’m seeing all this stuff through fresh eyes again…one of the great benefits of changing jobs. I’ll do my best to keep the blogging energy up and to provide some analysis. Though I’m sure my perspective will shift a bit…to what, I really don’t know, yet.

Open source grid computing takes off

This has been fun to watch. The Hadoop team at Yahoo! is moving quickly to push the technology to reach its potential. They’ve now adopted it on one of the most important applications in the entire business, Yahoo! Search.

From the the Hadoop Blog:

The Webmap build starts with every Web page crawled by Yahoo! and produces a database of all known Web pages and sites on the internet and a vast array of data about every page and site. This derived data feeds the Machine Learned Ranking algorithms at the heart of Yahoo! Search.

Some Webmap size data:

  • Number of links between pages in the index: roughly 1 trillion links
  • Size of output: over 300 TB, compressed!
  • Number of cores used to run a single Map-Reduce job: over 10,000
  • Raw disk used in the production cluster: over 5 Petabytes

I’m still trying to figure out what all this means, to be honest, but Jeremy Zawodny helps to break it down. In this interview, he gets some answers from Arnab Bhattacharjee (manager of the Yahoo! Webmap Team) and Sameer Paranjpye (manager of our Hadoop development):

The Hadoop project is opening up a really interesting discussion around computing scale. A few years ago I never would have imagined that the open source world would be contributing software solutions like this to the market. I don’t know why I had that perception, really. Perhaps all the positioning by enterprise software companies to discredit open source software started to sink in.

As Jeremy said, “It’s not just an experiment or research project. There’s real money on the line.”

For more background on what’s going on here, check out this article by Mark Chu-Carroll “Databases are hammers; MapReduce is a screwdriver”.

This story is going to get bigger, I’m certain.

A handy music playlist tool

I’ve been looking for a way to share playlists on my blog and elsewhere online for a long time. It’s been surprisingly hard to find a really convenient way to do it.

DRM and industry lockdown have been a big part of that, but there have also been too few technical ways to point to music files that are already publicly available. There are tons of legal MP3’s on the Internet that reside at readable URLs today.

Lucas Gonze and his team at Yahoo! solved this problem. They launched a source-agnostic embeddable media player. You can read more about it on YDN.

It’s fantastically simple. All you do is paste this reference to Yahoo!’s media player javascript code anywhere on your web page (I added it at the bottom of my blog templates):

<script type=”text/javascript” src=”http://mediaplayer.yahoo.com/js”></script>

Then you just add an HTML link somewhere on your web page to any MP3 file you want to see in your playlist.

That’s it. You’re already done. The link you just made will now include a small play button in front of it, and a mini media player will appear in the browser.

Here’s a short playlist I quickly put together to show how it works. The 4th track here is particularly relevant to my life:

Cut Chemist – The Garden
Young Einstein (Ugly Duckling) – Handcuts Soul Mix
They Might Be Giants- Birdhouse in Your Soul
LCD Soundsystem – Losing My Edge

The code for that playlist looks like this:

<a href=”http://download.wbr.com/cutchemist/TheGarden.mp3″> Cut Chemist – The Garden </a>
<a href=”http://www.uglyduckling.us/music/HandCutsSoulMix.mp3″> Young Einstein (Ugly Duckling) – Handcuts Soul Mix </a>
<a href=”http://midwesternhousewives.com/mix/The%20Might%20Be%20Giants-%20Birdhouse%20in%20Your%20Soul.mp3″> They Might Be Giants- Birdhouse in Your Soul </a>
<a href=”http://www.personal.psu.edu/users/s/m/smk291/muchies/LCD%20Soundsystem%20-%20Losing%20My%20Edge.mp3″> LCD Soundsystem – Losing My Edge </a>

They’ve included some other nice things in the code that give you some flexibility. You can create a shareable playlist file, and you can add cover art, for example.

What I like most, probably, is the architecture of the solution. Anyone who already links to MP3 files can just add the music player javascript code to their page templates, and it will just work immediately. You don’t have to force fit a heavily branded HTML badge into your web page. And since the links are all standard HTML href’s, the content of the playlist is search engine friendly.

It’s the first time I’ve seen a media player so closely aligned with the way the Internet works.

Lucas posts about the need to unlock how media files are referenced. He wants to take the complexity out of distribution and reduce the concept of music sharing and discoverability to the Internet’s roots with URLs as identifiers:

“Almost all online music businesses right now are in the distribution business, even if they see other functions like discovery or social connection as their main value, because they have no way to connect their discovery or social connection features with a reliable provisioning service from a third party. But provisioning is a commodity service which doesn’t give anybody an edge. They don’t want to import playlists from third parties because *that’s* where they are adding value.

Exporting playlists for others to provision, though, is a different story, and it makes much more sense from a business perspective. Let somebody else deal with provisioning. This is what it would mean for somebody like Launchcast or Pandora to publish XSPF with portable song identifiers that could be resolved by companies that specialize in provisioning.”

It seems Lucas is thinking about how to get music flowing around the Internet with the same efficiency that text has enjoyed. Very smart.

The problem with being popular (part 2)

One of the more interesting sciences, in my mind, is how information relevance is both determined, surfaced and then evolved.

In Fred Wilson’s recent Cautionary Techmeme Tale he argues that making news popular takes away its social context and therefore becomes meaningless. He found Techmeme more useful when its sources more closely resembled his network of friends:

“For years, I’ve been using curators to filter my web experience…Techmeme has been the killer social media curator for my world of tech blogs. Lore has it that it was created using Scoble’s OPML file. It doesn’t matter to me if that’s true or not, I love that story. Because my OPML file was unusable until I found Techmeme and after that I stopped reading feeds and started reading curated feeds.”

This feeds into a larger argument about why pop culture and the art of being or becoming popular can be a bad thing. Not long ago I was inspired by the movie “Good Night and Good Luck” to dive into this idea myself:

“The real problem with popularity-driven models is that they reduce both the breadth and depth of the sources, topics and viewpoints being expressed across a community. Popularity-driven models water down the value in those hard-to-find nuggets. They normalize coverage and create new power structures that interesting things have to fight through.”

This is exactly why personalization, recommendations and social media technologies really matter. They can solve this problem of creating conformist media consumption practices by creating relevance through networks of people rather than through networks of commercial institutions.

I haven’t used My Yahoo! as much as I’d like, but there is a simple function in it that I love which could ultimately create amazing benefits for people who want a human filter for the Internet. It’s called “Top Picks”.

“The Top Picks module automatically highlights stories from your page, based on the articles you have recently read on My Yahoo! The more stories you click on, the more you will see this module reflect your interests.”

Actually, the technology beneath it is not so ‘simple’ but the application of it here makes so much sense that it feels like it’s simple when you watch it work. It works by using implicit behaviors. I don’t have to tell it what I like. It learns.

If it could also show me what my social network is tapped into right now, then the experience would feel nearly complete.

Media researchers will note here that people need pop culture to feel connected to a greater whole. I believe that’s true, too. Television is an amazingly powerful community builder.

But I would gladly trade a powerful singular social voice tied together by networks of distribution ownership for a less unified but still loosely connected network of pop culture tied together by my personal activities and my social connections.

Investing in video at YDN

We’ve been playing around with video as a communications mechanism on Yahoo! Developer Network for a while now. Our casual attempts to generate interest in Yahoo! technologies through interviews, screencasts, tech talks, etc. have worked really well.

So, we hired a full time videographer/filmmaker named Ricky Montalvo and got him some decent gear to push the envelope a little further. And today we rolled out YDN Theater on the YDN web site to establish a home for all the work he has been producing.

The journey here started with a pretty lame but surprisingly successful screencast that Dan Theurer and I did to explain how browser-based authentication worked. It was blurry. We made mistakes. The subject matter was pretty abstract. And neither Dan nor I have particularly strong camera presence.

Regardless, it has been viewed over 19,000 times, so far.

We kept pushing with new types of videos such as partner showcases with people like Joyce Park, Adam Rifkin, and Leah Culver. We brought the camera to our various Hack Days and produced a particularly funny recap of the London event. And we recorded tech talks from our own staff at Yahoo! and presentations from guest speakers like Grady Booch, Joe Hewitt and David Weinberger.

By the time we found Ricky, we knew we were building a program that was going to be really interesting. Yet, we hardly spent any money other than a few cheap cameras and some basic editing tools including Camtasia at that point.

The success to date I think has been in large part due to the fact that we haven’t tried to pimp out our videos with any professional plastic gloss or staged demos. We also try to have a little fun with them. Jeremy Zawodny is a really good interviewer. His unassuming yet pointed questions get people to say things they otherwise wouldn’t include on any planned script. And the fact that the videos are raw with few cuts or edits make them feel real, too.

There are some good video program ideas floating around here that could be a lot of fun, but now we’re torn between how much time we want to spend building out the video offering and how much time we want to spend on all the other ways the team can evangelize Yahoo! technologies.

I’m not sure how to measure that decision just yet, but as long as people are consuming these shows we do with such enthusiasm we’ll probably tilt the scale in favor of doing more video whenever possible.

Ad networks vs ad exchanges

I spent yesterday at the Right Media Open event in Half Moon Bay at the Ritz Carlton Hotel.


Right Media assembled an impressive list of executives and innovators including John Battelle of Federated Media, David Rosenblatt of DoubleClick, Scott Howe of Microsoft, entrepreneur Steve Jenkins, Jonathan Shapiro of MediaWhiz, Ellen Siminoff of Efficient Frontiers, and Yahoo!’s own Bill Wise and the Right Media team including Pat McCarthy to name a few.

It was an intimate gathering of maybe 120 people.

Much of the dialog at the event revolved around ad exchange market dynamics and how ad networks differ from exchanges. DoubleClick’s Roseblatt described the 2 as analagous to stock exchanges and hedge funds…there are a few large exchanges where everyone can participate and then there are many specialized networks that serve a particular market or customer segment. That seemed to resonate with people.

The day opened with a very candid dialog between Jerry Yang and IAB President Randall Rothenberg where Jerry talked about his approach to refocusing the company and his experiences at Yahoo! to date.

Battelle’s panel later in the afternoon was very engaging, as well. The respective leaders of the ad technology divisions at Yahoo! (Mike Walrath of Right Media), Miscrosoft (Scott Howe of Drivepm and Atlas) and Google (David Rosenblatt of DoubleClick) shared the stage and took questions from John who, as usual, didn’t hold back.

The panelists seemed to have similar approaches to the exchange market, though it seems clear that Right Media has a more mature approach, ironically due in large part to the company’s youth. Microsoft was touting its technology “arsenal”. And DoubleClick wasn’t afraid to admit that they were still testing the waters.

I also learned about an interesting market of middlemen that I didn’t know existed. For example, I spoke with a guy from a company called exeLate that serves as a user behavior data provider between a publisher and an exchange.

There were also ad services providers like Text Link Ads and publishers like Jim Mansfield’s PhoneZoo all discussing the tricky aspects of managing the mixture of inventory, rates and yield, relationships with ad networks, and the advantages of using exchanges.

I’ve been mostly out of touch with the ad technology world for too long.

Our advanced advertising technology experiments at InfoWorld such as behavioral targeting with Tacoda, O & O contextual targeting services like CheckM8, our own RSS advertising, lead generation and rich media experiences were under development about 3 years ago now.

This event was a great way to reacquaint myself with what’s going on out in the market starting at the top from the strategic business perspective. I knew ad exchanges were going to be hot when I learned about Right Media a year ago, but I’m even more bullish on the concept now.

The positive outlook for Yahoo!

I was able to contribute a quote to a BusinessWeek.com story by Catherine Holahan on changes happening at Yahoo!. Unfortunately, she also quotes from a really old blog post of mine (July 2006) which feels like a lifetime ago, but she gets the point right that things have improved here:

“Yahoo: Bringing Geeky Back“:

“Business is anything but usual under cofounder Yang, who’s trying to return Yahoo to its roots as a technology company … Matt McAlister, director of Yahoo’s Developer Network, said in an e-mail that he’s “never been more optimistic about the future of the company,” thanks in large part to these developer initiatives. “I think Yahoo is focusing on the right things and moving in a very positive direction,” he added. “

She talks a little about the ways the company is opening up, and she’s right that the developer initiatives are really coming along at Yahoo! (despite kind of getting open API thing wrong…API’s != “source codes“).

It feels like the beginning of something good here at Yahoo!.

The magic of Hack Day

Even after seeing Hack Day work internally a bunch of times and now twice in the open format with external developers I’m still amazed every time. There’s something magical about the event.


Yes, ok, obviously, if you attract a bunch of creative people, give them the right kind of stage to express themselves and an audience of peers to listen to them then you are going to be surprised one way or another. And if you add some rules that make it feel like a game, then the competitive spirit will naturally motivate people to reach further.

It seems obvious now, but it amazes me all the same. I’ve never really experienced anything quite like it in the workplace.

In many ways, the experience both as a host and as a participant at the event feels similar to the intensity and even panic you get working at a startup that suddenly feels like it’s turning the corner. Everyone is operating at full speed. You know you’re on to something hot. But you have no idea what will happen next…and you’re pretty sure that you will fail if you stop running at full speed.

It’s also a little bit like the mid point in the sport season when you’re on a team that might actually win the championship this year. You know you can do something great, but you have to focus and make it happen. Intuition takes control of every decision. There’s no time for analysis.

One of the powerful lessons of Buddhism is the idea of letting go of the things we want to control. It’s incredibly difficult to throw an event where the outcome is so completely unknown. I can’t tell you how much time and energy we spend trying to remove all the rules and controls and precedents that come with being at a high profile company like Yahoo! in order to run Hack Day the right way.

Asking people to understand the event without experiencing it is a tall order. There’s always a “this will never work” look on their faces while you tell them that we invite a bunch of people over to build stuff. And then comes the panic while the actual hacking phase of the event silences the socializing aspect from earlier in the day. It can be uncomfortable hosting a party for a bunch of people who aren’t talking to each other.

This long quiet period carries on into the night, essentially a hum of keyboards banging away and murmers amongst team members, where you start to wonder if any of the hacks will be any good or if there will be enough to present or if anybody even cares. Part of you also knows the magic is happening right now…brains are crackling and the creative fire is blazing amongst these people who are intently focused on their computer screens.

The demos are a good indicator of the success of the event, but reading the follow up posts out in the blogosphere brings it home for us as hosts. Like the bride and groom after a wedding, we’re never totally sure how well the event went until people tell us. Here are some of my favorite quotes about Hack Day London:

Josh Clark: Lightning! Blimps! Submarines! And, um, Machine Tags!
The event was thoroughly engaging and altogether humbling. The amount of know-how, creativity and sheer geekery in the room was overwhelming. It’s just plain exciting to be part of a profession and community whose frontiers are expanding so fast.

Neil Ford
All in all, it was a fantastic, if tiring, two days. The total number of hacks presented at the end was 73, all of which were of a superb standard. I think it was impossible to leave the event uninspired.

Ryan Morrison: Land of the Living
I loved the event, I had one of the best times of my life and it’s re-inspired my love of getting down and dirty with code.

It’s incredibly gratifying to know that people enjoy the event as much as we enjoy putting it on. It isn’t an easy event to run, and knowing that people get something real out of it makes it all worth it.

If you were there at Alexandra Palace at Hack Day, we would love to hear what you’d like to see next time. We’re always looking to improve it. We’re watching the blogosphere, so post away, or feel free to comment.

Photo: Andy Piper

The Hack Day London Video

I’m heading back home from Hack Day London tomorrow. What a spectacular event.

I did my best to capture the behind-the-scenes action this time, as I think the Hack Day event process itself is really interesting, too. Of course, sharing the day-to-day work would be frightengly boring, but you can at least get a sense of what happens the day or so before Hack Day starts in this video here:

What’s easy to forget is that the event process itself is treated like a hack. We break the rules. We invent on the fly. We don’t know if it will work.

Anyhow, there’s more to come, I’m sure.

(Apologies for the horrible editing in this video…it’s my own hack contribution…unpolished, experimental, and a little bit broken.)

The next Yahoo! Hack Day to be in London in June

This event should be really good. I’ve been working with the local Yahoo! team in London and their partner BBC Backstage to keep the spirit of the original Open Hack Day alive, and I think the teams there have done that and raised the game significantly. If you are in Europe and oversee Internet developers, then you need to get your guys to this event.

Event organizer Tom Coates posted some thoughts on why this event is so great and what happens there:

“It’s a two day event, starting first thing on Saturday morning and running through to Sunday evening. We’ll have a whole bunch of speakers from Flickr, Yahoo! and the BBC to start us off. We’ll have food—mostly flat—to meet the dedicated needs of our guests. There may be booze. I’m not telling. If you want, you can stay awake all night or crash out in a corner in a sleeping bag. The only requirement or restriction (except for the legal ones, which you should probably read) is that you come to the event and try and build something, ideally using some of the stuff that all the organisations hosting the event have to offer. Did I mention it was free?”

More details on hackday.org.

UPDATE: Here are a few mentions about the event, so far:

Matt Cashmore, event producer on BBC’s Backstage team
Ian Forrester, also a member of the Backstage team

links to hackday.org
    get this for your page