Interesting perspectives from Web 2.0 Expo

Today’s Web 2.0 Expo in San Francisco provided some really good brain food.

Clay Shirky’s keynote was excellent. He talked about architecting a new world for the “cognitive surplus” that’s emerging as people pull themselves out of the historical sitcom hangover and invest their energy online. Matt Jones and Tom Coates shared some neat ideas on design for personal infomatics. And Twitter’s Alex Payne and Michael Migurski of Stamen Design presented learnings from the perspective of an API provider.

One little nugget I really liked was a minor point Migurski made when talking through the Oakland Crimespotting service. He noted that there are several standard formats commonly provided by most web services including HTML, JSON, serliazed PHP, RSS and XML.

But we often forget about simple Excel spreadsheets.

He showed how the Oakland Crimespotting site offers downloadable Excel spreadsheets detailing recent activity from particular police beats, for example.

One of the keys to opening up government data is making the case to the people who are best equipped to provide raw data that it needs to be posted directly to the Internet. Telling them they need to output JSON for data visualizations and mashups will do as much good as a slap in the face. Showing them a regularly updating Excel spreadsheet that is findable on a web page that they can email to their colleagues, friends and families is going to get them thinking differently and perhaps encourage their participation directly.

The crime data issue is going to be a big deal in the not too distant future, I’m sure. And as Mr. Coates and Mr. Jones noted in their talks on personal data design, it’s the details that really matter in this space. You can think about products and features all day, but the specifics that define how data is shared, how it becomes relevant and how it is presented will make or break the intent of any offering.

Designing Your API, Web 2.0 Expo 2008:

Interactive journalism: An amazing homicide mashup

I had the pleasure of interviewing Sean Connelly and Katy Newton for YDN Theater recently with YDN videographer Ricky Montalvo. They created the amazing (and award-winning) crime data mashup Not Just A Number in partnership with The Oakland Tribune.

Not Just A NumberAfter getting tired of watching the homicide count for 2006 climb higher and higher, they decided to humanize the issue and talk to the families of the victims directly. They wanted to expose the story beneath the number and give a platform upon which the community could make the issue real.

Statistics can tell effective stories, but death and loss reach emotional depths beyond the power of any numerical exploration.

Sean and Katy posted recordings of the families talking about the sons, daughters, sisters and brothers that they lost. They integrated family photos, message boards, articles and more along with the interactive homicide map on the site to round out the experience making it much more human than the traditional crime data mashup.

Here is the video (7 min.):

I also asked them if they had trouble getting data to make the site, and they said the Oakland Tribune staff were very supportive. There weren’t any usable open data sets coming out of the city, so they had to collect and enter everything themselves.

This, of course, is a very manual process. Given the challenge of getting the data Sean and Katy didn’t see how the idea could possibly scale outside of the city of Oakland.

SOmebody needs to take that on as a challenge.

I’m hopeful that efforts like Not Just A Number and the Open Government Data organization will be able to surface why it’s important for our government to open up access to the many data repositories they hold. And if the government won’t do it, then it should be the job of journalists and media companies to surface government data so that people can use it in meaningful ways.

This is a great example of how the Internet can empower people who otherwise have no voice or audience despite having profound stories to tell.

Making government more useful through data

A very interesting working group formed recently to drive better transparency in government through data. The Open Government Data organization has a simple aim:

“The group is offering a set of fundamental principles for open government data. By embracing the eight principles, governments of the world can become more effective, transparent, and relevant to our lives.”

They proposed that data will be considered open if it complies with the following qualifications:

1. Complete
2. Primary
3. Timely
4. Accessible
5. Machine processable
6. Non-discriminatory
7. Non-proprietary
8. License-free

This is a promising approach to driving high impact changes in the way government serves its people. Giving everyone greater access to relevant information that they already own is a noble pursuit.

I’ve explored this a little myself in some investigations of access to crime data [1, 2].

It’s no surprise that Adrian Holovaty of the Chicago Crime mashup fame (and now Every Block) is one of the founding members. Of course, there’s no better advocate for the free flow of information than Lawrence Lessig. And Tim O’Reilly will be a strong foundational force here. I’d love to see Jon Udell join, too, as his work has inspired a lot of people (myself included) to think differently about exposing and sharing data like this.

Good luck, guys!

Oakland Trib’s Not-Just-A-Number improves on crime data visualization

OJR’s Jim Wayne dives into Oakland Tribune’s “Not Just A Number” web site. The service won the Service Journalism Award from ONA for an amazingly powerful view of crime data.

The basic premise was to create a data visualization for Oakland homicide crime data that made the victims and, more importantly, the people in their lives real participants in the story rather than pure statistics (or just plain ignored entirely).

It’s a very powerful site and a model for all local newspapers to follow. It’s disappointing but no surprise the media creates these kinds of community services before local governments do. At least we’re getting more access to crime data.

Wayne also points to a crime data visualization from the Los Angeles Times called The Homicide Map that I wasn’t aware of.

They have a nice map mashup that takes a more statistical approach, yet they also include things like images of the victims.

Unfortunately, as Oakland Tribune producers Katy Newton and Sean Connelley point out, a mug shot is not a fair image to use for a violent crime victim in a statistical map. But I’m glad to see them exposing data that needs to be shared.

732 homicides in Los Angeles so far in 2007! Unbelievable.

Crime data stories

My Potrero Hill neighbors tell me that the sweet song of crackling firearms in the evening always begins again in May as the days get longer, hotter and schoolless.

Recently, I witnessed a sample of the gun play happening in the nearby projects, and I decided to do some of my own research to understand what’s going on. The first thing I found was that I wasn’t the only witness to this particular incident:

“Two of the bullets hit our daughters bedroom– one went through the wall and crossed a small portion of the room and lodged in another wall near her sliding glass door.

[The Police] told us that based on the 24 bullet shells they found up the hill on Missouri St. near the public housing, there were two guns involved, one of which was an AK47 the other was probably a 9mm pistol. The police have no idea who was firing the guns and given that there are not witnesses, there is not likely to be any resolution to the incident. The officers were confident that the two bullets that hit the condo were random and not targeted at us.”

There are lots of factors behind violent neghborhoods, and the San Francisco projects are pretty densely representative of many of those factors. But it really irritates me that guns are so prevalent in the area, and, in general, so prevalent in America.

So, I started my journey at the old PotreroHillSF Crime Mashup which apparently doesn’t work any more. There is an ongoing “Police Blotter” on the site, though, with some good reporting.


I then found the official San Francisco Police Department Crime Map. Of course, the data is wrapped in their own heavy-handed user interface and unavailable in common shareable web data formats. The tool is burdened with legal trappings and strangely fails to acknowledge homicides, though they offer an explanation:

“A homicide may not appear correctly on the map because:

  1. The incident was initially reported as an assault and the victim died some time later from the injuries.
  2. The incident was reported as an arson, and the body was not found until a later time.
  3. A body was found and the cause of death was not obvious to the officer making the incident report.”

I’m hoping that the City has more advanced reporting capabilities internally, as it seems pretty obvious that we have a data visualization failure going on here. I can see some data around assaults, robberies, larceny, vandalism, drug incidents, etc.

But the compelling visual storytelling is missing.

I want to know how many crime incidents in the projects this year involved guns. How many guns in these events are registered/unregistered? How many of the gun incidents were or became homicides vs non-gun related incidents? Where did the guns come from? What kinds are being used?

I suspect most guns aren’t registered which is an argument used by those who think a gun ban would be useless. People who want guns will find them, legal or not. But I also suspect that the victims aren’t carrying guns. Thus, the argument that people should have the right to own a gun to protect themselves isn’t a counterbalancing force. People who avoid violence won’t carry guns, legal or not.

As I progressed with this research I realized that somewhere in between raw data and overt campaigning is an interesting space. Data can help us learn and make more intelligent and informed decisions about how to manage and evolve our society and its rules.

Unfortunately, that space seems more difficult to find than it should be. I should be able to download data for myself or at least be able to visualize the stories behind the data in relevant pictures and charts.

Of course, there’s the fantastic ChicagoCrime.org web site which has done a lot to raise awareness about crime data. Despite the lack of available data from the local government, site owner Andrian Holovaty found a way to collect what he needed to make this site through an automated script:

“Each weekday, my computer program goes to the Chicago Police Department’s website and gathers all crimes reported in Chicago.”

The site has some great info (such as this screenshot of “Armed Robbery: Handgun Incidents”), though I still want to see an editorial lens on this data that puts a bit more meaning behind it.

For example, it only takes a glance to see in this series of Census images of San Francisco that the City is incredibly segregated, something I think many residents choose to ignore under the mask of open-mindedness. Even here, though, the story is incomplete without some intelligence wrapped around the data. What’s the trend? Is it becoming whiter? Where are people going who are leaving?

This same question punctures my happy place every time I exit onto Palo Alto’s University Avenue from Highway 101 and pass what is now a high end office park where one of the most dangerous areas in the country used to exist only a decade or so ago. I’m very pleased it’s a safer place, but do we understand the cost of that transition? Where did those people go? Are they better off?

Yahoo! colleague Micah Laaker pointed me to an interesting project he worked on back in 2002 and 2003 called the Denver Census Tract Animation Project. He worked with Citizen Mapmakers to trend movement of the African-American population in Denver from 1960 to 2000. Here’s a snapshot of their work:

I really like the way they visualized data to tell a story here. We need similar visualizations for crime data.

The InfoPlease “School Shootings” site gets closer to telling a story about guns just by focusing on a type of statistic and representing it. What a powerful domain name! However, the data here is still pretty raw and limited. This is hugely important information, but there’s an implicit argument here that should be made much more explicit with actionable information and analysis. In its current state it’s just telling us that there are a lot of school shootings (a surprising number in Europe, actually).


The Citizen Crime Watch site for New Orleans gets even closer to what I want to see. Similar to ChicagoCrime.org, they visualize with your standard data-on-a-map mashup, but the hover links point to coverage in the local media. I’m suddenly given a much more human window into the crime scene, and I can read about each event. For example, on April 9, 2007, there was a homicide in a trailer park:

“…Officers found Williams lying on the floor of the trailer with blunt-force trauma to her head. Emergency medical technicians declared her dead at the scene. An autopsy shows she had been beaten to death, said John Gagliano, chief investigator for the Orleans Parish coroner’s office.

The trailer is in a trailer park at 6801 Press Drive run by the Federal Emergency Management Agency. Although the trailer park is near the campus of Southern University, the chancellor, Victor Ukpolo, said neither faculty nor students live there.

The murder is being investigated by Detective Harold Wischan, who can be reached at (504) 658-5300.”

I’m very thankful for local reporting from sites like Nola.com, The Times Picayune, and community leaders such as Mike Lin of PotreroHillSF and the increasingly active Yahoo! Group Potrero Hill Parents Association who all help surface this kind of information, but it’s not enough. The City needs make it easier for its residents to both report on things that matter to us and to collect the data, filter it, and act on it.

People will always want greater access to information. This is particularly true in communities where poor decision-making creates mistrust:

“Under pressure from constituents who say New Orleans police stonewall requests for crime data, the City Council’s criminal justice subcommittee took police representatives to task Wednesday, calling for a faster, freer flow of public information…When asked for a written breakdown of policy and procedures relating to the release of public information, Maj. Michael Sauter, the head of technology, told the council most of that information was ‘not meant for the public.'”

Similarly, Rick Klau has begun experimenting with this kind of thing in response to the Magnetix toy recall incident. He calls it “Open source parenting” and observes that bottom-up community-driven politics is likely to be more successful than anything a politician can enable:

“If the government is under-staffed and under-funded to help parents avoid harmful toys, then why can’t we help ourselves?…Give thousands of parents the tools to easily identify harmful products, leverage the community’s ability to provide visibility to legitimate threats while minimizing less serious risks, and quickly disseminate information that could be instrumental in avoiding a serious accident.”

I’m suddenly wondering what role politicans will play if communities are able to form solutions to issues locally, nationally and internationally on their own. Maybe instead of legislators (or merely professional campaigners/marketers), politicians will become community managers.

I also start wondering what politicians do all day if they can’t sort out ways to curb violence in our neighborhoods. I don’t see why anyone living in this country or any other should have to worry about whether their child will be shot accidentally in his or her bedroom by stray AK47 bullets or intentionally while at school.

I’m convinced the answer is in the data that is already being collected in various government crime databases. And I’m sure the answer is related to gun access.

Where is Tufte when you need him?