johnkeefe.net

Do you want to ...

  Inform the citizens of New York?

  Help people understand their world?

  Root out corruption?

  Make a mark on society?

  Craft beautiful online projects and visualizations?

  ( Like this diversity map, this stop & frisk project and this election tracker? )

WNYC is growing our Data News Team to make high-impact visualizations and projects, and to help WNYC reporters and producers present the facts, expose corruption and explain our world. We've been pioneers in the field of crowdsourcing, data journalism and mapping -- even winning some prestigious awards for our work.

Now we're kicking it up a notch. Like to join us? 

What we have:

• An award-wining staff of reporters and producers
• A committed, innovative digital staff
• A mission to conduct journalism in the public interest
• Millions of engaged, passionate and active listeners and readers

What you have:

• A passion for news
• An attention to detail, a respect for fairness and a hatred of inaccuracy
• A user-centered approach to exploring information
• An appreciation for clean lines, clear stories and use of white space
• A genuine and friendly disposition, and an honest spirit of collaboration
• A bias toward sharing what you know, and helping others build on it

What you'll do:

• Huddle with reporters to figure out how we might help their stories with data, design and web technology
• Work as a team to turn ideas into realities in days or weeks, tops
• Learn from and build on successes and mistakes along the way
• Have your work consumed online and talked about on air to millions of New Yorkers

Head over to our official aplication for Interaction Designer and tell us all about you.

If you need more proof that it's valuable for journalist-programmers to show their work, here's some: WNYC's Live New Jersey Election Map.

Exactly one week after Albert Sun of the Wall Street Journal New York Times shared some of his work, we made this:

(Map isn't embeddable for licensing reasons; the live version is here.)

Here's what happened.

Last month I went to a Hacks/Hackers NYC meetup about mapping. There, Albert showed his WSJ Census Map Maker project and a map I had admired that has dynamic mouse-overs without using Flash. At one point, he showed his project's code repository and welcomed us to use and build on it.

The next day, I downloaded the code and tried to make a rough version of Albert's map, but using the shapes of New Jersey legislative disricts (downladed from the US Census, stored in this Fusion Table, which generates this KML file). After a little tinkering -- which includes a fix I've described in the comments below -- I managed to build one that works. I sent that to stellar coder Jonathan Soma, of Balance Media, who works with me to build interactives for WNYC.

I also reached out to Al Shaw, of ProPublica, who I knew (from another Hacks/Hackers Meetup) had wrestled with live Associated Press election data for Talking Points Memo. He had some great tips, which I passed along to Soma, too.

Also on the case were Balance's Kate Reyes and Adda Birnir, who crafted the map's design and user experience -- a particularly tricky task because each district elects one person for state senate and two people for state assembly.

A week later, as the results rolled in, WNYC's map was live and rockin' -- listing real-time returns for each district, and changing colors when races were called.

In the process, Soma built on Albert's work, and those modifications are now a part of the code base (see Github commits here and here).

And if you need proof that such work is valuable, the map was WNYC's No. 6 traffic-getter for the month -- despite the fact it was truly useful for about 4 hours late on the evening of an off-year election.

In September, I shared how WNYC makes news maps during a talk at the the Online News Association conference.

UPDATE: ONA posted a video of this presentation, which I've embedded here:

'Once Upon a Datum': Telling Visual Stories from Online News Association on Vimeo.

Here are my presentation slides (PDF), and here's a list of links to pages and sites I discussed in my talk:

Same-Sex Couples in NYC: http://www.wnyc.org/articles/wnyc-news/2011/jul/14/census-shows-rising-number-gay-couples-and-dominicans/

Hispanic Origins in NYC: http://www.wnyc.org/articles/wnyc-news/2011/jul/14/census-shows-rising-number-gay-couples-and-dominicans/

The New Littles: http://www.wnyc.org/shows/bl/clusters/2011/jun/02/june-guest-andrew-beveridge-and-new-littles/

Marijuana Arrests: http://www.wnyc.org/articles/wnyc-news/2011/apr/27/alleged-illegal-searches/

Contributions by Zip Code: http://empire.wnyc.org/2011/07/where-are-the-mayoral-candidates-raising-their-money/

Dollars in a District: http://empire.wnyc.org/2011/09/the-54th-assembly-campaign-contribution-breakdown-where-have-all-the-in-district-donors-gone/

NYC Hurricane Evacuation Map: http://wny.cc/EvacZones

NYC DataMine: http://www.nyc.gov/html/datamine/html/data/geographic.shtml

Shpescape: http://www.shpescape.com/

Hurricane Zones Fusion Table: http://www.google.com/fusiontables/DataSource?dsrcid=964884

Fusion Tables Layer Builder: http://j.mp/FusionBuilder or http://gmaps-samples.googlecode.com/svn/trunk/fusiontables/fusiontableslayer_builder.html

Layer-wizard map from presentation: http://dl.dropbox.com/u/466610/preso-map.html

 

A couple of years ago, I had our WNYC engineers use a plotter to print out this huge evacuation map PDF. Seemed like a good thing for the disaster-planning file. Just in case.

Then, back in June of this year, I was browsing the NYC DataMine (like you do), and realized New York City had posted a shapefile for the colored zones on that map.

UPDATE (Feb. 11, 2012): NYC has nicely revamped the DataMine since the summer Irene struck -- even mapping geographic files like this right in the browser. But it's actually tricker to find the shapefiles now. Here's the hurricane zones dataset. Click "About" and scroll down to "Attachments" for the .zip file containing the shapefiles. Or just use this shortcut.

I knew I could use the shapefile to make a zoomable Google map -- which would be a heckuvalot easier to use than the PDF. So I imported the shapefile into a Google fusion table. (It's super easy to do; check out this step-by-step guide.) Next, I added that table as a layer in a Google Map and tacked on an address finder I'd developed for WNYC's census maps.

Then I tucked the code away on my computer. Just in case.

Fast-forward to Thursday morning, as Irene approached. On the subway in to work, I polished the map and added a color key. It was up on WNYC.org by midmorning, long before the Mayor ordered an evacuation of Zone A.

When the order was announced, I used another fusion table to add evacuation center locations, updating that list with info from New York City's Chief Digital Officer Rachel Sterne. (The dots are gone now, since the sites are closed.)

I'm not at liberty to reveal traffic numbers, but the site where we host our maps received, um, a lot more views than it usually does. By orders of magnitude. Huge props to the WNYC.org digital team for keeping the servers alive.

As Hurricane Irene was approaching Puerto Rico, I noticed that the National Hurricane Center posts mapping layers for each element in their storm-track forecast maps.

Since their zoomable maps aren't embeddable, I made one that is. Feel free to use it:

Right now, I'm manually updating the map with new layers as they are issued, every three hours. I'm pretty close to having a script ready to handle that for me, based on information in the storm's RSS feed.

In the process of building this map, I learned how to use "listeners" to dictate the order the layers are rendered. For anyone trying to work that out, here's the code for how I did it.

When the New York census numbers arrived this week, we were ready. WNYC quickly published an interactive, sharable map so New Yorkers -- and our reporters -- could explore the new data and see the stories.

We built the map with free tools and timely help from some smart, kind people.

<p>scrolling="no">

The short story is that we mashed together population numbers and geographic shapes using Google Fusion Tables, and then used JavaScript and Fusion Tables' mapping features to make things pretty and interactive.

The long story is meandering and full of wrong turns. But here are the highlights, should anyone need a little navigation. Don't hesitate to contact me for more help and insight; I'm due to pay some forward.

Getting in Shape

First up: Shapes of the census tracts plotted on Earth. I downloaded New York's tracts from the U.S. Census Bureau's TIGER/Line Shapefile page. They also have counties, blocks, zip codes, and more.

Then I uploaded this "shapefile" -- actually collection of related files zipped together -- to Fusion Tables with a free, online tool called Shpescape. (Thanks to Google's Rebecca Shapley for sharing this key to my puzzle.)

Hello, Data

Census data is publicly available, but can be a hassle to handle. In fact, on the day each state's info was released, the files were available in a set that apparently requires one of two pricey programs -- SAS or Microsoft Access -- to assemble. 

So I got clean, assembled, comma-delimited files -- complete with 2000-to-2010 comparisons -- from the USA Today census team, which provided them as a courtesy to members of Investigative Reporters and Editors. Huge props to Anthony DeBarros and Paul Overberg, who crunched the New York numbers in a blazing 30 minutes.

By the way, IRE membership is $60 for professionals and $25 for students. Well worth it, and cheaper than either of those programs. If you're digging into census numbers and qualify, I recommend this route.

That said, every state's 2010 data is now available free from the Census Bureau's American Fact Finder. Navigating the site is a little tricky, and worth a separate post, but the bureau provides some tutorials, and there's very detailed PDF about each data field.

With data in hand, I uploaded it to Google Fusion Tables in another table.

Map Making

Next, I merged the shapes table and population table, using the unique tract ID to marry the data (the shape file calls it GEOID10. the IRE data calls it FULLTRACT). Note that the GEOID10 is formatted differently depending on whether you're using tracts, blocks, counties, etc., so be sure you've got the right match in both files.

Clicking Visualize -> Map shows a map. It'll be all default-red until you click on Configure Styles -> Fill Colors -> Gradient (or ->Buckets) and make different colors appear depending on values in the column of your choice.

Using the Share button makes the map viewable by others, and "Get embeddable link" does just that.

Adding Prettiness

I used the Fusion Tables "Configure info window" option to make custom pop-up bubbles on our maps. This actually required some nicer-looking data, such as a columns with rounded percentages and + or – signs. I added these using the free R statistical program, which I learned how to do from The New York Times' Amanda Cox at the 2011 Computer Assisted Reporting conference.)

Census tracts officially extend to the state lines, which made it look like a lot of people live in the Hudson River. So I had trimmed those tracts to the shorelines with a free mapping program called QGIS, using water shapefiles as a reference (those are here, in the drop-down menu).

After creating 12 merged Fusion Tables, I pulled them into one page using JavaScript and jQuery, with fantastic guidance from Joe Germuska at the Chicago Tribune (part of the team that built this great map).

The "Share/Embed this view" feature came together in two parts: 1) The JavaScript turns the current the latitude, longitude, zoom level and current map choice into a long URL that pops up when you click the Share/Embed link. 2) Using a nifty jQuery plug-in (updated link Dec. 2011), the map looks for those values in the URL that summoned it, and reorients to that map if they exist.

Prep Work

Clearly, not all of this could happen in a couple of hours on Data Day. I'd been tinkering, testing and tweaking for a few weeks using New Jersey's data, which came out much earlier.

I also wrote down, edited and revised every step I took to make the maps. So when the adrenaline was running I had a script to follow.

The WNYC Web Team also set up a slick, fresh project server, at project.wnyc.org, to host the html pages and track the traffic.

Fusion Function

Using Google Fusion Tables made it super easy to manage, map and serve up a lot of data. And the FT feedback team was fantastic about responding to questions and glitches I encountered along the way.

I did run into a couple of hiccups: slow load times and pop-up bubbles that failed to pop up. The first was a product of displaying so much data -- and I knew I was pushing things. The second was a Google glitch that their engineers managed to fix within a few hours, but was still spotty at times afterward.

Also, the Google Map engine starts dropping shapes when there are too many to show. So I funneled different counties' data into almost a dozen different layers, a workaround the Google folks showed me ahead of time.

That said, I had time to code and tweak lots of neat things because I didn't have to focus on building or running a database engine. Google's free services took care of that.

What Could Be Better?

Probably a lot. I wanted to let people to add comments, right on the map, but didn't have the chops or time to pull that off.

Another good thing would be a "Loading ..." indicator displayed while the map data is pulled into your browser, which I may yet add.

But what couldn't have been better was everything I learned, the help I got from other data folks and the support from my WNYC colleagues. Plus we gave New Yorkers a pretty nifty service and several great stories.

Need more details? Feel free to ask questions in the comments. Or drop me a line. I'll try to help, too.

We did something a little creative and unique at WNYC this past election night: We mapped the vote by "community type."

This revealed the diversity of the vote across New York State -- from the cities to the suburbs, boom towns and "service worker" centers -- in real time, on the air and on the WNYC home page.

And the diversity is striking. Despite Democratic wins in every statewide race, the Republicans running for state attorney general and comptroller "won" every community type outside "Industrial Metropolis" and "Campus & Careers" counties.

Patchwork Nation's Dante Chinni talked about this on air during WNYC's coverage election night, and has written more about it since.

The live map was a mashup of Patchwork Nation's unique take on the nation and the Associated Press's live vote totals. At the request of WNYC, Patchwork Nation programmers dove into the AP test results and quickly wove them into a new map based on PN's existing county maps -- customizing them for the event and adding real-time percentages by community type.

Bringing the Threads Together

In the months before the election, I had wondered how we might better understand the early returns -- those that come in typically between 9 p.m. and 10:30 p.m. -- which often don't match the final results. I wanted more clarity.

At a Hacks/Hackers Open-Source-a-Thon, I started playing with the election data with help from Al Shaw (then at TalkingPointsMemo, now at ProPublica) and Chrys Wu (of Hacks/Hackers and ONA fame).

That evolved into a little program I wrote in Sinatra that generated vote-total map at the left, shading counties darker as more of their precincts reported. It also helped me better understand how the data were structured, how to retrieve the numbers and what it might take to make a live map.

So when Chinni asked if WNYC had any county-level data sets we'd like to put through the Patchwork Nation treatment, I had the perfect candidate.

Consider, for a moment, the location of your nearest fire extinguisher.

And just how well did you use it last time?

Right. Chances are you're not prepared to skillfully put out a fire where you are sitting. At least you're not practiced.

But what if you had a non-emergency reason to use a fire extinguisher every once in a while? Maybe to clean your desk. Or a spill. (Nearest paper towel, anyone?) Using it occasionally would help insure that in case of a fire, you would both a) actually use it and b) use it well.

To better prepare our on-air and online operations for major breaking news, I've been promoting a point of view that says we shouldn't put our emergency tools, systems and skills "behind glass." Instead, we should incorporate those efforts into our everyday work. (I even believe we shouldn't put our energy into efforts that can't be used on a regular basis because in a crisis, we won't use them anyway.)

The best example of this is the daily production of our national morning program, The Takeaway with John Hockenberry and Adaora Udoji. The show's staff may just be the best breaking-news response team in public radio -- because they make the show in real-time every day, incorporating fresh news as they go. When the news happens to be really big, they're not just prepared ... they're already there.

A few other examples:

Map maker, map maker
In case of a civil emergency in New York, we'd want to quickly map shelters, closed roads, danger zones, escape routes. Even locate our staff. But we weren't prepared to whip together those kinds of maps in mere minutes. Now we're honing those skills by incorporating such work into everyday projects.

Information integration
When news hits the fan, information flies everywhere. Consolidating that data is key ... and also happens to be handy in everyday work. In the course of discussing a Mumbai-like terror attack in NYC, we discovered that our news-editing software can also check a listener email box. That's one less window to watch.

Nobody move
We designed our newsroom so that in a crisis nobody needs to change seats, which would move them away from familiar surroundings. As a byproduct, when something doesn't flow quite right during daily work, I try to make sure we address it now so we don't get caught off guard later.

Expected events as prototypes
In planning for election night, and now for the inauguration, we developed tools and techniques that will serve us again in a major unexpected event. We now know how to quickly rip up our station's home page to focus on a single topic. And in order to provide real-time election-night returns, we found new ways to clear the information path between the editors and the home page.

Share and share again
We share a statehouse reporter with stations across our region. So when former governor Eliot Spitzer imploded in a prostitution scandal, we didn't have to think twice about how to move information and audio between stations. We just used the FTP site and email list we use every day.

I've long been a fan of drills, and there are many more of those in our future. But by incorporating a little drill into our regular routine, we're better prepared for situations that are anything but.

[Photo by IamSAM. Some rights reserved.]

Two hours before our company holiday party, several of us were contemplating mass murder in Manhattan.

Not exactly Christmas cheer. But some good certainly came of it.

Really.

Several top thinkers and decision-makers reviewed the horrible events in Mumbai last month and considered how our news department would respond to a low-tech, coordinated attack on multiple locations in New York.

Here are some of our points of discussion, which could apply to any organization, journalistic or otherwise:

The Visioning Thing. We didn't do a full-scale drill, we simply took time to really visualize how things might happen. It was pretty powerful.

First we tried to think like terrorists and, as a group, picked three targets -- a transportation hub, a hotel and a shopping center. No sense naming them here; suffice it to say, we all knew each of them well and could picture the devastation and chaos.

Then we carefully imagined where each of our key people would be on a weeknight at 9:30, when Mumbai's night of hell began. Who lives where? Who's still at work? Who could get in fastest? What route would they take? Where would the first available reporters go? How would they stay safe? Think.

Our Civic Duty. We're journalists. It's second nature to pursue the facts and try to present them quickly, accurately in context. But as a broadcaster in a city under siege, our public service mission takes on new qualities, and raises questions. What do we do for people still in or near danger? Can we be better oriented to provide public warnings, safety and health info, comfort, maps, conversation, rumor control ... help? The conversation has started to adjust our operating Point of View, and could make a huge difference in how we serve our city (lowercase c) in those first few hours.

Information Overload. Phone calls, reporters, sources, Tweets, network audio, news wires, emails, web comments, TVs ... we easily came up with more than 20 distinct streams of audio, text, and visual information key to covering the story. In an era when all of this information is available to everyone on our staff, are we ready to monitor them all in a sophisticated, organized way? (Ah, no.)

Online, Under Pressure. Our methods for broadcasting have changed since 9/11 and The Northeast Blackout. We now use web alerts, social media, maps, and other tools to convey information. But when the adrenaline pumps, and minutes matter, we have to be ready to take advantage of all of these channels while maintaining our standards of accuracy and context.

Bias Toward Action. It's been 7 days since this meeting, and we're far better prepared than we were 8 days ago. What we've done, and are doing, is the subject of my next post, Break Glass Now.

[Photo by Andy Eakin]

My iPhone is my new flash recorder.

As a manager, I'm not often out collecting sound for air. But I've been carrying a Nagra flash recorder just in case I need to contribute in a crisis, happen upon breaking news, or want to capture an aural moment we might use.

Today, I gave the Nagra back so someone else can use it.

That's because Adam Hirsch, a producer on The Takeaway and a fellow iPhone geek, showed me an impressive, new (and currently free) iPhone app that records fantastic audio using the phone's built-in microphone. It's far better than any apps I've tried for this purpose, and good enough to impress our engineers.

It's called the iTalk Recorder, from Griffin. The recordings really do sound great. At the top setting, it makes CD-strength AIFF files at 44.1k and 16 bits. If you have a Mac, a nifty download allows you to transfer the audio from phone to computer over wifi.

It certainly won't replace our reporters' professional recording equipment. But in a pinch, or as a backup kit, it's fantastic.

For me, that's perfect. And one less thing carry.