When New York City released real-time bus info for a Brooklyn line, one of my colleagues wasn't happy.
"I want a phone number that will TELL me when the next bus is coming," she said.
I'll have it for you by the end of the day, I replied.
It was a bit of a gamble. More of a challenge to myself, really. But if "before midnight" counts as the end of the day, then I succeeded. Try it:
Dial 646-480-7193. When prompted, enter 308333 or any of the bus stop codes for the B63 line.
It's not journalism, but it is a working example of how someone can take public data and turn it into a useful tool, quickly. And at almost no cost.
How I Did It
First, I wrote a little program in Sinatra that sends a 6-digit bus stop code to the NYC Metropolitan Transit Authority API -- or application programming interface -- whenever someone hits my program's web address. (The API is a public portal to the live bus data. All you need is a little programmer know-how and a free key from the MTA. The technical details are right here.)
The API sends back 77 lines of information about the stop and the buses approaching it, including the one detail I want:
The next bus is three stops away. I use a Ruby tool called Nokogiri to find and extract just this number, which I drop into an amazingly simple web page. The page's entire output looks like this:
The next bus to arrive at 14th Street and Fifth Avenue heading north is 3 stops away.
That accomplished, I bought a phone number from Twilio for $1 a month, and $0.01/call. Twilio provides a telephone connection to web-based programs, which I first heard about during a demo at a TimesOpen event.
I set my new phone number to hit my program's URL whenever someone calls. By wrapping the text in special tags, Twilio recognizes it as a cue to talk:
The next bus to arrive at 14th Street and Fifth Avenue heading north is 3 stops away.
I later used Twilio's <Gather> tags to build what's essentially a web form to capture digits entered on a phone for any bus stop. I also added error catching, for when no buses are coming, and programmed it to announce the location of the following bus if the next bus is arriving.
Turns out that Twilio reads text a little too fast to be understood on an NYC street corner. So I rewrote the output to introduce pauses:
The next bus to arrive at ... 14th Street ... and ... Fifth Avenue ... heading ... north ... is ... 3 ... stops ... away.
Also, there's a bug in the API system that sends back server errors in certain conditions. Word is that the MTA has actually fixed this on their development servers, and that fix is being pushed to the public system pretty soon.
More to Come
I have a few enhancements up my sleeve, which should be done in a week or two. If you'd like to know when new tricks roll out, drop a note with "Bus Talk" in the subject line to john (at) johnkeefe.net. I'd love to hear your thoughts on how it could work better, too.
I'll also release the source code after those nifty updates. Let me know, too, if you're interested in that.
Photo by jbrau13 on flickr.
For 26 hours this weekend, a bunch of journalists and coders got together to make lots of great things designed to help the citizens of New York City.My blog post summarizing the event and all of the resulting projects is upon Hacks/Hackers.I helped Jenny 8. Lee, Chrys Wu and Stephanie Pereira organize the event. Then I joined a team working with digital heaps of NYC taxi trip data to make data visualizations and start some other projects. My favorite one is here (with a detail below), which is a representation of taxi usage for 24 hours, set around a clock. Beautiful. It was built by Zoe Fraade-Blanar using Processing and data crunched by the other teammates.
|Click image for full view|
I came away from the event with many new connections, excitement about learning Processing, some more skills in Sinatra and a note to check out Bees with Machine Guns(!)
My presentation before a room full of talented programmers next week Thursday will include hair salons, semi-trailer trucks and sidewalk chalk.It's TimesOpen 2.0, where digital tinkerers gather to talk about online data from the New York Times and the latest trends in information technology.Next Thursday is Mobile/Geolocation night (which is free). Presenters will include Mano Marks from Google, John Britton from Twilio, Matt Kelly from Facebook and me. I'll be talking about The Takeaway's Sourcing Through Texting project.My preso will likely be the lowest-tech of the bunch. Our aim was, and is, to connect journalists and citizen-sources using basic text messages, and our method was brainstorming, learning and prototyping in two neighborhoods -- Southwest Detroit and Miami's Little Haiti. We absorbed a ton. (And we sparked an investigative series on illegal truck traffic.)There are certainly opportunities here to mash up APIs and build on some nifty platforms. I'll talk about that, too. But as we continue working toward connecting with sources via texting, some of our best insights have come from coffee shop conversations, church bulletin announcements and short-codes scrawled on sidewalks.
The answers are still emerging. We're still making prototypes. Yet, yesterday the concept won a Knight-Batten Special Distinction Award for innovation in journalism. Since the award application went in, we've gone to Miami to run another experiment in Little Haiti, and Detroit's WDET aired a week-long series that evolved from the project.That the award effectively predates those happenings is a huge jolt of support for experimentation, design thinking in journalism and everyone who contributed to this unique collaboration.That includes folks from The Takeaway, Public Radio International, WNYC Radio, WDET Detroit, WLRN Miami, The Miami Herald, American Public Media's Public Insight Network, Mobile Commons, the Institute of Design at Stanford and the residents of Southwest Detroit and Miami's Little Haiti.---Sourcing Through Texting is a project of The Takeaway, which is produced by WNYC Radio and Public Radio International. It was made possible by a grant from the John S. and James L. Knight Foundation. Disclosure and disclaimer: I helped develop and produce this project. As always, the words here are my own and not those of my employer or any of the entities mentioned.
Today our education reporter had a bunch of data from New York State she was trying to match to schools in New York City. But the school codes used by the two governments look radically different.For example, PS 15 on the Lower East Side is known to the state as 310100010015; the city calls it 01M015.I once made a nifty formula to make the conversion(!), but a more straightforward and official approach involves the Excel spreadsheet found here. It lists all of the city schools, along with their addresses, various codes, and more. For a data cruncher, that's a secret decoder ring.What made me smile was that the only reason I knew this document even existed was because of a little prototype I tried during the first swine flu outbreak. That experiment wasn't robust enough to make it beyond this blog, but it taught me a lot ... including where to find this ring!
I've joined an exclusive club of New York City microbloggers.Twitter? Ha. Facebook? Kidstuff. We're talking bricks and mortar, baby.Quite literally.My missives scroll across the facade of WNYC's building in west SoHo, zipping into your field of view as a parade of little red lights. It's the WNYC News Zipper.As you walk to work or sit in traffic on Varick Street, I've got your eyeballs.
NY terror-plot suspects indictedNone of this 140-character stuff. Better to use just five words; seven max. (I used a nonessential adjective clause once. Lost everyone by the second comma.)
Media banned from covering Iran protestsAnd I know where you are, no fancy GPS required.
Building collapse on Reade Street, up aheadEven if it's partly cloudy in the Bronx, I am absolutely certain you're in a downpour.
This rain ends by eveningUser customization? Easy. I can sense you're in line for the Holland Tunnel on your evening commute home. So how about a little news about your governor and his chief rival?
Corzine, Christie speak to biz group toniteIt's tempting to simply repurpose our tweets or web headlines, feeding them automatically to the sign. But it's also clear that wouldn't be as special. Or impactful. Or memorable. So I've been recrafting our material specifically for my particular version of a hyperlocal, mobile user.I've been doing this for a few weeks as a prototype, and soon WNYC's editors, producers and hosts will feed lines to the sign. What I've learned by writing -- and watching -- those little red words will help our staff craft the phrases that catch your eye as you zip by.
In my quest for timely, interesting, understandable info-graphics, I've set up a prototyping challenge for myself: Upon finding news data, turn it into something visual, compelling and useful ... as fast as possible.I'm prepared to fail quickly and often.In this case, "as fast as possible" was three days to make, another two to find time to post it. The result is not wholly useful. And you can't absorb it quickly. And it's a little misleading.But it's a start.Here is what I made. It's a visual representation of the attendance rates for every public school in New York City on Thursday, May 21, 2009. The New York City Department of Education started posting this data the previous day as the Swine Flu/H1N1 outbreak was causing kids to stay home.It tells me two things off the bat:
1) Queens and Brooklyn schools had much lower attendance rates than Manhattan and Staten Island schools.2) Teens skip school on nice May days.No. 2 is apparent because almost every red square is a high school, which have notoriously low rates this time of year. For a better indication of potentially flu-related absences, I'd chart the difference between these absentee rates and a typical May day at each school ... which is info I don't have. Yet.Initially I published this in Google Maps, which was interactive and allowed you to click on schools for specific info. But Google Maps only plotted about 100 or so schools, and there are more than 1,000 here. Instead, I did it in Google Earth on my own computer and took a snapshot. Here's another.Kinda cool. Was fun to do.Next!----------Anatomy of the process:Daily absentee data from the school system is here.
An Excel spreadsheet with general data on each school is here.
I crossed these two data sets in Access to match school numbers with addresses.
I got the latitude and longitude for each address, in 500-line batches, here.
I spent a lot of time learning about KML files, writing them, failing, trying again.
I made colored icons in Photoshop, and used Excel to assign each school the correct icon.
I put all of the relevant data into one spreadsheet and fed it to this little helper ...
Which gave me this KML file ...
Which I fed to Google Earth, running on my Mac.
Consider, for a moment, the location of your nearest fire extinguisher.And just how well did you use it last time?Right. Chances are you're not prepared to skillfully put out a fire where you are sitting. At least you're not practiced.But what if you had a non-emergency reason to use a fire extinguisher every once in a while? Maybe to clean your desk. Or a spill. (Nearest paper towel, anyone?) Using it occasionally would help insure that in case of a fire, you would both a) actually use it and b) use it well.To better prepare our on-air and online operations for major breaking news, I've been promoting a point of view that says we shouldn't put our emergency tools, systems and skills "behind glass." Instead, we should incorporate those efforts into our everyday work. (I even believe we shouldn't put our energy into efforts that can't be used on a regular basis because in a crisis, we won't use them anyway.)The best example of this is the daily production of our national morning program, The Takeaway with John Hockenberry and Adaora Udoji. The show's staff may just be the best breaking-news response team in public radio -- because they make the show in real-time every day, incorporating fresh news as they go. When the news happens to be really big, they're not just prepared ... they're already there.A few other examples:
Map maker, map makerI've long been a fan of drills, and there are many more of those in our future. But by incorporating a little drill into our regular routine, we're better prepared for situations that are anything but.[Photo by IamSAM. Some rights reserved.]
In case of a civil emergency in New York, we'd want to quickly map shelters, closed roads, danger zones, escape routes. Even locate our staff. But we weren't prepared to whip together those kinds of maps in mere minutes. Now we're honing those skills by incorporating such work into everyday projects. Information integration
When news hits the fan, information flies everywhere. Consolidating that data is key ... and also happens to be handy in everyday work. In the course of discussing a Mumbai-like terror attack in NYC, we discovered that our news-editing software can also check a listener email box. That's one less window to watch. Nobody move
We designed our newsroom so that in a crisis nobody needs to change seats, which would move them away from familiar surroundings. As a byproduct, when something doesn't flow quite right during daily work, I try to make sure we address it now so we don't get caught off guard later. Expected events as prototypes
In planning for election night, and now for the inauguration, we developed tools and techniques that will serve us again in a major unexpected event. We now know how to quickly rip up our station's home page to focus on a single topic. And in order to provide real-time election-night returns, we found new ways to clear the information path between the editors and the home page. Share and share again
We share a statehouse reporter with stations across our region. So when former governor Eliot Spitzer imploded in a prostitution scandal, we didn't have to think twice about how to move information and audio between stations. We just used the FTP site and email list we use every day.