Modeling the 2020 vote with Observable

I've been interested in how voter turnout might affect the 2020 US election and I've wanted to play with Observable notebooks.

So I blended the two projects, and you can play with my live Observable notebook that does those calculations.

The result is an admittedly super-simplistic model of how things might turn out. But you can increase the percentage of Republican and Democratic voters nationwide and see what happens!

Notably, even if Democrats were able to boost turnout more than Republicans — say 107% vs 106% — Trump still wins.

As written, it doesn't consider nuances such as regional differences in voting turnouts, swing voters, or faithless electors. (It does, however, account for the unique ways Maine and Nebraska divide their electoral votes). But I learned a lot in the process ... and there's more to come.

All my calculations are visible in the Observable notebook itself, and the initial data prep is documented in a Github repository. For good measure, I put all the raw data in my Datasette library.

Minneapolis race and ethnicity data by neighborhood, served with Datasette

Minneapolis police report stops and other incidents by neighborhood, so I decided to calculate the racial makeup of those neighborhoods to make some comparisons — along the lines of what I've already done for New York, Chicago, and Washington, DC.

This time, though, I'm using Datasette.

I've seen creator Simon Willison tweet about Datasette, and with some extra time on my hands I took a look. It's so impressive!

With Datasette, one can publish data online easily, efficiently (even free!) and in a way that allows others to explore the data themselves using SQL and feed data visualizations and apps. At scale.

How is this not in every newsroom?

(Simon, by the way, has offered to help any newsroom interested in using Datasette — an offer I hope to take him up on someday.)

Minneapolis neighborhoods

Once again, I've married US Census blocks with other municipal zones, this time the official neighborhood map of Minneapolis.

That data is now online, served up with Datasette.

And with some nifty SQL queries, bookmarked as simple links, I can list the race and ethnic makeup of every neighborhood by raw number.

Or by percentage.

Lockdown loaves

It's become a coronavirus cliché, but for this week's #MakeEveryWeek I made sourdough bread. 

The twist: I made one loaf in the oven and one in a slow cooker.

It all started with sourdough starter, specifically this guide from Quartz colleague Tim McDonnell. This was a great project for my teens, incorporating chemistry, biology, and excellent smells.

Next was this incredibly fun and detailed sourdough recipe from Kitchn, which makes two loaves and relies on two oven-safe pots. Alas, our family has but one.

We do have a slow cooker, though. Could I make one of the loaves in that? The answer is yes!

Building a pulse oximeter

At-home pulse oximeters, those fingertip devices doctors use to measure the oxygen saturation in your blood, have been selling out everywhere thanks to the Covid-19 pandemic.

But as my Quartz colleague Amirta Khalid points out in this great article, most people don't need 'em. If your oxygen level is worryingly low, you'll know — you don't need a machine to tell you. Folks with some existing conditions, however, can use a pulse oximeter to help a remote doctor monitor their vitals or to adjust supplementary oxygen devices.

When Khalid mentioned she was working the story, it reminded me of the DIY "pulse ox" sensor Sparkfun sells. It, like other pulse oximeters, shines light into the skin and makes measurements based on how that light is absorbed. I've built heartbeat-driven projects before and had been exploring new ways to monitor pulse rates. So I got one.

Sparkfun warns in red letters that "this device is not intended to diagnose or treat any conditions," and I offer the same caution if you're tempted to build one. The process wasn't hard at all. I got it running quickly ... and then added an LED display for fun and flourish.

Here's how I made it, and the code, too.

Work-from-home "on air" light

I'm incredibly lucky to be both healthy and able to work from home during this coronavirus crisis. That means I spend large chunks of my day on video calls.

As a courtesy to my family, all of whom are also working and schooling from home, I've tried to warn them when they risk being broadcast to my colleagues. 

Now I have a fun "on air" light to help! And I've put the code online so you can make one, too.

DIY aquarium lights

Buy a new aquarium, and you often get hood lights that are ... meh. They're good enough, but not great.

There are plenty of high-quality replacement lights out there, but none of them had the nice, low profile of the plastic covers that came with this tank. So I decided to spruce up the existing illumination with some DYI lights — and even make them programmable with an Arduino.

That was more than a year ago. Now in coronavirus isolation, I finally made it happen.

Here's how.

Detecting feature importance in fast.ai neural networks

I'm working on a new neural network that tries to predict an outcome – true or false – based on 65 different variables in a table.

The tabular model I made with fast.ai is somewhat accurate at making those predictions (it's a small data set of just 5,000 rows). But to me even more interesting is determining which of the 65 features matter most. 

I knew calculating this "feature importance" was possible with random forests, but could I do it with neural nets?

It turns out I can. The trick is, essentially, to try the model without each feature. The degree to which the model gets worse with that feature missing indicates its importance – or lack of importance.

This blog post describes how to run this test, and this adaptation worked perfectly in my fast.ai notebook. Here's the code in a Gist:

Unfortunately, because my project uses internal Quartz analytics, I can't share the data or the charts I'm playing with. But with the code above, I can now "see into" the neural network and get cool insights about what's going on


Converting videos to images for machine learning

This week I kept to my summer of training plan, however the model-building I did was for a Quartz project we're not ready to share. But! I learned something super useful in the process: how to quickly turn videos into many still images.

For our latest project, I'm training a model to identify specific objects available to me – much like how I trained a model to identify items in the office.

The fastest way to get lots of images of an object is to take a video of it. And a quick way to turn that video into images – called an "image sequence" – is ffmpeg. It seems to convert from many formats like .mp4, .mov, .avi to lots different image formats such as .jpg and .png.

There's plenty more detail in the ffmpeg docs, but here's what I did that worked so quickly on my Mac:

brew install ffmpeg

I use Homebrew to put things on my Mac, so this went pretty quickly. I had to update my Xcode command line tools, but Homebrew is super helpful and told me exactly what I needed to do.

Next, I did this from the Terminal:

ffmpeg -i IMG_1019.MOV -r 15 coolname%04d.jpg

Here's what's going on:

  • -i means the next thing is the input file
  • IMG_1019.MOV is the movie I Airdropped from my phone to my laptop
  • -r is the flag for the sample rate.
  • 15 is the rate. I wanted every other image, so 15 frames every second. 1 would be every second; 0.25 every 4th second.
  • coolname is just a prefix I picked for each image
  • %04d means each frame gets a zero-padded sequence number, starting with 0001 and going to 9999– so my image files are named coolname0001.jpg, coolname0002.jpg, coolname0003.jpg, etc.
  • .jpg is the image format I want. If I put .png I got PNGs instead.

In mere moments I had a dozens of JPG files I could use for training. And that's pretty great.

A bot now updates my Slack status

One of my closest collaborators is a teammate far away — I'm in New York and Emily Withrow is in Chicago.

We stay connected chatting on Slack. But recently Emily asked if I could regularly update my Slack status to indicate what I was doing at the moment, like coding, meeting, eating. It's the kind of thing colleagues in New York know just by glancing toward my desk.

Changing my Slack status isn't hard; remembering do it is. So I built a bot to change it for me.