this space intentionally left blank

May 15, 2014

Filed under: journalism»professional

In These Times

On Monday, I'll be joining the Seattle Times as a newsroom web developer, working with the editorial staff on data journalism and web projects there. It's a great opportunity, and I'm thrilled to be making the shift. I'm also sad to be leaving ArenaNet, where I've worked for almost two years.

While at ArenaNet, I never really got to work on the kinds of big-data projects that drew me to the company, but that doesn't mean my time here was a loss. I had my hands in almost every product, ranging from the account site to the leaderboards to the main marketing site. I contributed to a rewrite of some of the in-game web UI as a cutting-edge single-page application, which will go out later this year and looks tremendously exciting. A short period as the interim team lead gave me a deep appreciation for our build system and server setup, which I fully intend to carry forward. And as a part of the basecamp team, I got to build a data query tool that incorporated heatmapping and WebGL graphing, which served as a testbed for a bunch of experimental techniques.

I also learned a tremendous amount about development in this job. It's possible to argue that ArenaNet is as much a web company as it is a game company, with a high level of quality in both. ArenaNet's web team is an (almost) all-JavaScript shop, and hires brilliant specialists in the language to write full-stack web apps. It's hard to think of another place where I'd have a chance to work with other people who know as much about the web ecosystem, and where I'd be exposed to really interesting code on a daily basis. The conversations I had here were often both infuriating and educational, in the way the best programming discussions should be.

Still, at heart, I'm not a coder: I'm a journalist who publishes with code. When we moved to Seattle in late 2011, I figured that I'd never get the chance to work in a newsroom again: the Times wasn't hiring for my skill set, and there aren't a lot of other opportunities in the area. But I kept my hand in wherever possible, and when the Times' news apps editor took a job in New York, I put out some feelers to see if they were looking for a replacement.

This position is a new one for the Times — they've never had a developer embedded in their newsroom before. Of course, that's familiar territory for me, since it's much the same situation I was in at CQ when I was hired to be a multimedia producer even though no-one there had a firm idea of what "multimedia" meant, and that ended up as one of the best jobs I've ever had. The Seattle Times is another chance to figure out how embedded data journalism can work effectively in a newsroom, but this time at a local paper instead of a political trade publication: covering a wider range of issues across a bigger geographic area, all under a new kind of deadline pressure. I can't wait to meet the challenge.

February 12, 2014

Filed under: journalism»industry

Last Against the Wall

I think most of us can imagine the frustrating experience of sharing a newspaper with the New York Times op-ed page. It must burn to do good reporting work, knowing that it'll all be lumped in with Friedman's Mighty Mustache of Commerce and his latest taxi driver. Let's face it: the op-ed section is long overdue for amputation, given that there's an entire Internet of opinion out there for free, and almost all of it is more coherent than whatever white-bread panic David Brooks is in this week.

But even I was surprised by the story in the New York Observer last week, detailing just how bad the anger between the journalists and the pundits has gotten:

The Times declined to provide exact staffing numbers, but that too is a source of resentment. Said one staffer, “Andy’s got 14 or 15 people plus a whole bevy of assistants working on these three unsigned editorials every day. They’re completely reflexively liberal, utterly predictable, usually poorly written and totally ineffectual. I mean, just try and remember the last time that anybody was talking about one of those editorials. You know, I can think of one time recently, which is with the [Edward] Snowden stuff, but mostly nobody pays attention, and millions of dollars is being spent on that stuff.”

First of all, the Times still runs unsigned editorials? And it takes more than ten people to write them? Sweet mother of mercy, that's insane. I thought the only outlet these days with an actual "from the editors" editorial was the Onion, and even they think it's an old joke. You might as well include an AOL keyword at the end.

And yet it's worth reading on, once you pick your jaw up off the floor, to see the weird, awkward cronyism that's not just the visible portions of the op-ed page, but its entire structure. Why is the editorial section so bad? In part, apparently, because it's ruled by the entitled, petty son of a former managing editor, who reports directly to the paper's publisher (and not the executive editor) because of a family debt. Could anything be more appropriate? As The Baffler notes:

What a perfect way to boil tapioca. Dynasties kill flavor. A page edited by a son because dad was kind of a big deal is a page edited with an eye to status and credentials. Hey, Friedman must be good—he won some Pulitzers. That’s a prize, you see, that Pulitzer thing. Big, big prize. We put it up on the wall. (Pause) Anyway, ready for a cocktail?

The Observer argues that the complaints from the newsroom at large are professional, not budgetary: reporters are angry about shoddy work being published under the same masthead as their stories. But it's hard to imagine that money doesn't enter into it at all. A staff of ten or more people, plus hundreds of thousands of dollars for each of the featured op-ed writers, would translate into serious money for journalism. It would hire a lot of staff, pay for a lot of equipment. You could use it to give interns a living wage, or institute a program for boosting minority participation in media. Arguably, you could put it into a sack and sink it into the Hudson, and still end up ahead of what it's currently funding.

Of course, most papers don't maintain a costly op-ed section, so it's not like this is an industry-wide problem. I don't know that I would even care, normally, beyond the sense of schadenfreude, except for the fact that it's such a perfect little chunk of journalistic mismanagement: when finances get strained, the cuts don't get made from politically-connected fiefdoms, or from upper-level salaries. They get taken from the one place that should be protected, which is the newsroom itself.

Call me an anarchist, but the most depressing part of the whole debate is that it's focused on how big the op-ed budget should be, or how it should be run, instead of whether it should exist at all. What's the point of keeping it around? Or, at the very least, why populate it with the same bland, predictable voices every day? One of the things I respect about the New York Times is the paper's forays into bucking conventional wisdom, from the porous subscription paywall to its legitimately innovative interactive storytelling. There's a lot of romance and tradition in the newsroom, but the op-ed page shouldn't be a part of it. I say burn it to the ground, and let's see what we can grow on the ashes.

January 10, 2014

Filed under: journalism»new_media

App-y New Year

At the end of January, I'll be teaching a workshop at the University of Washington on "news apps," thanks to an offer from the outgoing news app editor at the Seattle Times. It's a great opportunity, and a chance to revisit my more editorial skills. From the description:

This bootcamp will introduce students to the basic components of creating news applications, which are data-powered digital stories tied together through design, programming and journalism. We’ll walk through all the components of creating a news application, look at industry examples of what works and what doesn’t, and learn the basic coding skills required to build a news app.
Sounds cool, but it's still a wide-open field — "data-powered digital stories" covers a huge range of approaches. What do you even teach, and how do you do it in two 4-hour workshops?

It turns out that for almost any definition of "news app," there's an exception. NPR's presidential election board is a data-powered news app, but it's not interactive beyond an auto-update. Snow Fall is certainly a news app, but it's hard to call it "data-powered." How can we craft a category that includes these, but also includes traditional, data-oriented interactives like The Atlantic's Netflix Genre Generator and the Seattle Times mayoral race comparison? More importantly, how do we get young journalists to be able to think both expansively and productively about telling stories online?

That said, I think there is, actually, a unifying principle for news apps. In fact, I think it cuts to the heart of what draws me to web journalism, and the web in general. News apps are journalistic stories told via hypermedia — or, to put it simply, they have links.

A link seems like a small thing after years on the web, so it's good to revisit just how fundamentally groundbreaking they are. Links can support or subvert their anchor, creating new rhetorical devices of their own. At the most basic level, they contextualize a story. More abstractly, they create non-linearity: users explore a news app at their own pace and with their own priorities, rather than the direct stream of narrative from a text story.

A link is a simple starting place. But it starts us down a path of thinking about more complicated applications and usage. I'm fond of saying that an interactive visualization is constructed in many layers, with users peeling open the onion as far as they may want. If we're thinking in terms of other hypertext documents (a.k.a., the TV Tropes Rabbit Hole) from the start, we're already prepared when readers use similar interaction patterns to browse data-based interactives — either by shallowly skipping around, or diving in depth for a specific feature.

By reconceptualizing news apps as being hypermedia instead of a specific technology or group of technologies, such as mapping or graphing, introducing students to web storytelling gets a lot easier — particularly since I won't have time to teach them much beyond some basic HTML and CSS (in the first workshop) and a little scripting (in the second).

It also leaves them plenty of room to think creatively when presenting stories. I'd love for budding news app developers to be as interested in wikis and Twine as they are in D3 and PostGIS. Most importantly, I'd love for an appreciation of hypertext to leak into their writing in general, if only to reduce the number of print die-hards in newsrooms around the country. You don't have to end up a programmer to create new, interesting journalism that's really native to the web.

July 24, 2013

Filed under: journalism»new_media

The Narrative

I had planned on writing a post about Nate Silver's departure from the New York Times this week, but Lance pretty much beat me to it:

Silver is now legendary for being a numbers guy. But there aren't going to be any useful numbers for analyzing the next Presidential election until the middle of 2015 at the earliest. The circumstances under which the election will take place---the state of the economy, whether we're at war or peace, the President's popularity and if and how that will transfer to the Democratic nominee, what issues are galvanizing which voters, etc.---won't make themselves known and so won't show up as numbers in polls at least until then. And until then, everything said about the election is idle speculation, and we know how Silver feels about idly speculating.

But we also know that the most incorrigible idle speculators believe idle speculation is the point.

It's well worth the time to read the whole thing.

I've seen some people assert, in light of this departure, that lots of people could do what Silver did for the Times: his models weren't that complicated, after all, and how hard can it be to write about them? I think this dramatically underestimates the uniqueness of FiveThirtyEight and, to some extent, signifies how threatening it really was to political pundits.

There are, no doubt, a few journalists who could put together Nate Silver's models, and then write about them with clarity. I don't think anyone doubted that evidence-driven political reporting was possible. What he did was show that it could be successful, and that it could draw eyeballs. I think it was John Rogers who said that the best thing about blogging was not the enabling effect for amateurs, but for experts. Suddenly people with actual skills--economists, historians, political scientists, statisticians--could have the kind of audience that op-ed pages commanded.

This should not have been a surprise for newspapers, except that the industry has spent years convincing itself that investigative teams and deep expertise in a beat aren't worth funding. To be fair, the New York Times has put money behind a lot of data journalism in the past few years. If they can't keep the attention of someone like Silver, who can? I guess we're going to find out.

March 13, 2013

Filed under: journalism»industry

Pay Me

They always want the writer to work for nothing. And the problem is that there's so goddamn many writers who have no idea that they're supposed to be paid every time they do something, they do it for nothing! ... I get so angry about this, because you're undercut by all the amateurs. It's the amateurs who make it tough for the professionals, because when you act professional, these people are so used to getting it for nothing, and for mooching...

--Harlan Ellison

Last week, Nate Thayer wrote a well-linked post about being asked to write for The Atlantic for free--well, for "exposure," which is free in a funny hat. It's gotten a lot of attention in the journalism community, including a good piece on the economics of web-scale journalism by Atlantic editor Alexis Madrigal.

I read this kind of stuff and think that I have never been happier to find a niche within journalism that makes me marketable. I mean, not that marketable: I had to switch industries when I moved out of DC, after all. But inside the beltway, I didn't have to freelance anymore, and I would have had plenty of options if I decided to leave CQ and head somewhere else. Data journalism was good to me, and I can't imagine having to go back to the scramble of being just a writer again.

But beneath that relief, I feel angry. And the fact that Madrigal can write a well-reasoned piece about why they're asking people to write for free doesn't make me any less angry. The fact that Ta-Nehisi Coates, who I respect greatly, can write about how writing for free launched the best part of his career, doesn't make me feel any less annoyed. I'm getting older but I'm still punk enough that when someone tells me the system is keeping us down, my response isn't to say, "well, I guess that's just how it is." The system needs to change.

Let's be clear: I don't expect writers to make a lot of money. They never have. People don't get into journalism because they expect to be rich. But writing--serious writing, not just randomly blogging on your pet peeves like I do here 90% of the time--is hard work. The long-form pieces that I've done have been drawn-out, time-consuming affairs: research, interviews, collecting notes, writing, rewriting, editing, trimming, and rewriting again. People think that writing is easy, but it's not, and it should be a paid job. (Even when it's not paid, it's not easy: I've been editing this post for three days now.)

As Ellison says, when publications can get the work for free, it makes it really hard to be paid for your writing. I'm not sure I'd phrase it with the same antipathy for "amateurs" (let's be clear: Ellison is a terrifying human being that I happen to agree with in this particular case), but it's certainly true that the glut of people willing to write for free causes a serious problem for those of us who write (or have written) for a living. They're scabs, in the union sense: they take work that should be paid, and drive down the cost of labor (see also: unpaid musicians).

And journalism is an industry increasingly dependent on free writing labor (or, even worse, perpetual unpaid internships instead of paid staff). As Cord Jefferson (in, of all places, Gawker) notes,

All in all, the creative landscape is starting to look more toxic than it's been in our lifetimes: Artists with million-dollar checks in their pockets are telling other artists that they shouldn't expect to get paid; publications are telling writers that they shouldn't expect to get paid, either; and meanwhile everyone wonders why we can't get more diversity in the creative ranks. One obvious way to reverse media's glut of wealthy white people would be to stop making it so few others but wealthy white people can afford to get into media. But in the age of dramatic newsroom layoffs and folding publications, nobody wants to hear that.
When your publishing model depends on people writing for free, there are a lot of people who aren't going to get published. I couldn't afford internships during college, meaning that I had a hard time breaking in--but I was still relatively lucky. I worked in office jobs with flexible hours and understanding bosses. If I wanted to take an early lunch break in order to do a phone interview, I could. I had evenings free to work on writing and research. I could take jobs that paid 10¢ a word, because I only had a day job. A lot of people don't have that chance, including a disproportionate number of minorities.

It adds insult to injury when you look at some of the people who are published precisely because they could afford internships and writing for free. Sure, it's wrong to base an argument on a few highly-visible outliers. But it's hard not to be a little furious to see the NYT sending good money to Tom Friedman (the obvious travesty), or Roger Cohen, or David Brooks when the industry claims it can't offer new writers recompense. It burns to see The Atlantic insisting that paying people isn't sustainable when they gave Megan McArdle (a hack's hack if there ever was one) a career for years, not to mention running propaganda for the Church of Scientology. If you're going to claim that you're trying as hard as you can to uphold a long-standing journalistic legacy in tough economic times, you'd better make sure your hands are clean before you hold them out in supplication.

I am skeptical, personally, of claims that the industry as a whole can't afford to pay writers. I have heard newsroom financials and profit margins, both for my own employer and for others. The news is no longer a business that prints money, but it remains profitable, as far as I can tell--if not as profitable as management would often like. Perhaps that's not true of The Atlantic: I don't know the details of their balance sheet, although this 2010 NYT article says they made "a tidy profit of $1.8 million this year" and this 2012 article credits them with three years of profitability. That's an impressive bankroll for someone who claims they don't have the budget to pay writers for feature work.

That said, let's accept that I am not an industry expert. It's entirely possible that I'm wrong, and these are desparate times for publications. I can't solve this problem for them. But I can choose a place to stand on my end. I don't work for free, unless it's explicitly for myself under terms that I completely control (i.e., this blog and the others that I fail to maintain as diligently), the same way that I don't take gigs from paying musicians just because I like playing in front of an audience.

Coates may defend working for free, because it got him a guest spot at the publication where he now works. But to me, the most important part of the story is that he got that spot on the strength of his blogging, which drew the attention of other writers and editors. You want exposure? There's nothing wrong with making it for yourself. Please start a blog, and hustle for it like crazy. But don't let other people tell you that it's the same as a paycheck--especially when they're not working for "exposure." They're on salary.

Is there a chance that, as with Coates and so many others, that exposure could lead to better gigs? Sure, the same way that a musician might get discovered while playing folk covers at a Potbelly sandwich shop. But it's a lottery, and pointing to successful writers who came up that way ignores the order of magnitude more that wrote for exposure and promptly sank into obscurity. You can't pay your rent with publicity, and you never could. We're professionals, and we should demand to be treated that way.

December 12, 2012

Filed under: journalism»industry

The Platform

Last week, Rupert Murdoch's iPad-only tabloid The Daily announced that it was closing its doors on Thursday, giving it a total lifespan of just under one year. Lots of people have written interesting things about this, because the schadenfreude is irresistable. Felix Salmon makes a good case against its format, while former staffer Peter Ha noted that its publication system was unaccountably terrible. Dean Starkman at CJR believes, perhaps rightly, that it will take more than a Murdoch rag going under to form any real conclusions.

Around the same time, Nieman Lab published a mind-bogglingly silly pitch piece for 29th Street Publishing, a middleman that republishes magazine content as mobile apps. "What if getting a magazine into Apple's Newsstand was as easy as pushing the publish button on a blog?" Nieman asked on Twitter, demonstrating once again that the business side of the news industry will let nothing stand between it and the wrong questions.

The problem publications face is not that getting into Apple's storefront is too hard--it's that they have a perfectly good (cross-platform) publishing system right in front of them in HTML ("as easy as pushing the publish button on a blog," one might say) and they're completely unwilling to find a business model for it other than throwing up their hands and ceding 30% of their income (and control of their future) to a third party in another industry with a completely different set of priorities. (Not to mention the barriers to search, sharing, and portability that apps throw up.)

What publishers need to be doing is finding a way to monetize the content that they've already got and can already publish using tools that are--well, probably not quite as easy as blogging, but undoubtably far easier than becoming a mobile software developer. One way to do that is with a leaky paywall: it's been a definite success for the NYT, and the Washington Post is considering one. I suspect that when calmer heads prevail, this will become a lot more common. The problem with paywalls is mobile: even if consumers were not conditioned to want "apps," sign-in on mobile is a frustrating user experience problem.

But let's say apps remain a hot topic in news boardrooms. I've been thinking about this for a few days: how could the news industry build a revenue model out of the best of both worlds, with clean mobile HTML deployed everywhere but leveraging the easy payment mechanism of an app store--assuming, in fact, that "payment is hard" is actually a problem the industry has, and given the NYT's success, I'm not honestly sure that it is. My best solution takes inspiration from two-factor authentication (which everyone should be using).

My plan goes like this: just like today, you visit the app store on your platform of choice. You download a yearly "subscription key" application, pay for it in the usual way, and then open it. Behind the scenes, the app talks to the content server and generates a one-time password, then opens a corresponding URL in the default site browser, setting a cookie so that further browser visits will always be signed in--but you as the user don't see any of that. All you see is that the content has been unlocked for you without any sign-in hassle. Next year, you renew your subscription the same way.

In an ideal world, there would be a standard for this that platform authors could implement. Your phone would have one "site key" application (not without precedent), and content publishers could just plug add-on apps into it for both purchasing and authentication. Everyone wins. But of course, that's not a sexy startup idea for milking thousands of dollars from gullible editors. Nor is it helpful for computer companies looking to keep you from leaving their platform: I'm pretty sure an application like this violates Apple's store rules. Personally, that's reason enough for me to consider them unacceptable, because I don't believe the correct response to exploitation is capitulation. That's probably why nobody lets me make business decisions for a major paper.

Assume we can't publish an app: two-factor auth still works in lots of ways that are mobile-friendly, post-purchase. You could visit the website, click a big "unlock" button and be sent a URL via text message, e-mail, Facebook, Twitter, or whatever else you'd like. A site built in HTML and monetized this way works everywhere, instead of locking you into the iPad or another single platform. It lets the publisher, not a third party, retain control of billing and access. And it can be layered onto your existing system, not developed from scratch. Is it absolutely secure? No, of course not. But who cares? As the Times has proven, all you need to do is monetize the people who are willing to pay, not the pirates.

This is just one sane solution that lets news organizations control their own content, and their destiny. Will it happen? Probably not: the platform owners won't let them, and news organizations don't seem to care about having a platform that they themselves own. To me this is a terrible shame: after years of complaining that the Internet made everyone a publisher, news organizations don't seem to be interested in learning that same lesson when the shoe is on the other foot. But perhaps there's an upside: for every crappy app conversion startup funded by desparate magazine companies, there are jobs being created in a recovering economy. Thanks for taking one for the team, journalism.

November 13, 2012

Filed under: journalism»new_media»data_driven

Nate Silver: Not a Witch

In retrospect, Joe Scarborough must be pretty thrilled he never took Nate Silver's $1,000 bet on the outcome of the election. Silver's statistical model went 50 for 50 states, and came close to the precise number of electoral votes, even as Scarborough insisted that the presidential campaign was a tossup. In doing so, Silver became an inadvertent hero to people who (unlike Joe Scarborough) are not bad at math, inspiring a New Yorker humor article and a Twitter joke tag ("#drunknatesilver", who only attends the 50% of weddings that don't end in divorce).

There are two things that are interesting about this. The first is the somewhat amusing fact that Silver's statistical model, strictly speaking, isn't actually that sophisticated. That's not to take anything away from the hard work and mathematical skills it took to create that model, or (probably more importantly) Silver's ability to write clearly and intelligently about it. I couldn't do it, myself. But when it all comes down to it, FiveThirtyEight's methodology is just to track state polls, compare them to past results, and organize the results (you can find a detailed--and quite readable--explanation of the entire methodology here). If nobody has done this before, it's not because the idea was an unthinkable revolution or the result of novel information technology. It's because they couldn't be bothered to figure out how.

The second interesting thing about Silver's predictions is how incredibly hard the pundits railed against them. Scarborough was most visible, but Politico's Dylan Byers took a few potshots himself, calling Silver a possible "one-term celebrity." You can almost smell sour grapes rising from Byers' piece, which presents on the one side Silver's math, and on the other side David Brooks. It says a lot about Byers that he quoted Brooks, the rodent-like New York Times columnist best known for a series of empty-headed books about "the American character," instead of contacting a single statistician for comment.

Why was Politico so keen on pulling down Silver's model? Andrew Beaujon at Poynter wrote that the difference was in journalism's distaste for the unknown--that reporters hate writing about things they can't know. There's an element of truth to that sentiment, but in this case I suspect it's exactly wrong: Politico attacked because its business model is based entirely on the cultivation of uncertainty. A world where authority derives from more than the loudest megaphone is a bad world for their business model.

Let's review, just for a second, how Politico (and a whole host of online, right-leaning opinion journals that followed in its wake) actually work. The oft-repeated motto, coming from Gabriel Sherman's 2009 profile, is "win the morning"--meaning, Politico wants to break controversial stories early in order to work its brand into the cable and blog chatter for the rest of the day. Everything else--accuracy, depth, other journalistic virtues--comes second to speed and infectiousness.

To that end, a lot of people cite Mike Allen's Playbook, a gossipy e-mail compendium of aggregated fluff and nonsense, as the exemplar of the Politico model. Every morning and throughout the day, the paper unleashes a steady stream of short, insider-ey stories. It's a rumor mill, in other words, one that's interested in politics over policy--but most of all, it's interested in Politico. Because if these stories get people talking, Politico will be mentioned, and that increases the brand's value to advertisers and sources.

(There is, by the way, no small amount of irony in the news industry's complaints about "aggregators" online, given the long presence of newsletters like Playbook around DC. Everyone has one of these mobile-friendly link factories, and has for years. CQ's is Behind the Lines, and when I first started there it was sent to editors as a monstrous Word document, filled with blue-underlined hyperlink text, early every morning for rebroadcast. Remember this the next time some publisher starts complaining about Gawker "stealing" their stories.)

Politico's motivations are blatant, but they're not substantially different from any number of talking heads on cable news, which has a 24-hour news hole to fill. Just as the paper wants people talking about Politico to keep revenue flowing, pundits want to be branded as commentators on every topic under the sun so they can stay in the public eye as much as possible. In a sane universe, David Brooks wouldn't be trusted to run a frozen yoghurt stand, because he knows nothing about anything. Expertise--the idea that speaking knowledgably requires study, sometimes in non-trivial amounts--is a threat to this entire industry (probably not a serious threat, but then they're not known for underreaction).

Election journalism has been a godsend to punditry precisely because it is so chaotic: who can say what will happen, unless you are a Very Important Person with a Trusted Name and a whole host of connections? Accountability has not traditionally been a concern, and because elections hinge on any number of complicated policy questions, this means that nothing is out of bounds for the political pundit. No matter how many times William Kristol or Megan McArdle are wrong on a wide range of important issues, they will never be fired (let's not even start on poor Tom Friedman, a man whose career consists of endlessly sorting the wheat from the chaff and then throwing away the wheat). But FiveThirtyEight undermines that thought process, by saying that there is a level of rigor to politics, that you can be wrong, and that accountability is important.

The optimistic take on this disruption is, as Nieman Journalism Lab's Jonathan Stray argues, that specialist experts will become more common in journalism, including in horse race election coverage. I'm not optimistic, personally, because I think the current state of political commentary owes as much to industry nepotism as it does to public opinion, and because I think political data is prone to intentional obfuscation. But it's a nice thought.

The real positive takeaway, I think, is that Brooks, Byers, Scarborough, and other people of little substance took such a strong public stance against Silver. By all means, let's have an open conversation about who was wrong in predicting this election--and whose track record is better. Let's talk about how often Silver is right, and how often that compares to everyone calling him (as Brooks did) "a wizard" whose predictions were "not possible." Let's talk about accountability, and expertise, and whether we should expect better. I suspect Silver's happy to have that talk. Are his accusers?

February 13, 2012

Filed under: journalism»political

Red Letter Day

Ah, budget day: the most annoying day of a data journalist's year. Even now that I'm no longer covering Congress, it still bugs me a little--except now, instead of being frustrated by the problem of finding stories, I'm just annoyed by the coverage itself. Few serious policy documents create so much noise from so little data.

For those who are unaware, on the night before budget day, each senator or representative places a constituent's tooth under their pillow before going to bed. While they're asleep, dreaming of filibusters and fundraising, the White House Chief of Staff creeps into their bedrooms and takes the tooth away, leaving a gift in return. Oh, the cries of joy when the little congresscritters wake to find a thick trio of paperback budget documents waiting for them!

Casting the budget as a fairy tale isn't as snarky as it might seem, because the president's budget is almost entirely wishful thinking. The executive branch, after all, does not control the purse strings of government--that power lies with the legislature. The budget is valuable in that it sets an agenda and expresses priorities, but any numbers in it are a total pipe dream until the appropriations process finishes. And if you want an example of how increasingly dysfunctional Congress has become, look no further than appropriations.

Although money for the next fiscal year is supposed to be allocated into appropriations bills by October 1st (the start of the federal fiscal year), they are increasingly late, often months late. In the meantime, Congress passes what are called "continuing resolutions"--stopgap measures that fund the government at (usually) reduced levels until real funding is passed. You can actually see the delays getting worse in a couple of graphics that my team put together at CQ: first, the number of "bill days" delayed since 1983, and then the number of "bill months" delayed by committee since 1990. Needless to say, this probably isn't helping the federal government run at its most efficient.

The connection between the president's budget and the resulting sausage is therefore tenuous at best (don't even get me started on tracking funds through appropriations itself). Even worse, from the perspective of a data-oriented reporter, is that the numbers in the budget are not static. They are revised multiple times by the White House in the months after release--and not only are they revised, they are often revised retroactively as new economic data comes in and the numbers must be adjusted to fit the actual policy environment. So even if we could talk about budget numbers as though they were "real money," the question remains: which budget numbers? And from when?

During my first couple of years at CQ, around January I would sit down with the Budget Tracker team and the economics editor, and propose a whole series of cool interactive features for budget season. And each time, they would politely and carefully explain all these caveats, which collectively added up to: we could talk about the budget in print, where numbers would not be charted against each other, and we could talk about the ways the budget/appropriations process is broken. But there simply isn't enough solid data to graph or visualize those numbers, since that lends them a visual credibility that they don't actually have.

The result is that I find budget day frustrating, even after leaving the newsroom, because it feels like a failure--something we should have been able to explain to our readers more fully, but couldn't quite grasp ourselves. Simultaneously, I often find coverage by other outlets annoying because they report on the budget as thought it's more meaningful than it actually will be, or they'll chart it across visualizations as though imaginary numbers could be compared to each other (there is an element of jealousy to this, no doubt: it must be nice to work in a place where you can get away with a little editorial sloppiness). It's a shame, because the budget itself is not broken. As an indication of what the White House thinks is important for the upcoming year, it's a great resource. But it is not a long-term financial plan, and shouldn't be reported as such.

January 18, 2012

Filed under: journalism»new_media

Your Scattered Congresses

Once more with feeling: today, I'm happy to bring you my last CQ vote study interactive. This version is something special: although it lacks the fancy animations of its predecessor, it offers a full nine years of voting data, and it does so faster and in more detail. Previously, we had only offered data going back to 2009, or a separate interactive showing the Bush era composite scores.

We had talked about this three-pane presentation at CQ as far back as two years ago, in a discussion with the UX team on how they could work together with my multimedia team. Our goal was to lower the degree to which a user had to switch manually between views, and to visually reinforce what the scatter plot represents: a spatial view of party discipline. I think it does a pretty good job, although I do miss the pretty transitions between different graph types.

Technically speaking, loading nine years of votestudy data was a challenge: that's almost 5,000 scores to collect, organize, and display. The source files necessarily separate member biodata (name, district, party, etc) from the votestudy data, since putting the two into the same data structure would bloat the file size from repetition (many members served in multiple years). But keeping them separate causes a lag problem while interacting with the graphic: doing lookups based on XML queries tends to be very slow, particularly over 500K of XML.

I tried a few tricks to find a balance between real-time lookup (slow interaction, quick initial load) and a full preprocessing step (slow initial load, quick interactions). In the end, I went with an approach that processes each year when it's first displayed, adding biodata to the votestudy data structure at that time, and caching member IDs to minimize the lookup time on members who persist between years. The result is a slight lag when flipping between years or chambers for the first time, but it's not enough to be annoying and the startup time remains quick.

(In a funny side note, working with just the score data is obscenely quick. It's fast enough, in fact, that I can run through all nine years to find the bounds for the unity part of graph to keep it consistent from year to yearin less than a millisecond. That's fast enough that I can be lazy and do that before every re-render--as long as I don't need any names. Don't optimize prematurely, indeed.)

The resulting graphic is typical of CQ interactives, in that it's a direct view on our data without a strong editorial perspective--we don't try to hammer a story through here. That said, I think there's some interesting information that emerges when you can look at single years of data going back to 2002:

  • The Senate is generally much more supportive of the president than the House is. While you can't directly compare scores across chambers (because the votes are different), the trend is striking. It's well known that House members tend to be more radical than senators, but I suspect the difference is also procedural: in the House, the leadership controls the agenda much more tightly than in the Senate, which can be held up by filibuster. As a result, the House may vote on bills that would never reach the Senate floor, just because the majority party can force the issue.
  • Although the conventional wisdom on the left since the Gingrich years has been that Republican discipline is stronger for political reasons, I'm not sure that's entirely borne out by these graphics. Party unity over the last nine years appears roughly symmetrical most of the time, while presidential support (and opposition) appears to shift in direct response to the strength of the White House due to popularity and/or election status. 2007-2009 was a particularly strong time for the Democrats in terms of uniting around or against a presidential agenda, for obvious reasons. This year the Republicans rallied significantly, particularly in the House.
  • There is one person who's explicitly taken out of the graphs (and not removed due to lack of participation or other technical reasons). That person is Zell Miller, everyone's favorite Bush-era iconoclast. If you're like me, you haven't thought about Zell Miller in 6 or 7 years, but there he was when I loaded the Senate file for the first time. Miller voted against his party so often that he had ridiculously low scores in 2003 and 2004, resulting in a vast expanse of white space on the plots with one lonely blue dot at the bottom. Rather than let him make everyone too small to click, I dropped him from the dataset as an outlier.
All of this, of course, is just my amateur political analysis. While I'm arguably more informed (possibly too informed!) about congressional practice than the average person, I'm no expert. For that, you may want to check out CQ's always-fantastic editorial graphics on the votestudies, which show in more detail the legislative trends of the last few decades. It's very cool stuff.

Finally, I did mention that this is my last CQ votestudy interactive. It's been a fantastic ride at Congressional Quarterly, and I'm grateful for the opportunities and education I received there. But it's time to move on, and to find something closer to home here in Seattle: at the end of this month, I'll be starting in a new position, doing web development at Big Fish Games. Wish me luck!

November 9, 2011

Filed under: journalism»new_media


As the deadlines creep forward for the Joint Special Committee on Deficit Reduction, my team at CQ has put together a package of new and recent debt interactives covering the automatically-triggered budget cuts, the proposals on the table, the schedule set for committee action, and more.

The centerpiece of the package is a "reactive document" showing how the automatic cuts will go into effect if Congress does not pass cuts totalling $1.2 trillion by January 15. A series of sliders set the size of the hypothetical cuts, and the text and diagrams of the document adjust themselves to match. It's a neat idea, and one that's kind of a natural match for CQ: wordy, but still wonky.

Like a lot of people, I encountered the idea of reactive documents through Bret Victor's essay Explorable Explanations. Victor is an ex-Apple UI designer who wants to re-think the way people teach math, and reactive documents are one of the tools he wants to use. His explorations of learning design via reactive documents, such as Up and Down the Ladder of Abstraction, are breathtaking. As he writes,

There's nothing new about scenario modeling. The authors of this proposition surely had an Excel spreadsheet which answered the same questions. But a spreadsheet is not an explanation. It is merely a dataset and model; it cannot be read. An explanation requires an author, to interpret the results of the model, and present them to the reader via language and graphics.

The reactive document integrates spreadsheet-like models into authored text. It can be read at multiple levels, depending on the reader's level of interest. The hurried reader can skim it. The casual reader can read it as-is. The curious reader can adjust the author's scenarios. The engaged reader can explore scenarios of his own devising.

Unlike a spreadsheet, the barrier to exploration here is extremely low -- simply click and drag. This invites casual readers to become engaged and start exploring. It transforms readers from passive to active.

Victor's idea is a clever one, and as someone who often describes interactives using the same "layered reading" mechanism, it appeals to my storytelling sense. I also like that it embraces the original purpose of the web--to present hypertext documents--without sacrificing the rich interactions that browser applications have developed. That said, I'm not entirely convinced that reactive documents like this are actually terribly useful or novel.

The main problem with this method of presenting interactive information is that it's actually really burdensome for the playful user. It's easy to read, but if you change anything, you have to basically either read and process the entire paragraph again, or you have to learn to pick out individual changes and their meaning from a jumble of words. Besides, sometimes words are not a very good description of an effect or process--imagine describing complex machinery only in paragraph form.

Victor also has some examples that avoid this flaw by making the reactive document incorporate diagrams and graphs alongside his formulas. These are great, but they also illustrate the fact that, once you make reactive "documents" more visual and take away the intertextual trickery, they're really just regular interactives. They're stunningly designed, and I'm always in favor of more multimedia, but there's nothing new about them.

This probably comes off as a little more adversarial to the concept of reactive documents than I actually am, most of which is just my rhetorical background leaking out. I think they're neat, and I would guess that Victor himself thinks of them less as a complete solution and more as a different shade in his teaching palette. In some places, they're helpful, in others not so much.

As an Excel enthusiast, though, I do take exception to Victor's description of spreadsheets as something that "cannot be read," with a high barrier to entry. People read and create spreadsheets all the time, although (to my frustration) they often use them as layout tools. But a spreadsheet that's already set up for someone and locked up to prevent mistakes is barely any more difficult to use than his draggable text--the only real difference is the need to type a number. Regular people may find spreadsheet formulas difficult to connect with cells, but those same people are unlikely to be creating Victor's reactive documents either.

Ultimately, I'm wary of claims that any tool is a silver bullet for education or explainer journalism. It's easy to be blinded by slick UX, and to forget that we're basically just re-inventing storytelling tools used by great teachers for centuries. That shouldn't eliminate interactive games and illustrations from our kit. But reading Victor's site, it's easy to give the technology credit for its thought-provoking qualities, when the credit really goes to his lucid, considered reasoning and clear writing (both of which mean that the technology is well-applied). Sadly, there's no script for that.

Future - Present - Past