Over the weekend we soft-launched our Seahawks Fan Map project. It's a follow-up on last year's model, which was built on a Google Fusion Table map. The new one is better in almost every way: it lets you locate yourself based on GPS, provides autocomplete for favorite players, and clusters markers instead of throwing 3,000 people directly onto the map. It's also built using a couple of interesting technical choices: Google Apps Script and "web components" in jQuery.
The answer, as with many newsrooms, is heavy use of Google Sheets as an ad-hoc CMS. I recently added the ability to for our news app scaffolding to pull from Sheets and cache the data locally as JSON, which is then available to the templating and build tasks. Once every few minutes, a cron job runs on a machine in our newsroom, which grabs the latest data and uploads a fresh copy to the cloud. Anyone can use a spreadsheet, so it's easy for editors and writers to update the data or mark a row as "approved," "featured," or "blocked."
Getting data from the form into the sheet is a more interesting answer. Last year's map embedded a Google Forms page, which is the source of many of its UI sins: they can't be styled, they don't offer any advanced form elements, and they can't be made responsive. Nobody really likes Google Forms, but they're convenient, so people use them all the time. We started from that point on this project, but kept running into features that we really wanted (particularly browser geolocation) that they wouldn't support, so I went looking for alternatives.
Combining the web endpoint for Apps Script with our own custom form gave us the best of both worlds. I could write a form that had pretty styling, geolocation, autocomplete, and validation, but it could still go through the same Google Docs workflow that our newsroom likes. Through the API, I could even handle geocoding during form submission, instead of writing a separate build step. The speed isn't great, but it's not bad either: most of the request time is spent getting a lock on the spreadsheet to keep simultaneous users from overwriting each other's rows. Compared to setting up (and securing) a server backend, I've been very happy with it, and we'll definitely be using this for other news apps in the future.
I'm a huge fan of Web Components and the polyfills built on top of them, such as Polymer and Angular. But for this project, which does not involve putting data directly into the DOM (it's all filtered through Leaflet), Angular seemed like overkill. I decided that I'd try to use old-school technology, with jQuery and I Can Haz templates, but packaged in a component-like way. I used AMD to wrap each component up into a module, and dropped them into the markup as classes attached to placeholder elements.
The result, I think, is mixed. You can definitely build components using jQuery — indeed, I'm very happy with how readable and clean these modules are compared to the average jQuery library — but it's not particularly well-suited for the task. The resulting elements aren't very well encapsulated, don't respond to attribute values or changes, and must manually handle data binding and events in a way that Polymer and Angular safely abstract away. Building those capabilities myself, instead of just using a library that provides them, doesn't make much sense. If I were starting over (or as I consider the additional work we'll do on this map), it's very tempting to switch out my jQuery components for Angular directives or Mozilla's X-Tags.
That said, I'm glad I gave it a shot. And if you can't (or are reluctant to) switch away from jQuery, I'd recommend the following strategies:
The map you see today is only the first version — this is one of the few news projects I plan to maintain over an extended period. As we get more information, we'll add shaded layers for some of the extra questions asked on the form, so that you can see the average fan "lifespan" per state, or find out which players are favorites in countries around the world. We'll also feature people with great Seahawks stories, giving them starred icons on the map that are always displayed. And we'll use the optional contact info to reach out to a "fan of the week," making this both a fun interactive and a great reporting tool. I hope you enjoy the map, and if you're a Seahawks fan, I'll see you there!
This fall, I'll be teaching ITC 210 at Seattle Central College, which is the capstone class of the web development program there. It's taught as a combined class with WEB 210 (the designer's capstone). The last time I taught this course, it didn't go particularly well: although the goal is for students to implement a WordPress site for a real-world client, many of them weren't actually that experienced with the technology.
More importantly, they had never been taught any of the development methods that let teams work together efficiently. I suggested some of the basics — using source control, setting tasks, and using a "waterfall" structure — but I didn't require them, which was a mistake. Under pressure, students fell back on improvised strategies, and many of them ended up in a crunch as a result.
For the upcoming quarter, I plan to remedy those mistakes. But to do so, it's helpful to look at the web development program from a macro level. What is that we're trying to do here, and what should this capstone class actually mean to students?
Although the name has changed, Seattle Central is still very much a community college, and this is very much a trade program. We need to focus on practical job skills, not on CS theory. And so while the faculty are still working on many of the details, one of our goals for curriculum redesign was to create a simple progression between the three web applications classes: first teach basic programming in ITC 240, followed by an MVC framework in ITC 250, and finish the process with a look at development processes (agile, waterfall, last-minute panic, etc.) in ITC 260. By the end, students should feel like they can take a project from start to finish as part of a team in an organized fashion.
Of course, just because that's what our intentions were doesn't mean that it's working out that way. These changes are large shifts in the SCC curriculum, and like steering an Oldsmobile, those take time. So while it would be nice to assume that students have been through the basics of project management by the time that they reach the capstone, I can't count on it — and even then, they probably won't have put it to practice in teams, since the prior classes are individually-graded.
To bring this back to ITC 210, then, we have two problems. First, students don't know how to manage development, because they've spent most of their time just learning how to code. Second, the structure of the class hasn't historically encouraged them to develop those skills. Assignments on the development side tend to be based around the design milestones, which makes their workload "lumpy:" a lot of waiting for design resources, followed by an intense, panicky burst at the end. This may sometimes be an accurate picture of the job, but it's a terrible class experience. Ideally, we want the developers to be working constantly throughout the quarter.
So here's my new plan: this year, ITC 210 will be organized for students around a series of five agile sprints, just like any real-world coding project. At the start of each sprint, they'll assign time and staff to tasks, and at the end of each sprint they'll do a retrospective to help determine their velocity. Grades will be largely organized around documentation of this process. During the last sprint, they'll pick up another team's site and file bugs against it as QA, while fixing the bugs that are filed against them.
This won't entirely smooth out the development process — devs will still be bottlenecked on design work from time to time — but it will make it clear that I expect them to be working the entire time on laying groundwork. It'll also familiarize them with the ways that real teams coordinate their efforts, and it will force them to fit into a common workflow instead of fragmenting into a million angry swarms of random methodology.
I tend to make fun of programmers for thinking that they're the only ones who can invent a workflow, but it's easy to forget that coordinating a team is hard, and nobody comes by it naturally. I made that mistake last time around, and although we scraped by, there were times when it was rough. This quarter, I'm not giving students a choice: they'll work like a regular software team, or they'll fail the course. It may seem harsh, but I think it'll pay off for them when it comes time to do this for a living.
I would love to have been in the meeting where someone pitched Guardians of the Galaxy. "We're going to take all the good will you've built up through the Marvel comic movie franchise, and then spend it on a space movie with characters that nobody really knows, one of whom is a heavily-armed raccoon." And then even weirder, it worked: Guardians is pretty good. Maybe it tells when it should show a little too often, but it never stopped me from enjoying myself. It's got a great soundtrack, good writing, well-done special effects, and most importantly, a really watchable cast.
Of course, this has been the case with most of the Marvel movies. I mean, let's be honest about, say, the Thor franchise, which have been a fun pair of movies considering that they're composed almost entirely of gibberish: Norse gods with British accents (who are actually aliens) fighting against elves and ice trolls (who are also aliens)! The whole thing is completely incoherent, but nobody cares because of the casting: everybody onscreen is good-looking, compulsively charming, and clearly having fun with a very silly premise.
But there's one thing that's been bugging me about the Marvel flicks, including Guardians, which is their endings. Namely, that they've all got the same one: the bad guys summon/control/take over a huge flying object, which immediately crashes headlong into a city.
Explosions follow, while the heroes rush to tackle the portal/controller/big bad at the wheel. Lots of buildings fall over in the process, and people run through the streets while looking up and behind them (oddly enough, hardly anyone ever trips). Lance Mannion refers to it as the "obligatory ad for the video game," and while that's harsh it's not inaccurate, because it does feel a little bit (between the overused CGI and the framing) like watching someone else play God of War. A lot of money went into it, and someone's clearly having a good time, but it's not necessarily you.
And to be clear, Marvel's not the only company writing screenplays this way. Star Trek: Into Darkness, for example, was a movie that committed every sin in the screenwriting book (and then added a few) but arguably the worst part was the meaningless and cruel spaceship crash at its climax. Over at Fox, X-Men: Days of Future Past has Magneto tossing an airborne baseball stadium at the White House. Huge flying objects are the new glass jail cell.
The problem with these pyrotechnics isn't just that they're repetitive and tasteless (although they are both), it's that they're ineffective: by destroying a huge chunk of a city, the writers are aiming for huge cinematic stakes, but by portraying it from a wide-angle lens (and, in PG-rated films, refusing to show any of the resulting bodies and carnage) there's no sense of drama. It's just computer-generated buildings falling over in the distance: who cares?
For a movie like Guardians, which has no real point other than to be a fun space adventure, it's bad enough that there's a good fifteen minutes of watching architecture instead of the characters that are the real draw. But it's more acutely frustrating for something like The Winter Soldier, which spends its first 90 minutes referencing the modern surveillance state, which is a tricky, subtle political problem. And nothing says "tricky" and "subtle" like sending three flying aircraft carriers through a building in Washington, DC.
Maybe that's expecting a bit much from a huge media property with a multi-year cinema domination plan. Marvel wants to put people into seats twice a year, and if that means making the same movie over and over again, that's fine. It's certainly never stopped anyone else (see also: Transformers and Harry Potter). If they're not even sure they can make a movie starring a woman, the chances they'll mess with the formular are pretty slim.
But look at it this way: now you know when you can take a bathroom break without missing anything. I figure you can stay away until the end of the credits, at which point you'll learn which comic-book movie will drop a giant metal object on Paris next year. If we're all very lucky, it'll be Squirrel Girl's turn eventually.
This week, my interactive work for the Seattle Times examines the bidding wars that are part and parcel of being one of the fastest growing cities in the country. It's got everything you need to be horrified by your local real estate market: high prices, short days-on-market, and a search function to see how dire it is next door. It is also the third or fourth interactive that I've built with Angular this year (source code here). There aren't a lot of people building news apps with Angular, which I find amazing: if your goal is to surface data on a deadline, I'd argue it's the best option out there.
Let's review what Angular brings to the table. At the most basic level, it's a library for doing two things:
By contrast, when I look at code written in D3 (seemingly the most popular library for doing news visualizations), I see an entirely different set of priorities:
After years of debugging spaghetti code in jQuery, this design seems both familiar and ominous, particularly the lack of templating and the long call chains. I've written my fair share of apps this way, and they tend to sprawl out into an unstructured, unmaintainable mess. That may not be a problem for the New York Times, which has more budgetary and development resources than I'll ever have. But as the (for now) only developer in the Seattle Times newsroom, I need to be able to respond instantly to feedback from designers, editors, and reporters. One of my favorite things to hear is "we didn't expect a change so fast!" Angular gives me the agility I need to iterate rapidly, try things out, and discard what doesn't work in favor of what does.
Speed and structure are good reasons to use Angular in a newsroom, but there's another, less obvious incentive. Angular is basically training wheels for Web Components: although it lacks the Shadow DOM, it includes equivalents for custom elements and HTML imports. It's a short hop from Angular to libraries like Polymer, and from there to a whole world of deadline-friendly tooling and reuse. Make no mistake, this is the future of web development, and it can't get here soon enough: I'd love to be able to simply send off an <interactive-feature> tag to the web producers, and I imagine they'd appreciate it too. The Google Web Components tags would be a similar godsend.
For me, this makes using Angular a no-brainer. It's fast, it's effective, it's great for visualizations, and it's forward-thinking. It shocks me that more people haven't seen its advantages — but then, given the way that most newsroom hackers seem to think of the browser as "that embarrassing thing that loads my server code," it probably shouldn't be surprising.
We've just released a new interactive I've been working on for a couple of weeks, this time exploring the Oso landslide earlier this year. Our timeline (source) shows... well, I'll let the intro text explain it:
The decades preceding the deadly landslide near Oso reflect a shifting landscape with one human constant: Even as warnings mounted, people kept moving in. This interactive graphic tells that story, starting in 1887. Thirteen aerial photographs from the 1930s on capture the geographical changes; the hill is scarred by a succession of major slides while the river at its base gets pushed away, only to fight its way back. This graphic lets you go back in time and track the warnings from scientists; the failed attempts to stabilize the hill; the logging on or near the unstable slope; and the 37 homes that were built below the hill only to be destroyed.
The design of this news app is one of those cases where inspiration struck after letting its idea percolate for a while. We really wanted to showcase the aerial photos, originally intending to sync them up with a horizontal timeline. I don't particularly care for timelines — they're basically listicles that you can't scan easily — so I wasn't thrilled with this solution. It also didn't work well on mobile, and that's a no-go for my Seattle Times projects.
One day, while reading through the patterns at Bocoup's Mobile Vis site, it occurred to me that a vertical timeline would answer many of these problems. On mobile, a vertical scroll is a natural, inviting motion. On desktop, it was easier to arrange the elements side-by-side than stacked vertically. Swapping the axes turned out to be a huge breakthrough for the "feel" of the interactive — on phones and tablets that support inertial scrolling for overflow (Chrome and IE), users can even "throw" the timeline up the page to rapidly jump through the images, almost like a flipbook. On desktop, the mouse wheel serves much the same purpose.
On a technical level, this project made heavy use of the app template's ability to read and process CSV files. The reporters could work in Excel, mostly, and their changes would be seamlessly integrated into the presentation, which made copy editing a cinch. I also added live reload to the scaffolding on this project — it's a small tweak, but in group design sessions it's much easier to keep the focus on my editor for tweaks, but let the browser refresh on another monitor for feedback. I used Ractive to build the timeline itself, but that was mostly just for ease of templating and to get a feel for it — my next projects will probably return to Angular.
All in all, I'm extremely happy with the way this feature turned out. The reporting is deep (in a traditional story, it would probably be at least 5,000 words), but we've managed to tell this story visually in an intuitive, at-a-glance format, across multiple device formats. Casual readers can flip through the photos and see the movement of the river (as well as the 2014 devastation), while the curious can dig into individual construction events and warning signs. It's a pretty serious chunk of interactive storytelling, but we're just getting started. If you or someone you know would like to work on projects like this, feel free to apply to our open news app designer and developer positions.
Hey, remember that huge Facebook controversy? No, not when they tried to emotionally manipulate thousands of people for a study of dubious worth. Also not when they leaked random purchases to all your friends, thus exposing your secret SexyKiltsAndHikingBoots.com purchases to the world (what? It's a Seattle thing, all right?). Probably not when they complained about the news culture that they themselves had created. Maybe it was when Facebook removed privacy options (or just changed them around in one of the site's near-constant redesigns).
Actually, I'm not really sure what I'm talking about either. To put it in the Upworthy headline-speak that clogs your news feed until you sigh and grudgingly switch it back to "most recent" mode again, "This social network has a bad habit of treating its users like lab rats. You won't believe how little they care!"
At some point, I resigned myself to the fact that I'm not going to quit Facebook anytime soon, no matter how bad their behavior is. Most people won't. In my case, even if I don't use it often, it's the only place where local dance events get publicized — without it, my ability to participate gets curtailed sharply. For other friends of mine, Facebook messages are a primary means of communication, over e-mail or even SMS. It's how we keep in touch with each other, even just in an ephemeral, transitory sense.
You may remember, however, that when Facebook first started along the path of "we know what you should see better than you do," there was a scrappy crowdfunding effort in response, for a decentralized social network run by users, for users. Diaspora took in a huge amount of money, showed no progress for a couple years, shut down, open sourced itself, and is now puttering along with roughly 15 thousand users. Which is, to be honest, what most reasonable probably saw happening anyway, but we live in a world where people collectively donate $30,000 for potato salad, so maybe a little perspective is in order.
Even though this is all fairly predictable, as someone with some interest in self-hosted versions of cloud services, I'm intrigued by the question of why we don't have a popular, decentralized, open-source alternative to Facebook. I don't think it's a difficult question to answer — rather, it's interesting because there are actually multiple reasons that it doesn't happen, and they point out bigger problems with cloud-based computing for everyday people. Facebook is a great case study for this, because many programmers have a habit (particularly when a little tipsy) of pointing out that they could write a simple Facebook clone in a weekend. This is both true, and entirely missing the point.
To see why, we have to look at a seemingly unrelated incident: Facebook's $1 billion purchase of Instagram. Why so much? It's not because it was an equal competitor: filtered photos don't compete with the Facebook news feed directly. But it had a hook that would pull users — easy sharing using your phone camera instead of a keyboard — and once the audience is there, upgrading to a Facebook-like feature set is easier. In other words, Instagram wasn't valuable because it was like Facebook. It was valuable because it was different enough to get users' attention, but close enough to serve the same social needs.
Every existing competitor to Facebook has a compelling hook. If Instagram has sharing, Snapchat has its (supposed) privacy features, and LinkedIn has a ton of annoying recruiters sending e-mails to random users. Unfortunately, open source is not good at figuring out product hooks — it tends to excel at imitation and evolution. An open-source social network would probably be better written than Facebook, but without that showcase feature, nobody will join. And a social network with no people in it is worthless. Lesson one: find a hook that's not "is open source." I suspect gaming is a possible contender, but it's proven elusive so far (both for small players like OpenFeint and the big vendors).
Assume we have our hook: how do users join up? Other social networks are free, they have nice onboarding procedures, and they don't require you to do any deep thinking about anything other than your relationship status. By contrast, when you join Diaspora, you need to choose a "pod" based on its physical location, size, and software version before you're allowed to sign up. I am shocked — shocked — that this has not taken off.
Home-built social networks tend to be decentralized, and they're often proud of that fact: letting users choose where to put their data, and how it's used, is a huge win for privacy. But decentralized services are more complicated, and require more work from their users. It's even worse if people are expected to actually self-host: the most successful self-hosted web app on the Internet today is Wordpress, and yet being forced to install Wordpress on a cheap, randomly chosen hosting provider is punishment for shoplifting in some countries. Lesson two: web app installation shouldn't be a trial.
There's a lot of thought going into this problem — Docker, for example, is a system for baking web apps into "containers" that can be installed and uninstalled almost like mobile apps. These containers travel with all their dependencies, and configuration, so there's no need to worry about what your host does or doesn't support, or what their particular weird setup is. But until then, the answer for most people tends to be "use a hosted solution a la Wordpress.com," which tends to the defeat the purpose of decentralized software.
Without learning these lessons, cloud computing stays out of the hands of regular people, and the hope of a personal Facebook with it. Of course, they're hardly a panacea, and they're certainly not a solution for social networking anway. At this point, it almost doesn't make a difference what Facebook does, or how badly it abuses people. I think this is part of why people get so angry about it: that feeling of helplessness. We can't code our way out of this problem, or leave our friends and family behind. All we can do is hold our noses and soldier on.
As I mentioned last week, the project scaffolding I'm using for news apps at the Seattle Times has been open sourced. It assumes some proficiency with NodeJS, and is built on top of the grunt-init command.
There are many other newsrooms that have their own scaffolding: NPR has one, and the Tribune often builds its projects on top of Tarbell. Common threads include the ability to load data from CSV or Google Sheets, minifying and templating HTMl with that data, and publishing to S3. My template also does those things, but with some slight differences.
If you're working on newroom development, or static app development in general, please feel free to check it out, and I'd appreciate any feedback you might have.
My first interactive feature for the Seattle Times just went live: our Top Northwest Companies features some of the most successful companies from the Pacific Northwest. It's not anything mind-blowing, but it's a good start, and it helped me test out some of the processes I'm planning on using for future news applications. It also has a few interesting technical tricks of its own.
When this piece was originally prototyped by one of the web producers, it used an off-the-shelf library to do the parallax effect via CSS background positions. We quickly found out that it didn't let us position the backgrounds effectively so that you could see the whole image, partly because of the plugin and partly because CSS backgrounds are a pain. We thought about just dropping the parallax, but that bugged me. So I went home, looked around at how other sites (particularly Medium) were accomplishing similar effects, and came up with a different, potentially more interesting solution.
The table at the bottom of the page is written as an Angular app, and is kind of a perfect showcase for what Angular does well. Wiring up the table to be sortable and filterable was literally only a few minutes of work. The sparklines in the last column are custom elements, and Angular's filters make presenting formatted data a snap. Development for this table was incredibly fast, and the performance is really very good. There are still some issues with this presentation, such as the annoying sticky header, but it was by far the most painless part of the development process.
I'm not planning on using Dullard at the Times (sorry, Pat!) only because I want to be able to bring people onboard quickly. So I'm going with the standard Grunt task runner, but breaking up its tasks in a very Dullard-like way and using it to automate as much as possible. There's no hand-edited code in the Top Companies graphic — only templates and data merged via the build process. Reproducing these stories, or updating them later, is as simple as pulling the repo (or, in this case, both repos) and running the Grunt task again.
That simplicity also extends to the publication process. Fast deployment means fast development and fewer mistakes hanging out in the wild when bugs occur. For Seattle Times news apps, I'm planning to host them as flat files on Amazon S3, which is dirt-cheap and rock-solid (NPR and the Chicago Tribune use the same model). Running a deployment is as simple as grunt publish. In testing last night, I could deploy a fixed version of the page faster than people could switch to their browser and press refresh. As a client-side kind of person, I'm a huge fan of the static app model anyway, but the speed and simplicity of this solution exceeded even my expectations.
As I work on these story templates, I'll be open-sourcing them and sharing my ideas. The long and the short of it is that working in a newsroom is unpredictable: crazy deadlines, no requirements to speak of, and wildly different subject matter. This kind of technical architecture may seem unrelated to the act of journalism, but its goal is to lay the groundwork so that there are no distractions from the hard part: telling creative news stories online. I want to worry about making our online journalism better, not debugging servers. And while I don't know what the final solution for that is, I think we're off to a good start.
This quarter, I've been teaching ITC 240 at SCC, which is the first of three "web apps" classes. They're in PHP, and the idea is that we start students off with the basics of simple pages, then add frameworks, and finally graduate them to doing full project development sprints, QA and all. As the opening act for all this, I've decided to make a foundational part of the class focused on security.
I've done my best to cultivate paranoia in my students, both by telling them horror stories (the time that Google clicked all the delete links on a badly-hidden admin page, that time when the World Bank got hacked and replaced with pictures of Wolfowitz's socks) and by threatening to attack their homework every time I grade it. I'm not sure that it's actually working. I think you may need to be on the other end of something fairly horrific before it really sinks in how bad a break-in can be. The fact that their homework usually involves tracking personal information for my cat is probably not helping them take it seriously, either.
The thing is, PHP doesn't make it easy to keep users safe. There's a short tag for automatically echoing values out, but it does no escaping of HTML, so it's one memory lapse away from being a cross-site scripting bug. Why the <?= $foo ?> tag doesn't call htmlentities() for you like every other template engine on the planet, I'll never know. The result is that it's trivial to forget to sanitize your outputs — I myself forgot for an entire week, so I can hardly blame students for their slipups.
MySQL also makes this a miserable experience. Coming from a PostgreSQL background, I was unprepared (ha!) for this. Executing a prepared query in MySQL takes at least twice as many lines as in its counterpart, and is conceptually more difficult. You also can't quote table or column names in MySQL, which means that mysqli_real_escape_string is useless for queries with an ORDER BY clause — I've had to teach students about whitelists instead, and I suspect it's going in one ear and out the other.
It may be asking a little much of them anyway. Most of my students are still struggling with source control and editors, much less thinking in terms of security. Several of them have checked their passwords into GitHub, requiring a "password amnesty" where everyone got reset. I'd probably be more upset if I didn't think it was kind of funny, and if I wasn't pretty sure that I'd done the same thing in the past.
But even if they're a little bit overwhelmed, I still believe that students should be learning this stuff from the start, if for no other reason than that some of them are going to get jobs working on products that I use, and I would prefer they didn't give my banking information away to hackers in some godforesaken place like Cleveland. Every week, someone sends me a note to let me know that my information got leaked because they couldn't write a secure website — even companies like eBay, Dropbox, and Sony that should know better. We have to be more secure as an industry. That starts with introducing people to the issues early, so they have time to learn the right way as they improve their skills.