Over the Thanksgiving holiday, when I wasn't busily digesting as much cornbread stuffing as I could eat, I spent some time running WebPageTest against various projects that the Seattle Times Interactive team has built. The news industry as a whole may not care about speed, but I do, and I want our pages as fast as possible — especially the ones that are embedded in the regular CMS via responsive frames.
After all the testing, I'm generally pleased by how our stuff stacks up, especially when compared against the rest of the site. We have some advantages, of course: our pages typically have fewer ads, and we can strip down the page for maximum efficiency. But it's also the result of a lot of hard work on our news app template, ensuring that every project comes with smart decisions built in. I genuinely think that all news pages could be this fast, so it's worth talking about how we've made it happen, especially for other news organizations that use a similar flat-file approach to their interactives.
All of that is pretty standard best practice, but we've also learned that Browserify can be dangerous if you're not careful. A lot of NPM modules are published with the unminified, debug version of the library as the default export from the module. Angular in particular is bad about this: running require("angular") on its own will load a file filled with comments and documentation, totalling more than a megabyte in size (even after gzip, it's still more than 200KB). That's huge!
As a result, one of our production checklist items is to make sure that we are loading the minified version of any external libraries. We also use the browser property in our package.json file to alias common libraries to their minified versions, so that when we require Angular, jQuery, or Leaflet, it automatically defaults to the smallest file.
Like a lot of newsroom developers, my team hosts files on Amazon S3, mostly because it's cheap and reliable. People like to think about S3 as though it's just a normal, heirarchical flat-file server, like Apache or Nginx, but it's not. S3 is really a key-value store: you put in a path, and it spits back a prerecorded reply, including the headers.
If you think of S3 as a server, you'll expect it to do a bunch of things that it doesn't actually do. For example, it doesn't set a cache expiration date, and it doesn't know about content types. It also doesn't understand about Gzip compression, so it'll merrily serve your files in their uncompressed form, making them way bigger than they need to be, even if the browser requests the compressed version.
We get around this by running a compression stage on any text-based file during deployment, and setting the headers for the stored object to match. This does mean that theoretically, a browser that doesn't support Gzip will be unable to request that content, because S3 will always respond with compressed content no matter what Accepts-Encoding header the browser sends. Luckily, every browser since IE4 supports it.
I love Angular. If you want to quickly generate a visualization with powerful tools for filtering and data binding, you can't do much better. I personally think it's an order of magnitude better than D3. But Angular can also be brutally slow: its change detection algorithm requires a lot of time and memory as a tradeoff for developer convenience.
On a recent project that looked at animal imports, we started with Angular as a way to test out the visualization, but soon noticed that it was taking three or four seconds just to parse and apply the data. On a desktop, that time is a drag. On mobile, it's likely to get the tab terminated, or convince readers that there's something wrong with it.
Even jQuery can be optional these days. Because we compile ES6 down with Babel, a lot of DOM code that would be ungainly can become elegant. Template strings and arrow functions alone have allowed us to cut out DOM libraries entirely, and as a result many of our interactives consist of no external libraries at all. If you haven't checked into the advantages of using Babel in your build process, it's well worth another look.
The number one contributor to page load time is not written by journalists: it's the third-party ad code that runs on the page. There may be only so much you can do about this, since it pays the bills, and of course it may not even apply on embedded graphics. But on our standalone pages, I've taken a strong stance on implementing all code ourselves whenever possible. For example, although our commenting system usually requires multiple scripts loaded synchronously, I wrote a loader that runs through and adds them asynchronously, and only after a user clicks on the "view comments" banner. We can't avoid the hit, but we can delay it until well after the rest of the page has had a chance to render.
Once you've delayed scripts with the async attribute, trimmed the size of those scripts and compressed them, and deferred as much third-party code as you can, what's left over? In our case, this is where we start getting into the structure of the actual interactive, and how it loads itself. For most interactives, we embed data directly into the page, but beyond a certain size it becomes worthwhile to grab it via AJAX instead.
Finally, a note of caution: as much fun as it is to squeeze every last millisecond out of the browser, I'm a little uncomfortable making it the alpha and omega of the job. Ultimately, our goal is to inform people — we'd like that to be fast, but a fast page with bad or misleading reporting is still a failure.
What I like about front-end speed is that it serves as a useful proxy for site quality. A site that's fast can't load too many ads. It can't serve too many tracking scripts. It has to put the reader first. It's easy, much of the time, to chip away at performance in the name of business metrics: loading an additional analytics script to get more information, or an obnoxious ad for a short-term revenue boost.
But if you put speed first, every decision has to start from the perspective of "what's good for the reader?" It's hard to measure the impact of good journalism, but we can have metrics for speed and other technical aspects of the presentation. We can spend more time on the former if we have strong, user-centric guidelines on the latter. If we want people to give us money over the long term, that seems like the only healthy strategy to me.
Rachel Nabors writes that that you literally cannot pay her to attend a conference without a code of conduct:
When I promised not to go to conferences without Codes of Conduct, I wasn’t paying lip service to a trend, doing the popular thing to gain brownie points with my feminist besties. I meant every word. It is my greatest fear in life that something bad would happen to someone attending a conference I attracted them to.
It was weird the other day to realize that I'm actually becoming a mentor figure for some people: I have interns, I teach classes, I'm now the co-organizer of the local Hacks/Hackers chapter. I'm still basically nobody, but I will also make this pledge: if a conference does not have a code of conduct (or its equivalent), I won't go, as a speaker or an attendee. It's important to me that industry gatherings be places where people are comfortable and safe, no matter who they are. Everyone deserves that much consideration, and while a code of conduct is not a guarantee of safe behavior, it's a good start.
It's disappointing, but perhaps to be expected, that several high-profile white dudes have decided that the most important thing they can do this year is fight against codes of conduct. Many of them feel attacked and want to circle the wagons instead of listening to voices from the community. That's too bad for people who attend conferences — but part of the point of the pledge is that it should punish organizers who don't think creating a standard framework for conference behavior is a priority. If they can't get speakers or attendees to come, sooner or later they don't have an event.
Let's be clear: nobody is entitled to run a conference, and helping people with their careers once upon a time does not excuse you from being a good citizen. If the pressure causes people like Jared Spool and Mike Monteiro to change their mind, everyone benefits. If it doesn't, and they go the way of the dinosaurs, well... that's how evolution works. We'll survive without them.
But I think this is a great opportunity to re-examine one particular community role that I see a lot, both in tech and outside. You know the one: that guy (almost always a man) whose schtick is being the "honest teller of truth," which really means "rude, petty, and abusive in kind of a funny way about stuff we all agree on." A lot of times, we tolerate that behavior because honestly, it is satisfying to have someone saying what we're all thinking about clients who won't pay, or bad design, or ugly code. I have sometimes thought of myself as that person, but I'm trying not to be anymore.
The problem with the "angry truth-teller" is that it stops being funny when they suddenly turn those tools on their allies. When that happens, you realize that it was never actually funny: you just didn't care about being respectful, because you didn't think the target was worthy. Unfortunately, a lack of empathy is not the same as a sense of humor. There might be a thin line between "comedian" and "jerk," but unless you're Don Rickles I don't really see the point in intentionally crossing it.
In the end, life is too short to give money and attention to people who can't have a little empathy over something as silly as tech conference administration. We are not required to make them leaders in our community, nor are we required to keep them as leaders if we decide that the negatives of their input outweigh the positives. There are lots of technically-skilled people out there who can be right about an issue — or wrong on it, even — without being entitled, nasty jackasses about it. Let's boost them instead.
Earlier today I took the wraps off of the private repo for our Chambers Bay interactive flyover. You can find the source code here, and a post on our dev blog about it here. It was a really fun challenge, and a rare example of using WebGL in a news capacity (the NYT did the Dawn Wall, but that's the only one I can remember recently).
From a technical standpoint, this was my first three.js project, and the experience was largely positive. I think there's a strong case to be made that three.js is basically jQuery for WebGL: sure, you don't need it, but it only takes a couple of features to make it worthwhile. In this case, I didn't particularly feel like writing a model loader or a scene graph. There are still plenty of hooks to write the parts that I do enjoy, like the fragment shader for the landscape (check out that sweet dithering), or the UI for directing the camera. Sure, three.js is a relatively large library, but I'm loading 4MB of textures and another 4MB of gzipped landscape model, so what's a few more hundred KB of code?
WebGL itself runs surprisingly well these days, although failure modes do not seem to be its strong suit. For example, the browser may have WebGL support enabled, but then crash when it tries to render (or it may be lying about support, as with the remote VM sessions used in Times meeting rooms). That said, I was astonished to find that pretty much everything (mobile included) could run the landscape at a solid frame rate, despite the fact that it's a badly-optimized mesh with 150,000 triangles. Even iOS, which usually falls over and dies when WebGL pushes past its skimpy RAM limits, was able to run smoothly once I added a low-res texture for it to use.
This was an ambitious project using some pretty cutting-edge web technology, which makes it interesting in light of arguments that the web suffers from "featuritis". After all, when you're talking about feature overkill, WebGL is a barnstormer. But this story would have been tough to tell another way, and it would have never had the same reach siloed in an app store.
Or take Paul Ford's mind-boggling What is Code? in Businessweek this month: behind Bloomberg's Trapper Keeper design aesthetic, it's a powerful article that integrates animations, videos, and interactive demonstrations with the textual message. Ironically, I saw many of the same people that criticize web apps going wild for Ford's piece, a stance that I can only attribute to sophistry.
My team at the Seattle Times has gradually abandoned the term "news apps," since everyone who hears it assumes we're actually writing iPhone clients for the paper. As a term of art, it has always been clumsy. But it does strike at a crucial quality of what we do, which occupy a gray area between "text" and "program." And if it seems like I'm touchy about pundits who think we should abandon the web, this is in large part the reason why.
Arguments that browsers should just go back to being a document viewers ignore the fact that HTML is not just a text format: it's a hypermedia format, and those have always blurred the already-fuzzy lines between data and code (see also: Excel, Hypercard, and IPython notebooks). It's true that the features of the web platform are often abused. Nobody likes slow navigation, ad popups, or user tracking scripts. But it's those same features that make new kinds of storytelling possible — my journalism is built on the same heavily-structured, "over-tooled" web platform that critics find so objectionable. I wouldn't give that up for all the native apps in the world.
It's an accepted truth on the web that fast pages are better for users — people stay on them longer, follow more links from them, and generally report being happier with them. I think a lot about performance on my projects, because I want readers to be thinking about the story, not distracted by slow load times.
It's possible that I've been more aware of it, just because I've been working on a project that involves smoothly animating a chart using regular HTML instead of canvas, but it seems like it's been a bad month for that kind of thing. First Peter-Paul Koch wrote a diatribe about client-side templating, insisting that it's a needless performance hit. Then Flipboard wrote about discarding traditional elements entirely, instead rendering everything to a canvas tag in pursuit of 60 frames/second animations. Ironically, you'll notice that these are radically different approaches that both claim they create a better experience.
Instead of just sighing while the usual native app advocates use these posts to bash the web, and given that I am working on a page where high-performance mobile animations are a key part, I thought it'd be nice to talk about some experiments I've run with the approaches found in both. There are a lot of places where the web platform needs help competing on mobile, no doubt. But I'd prefer we talk about actual performance problems, and not get sidetracked into chasing down scattered criticisms without evidence.
At the other extreme is Flipboard's experiment with canvas rendering. Instead of putting everything in the document, like normal websites, they put a full-screen canvas image up and render everything — text, images, animations, etc. — manually to that buffer. You can try a demo out on your device here. On my Nexus 5, which is a reasonably new device running the latest version of Chrome, it's noticeably choppy. My experience with canvas is that Chrome's implementation is actually much faster than Safari, so I don't expect it to be smooth on iOS either (they've blacklisted tablets, so I can't be sure).
In order to get this "fluid" experience, here's what the Flipboard team threw away:
Again, I'm not claiming that my use case is a perfect analogue. I'm animating a graphic in response to a single button press, and they're attempting to create an "infinite scroll" (sort of — it's not really a scroll so much as an animated pager). But this idea that "the DOM is lava" and touching it will cause your reader's phones to instantly burst into flames of scorn seems patently ridiculous, especially when we look back at that list of everything that was sacrificed in the single-minded pursuit of speed.
Performance is important, and I care deeply and obsessively about it. As a gamer and a graphics nerd, I love tweaking out those last few frames per second, or adding flashy effects to a page. But it's not the most important thing. It's not more important than making your content available to the blind or visually impaired. It's not more important than providing standard UI actions like copy-and-paste or "open in new tab." And it's not more important than providing a fallback for older and less-powerful devices, the kind that are used by poor readers. Let's keep speed in perspective on the web, and not get so caught up in dogma that we abandon useful techniques like client-side templating and the DOM.
Similar to our news app template, I've put together a Grunt scaffolding for creating bundled custom elements, including HTML templating and CSS, all in a single standalone file. It's our component template — or, as I like to call it, the Poor Journalist's Polymer.
My second testbed project is a Leaflet map element that uses custom HTML to set the map configuration without ever writing a line of JSON (unless you really want to). It's intended to make mapping simple and fast for web producers, while still offering plenty of power for people like me who just want the boilerplate out of the way. Leaflet's a great candidate for this kind of declarative approach, and I think this is a really promising demo for the power of custom elements.
For standalone components like these, the template seems to be working well. I haven't yet solved the problem of easily embedding them in highly opinionated news apps, due to the way that dependencies are handled. It's useful for custom elements to be able to bundle their CSS and other assets into their package, similar to the way that HTML imports and shadow root offer embedded styles, but that means they may not integrate well into projects that already have their own build system. As far as I can tell, the best solution for now will probably be to load the packages from Bower and require() the standalone files from its build directory, which should work with whatever module system you like.
We came very close to using WebGL for a Seattle Times special report that will come out next week. Now that iOS 8 has shipped with support for WebGL, albeit in an unstable and slightly buggy form, it's common enough that I felt comfortable using it (with a scaled-down 2D fallback) for our audience. In the end, we went with a different design language and shelved the WebGL experiments, but the experience has left me very excited about the potential for mainstream usage.
There's an obvious parallel here, which is the first two major versions of Android. Because it was designed to run on low-end hardware, Android drew all its UI via software until 3.0 (and hardware acceleration didn't become widespread until 4.0). The resulting lag was never as bad as critics claimed, but it did mean that a lot of Android looked and felt a bit utilitarian. You wouldn't see something like Material Design emerge until the system supported using the GPU for rendering ordinary UI.
It's not a coincidence that Google's moving to Material Design on both Android and the web. Its design language — a smoothly-animated world of flat, geometric shapes — is attractive, but more importantly it's well-matched to the kinds of flat, geometric shapes that can be animated fluidly in a browser, using the 3D acceleration that's already built into the composition layer. Web Components will give developers a way to package those elements up, and make them reusable. Flexbox makes their layouts scalable and responsive.
But for the web platform to move forward, we need more than just a decent look and feel. We need the ability to write the kinds of applications that people insist that it can't run. WebGL is a step in that direction: graphics with near-native speed and capability, instantly deployed and paired with a surprisingly powerful UI toolkit. The kinds of apps and experiences we can write othe web, for a mainstream and mobile audience, just got a lot bigger. And I for one am looking forward to pushing those boundaries as much as I can.
Paul Kinlan's post, Add-to-homescreen Is Not What the Web Needs, is only the most recent in a long-running debate surrounding "apps" on mobile, but it is thought-provoking. Kinlan, who cheerleads for the Web Intents integration system in Chrome, naturally thinks that having an "add-to-homescreen" option misses the point:
I want to see something much more fundamental. The web offers something far richer: it encourages lightweight usage with no required installation and interaction with on-demand permissions. I never want to see an install button or the requirement to understand all the potential permissions requried before trying the app. The system should understand that I am using an app and how frequently that I use it and it should then automatically integrate with the launch points in the OS.
Kinlan has a great point, in that reducing the web to "just another app" is kind of a shame. The kinds of deeper integration he wants would probably be prone to abuse, but they're not at all impossible. Mozilla wants to do something similar with Firefox OS, although it probably gets lost in the vague muddle of its current state. Worse, Firefox OS illustrates the fundamental problem with web "apps" on mobile, and it's probably going to take a lot more than a clever bookmark to solve the problem. That's because the real problem with the web on mobile is URLs, and nobody wants to admit that.
As a web developer, I love URLs. They're the command line of the web: a powerful tool for organizing information and streaming it from place to place. Unfortunately, they're also like the command line in other ways: they're arbitrary, much-abused, and ultimately difficult to type on mobile. More importantly, nobody who isn't a developer really understands them.
There is a now-infamous example of the fact that people don't understand URLs, which you may remember as the infamous Facebook login of 2010. That was the point at which the web community realized that for a lot of users, logging into Facebook went a lot like this:
As a process, this was fine until ReadWriteWeb actually published a story about Facebook's unified login that rose to the top spot in the Google search listings, at which point hundreds of people began commenting on the article thinking that it was a new Facebook design. As long as they got to Facebook in the end, to these people, one skinny textbox was basically as good as another. I've actually seen people do this in my classes, and just about ground my teeth to nubs watching it happen.
In other words, the problem is discovery. An app store gives you a way to flip through the listings, see what's popular, and try it out. You don't need to search, and you certainly don't need to remember a cryptic address (all these clever .io and .ly addresses are, I'm pretty sure, much harder to remember than plain old .com). For most of the apps people use, they probably don't even scroll very far: the important stuff, like Facebook and Candy Crush, is almost certainly at the top of the store anyway. Creating add-to-homescreen mechanisms is addressing the wrong problem. It's not useless, but the real problem is not that people don't know how to make bookmarks, it's that they can't find your web app in the first place.
The current Firefox OS launcher isn't perfect, but it at least shows someone thinking about the problem. When you start the device, it initially shows a search box titled "I'm thinking of...". Tap into the box and even before you start typing it'll instantly start showing a set of curated sites sorted into categories like "social" and "games." If you want isn't there, you can continue to search the web as a whole. Sites launched from this view start in "app mode" with no URL bar, even though they're still just web sites and nothing's technically been installed. Press the bookmark button, and it's added to your homescreen. It's exactly as seamless as we've always claimed the web could be.
On top of this, sadly, Mozilla adds the Marketplace app, which can install "packaged" apps similar to Chrome OS. It's an attempt to solve the discoverability problem, but it lacks the elegant fluidity of the curated results from the launcher search (not to mention that it's kind of confusing). I'm not wild about curation at the best of times — app stores are a personal pet peeve — but it serves a purpose. We need both: an open web, because that's the spirit of things, and a market destination, because it solves the URL discovery process.
What we're left with is a tragedy of the commons. Mozilla's marketplace can't serve the purpose of the open web, because it's a curated and little-loved space that's only for Firefox OS users. Google is preoccupied with its own Chrome web store, even though it's certainly in a position to organically track the usage of web apps via user searches. Apple couldn't care less. In the meantime, web app discovery gets left with the scraps: URLs and search. There's basically no way, other than word of mouth, that your app will be discovered by normal people unless it comes from an app store. And that, not add-to-homescreen flaws, is why we can't have nice things on the web.
After a busy couple of weeks, Seattle went and won the Super Bowl, leading to the world's most polite celebration in our neighborhood:
There was another prize for the weekend: a friend of ours gifted us a Chromecast, which will be much appreciated since there's currently no way to watch HBO on the PS4. On Monday, Google released the public SDK for the platform, so I decided to poke around a bit.
Chromecast has a decidedly-odd way of loading content. The device itself is just a thin shell around a Chrome window, and it loads web pages like any other browser. But there's no keyboard of any kind, so how does it know which page to load? The answer is that each "app" has an ID listed with Google, corresponding to a set of URLs that the developer provides. When a mobile app or a computer running Chrome triggers the Chromecast, it sends the app ID, which the device then sends to Google and gets a URL in return (or, if the app hasn't been listed, it does nothing). From that point on, you can send messages to the page over via Google's cloud, and your page can do whatever you want it to do. Getting your pages linked to an application ID on the Chromecast lookup servers costs $5.
In the time since I last wrote about Caret, it's jumped up to 1.0 (and then 1.1). I've added tab memory, lots of palette search options, file modification watches, and all kinds of other features — making it legitimately comparable with Sublime. I've been developing the application using Caret itself since version 0.0.16, and I haven't really missed anything from native editors. Other people seem to agree: it's one of the top dev tools in the Chrome web store, with more than 1,500 users (and growing rapidly) and a 4.5/5 star rating. I'm beating out some of Google's own apps, at this point.
Belle's reaction: "Now you charge twenty bucks for it and make millions!" It's good to know one of us has some solid business sense.
Caret is already designed around message-passing for its internal APIs (as is Ace, the editing component I use), so it won't be too difficult to add external hooks, but it'll never have the same power as something like Sublime, which embeds its own Python interpreter. I can understand why Google made the security decisions they did, but I wish there was a way to relax them in this case.
I figure I have roughly six months to a year before Caret has any serious competition on Chrome OS. Most of the other editors aren't interested in offline editing or are poorly-positioned to do so for architectural reasons. The closest thing to Caret from the established players would be Brackets, which still relies on NodeJS for its back-end and can't yet run the front-end in a regular browser. They're working on the latter, and the former will be shimmable, but the delay gives me a pretty good head start. Google also has an app in the works, but theirs looks markedly less general-purpose (i.e. it seems aimed specifically at people building Chrome extensions only). Mostly, though, it's just hard to believe that someone hadn't really jumped in with something before I got there.
Between Caret, Weir, and my textbook, this has been a pretty productive year for me. I'm actually thinking that for my next project I may write another short book — one on writing Chrome apps using what I've learned. The documentation from Google's end is terrible, and I hate to think of other people having to stumble through the APIs the way I did. It might also be a good way to get some more benefit out of the small developer community that's forming around Caret, and to find out if there's actually a healthy market there or not. I'm hoping there is: writing Caret has been fun, and I'd to have the chance to do more of this kind of development in the future.
At this point, Caret has been in the Chrome Web Store for about a week. I think that's long enough to say that the store is a pretty miserable experience for developers.
When I first uploaded it last week, Caret had these terrible promo tiles that I threw together, mostly involving a big pile of carrots (ba dum bum). At some point, I made some slightly less terrible promo tiles, stripping it down to just bold colors and typography. Something set off the store's automated review process, and my new images got stuck in review for four days — during which time it was stuck at the very bottom of the store page and nobody saw it.
On Tuesday, I uploaded the first version of Caret that includes the go-to/command palette. That release is kind of a big deal--the palette is one of the things that people really love about Sublime, and I definitely wanted it in my editor. For some reason, this has triggered another automatic review, this one applied to the entire application. I can unpublish Caret, but I can't edit anything — or upload new versions — until someone checks off a box and approves my changes. No information has been provided on why it was flagged, or what I can do to prevent these delays in the future.
Even at the best of times, the store takes roughly 30 minutes to publish a new version. I'm used to pushing out changes continuously on the web, so slow updates drive me crazy. Between this and the approval hijinks, it feels like I'm developing for iOS, but without the sense of baseless moral superiority. What makes it really frustrating is the fact that the Play store for Android has none of these problems, so I know that they can be solved. There's just no indication that the Chrome team cares.
I was planning on publishing a separate, Google-free version of the app anyway, so I worked out how to deploy a standalone .crx file. The installation experience for these isn't great — the file has to be dragged onto the Chrome extensions list, and can't just be installed from the link — and it introduces another fun twist: even though they promised it would be possible for years, there's no way to download the private key that Google uses in the Chrome store, meaning that the two installations are treated as completely different applications when installed.
Fair enough: I'll just make the standalone version the "edge" release with a different icon, and let the web store lag behind a little bit. As a last twist of the knife, generating a .crx package as part of a process that A) won't include my entire Git history, and B) will work reliably across platforms, is a nightmare. Granted, this is partly due to issues with Grunt, but Chrome's not helping matters with its wacky packaging system.
All drama aside, everything's now set up in a way that, if not efficient, is at least not actively harmful. The new Caret home page is here, including a link to the preview channel file (currently 3 releases ahead of the store). As soon as Google decides I'm not a menace to society, I'll make it the default website for the store entry as well.
The problems with Google's web store bug me, not so much because they're annoying in and of themselves, but because they feel like they miss the point of what packaged web apps should be. Installing Caret is fast, secure, and easy to update, just as regular web apps are. Developing Caret, likewise, is exactly as easy and simple as writing a web app (easier, actually: I abuse flexbox like crazy for layout, because I know my users have a modern browser). Introducing this opaque, delay-ridden publication step in between development and installation just seems perverse. It won't stop people from using the store (if nothing else, external installation is too much of a pain not to go through the official channel), but it's certainly going to keep me from enjoying it.