this space intentionally left blank

October 3, 2014

Filed under: tech»web

WebGL and beyond

We came very close to using WebGL for a Seattle Times special report that will come out next week. Now that iOS 8 has shipped with support for WebGL, albeit in an unstable and slightly buggy form, it's common enough that I felt comfortable using it (with a scaled-down 2D fallback) for our audience. In the end, we went with a different design language and shelved the WebGL experiments, but the experience has left me very excited about the potential for mainstream usage.

It's probably easier to understand why WebGL is exciting by looking at what the regular 2D canvas does badly. 2D canvas is terrible at combining or masking its rendering functions (globalCompositeOperation in particular is dog-slow in Firefox). It doesn't give users easy access to the image data directly, which is frustrating in a bitmap-based drawing API. It doesn't like changing colors or styles frequently. But its biggest weakpoint is actually that it's tied so closely to JavaScript, which is a single-threaded language running in the browser event loop. The more pixels you touch in detail, the slower it gets.

WebGL, by contrast, is great at blending, filtering, and masking. But most importantly, WebGL code moves most (if not all) of the math-heavy graphics code you'd normally write in JavaScript — scaling and transforms, patterns, and color — over to the GPU. Your graphics card is a massive parallel-processing machine, so all your drawing occurs simultaneously, not sequentially. You can alter every pixel in the frame, if you want, and it'll barely take any more time than if you change only a few.

Once I spent a little time writing some simple shaders, I realized that there's a whole range of experiences you can write in WebGL that simply aren't possible on a 2D canvas. I could shift the colors or custom-filter an image on a per-pixel basis. I wrote a dust simulation that simulated thousands of motes on a low-end machine, even with the physics still running in JavaScript. I even created a faux-3D effect a la Depthy, by displacing each pixel by the value of a second texture's lightness and the mouse position.

None of these experiments involved 3D math in any way. They're not spinning teapots, or Unreal Engine demos, or elaborate parallax effects. I suspect that the real value of WebGL isn't going to be from any of those things. It's going to be the fact that it gives the web platform the free-drawing capability of canvas, but uncoupled from the JavaScript execution model that it's been shackled to.

There's an obvious parallel here, which is the first two major versions of Android. Because it was designed to run on low-end hardware, Android drew all its UI via software until 3.0 (and hardware acceleration didn't become widespread until 4.0). The resulting lag was never as bad as critics claimed, but it did mean that a lot of Android looked and felt a bit utilitarian. You wouldn't see something like Material Design emerge until the system supported using the GPU for rendering ordinary UI.

It's not a coincidence that Google's moving to Material Design on both Android and the web. Its design language — a smoothly-animated world of flat, geometric shapes — is attractive, but more importantly it's well-matched to the kinds of flat, geometric shapes that can be animated fluidly in a browser, using the 3D acceleration that's already built into the composition layer. Web Components will give developers a way to package those elements up, and make them reusable. Flexbox makes their layouts scalable and responsive.

But for the web platform to move forward, we need more than just a decent look and feel. We need the ability to write the kinds of applications that people insist that it can't run. WebGL is a step in that direction: graphics with near-native speed and capability, instantly deployed and paired with a surprisingly powerful UI toolkit. The kinds of apps and experiences we can write othe web, for a mainstream and mobile audience, just got a lot bigger. And I for one am looking forward to pushing those boundaries as much as I can.

Future - Present - Past