this space intentionally left blank

February 27, 2014

Filed under: tech»coding

Just Use Ed

There's a regular, recurring movement to replace text-based programming with some kind of graphical version. These range from Scratch (offering "blocks" to make text syntax more friendly) to Pure Data (node-based dataflow programming). Rarely do any of them take off (Scratch and pd are successful within education and audio, respectively, but little-used elsewhere), but that doesn't stop anyone from trying.

It may be the fact that I started as a writer, or that I was a language nut in college, but I've always felt that text-based programming doesn't get a lot of respect. The written word is one of the great advances of civilization. You can pack a lot of meaning into a line of text, and code is no different. Good source code can range from whimsical to workmanlike, a gamut that's hard to imagine existing in the nest of wiring that is the graphical languages.

As a result, text editing is important to me. It's important to a lot of people, but most of them don't write an editor, and I ended up doing that. I figured I'd write up some notes on the different ways people have written their editors, and why I picked one model in particular for Caret. It may be news to many people that there are even multiple models to consider, but that's programming for you: there's at least four ways to put letters into a document, and bitter wars between factions for each of them.

The weirdest editor still in common usage, of course, is Vim. Born from the days when network connections were too slow to actually update text in realtime, Vim uses a shorthand language for text editing. You don't hold delete until some amount of text is gone in Vim — instead, you type "d2w", meaning "delete two words." You also can't type directly until you switch into the "insert" mode with the "i" or "a" commands. Like many abusive subcultures, people who learn this shorthand will swear up and down that it's the only way to work, even though it's clearly a relic of a savage, bygone age.

(Vim and Emacs are often mentioned in comparison to each other, because they tend to be used by very similar kinds of people who, nevertheless, insist that they're very different. I don't really know very much about Emacs, other than it's written in Lisp and it's not as eyeball-rolling weird as Vim, so I'm ignoring it for the purposes of this discussion.)

Acme tends to look a little more traditional, but it is actually (I think) more radical than Vim, because it redefines the relationship between interface and editor. Acme turns all documents into hypertext: middle clicking a filename opens that file, and clicking a word (like "copy" or "paste") actually runs that command (either in a shell, or in Acme). There's no fixed interface in Acme, just a set of menu bars that are also text fields. I love the elegance of this idea, where a person builds an text editor's UI just by... editing text.

Which brings us to Sublime. I've been very clear that Caret is modeled closely on Sublime, with a few changes to account for quirks of the platform and my own preferences. That's partly because it's generally considered the tool of choice for web developers, and partly because it's genuinely the editor that has my favorite workflow tools. Insofar as Sublime has a philosophy, it is to prioritize clarity and transparency over power. That's not to say it's not powerful — it certainly is. But it tries to be obvious in a way that other editors do not.

For example, say you need to change a variable name throughout a function. Instead of immediately writing a regex or a macro, Sublime lets you select all the instances of that variable with the mouse or keyboard, which creates multiple cursors. Then you just type the new name. It's not as powerful as a regular expression, but 90% of the time, it's probably what you wanted to do anyway. Sublime's command/go-to palette is another smart-but-obvious idea: instead of hunting through the menu or the filesystem, open the palette and type to fuzzy-filter the list. It's the speed of a command line without the hostility.

To paraphrase an old saw, the best feature is the one you have with you. That's why putting the command palette in Caret was a must, since it puts all the menu items just a few keystrokes away. Even now, I don't always remember where a given menu item is in the toolbar in my own editor, because I hardly ever use the mouse. There was a good week when menus looked completely wrong, and I never even noticed.

The reason I've started looking over other editors now is that I think Caret can reach for more than just parity with Sublime. I'm intrigued by the ways that Acme makes it easy to jump around files, and lately I've been thinking about what it means to be an editor built in "web technology." Adding the ability to open links from a URL is a given, but it's only the start: given that OAuth provides a simple, standard method of authenticating against a remote server, a File implementation for Caret could easily open files against service endpoints for something like Github or Ghost in a generic way. It would be a universal cloud editor, but easily capable of running locally.

Of course, Caret won't be the last editor to try something different (just this week, Github announced their own effort), but it's still pretty amazing how many ways we have to solve a simple problem like "typing letters into a file." As a writer and a coder, I love being spoiled for choice.

February 6, 2014

Filed under: tech»web

Chromecastic

After a busy couple of weeks, Seattle went and won the Super Bowl, leading to the world's most polite celebration in our neighborhood:

There was another prize for the weekend: a friend of ours gifted us a Chromecast, which will be much appreciated since there's currently no way to watch HBO on the PS4. On Monday, Google released the public SDK for the platform, so I decided to poke around a bit.

Chromecast has a decidedly-odd way of loading content. The device itself is just a thin shell around a Chrome window, and it loads web pages like any other browser. But there's no keyboard of any kind, so how does it know which page to load? The answer is that each "app" has an ID listed with Google, corresponding to a set of URLs that the developer provides. When a mobile app or a computer running Chrome triggers the Chromecast, it sends the app ID, which the device then sends to Google and gets a URL in return (or, if the app hasn't been listed, it does nothing). From that point on, you can send messages to the page over via Google's cloud, and your page can do whatever you want it to do. Getting your pages linked to an application ID on the Chromecast lookup servers costs $5.

Five dollars is a low price, but it's more than I really want to pay for a glorified DNS. I'm a little dismayed by the restrictions on the open web — I'd like the option to just send a URL directly. I'm also holding out for a pure JavaScript API, instead of piggybacking on the Chrome extension. So I probably won't be writing any Chromecast apps any time soon. But it's certainly not for a lack of ideas. The interaction model that Chromecast uses — where the screen is just a dumb display, but it can receive commands from other web-accessible devices — is strikingly similar to Microsoft's SmartGlass model. And where Microsoft seems to see it as a way to create companion apps for XBox, I think it's interesting to think about how this "distributed I/O" model could be used for standalone applications.

  • The Chromecast isn't going to rival any consoles, but it wouldn't have to be for a lot of group gaming experiences. Just having a screen that could be used as a scoreboard, or a trivia question where phones are used as buzzers, would be a cool usage that doesn't require precise controls or rich graphics. Turn-based games could easily use the screen as a board overview, while letting people zoom in and move their pieces from their local touchscreen. It also provides an interesting split between public and private information for players that many video games (excepting the Wii U and Dreamcast) couldn't duplicate.
  • I love maps. I think they're the real face of augmented reality, as any regular traveler can attest these days. But they don't have to be mobile. A Chromecast could easily serve as a map up on your wall, updated with whatever information you find interesting. Maybe that's as simple as the weather, but imagine being able to tag it with RFID information or last-known positions for people in your household. Systems like Google Now, which learn from your schedule, could even post notifications for the buses that are coming or traffic problems that you're likely to face.
  • Along those same lines, a simple dashboard could be helpful for businesses and individuals. Being able to throw metrics up on the wall with a web browser is not a new thing, but tying it to a smart, feed-aware service would open up all kinds of new tricks, like being able to leave yourself notes via a hashtag on social networks. There's not really any input needed: it's just a passive display of whatever you want to keep yourself caught up on, in an easy at-a-glance format.
  • Finally, it's probably just all the public speaking I've been doing lately, but it's tempting to think that a presentation app for Chromecast would be super-helpful for speakers. A lot of times, when I go to a meetup or a new classroom, it's hard to predict what kind of video hookups the projector will have, assuming that they even have a projector. But many times, there will be a big-screen LCD TV, with a handy HDMI input. Being able to carry a Chromecast with me to make my presentations, especially if the speaker notes can be viewed separately, would be awesome.
When we talk about the web being device-agnostic, the Chromecast is a perfect example of what we're talking about. It's radically different from other web clients: low DPI on a big screen, no local input, and unpredictable performance. But that's the power of the platform — as a toolkit, its reach is unparalleled. And the restrictions prove to be exciting inspiration for new uses, just as touchscreens came with their own unique challenges and advantages. I don't know if Chromecast is going to be successful, but the hacks for it are going to be really interesting.

November 22, 2013

Filed under: tech»coding

Plug In, Turn On, Drop Out

This is me, thinking about plugins for Caret, as I find myself doing these days. In theory, extensibility is my focus for the next major release, because I think it's a big deal and a natural next step for a code editor. In practice, it's not that simple.

Approach #1: Postal Services

Chrome has a pretty tight security model applied to packaged apps, not the least of which is a strict content security policy. You can't run code from outside your application. You can't construct code using eval (that's good!) or new Function (that's bad). You can't add new files to your application (mostly).

Chrome does expose an inter-app messaging system similar to postMessage, and I initially thought about using this to create a series of hooks that external applications could use. Caret would broadcast notifications to registered listeners when it did something, and those listeners could respond. They could also trigger Caret commands via message (I do still plan to add this, it's too handy not to have).

Writing plugins this way is neatly encapsulated and secure, but it's also going to be intensely frustrating. It would require auditing much of Caret's code to make sure that it's all okay with asynchronous operation, which is not usually the case right now. I'd have to make sure that Caret is sufficiently chatty, because we'd need hooks everywhere, which would clutter the code with broadcast/response blocks. And it would probably mean writing a helper app to serve as a patchboard between applications, and as a debugging tool.

I'm not wild about this one.

Approach #2: Repo, Man

I've been trying to think of a way around the whole inter-app messaging paradigm for about a month now. At the same time, I've been responding to requests for Git and remote filesystem support, which will not be a core Caret feature. For some reason, thinking about the two in close proximity started me thinking along a new track: what if there were a way to work around the security policy using the HTML5 file system? I decided to run some tests.

It turns out this is absolutely possible: Chrome apps can download a script from any server that's whitelisted in their manifest, write that out to the filesystem, and then get a special URL to load that file into a <script> tag. I assume this has survived security audits because it involves jumping through too many hoops to be anything other than deliberate.

The advantages of this approach are numerous. Plugin code would operate directly alongside of Caret's source, able to access the same functions and modules and call the same APIs that I use. It would be powerful, and would not require users to publish plugins to the Chrome store as if they were full applications. And it would scale well--all I would need to do is maintain the index and provide some helper functions for developers to use when downloading and caching their code.

Unfortunately, it is also apparently forbidden by the Chrome Web Store policies, which state:

Packaged apps should not ... Download or execute scripts dynamically outside a sandboxed environment such as a webview or a sandboxed iframe.
At that point, we're back to postMessage unless I want to be banned from the store. So much for the workaround.

Approach #3: Local hosting

So how can I make plugins work for end users? Well, honestly, maybe I don't. One of the nice things about writing developer tools, particularly oddball developer tools, is that the people using them and wanting to expand on them are expected to have some degree of technical knowledge. They can be trusted to figure out processes that wouldn't necessarily be acceptable for average computer users. In this case, that might mean running Caret as an unpacked app.

Loading Caret from source is not difficult--I do it all the time while I'm testing. Right now, if someone wants to fork Caret and add their own features, that's easy enough to do (and actually, a couple of people have done so already). What it lacks is a simple entry point for people who want to contribute functionality without digging into all the modules I've already written.

By setting up a plugins directory and a little bit of infrastructure, it's possible to reach a middle ground. Developers who really want extra packages can load Caret from source, dump their code into a designated location, and have their code bootstrapped automatically. It's not as friendly as having web store distribution, and it's not as elegant as allowing for a central repo, but it does deliver power without requiring major rewrites.

Working through all these different approaches has given me a new appreciation for insecurity, which sounds funny but it's true. Obviously I'm in favor of secure computing, but working with mobile operating systems and Chrome OS that strongly sandbox their code tends to make a person aware of how helpful a few security holes can be, and vice versa: the same points for easy extension and flexibility are also weak points that can be exploited by an attacker. At times like this, even though I should maybe know better, that tradeoff seems absolutely worth it.

November 14, 2013

Filed under: tech»coding

For Free Ninety Nine I'll Beat 99 Acts Down

Assuming that the hamster powering the Chrome web store stats is just resting, Caret clicked over to 10,000 installations sometime on Monday. That's a lot of downloads. At a buck a piece, even if only a fraction of those people had bought a for-pay version, that might be a lot of money. So why is Caret free? More importantly, why is it free and open source? Ultimately, there are three reasons:

  1. I feel like I owe the open source community for the value I've gotten from it (basically, everything on the Internet), and this is a way to repay that debt.
  2. Caret isn't really just mine. It's heavily influenced by Sublime, and builds on another open source project for its text processing. As such, it feels awkward to charge money for other peoples' work, even if Caret's unique code is significant in its own right.
  3. I think I get more value (i.e. job marketability, reputation, skill practice) out of being the person with a chart-topping Chrome app, in the long term, than I would get from sales.

Originally, I had planned on writing about how I reconcile being a passionate supporter of paid writing while giving away my hobby code, but I don't actually see any conflict. I expect a paycheck for freelance coding the same way I expect it for journalism — writing here (and coding Caret) doesn't directly benefit anyone but me, and it doesn't really cost me anything.

In fact, it turns out that both industries also share some uncomfortable habits when it comes to labor. Ashe Dryden writes:

Statistically, we expect that the demographic breakdown of people contributing to OSS would be about the same as the people who are participating in the OSS community, but we aren't seeing that. Ethnicity of computing and the US population breaks down what we would hope to see as far as ethnicity goes. As far as gender, women make up 24% of the industry, according to the same paper that gave us the 1.5% OSS contributor statistic.

Dryden was responding to a sentiment that I've seen myself (and even been guilty of, from time to time): using a person's open source record on sites like GitHub as a proxy for hireability. As she points out, however, building an open source portfolio is something that's a lot easier for white men. We're more likely to have free time, more often have jobs that will pay for open source contributions, and far less likely to be harassed or dismissed. I was aware of those factors, but I was still shocked to see that diversity numbers in open source are so low. We need to do better.

As eye-opening as that is, I think Dryden's middle section centers around a really interesting question: who profits?

I'd argue that the people who benefit the most from the unpaid labor of OSS as well as the underpaid labor of marginalized people in technology are business owners and stakeholders in these companies. Having to pay additional hundreds of thousands or millions of dollars for this labor would mean smaller profit margins. Technology is one of the most profitable industries in the US and certainly could support at least pay equality, especially considering how low our current participation is from marginalized people.

...Open source originally broke us free from the shackles of proprietary software which forced us to "pay to play" and gave us little in the way of choices for customization. Without realizing it, we've ended up in a similar scenario where we are now paying for the development of software that large companies financially benefit from with little cost to them.

Her conclusion — that the community benefits, but it's mostly businesses who boost their profits from free software — should be unsettling for anyone who contributes to open source, and particularly those of us who see it as a way to spread a little socialist good will. For this reason, if nothing else, I'll always prefer the GPL and other "copyleft" licenses, forcing businesses to play ball if they want to use my code.

November 5, 2013

Filed under: tech»innovation

Patently Nonsense

Now that Apple and Microsoft (with a few others) have formed a shell company in order to sue Google over a bunch of old Nortel patents, and IBM has accused Twitter of infringing on a similar set of bogus "inventions," it's probably time again to talk about software patents in general.

Software Patents: Frequently Asked Questions

Q. When it comes to software patents...

A. No.

Q. Hang on. Are they a valid thing? Are any of these suits valid?

A. See above: no, and no.

Q. Why not?

A. Because software patents, as a description of software, are roughly like saying that I could get a patent for the automobile by writing "four tires and an engine" on a piece of paper. They're vague to the point of uselessness, and generally obvious to anyone who thinks about a problem for more than thirty seconds.

Q. Yeah, but so what? Isn't the point of patents to create innovation? Haven't we had lots of innovation in software thanks to their protection?

A. True, the point of patents is to make sure that people let other people use their inventions in exchange for licensing fees, which is supposed to incentivize innovation. In patents for physical inventions, that makes sense: I need to know how to build something if I want to use it as a part of my product, and the patent application will tell me how to do that. But software patents are not descriptive in this way: nobody could write software based on their claims, because they're written in dense legal jargon, not in code.

Let's take one of the patents in question, IBM's "Programmatic discovery of common contacts." This patent covers the case where you have two contact lists for separate people, and you'd like to find out which contacts they have in common. It describes:

  1. Having one local list of contacts,
  2. getting a second from the server,
  3. comparing them, and
  4. presenting a list of hyperlinks to the user.
That is literally the extent of detail present in this patent. Now, do you think you can build a contact matching system by licensing that patent? Or will you do what every other software company on earth has done, and invent the stupid thing yourself (a task that would take a decent coder a couple of days)?

As for the innovation argument, it's impossible to prove a negative: I can't show you that it wasn't increased by patents. But consider this: most of the companies that we think of as Internet innovators are strikingly low on patent holdings. Reportedly, Twitter owns nine, and has pledged not to sue anyone over them. Google's only applied for a few, although they have purchased many as a defensive tactic. Facebook is not known for licensing them or taking them to court (indeed, just the opposite: enter the Winklevii). For the most part, patent litigation is limited to companies who are either no longer trailblazers (Microsoft), are trying to suppress market competition (Apple), or don't invent anything at all (Intellectual Ventures). Where's the innovation here?

This American Life actually did a pair of shows on patents, strongly arguing that they've been harmful: companies have been driven out of business by patent trolls. Podcasters have been sued for the undeniably disruptive act of "putting sound files on the Internet." The costs to the industry are in the billions of dollars, and it disproportionately affects new players — exactly the kind of people that patents are meant to protect.

Q. So there's no such thing as a valid software patent?

A. That's actually a really interesting question. For example, last week I was reading The Code Book, by Simon Singh. Much of the book is taken up by the story of public key encryption, which underlies huge swathes of information technology. The standard algorithm was invented by a trio of researchers: Ron Rivest, Adi Shamir, and Len Adleman. As RSA, the three patented their invention and successfully licensed it to software firms all over the world.

The thing about the RSA patent is that, unlike most software patents, it is non-obvious. It's extremely non-obvious, in fact, to the degree that Rivest, Shamir, and Adleman literally spent years thinking about the problem before they invented their solution, based on even more years of thinking on key exchange solutions by Whitfield Diffie, Martin Hellman, and Ralph Merkle. RSA is genuinely innovative work.

It is also work that was independently invented by the espionage community several years before (although obviously they weren't allowed to apply for patents). Moreover, a lot of the interesting parts of RSA are in the math, and math is not generally considered patentable. Nevertheless, if there's anything that would be a worthy software patent, RSA should qualify.

It goes without saying that matching contacts, or showing search ads, or scrolling a list based on touch, are not quite in the same league. And it's clear that patents are not creators of value in software. People aren't buying Windows computers or iPhones because they're patented. They're buying them because they run the software people want, or run on good-looking hardware, or any number of other reasons. In other words: software is valuable because of what it does, not because of how.

Q. So software shouldn't have any protection?

A. Sure, software should be protected. In fact, it already is. Code can be copyrighted, and you can sue people who take it and use it without your permission. But that's a different matter. Copyright says you can sue people who publish your romance novel. Software patents would be like suing anyone who writes about boy-meets-girl.

Q. Okay, fine. What's the answer?

A. Ultimately, we need patent reform. The steps for this are the same as any other reform, unfortunately: writing letters, asking your political candidates, and putting pressure on the government. It's not a sexy recommendation, but it's effective. If we could frame these as another type of frivolous lawsuit, the issue may even get some traction.

Personally, though, I'm trying to vote with my wallet. I'd like to not give money to companies that use patents offensively. Incidentally, this is why I'm cautiously optimistic for Valve's Steam Machines: it's much harder for me to not give money to Microsoft, since I play a lot of games on Windows (and XBox). A Linux box with a good game library and a not-terrible window manager would make my day.

Finally, there's a community site that can help. Ask Patents is a forum set up by the people who run the Stack Overflow help group for programmers. It takes advantage of a law that says regular people can submit "prior art" — previously-existing examples of the patented invention — that invalidate patents. Ask Patents has successfully blocked bad software patents from being granted in the first place, which means that they can't be used for infringement claims. Over time, success from finding prior art makes patents more expensive to file (because they're not automatically granted), which means fewer stupid ones will be filed and companies will need to compete in the market, not the courtroom.

October 21, 2013

Filed under: tech»web

Caret 1+

In the time since I last wrote about Caret, it's jumped up to 1.0 (and then 1.1). I've added tab memory, lots of palette search options, file modification watches, and all kinds of other features — making it legitimately comparable with Sublime. I've been developing the application using Caret itself since version 0.0.16, and I haven't really missed anything from native editors. Other people seem to agree: it's one of the top dev tools in the Chrome web store, with more than 1,500 users (and growing rapidly) and a 4.5/5 star rating. I'm beating out some of Google's own apps, at this point.

Belle's reaction: "Now you charge twenty bucks for it and make millions!" It's good to know one of us has some solid business sense.

The next big milestones for Caret will focus on making it better at workflow, especially through plugin integration. Most editors go from "useful" to "essential" when they get a way for users to extend their functionality. Unfortunately, the Chrome security model makes that much more difficult than it is with compiled binaries: packaged apps aren't allowed to expose their internals to external JavaScript, either through script tags or eval(). The only real extension mechanism available is message-passing, the same as web workers.

Caret is already designed around message-passing for its internal APIs (as is Ace, the editing component I use), so it won't be too difficult to add external hooks, but it'll never have the same power as something like Sublime, which embeds its own Python interpreter. I can understand why Google made the security decisions they did, but I wish there was a way to relax them in this case.

I figure I have roughly six months to a year before Caret has any serious competition on Chrome OS. Most of the other editors aren't interested in offline editing or are poorly-positioned to do so for architectural reasons. The closest thing to Caret from the established players would be Brackets, which still relies on NodeJS for its back-end and can't yet run the front-end in a regular browser. They're working on the latter, and the former will be shimmable, but the delay gives me a pretty good head start. Google also has an app in the works, but theirs looks markedly less general-purpose (i.e. it seems aimed specifically at people building Chrome extensions only). Mostly, though, it's just hard to believe that someone hadn't really jumped in with something before I got there.

Between Caret, Weir, and my textbook, this has been a pretty productive year for me. I'm actually thinking that for my next project I may write another short book — one on writing Chrome apps using what I've learned. The documentation from Google's end is terrible, and I hate to think of other people having to stumble through the APIs the way I did. It might also be a good way to get some more benefit out of the small developer community that's forming around Caret, and to find out if there's actually a healthy market there or not. I'm hoping there is: writing Caret has been fun, and I'd to have the chance to do more of this kind of development in the future.

September 12, 2013

Filed under: tech»web

Beta Caret-ene

At this point, Caret has been in the Chrome Web Store for about a week. I think that's long enough to say that the store is a pretty miserable experience for developers.

When I first uploaded it last week, Caret had these terrible promo tiles that I threw together, mostly involving a big pile of carrots (ba dum bum). At some point, I made some slightly less terrible promo tiles, stripping it down to just bold colors and typography. Something set off the store's automated review process, and my new images got stuck in review for four days — during which time it was stuck at the very bottom of the store page and nobody saw it.

On Tuesday, I uploaded the first version of Caret that includes the go-to/command palette. That release is kind of a big deal--the palette is one of the things that people really love about Sublime, and I definitely wanted it in my editor. For some reason, this has triggered another automatic review, this one applied to the entire application. I can unpublish Caret, but I can't edit anything — or upload new versions — until someone checks off a box and approves my changes. No information has been provided on why it was flagged, or what I can do to prevent these delays in the future.

Even at the best of times, the store takes roughly 30 minutes to publish a new version. I'm used to pushing out changes continuously on the web, so slow updates drive me crazy. Between this and the approval hijinks, it feels like I'm developing for iOS, but without the sense of baseless moral superiority. What makes it really frustrating is the fact that the Play store for Android has none of these problems, so I know that they can be solved. There's just no indication that the Chrome team cares.

I was planning on publishing a separate, Google-free version of the app anyway, so I worked out how to deploy a standalone .crx file. The installation experience for these isn't great — the file has to be dragged onto the Chrome extensions list, and can't just be installed from the link — and it introduces another fun twist: even though they promised it would be possible for years, there's no way to download the private key that Google uses in the Chrome store, meaning that the two installations are treated as completely different applications when installed.

Fair enough: I'll just make the standalone version the "edge" release with a different icon, and let the web store lag behind a little bit. As a last twist of the knife, generating a .crx package as part of a process that A) won't include my entire Git history, and B) will work reliably across platforms, is a nightmare. Granted, this is partly due to issues with Grunt, but Chrome's not helping matters with its wacky packaging system.

All drama aside, everything's now set up in a way that, if not efficient, is at least not actively harmful. The new Caret home page is here, including a link to the preview channel file (currently 3 releases ahead of the store). As soon as Google decides I'm not a menace to society, I'll make it the default website for the store entry as well.

The problems with Google's web store bug me, not so much because they're annoying in and of themselves, but because they feel like they miss the point of what packaged web apps should be. Installing Caret is fast, secure, and easy to update, just as regular web apps are. Developing Caret, likewise, is exactly as easy and simple as writing a web app (easier, actually: I abuse flexbox like crazy for layout, because I know my users have a modern browser). Introducing this opaque, delay-ridden publication step in between development and installation just seems perverse. It won't stop people from using the store (if nothing else, external installation is too much of a pain not to go through the official channel), but it's certainly going to keep me from enjoying it.

September 5, 2013

Filed under: tech»web

Caret

As I mentioned in my Chromebook notes, one of the weak points for using Chrome OS as a developer is the total lack of good graphical editor. You can install Crouton, which lets you run Vim from the command line or even run a full graphical stack. But there aren't very many good pure text editors that run within Chrome OS proper — most of the ones that do exist are tied to hosted services like Cloud9 or Nitrous. If you just want to write local files without a lot of hassle, you're out of luck.

I don't particularly want to waste what little RAM the Chromebook has running a whole desktop environment just for a notepad, and I'm increasingly convinced that Vim is a practical joke perpetuated by sadists. So I built the Chrome OS editor I wanted to have as a packaged app (just in time!), and posted it up in the store this weekend. It's 100% open source, of course, and contributions are welcome.

Caret is a shell around the Ace code editor, which also powers the editor for Cloud9. I'm extremely impressed with Ace: it's a slick package that provides a lot of must-have features, like syntax highlighting, multiple cursors, and search/replace, while still maintaining typing responsiveness. On top of that base, Caret adds support for tabbed editing, local file support, cloud settings storage, and Sublime-compatible keystrokes.

In fact, Sublime has served as a major inspiration during the development of Caret. In part, this is just because it's the standard for web developers that must be met, but also because it got a lot of things right in very under-appreciated ways. For example, instead of having a settings dialog that adds development complexity, all of Sublime's settings are stored in JSON files and edited through the same window as any other text files — the average Sublime user probably finds this as natural as a graphical interface (if not more so). Caret uses the same concept for its settings, although it saves the files to Chrome's sync service, so all your computers can share your preferences automatically.

The current release of Caret, 0.0.10, is usable enough that I think you could do serious editing with it — I've certainly done professional work with less effective tools, including the initial development on Caret itself — but I'm on a roll adding features and expect to have a lot of improvements made by the end of next week. My first priorities are getting the keybindings into full working condition and adding a command palette, but from that point on it's mostly just polish, bugfixes, and investigating how to get plugin support past Chrome's content security policy. Once I'm at 1.0, I'll also be posting a standalone CRX package that you can use to install Caret without needing a Google account (it'll even auto-update for you).

Working with Chrome's new packaged app support has been rough at times: there are still a lot of missing capabilities, and calling the documentation "patchy" is an insult to quilts everywhere. But I am impressed with what packaged apps can do, not the least of which is the ease of installation: if you have Chrome, you can now pretty much instantly have a professional-grade text editor available, no matter what your operating system of choice. This has always been a strong point for web apps anyway, but Chrome apps combine that with the kinds of features that have typically been reserved for native programs: local file access, real network sockets, or hardware device access. There's a lot of potential there.

If you'd like to help, even something as simple as giving Caret a chance and commenting with your impressions would be great. Filing bugs would be even better. Even if you're not a programmer, having a solid document editor may be something you'd find handy, and together we can make that happen.

August 1, 2013

Filed under: tech»web

Learning WebGL

Once Microsoft announced IE 11 will offer WebGL, that was pretty much the last straw: Apple may drag their feet at enabling it in Safari, but everyone else seems to have decided that it's secure enough and capable enough for production. I still think it's a little nutty, but I don't really have an excuse to avoid it anymore. So when an EaselJS-based visualization at work started having performance issues, I wrote a WebGL shim as a learning project.

I stand by my earlier impressions of the WebGL API--it's clumsy and ill-suited to JavaScript--but the performance is undeniably there. Rendering through EaselGL is often orders of magnitude faster than vanilla Easel, particularly when it comes to mouseover responsiveness. It's worth struggling through the learning process if you find that canvas is becoming a bottleneck for your application. I think a more interesting question is why so many WebGL tutorials are awful. And they are awful:

  • They hide the WebGL boilerplate behind library code--say, a function call that loads shaders or does matrix math--and force readers to dig through the source to figure out what mat4mult() or loadShader() does.
  • Even when they leave the code all in one place, they treat the often-confusing GL code as boilerplate, and don't explain it. Why do I need to call bindBuffer()? What do the six (!) parameters of vertexAttribPointer() actually control? What's the deal with all the constants, like gl.STATIC_DRAW?
  • Since WebGL is not a 3D API, they require a lot of 3D math. Most people teach this badly, assuming they don't double down on the mistakes above and just hide it behind calls to library functions. As a result, it's easy to get lost, and hard to reach the "putting shapes on the screen" feedback loop that keeps students engaged.

Of course, these are not uncommon mistakes in programming tutorials. In fact, they're extremely common--I just haven't had to learn anything from scratch in a while, and had forgotten how confusing the process could be. Anyone writing for beginners would do well to keep these errors in mind.

There are a few walkthroughs that I found more helpful. As I've mentioned, Greg Tavares's series on WebGL was eye-opening, and Brandon Jones provided the only worthwhile explanation of attribute array setup that I found. Between those two, and countless Google searches, I managed to cobble together a basic understanding of how the GL state machine actually works.

As a way of distilling out that knowledge, I've assembled the WebGL demo script that I would have wanted when I started out. It uses no external code--everything's right there on the same page. It explains each parameter that's used, and what each function call does. And it's only concerned with drawing a basic 2D shape--no matrix math is involved. It's stored in a Github Gist, so feel free to file pull requests against anything you find confusing. Also, feel free to look through EaselGL: it's a bit more advanced and I need to add more comments, but as a 2D API I think it's quite a bit easier to understand than the typical game library, particularly for ex-ActionScript developers like myself.

July 19, 2013

Filed under: tech»web

Over the Top

By the time you read this, I'll have been running Weir as my full-time RSS reader for two and a half weeks, starting on July 1. It's going well! Having just added OPML export, so that I can switch if it stops being worth the trouble, I've had a chance to sit back and consider some lessons learned from the project.

  1. Eight megabytes does not seem like a lot of data these days, but it adds up. Before I started culling feeds (and before I added Gzip support to the request service), Weir was pulling down roughly eight megs of data with each fetch, at ten minute intervals. That's 48MB per hour, a small amount that adds up to over a gigabyte per day. By default, I get 24 gigs per month of transfer allowance on my server. Something had to go.

    It's interesting to note, by the way, that this is something that RSS services like Newsblur or Feedly don't worry so much about, because the cost of each feed is spread across all subscribers. I didn't cost Google Reader as much traffic as Weir requires on its own.

  2. So I started unsubscribing. The majority of my original subscription list in Google Reader came from Paul Irish's front-end feed collection, and while I had already unsubscribed from the crazy people, it turns out most of the other blogs in that collection were dead. Even with 304 support added in, Weir was downloading a ton of RSS, only to discard much of it as being past the configured expiration date. I don't think this means blogging is a thing of the past, personally, but it's clearly down from its heyday in favor of social services, particularly (in the technical community) Google+.
  3. That said, using Feedburner seems to be a clear indication that you weren't that interested in blogging anyway, because it makes up a disproportionate amount of the abandoned or simply broken RSS feeds on my list. I suspect this is because it signals a lack of ownership. If you care about your feed, you maintain it yourself.
  4. Even with feeds that work, sometimes connections fail, or things break, just because it's the wild and crazy web out there. Taking that into consideration from the start, and tracking the last result for every feed, was one of the smarter things I did. I should probably be tracking more, but I'm too lazy to do real logging.
  5. Feeds are messy, and sanitization is hard. People inject all kinds of styles into their RSS. They include height and width attributes that don't play well with mobile. They put things into tables. They load scripts that I don't want to run, and images that I'd like to defer until their containing post is activated. Right now, I'm using document.implementation.createHTMLDocument() to make a functional (but "dead") DOM, then running a sanitization task over that, but figuring out that process--and making it watertight--has not been easy.
  6. In fact, working with RSS--ostensibly a "machine-readable" format--tends to drive home just how porous the web can be, and how amazing it is that it works at all. Take RSS date formats discovered by the developer of another reader web app, for example. I'm relatively isolated from the actual parsing, but there's code in Weir to work around buggy HTTP responses, missing feed information, and weird characters. Postel's Law has a lot to answer for, in my opinion.
  7. "Worse-is-better" really works for me as a personal development philosophy. My priority has been to get things running, no matter how badly--hacks get added to the .plan file and addressed later on, when I have time to figure out a graceful solution. This has kept my momentum high, and ensured that I don't get bogged down with architecture that I might not even need.
  8. The value of open-source software for this project can't be overstated. There's no way I could have built Weir on my own. In addition to Node, of course, I'm using a number of open-source modules for parsing feeds, handling two-factor auth, and storing posts in the database. Because of open source, I can patch those various libraries together, add my own code on top, and have a newsreader application that does everything I need. Weir doesn't stand on the shoulders of giants--it stands on the shoulders of countless other people, each giving a little bit back to the wider community.

Past - Present - Future