Mike West - Posts Only

Secure Chrome extensions: Content Security Policy

2011-10-30T00:00:00+02:00

After reading the Content Security Policy primer that I wrote earlier this month, you should have a good idea of the benefits that CSP can offer a website developer. Whitelisting known-good resource origins, refusing to execute potentially dangerous inline JavaScript, and banning the use of eval are all effective mechanisms for mitigating cross-site scripting attacks. Implementing and enforcing such policies is simply a good idea.

Recognizing these advantages, Chrome’s engineers have extended CSP’s reach beyond websites into other areas of the browser where policies can serve as a defense against attacks. Most notably, policies can be applied to packaged applications and extensions. Both can make use of Chrome’s powerful and modular set of extension APIs, which give developers capabilities above and beyond what it makes sense for the web platform to offer normal websites. To whatever extent possible, therefore, developers need to ensure that those additional capabilities aren’t leaked out to the broader web. Content Security Policies mitigate this risk by helping developers ensure that the only code that runs with elevated privileges is their own.

Defining policies for an extension is quite straightforward, and we’re working to add them to all the sample extensions in the Chromium project. Walking through one of these samples together should give you a good idea of how it all works.

Implementation

The Mappy sample extension injects a content script into every page a user visits, searches the page for addresses, and passes them back to the extension so that a map can be displayed in the extension’s popup. This is exactly the sort of dangerous scenario in which we need to worry about malicious websites exploiting the extension’s permissions, perhaps by cleverly embedding JavaScript into the address which might then be executed if we’re not careful when it’s passed back to the extension’s context. A content security policy is absolutely necessary.

The first step towards securing the extension is to define exactly which resources it needs to load. Mappy’s needs are pretty simple: it loads CSS and JavaScript from the local extension package, connects to https://maps.googleapis.com to geolocate an address, and finally displays a static map image from https://maps.google.com.¹ A sufficiently paranoid policy would use a default-src policy directive to deny access to everything by default, and then specifically whitelist only those resources via style-src, script-src, connect-src, and img-src directives. I’ve annotated Mappy’s definition (which does exactly that) below:

# Block everything, then whitelist from there.
default-src 'none';

# Accept CSS from the extension's package.
style-src 'self';

# Accept JavaScript from the extension's package.
script-src 'self';

# Allow XHR connections over HTTPS to Google Maps APIs.
connect-src https://maps.googleapis.com; 

# Allow images from Google Maps to load over HTTPS.
img-src https://maps.google.com;

For normal websites, content security policies are delivered via an HTTP header. Chrome extensions use the same policy syntax, but define policies in the package’s manifest JSON as a simple key-value pair. Simply take the policy you’ve defined, and add it to manifest.json: you can see how Mappy’s CSP is defined by visiting Chromium’s source repository.

With the manifest changes in place, the next step is to take a close look at the extension’s code to make sure we comply with the new restrictions we’re defining. Generally speaking, this means ensuring that CSS and JavaScript is moved out from the extension’s background page and popup into separate files. Most of the time, this is a straight copy/paste job, simply moving code wholesale from an inline <script> block into a background.js file. For Mappy, the background.html change is captured in this diff, and the popup.html change is visible in this diff.

That’s it! Once you’ve defined a policy, and adjusted your extension’s code to match it, just test the extension locally, bump the version number, and push the update to your users. They’ll be more secure, and the world will be a (slightly) better place.

Next Steps

Your task is clear: update the extensions you own! It’s very straightforward work, and a real win for security. Content Security Policy support was added to Mappy in revision 106043; take a look at the code review to get some implementation ideas for your own extensions.

At the same time you’re working on your extensions, the Chromium team is working to set a better example by updating the 70 or so sample extensions and applications that Chromium makes available to include the new content_security_policy manifest entry. It’s easy work – following the steps listed above takes less than half-hour on average – but 70 is a big number, so we’re a bit behind. You can follow our progress on that effort at crbug.com/92644: star the bug if you’re interested in updates, and pitch in if you’re interested! Patches are welcome: just send any code reviews my way (mkwst@chromium.org).

Whenever possible, load resources over HTTPS rather than unencrypted HTTP. This has a number of benefits for privacy and security, most importantly (for our current context) a guarantee that the code you’re loading hasn’t been modified since it left the server. Active network attackers need to do a lot more work to inject code into a HTTPS stream, wheres it’s a trivial task via HTTP.↩

Content Security Policy: A Primer

2011-10-11T00:00:00+02:00

The browser is not a safe programming environment. It is inherently insecure. – Douglas Crockford, “Ajax Security”

The web’s security model is fundamentally broken, and has been since the beginning. Browsers trust the code they receive from a website, so each chunk of JavaScript that executes on a page runs with access to the entire origin’s data. Cross-site scripting (XSS) attacks exploit this trust, injecting malicious code and other resources into a site in a wide variety of ways. At best, sites are defaced. At worst, user session data is compromised and leaked to untrusted third-parties.

It is, of course, possible to completely eliminate this class of attacks by properly escaping every bit of data that makes it onto a site. A brief glance back at the web’s history shows that this defense is unlikely to be effective; it’s simply a tough problem, given the convoluted set of escaping rules in various HTML contexts, differences in cross-browser implementation, and browser bugs (like UTF-7 support). Even with our solid, modern understanding of the issue, and a variety of secure frameworks, it remains trivial to accidentally create a hole, and one hole is all it takes.

Content Security Policy (CSP) is a relatively new addition to the web platform that promises to mitigate the risk of XSS attacks by giving administrators fine-grained control over the data and code that ought to be allowed to run on their site. The feature boils down to a whitelisting mechanism for images, script, style, and a variety of other resource types. A site declares an acceptable list of origins for each data type it cares about via a straightforward HTTP header, and browsers that support the feature simply refuse to load resources that aren’t listed, thereby ensuring that user data can’t be leaked to third-parties. Browsers that don’t support the feature simply ignore the header, and proceed as per usual.

Though the feature is still very much bleeding-edge, implementations of the draft specification exist in both Firefox 4+ and Chrome 16+, and the W3C has proposed a Web Application Security Working Group which includes CSP as a top-level deliverable. All that in mind, CSP is practically available right now, has no negative effects on browsers that don’t support it, and is already being used successfully by some pretty high-profile sites (like Twitter¹). The implementations are being iterated upon, and things are still a bit fast and loose, but getting involved now is certainly something I’d recommend.

mkw.st’s Policies

On mkw.st, I’m using one stylesheet hosted on mkw.st, one of Google’s web fonts, and a few inlined data: images. I’d like to ensure that that remains the case, and that I don’t accidentally load script from my friends at evil.com.

I’ve configured my server to generate the following headers when loading the site:

$ curl -Ik https://mkw.st/
HTTP/1.1 200 OK
Server: nginx/1.0.8
Date: Sat, 08 Oct 2011 10:38:40 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 7940
Last-Modified: Sat, 08 Oct 2011 05:07:16 GMT
Connection: keep-alive
Vary: Accept-Encoding
Strict-Transport-Security: max-age=604800
X-WebKit-CSP: default-src 'self'; img-src 'self' data:; font-src https://themes.googleusercontent.com;
X-Content-Security-Policy: default-src 'self'; img-src 'self' data:; font-src https://themes.googleusercontent.com;
Accept-Ranges: bytes

The interesting bits are X-Content-Security-Policy and X-WebKit-CSP², both of which contain a simple, semicolon-separated list of policy directives. Each directive consists of a type followed by a set of one or more “source expressions” that define the policy’s limitations. These expressions generally consist of scheme/host/port groupings, but can also contain wildcards and special keywords like 'self' (which expands to the current site’s origin) and 'none' (which ought to be self-explanatory). Let’s look at each in turn.

The first directive is of type default-src, which specifies a catch-all for content types that aren’t explicitly listed. I haven’t created a policy for objects (object-src), for example, so any request for video or Flash content would use the policy specified here. In this case, I’m specifying a single source: 'self'³. This means that resources that hit default-src can only be loaded from the same origin as the page itself. You may see references to allow in older CSP documentation: default-src replaces that keyword.

You likely won’t be shocked to learn that the next directive,img-src, whitelists sources for image resources. Here, images are limited to data: URLs, and to 'self'.

Equally unsurprisingly, the directive of type font-src allows the site to load font resources from https://themes.googleusercontent.com/, which is taken quite literally and exclusively. Fonts can only be loaded from themes.googleusercontent.com, and only over HTTPS. Any request for a font from a different host or scheme would throw an error. It’s important to note here that the value of the font-src overrides default-src completely. They aren’t merged: I won’t be able to load fonts from 'self' unless I explicitly add it to font-src’s definition.

Nothing’s free.

In order to function properly, Content Security Policy imposes a few restrictions on sites that make use of it. Two in particular are worth calling out here.

CSP enforces a strict separation of behavior and semantics by refusing to execute inline script, and an equally strict separation of presentation and semantics by refusing to apply inline styles. This follows directly in line with the general recommendation of the web standards community, and allows CSP to clearly identify the source of each piece of data on a page. This identification is essential, as without it the whitelisting mechanism is ineffective. In addition, banning inline script and style substantially reduces the effect of cross-site scripting attacks that rely on JavaScript or CSS reflection: attackers will be forced to compromise a whitelisted resource, which is a much higher bar.

It’s fairly obvious that this restriction will block execution of code contained within <script> blocks, but also note that it prevents event handlers from being written inline (<a … onclick="JAVASCRIPT"> ought to be rewritten to use addEventListener(…)). Likewise, javascript: URLs won’t work.

It’s also the case that data: URLs are blocked by default, as they can contain arbitrary content that can’t be easily verified as being safe, or as really being intentionally written by your server. If you’d like to allow data: URLs, you’ll have to do so on a type-by-type basis. See above, for example, where I’m explicitly allowing data: URLs for image resources, but not for any other resource type.
Relatedly, raw strings will not be converted to code. This means that eval is right out, as are a variety of other eval-like mechanisms (setTimeout with a string argument, new Function(…), and so on). Since eval is evil, this isn’t a particularly onerous restriction, but it does mean that you’ll need to parse JSON content without simply executing it, either with something like json.js or with a built-in JSON object.

HTTP Strict Transport Security and You

2011-10-07T00:00:00+02:00

With a simple Wi-Fi packet-sniffer, intercepting login cookies over the air is far easier than it ought to be. Firesheep demonstrated this vulnerability definitively, showing the public exactly how trivial it is to hijack unencrypted HTTP sessions. So, we learned an important lesson: running HTTPS on your servers mitigates a large chunk of risk, as encrypting the traffic between your site and its users prevents attackers from grabbing cookies, and thus from hijacking a user’s session. Encryption also ensures that your content is delivered without modification, which reduces the ability of man-in-the-middle attackers to intercept your users’ traffic when they’re using untrustworthy networks. To whatever extent possible, you should use HTTPS, and ensure that you force encryption for important cookies by marking them as Secure.

Unfortunately, this isn’t a complete solution. Users generally don’t type https://example.com/ into their browser’s address bar. They generally either type example.com or http://example.com/, and you’re going to have a hard time reminding them to consistently use https when visiting your site. You can redirect users from HTTP to HTTPS, but that leaves a window of opportunity for attackers: your request to the HTTP site is unencrypted, and potentially vulnerable to attack before you can be redirected. It would be significantly safer if you could avoid that HTTP request. This is the problem that HTTP Strict Transport Security (HSTS)¹ is meant to address.

Adding a Strict-Transport-Security header to your server’s HTTPS responses will inform browsers that your website should only be accessed via HTTPS. Normal HTTP connections will be rewritten client-side, before any network traffic is generated. Sending the following header:

Strict-Transport-Security: max-age=31556926; includeSubDomains

would instruct a browser that supported the draft standard that it shouldn’t touch your site (or any of its subdomains) over HTTP for a year. This substantially reduces the surface exposed to attack: as long as the very first HTTP request went through correctly, your users will never use an insecure connection to your website again, regardless of what they type in the address bar.

You will need to ensure that you can actually serve HTTPS responses for every request (on any port) to your host, however, as HTTP simply won’t be an option anymore. This is a Good Thing™, even if it might mean a little work for you up front.

None of this is new (just new to me), so browser support is already relatively good: HSTS is supported in Firefox 4+, and Chrome². Opera appears to be waiting for issues open against the draft standard to be resolved, and for the draft to move to RFC status. Looking at the list, I don’t see any significant upcoming changes to the header structure: I’d recommend implementing it for your sites now.

Sound interesting?

As an example, I’ve just implemented this for mkw.st (where I’ll likely be moving this site at some point in the relatively near future). It took about a half-hour, all told, and most of that was waiting for the certificate to be mailed to me. Here’s what I needed to do to set things up:

I generated a certificate request on my server, which also involved creating a new private key. Both were trivial, and RapidSSL’s detailed walkthrough was helpful.
Using the newly generated CSR, I purchased a certificate via CheapSSLs. They seem a bit… well… cheap, so I’m not sure I’d recommend them for a high-risk site. For my not particularly valuable site, however, they’re brilliant. A 3-year cert (from RapidSSL) was $27 (paid via PayPal), and showed up in my inbox within 20 minutes of purchase.
I configured my server to use the certificate. For Nginx, this just meant recompiling the server with SSL support, and adding a new server definition that listened on port 443, and had reference to both the private key and certificate generated above. Nginx’s documentation is better than I remembered it being.

For a single-server site like mine, this is a piece of cake, and certainly worth playing around with.

Adam Barth co-wrote the HSTS spec, and he’s coincidentally just started a cleverly titled web security blog that I suspect will be very much worth reading: Scheme/Host/Port ↩
Chrome actually attempts to avoid even the initial one-time HTTP request for websites that have requested extra protection. Sites like PayPal and Twitter are hard-coded as requesting HSTS (the complete list is visible in net/base/transport_security_state.cc).↩

Chrome Privacy

2011-09-24T00:00:00+02:00

Dave Winer ends an otherwise quite reasonable piece about his concern at Facebook’s “frictionless sharing” with a non sequitur attack on Chrome for, as far as I can tell, nothing it’s actually doing:

One more thing. Facebook doesn’t have a web browser, yet, but Google does. It may not be possible to opt-out of Google’s identity system and all the information gathering it does, if you’re a Chrome user.

Ben Brooks picked up on that, adding:

Read Winer’s take on this, it’s pretty creepy of Facebook — what’s more scary is the possibility of Google doing this to Chrome users.

A direct response to this sort of speculation about Chrome’s potential for evil, slipped in as a conclusion to a discussion about an unrelated company being evil, is difficult, so let me step back a bit. Chrome founded a privacy team here in Munich back around the release of Chrome 4. I’m proud to be a small part of this clever group of developers who care about making Chrome’s use of data transparent, and giving you control over how it’s used to whatever extent possible. We build APIs that enable privacy-relevant extensions and apps to fine-tune a browser with an already good set of privacy features, and review features built by other teams for potential impact on user’s private information. I’m biased, but I think we do a decent job.

So, to Dave, Ben, and everyone else: If you see Chrome doing something you don’t like with your information, or you have specific questions about some feature or another, send an email to the team at privacy@chromium.org, or to me directly, mkwst@chromium.org. You can also file a bug on the public tracker at new.crbug.com and mail me the bug ID. I’ll make sure it gets triaged into the right team.

With that in mind, here’s a direct answer to what I think is being criticized: The browser doesn’t secretly send information about your browsing habits to Google, nor will it. The feature that comes closest is Sync, which is a) opt-in, b) encrypted locally before being sent to Google, optionally with a password separate from your Google account.

Update:

Some good discussion is taking place on Hacker News.
I’ve removed an unintentional ad hom that was personally inflammatory: Sorry Dave, I honestly didn’t mean to offend you, just to argue with you. The focus of my frustration is both the suggestion that Chrome may at some point in the future start being evil, and the way it was casually slipped as a conclusion into a discussion of an unrelated company arguably being evil. I hope that remains clear. This article’s history is available on GitHub, so the edits I’ve made remain transparent.

I'm on Technikwürze

2011-06-05T00:00:00+02:00

I sat down with Technikwürze’s Marcel Böttcher way back at the beginning of February to talk about the exciting new release of Chrome 9 to the stable channel, and a few other bits and pieces of the Chrome ecosystem. That interview (in German) is just now seeing the light of day as Technikwürze 178. After listening to it last night, I think it generally went pretty well.

I got a good chance to talk about the (then shiny and new) world of WebGL. It’s become hugely popular in the meantime, and is supported with GPU acceleration across the board in modern browsers with generally excellent performance characteristics. Libraries have sprung up like Mr.doob’s amazing three.js that make working with WebGL easy and fun. ro.me is a great example of what can be creatively accomplished with the technology, and I’m really excited about seeing where the web will take things that it has full-speed access to OpenGL goodness from JavaScript.

I touched on the Chrome Web Store, which has had a very successful launch in the States, and is ramping up (albeit slower than I’d like) for a full international release later this year. You can see some of the results of that effort already, as the US store has been fully localized to make it accessible to users no matter what language they speak, and it’s only going to get better as the year progresses.

I was able to highlight the focuses of Google’s engineering center in Munich (we’re hiring! send me CVs!), especially those of the Chrome team that’s doing exceptional work on privacy and enterprise features of the platform. I touched on some of the thought behind Chrome’s release process, autoupdates, and the extension API framework. We dove briefly into some of the privacy-related work Google has done, highlighting Keep My Opt-outs which we released in January, and pointing towards some future extension APIs that will enable developers to bring some great privacy protections (think TorButton and NoScript) to the Chrome platform.

These are all relatively timeless bits and pieces that I’m glad I was able to share, and I hope that listeners find it interesting, even if it’s all framed in a context that’s 4 months too late. I’m a bit disappointed that it’s just coming out now rather than back then when it would have been a bit more relevant. It’s amazing to see what’s happened in the last 4 months (IE9 and FF4 have both been released in the meantime, we’ve pushed two new stable releases of Chrome, I/O happened, etc.), and pushing this interview out four months later simply doesn’t reflect the things that I’d like to be focusing on now. It sounds like I’m talking about quite old things as though they were new, which is a bit confusing for everyone involved. I’m hopeful that we can get together again in the relatively near future to talk about some of the things that are interesting today and tomorrow.

Things I’ll say differently next time.

First and foremost, listening to myself speak German is a painful experience. I mean, it’s tough to listen to myself speak English; listening to myself try and fail to wrap my tongue around a few phrases (repeatedly) is a bit embarrassing. sigh I’ll get better at it.

I’d also like to make two corrections. I’m not sure what one would call “errata” for a podcast, but that’s what I’ll outline here. Let’s start with the one or two things I said that jumped out at me as being clearly idiotic, even back in February:

I sold some of my coworkers short: of course Google has lobbyists in Berlin, but that’s not at all the extent of our efforts there. Among other things, there’s a large cooperation with academic institutes there to set up a research institute focused on internet and society.
I talked about Native Client as something that would trigger a review when uploading an item to the Chrome Web Store. I meant to refer to NPAPI plugins that can execute native code on your computer. If you write a Chrome extension or packaged application that relies on an NPAPI plugin, you’ll go through a review process to verify that someone’s responsible for the code, as there’s no practical limit on what such a plugin can do to a user’s computer (and we flag it as such in the store as being able to access “All data on your computer and the websites you visit”). These types of plugins also won’t run on ChromeOS, full stop.

While Native Client is in your head, however, I’ll note that it is right now in Chrome behind a flag, and there are indeed projects written using it (NaClBox being the most recent brilliant example that I’m aware of), but it’s not something for the general public at the moment. It’s experimental. An amazing piece of technology that’s in earnest development, but experimental. The Native Client team gave a talk at I/O (“Beyond JavaScript: Programming the Web with Native Client”) that’s worth watching if you’re interested in the current status of the project.

I also mentioned casually that Native Client supports not only C/C++ but also Java and maybe Python. Again, I should have referred to NPAPI plugins (which actually do somewhat support Python via projects like pyplugin). Native Client is C/C++ only at the moment. Though there’s no theoretical reason the framework couldn’t work foremost other languages, there’s no compiler, nor do I know of any effort underway to write one.

Updates.

The market share numbers from February which I quoted are, of course, out of date. At I/O we talked about reaching 160 million users, and StatCounter puts Chrome usage in their slice of the internet at right around 20% at the end of May. What I wish I had mentioned is that these numbers are nice to throw around in conversation, but the most important thing for a particular application is its own demographics. Look at your own logs to determine the distribution of browsers in your audience.
The WebM project has since submitted an update to the bitstream specification (“VP8 Data Format and Decoding Guide”) as an Internet Draft to the IETF.
With regard to the application metadata format that the Chrome Web Store uses, Mozilla’s implementation of a similar idea is progressing nicely, but there’s still a gap between the functionality it offers and the model Chrome is running with at the moment. Discussion regarding those differences, and the direction in general, are happening in public on a variety of lists (mozilla-labs for example).
Chrome’s quick, multi-channel release cycle has been more successful than I thought possible. Firefox has adopted a similar system, and Microsoft is already pushing out “platform previews” of IE10. This is just plain good for the web.

And that’s about it…

This was the first time I’ve participated in a podcast of any sort. It was a good experience, modulo the gap between recording and publication, and I look forward to talking into a microphone at some point again. Thanks for inviting me onto the show, liebe Technikwürzler. Till next time… :)

A Quick git/vim Workflow Tip

2011-04-10T00:00:00+02:00

Git makes it incredibly easy to work on a lot of a project’s features at once, hopping quickly back and forth between branches. I love this ability, but I hate remembering what exactly it was that I was working on in a particular branch. I know it had something to do with a particular bug, but I’ve no idea anymore which files I was fiddling around with to fix it.

Enter git vim. I’ve set up two trivial alias in my ~/.gitconfig file to either show me a list of files effected by a particular revision or range, and to directly open them in my editor of choice so that I’m back to work as quickly as possible.

[alias]
  ...
  fshow = ! sh -c 'git show --pretty="format:" --name-only $1 | grep -v "^$" | uniq | sed -e "s#^#`git rev-parse --show-toplevel`/#"' -
  vim   = ! sh -c 'vim `git fshow $1`' -
  mate  = ! sh -c 'mate `git fshow $1`' -
  edit  = ! sh -c '$EDITOR `git fshow $1`' -

The first alias calls git show to get a list of all the files touched by a revision or range, filters out empty lines, and ensures that the paths are absolute after deduplicating the list of files. Any valid revision or range will work: HEAD, 37ad159, master.., etc.

To see a list of files changed over the last four revisions, I could type:

$ git fshow HEAD~5..
lib/rocco/layout.rb
lib/rocco/layout.mustache
lib/rocco.rb
rocco.gemspec
test/helper.rb
test/test_reported_issues.rb

To open those files for editing, I’d use the second alias:

$ git vim HEAD~5..

If I wanted to open the files in TextMate instead, I’d use the third:

$ git mate HEAD~5..

Or, if I was clever enough to set up an $EDITOR environment variable (which I should do, given that it’s used all over the place on the shell), I could use the last alias to open the files in whatever program that was set to:

$ git edit HEAD~5..

I can hop from branch to branch, and open all the relevant files quickly and easily. It’s a small thing, but it’s made a big difference in my workflow over the last week or three.

Intro to IndexedDB

2010-12-14T00:00:00+01:00

Yesterday at the Silicon Valley GTUG meetup, I gave a presentation introducing the IndexedDB API. I’ve thrown the slides on Slideshare, but the transcription there is absolutely miserable. I’ll reproduce it here in a readable format, and add a few notes where appropriate.

Video

Embedded Slides

Intro to IndexedDB (Beta)

Slide Transcript

IndexedDB: Mike West, @mikewest, mkwst@google.com, SV GTUG, 2010.12.14
Beta: The IndexedDB API is incredibly beta. It’s only implemented in Firefox 4 and Chrome dev channel, so it’s not anything that can be used for production projects in the near future. Microsoft and Opera are contributing to the spec, however, and Google is working on pushing the code upstream to Webkit itself, so this looks like something that will be more and more relevant going forward.

Since the spec’s not finished, and everything’s in dev mode, this is a great time to examine the API, and experiment. We need to play around with this code, and feed our experience back into the standards bodies and browser vendors: that’s the best way to ensure that things work the way we want them to when everything’s solidified.
Offline: One of the most exciting recent realizations in web development is that the offline bits of the HTML5 suite of specifications are really ready for widespread use. It’s possible to store arbitrary amounts of information on a user’s computer without resorting to opaque hacks like Flash storage, while at the same time making that information available in useful ways to your program’s code. This opens up a whole new world of sites and applications which we’re only just beginning to appreciate. Offline’s important, and not just because of the Web Store.
Storage Options: What I’d like to do here is take a very brief survey of the landscape for context, and then dive into one particular feature that I think will become important in the near future: IndexedDB.
Cookies: These aren’t offline at all, but they’re relevant to the general context of how web applications store data at the moment. The image on this slide is Luigi Anzivino’s “Molasses-Spice cookies” (which look delicious).
Cookies
- Simple, key-value pairs, “shared’ between server and client.
- Excellent for maintaining state, poor for anything else, as they are unstructured, and incur a signiﬁcant overhead for each HTTP request.
Local Storage: The image on this slide is Evan Leeson’s “Toasters”.
Local Storage
- The simplicity of cookies, tuned for higher-capacity, client-side-only storage.
- Dead simple API:
```
localStorage.setItem( ‘key’, ‘value’ );
localStorage.getItem( ‘key’ ); // ‘value’
```
- Values are unstructured strings:
  - Filtering and search are O(n), unless you layer some indexing on top.
  - Structure requires JSON.stringify & JSON.parse
WebSQL: The image on this slide is Nick P’s “file cabinet to heaven”, which is a pretty accurate representation of life with WebSQL. Stacking file cabinets on top of each other certainly provides you with the possibility of well organized storage, but that doesn’t mean it’s a good idea.
WebSQL
- A real, relational database\n implementation on the client (SQLite)
- Data can be highly structured, and JOIN enables quick, ad-hoc access
- Big conceptual overhead (SQL), no finely grained locking
- Not very JavaScripty, browser support is poor (IE and Firefox won’t implement it), and the spec has been more or less abandoned.
File API: The image on this slide is Davide Tullio’s “Hard Disk in B&W.
File API: I know nothing about the File API, but Seth does! And his presentation is right after mine, so I’ll be all ears. :)
IndexedDB: The image on this slide is Robin Riat’s “Kanuga library card catalog”
IndexedDB:
- Sits somewhere between full-on SQL and unstructured key-value pairs in localStorage.
- Values are stored as structured JavaScript objects, and an indexing system facilitates filtering and lookup.
- Asynchronous, with moderately granular locking
- Joining normalized data is a completely manual process.
IndexedDB Concepts
Practically everything is asynchronous. Callbacks are your friends.
Databases are named, and contain one or more named Object Stores
A diagram of how a database might look, containing a single object store and a set of objects.
Object stores define a property (similar to a primary key) which every stored object must contain, explicitly or implicitly (autoincremented).
The same diagram as #18, with IDs added.
Values in an Object Store are structured, but don’t have a rigidly defined schema. Think document database, CouchDB. Not MySQL.
The same diagram as #20, with differing data added for various objects.
Object Stores can contain one or more Indexes that make filtering and lookup possible via arbitrary properties.
The same diagram as #22, with a subset highlighted (as though they were filtered out).
IndexedDB API: Now we’ll dive into some JavaScript. Lovely, lovely JavaScript.
It’s beta. Again. This is a reminder. :)
Vendor Prefixes: webkitIndexedDB & moz_indexedDB

Code:

// Deal with vendor prefixes
if ( "webkitIndexedDB" in window ) {
  window.indexedDB      = window.webkitIndexedDB;
  window.IDBTransaction = window.webkitIDBTransaction;
  window.IDBKeyRange    = window.webkitIDBKeyRange;
  // ...
} else if ( "moz_indexedDB" in window ) {
  window.indexedDB = window.moz_indexedDB;
}
if ( !window.indexedDB ) {
  // Browser doesn’t support indexedDB, do something
  // clever, and then exit early.
}

Database Creation

Code:

var dbRequest = window.indexedDB.open(
  “AddressBook”,        // Database ID
  “All my friends ever” // Database Description
);

// The result of `open` is _not_ the database.
// It’s a reference to the request to open
// the database.  Listen for its `success` and
// `error` events, and respond appropriately.
dbRequest.onsuccess = function ( e ) { ... };
dbRequest.onerror   = function ( e ) { ... };

Databases are versioned…

Code:

// The `result` attribute of the `success` event
// holds the communication channel to the database
dbRequest.onsuccess = function ( e ) {
  var db = e.result;
  // Bootstrapping: if the user is hitting the page
  // for the first time, she won’t have a database.
  // We can detect this by inspecting the database’s
  // `version` attribute:
  if ( db.version === “” ) {
    // Empty string means the database hasn’t been versioned.
    // Set up the database by creating any necessary
    // Object Stores, and populating them with data
    // for the first run experience.
  } else if ( db.version === “1.0” ) {
    // 1.0 is old!  Let’s make changes!
  } else if ( db.version === “1.1” ) {
    // We’re good to go!
  }
  // ...
};

… and versioning is asychronous.

Code:

dbRequest.onsuccess = function ( e ) {
var db = e.result;
if ( db.version === “” ) {
  // We’re dealing with an unversioned DB.  Versioning is, of
  // course, asynchronous:
  var versionRequest = db.setVersion( “1.0” );
  versionRequest.onsuccess = function ( e ) {
    // Here’s where we’ll set up the Object Stores
    // and Indexes.
  };
}
// ...   };

Creating Object Stores and Indexes

Code:

dbRequest.onsuccess = function ( e ) {
var db = e.result;
if ( db.version === “” ) {
  var versionRequest = db.setVersion( “1.0” );
  // Setting a version creates an implicit Transaction, meaning
  // that either _everything_ in the callback succeeds, or
  // _everything_ in the callback fails.
  versionRequest.onsuccess = function ( e ) {
    // Object Store creation is atomic, but can only take
    // place inside version-changing transaction.
    var store = db.createObjectStore(
      "contacts",  // The Object Store’s name
      "id",        // The name of the property to use as a key
      true         // Is the key auto-incrementing?
    );
    // ...
  };
}
// ...   };

More code:

dbRequest.onsuccess = function ( e ) {
  var db = e.result;
  if ( db.version === “” ) {
    var versionRequest = db.setVersion( “1.0” );
    versionRequest.onsuccess = function ( e ) {
      var store = db.createObjectStore( "contacts", "id", true );
      store.createIndex(
        “CellPhone”,  // The index’s name
        “cell”,       // The property to be indexed
        false         // Is this index a unique constraint?
      );
    };
  }
  // ...
};

Writing Data (is asynchronous)

Code:

// Assuming that `db` has been set somewhere in the current
// scope, we use it to create a transaction:
var writeTransaction = db.transaction(
  [ “contacts” ],           // The Object Stores to lock
  IDBTransation.READ_WRITE  // Lock type (READ_ONLY, READ_WRITE)
);
// Open a contact store...
var store = writeTransaction.objectStore( “contacts” );
// ... and generate a write request:
var writeRequest = store.add( {
    “name”:  “Mike West”,
    “email”: “mkwst@google.com”
} );
writeRequest.onerror = function ( e ) {
    writeTransaction.abort();
};
// Transactions are “complete” (not “committed”?) either when
// they fall out of scope, or when all activities in the
// transaction have finished (whichever happens last)
writeTransaction.oncomplete = function ( e ) { ... };

Reading Data (is asynchronous)

Code:

// Assuming that `db` has been set somewhere in the current
// scope, we use it to create a transaction:
var readTransaction = db.transaction(
  [ “contacts” ],           // The Object Stores to lock
  IDBTransation.READ_ONLY   // Lock type (READ_ONLY, READ_WRITE)
);
// Open the `contact` store...
var store = readTransaction.objectStore( “contacts” );
// ... and generate a cursor to walk the complete list:
var readCursor = store.openCursor();
// Setup a handler for the cursor’s `success` event:
readCursor.onsuccess = function ( e ) {
  if ( e.result ) {
    // You now have access to the key via `e.result.key`, and
    // the stored object via `e.result.value`.  For example:
    console.log( e.result.value.email ); // mkwst@google.com
  } else {
    // If the `success` event’s `result` is null, you’ve reached
    // the end of the cursor’s list.
  }
};

Querying (is asynchronous)

Code:

var t = db.transaction( [ “contacts” ], IDBTransation.READ_ONLY );
var s = t.objectStore( “contacts” );
// ... and generate a cursor to walk a bounded list, for example
// only those names between M and P (inclusive)
var bounds = new IDBKeyRange.bound(
  “M”,  // Lower bound
  “Q”,  // Upper bound
  true, // Include lower bound?
  false // Include upper bound?
);
var readCursor = store.openCursor( bounds );
// Setup a handler for the cursor’s `success` event:
readCursor.onsuccess = function ( e ) {
  // process `e.result`
};

Further Reading:
- The IndexedDB Spec
- Firefox 4: An Early Walkthrough of IndexedDB
- Mozilla Developer Docs
Questions?, Mike West, @mikewest, mkwst@google.com, http://mikewest.org/

projects.mikewest.org

2010-11-02T00:00:00+01:00

“What has Mike West been up to recently?” This is one question that you almost certainly haven’t been asking yourself recently. That’s a shame, really, as I’ve got a few small projects floating around that I think you might be interested in paying attention to.

If you’ll indulge me, I’ll run through them in no particular order:

VimRoom is a Vim plugin that replicates the best features of WriteRoom in my favorite editor. I do most of my writing (code and otherwise) in Vim, as the keyboard shortcuts I’ve painstakingly assembled are hardwired into my fingers (and, I suppose, because I’m a huge nerd). Combined with iTerm2’s brilliant fullscreen mode, VimRoom offers a great way to shut out the myriad distractions floating around my iMac, and focus on actually getting words onto the screen: I’m pretty pleased with how it’s turned out, and I hope it might be useful for others.

static_gettext is a (tiny) internationalization framework that makes it possible to generate arbitrary localizations of static source documents. Dynamic frameworks like Django or Rails have good systems in place to deal with the tough problems of removing hard-coded strings from source code and templates. Less complex, static websites aren’t going away, however, and static_gettext has solved the problem for me nicely, generating industry-standard *.po files that can be easily handed off to translators. The framework is currently seeing active use on the HTML5 Boilerplate site for the German translation (which I helped out with), and a few other localizations are in the pipe.

JSLint Utils is a command-line wrapper for JSLint that makes it possible to quickly lint your JavaScript and CSS files via the magic of either Node.js (fast!) or Rhino (not so fast!). Lint results can be output as XUnit-style XML files, making it possible to cleanly integrate into a variety of continuous integration systems (Hudson, for example), meaning that you don’t even have to think about it: you just get friendly emails after every commit you screw up. It’s a lifesaver!

PyPlaceholder is a command-line Python script that generates placeholder PNGs for use in… well… wherever you might use placeholder PNGs. It’s made my life a little simpler while putting together mockups, and it has a few tiny features that I didn’t see on any of the surprisingly numerous alternatives. I’ve additionally wrapped it up in a Flask application, flask-pyplaceholder, to generate images on the fly via placeholder.mikewest.org.

Send To Instapaper is a stunningly well-named Chrome extension that has no purpose other than to replicate the standard Instapaper-bookmarklet functionality with two small additions: it adds a keyboard shortcut, and it’s streamlined, clean, and pretty.

And now, the point…

I read Jesper’s Doc Robot this morning, and found myself nodding vigorously. I disagree about the inherent valuelessness of all autogenerated documentation (as I’ve also been doing a good bit of work on Ryan Tomayko’s Rocco, which autogenerates lovely documentation if you do the work of coding in a literate fashion), but I think he makes a great point about the way we generally treat documentation as an afterthought. I’d like to share the solution I’ve come up with for my code.

You’ll notice that these projects that I’ve linked to above all have landing pages that more or less clearly explain what the project is, how it works, and what your next steps might be, should you be interested enough to want to try it out for yourself. Each of those landing pages also has a clear changelog, and an RSS feed that you can subscribe to, should you be interested in keeping yourself up to date. There’s even an RSS feed for the site, in case you’d just like to have a good handle on what it is that I’m up to.

I think this is the bare minimum a project needs to offer in order to begin to serve an audience. I’ve put together projects.mikewest.org in order to have a clear place to put any and all documentation I generate for the projects I’d like to present to the public. The ever so clever Tim Huegdon has put together a similar project listing with good documentation, and I’d like to encourage you to do so as well.

GitHub pages makes the process of putting together a basic website hassle-free, and you can take a look at the project page repository to see how I’ve bent Jekyll to the task of generating RSS feeds and embedded changelogs. It’s not rocket science, but it took a bit of experimenting that you can spare yourself. Use the repo as a template. There’s nothing there I wouldn’t gladly share.

JSLint needs some Bad Parts

2010-05-16T00:00:00+02:00

One of the few tools that I consider truly indispensable when developing websites is JSLint. A marvel of applied compiler theory, the program has saved me from stupid mistakes more times than I care to remember. I have it set up to automatically sanity-check my code every time I push a commit to SVN, and it’s generally my best friend in the whole wide world.

JSLint isn’t always perfect, however, especially when used to check CSS files. I think Douglas Crockford is a JavaScript wizard, but I’m somewhat less blindly acquiescent when it comes to his take on the Good Parts of style sheets. For better or worse, browsers are much less similar in their interpretation of what I mean when I write a particular style, and certain hacks and workarounds are par for the course when putting together a layout of any complexity. JSLint’s CSS checks do a good job warning about certain types of errors (missing semicolons, for instance), but a relatively terrible job when it comes to some of the necessary evils of practical development. In short, JSLint’s CSS mode needs some Bad Parts to be really usable for me.

In itself, this wouldn’t be an issue. Crockford has kindly released JSLint as open source (under a unique (to say the least) “Do no evil” license), and I’m generally able to puzzle out how to go about adding support for the features that I believe are necessary. Getting those patches into the mainline JSLint project is, unfortunately, often a miserable process. I have two issues with the current process:

First, there’s no public repository (that I’ve come across). Crockford is pretty good about announcing changes on the JSLint mailing list, and he keeps a timestamp in the current fulljslint.js file, but that’s hardly a solid basis for contributing patches. This isn’t a show stopper, it just makes things harder to follow than they could be, and more difficult to give back to the project than it should be. I asked about this in May 2009, and was met with silence. Ah well.

Second, the aforementioned list is easily the most (passive-)aggressive I’ve been on recently. The (true) warning that “JSLint will hurt your feelings” goes double for posting questions or suggestions, especially as a newbie. I almost fell off my chair when I saw this question (“I have this problem, and I’ve found this workaround. Why is it a problem in the first place?”) received this response (“Thanks for the report. JSLint now no longer accepts your workaround.”) Crockford is a smart guy, and he’s opinionated in the best possible way, but he couples that with a periodic lack of tact that’s really influenced the way the list as a whole works. It’s simply not a fun place to contribute ideas or code, and that’s a shame. Arguing about the validity of suggestions is simply not something I’m willing to do every single time I make a suggestion, especially when the suggestions are quite often met with silence from the one guy who actually has commit access to the mainline trunk.

So, to resolve these issues for myself, I’m forking JSLint. I fully expect this to be of interest to almost exclusively myself, but I’ll find it useful, and I’ll hold out hope that some of the patches will eventually make it back into JSLint proper.

To resolve the issue of a base “official” repository, I’ll pull down a copy of the current iteration of JSLint on a daily basis, and mirror it on GitHub (jslint/mirror). My patches will be pushed to jslint/master (along with QUnit-based regression tests), and I’ll do my best to keep master merged with the latest from mirror, and the changes continually rebased on top of mirror into jslint/rebased for ease of application (in the unlikely event that Crockford pays attention :) ).

I’ll keep the changelog up to date as I add the functionality I find that I need to do my job. If you find the idea useful, I hope you’ll help me out. Reasonable patches most humbly accepted (although my arbitration of “reasonable” might hurt your feelings too).

A JavaScript Detection Pattern

2010-03-06T00:00:00+01:00

Progressive enhancement of our sites and applications has become a relatively well accepted best practice for web development. It simply makes sense to implement core functionality in a universally accessible way before layering new behaviors and possibilities for interaction on top via JavaScript.

This implementation model has one drawback that we need to consider. If we render a basic version of a module before reworking it with JavaScript, it’s possible that visitors would briefly be exposed to the unenhanced module, only to see it morph into something new before their eyes. This distraction is especially likely when we consider the well-known performance benefits associated with loading JavaScript at the very bottom of a page. The page as a whole will load well before JavaScript is completely parsed and executed, leaving a space of time during which the raw versions of your modules are visible.

To avoid this issue, I’d suggest inserting the following code directly after opening your page’s body element:

<script>document.documentElement.className += " js";</script>

When a visitor comes to the site with JavaScript enabled, this snippet will add a js class to the document’s root node (in most cases, the html element). This gives us a styling hook that allows us to generate CSS rules that only take effect when JavaScript is present. Prefixing rules with a .js selector ensures that they only apply when this script has executed, meaning that the visitor must have JavaScript enabled in her browser. We can use this information to pre-style the widgets in preparation for the JavaScript manipulations we’ll do later on.

I’ve put together a demonstration of this JavaScript detection technique in action. It’s worth visiting that demonstration both with and without JavaScript enabled, simply to get a feel for what’s possible. It’s not a perfect solution (JavaScript could error out somewhere in the middle of the page, for instance), but I find it to be a reasonable compromise that avoids distracting flashes of incompletely styled content.

That demonstration, however, is pretty abstract. Let’s look at a more practical example of how this could improve a site’s usability. Take Twitter, for instance: with JavaScript enabled, clicking on the “Sign in” button in the top right-hand corner of the page exposes the login form, which is otherwise hidden away. Without JavaScript, the form doesn’t show up at all, users are instead directed to a separate page that just displays this form.

This is a reasonable fallback, all things considered (and certainly better than sites like CNN, whose login link goes nowhere without JavaScript). Still, I see it as a missed opportunity as the login form’s HTML code is delivered to the user on the homepage regardless, it’s simply hidden by default. I think a better decision would have been to design the page such that the login form shows up for all users, and is simply presented differently for users with JavaScript.

I’ve implemented a demonstration of how this might work by copying down the Twitter homepage’s code, adding the JavaScript detection snippet from above, moving the login form HTML lower down on the page, and adding a few lines of CSS to make things halfway usable (a caveat: the twitter demo looks good in Chrome/Webkit, decent in Firefox, and miserable in IE: Twitter’s HTML is dependent upon server-side browser detection, which I’m not going to attempt to recreate here.). I’m no designer, but it seems like a step in the right direction to me, and adds a bit of bulletproofing with next to no effort expended.

With this in your toolkit, I don’t think there’s any excuse for mandating a hard requirement for JavaScript for most interactive widgets. Certainly complex applications would be hard-pressed to come up with ways of presenting the same functionality to all users, but I think we can all agree that “simple” functionality (like logging in) should be equally available to everyone, and we should absolutely take that into account when building websites.

Update: Tim Huegdon mentioned, rightly, that class isn’t actually a valid attribute on the html element, and that it might be better to set the js class on the document’s body element instead. It’s a valid point, one which ought not be ignored out of hand. I’m sticking with document.documentElement for a simple reason: I know it works stably in every browser I’ve tested (IE6+, FF2+, Safari 2+, Opera 9.5+). I’ve heard anecdotal evidence of problems in IE caused by manipulating the document’s body while it’s loading (“Operation Aborted”, and the like), which I’ve never experienced with this technique, and which I’d like to avoid.

Update to the Update: Martijn van der Ven notes, also rightly, that HTML5 allows class attributes (as well as all other global attributes) on the html element. One more reason to switch over to the HTML5 doctype, if you ask me.

Mike West - Posts Only

Secure Chrome extensions: Content Security Policy

Implementation

Next Steps

Content Security Policy: A Primer

mkw.st’s Policies

Nothing’s free.

Further Reading

HTTP Strict Transport Security and You

Sound interesting?

Chrome Privacy

I'm on Technikwürze

Things I’ll say differently next time.

Updates.

And that’s about it…

A Quick git/vim Workflow Tip

Intro to IndexedDB

Video

Embedded Slides

Slide Transcript

projects.mikewest.org

And now, the point…

JSLint needs some Bad Parts

A JavaScript Detection Pattern