davidflanagan.com

My colleagues at the Mozilla Foundation want to "create a web literate planet" and "educate a generation of web makers". Some are working to figure out just what those phrases mean. While others are just going out and teaching stuff.

I signed up to write about web literacy, but keep finding myself pulled back to teaching basic JavaScript programming as a webmaker skill and as a critical thinking skill. (And today's fireside chat about "algorithmic thinking" has only fanned the flames of that desire.)

I've created a very simple JavaScript-based turtle graphics library (and even tried it out on some second graders) and have been pondering some other kinds of math and science explorations that can be facilitated with JavaScript, SVG, and other web technologies. (Nothing published yet; I'll blog here when I've pushed a repo to github.)

So we're teaching webmakers by showing them how to remix websites, create online valentines, and (if I have my way) use turtles to draw pretty patterns. Each of these activities has a way (we'll, mine doesn't yet) to publish the results to a public web page so they can be shared with friends, shown to parents, and so on.

I had a minor epiphany about this today, which is why I'm writing this now. Here it is: we ought to allow junior webmakers to collect everything they publish, as well as the badges they earn, into an online "webmaker scrapbook". It would be like a blog or github for kids, but more colorful and fun.

Having a scrapbook helps with continuity: a young person might start something at one hackasaurus event, and then come back to another and continue it or create a new version of it. Keeping everything together also allows participants to go back and see how much they've learned and how far their skills have grown since they started. And I think the scrapbook metaphor sets a good tone for the collection of works: these are things that a young webmaker should be proud of and want to preserve and remember.

Finally, for kids who like scrapbooking, this idea gives them another compelling reason to learn HTML and CSS skills: so they can decorate their scrapbook in fun ways.

I'm not sure that I'm making any sense here. Your feedback is welcome in the comments!

Mark Surman, the head of the Mozilla Foundation, has proposed that as part of Mozilla's mission ("to promote openness, innovation and opportunity on the web") we should work to "create a web literate planet". His vision inspires me: if the web is going to continue to be a revolutionary force in human affairs, we need a critical mass of citizens who understand the web deeply, who actively participate and build the web, and who can not be turned into mere consumers. So I've taken a break from my work on dom.js to work on education and web literacy.

Michelle Levesque works for the Mozilla Foundation and has been thinking a lot about web literacy under the name "webmaker skills". That is, she's working on listing the skills that Mozilla feels are most important in order for someone to be "web literate", where literacy means not just "reading" the web, but writing, or "making" it, too.

Mozilla is teaching webmaker skills through a number of really interesting projects and experiments including:

• HTML and CSS editing with Hackasaurus and Lovebomb.me.
• Web movies with Popcorn.js and PopcornMaker.
• Putting web makers into newsrooms through the Knight-Mozilla News Technology Partnership.

These projects tend to approach web literacy stealthily, and teach or promote it as a side effect of making cool stuff online. Given my background as a writer, however, I'll probably take a more direct approach and just write about web maker skills. (I don't expect that any of the teens learning HTML with Hackasaurus will ever read what I write, but perhaps the adults teaching them will...)

I can't write about web literacy the way I write about JavaScript, of course. Many webmaker skills are probably ones that are best learned by doing rather than by reading. And no one really knows exactly what we mean by "web literacy" anyway. Still, there are plenty of web literacy topics that can be addressed in prose, from "what is a web browser?" and "what is the difference between a URL and search query?" to "how do I write HTML and CSS" and "what is DNS?" Other crucial web literacy topics (harder ones to write about) include:

• Identity, anonymity and privacy on the web.
• Copyright, fair-use and other IP laws and the web.
• The difference between the open web and walled gardens erected within the web.
• How to detect scams and misinformation on the web.
• The seedy neighborhoods of the web: typo squatters, aggressive SEO, comment spam, etc.
• The dark side of the web: phishing, malware, etc.

(Please feel free to suggest other web literacy topics in the comments. Or if you know of particularly good writeups on these or other topics, let me know.)

I'm actually debating whether to use the term "web literacy" at all, or to just stick to "web skills". The word "literacy" carries a lot of academic baggage, and that some of that baggage, like the "digital native" vs. "digital immigrant" distinction, is bogus. The literacy metaphor is really quite appealing, but it might actually be more trouble than it is worth.

You may notice that one of the web skills not listed above is JavaScript programming. I've thought a lot about the question of how to teach kids to program with JavaScript, and it is an area that I'd love to work on. But its a hard problem, and others are tackling it, so it may not make sense for Mozilla to jump in. Still, its very tempting to put together some JavaScript explorations for kids.

The Mozilla Firefox Challenge celebrity fundraiser contest has ended, and the JavaScript community's entry, TeamJS did amazingly well:

• We raised $25,000 for Feeding America, a US charity that collects surplus food on a massive scale and distributes it to a national network of food banks that in turn distribute it to hungry people.
• The $25,000 we raised will enable Feeding America to obtain $400,000 worth of food, which works out to 200,000 meals for people who might otherwise not eat.
• After spending a week or two in first place we ended up 8th in the contest, behind 5 of the original celebrity entrants, and ahead of 7 of the celebrities. We raised more money than Barbra Streisand, Sean Penn, Will Ferrell, and four others, all of whom had a $5,000 head start courtesy of Mozilla.
• I'd like to believe that we also strengthened our JavaScript (and web developer) community, too!

$25,000 is a lot of money and it will do good for a lot of people. My heartfelt thanks to everyone who made this happen! You know who you are, and if you don't you can find the names of the non-anonymous donors at the TeamJS page. Look for the scrolling list, or use view-source to see the complete list more quickly. (Incidentally, even though the contest is over, you can still make donations to Feeding America through that page, so if your name isn't on that list, you can still make it appear.)

There are a few people and organizations that I'd like to single out for special thanks:

I started TeamJS on a lark (okay, because I was bored with my programming project) and I really can't take much credit for its success. I am proud of myself, however, for getting Chris Williams involved. As I said to him in my first email, "I think this might be your kind of crazy". It turns out I was right. Chris kicked things of with a big crazy donation of $5,000 from JSConf and then whipped the JavaScript community into a giving frenzy by offering a conference discount to anyone who gave $100 to TeamJS. Without Chris's early generosity I don't think TeamJS would have ever reached the $5000 mark. But it isn't just Chris's financial generosity: he's a natural leader and community builder and has the chutzpah to tweet outlandish statements like "the most important thing you'll do today is donate to TeamJS". It just kind of makes sense when Chris says it. Oh, and also, the black and yellow JS logo that TeamJS used was Chris's gift to the JavaScript community. Thank you, Chris, for all that you do!

I also want to thank the team at Jupiter JavaScript Consulting for a $1000 group gift. If I recall correctly, it came near the end of that first crazy day of donations after Chris kicked things off, and propelled us past $10,000 and into the big time.

As the contest drew to a close, TeamJS was languishing at around $14,000 All our momemtum was gone, and we were slipping down the leaderboard into 12th place. Out of the blue Cody Lindley matched Chris William's awesomeness and donated $5,000. (Cody is the author of JavaScript Enlightenment, which, by all reports, is an excellent book.) Cody inspired me to increase my own donation, and his big give really brought the excitement back. Thank you Cody! Your $5,000 donation put us within reach of our $25k goal and brought in almost $5,000 in smaller donations: on the last day we had about 60 new donations that brought us up to $23,780 just before the contest ended at midnight.

But the story, and the thank yous, don't end there. At 7 minutes to midnight, Peter deHaan donated $1,220 to bring us to exactly $25,000. How perfect for a team of geeks: we met our goal to 5 significant digits and finished exactly on schedule Thank you Peter, for closing out the contest in style: it was a perfect ending!

What a ride! Thank you everyone for being so awesome!

I've been thinking a lot recently about Mozilla's mission to promote the open web. Part of that mission, a part I'd like to be involved in, has been described as "educating the next generation of web makers" and as "creating a web-literate planet". And an important part of web literacy (and web making, of course) is the realization that the internet is not just for consumption of information, but that it is writable.

Do you remember what the web was like before blogs and wikis? The web was only writable if you were a "webmaster". (Does anyone even use that term anymore?) This meant, at a minimum that you could edit HTML and understood how to use ftp. (And a working knowledge of <frameset> and <marquee> was nice, too. :-)

Blogs changed this by making it radically simpler to publish content to the web. You didn't have to be a webmaster anymore, you could just be a blogger. It made a huge difference at the time. In this age of Twitter, though, blogging feels like a chore. My blogging software runs sluggishly on my webhost, there's always comment spam to deal with, RSS feeds are always a mystery, and my templates are out-of-date (I've been meaning to take the ads off the site, but haven't made the time.)

I don't use Facebook, but I gather that publishing content in Facebook is much easier than blogging. But (and this is a really important point to understand) Facebook isn't the open web, so it doesn't count.

It seems to me if the web is going to be truely writable, we need a paradigm shift that makes it even easier to create and publish content than blogging software does.

Atul Varma and Jess Klein have taken a step in this direction with lovebomb.me and the related Webpage Maker and webpad.hackasaurus.org. Their goal is to teach kids how to write HTML and CSS. But they don't want to send the kids off on a quest to register a domain, purchase a hosting plan and learn how to use scp for uploading pages. If kids are going to learn to create and edit web pages, they need a way to publish their work and immediately show it to friends, parents and teachers. So what each of these sites does is host the user's content for them at a short random URL, in the same way that sites like pastebin.com and jsbin do.

This is a great solution for lovebomb.me, hackasaurus, and jsbin: those sites have a relatively small community of users and are used for relatively specific purposes. But it doesn't scale to make the whole web writable (maybe it could scale if Google hosted it, and the "writable web" was just a collection of Google Docs documents :-). In addition to the issue of not having unlimited storage space, there is also the administrative and legal issues of how to deal with the inevitable problems of spam, malware, pornography, and hate speech that will come to infest anonymously writeable sites like these.

Many of us are already publishing our content all over the web, with the help of commercial or ad-supported services:

• blog posts at blogger.com, etc.
• photos at flickr.com, etc.
• source code at github.com, etc.
• music at mp3locker.com, etc.
• documents and spreadsheets at docs.google.com

So that brings me, finally, to the point of this post: who stores the writable web? Publishing content to the web ought to be as easy as File -> Save As. But where does that content go? Do I have to own a domain and pay for webhosting to be a web maker, or can I use a service to handle the storage details for me? And can I publish all my content in once place, or do I have to rely on different storage services for different types of content?

I don't have any answers, or even any proposals, really. Only questions:

• Can we decouple the way we refer to web content from the details of where that content is hosted?
• Do we need some higher-level URI scheme that allows web makers to store all their content in a single hierarchical namespace that is independent of domain names of the services that actually store that content?
• Can identity service providers (like BrowserId) tackle this problem by defining a namespace that refers to all the content defined by or owned by each identity?
• Can storage service providers define a working "Save As" feature for the writable web?

I'm a big fan of subscription music: I love the model of being able to listen to whatever you want whenever you want. Its all out there on the web and I don't have to be bothered with actually owning any of it. Compared to the cable TV or internet bill (or any other utility, really) $10/month seems like a small price to pay for unlimited music.

I'm an old-timer at this: I've been a Rhapsody subscriber for 5 years. If you've used Rhapsody, you probably know that it sucks. Their Windows client is built around IE. They still after all these years don't have a Mac or Linux client. Their servers often seem sluggish. Given the nature of the buisness they're in, their legal team probably dwarfs the size of their developer team. They haven't done anything innovative in years. I'm ready to get out.

There have always been other music services, but until recently, they've been about purchasing music, or have been radio format where you can't choose exactly what you listen to. But finally, Rhapsody has competition in the subscription business with services like MOG, Spotify and Rdio. MOG now has an app for my Squeezebox players, so I've signed up for a 2 week trial and am liking what I see so far. (Aside to Spotify: Facebook-only login? Really? I hear good things about you, but I'm not going to create a Facebook account to try you out.)

The problem is that after 5 years on Rhapsody, my family has hundreds of albums in "My LIbrary" (Rhapsody's term for your set of favorites) and quite a few long custom playlists. I don't want to start over from scratch to re-favorite those albums and re-create those playlists. Rhapsody owns my data and they've got me locked in. (More accurately, perhaps, Rhapsody controls my data and that's a barrier to me leaving them for a better service.)

Frankly, I'm shocked that new services like MOG have entered this market without an "import playlists" feature. Web browsers can import the bookmarks of other browsers. One of the first things MOG should have done when I signed up is to ask me for my Rhapsody credentials so that it could copy everything over. Seems like a no-brainer and a huge money-maker, so why hasn't anyone done this? Maybe there is fine-print in the Terms of Service that prevent it? Maybe all of the services are afraid they'd lose more customers than they'd gain if this became the norm?

Really, though, having MOG import my Rhapsody favorites and playlists isn't the best model anyway: then MOG controls my data and I'll run into this problem all over again if I decide to switch to some other service. The right model is that every music listener should own and control their own musical preferences. I should be able to edit my playlists using 3rd-party applications (personally, I'd like to use emacs!) and be able to sync them with any subscription services I choose. Logitech has an opportunity here to serve the owners of their Squeezebox players. Squeezebox owners (myself included) typically enter their subscription service credentials into the mysqeezebox.com site, so they are automatically in a perfect position to create a playlist transfer squeezebox app.

Really, though: this is probably a job for a trusted non-profit organization dedicated to protecting the open web: Mozilla! I imagine it would be a great project for Mozilla Labs. (I work for Mozilla, but not in Labs. I'm not speaking for Mozilla or Mozilla Labs in this post, of course.)

Technically, this shouldn't actually be too hard. Any subscription service with a Web interface (I think that is most or all of them) has an HTTP-based API for listing and editing playlists. And any service that supports Squeezebox players has a stable API that they're willing to let other devices rely upon. The Squeezebox code is open-source so it might even be possible to use some of that code. Listing a user's favorites and getting the contents of her playlists ought to be pretty easy. Adding favorites to a new service should also be relatively easy, I think. Creating and populating new playlists on a new service is probably the hardest thing. I don't think this is doable on a Squeezebox, so it may involve some reverse engineering of each service's web API.

The playlist migration and sync application I imagine would run in a web browser and would store music service credentials and playlists locally, with no server-side involvement. We'd have to write import and export modules for each music service and then also define modules for importing and exporting playlists to sharable open formats like plain-text, XML, and CSV.

I wonder if I can get something basic working before my 14-day free trial at MOG expires?
If I write any code, I'll put it up on github.

I'd love your thoughts! Please share your comments below.

If you weren't paying attention to the #teamjs hashtag on Twitter on Friday, you missed out on a lot of excitement and a great JavaScript community moment. If you were part of the fun, then you probably know all this already, and should see TeamJS Update and What's Next instead.

TeamJS is a group of JavaScript programmers and other web developers who are raising money for Feeding America as part of the Mozilla Firefox Challenge.

The Mozilla Firefox Challenge is a celebrity fundraiser, sponsored by Mozilla. Mozilla got 12 celebrities involved, donated $5,000 to their favorite cause and asked the celebrities to compete to raise more money through CrowdRise for that cause. Whichever charity raises the most money before January 11th will win an additional $25,000 from Mozilla.

But the competition isn't limited to the initial 12 celebrities: there are no celebrities behind TeamJS, and we didn't get any seed money from Mozilla. TeamJS started off as a lark to see whether the JavaScript and web developer community could donate some money and get the black-on-yellow JavaScript logo on the leaderboard among the celebrities faces.

But then something amazing happened! On Friday morning Chris Williams, the big-hearted organizer of JSConf (JavaScript's premiere conference) donated $5000 on behalf of JSConf. That put us on an even playing field with the celebrities and, when combined with donations from Wednesday and Thursday, it catapulted us into the middle of the pack.

That seed money opened the floodgates and the money began to pour in. When I left the office Friday evening, TeamJS was in second place. When I got home, we were first. The evening ended with another big $1,000 donation from Jupiter Consulting that brought us close to $11,000 in total donations. We were no longer just in first: we had a $2,000 lead and had left the celebrities in our dust. (As I write this Sunday night, we have raised $12,211 and second place is Sophia Bush with $9,990.)

It was a really amazing day, but what it means is that this is no longer a lark: we could actually win this contest. Please join us and let's make this an amazing week!

Read more about our charity, Feeding America and give whatever you can at the TeamJS page.

Friday was a lot of fun! We went from nowhere to first place with $12,000 in donations for Feeding America. But we were so successful that we actually have a chance to win, which means that we can't just stop now. So this post is about what has happened over the weekend and what comes next.

First, of course: Please give if you haven't yet, and please keep publicizing this good cause.

As of 9:30pm on Sunday, we had raised $12,211 from 84 donors. The (skewed) average donation was $145. The mode was $100 and the median was $50.

I've discovered that Feeding America has some really well-done videos that explain the hunger problem in America and what they're doing to address it. I've linked to my favorites from the TeamJS page at crowdrise.com

Speaking of the TeamJS page, I've I've made some changes to it:

• It includes more information about Feeding America, including the videos mentioned above
• It includes a bit of information about what "JS" is, since now that we're in first place, I expect that non-geeks will be clicking through to our page.
• I've upload JSconf and Jupiter Consulting logos to the page and they display as small icons beneath the big JavaScript logo, in recognition of the major gifts from those two businesses. (There are three spots left on the page: if you've got a business, a $1000 donation will get your logo on the page, until some other business gives more and knocks you out of the spot.)

I've got to hand it to crowdrise: they do a great job at online giving. I don't like that I have to reload the entire page to check for new donations, but the site is responsive and they do imbue the whole process with a sense of fun!

I've noticed that many donors (about a third or so) are creating crowdrise accounts and formally joining TeamJS as a fundraiser. That's great. Anyone who does that can use their own page as a donation URL to publicize and gets public credit for the donations they've brought in. You can post a photo, and you can participate in team discussions on the site, if we ever have any. But you don't have to do that to be part of TeamJS. If you donate, you're part of the team!

We're comfortably ahead for now, but I don't think we can dismiss the competition. These are people with hundreds of thousands of twitter followers, and they haven't really gotten serious yet. Also, three of the celebrites are old hands at fundraising on crowdrise. Barbra Streisand, Sophia Bush, and Ed Norton are the #2, #3 and #4 all-time biggest fundraisers on the site. So we should expect them to come on strong. (And if they do come on strong, it is in part because we've forced them to work harder, and everybody's cause wins!)

So what's next? How can we keep the momentum going? Please share your ideas in comments, by email, on twitter, or on the crowdrise site. Should we have a TeamJS presence on Facebook or Google+? Go for it! Can we get other programming language communities involved? Can we attract other business donations? Can we broaden the appeal of the campaign beyond the tech community? Everyone one is welcome to dive in and help in whatever way you can!

(If you don't live in the US, you can skip it.)

The US Census Bureau calculates that in 2008, 15% of US households were "food insecure". A slightly older version of that data is broken down by county and state in this excellent interactive map.

With continued high unemployment, that 15% figure has surely gone up since 2008. These are the households that may have to choose between food and heat this winter. If, like me, you can't even imagine what it must be like to have to make that choice, you're probably in a position to help.

Its just crazy that there's a major hunger problem in a country with huge food surpluses. That's why one of my favorite charities is Feeding America (you may have heard of them under their old name "America's Second Harvest). They're a national organization that collects surplus before it is thrown out and distributes it to a network of food banks that distribute it to people in need.

And they're very efficient at it: each dollar donated provides eight meals. And they've got very low administrative and fundraising overhead: 96% of their money goes to their hunger programs.

Please help, if you can.

And if you're going to donate money to Feeding America, a fun way to do it is to participate in the Mozilla Firefox Challenge as part of TeamJS: I'm challenging the US JavaScript and webdev community to donate more to this cause than a dozen Hollywood celebrities can raise for their causes. I think it would be great to see the bright yellow JS logo (and the Rhinoceros from my book because I think it is funny :-) in the middle of that pack of celebs. (Click on the Firefox Challenge link above for a visual of what I'm talking about.)

I've been flogging this pretty hard on Twitter (#teamjs and #fxchallenge), with fairly disappointing results. I'm not ready to give up, but I am feeling discouraged, and I could really use some help. If you think the challenge is a fun way to raise money for a good cause, please help publicize it. I don't use Facebook or Google+, so help with those networks would be especially great. If you sign up for an account on crowdrise.com, you can even join the TeamJS team.

Update: discouraged no more! Chris Williams and JSConf have made a big donation and a huge publicity push. TeamJS is now in 4th place!

And if you don't live in the US (but read this anyway) and would like to raise money for a cause that is not US-centric, you can create an account at crowdrise.com and then create your own team in the Firefox Challenge-maybe you can embarrass the US JavaScript community into giving more!

Mozilla is sponsoring The Mozilla Firefox Challenge, a celebrity fundraiser. Mozilla has donated $5000 to the favorite charity of 12 Hollywood celebrities and will donate $25,000 more to the charity of whichever celebrity can raise the most for their cause.

I want to challenge the JavaScript community, and the web developer community, and heck, the whole geek community to go head to head with the celebrities and raise more money than they can. So I've formed TeamJS to compete against the celebs in the challenge. We don't get the head start of $5k seed money, but we can still win the $25K prize!

The charity I've picked is Feeding America, which describes itself as a sort of food bank for food banks. They're a national organization that collects surplus food and distributes it, via a network of independent food banks, to those who are hungry. And they're remarkably efficient at what they do: each dollar donated provides 8 meals.

I kicked things off with a $500 donation. Please contribute what you can, and spread the word!

(Feeding America is a US charity, but the international JavaScript community is more than welcome to participate. I've asked crowdrise if they'll allow me to designate different charities on different days, so we can pick others that appeal to geeks in other parts of the world, but I don't actually know if that will be possible.

Update: crowdrise.com says that I can't switch charities once donations have been made. If you want to participate but want to give to a different charity, you can form a new team. TeamJS.eu anyone? Since crowdrise is in the US, the charities have to be registered in the US, but that still leaves lots of good ones with an international focus... )

Back in January I posted an IndexedDB example that showed how to initialize and query a database of zipcodes using the HTML5 IndexedDB API.

That API has been evolving, and Kyle Huey from Mozilla has updated my example to work with the latest version of the API, which is implemented in Firefox 10. The main difference seems to be in the way that the database is initialized. Instead of the old setVersion() magic, you now set up the database in the handler of the "upgradeneeded" event.
That's quite a bit more sensible.

Kyle's version of the demo is here. The first query you make triggers the initialization of the database. It looks like there might be a problem with that first query not succeeding. But subsequent queries do work.

Thanks Kyle!

I had a little twitter rant this morning about classes in JavaScript. (Actually, the rant was really about the attitude that seems to be behind much of the "don't add classes to JavaScript" commentary I see.) So I should probably explain in somewhat longer form why I want classes added to the language.

• The JS standard library is object oriented: we write a.map(f), not map(a,f). If the standard library defines classes, the language should make it easy to define classes.
• A number of popular JS libraries include convenience methods for defining and/or extending classes. This demonstrates to me that developers want to use classes and JavaScript, but that they want a simpler way to define them.
• Once we get used to the idea of a class keyword in the language, I think it is going to be really nice to be able to dash off little classes with that keyword, instead of the two step define the constructor and then set up its prototype object technique that we use now. In particular, I think it is going to feel really old-fashioned to define classes the old way once we've got modules in the language. I can't articulate this point well, but it is actually the one that is most compelling to me. We're getting modules in the next version of JS, and I don't think that the language will feel right without a class syntax to go along.
• The classes being proposed for JavaScript are all syntax sugar for existing patterns. Adding a class keyword will not change JS in any fundamental way, it will just make it easier to use the patterns that we already use. If you like defining your classes the old way, you'll still be able to do that. And you'll still be able to add methods to classes with C.prototype.new_method = function() {...} as you can today. As Brendan Eich would say, this is just "paving the cow paths". Here's what the classes proposal [Update link fixed now] describes as its motivation:

  ECMAScript already has excellent features for defining abstractions for kinds of things. The trinity of constructor functions, prototypes, and instances are more than adequate for solving the problems that classes solve in other languages. The intent of this strawman is not to change those semantics. Instead, it's to provide a terse and declarative surface for those semantics so that programmer intent is expressed instead of the underlying imperative machinery.

  I understand the reluctance that many JS programmers feel about adding classes. I feel it too, a little bit. A class keyword isn't "javascripty". I like, and understand the appeal of Allen Wirfs-Brock's approach: He defines extensions that make object literals more expressive, and then uses the <| and {. operators to create a simple class definition pattern. But the problem is that it is still a pattern rather than a new syntactic form. And to quote Dave Herman: "patterns are the sincerest form of feature request": if we're using patterns to define classes, then the language ought to just support classes.

  My desire to have classes in the next version of JavaScript of course has no bearing on whether we'll actually get them. The classes proposal linked above is apparently not doing well in committee, for reasons that I haven't tried to understand. There have been recent attempts to revive classes with simpler class proposals, so maybe there is still some hope.

I've recently blogged about CoverMonkey, a code coverage tool for Mozilla's SpiderMonkey JavaScript interpreter and my JavaScript HTML parser.

One of the really useful things you can do with a code coverage tool is check whether your test suites are actually exercising all of the code you're trying to test. (A hat tip here to my colleague Donovan Preston who insisted on good tests for dom.js-CoverMonkey was his idea.) There is already a very good test suite for HTML parsers and I was able to get my HTML parser to pass those tests.

That test suite is large and comprehensive, but when I ran CoverMonkey over the tests I found that there about 100 lines (out of some 5000) in my parser that did not get run by the tests. These were for pretty obscure things. Like what happens if you put a <!DOCTYPE> after <body>, or what happens if you have a NUL (\u0000) character in a <script> tag or what if you use &NotEqualTilde; character reference, which expands to two 16-bit characters rather than just one) in RCDATA state (like in a <xmp> element) ?

I ended up writing 65 test cases to improve code coverage for my parser up to almost 100%. (There is still some untested code for handling <noscript> tags when scripting is disabled because the way the test suite is set up does not allow testing with scripting disabled). Anne van Kesteren was kind enough to grant me commit rights to the html5lib repo, and my tests are now part of the suite: here and here.

I'm pretty pleased with this. It is a small thing, and there are only a handful of developers who will ever need to run these particular tests, but being able to use CoverMonkey to find and fix these few remaining holes in an otherwise excellent set of tests was pretty cool.

(If you are one of those handful of developers who might actually run these tests, note that my parser does not do validation and does not report parse errors. My tokenizer test cases include parse errors where I think they should be reported, but I can't actually verify with my own parser that there are supposed to be errors in those spots. So the tokenizer tests might have errors related to error reporting. The parser test cases also do not include any errors, but I doubt that will break things for anyone, since the errors in the parser tests aren't standardized in any way and can't really be compared.)

So, the moral of the story is: validating your test suites with a code coverage tool really works!

The project I've been working on in Mozilla's research group is dom.js: an implementation of the browser DOM in JavaScript. I've gotten much of the core DOM 4 document tree functionality implemented, and have started on DOM features defined by the HTML spec.

One of those features is the innerHTML property. And to implement it, I need an HTML parser. So I dove in and implemented one. Here are some of the things I learned:

I've turned off comments for all entries on this blog, since I had over a thousand spam comments over the last couple of weeks. Most of them never got published, but dealing with even the small fraction that got through the comment spam filters was too much work.
I think the problem was that I hadn't been closing old comment threads, so the blog presented a good target to comment spammers. Maybe now if I just leave the comments open for the most recent one or two entries the spam problem won't be so bad.

I started work at Mozilla in May, and haven't had a chance to update my blog since then. I've been working on a project to implement the DOM in JavaScript. One of the interesting side-projects that has come out of this is a code coverage tool for SpiderMonkey (Mozilla's js interpreter). I'm calling it CoverMonkey, and have just created a repo for it on github. Take a look at some example output.

Debug builds of SpiderMonkey have a -D option that causes them to output opcode-by-opcode disassembly of every file they've run, along with execution counts for every opcode. CoverMonkey analyzes this output and displays coverage statistics and generates color-coded versions of your source code to highlight uncovered and partially covered lines.

The first big caveat about CoverMonkey is that this is not a tool you can use in the browser. It works with standalone versions of SpiderMonkey only. So it is only useful if you're using pure JavaScript. The second caveat is that you'll need a debug build of SpiderMonkey, which means that you'll probably have to build it yourself. Another, smaller caveat: CoverMonkey is a JavaScript program, but unfortunately SpiderMonkey cannot create files, so CoverMonkey requires Node.

The examples from the brand-new 6th edition of JavaScript: The Definitive Guide are available for download from O'Reilly and from Github.

These are ten of my favorites from the book. Note that many of these use new features of ES5 or of HTML5, and will only work in the newest browsers:

1) Example 1-1 is is an extended example in the first chapter of the book, intended to show readers a simple but non-trivial example of JavaScript. This is the loan calculator example from the last edition, but made much more interesting with the addition of client-side graphics, localStorage, and Ajax.

2) Example 9-7 emulates Java-style enumerated types in JavaScript. It demonstrates that JavaScript's prototype-based inheritance is so flexible that factory methods can be normal object factories or even class factories. That example is a little clearer if you look at some code that uses it.

3) Example 9-16 defines a class hierarchy of abstract and concrete Set classes. This one is a favorite because it involves data types and API design. Chapter 9 includes a number of other Set examples, too.

4) Example 9-23 demonstrates the ES5 Object.defineProperty() method and defines a convenient way to inspect and modify the attributes of the properties of an object. It may not be practical, but I think it is a beautiful hack.

5) Example 15-10 is a simple stream-like API wrapped around the innerHTML property of an element. When you're generating text (a table, for example) for display it is sometimes easier to pass each chunk that you compute to a write() method than it is to concatenate it all together and set it on innerHTML.

6) Example 21-03 is an analog clock implemented as an SVG graphic with scriptable hands. I love client-side graphics, and this is a favorite of mine because making the hands rotate is so simple with SVG transforms.

7) Example 21-06 draws a fractal Koch snowflake using the <canvas> tag. I like it because it draws the same line over and over again, but uses transformations to make the line appear at different locations, orientations and sizes. You can view the snowflakes live in your browser.

8) Example 21-13 is another graphical example: it draws sparklines. This one is a favorite just because sparklines are so cool.

9) Example 22-1 uses the HTML5 geolocation API to find out where you are then uses the Google Maps API to obtain a static map of your location. I like it because geolocation (via wifi networks) is just pure magic!

10) Example 22-15 is a long example that demonstrates the IndexedDB API. I like it because the idea of a client-side database in a web browser is crazy and cool. This one is really cutting-edge, but if you're running Firefox 4, you can try it out live.

Wow! My previous post about the piracy of my JavaScript book generated an amazing number of insightful comments (and some idiotic ones, too). I don't think I can cover this issue comprehensively, but here are a few responses.

• First of all, let me restate the main points of my earlier post: it is personally very discouraging to me to see my books pirated, and I am dismayed by how easy it has become in the last few years to find illegal copies of my books. My suggestions about how Google might alter its suggestions and search results were sincere but secondary.

• Also: I understand (and stated in the post, though not as explicitly as I could have) that piracy has less financial impact on me than other trends affecting the publishing industry: technical books in general seem less relevant today than they did in the past. Nevertheless, I get myself worked up over piracy because it feels personal.

• Mark Pilgrim wrote a prickly response to my post. He makes a few good points (I did not know, for example, that Google was already filtering terms like "rapidshare" out of its suggestions). But he also gets worked up over the fact that when I wrote my post my book was not yet actually available from the download sites. (It is now, of course.) I disagree with Mark's main point "the book is dead". The tech book industry is ailing (as my declining royalties checks illustrate), but even tech books aren't dead yet.

• I suspect that this tweet of mine about why Mark could release a free version of his book and I could not may have had something to do with the prickliness of Mark's response. In fact, a far more relevant distinction (that would not have fit into a tweet) is that Mark's book was a new one, and he had an agreement with O'Reilly that allowed him to do that, where mine was a new edition of an old book, still governed by a contract from 1997 or thereabouts. Releasing a free version of my book is simply not a legal option I had. (Had I asked for it, O'Reilly might have agreed, but it would have been their decision, not mine.) Mark: if I offended with that tweet, I apologize.

• O'Reilly ought to do a better jobs of SEO so that its ebook sales pages are at the top of the Google search results for typical pirate searches.

• No one knows how to price ebooks yet. I think that O'Reilly is actively working on this. I agree that it is absurd that Amazon's kindle price is higher than the print price, and that O'Reilly's ebook price is so high, too. I don't think I have any control over this. Note John Seifarth's comment, though: he had registered his 5th edition with O'Reilly and they offered him a $5 electronic copy of the 6th edition.

• If I were to write a completely new book, I might well try self-publishing it at a much lower price point that enabled me to retain a much larger share of the revenue. This would mean, however, that it wouldn't be professionally indexed or typeset or copyedited.

• I'm surprised how many commenters were willing to confess to book piracy on the blog of an author who is upset about book piracy. You do know that blog comment systems always log your IP address, don't you?

• I agree with many commenters that ebooks could be so much better than they are today. O'Reilly is thinking about this, I know. Being able to push updates out to ebook customers would be a really big win.

• A number of commenters suggested that I put a donation button up on my website so that people who enjoyed, but did not pay for, my books could send me money. I might try it, though there might be legal issues. And it also feels a bit like surrender, or a way of giving my blessing to piracy.

• Someone calling his or herself "Leonid Brezhnev" has drunk the pirate kool-aid and articulates it this way:

  When you steal something tangible, the owner loses what he had.

  When you copy something digital, the owner has lost nothing.

  Copyright is meaningless when there are no physical limitations. Reality is knocking at your door and taking your money - it's up to you to get another job or try to screw up other people's liberties trying to enforce an artificial law.

  Finally, I find it offensive that you claim ownership for the equivalent of one long number.

  First of all, Leonid, copyright laws predate digital media by many, many years. They prevent publishers from simply re-printing and re-releasing each other's books. There are different laws that prevent publishers from breaking into each other's warehouses and stealing books. Copyright law establishes a framework within which publishing can work as a business. Without it, there would just be a race to the bottom, margins would get very thin, and only crap would be published. Maybe the fair-use exceptions are too narrow or the copyright protection lasts for too long, but the basic idea of copyright has been around for so long because it serves an important purpose. Copyright really has nothing to do with digital vs. analog or electronic vs. physical.

  Justifying piracy by equating a book with a long number is just silly. If you want to critique copyright from a Marxist or Anarchist perspective (as a tool for wealthy publishers to oppress the masses, perhaps) you could probably do that in an intellectually consistent way, and I'd at least listen to you respectfully. But the number thing is ridiculous.

Every time a new book of mine comes out, I get myself worked up about piracy. I've been tweeting about it this time, but there are nuances that require a longer form. The title of this post is a reference to Google's suggestions to people searching for my book:

Maybe if enough people link to this post, then those search suggestions will bring would-be pirates to this page instead of to the ebook piracy sites :-) [Update: Hooray, as of tonight, for each of the search suggestions pictured above, this blog post is ranked higher than any ebook download site! Sweet symbolic victory!]

So, in no particular order, but in more than 140 characters, here are some of my thoughts about book piracy.

I'm well-aware of Tim O'Reilly's public statements about piracy. And I agree with some of them. Obscurity is a worse problem than piracy, for example. And I know that some authors have been successful at selling books even while making them free (and legal) for download. I know that some genre fiction authors have been successful by self-publishing ebooks for 99¢. (I don't know that world well enough to link to any of those authors, though.) For most of these authors, however, "success" means "I'm making more money selling books for 99¢ than I did when I sold them for $3.99". There are only a few lucky authors who can actually make a living and support a family by selling 99¢ ebooks. I don't think self-published 99¢ ebooks is the way forward for technical books.

For 15 years I've been one of those lucky authors who has been able to support himself and his family almost entirely on book royalties. But the publishing industry has been in decline and my royalties checks have decreased more-or-less steadily since the dot-com bust, and I've now decided that I need to look for a salaried job. This is kind of the end of an era for me. (So if I sound whiny, keep in mind what is at stake for me. And if I can't whine on my own blog, where can I whine? :-)

I do not know whether or to what extent piracy is responsible for my declining income. I suspect that the internet and the transition from print books to ebooks has more to do with it than piracy does. But I also suspect that piracy has a non-trivial impact, too.

But beyond the non-quantifiable financial impacts, I can report that, to me, the piracy of my books is profoundly discouraging. When my Ruby book came out in 2008 I was sad to discover that pirated copies were available within a week or so of the book's release. When my jQuery pocket reference came out earlier this year, I was shocked to discover that Google was giving the ebook download sites higher placement than reviews of the book. And now JavaScript: The Definitive Guide is out. I don't have a copy of it yet, but illegal copies are free for anyone who wants one. And Google will suggest those illegal downloads to anyone who tries to research the book (see the screenshot). I've worked really hard on this book, and I've got to say that this just feels like a kick in the gut.

I was trying to be provocative when I tweeted the question "Does Google enable piracy?" But I do think it is a valid question. If Google indexes sites like ebookee that link directly to download sites and makes it easy to find the pirated content you want and even offers suggestions on what to search for, I think there is a case to be made that they're encouraging piracy. And an important part of this is that the ease of finding illegal ebooks has removed any stigma involved. Malware sites get flagged "may harm your computer". Porn images get filtered by default in Google image search. But illegal copies of ebooks? They're just out in the open--it must be okay to download them.

I know that electronic piracy cannot be defeated. And I don't think we should (or can) lock everything down with draconian DRM. But I also think that a laissez-faire attitude toward piracy ("well, it is better than obscurity" or "its going to be pirated anyway, so you might as well just make it free") is the wrong answer. Even if publishers cannot win against pirates, they should at least fight for a stalemate rather than accept defeat.

Here are some small steps that might help:

• Google could filter its search suggestions so that they do not actively suggest piracy. I suspect that Google already filters the suggestions offered when someone enters the name of a porn star, for example. Google already has a database of copyrighted books (Google Books) so they could easily filter the suggestions offered when someone searches for a book title.

• Google could flag (without filtering) search results that are likely links to pirated content. Google already flags some results with "this site may harm your computer". Why not flag pirate sites: "Downloading content from this site may result in legal action by the copyright holder" or "Downloads from this site may be illegal". Or nice and simple: "this site may harm your karma".
  

I'll close by quoting the sidebar I titled "A Note about Piracy" in the preface of my new book:

If you are reading a digital version of this book that you (or your employer) did not pay for (or borrow from someone who did) then you probably have an illegally pirated copy. Writing the 6th edition of this book was a full-time job, and it took more than a year. The only way I get paid for that time is when readers actually buy the book. And the only way I can afford to work on a 7th edition is if I get paid for the 6th.

I do not condone piracy, but if you have a pirated copy, go ahead and read a couple of chapters. I think that you'll find that this is a valuable source of information about JavaScript, better organized and of higher quality than what you can find freely (and legally) available on the Web. If you agree that this is a valuable source of information, then please pay for that value by purchasing a legal copy (either digital or print) of the book. On the other hand, if you find that this book is no more valuable than the free information on the web, then please discard your pirated copy and use those free information sources.

Update: Thanks everyone for the kind words and the thoughtful comments! I've replied to most of your comments tonight, but I really have to get serious about writing my jsconf presentation, so I may not be able to reply to any more for a couple of days.

Update: More great comments! I'm really wishing I had a threaded comment system on this blog. Its never been an issue before

Firefox 4 partially implements the TypedArray specification, but, sadly, leaves out the DataView class. Typed arrays are cool, but they use native byte endianness, which means that you should not use them for working with data from files or from the network. The DataView class allows control over byte order, which is why it is important.

I've implemented DataView for Firefox 4, using the existing TypedArray classes that Firefox does support. (Chrome and, I assume, Webkit nightlies, already have DataView, and do not need this library.)

The DataView API is sort of sucky, so the next step is to define a useful binary API on top of DataView.

Update: This morning I see phrases like "radiation released into the environment" (dailykos.com blog), "low levels of radiation wafting into Tokyo" (Reuters news story) and "fleeing...amid fears that radiation...will reach the city. " (ctv.ca)
The authors of each of these quotes have confused radiation with radioactivity. Radioactivity is much worse than radiation. If you don't understand the difference, please read on.

I didn't really understand the difference between radiation and radioactivity until about 5 years ago, so for all of you who have also missed the distinction, consider this a public service announcement. (But keep in mind that I am not a physicist, and may have some of the details wrong here.)

Radiation is energetic particles, such as photons (gamma rays), electrons (beta particles), helium nuclei (alpha particles) or neutrons (the really deadly ones). They have enough energy that when the collide with cells in your body, they can cause damage to the cells. If enough of your cells are damaged by radiation, you get sick. If the radiation damages your DNA, it may increase your chance of getting cancer.

You can protect yourself from radiation with shielding. The amount of shielding required depends on the type of particle and how much energy it has. Alpha particles are easiest to shield against; neutrons are hardest, I think. Radiation also decreases dramatically with distance, so if you are far away from radiation, you're safe from it. [In comments, FishyJoe correctly points out that the UV radiation from the sun can still be dangerous at great distances.] When the energetic particles being radiated collide with something, then their energy dissipates and they are no longer dangerous.

Radioactivity is the process by which certain substances (such as Uranium) release radiation. In contexts like the unfolding nuclear disaster in Japan, the term "radioactivity" is also used to mean those radioactive substances themselves. And that is a completely different and scarier thing.

If a failed nuclear power plant is releasing radiation that means that people nearby (like just outside of it) are at danger of being irradiated if they don't have proper shielding. Just like if you had lots of xrays without the lead shield.

But if a failed nuclear power plant is releasing radioactivity, that means that little bits of radioactive matter are floating around, blowing with the wind, being breathed in, being tracked around on people's clothes and shoes and so on. And each little radioactive particle is releasing radiation that whole time This is particularly bad if the particles get inside your body. This is the danger of radioactive fallout.

I see the phrase "radiation leak" used in the press about Japan. That is a strange phrase if they're really talking about radiation. Radiation escapes, maybe, but I don't think I'd talk about it leaking as if it were a liquid or a gas. I fear, therefore, that the phrase "radiation leak" as used in news stories really means "leak of radioactive substances". (The same thing goes for the phrase "radiation released into the environment".) A release of radioactivity is a lot worse, and would explain why people who live 20km away from the plants are being told to stay indoors. Radiation being released from a crippled reactor would not be dangerous at 20km, but radioactive isotopes being released into the air 20km away can come to you and get inside your body and expose you to radiation from up close.

The units used to measure radiation and radioactivity are really confusing. Physicists use one set of units to describe the radiation emitted by a radioactive source and biologists and nuclear medicine people use other units to measure the radiation dose that is actually absorbed. And there are old units and SI units. 1 Becquerel (Bq) is one radioactive decay per second. 1 Curie (Ci) is 3.7 billion Bq. Millicuries (mCi) are more commonly used that Curies. Becquerels and Curies just measure how many particles of radiation are being emited, but don't say anything about the energy of those particles.

The unit I've seen used about the Japan disasters are Sieverts. These measure the a dose of radiation and attempt to quantify the human biological effects of radiation. The older, non-SI unit is rem, which stands for "Roentgen Equivalent Man". (Roentgen is yet another unit, that has something to do with how much ionization in air is produced by radioactivity.) Note that a dose of radiation is the product of the amount of radiation and the time you're exposed to it. So you'll see radiation levels quoted as some number of millisieverts per some unit of time. If you were exposed to that level of radiation for that amount of time, that is the radiation dose you would receive.

I've read that Japanese authorities are distributing potassium iodide (KI). So I'll end with a note about that. Some of the common radioactive substances that occurs in nuclear reactions are isotopes of Iodine. As radioactive elements go, they're not all that bad. But, and this is a big but: our bodies concentrate iodine in our thyroid glands. So if you breathe or eat radioactive iodine, it will quickly concentrate in your neck, and may well cause thyroid cancer. If, however, your system is already flooded with iodine, then your thyroid doesn't need anymore, and most of that radioactive iodine will instead be flushed out of your body in your urine, and will do much less harm.

So, if you take KI before you are exposed to radioactive iodine, it will protect you, to some extent. It does not protect you from radiation. And it does not protect you from other kinds of radioactivity, such as Strontium which can concentrate in your bones. And, if you have already had your thyroid gland removed (because of thyroid cancer) then there is no need to take KI.

That's just about all I know on these matters. I hope you found it helpful. If I've made mistakes, feel free to correct me in the comments.

The online Rough Cut edition of JavaScript: The Definitive Guide has been updated, and is now feature complete. All the chapters, examples, and reference entries that will be in the book are finally in the rough cut. (The index has not been created yet, though).

If you're willing to order the printed book, at the cover price, for delivery in May, you can get online access to the rough cut for free right now: http://oreilly.com/catalog/9780596805531/.

One caution: the client-side reference section is not fomatted quite right in this rough cut edition: the class hierarchy information has been appended into the title of the reference entires. So when you look up the File object, for example you see a page named FileBlob (because File inherits from the Blob) And if you look up Form, the page title is "FormNode, Element" because Form inherits from Node and Element. Sorry about that!

The online Rough Cut edition of JavaScript: The Definitive Guide has been updated, and is now feature complete. All the chapters, examples, and reference entries that will be in the book are finally in the rough cut. (The index has not been created yet, though).

If you're willing to order the printed book, at the cover price, for delivery in May, you can get online access to the rough cut for free right now: http://oreilly.com/catalog/9780596805531/.

One caution: the client-side reference section is not fomatted quite right in this rough cut edition: the class hierarchy information has been appended into the title of the reference entires. So when you look up the File object, for example you see a page named FileBlob (because File inherits from the Blob) And if you look up Form, the page title is "FormNode, Element" because Form inherits from Node and Element. Sorry about that!

A few commenters on my last post were curious about my decision to include jQuery coverage in my JavaScript book. There was even a little bit of outrage on Twitter. So I thought I'd clarify:

The 6th edition of my book covers jQuery because:

• jQuery is probably the most popular framework

• it is small enough that I can cover it in one chapter

• for document manipulation, event handling, and simple animations, jQuery gets a lot of things right

One chapter of my book is dedicated to jQuery, and jQuery is discussed only in that chapter. I do not use jQuery in examples in other chapters.

The book also includes an introduction to Node as part of a chapter that also covers Rhino.

I turned in the final chunk of js:tdg6 to the O'Reilly production department yesterday. So I'm basically done with the book! (Well, okay, I still have to update the preface, check the galleys etc., but the serious writing is finally, finally done.)

Here are some stats:

• Covers: ECMAScript 5, HTML5, jQuery

• Words: 396,582

• Pages: 1024 (nice number, but the index will add another 50 or so...)

• Lines of code: ~11,500

• Chapters: 22 (plus core and client-side reference sections)

• New or completely rewritten chapters: 15

• Months of work: 21

I don't know when you'll be able to get a copy. Its out of my hands now and I don't yet know the production or printing schedule. A long book like this takes a while though.
You can buy a digital copy of the Rough Cut however.