I signed up to write about web literacy, but keep finding myself pulled back to teaching basic JavaScript programming as a webmaker skill and as a critical thinking skill. (And today's fireside chat about "algorithmic thinking" has only fanned the flames of that desire.)

I've created a very simple JavaScript-based turtle graphics library (and even tried it out on some second graders) and have been pondering some other kinds of math and science explorations that can be facilitated with JavaScript, SVG, and other web technologies. (Nothing published yet; I'll blog here when I've pushed a repo to github.)

So we're teaching webmakers by showing them how to remix websites, create online valentines, and (if I have my way) use turtles to draw pretty patterns. Each of these activities has a way (we'll, mine doesn't yet) to publish the results to a public web page so they can be shared with friends, shown to parents, and so on.

I had a minor epiphany about this today, which is why I'm writing this now. Here it is: we ought to allow junior webmakers to collect everything they publish, as well as the badges they earn, into an online "webmaker scrapbook". It would be like a blog or github for kids, but more colorful and fun.

Having a scrapbook helps with continuity: a young person might start something at one hackasaurus event, and then come back to another and continue it or create a new version of it. Keeping everything together also allows participants to go back and see how much they've learned and how far their skills have grown since they started. And I think the scrapbook metaphor sets a good tone for the collection of works: these are things that a young webmaker should be proud of and want to preserve and remember.

Finally, for kids who like scrapbooking, this idea gives them another compelling reason to learn HTML and CSS skills: so they can decorate their scrapbook in fun ways.

I'm not sure that I'm making any sense here. Your feedback is welcome in the comments!

Michelle Levesque works for the Mozilla Foundation and has been thinking a lot about web literacy under the name "webmaker skills". That is, she's working on listing the skills that Mozilla feels are most important in order for someone to be "web literate", where literacy means not just "reading" the web, but writing, or "making" it, too.

Mozilla is teaching webmaker skills through a number of really interesting projects and experiments including:

• HTML and CSS editing with Hackasaurus and Lovebomb.me.
• Web movies with Popcorn.js and PopcornMaker.
• Putting web makers into newsrooms through the Knight-Mozilla News Technology Partnership.

These projects tend to approach web literacy stealthily, and teach or promote it as a side effect of making cool stuff online. Given my background as a writer, however, I'll probably take a more direct approach and just write about web maker skills. (I don't expect that any of the teens learning HTML with Hackasaurus will ever read what I write, but perhaps the adults teaching them will...)

I can't write about web literacy the way I write about JavaScript, of course. Many webmaker skills are probably ones that are best learned by doing rather than by reading. And no one really knows exactly what we mean by "web literacy" anyway. Still, there are plenty of web literacy topics that can be addressed in prose, from "what is a web browser?" and "what is the difference between a URL and search query?" to "how do I write HTML and CSS" and "what is DNS?" Other crucial web literacy topics (harder ones to write about) include:

• Identity, anonymity and privacy on the web.
• Copyright, fair-use and other IP laws and the web.
• The difference between the open web and walled gardens erected within the web.
• How to detect scams and misinformation on the web.
• The seedy neighborhoods of the web: typo squatters, aggressive SEO, comment spam, etc.
• The dark side of the web: phishing, malware, etc.

(Please feel free to suggest other web literacy topics in the comments. Or if you know of particularly good writeups on these or other topics, let me know.)

I'm actually debating whether to use the term "web literacy" at all, or to just stick to "web skills". The word "literacy" carries a lot of academic baggage, and that some of that baggage, like the "digital native" vs. "digital immigrant" distinction, is bogus. The literacy metaphor is really quite appealing, but it might actually be more trouble than it is worth.

You may notice that one of the web skills not listed above is JavaScript programming. I've thought a lot about the question of how to teach kids to program with JavaScript, and it is an area that I'd love to work on. But its a hard problem, and others are tackling it, so it may not make sense for Mozilla to jump in. Still, its very tempting to put together some JavaScript explorations for kids. ]]> tag:www.davidflanagan.com,2012://1.223 2012-01-13T05:01:19Z 2012-01-23T17:07:38Z

That API has been evolving, and Kyle Huey from Mozilla has updated my example to work with the latest version of the API, which is implemented in Firefox 10. The main difference seems to be in the way that the database is initialized. Instead of the old setVersion() magic, you now set up the database in the handler of the "upgradeneeded" event.
That's quite a bit more sensible.

Kyle's version of the demo is here. The first query you make triggers the initialization of the database. It looks like there might be a problem with that first query not succeeding. But subsequent queries do work.

Thanks Kyle!

• The JS standard library is object oriented: we write a.map(f), not map(a,f). If the standard library defines classes, the language should make it easy to define classes.
• A number of popular JS libraries include convenience methods for defining and/or extending classes. This demonstrates to me that developers want to use classes and JavaScript, but that they want a simpler way to define them.
• Once we get used to the idea of a class keyword in the language, I think it is going to be really nice to be able to dash off little classes with that keyword, instead of the two step define the constructor and then set up its prototype object technique that we use now. In particular, I think it is going to feel really old-fashioned to define classes the old way once we've got modules in the language. I can't articulate this point well, but it is actually the one that is most compelling to me. We're getting modules in the next version of JS, and I don't think that the language will feel right without a class syntax to go along.
• The classes being proposed for JavaScript are all syntax sugar for existing patterns. Adding a class keyword will not change JS in any fundamental way, it will just make it easier to use the patterns that we already use. If you like defining your classes the old way, you'll still be able to do that. And you'll still be able to add methods to classes with C.prototype.new_method = function() {...} as you can today. As Brendan Eich would say, this is just "paving the cow paths". Here's what the classes proposal [Update link fixed now] describes as its motivation:

  ECMAScript already has excellent features for defining abstractions for kinds of things. The trinity of constructor functions, prototypes, and instances are more than adequate for solving the problems that classes solve in other languages. The intent of this strawman is not to change those semantics. Instead, it's to provide a terse and declarative surface for those semantics so that programmer intent is expressed instead of the underlying imperative machinery.

  I understand the reluctance that many JS programmers feel about adding classes. I feel it too, a little bit. A class keyword isn't "javascripty". I like, and understand the appeal of Allen Wirfs-Brock's approach: He defines extensions that make object literals more expressive, and then uses the <| and {. operators to create a simple class definition pattern. But the problem is that it is still a pattern rather than a new syntactic form. And to quote Dave Herman: "patterns are the sincerest form of feature request": if we're using patterns to define classes, then the language ought to just support classes.

  My desire to have classes in the next version of JavaScript of course has no bearing on whether we'll actually get them. The classes proposal linked above is apparently not doing well in committee, for reasons that I haven't tried to understand. There have been recent attempts to revive classes with simpler class proposals, so maybe there is still some hope. ]]> tag:www.davidflanagan.com,2011://1.214 2011-11-11T23:08:57Z 2011-11-29T04:33:29Z

  I've recently blogged about CoverMonkey, a code coverage tool for Mozilla's SpiderMonkey JavaScript interpreter and my JavaScript HTML parser. One of the really useful things you can do with a code coverage tool is check whether your test suites are...
  David I've recently blogged about CoverMonkey, a code coverage tool for Mozilla's SpiderMonkey JavaScript interpreter and my JavaScript HTML parser.

  One of the really useful things you can do with a code coverage tool is check whether your test suites are actually exercising all of the code you're trying to test. (A hat tip here to my colleague Donovan Preston who insisted on good tests for dom.js-CoverMonkey was his idea.) There is already a very good test suite for HTML parsers and I was able to get my HTML parser to pass those tests.

  That test suite is large and comprehensive, but when I ran CoverMonkey over the tests I found that there about 100 lines (out of some 5000) in my parser that did not get run by the tests. These were for pretty obscure things. Like what happens if you put a <!DOCTYPE> after <body>, or what happens if you have a NUL (\u0000) character in a <script> tag or what if you use &NotEqualTilde; character reference, which expands to two 16-bit characters rather than just one) in RCDATA state (like in a <xmp> element) ?

  I ended up writing 65 test cases to improve code coverage for my parser up to almost 100%. (There is still some untested code for handling <noscript> tags when scripting is disabled because the way the test suite is set up does not allow testing with scripting disabled). Anne van Kesteren was kind enough to grant me commit rights to the html5lib repo, and my tests are now part of the suite: here and here.

  I'm pretty pleased with this. It is a small thing, and there are only a handful of developers who will ever need to run these particular tests, but being able to use CoverMonkey to find and fix these few remaining holes in an otherwise excellent set of tests was pretty cool.

  (If you are one of those handful of developers who might actually run these tests, note that my parser does not do validation and does not report parse errors. My tokenizer test cases include parse errors where I think they should be reported, but I can't actually verify with my own parser that there are supposed to be errors in those spots. So the tokenizer tests might have errors related to error reporting. The parser test cases also do not include any errors, but I doubt that will break things for anyone, since the errors in the parser tests aren't standardized in any way and can't really be compared.)

  So, the moral of the story is: validating your test suites with a code coverage tool really works!

  ]]> tag:www.davidflanagan.com,2011://1.213 2011-10-27T06:21:53Z 2011-11-29T04:38:24Z
  The project I've been working on in Mozilla's research group is dom.js: an implementation of the browser DOM in JavaScript. I've gotten much of the core DOM 4 document tree functionality implemented, and have started on DOM features defined...
  David The project I've been working on in Mozilla's research group is dom.js: an implementation of the browser DOM in JavaScript. I've gotten much of the core DOM 4 document tree functionality implemented, and have started on DOM features defined by the HTML spec.

  One of those features is the innerHTML property. And to implement it, I need an HTML parser. So I dove in and implemented one. Here are some of the things I learned:]]>

• Long ago, browser vendors made the choice that browsers would not display error messages when given malformed HTML, but instead attempt to display the malformed input in some reasonable way. For web compatibility, all browsers today must do that. XHTML was a attempt at imposing a strict syntax on the web, and it failed.
• So for web compatibility, browsers must all parse HTML liberally and accept malformed input. And for web interoperability, browsers must all parse malformed HTML in the same way.
• The HTML parsing specification is long and complicated. This is because in addition to specifying exactly how to parse valid HTML inputs, it also specifies exactly how to parse all possible invalid inputs. A standardized parsing algorithm is, in fact, one of the most important (but least appreciated) features of HTML 5 and represents a huge amount of work by Ian Hickson.
• The HTML spec defines the parser in terms of a tokenizer stage and a tree builder stage. And both stages are defined as state machines (which you may remember from undergrad CS classes). The prose description of a state machine is verbose, but the translation to JavaScript code is fairly mechanical (in fact, I wrote a simple script to help me translate most of the tokenizer state machine from HTML prose to JavaScript code).
• The advantage of a state machine specification is that it can account for every possible input. What happens, for example, when the tokenizer sees <a href=>, for example, or reaches the end-of-file while in the middle of a comment? Since the tokenizer is specified as a state machine unexpected inputs like these are just state transitions and have a well-defined behavior.
• The flip side is that by specifying the HTML parser in terms of state machines, the HTML spec never explicitly says how the parser is supposed to behave when it is given <a href=> or other malformed input. The behavior is implicit in the transitions of the state machine, but to figure out what the parser does for any given input, you have to read through the spec, simulating the state machines in your head. There is not a formal grammar like you'd see when reading a programming language specification, just an algorithmic description of how the parser behaves.
• The HTML specification includes specific instructions on how to handle the </sarcasm> end tag. I think its a joke, though, since the instructions say to treat it like any other end tag. :-)
• The HTML specification integrates the SVG and MathML grammars, and as part of that, HTML adopts all the named character entities (things like &lt; and &quot;) of MathML. There are now 2125 named character references you can use, including &CounterClockwiseContourIntegral; and &InvisibleComma;!
• An HTML parser implementation cannot be separated from the HTML and DOM implementation of which it is a part. I had always assumed that an HTML parser could be written generically to build up a DOM parse tree using public DOM APIs like document.createElement() and Element.appendChild(). It turns out that there are a number of places where the parser must manipulate DOM nodes in ways that are not possible with the public API. These include setting the compatMode property of the Document object and setting the form property of HTML form elements like input fields and buttons.
• There is a really nice HTML parsing test suite (the tokenizer and tree-construction directories are the most important ones) and I gather that the major browser vendors all use it. Strangely, the documentation for using the test data is at a completely different site.

It took me a couple of weeks, but I learned a lot about HTML parsing and ended up with a working parser! The source code weighs in at over a quarter megabyte and 6300 sloc. (Though to be fair, I should note that the object literal that holds those 2125 named character references takes up over 1000 of those lines of code.) It passes the tokenizer and tree construction tests mentioned above, and it can parse the HTML specification (over 6 megabytes of HTML) in under 8 seconds.

If I can brag a bit, this 8 second speed easily beats the two other JavaScript-based HTML parsers that I know of. The Node-based parser at https://github.com/aredridel/html5 displays some kind of quadratic behavior (doubling the size of the HTML to be parsed quadruples the parsing time) and as far as I can tell (I gave up after 5 minutes or so) it is just not capable of parsing the HTML spec, at least not when used with the Node-based DOM library at https://github.com/tmpvar/jsdom.

The other JavaScript HTML parser I know about is the Validator.nu parser by my Mozilla colleague Henri Sivonen. This is a Java-based parser, but it includes a custom Java-to-C++ translator, and when translated to C++, it is used as Firefox's HTML parser. The same codebase can be translated to JavaScript using Google's GWT parser, but the result is not terribly efficient. Parsing the HTML specification takes 18 to 20 seconds with the GWT version of Henri's parser. (I thought seriously about modifying Henri's Java-to-C++ translator to output JavaScript instead of C++, but in the end, I decided that it would take a similar amount of time and would be more fun and educational to just write my parser from scratch.)

If you've read this far, here is where I confess that my parser isn't even done yet. It does not yet execute scripts, which means that it doesn't handle document.write() calls. That's next on my list, but its kind of scary. You'd think that executing a script in a JavaScript-based parser would just amount to an eval() call. But there's a lot more to it than that. Another colleague of mine described these sections of the spec as "brain-meltingly" complex. We'll see. I'm keeping my fingers crossed that none of the strange cases where document.write() inserts new scripts require me to refactor the parsing code in major ways! ]]> tag:www.davidflanagan.com,2011://1.212 2011-09-05T06:08:57Z 2011-09-05T06:16:25Z

Debug builds of SpiderMonkey have a -D option that causes them to output opcode-by-opcode disassembly of every file they've run, along with execution counts for every opcode. CoverMonkey analyzes this output and displays coverage statistics and generates color-coded versions of your source code to highlight uncovered and partially covered lines.

The first big caveat about CoverMonkey is that this is not a tool you can use in the browser. It works with standalone versions of SpiderMonkey only. So it is only useful if you're using pure JavaScript. The second caveat is that you'll need a debug build of SpiderMonkey, which means that you'll probably have to build it yourself. Another, smaller caveat: CoverMonkey is a JavaScript program, but unfortunately SpiderMonkey cannot create files, so CoverMonkey requires Node.