The Sourcerer

Got Style?

There's a new JavaScript library from Google on the block. So, I'm of course out to absorb the good bits for Chiron. I've got to wonder some times whether people have started to use ideas from my code.

Compare this bit from Google's JavaScript library with this one in Chiron.

I don't hear much about people using Chiron. Could this be a sign that people have been reading it and absorbing the good bits? There are other parallels. For example, some of the function names and file names match up with Chiron too, like base.js and inherits (instead of Doug Crockford's recommended term, begets).

Re: Recommendations for the Fourth Edition of the ECMAScript Programming Language Specification

Doug Crockford has posted some recommendations to improve JavaScript.

I have some comments in the context of my experience with live-patching JavaScript issues with libraries.

New Features

Function Reflection

Doug recommends that function objects contain a "name" property. Some versions of JavaScript actually have this already. This could be handy but I haven't needed this and would not be likely to use it. I was surprised that FireFox had this when I used a Function in a with block. I've learned my lesson.

However, Doug also recommends the addition of an arguments member to Function objects. This is a critical feature. The arguments array would contain the names of all of the functions declared argument variables. Function objects already have a length member that corresponds to the length of this hypothetical arguments array. There was some buzz about using this value for method overloading and it permitted me to create an elegant partial application decorator. If we had access to the names of a function's argument declarations, I could implement a decorator for supporting Python-style positional and named arguments. Decorated functions would accept an array/list of positional arguments and an object/dict/map of keyword arguments and the decorator would translate them to an array for a wrapped Function.apply. That decorator or further decorators could support default values and automatic argument length assertions.

typeOf

Doug notes that JavaScript's typeof "function" is broken because it returns the wrong strings for null and arrays. All that and he hints that using camelCase instead of lowercase would have been more in keeping with the language's aesthetic. He recommends that we deprecate typeof and replace it with a fixed version called typeOf. I would go a step further. typeof returns a string. I propose that it ought to return the type object. This would jive well with instanceof, instanceOf, or isInstance. This means there should be a Null type that would construct null and an Undefined type to construct undefined.

Object Methods

Don't Enum

I don' particularly care whether Object has a dontEnum method since I don't augment base types and wrap them when I need reliable collection types, but this would help ameliorate integration woes with prototype so it's a good idea.

Array of Keys

Doug wants a Object.arrayOfKeys method. I want a keys method that returns a Set. I would also like keysIter and, less emphatically, keysArray. I don't want the name arrayOfKeys because it introduces an unnecessary Of and doesn't sort as well as keys*.

Array of Values.

Doug also wants Object.arrayOfValues. I want values and valuesIter. These should return Array and Iter object respectively.

toJsonString

Object.toJSONString(memberName). The first thing that comes to mind to improve this idea is to rename it toJsonString; I prefer to consider acronyms words and I would have also called XMLHttpRequest XmlHttpRequest. This name jives well with toString, but I would have called that string. But, I actually would call this repr and made it a global function. repr would work on arbitrary objects, a subset of which would be serializable JSON. The behavior of repr would be overridable by providing a repr method on an object.

Beget Object

Doug recommends that Prototype inheritance be a little more lucid by providing a beget function that returns a new, empty Object that inherits from the previous. I like this idea. I called it inherit in Chiron.

isEmpty

isEmpty would return whether an object has any properties, other than those inherited. Sounds fine to me, although I probably would have just used number and boolean casting. This is a good idea too though, because in JavaScript, it's hard to distinguish an Object that is being used as the base of the inheritance tree and an Object that is being used as a hash-map. To that end, I recommend the addition of a Map type. Map would have isEmpty, number, and boolean functions and would implicitly cast to Number and Boolean based on their length that would be reflected publically by the getLength function.

Array Methods

Doug recommends the addition of some of Mozilla's JavaScript 1.6 Array methods. I support all of them. However, some nomenclature refinements and specifications:

indexOf should be augmented by a find function that throws a KeyError instead of returning a negative index.

lastIndexOf should be augmented by a findLast function similar to find.

every should be called all. This function should accept an iterator or iterable and short circuit on the first failure.

filter should be called where. Filtering implies the opposite of finding. To filter something is to remove it from a stream if it passes a particular condition. where implies that the outgoing stream should contain only those elements from the original stream that affirm the guard.

forEach is good as is. I would like to specify that it should return this so that forEach calls can be chained. This has implications on iterations that are partially consumed by a forEach call that throws a StopIteration exception in its continuation.

map is good as is. As a member of an array, each should be a synonym. These functions should also be declared in global scope with opposite argument order: map(function, collection, [context]) vs each(collection, function, [context]).

some should be called any. This function should accept an iterator or iterable and short circuit on the first success.

String Methods

trim is good as is. There should also be trimBegin and trimEnd or trimFirst and trimLast.

string.eval is a good idea, but I've got my own ways to launder the scope chain for eval.

Date Methods

Date.toJSONString and Date.toISOString are both great ideas; not having them has been a problem for me in the past. I recommend calling them toJsonString and toIsoString.

Corrections

Reserved Words

Doug argues that identifiers in object literal notation and member dot notation should allow reserved words. I concur. It's a small syntax shortcut that identifiers used in member selection and object literal notation are not required to be enquoted; the language should be equally permissive for both syntax forms.

Doug also argues that the list of reserved words is too long. I have mixed feelings. On one hand, having a long list of reserved identifiers leaves open the door for future advances in the language (some of which include type annotations, which I find dubious). On the other hand, they muddy the name space for current code. On the latter note, it's important that all browsers are equally strict. Safari, at the moment, is much more strict than other browsers, so I've been surprised.

Object Literal Notation

I agree that commas should be more regular throughout the language. They should be permitted after any value, including the last in an Object or Array literal without affecting the length of either.

arguments

arguments should definitely be an instance of an Arguments type that inherits from Array.

Inner Functions and the Context Object (this)

I agree that this should not be the global object (window) in inner or anonymous functions. this should be acquired from the scope chain in such closures. Doug claims that this is not the standard. I recall being corrected on this point, but I also recall having been under the same impression. I leave this as an exercise to the reader to determine the current state of affairs in various browsers.

Tail Recursion

Tail recursion would be nice.

Deprecation

Primitive wrappers should be eliminated. Boxing is almost never necessary. In fact, the use of the new keyword could be completely obviated and code would become much more reusable since there would be no distinction between a factory method and an object constructor.

I actually use the with statement in a couple cases that are really important. I don't recommend using it for its designed purpose, but I really do need it to stay in the language for my module system and my JavaScript shell to keep working. FireBug, in particular, depends on this feature.

Semicolon insertion was silly. Off with its head.

I don't particularly care about arguments.callee.

typeof has to go.

Again, eval is a necessary part of my perverse world. I can do my own laundry. There are, however, irresponsible uses of eval that I do not condone.

De Schpugeti

You've heard of syntactic sugar. After a syntax and vocabulary provides all of the normal forms necessary to express all of the programs possible in a language, yummy shortcuts bridge the gap between expressiveness and usability. For example, c++ is a delicious variation of (temp = c, c = c + 1, temp).

You've also heard of CRUD, an acronym for the normal forms for a data programming interface (databases, data structures). If you design an interface library for data and fail to cover all of these bases, you deserve what's coming to you — and I can't promise that it won't involve a big hole in the ground and Hello Kitty.

CRUD stands for create, read, update, and delete. Those functions encompass all of the necessary data interactions; however, they are not sufficient. While CRUD may provide all the nutrients, vitamins, and minerals your data API might need, it certainly won't taste good without sugar.

Enter: Chiron.

del, set, cut, has, put, get. De Schpugeti. All of these names are three letters long, should be familiar by their names, and are sufficient in addition to necessary. These methods are the interface of Chiron's Dict, for manipulating item data: data that have keys and corresponding values. They are also implemented in a consistent manner by List presuming that indices are like keys.

del

Deletes an item for a key.
Returns the mutated collection.
For ordered collections, accepts a beginning key and an ending key, deleting all items in that range of keys excluding the ending key.
Implemented in terms of remove.

set

Adds or overwrites an item for a key.
Returns the mutated collection.
Implemented in terms of insert.

cut

Deletes an item for a key.
Throws a key error of no item has that key.
Returns the corresponding value.
Implemented in terms of get and del.

has

Returns whether key is among items.
Implemented in terms of has.

put

Adds an item for a key.
Throws a key error if there is already an item with that key. returns the mutated collection.
In an ordered collection, reserves the right to change the keys of some other items to make room for the new item.
Implemented in terms of set and has.

get

Returns a value for the given key.
Throws a key error if there is no value for the given key.
Implemented in terms of retrieve and has.

Also, Lists, Sets, and, by extension, Dicts have functions for managing collections of values that do not have keys. Dict is a Set of items instead of mere values that are compared and hashed on their key.

has

Returns whether a collection contains a particular value.

insert

Adds or overwrites a value.
Returns the mutated collection.
Defaults to an implementation in terms of find and set.

retrieve

Returns the contained value that is naturally equivalent to a given value.
Throws a value error if none exists.
Defaults to an implementation in terms of find and get.

remove

Removes a value.
Throws a value error if the value isn't in the collection.
Returns the mutated collection.
Defaults to an implementation in terms of find and del.

discard

Removes a value if it exists.
Returns the mutated collection.
Implemented in terms of has and remove.

find

Returns the key for a given value.
Accepts an optional equivalence relation to override the default of eq.

Now accepting ideas for a good mnemonic.

The fun trick is that List, Dict, and Set implement all of these functions. While it may be more appropriate to use one collection type for one kind of data or one algorithm, you can use any of these types interchangeably.

Integrating Simile

Not to name names, but I've been working on integrating code from Simile from MIT into Chiron. Refactoring an existing JavaScript project highlights all the things you get for free in Chiron.

Simile has it's own XHR engine, DOM event wrappers, DOM layout and style functions, PNG transparency solution, and a SortedArray type that provides binary search functions. Here are some of my observations.

• Simile's layout getSize was better than mine. I will rectify this.
• Not having a module system makes us reinvent the wheel: frequently.
• It's hard to write a good XHR engine. There are a lot of XHR modules out there, most of them have some issue or another: missing browser, doesn't report OK status on local files, doesn't unify browser caching inconsistencies, doesn't support timeouts, doesn't expose XML (the X in AJAX) in IE, or so on. If you're going to make a new one, you should use these as references and do some serious research, development, and testing. Otherwise, you should copy or use the best of them (jQuery, in my opinion). Also, it needs to support asynchronous (the A in AJAX) requests, and you need to use them as often as possible.
• Not having a solid, modular library makes us lazy. The inconvenience of name-spacing makes us lazy. This causes us to write sloppy code. For example, we should always use an enquote function when we're string interpolating HTML attributes and an inoculate function when we're interpolating HTML, or we should use DOM functions or a DOM wrapper API to generate our HTML.
• As I integrate code from other libraries, a pattern emerges. In my first pass, I collapse the name-spaces. Every module is a name-space, so all the manual creating of hierarchies like Simile.DOM (Simile = {} presumed, then Simile.DOM = {}, then endless repetition of Simile.DOM to augment or use its contents) is unnecessary and undesirable.

Referencing URL's of resources, like other scripts and images, relative to the URL of the script you're currently in, is hard. Starting from scratch, this usually means you're going to have a global URL constant. This means domain-coupling. Maybe you make the URL relative to the root. This means domain-coupling. Maybe you provide it as a configuration variable. This means site-coupling. Maybe you scrape the script tags on the page for the URL of your script then resolve the URL relative to your own URL. This means you're going to write a lot of slow code for what you perceive to be little value. In Chiron, you can get a function called resolve from http.js that resolves a URL relative to a base URL. Chiron also provides your modules with a moduleUrl variable that is the URL of the script you're in. resolve also implicitly uses this variable as your base URL if you don't provide a second argument(include('http.js'); resolve('images/blah.gif')).

Chiron grabs the script tag href of modules.js and removes the script object from the DOM (so other scripts can't sniff it) exactly once, since it needs that URL to resolve other module URL's. From there, Chiron keeps track of where all of your modules are relative to it and provides that information to each module.

• About SortedArray:
• A collection type should create an empty instance if you pass no arguments in.
• A collection type should populate itself from the values of another collection if you pass one in as its first argument. This should always be the first argument, even if you frequently create empty collections with overrides on later arguments. Force your user to pass in a null or undefined.
• Try to accept null and undefined as equivalent unless the distinction is meaningful.
• Try to distinguish null and undefined from 0 in all meaningful cases.
• Invariants like "sorted" are a promise. Guarantee your invariants across all function calls, including construction. If this means an unacceptable performance degradation, permit the user to suppress whatever code you need to verify the invariant if they are willing to provide treated data.
• If there is a reasonable default, it should always be implicit. I should not have to explicitly send the global compare function into a SortedArray if I want a SortedArray of types supported by compare.
• Not having a system of base types makes for noisy API's where names from different organizations have different meanings. For example, find functions should always accept the same kinds of arguments and return the same kinds of values. Simile's name choices are very close to mine, to the effect that they could almost be used as partially implemented duck types for mine, but some of the names would have to be realigned. find in Simile accepts a comparator and returns an acceptable index to insert or remove a particular element. find in Chiron returns an index or key at which an item can be inserted or removed, and guarantees that it will be the first occurrence of a given value (not a comparator). It was very easy to refactor SortedArray to subscribe to the strict model. Also, removeAll needed to be clear, length and getCount both needed to be getLength, getIterator needed to be iter, next needed to throw StopIteration once in a while, among others.

I'm looking forward to having a semblance of Simile Timeplot and Timechart in the Chiron family.

Polymorphic repr

I'm not about to debate whether debugging is a valuable exercise in JavaScript, nor whether introspection or reflection tools are useful, nor whether they would be especially helpful in a dynamic language. Ruby has inspect. Python has repr. The Chiron JavaScript library has repr too.

Notionally, repr is the inverse of eval for a reasonable subset of JavaScript. There are a lot of object hierarchies that cannot be reconstructed from a repr serialization, but as a debugging tool, repr is indispensable.

repr is a polymorphic function you can import from base.js. If you pass repr an object that implements a repr member function, repr will defer to your overridable repr. Otherwise, repr returns reasonable defaults for other types. For example, repr provides defaults for Array, Object, String, Number, Boolean, and Date. repr also recursively represents members of arrays and objects, but provides circular reference protection by tracking visited objects in a memo Set.

Chiron's debugger uses repr to convert the value of an expression on the command line to a human-readable string.

j$ 1
1
j$ repr(1)
"1"
j$ "hi"
"hi"
j$ repr("hi")
"\"hi\""
j$ true
true
j$ {a: 10}
{"a": 10}
j$ [1, 2, 3]
[1, 2, 3]
j$ [{a: 10}]
[{"a": 10}]
j$ var a = {}; a.a = a; a
{"a": <cycle>}
j$ type()()
<instance run.html#0 0>
j$ type({'repr': function () {return "x"}})()
x

modules.js

We've posted modulesjs.com this evening to proliferate what could be the most important piece of code I've ever written, modules.js. This JavaScript is the core of my Chiron JavaScript library project for the Tale game user interface, but could stand alone as the core of any number of JavaScript library projects. Every popular JavaScript library could benefit from a module system that handles dependencies and isolates name spaces like Python or Ruby. Read the tutorial and look over the code. Find me on an instant messenger or my face in the big blue box or send me an email and tell me what you think.

Thanks go out to Ryan Witt for managing the server and domain, my sister Kathleen Kowal for the graphic design, Ryan Paul, Ryan Ernst, and Mike Stone for proofreading and particularly to Ryan Paul for letting me expound at him nigh daily for the last decade.

Chiron Automated Tests

It's worth nothing that I've posted the current results for automated unit testing for Javascript library.

The automated test system uses a web browser, an HTTP daemon, a client web page, and a batch of javascript unit test files. A script launches the daemon with an argument list containing a manifest of all of the javascript unit test files. The daemon serves all of the Chiron files, a logger, a reporter, and a test file-name iterator. The script then opens a web browser and navigates to the client page. The client page supplants the usual test.js module with its own autoTest.js module (in an aspect oriented fashion) so that the unit test assertions and timers get logged to the server via an HTTP request. Then, the client page iteratively requests a test URL from the server and opens the test in a new window. After all of the unit tests have run, the client opens the test results page.

In the future, I would like to use up a database and leave the test daemon running before and after each Subversion commission so that test results can be gathered from a suite of platforms. Then, there would be a catalog of performance and coverage trends.

Cixar Javascript Library and Names

The name "Cixar Javascript Library", or CJL, has numerous virtues. It has a three letter acronym, a full name long enough to asphyxiate a cheetah, conjures absolutely no mental image, nor alludes to any works of art or literature.

I'd like to announce that, having rigorously contemplated the issue from a vantage about two feet above my naval (that is, half of a meter for our dear friends huddled safely distant from the sanity interference field), I've decided to name the Cixar Javascript Library. My requirements are succinct:

• one word,
• allegorical, preferably ambiguous,
• eminently brandable,
• subtly humorous,
• an alternate spelling in Unicode,
• If possible, the name must enamour an attractive girl no more than five years older or younger than me before Monday.

I browsed Wikipedia and discovered Ajax's centaur tutor, Chiron. So, there you have it, the library's name is Chiron. I leave it as an exercise for the reader and my fate/doom to discover whether it will suffice. However, I will explicitly state that brandability is fulfilled. Chiron, it turns out (back me up on this, Wikipedia), was really half-cow, the only Centauren to grace the Hellenic Pantheon.

For prospective users of Chiron, I have a question of nomenclature preference. Use of the Chiron library requires the HTML author to include exactly one script tag in their header, modules.js (13KB shrunk but not compressed). From there the author can opt to use whatever modules they will.

Central to the Chiron library is a module currently called javascript.js that contains exceptions (KeyError), the type system (type), common types (Dict, List, Iter), and a wealth of common functions (forEach, stripEnd) (currently 36KB shrunk but not compressed). Since I would like to give the user the option of not loading this module, most modules explicitly include it, even though in a language like Python, these primitives would be implicitly available.

I called the module javascript.js since it would include primitives that in Python would be part of the language. In Python, this module would probably be called __builtins__.py. In any other Javascript library, precedent suggests that that module would be called base.js.

Which of the following should I call this module of language primitives?

• javascript.js,
• builtins.js,
• base.js,
• your idea here.

Please send me an email and weigh in.

Javascript and Load Time

I conducted a quick experiment to assess whether it is worth worrying about the size and distribution of Javascript modules. I gathered the following numbers on my Linux box at home (fast) connected to Cixar (fast) over my home Internet connection (average today).

ping cixar.com	30ms
ping google.com	80ms
HTTP overhead	30ms
bandwidth	0.1KB/ms
patience	1500ms
maximum size	150KB
Chiron core	100KB
Chiron core shrunk	50KB
Chiron core shrunk compressed	13KB

I've decided not to worry about the size or distribution of Chiron until its core is five times bigger.

Higher Order Functions in Javascript

Recently I discovered that a friend of mine isn't using "Higher Order Functions" in their Javascript. I cried. However, I fear, this is common-place, perhaps some artifact of a latent and ubiquitous fear of closures instilled by Internet Explorer and the folklore that arises in the absence of reliable, or believable, information. In any case, I'm resolved to expound the virtues of this lesser known programming paradigm among object oriented language aficionados.

Higher Order Functions are functions that accept or return functions. This variety of insanity is possible in functional languages, or languages that provide closures. It is of course possible to accept and return function pointers in C or its ilk, but these are not sufficient for Higher Order Functions because you cannot create functions that can refer to the local variables of the function that instantiated them. This is the crux.

It is helpful to think of a name space as an Object. Calling a function creates a local name space, rooted in the name space in which the function is declared. Unlike in assembly languages, these name spaces can persist, as Objects do, long past the lifetime of the function call that created them. Like Objects, a name space exists as long as a reference exists referring to it. However, among name spaces, the only way to refer to a name space is to create a name space rooted therein. This means that the local variables of a function call exist as long as another function's name space refers to it.

Permit me to offer a practical example, one that any one can use every day of the week. Sort functions in most languages accept an optional comparator function as an argument. C++'s Boost library for example accepts predicates. This is, of course, an insane hack since C++ isn't designed for functional fu. Python provides sorted and accepts a comparator. You don't have to believe me; the important point is that Javascript follows suit.

This is the default sort behavior.

var items = [3, 1, 2];
items.sort();
assert(items == [1, 2, 3]);

A comparator is a function that accepts two arguments and returns an integer indicating whether the arguments are equivalent, sorted in ascending order or in descending order. The result of a comparison call can be compared to 0 (on the right hand side of an expression) with ==, > and < and the result will be exactly the same as if the operands were replaced with the comparison arguments respectively. So, if compare(a, b) == 0, then a == b, if compare(a, b) < 0 then a < b. The simplest compare function provides these results for numbers.

var compare = function (a, b) {
	return b - a;
}

This example, provides a reverse comparator; a comparator that will sort numbers in descending order.

var compare = function (a, b) {
	return -(b - a);
};
var items = [3, 1, 2];
items.sort(compare);
assert(items == [3, 2, 1]);

Of course, there are all manner of objects that you might want to sort, and not all of them are strings and numbers. Supposing you had an array of people, you might want to sort them by their last name. You can do this with a comparator.

var compare = function (a, b) {
	if (a.lastName b.lastName) {
		return -1;
	} else if (a.lastName == b.lastName) {
		return 0;
	} else {
		return 1;
	}
};
people.sort(compare);

Of course, it would be cumbersome to give highly descriptive names to every block of code you wrote and used once, so it's no more reasonable to have to name all of your comparators. You can just use an anonymous function.

people.sort(function (a, b) {
	if (a.lastName b.lastName) {
		return -1;
	} else if (a.lastName == b.lastName) {
		return 0;
	} else {
		return 1;
	}
});

However, you will find that all of your comparators will have the same shape. They will probably all test equality and lessity. It would be beneficial if you had a function that would create comparators for you just based on the attribute of the object you want to compare. Enter higher order functions. Let's construct one such function.

We want to create a function that accepts two elements and returns a number reflecting their transitive relationship, a comparator, based on some attribute. So, its clear that our function will accept an attribute name, and return a function.

/* the higher order function: */
var by = function (attribute) {
	/* the comparator: */
	return function (a, b) {
	};
};

Lets see the function in full.

/* the higher order function: */
var by = function (attribute) {
	/* the comparator: */
	return function (a, b) {
		if (a[attribute] b[attribute]) {
			return -1;
		} else if (a[attribute] == b[attribute]) {
			return 0;
		} else {
			return 1;
		}
	};
};

The important leap here is that the comparator can refer to the attribute from its enclosing name space, even though that function will have completed and been popped off the "stack" long before we ever call our comparator.

We can use our mighty by function to sort people by last name much more readably and succinctly.

people.sort(by('lastName'));

For good or ill, this will work for rigid arrays of arrays (tables) as well, if you provide an integer instead of an attribute name. I say this with half of a heart because it's a Pyrrhic victory. The price of being able to perform member selection and array indexing with the same, generic syntax is the inability to distinguish member selection for associative array (or hash) indexing, but that is a tale of woe for another day.

var table = [
	['Kris', 'Kowal', 3],
	['Ryan', 'Paul', 2],
	['Ryan', 'Witt', 1]
];
table.sort(by(1));

In any case, this demonstrates the power of higher order functions in your every day life. If you don't like the for (var i = 0; i length; i++) pattern (as you ought be weary of by now), modern Javascript implementations provide a higher order function, Array.forEach. From the pantheon of functions you will also find map, fold or reduce, filter, compose, partial, and others. Happy coding.

Javascript Events

This evening, I finally got around to implementing event base classes for the Cixar Javascript Library for Tale. These events resemble W3C DOM events but add some much needed complexity under the hood.

I've implemented Event, Dispatch, and Observable. Event is a base-class for events that implements the usual stopPropagation and cancelDefault. Dispatch creates a function object for dispatching an Event with optionally overridden event type or event factory and default function. Observable can be mixed into any type to permit users to listen to method invocations. This is the beginning of the fun. Dispatch implements call and listen as expected. Additionally, Dispatch implements listenBefore and listenAfter so you can attach event listeners that will fire before any other listener as yet attached, or after the default function and any other listeners have dispatched. Additionally, all of these listener functions return a new Dispatch object so you can assure that your listeners will fire in order relative to each other. This behavior is recursive, so you can attach a listener before or after any other listener.

Here's the unit test.

Don't Repeat Yourself: or Javascript DOM Initializers

I think I did something clever. Once again, it's time to fight the losing battle of explanation. I find that I've been repeating myself a lot in Javascript. Worse, I've had to compromise my ideals pretty frequently. Consider the following Javascript.

<html>
	<head>
		<script src="/javascript/modules.js">
	</head>
	<body>
		<div id="clock_0"></div>
		<script>
			require('clock.js', function (clock) {
				clock.Clock(
					document.getElementById('clock.js')
				);
			});
		</script>
	</body>
</html>

In this example, I've installed the Cixar Javascript module loader. This gives me a global function called require that fetches javascripts and loads them as singleton modules and returns the module object or calls a given function closure with the module object. I then populate my document with HTML and bless the HTML with Javascripts. In this case, I set up a clock. The details of how this clock works are hidden in a Clock type that is in turn hidden within a clock module. This particular setup requires that any blessed DOM element have an identifier so that it can be in turn fetched by name in the Javascript.

It turns out that this pattern of laying out out a DOM element like a div followed by a script to imbue it with DHTML powers is pretty common in my Ish project. This makes for a lot of script blocks but also makes the server side code rather orthogonal and elegant. It's not that bad, and I am proud it's this clean so far.

However, it could be much cleaner. Consider this example that I implemented today.

<html>
	<head>
		<script src="/javascript/modules.js"></script>
		<script>require('init.js')</script>
	</head>
	<body>
		<div require="clock.js#Clock"></div>
	</body>
</html>

Observe that with this approach, the div element does not have an identifier nor a corresponding script block. Instead, I've loaded an init.js module (the name will change when you email me a better one). The init.js module sets up a page load listener. When the page loads, the initializer scans the entire DOM for elements with require or requireRelative attributes. It then loads the given module, extracts the given function from the module, and blesses the corresponding element with that function.

I suspect this has a hidden advantage in addition to elegance. With the previous strategy, every DOM element had a corresponding getElementById call. I suspect that getElementById traverses the DOM until it finds an element with the given ID (plausibly, the browser tracks an (id, element) mapping dynamically, but this would be tricky to implement correctly). The old approach requires one of these traversals for every blessed DOM element. This new approach only requires one traversal of the document per page load.

So, two questions for the audience. What should this module be called instead of init.js? Alternately, should this behavior be automatically installed by modules.js?

Programming in Javascript

I'm going to attempt to explain computer programming. This will be interesting for my sister since she's nominally curious about the topic, and as I stumble through increasingly absurd (but commonly accepted) metaphors, I'll accidentally illustrate how odd the mind of a programmer must really be.

Programming is like writing a recipe book.

/* this is a recipe */
var ingredients;
mix(ingredients);
cook();

Running a program is like having a toddler make a gourmet meal by following the instructions in your recipe book (or toddlers in any number of kitchens capable of communicating with each other over the phone in ostensibly incomprehensible sequences of "gah" and "goo-goo" that mysteriously can also convey food). The recipe book is also a "make your own ending" novel. As the toddler reads instructions it will find "branches" like, "Making bread, step 567: to apologize to the butler, go to chapter 10. To bludgeon the butler with a monkey-wrench, go to chapter 829." However, "branches" or "gotos" are fairly uncommon.

/* this is not Javascript, but it looks a bit like it */
if (choice == 'apologies') {
	goto apologies;
} else {
	goto bludgeon;
}

Recipe writers use "functions" instead. A function is a chapter in the recipe book, a mini-recipe, wherein the author provides detailed instructions for doing things like "beat an egg". You can tell an infant to go to any of these chapters to do some particular task and then go back to the part of the recipe where they started, like "Making bread, step 568: knead dough. Go to chapter 12 with this dough and bring it back to step 569 when you're done."

var bread = function () {
	var eggs;
	var flour;
	var dough;
	dough = mix(eggs, flour);
	knead(dough);
	bake(dough);
	return dough;
}

To keep track of where they are in the recipe, the toddler keeps a stack of dishes, upon which, the toddler arranges peas and carrots with chapter, page, and step numbers from the recipe book.

I'm making bread
	I'm mixing eggs with flour
		I'm breaking an egg
			I'm picking bits of shell out of the yolk

This stack of dishes is often very tall.

With Javascript, this stack is usually about 10 plates. Apple's web browser, Safari, lets you stack 100 plates. Firefox's implementation of Javascript lets you juggle a stack of 1000 plates. Microsoft's Internet Explorer 6 handles 1125 plates. Opera gives you room for a whopping 3345 plates.

You might wonder why the toddler (in this case Hansel and Gretel) can't arrange the peas and carrots on the recipe book itself whenever it gets to a new chapter, letting it know where it needs to go back. There are some tricky chapters that actually, and often by way of long paths, refer back to themselves. These are called "recursive" functions and they have the special attribute that they must be "re-entrant", meaning that any number of toddlers can have any number of plates referring back to it without getting lost in their recipes or mixing up each other's ingredients. Recursion is handy because you can write chapters like "Find the spoons" that go like this: "If there's more than one place to look, divide those places in half. Find the spoons in north half. If you don't find them there, find the spoons in the south half. If you don't find them there, cry." Seriously, it's handy, and if you write the chapter well enough, it'll go a lot faster than, "Look in a new place. If you don't find the spoons, try again."

var find = function (places, spoons) {
	if (count(places) > 1) {
		try {
			find(beginHalf(places), spoons);
		} catch (tantrum) {
			find(endHalf(places), spoons);
		}
	} else if (has(places[0], spoons)) {
		return places[0];
	} else {
		throw new Tantrum("Wah!");
	}
};

The next nice thing about functions is that the recipes don't have to be particularly specific. This means you could write a chapter in your recipe book entitled, "Find Something". That way, the recipe writer can call for the toddler to "Find Spoons", "Find Monkey-wrench", or "Bludgeon Butler with Monkey-wrench". Writing recipes is about being vague. You never know when you're going to need to do something again and you don't want to write out every detail.

var bludgeon = function (something) {
};

So, you're keeping track of a lot of ingredients, so knowing what to call them can be a bit of a bear. You want to be sure that when you call for "the apple", you don't just get any apple; it has to be the relevant apple that the toddler just skinned. Most of the time, you keep track of names with the outline of your recipe. If you refer to an "apple", you read backwards first in the chapter, then the book the chapter is in, then the volume. You don't look in other chapters, books, or volumes because they might have their own apples.

var volume = function () {
	var apple = new Apple();
	var chapter = function () {
		var orange = new Orange();
		var step = function () {
			compare(apple, orange);
		};
	};
};

Many languages leave it at that. However some languages have "closures". It's a weird word for a simple idea. A closure is where you keep track of the plate you were on when you read the function. Sometimes reading a function is part of the recipe. When you do that, you're making it so that when you call that function, you can use any of the ingredients on your plate.

var makeACounterFunction = function () {
	var whereImAt = 0;
	return function () {
		var whereIWas = whereImAt;
		whereImAt = whereImAt + 1;
		return whereIWas;
	};
};
var counterFunction = makeACounterFunction();
assert(counterFunction() == 0);
assert(counterFunction() == 1);
assert(counterFunction() == 2);

Some languages are object oriented. There are a lot of interesting things about object oriented programming, but the short of it is that your ingredients can each have their own recipes and sub ingredients. You don't have to tell the function which ingredient to work on. You can just talk about it as "this".

var Egg = function () {
	this.break = function () {
		break(this);
	};
};
var egg = new Egg();
egg.break();

So, I'm going to gloss over a few details and descend into madness now.

Javascript Module Loader

I had an epiphany when writing a module loader. The trouble is that I didn't want modules to have access or ability to mess with the module loader's local variables. Since I was just calling eval with the text of the module, it had all of this access for reading and writing. Since I don't trust even myself, this was not acceptable. The problem was finding a way to sweep all the names under the carpet.

Here's the trivial solution:

(function () {
	/* module loader */
	this.require = function (url) {
		var require = function (url) {
			/* this require is for the module */
		};
		var text = http.requestText(url);
		eval(text);
	}
})();

Here's a less naive solution:

var evalModule = function (text, require) {
	eval(text);
};
(function () {
	this.require = function (url) {
		var text = http.requestText(url);
		var require = function (url) {};
		evalModule(text, require);
	};
})()

That solution still leaves evalModule as a variable that the module could potentially supplant. I needed to isolate evalModule. My solution uses a member variable of a jail object which eventually gets populated with the evalModule function without giving the closure access to either jail or itself..

(function () {
	var jail = {};
	this.require = function (url) {
		var text = http.requestText(url);
		var require = function (url) {};
		evalModule(text, require);
	};
	return jail;
})().evalModule = function (text, require) {
	eval(text);
}

Of course, it's not enough to isolate the module in a closure. I also give it a Module object as its context object.

(function () {
	var jail = {};
	this.require = function (url) {
		var text = http.requestText(url);
		var require = function (url) {};
		var module = new Module(url);
		evalModule.call(module, text, require);
	};
	return jail;
})().evalModule = function (text, require) {
	eval(text);
}

So, modules look like this:

var module = this;
assert(this != window);
with (require('module.js')) {
	/* go do stuff without fear of name collisions */
}

Javascript Functions, Accelerated Introduction

In Javascript, functions don't really have intrinsic names. The name is just a variable you've assigned an anonymous function object to.

function a () {}
/* is the same as */
var a = function () {};

You can tell a function that it should use some other object as "this".

var egg = new Egg();
var weasel = new Sweater();
egg.break.call(weasel);

You can also send any number of arguments to the function, even if it doesn't ask for them. Then, inside the function, you can get at all the extra arguments with the arguments variable.

(function (a, b) {
	assert(a == 10);
	assert(b == 20);
	assert(arguments[1] == 20);
	assert(arguments[2] == 30);
}).call(this, 10, 20, 30);

You can send arguments to a function from an Array.

(function (a, b, c) {
	assert(a == 1);
	assert(b == 2);
	assert(c == 3);
}).apply(this, [1, 2, 3]);

You can even make up an array of arguments with a program.

var array = [1];
array.push(2);
assert(array == [1, 2]);
var sum = function (a, b) {
	return a + b;
}
assert(sum.apply(this, array) == 3);

You don't have to give a function all of the arguments it asks for.

(function (a, b, c) {
	assert(a == 10);
	assert(b == 20);
	assert(c == undefined);
})(10, 20);

You can run programs/recipes that you get as quoted data.

eval("var a = 10; a = a + 1");

When you eval, you can see all the data in the caller's closure.

var a = 10;
eval('a = 20');
assert(a == 20);

An interesting thing about putting Javascript in HTML is that all of the scripts have the same closure.

<script>
	var global = true;
<script>
<script>
	assert(global);
<script>

You can wrap a bit of code in a closure so that its names are all its own, can't conflict with other scripts.

var global = true;
(function () {
	var local = true;
})();
/* local is not visible here */

In a web browser, the context object for the entire program is the Window object containing the script.

assert(this == window);
(function () {
	assert(this != window);
	assert(this == 10);
}).call(10);

The window is also the base closure for your program, so you can access everything in it as if it were a local variable.

window.document == document

You can play this trick with any object's member data or functions using the with block.

with ({'a': 10}) {
	assert(a == 10);
}

Javascript Modules Revisited

For those of us who retch at the mention of both "Java" and "Javascript", today's post brings us deeper into my realm of insanity, "deep in the shit", to quote Colbert. Yes, it's no picnic, but the development world has its human moments. Moments where hubris, playful cynicism, and violent indignation must give way to humility. People, harken to my words, hidden among the copious piles of refuse, there's some good code in Dojo.

Don't run off and commit resources to it; I'm just saying it's a good read for someone looking for snippets to steal. If you need something that works today, it might be a good option, but hold your horses; I haven't finished reading the alternatives.

I've mentioned that Dojo has a module system. It plainly doesn't meet our needs. Let's be explicit about our requirements

Module system must:

• be the first and only Javascript file explicitly loaded with a script tag in the header of a document
• have a light footprint on the global context object (window in browsers), ideally one function
• isolate each module in its own closure
• isolate each module with its own context object, preferably the module object
• leave no conditional module management code to users
• provide singleton module objects that are loaded once per page
• provide modules as shallow objects, encouraging the library user to obey the "Law of Demeter"
• handle synchronous and asynchronous pattern interface
• permit the library user to require a module under any pseudonym it desires, or vaguely, provide the flexibility of Python's import syntax, with suitable analogs for import module, import module as m, from module import *, from module import part as p.
• permit the library user to move an entire directory of scripts without breaking any of its references to other modules within the same directory tree nor break references to global modules

Dojo does not appear to follow all of these edicts. For example, all modules use the global object, window, as their context object. Every top level module, dojo for example, leaves a footprint on the global object. Dojo encourages Demeter violations like, dojo.component.subcomponent.WhatYouReallyWant. The list of annoyances goes on.

However, Dojo does load modules synchronously, ironically with the so called "Asynchronous XML HTTP Request". My solution in Cixar Javascript, modules load asynchronously using DOM or write tricks. Modules register themselves when they're text loads. Register isolates the context object and closure.

Neither my solution nor Dojo's solution address the problem of mobile libraries. This is a critical feature for a real module system, since it would permit authors across the web to isolate a tool in a directory and permit users to place and name that directory anywhere and any way they want without having to modify the contained code.

The problem with my system was that it requires modules have to register their location:

this.register('module.js', function (module) {

This was necessary because of the non-blocking, asynchronous module loader. There was no way for a Javascript to recognize what module it was supposed to be since the module scripts would load in an indeterminate order.

Enter Dojo's module loader code. This bit of AJAX is pretty good, with lots and lots of embedded knowledge of multiple Javascript environments and their eccentricities. Using it as a reference, I wrote a new module loader system that provides both synchronous and asynchronous forms for require, and obviates the need for register calls completely.

A module:

/* import module */
var module = require('module.js');

/* import module, asynchronously */
require('module.js', function (module) {
});

/* import module as m */
var m = require('module.js');

/* from module import * */
with (require('module.js')) {
}

/* from module import component */
var component = require('module.js').component;

/* from module import component as c */
var c = require("module.js').component;

Most of all, the register function no longer exists. Modules have their own closure and singleton context object. Modules have their own require function so (eventually) they will be able to load modules relative to their own URL rather than the URL of the containing page. Here's what it looks like:

(function (url) {
var require = function (subUrl) {...};
eval(http.requestText(url));
}).call(modules[url], url);

Meanwhile, I also re-imagined Dojo's HTTP request mechanism, the heart of AJAX. Since the module system requires HTTP requests, it creates a pseudo-module that users can later require as http.js. The HTTP module supports the same abbreviated syntax for both synchronous and asynchronous requests as require, and provides forms for retrieving the HTTP response, text, XML, and the XML document element object. The module includes most of Dojo's code paths for multiple browsers, environments, and platforms and provides a wrapper for the HTTP response object that cleaves more strictly to an observable style and handles some odd cross browser cases.

var text = http.requestText('text.txt');

http.requestXML('xml.xml', function (xml) {
});

var response = http.request('location.html');
if (response.isOk()) {
var text = response.getText();
}

Javascript Libraries

I've been poring over Javascript libraries lately, trying to ascertain the extent of their features, how much implementation overlap there is among them, and which ones implement certain features best. The idea is that I'm going to integrate the best tidbits into Cixar Javascript. I'm reading:

• Yahoo's YUI,
• Dojo,
• Prototype,
• Mootools,
• Mochikit, and
• Scriptaculous, plus outliers and extensions like
• Flapjax,
• Mooshow,
• iBox, and
• Lightbox.

Please contact me if I need to add a library to my reading list.

I've got some general impressions so far. Dojo and Mochikit are by far the heaviest. They're loaded with lots and lots of well tested unreadable undocumented components. Dojo supports its own weight because it's the only Javascript library that has a module loader, albeit a module loader that loads modules into the global name-space. YUI, Mootools, and Scriptaculous are comparatively higher quality but smaller and limited to their size by the lack of a good module system. YUI is the best documented, having lots of Javadoc style comments. Mootools, I think proves to be the highest quality code for the few things it does and does rather well. Scriptaculous and its base library Prototype, spun off the Ruby on Rails project, both seem to work pretty well, but the code is criminally bad, showing lots of signs of having been designed by whim. I get a very strong "the least that would work" vibe from it.

Dojo's module system intrigued me because it loads modules synchronously with AJAX and Javascript's eval. I've considered applying the same ideas for Cixar Javascript's module system. However, the current Cixar Javascript module system puts the power of nomenclature into the hands of the module writer, not the hands of Kurremkarmerruk, the Wizard who knows and manages all (or most) of the names in the universe. I think that the "CJS" syntax is verbose because of this feature, not because it loads modules asynchronously. Since loading modules asynchronously may have higher performance in the long run, I don't see much reason to adopt Dojo's approach. However, Dojo does include a lot of good snippets that seem to embody a lot of working knowledge on how to make Javascript work in many, many environments, so I do intend to study it throughly.

Let me back these statements up with some code. This is the Dojo syntax for defining, loading, and using a module:

dojo.provide('dojo.provided_module');
dojo.require('dojo.required_module');
dojo.provided_module.provision = dojo.required_module.provision;

This is Cixar Javascript's:

this.register('cixar/provided_module.js', function () {
this.require('cixar/required_module.js', function (that) {
this.provision = that.provision;
});
});

I particularly like using Javascript's with block to effectively bring all of a module into my name-space so that I don't have to qualify any of its names.

this.register('my.js', function () {
this.require('javascript.js', function (js) { with (js) {
assertEq(js.List, List);
}});
});

I'm getting comfortable with the parentheses and braces because I feel that localizing names is really important for the long term maintainability and extensibility of the world of Javascript.

From this preliminary analysis, I think I will be integrating most of Mootools code since it's small and high quality. I will probably also integrate parts of Dojo and Mochikit since their feature coverage is quite extensive, but porting whole modules is probably out of the question. In any case, I will try to bring a comprehensive set of features into CJS so that people coming from any of these frameworks will find some degree of familiarity, if not because of their experience with their respective Javascript libraries, then because of their familiarity with the server side scripting language on which that library was based.