Narcissus addon available

Today I uploaded a plugin for Narcissus JavaScript.  Details can be found here.

It requires Firefox 4, so to test this out, you will need to build from source for the time being.  Longer term, the plan is to swap out the JS engine at will.  But, it is at least usable for experimenting currently.  Enjoy!

More Awesomeness: jstests.py

In working with Narcissus, I've uncovered a jewel of a resource in Mozilla's JavaScript unit tests.  The value of unit tests is well understood.  The most critical quality, however, is that they are maintained.  Mozilla has done an exemplary job of this.  The unit tests are constantly growing.  As I write this, there are just shy of 3000 unit tests.  All of them pass with the Tracemonkey engine.

The organization of the tests is well thought out.  Each version of JavaScript has its own test directory.  So if you don't care about certain batches of tests, it is easy to ignore them.

Unfortunately, as with many other things at Mozilla, there is an abundance of out of date documentaion.  The JS Test Library page refers to a Perl script, which appears to be more or less abandoned.  Instead, there is a python script 'jstests.py', written by Dave Mandelin in his copious amounts of free time.  Unfortunately, the old version of unit testing documentation comes up first when searching for it on Google.  Alas.

One thing that is sadly not emphasized is just how easy it is to plug in a different engine to use these tests.  In fact, I have a "shell" for Narcissus JS, written in 30-some lines of Python code.

To hook into the testing framework, you need a "shell" that supports two command line options:
1) -e, which interprets a string of JavaScript code.
2) -f, which handles a JavaScript file.
It must be possible to support multiple files and strings of code, though this is not a particularly difficult challenge.

Some tricks:

• The -m option will let you specify your own module of JS tests.  This is also a convenient way to insert a little JavaScript into the testing process to handle tests relying on non-standard JS features.
• The -x option allows you to ignore a list of tests.  Since Mozilla includes tests for special Firefox-only features, you will want to use this.
• The --failure-file option will write the failing unit tests to the specified file, useful as input for the -x option.

On another note, Narcissus now runs without special extensions in Firefox 4.  Huzzah!  Now to make it interact smoothly with the DOM...

This summer I am working at Mozilla. It is an awesome, awesome place to be. The people here are friendly and amazingly bright. Plus there is a dinosaur in the lobby. How can you beat that?

My work has been looking at Narcissus JavaScript. Narcissus is a JavaScript implementation written in JavaScript, and as such, it is an ideal vehicle for playing around with the language.

Narcissus relies on a few special extensions, which require a special build. It is not hard to build, but it is not all that well documented either. Here is the process that I use (in OSX):

  $hg clone http://hg.mozilla.org/tracemonkey/
  $cd tracemonkey/js/src/
  $autoconf213
  $./configure --enable-narcissus
  $make

With that, you are good to go. From the command line, ./js will give you a tracemonkey shell. Here, you can use evaluate(prog) to test out narcissus.

With some adjustments, Narcissus should run within a standard Firefox build. I am down to a handful of failing unit tests. After that, running Narcissus within a browser should be as simple as loading the Narcissus js code.

NetBeans -- The Least Terrible IDE

My favorite text editor is MacVim. It lays a nice Aqua interface over the top of the classic editor. It is not perfect. I could rant for hours on its shortcomings, but it is still a shorter rant than I have for Aquamacs and TextMate, and these are the only two other text editors that I find even usable.

But text editors and IDEs are not the same thing, despite the fact that they do many of the same tasks. When I start working with a Java or Scala project, I want the advanced functionality that a full-featured IDE offers. (And yes, I am aware of Eclim, but that does not really change the fact that an IDE has been brought into the process.)

I have played around with a number of IDEs over the years; I have yet to settle on one. Eclipse is the classic, with extensions and plugins for about everything. Now that IntelliJ has an open-source version, I have been spending a good deal of time with that. Nonetheless, NetBeans is still, in my opinion, the best.

It is not that it is the prettiest. In fact, it is by far the ugliest of the three on any other system besides OSX, and still not that pretty even then.

It is not that it has the best Scala plugin. While I have had more success with the NetBeans plugin than the Eclipse one, I have found IntelliJ's to be more full-featured (albeit a little more of a hassle to install, since the version offered through their plugin manager does not seem to work. But that is another rant...).

And while I love some of the plugins available (the jVi plugin in particular being the best vi emulator that I have seen for any IDE or editor), they still pale compared to the wealth offered by Eclipse.

No, these are pluses, but relatively minor points. The reason that I keep returning to NetBeans, in a word, is Ant.

There is a fantastic article by Jeff Atwood titled "F5 is not a build process". This comes close to the core of the issue: every project should have a build process that functions outside of the IDE. This is a problem I have seen even with some open-source projects. Instructions like "start up NetBeans" or "download the Eclipse plugin" should never appear for any tool that is not specific to an IDE.

Unlike Eclipse and IntelliJ, NetBeans does not have its own special build tools. It defers this to Ant (or yes, you can use Maven instead). As a result, the build process is available outside of NetBeans. In my experience, projects designed with the other IDEs only create a build process as an afterthought.

This is huge. It means that the IDE actually takes care of the build process for you, as is just and proper. It also, however, allows you some extra options when things go wonky.

I do some weird things with my IDE. Using Scala is one of the more tame uses, but even that can start causing problems when you combine it with Antlr or JavaCC. And God help you if you want to add unit tests into this mix. I have managed to confuse and break every IDE I have used, to one degree or another. (To be fair, IntelliJ has generally handled more weird stuff than the others).

The beautiful thing about NetBeans is that when it stumbles, it can still function. For one project, NetBeans gave me hundreds of spurious errors, but the build process still worked.

NetBean's generated ant files include a number of hooks where you can add in your own ant tasks, a feature that I have found invaluable. Furthermore, if something in NetBean's ant files is horribly broken, you can replace it with your own; not something that I would recommend, but it can save you in extreme cases.

And so, despite NetBeans numerous quirks and limitations, I still list it as my favorite IDE. And while I believe the NetBeans team will address the product's current limitations sooner or later, I doubt that either the IntelliJ or Eclipse teams will ever fix what I see as a fundamentally broken architectural decision.

What main method?

I've been looking for an excuse to learn more Scala for a while. Currently, I am working on implementing the TAPL interpreters in Scala.

I ran into a problem that took me a while to figure out. To illustrate, I'll give a simple example. Consider this main method:

def main(args: Array[String]) = {
var sum = 0
for (s <- args) { sum += s.toInt }
if (args.length > 0) {sum/args.length} else 0
}

This simply averages a list of numbers specified from the command line. But when I try to run it, I get this error:

Exception in thread "main" java.lang.NoSuchMethodError: main
Java Result: 1

With a very slight change, this will work:

def main(args: Array[String]) = {
var sum = 0
for (s <- args) { sum += s.toInt }
println(if (args.length > 0) {sum/args.length} else 0)
}

This just prints out the average, no complaints.

So what is the difference? Why does printing out the result make a difference? The answer requires a little understanding of the way that Scala works.

Scala relies on type inference. This means that it tries to determine the types of your variables (including methods) from analyzing your program. This is not the same as dynamically typed languages like Perl or Ruby -- the type checking is still done at compile time. The compiler is smart enough to infer the types of your variables from the context.

But there is a problem here. It also has implicit returns. The program will automatically return the last executed expression in the program. This has been a popular feature in many programming languages, including the Lisps, Ruby, and the various ML dialects (of which Scala itself is something of a shirttail cousin). Personally I despise this feature, but I am in the minority.

Combining these features with Java programmer expectations leaves a nice little trap. The main method must not return anything. (Or if you rather, its return type must be Unit). If we copy the second version into the interpreter, we will see this:


main: (Array[String])Unit

The second version works because it prints out the results -- println has a return type of Unit. However, the if statement in this example has a return type of Int. The first version of the program, although it looks nearly identical, has a different type:

main: (Array[String])Int

As a result, Scala looks for main: (Array[String])Unit but cannot find it and returns an error message. This is easy to fix:

def main(args: Array[String]): Unit = { ... }

Or better yet, we can just remove the equals sign from the original:


def main(args: Array[String]) { ... }

Interestingly, the original version actually catches a problem in my code; I am calculating an average, but then I do nothing with it. More often than not, if you see this error, you are probably doing something wrong -- why calculate a value for no reason?

Still, when you are learning a language, a stumbling block like this can be confusing. The error message is clear enough to anyone who understands Scala. My only suggestion would be to add a special case for main specifically. It is a slight hair in the implementation, but it can save clueless newbies (like me) a little frustration, and really... what other kind of method would be named "main"?

Building Hyype.net with Helma Object Publisher

Today, Hyype.net was launched. Hyype.net is a micro-blogging sports site. It is designed to encourage loud, boisterous, trash-talking about each others favorite teams. We have some ideas of where we'd like to expand to, but the first part is completed and live.

This blog entry is not about Hyype.net though. This is about building Hyype.net, and more specifically, about my experiences with Helma Object Publisher. Helma is a web-development framework using Mozilla's Rhino JavaScript as the server-side language.

Disclaimer: I am not an expert at using Helma. There may be better ways to organize things. Hopefully, some of the Helma guys will step in and point out where I did things the stupid way. This entry is about a newbie's experience learning the framework and building something useful with it.

Getting Started

After having struggled with installing a number of different frameworks, Helma was absolute bliss to get up and running. Inside of a minute I was able to download it, configure it, and get it running. At that time, I was presented with this:


It was a powerful feel-good moment. Not only was it easy to get up and running, but I was presented with something that was actually useful instead of some stupid splash page. This page also has links to a tutorial and documentation.

The tutorial was a great place to start, and after an hour of going through that, I felt like I knew what to do.

HOPped up on objects

At about this point, my romance with Helma began to sour. What exactly is a HopObject? Why is there 'Root' and 'Global'? Why the @#$@! do my macros work here, but not there?

The organization of a Helma project at first seems bizarre. Combined with some subtle differences in how functions work (Why doesn't renderSkin() work like this.renderSkin()?), I slowly developed the feeling that Helma was well-polished on the surface, but buggy and horribly organized. I compared it to PHP, which is about the dirtiest insult you can hurl at a Java-land framework.

And then, all of a sudden, I got it. The pieces clicked together, the organization made sense, and I was left reeling. I had to lie down for a minute until the world stopped spinning.

Here is the key thing you need to understand about Helma: it is an object-oriented framework.

That might not sound like a big point. After all, most frameworks today are built with object-oriented languages and overrun with objects. But that is not quite the same thing. Take Rails, for instance. It has a clean organization with model objects, controller objects, and views (which are probably objects too—I don't really know).

In contrast, each Helma application is one massive, amorphous object. It is like the difference between classes and prototypes. This is the type of organization that makes complete sense if you are thinking in a JavaScripty way.

One subtle distinction about JavaScript programs is that each one runs in the global object. It has no name, we never really see it, but it is there. (This was a crucial point when I was building JOMP—top-level functions are methods of the global object and need no special handling). Helma has just taken this idea and build a web framework on top of it. My master's thesis spends a great deal of time extolling the benefits of JavaScript's object system, so I was a little embarrassed that it took me so long to grock Helma's organization.

Each folder in Helma corresponds to some type of HopObject. These folders will hold the database configuration (if needed), the actions (more on those later), the methods, and the skins (html templates).

Global is the global object where all JavaScript objects live. Before I understood it properly, I had a fair amount of the skins stored there. Now, I have only a single macro. I'm not really sure I even need that. This is probably good form — global===bad, as Douglas Crockford would say.

Root is the single massive object that is the web application. This was the single most crowded directory for Hyype.net. Anything that did not clearly belong to a single object ended up here.

There is also HopObject, which serves as the parent object of all other Helma objects, including Root itself. I probably should have used this more, but this directory is empty for Hyype.net.

Every other directory corresponds to some object specific to your application. As it turns out, Hyype.net really only needed 3: Users, Profiles, and Thread.

For the rest of this article, remember that everything is owned by some object. A skin or an action belongs to some object—HopObjects are not passive lumps of data like they are in other webdev frameworks.

Persistence

The core use of objects in almost any web framework is to serve as containers for data. This is easy to do in Helma. It does take some configuration, but that is pretty minimal. Here is the User configuration for Hyype.net:

_db         = myDataSource
_table      = user
_id         = id

username    = username
password    = password
email       = email
verified    = verified
createdAt   = created_at
updatedAt   = updated_at

profile = object (Profile)
profile.foreign = user_id
profile.local = id

This is a little more work than it would be in Rails, but it is not too bad. It would be nice if Helma could generate a base config, but it would really only save a minute of work. Better yet would be if it could default to the same names as the DB fields. The associations are also a little trickier than in Rails, but not bad. The documentation is fantastic, however, so I was able to figure things out without too much trouble.

One note: In good Helma form, the database field names should have been in all caps. I ignored that, but it would have made the configuration a little clearer. With all lowercase, it is not as obvious which side is which.

HACking something together

Every action corresponds to a url. While there are different ways to do it, the standard approach (and the one I used) is to create a separate .hac file for each action. Here is a simple one:

res.data.title = "About Hyype.net";
res.data.selected = "about";

res.data.body = this.renderSkinAsString("about");

this.renderSkin("layout");

"res" stands for "response". The data property stores the variables that your template will need. The renderSkin methods will write out the templates.

I would have liked Rails' style layouts. If there is a built-in setup for this in Helma, I could not find it. Rolling my own was easy enough though. I did this by creating a skin cleverly named "layout". Within it, there is a body variable. Using "renderSkinAsString" I could fill out this spot and then render the layout itself.

Skins

My last project involved JavaServer Faces. Before that, it was a PHP project. They represent the two extremes. JSF is rigid and unfriendly (and, perhaps worst of all, not html). PHP lets you do whatever you like, even things you should never, never do.

Helma's skins take a nice middle ground. They don't allow arbitrary code, but they do allow you to call macros (special JavaScript functions). The result is that no code bleeds into the html. Some html did bleed into the code, however, but I managed to keep this relatively minimal.

One really cool thing about Helma's organization is that objects can render themselves. They can contain their own skins. For example, Thread has templates for being rendered as clips (displayed on the left rail), details (for a more complete view of the thread), and myClip (for the "My Hyype" page). Within the main page, each thread is rendered with 'str += hyype.renderSkinAsString(clipSkin);' The current thread is displayed on the center of the page with 'res.data.body = selectedThread.renderSkinAsString("details");'

It seems like an unusual way of organizing templates, but I found it very effective. Unlike objects in other frameworks, HopObjects know how to dress themselves.

Here is the template for Hyype.net's threads in clip view.

  <div class="bb-box"><div class="ctl"><div class="ctr"></div></div><div class="pad">
   <a href="<% rootPath %><% this.posterProfileHref %>"><img src="<% this.smallPic %>" alt="<% this.poster %>" title="<% this.poster %>" class="bb-img" /></a>
   Posted by<br />
   <a href="<% rootPath %><% this.posterProfileHref %>" class="lnk-green"><% this.poster %></a>
   in <a href="<% rootPath %>main?cat=<% this.categoryName %>" class="lnk-green"><% this.categoryName %></a><br />
   on <% this.postedOn %>
   <div class="comm"><a href="<% rootPath %>main?threadID=<% this.id %>&pageNum=<% request.pageNum %>" class="clip-title"><% this.title %></a></div>
   <% this.agreeDisagree %>
  </div><div class="cbl"><div class="cbr"></div></div></div>

Utilities

Helma offers a rich set of utilities as part of its core and a number of optional extensions. And if that is not enough for you, the Jala Project offers a bunch more modules. For the most part, these are well-documented and easy to use. The one time I had a problem, a note to the Helma mailing list quickly answered my question.

One problem spot I did run into was trying to include Java code in my project. For the most part, this is not needed. Rhino allows you to call Java. However, I wanted to get the MD5 hash of a string for comparing passwords. I could not call the methods I needed in Rhino, since it was ambiguous to Rhino which method I was calling.

I wrote the method I needed in Java, but there was no good place to store the class file. Instead, I put the class file in a jar and added it to the lib/ext/ directory. This was not the cleanest process in the world, especially since it is shared by all projects, but it did the job.

Deployment

Once we had the application ready to deploy, life got a touch more challenging. We went off the beaten path a bit by not using Antville. Things went more or less smoothly until we tested out uploading images.

The server crashed. We reconfigured a few things. It crashed again. After several hours of research and troubleshooting, we nailed down the problem to our version of Java. Specifically, it was using GNU's implementation, which ignored some of the flags we were passing it. I still have not totally forgiven GNU.

While I think Helma is blameless for this, it is one risk of using a framework that is not as widely used as PHP, JSF, etc. Fortunately, the community is friendly and active, which usually more than makes up for this.

Conclusion

Helma has gotten to be my favorite webdev framework. Given any choice in the matter, this is what I'd use for a new project. Helma seems to be in active development, and I'll be excited to see where it goes from here. If you have a new project coming up, give Helma a look.

Yes, We're Open

90% of the time when I need some work done on my car, I take it to Sears' Automotive Center. This is despite the fact that they are usually unable to fix anything more than the basics.

The main reason I keep going there is that it is incredibly convenient. When I have tire problems driving home on a Friday night, they are open. If I want to drop my car off on a Saturday and then walk to the movie theater in the same shopping mall, I can do it.

In contrast, the Ford dealership has generally done a better job of fixing more complicated problems. But they are only open 9-5, and they are in the middle of nowhere. I can only read outdated snowboarding magazines for so long. Also, they charge twice as much, so I take my car there only in desperation.

Convenience Matters

This should not come as a surprise. Unfortunately, the state of many software projects out there would suggest otherwise. Far too many of them have terrible installation instructions and horrid documentation.

If you want people to use your tools, put yourself in the position of your new users. What are the pain points? Where is this confusing? You won't catch all possible problems, but the more you do catch, the more users you will be able to win over.

I've been working on XMUltra lately. My initial efforts have largely been focused on just this part -- how to make it easy for a new user to get started. I have a ways to go yet, but I think the initial public launch had much better tips for getting started than the internal documentation.

Eclipse vs. Emacs

These two have very different strengths. Emacs is the more powerful of the too, or so the Emacs elite claim. (Not being an Emacs power-user myself, I have to take their word for it).

Eclipse is decidedly easier to get started with. For one, it uses the standard key bindings, so that ctrl-x, ctrl-c, ctrl-v, ctrl-s, etc. all work as you would expect. In addition, it is decidedly prettier. It has nice file open dialogs and more or less intuitive preferences settings.

Actually, Aquamacs comes pretty close here. It takes advantage of the fact that Macs have both a command and a ctrl key, so you can still use command-x, command-v, etc like you would in any other program, but use ctrl for emacs key sequences. It also has nice GUI interfaces overlaying the Gnu Emacs versions, so you can get decent looking file open dialogs (in addition to the regular Gnu emacs versions).

But Emacs and all of its touted power fails miserably in actually being able to transfer that power. Yes, I'm sure it is much easier to write an Emacs plugin than to write an Eclipse plugin, but that is only half of the story. After I write one, someone else has to install it, or nobody but me benefits from it.

In Eclipse, I have to type in a url into a package manager.

For Emacs, let's take a look at a couple of examples of installation instructions:

• JDEE includes steps like "Download the latest versions of Eric Ludlam's speedbar, eieio, and semantic bovinator packages and install them on your system, each in their own directory." Great.
• HtmlModeDeluxe: "First, you must install all four modes needed. The php-mode I tested so far and which seem to work well are Turadg Aleahmad's 1.0.2 and 1.0.4. As I had several reports during the last few weeks that the current mmm-mode cvs does NOT work, I now recomment to first try..."
• Emacs WebDev: "You will need the following packages: *psgml *mmm *generic-x *php-mode *css-mode *xxml *tidy"

Hmm... So do I have semantic bovinator installed on my system already? How do I tell? What version do I have? And, most importantly...

WHEN THE @#$%! DID I BECOME A FLIPPIN PACKAGE MANAGER!!!

This is the single biggest failure point of Emacs. There are some attempting to fix this, but one really, really, really needs to be made part of the Gnu Emacs code base.

A Tale of Two Frameworks

I'm a big fan of Rhino JavaScript. Enough so that I built a JSF-based web framework for it. I spent some time looking around at what other frameworks existed.

One of the first I found was Phobos. They have nice webcasts, and it looks pretty cool. But... For one, Java.net is painfully slow to load. Some might be turned off before they ever see the homepage. But if you have some patience, you might make is as far as the download page and see this in the installation instructions: "You need to have the NetBeans IDE installed."

Wow. So in one fell swoop, the Eclipse users have already been turned away. But maybe this is just for tool support. Perhaps this will work just fine as long as Glassfish is installed. And what does the website say? <cue the crickets>

In contrast, let's take a look at Helma. Here we have a nice, prominent download link. After downloading it and following the instructions, I had Helma running with a useful admin tool inside of a minute. It even has a default file-system persistence setup in case you don't have a database installed. The first experience is lots of happy juju.

So take a wild guess which framework I am using now.

Helma takes some getting used to, but it is worth the effort. The organization is very cool... but very different. While I did some head scratching before I figured it out, I made the effort because Helma had already won me over. I wanted to like Helma, and so I spent the time to find its cool features. Also, the documentation pointed out those features for me. (And seriously... If you are a Rhino fan, check out Helma -- I actually prefer it over Ruby on Rails.)

As Simple As Possible, But Not Simpler

One final note... Sometimes you cannot make things as simple as you would like. If you are building a complicated product, it may be that a complex installation is required. But challenge every pain point. Would a Rails-style sensible default be a wise move? The answer is usually yes.

Back to my mechanic... Before Sears, I went to a private mechanic. The work was fantastic, the prices were fair, and I went there despite the hassle of driving to and waiting in the middle of nowhere.

Unfortunately, he went out of business.