Postbox: How not to do customer support

I am generally suspicious of large organizations, and I am often in areas with poor wifi access, so I have never totally bought in to the idea of relying on Gmail to keep track of my mail for me.  While I do use their product, I prefer to use a Desktop client to back up my messages and give me a little more flexibility in how I use my email.

Unfortunately I am in the minority, and desktop mail applications are dying off rapidly.  Thunderbird has largely been abandoned.  The developers of Sparrow have stopped working on it after Google acquired the company.  Postbox is one of the few options left, and for the most part I like the product.  It works cleanly with Gmail and has a pleasant interface, as my tastes go.  There are a few bugs, as there are with all products, though nothing out of the ordinary.

However, they have cut themselves off from their customers as completely as they can.  If you look at http://support.postbox-inc.com/entries/21694767-How-to-Get-Help-with-Postbox, every option there is designed to avoid the burden of dealing with their customers.  There is no way to report issues or even to post on a forum.  A year-old message reports "we're currently evaluating a number of different Forum solutions that will enable Postbox users to collaborate and share ideas."

Dealing with customers can be annoying; ignoring them is fatal.

Nothing is more irritating to users than having a problem and not even being able to report it.  The company has no way of learning about which issues matter to their customers, or even learning about these issues in the first place unless they themselves have the same problems.  It is a damning flaw in a fine product, and it is likely to cripple Postbox going forward.  I hope that they address it in the near future, though I am not hopeful.

Tales of DRM

The struggle to find a balance between the rights of artists and the rights of their customers seems to be ongoing.  Each new medium seems to go through a struggle before the balance is found.  Music seems to have settled, or at least to be quiescent.  Video and books are still a little more turbulent.

Rewarding artists for their efforts is important, but in my own experiences, DRM-free content has been a great tool for making me aware of new content and getting me to part with my hard-earned dollars.  On the other hand, content locked down has driven me away from some purchases.  Some examples:

• MP3.com first introduced me to a huge variety of independent music.  Through that site, I developed a taste for Celtic music and ordered albums from a dozen artists that I would never have come across otherwise.  In one particular case, I discovered a band that was playing at a bar three blocks from my apartment.  I went to their show and bought all three of their albums at the show.  (I wept when MP3.com was destroyed).
• On another front, I have stopped buying videos from iTunes due to issues with DRM.  My previous apartment complex provided pathetically slow download speeds, so streaming videos were not a realistic option.  But when I purchased an HD version of "Downton Abbey" through iTunes, I discovered that I was unable to play it on my screen.  Emailing support, they nicely informed me that I would have to buy an Apple monitor.  In the meantime, I could watch the content I had purchased on my iPhone...
• While I have paid for locked-down content on my Kindle, my purchase of tech books has skyrocketed since I discovered several sites that sell their content DRM-free.  The Kindle works great, but for technical books I like to read them and mark them up using iAnnotate on my iPad.

The internet provides a wonderful way for consumers to find new content, but the freedom for customers to use that content on the devices of their choosing can be an important selling point.  DRM too often restricts many important use cases.

Of course, lack of DRM on a product can lead to an increase in casual piracy.  I came across a PDF of Programming in Lua by Roberto Ierusalimschy.  After reading through it for a bit, I noticed "Prepared for [name redacted]" written across the bottom of each page.  With a pang of conscience, I deleted the PDF.

Reading "Zorro"

Today I finished my Amazon.com preview of Isabelle Allende's "El Zorro".  I am not a native Spanish speaker, so this is s significant accomplishment for me -- I've read roughly 10% of the book.

I have always been a fan of swashbuckling tales, and I wanted to develop my Spanish, so this book was an obvious choice.  I bought a paper version of the book 7 years ago, shortly after it came out.  As of this last Christmas, I had made it through about 4 pages with a supreme amount of effort.  It is not that the story was not engaging, but my Spanish was simply not up to the task.  With a thick Spanish-English dictionary in hand, it would take me roughly an hour a paragraph.

So what changed?  Did I suddenly move to Mexico?  Spanish immersion classes perhaps?  Nope.  I got a Kindle for Christmas.

I bought a Spanish-English dictionary for the Kindle, and with that in hand, looking up a word just involves a click of a button.  It has made more of a difference than I ever expected.  Starting from the beginning again, I had passed up my old position in an evening, and completed the first chapter in a weekend.  I have not read every day or even every week, but nonetheless after 3 and a half months, I have finished my sample.

I have now purchased the book, and I am expecting to speed up even more.  Previously, the dictionary lacked several words, and the translation option was disabled for a book sample.  I ended up losing time trying to deduce an unknown word by context.  Now I can translate whole paragraphs with just a couple of clicks.

I wonder what this will mean for the future of foreign language eduction.  Will a Kindle (or equivalent) become a requirement for every serious language student?

Vim, Emacs, and Rome

After visiting France, my wife and I spent a week visiting Florence and Rome before returning back to the States.  The architecture of Europe is amazing.  Even small towns have cathedrals about like we have Starbucks -- one on every corner.  However, of all that I have seen, Rome's subway system struck me as the most awe-inspiring engineering feet that I have seen.

That is not to say that Rome has a particularly good subway system.  In fact, it struck me as pitiful compared to Paris or London.  In Paris, the subway will take you within a block or two of wherever you want to go.  In Rome, there are only two lines in the entire city, which form a giant plus.  You are lucky if your destination is six or seven blocks away from the closest stop.

The problem is that Rome has a rich and ancient history, and any time you dig twenty fit down, you will probably hit some of it.  Work must stop so that an archaeological team can investigate it.  In this context, it strikes me as nothing short of miraculous that someone could cut a continuous line across Rome.  Twice.  Whoever had the vision, the political savvy, and the sheer force of will to accomplish it is a force to be reckoned with, and a (wo)man to be feared.

Returning home, I discussed editors with one of my friends.  He and I are both editor junkies and have had phases of being Vim and Emacs fanboys.  However, my friend has recently been experimenting with SublimeText.  When I asked him why he was switching, he responded that it was for ease of writing extensions.

Both Vim and Emacs are highly extensible.  Both are old, even ancient as editors go.  Despite the extensibility of these editors, their age makes them difficult to adapt sometimes.  I was struck with a parallel to Rome and its challenges building its subway system.

Is it possible to streamline and revitalize these editors?  Or are all large, successful software projects doomed to be mired in their own histories?

Election time is coming soon.  In a few more weeks, I'll be able to turn on my radio without hearing election ads.  Sweet bliss...

For the most part, I think everyone knows which candidates they will vote for.  You like Obama, you like Romney, or you like neither and have to hold your nose when you vote.  Either way, you probably were decided on your choice months ago.  I know that I was.

The California propositions are a different story.  They are generally confusing, and often deliberately so.  Personally, I am not a fan of the proposition system in general -- a lot of bad laws seem to come from good intentions.  But that is another rant.

When the propositions come up, I like to look at the endorsements coming from different newspapers.  Each newspaper is at least a somewhat disinterested party, and it is the paper's business to understand the issues for each proposition.  This time, I thought that it would be fun to go about it a little more formally.

I went to Wikipedia and took the list of California's largest papers by circulation.  I then visited each paper's website and searched for their endorsements.  (I was not able to find the endorsements for the Orange County Register, Investor's Business Daily, or La Opinión).  The results are summarized below:
 

Proposition:	30	31	32	33	34	35	36	37	38	39	40
Los Angeles Times	YES	no	no	no	YES	no	YES	no	no	YES	YES
San Jose Mercury News	YES	YES	no	no	YES	YES	YES	no	no	YES	YES
San Francisco Chronicle	YES	YES	no	no	YES	YES	YES	no	no	no	YES
San Diego Union-Tribune	no	YES	YES	no	-	YES	YES	no	no	no	YES
Sacramento Bee	YES	no	no	no	YES	no	YES	no	no	YES	YES
The Press-Enterprise	no	no	YES	no	no	no	YES	no	no	YES	YES
Fresno Bee	YES	YES	no	no	no	no	YES	no	no	YES	YES
Los Angeles Daily News	YES	YES	YES	no	YES	YES	YES	no	no	no	YES
Long Beach Press-Telegram	YES	YES	YES	no	YES	YES	YES	no	no	no	YES

There is a surprising amount of consensus between the papers.  For almost half of the propositions, the choice was unanimous.  Here is the breakdown for different issues.

Unanimous for:

• Prop 36: Revise 3 strikes law.
• Prop 40: State senate redistricting.

Strong support:

• Prop 30: Governor's tax increase.
• Prop 31: Legislative reforms.
• Prop 34: End death penalty.

Contentious:

• Prop 32: Paycheck protection.
• Prop 35: Human trafficking.
• Prop 39: Out-of-state corporate tax.

Unanimous against:

• Prop 33: Auto insurance.
• Prop 37: GMO labeling.
• Prop 38: Munger tax plan.

Prop 37 is particularly interesting.  Judging by the number of signs I see around town, it would seem to enjoy strong support among voters.  While there have been a number of seemingly disingenuous ads attacking it, the papers raise some fairly cogent arguments against the proposition.  Even the Santa Cruz Sentinel (not on the list) opposes it:

Citizens would be empowered to sue grocers they believe to be selling unlabeled GE foods, without needing to prove any damages. Clearly, this provision would create even more lawsuits. And who would this benefit? Lawyers. That's what happened after voters in 1986 approved Prop. 65, requiring disclosure of toxic chemicals. The result has been more than 16,000 legal actions. Some were warranted, others were aimed at forcing businesses to settle quickly rather than pay for court costs.
...
The initiative may be well intentioned, but it creates more problems than it solves. Vote no on Proposition 37.

The comments on the article are perhaps predictable:

• "It just goes to show you the hands of the biggest corporations (Monsanto, Dupont, etc) reach far and wide."
• "Who paid for this article to be published? Could it be the no on prop 37 group??"
• "I guess when you don't have the truth on your side, your only option is to confuse people."
• "Absolutely Irresponsible journalism."

Politics as usual. :-)

Why is there an app for that?

I've noticed a disturbing trend lately when I browse the internet with my iPhone or iPad.  It seems like every site that I visit wants me to download their custom iPhone/iPad app.  News sites seem particularly guilty of this quirk.

The main purpose for these apps seems to be storing data offline.  If you have a number of articles cached, you can go read them in your favorite wifi-free park.  Of course, many of these apps seem poorly designed.

Flipboard works quite well, which eliminates any valid reason for a news-related iPhone app that I can dream up.  Since it is now available on Android, it would seem to be much wiser for news sites to focus on creating a quality version of the site for mobile users, and leave the app development to Flipboard.

Also, why don't browsers allow for the same functionality that Flipboard provides?  If users want to be able to review content offline, shouldn't developers be given the tools to create that experience?

Free My Phone

Sprint is a distant third in the iPhone market.  In a somewhat desperate move, they have launched a series of ads about their unlimited data plans.

crickets.

I still have unlimited data with AT&T.  I have not exactly been ecstatic about the service, but I have stayed with them.  Verizon seems to be more popular these days, but AT&T has marginally better coverage in the areas I care about.  More to the point, switching plans is a marginal pain, and I have never seen a reason to bother.

Is unlimited data worthwhile to me?  No.  I'd drop it in a heartbeat if a better offer came by.  So what would it take?

UNLOCK.  MY.  PHONE.

I paid for an expensive phone and I cannot take it with me we when I change networks.  That irritates me immensely.  Sprint, unlock my phone, and I'll switch to your network in a heartbeat.

The Ethics of Piracy

I'm not especially interested in the strict legality of "piracy".  The information age has put the rights of artists and the rights of end users into conflict.  The courts and the legislatures of the world are working on sorting it all out.  Things will change, and a new balance will be established.  I have definite feelings about what the balance should be, but in the meantime, what is the path of the righteous?

I have watched movies and TV shows online for free.  When possible, I try legitimate channels.  I am a fan of Hulu and Netflix.  Some networks make their shows available online.  However, there are times that legitimate channels do not work.  Either the show is not available for free, or there are absurd restrictions.

While I was in France, I was not able to watch several shows online since I was not in the United States.  Why should that matter?  Why should the pilot of the Walking Dead only be available to me if I am in the United States?  Companies are free to make their content available to whomever they wish, though the restriction does seem odd.

I have DVDs in my collection with Chinese subtitles.  I have watched movies on YouTube that are almost certainly not authorized.  Where should I draw the line?

Entertainment companies have been vicious in protecting their cartel.  I have little love for them, and I have no ethical qualms about removing their restrictions for materials that I have purchased.  Nonetheless, my actions have not always been ethical.

Is it ethical to buy DVDs from China?  Is it ethical to watch YouTube videos of copyrighted material?  I believe that the answer is no.

Is it ethical to support a corrupt system that is taking away the rights of its users?  I am not totally sure.  For the time being, it seems the lesser of two evils.

Perhaps the best bet is to limit yourself to free material.  Libraries make a great deal available.  Video rentals are cheap.

We live in dangerous times.  We need to be vigilant for our rights, and respectful of the rights of others.  The convenience of the internet makes it easy to violate both.

I have a Hidden Markov Model... Now What?

I have been working on creating hidden Markov models (HMMs) for computer viruses.  Now that I have them, I'm running into an interesting complication.  Namely, what can I do with them?

With an HMM, you can get the statistical probability for any particular series of observations.  For a very simple case, consider a loaded die.  50% of the time, it will roll a 6.  Otherwise, it will roll a number between 1 and 5 (10% chance each).  Once you have your model built up, you can determine the probability of a series of rolls.

So, continuing the example, pretend that you observe 10 sixes being rolled in succession.  What are the odds that this sequence would have been rolled with the loaded die?  (1/2) ^ 10, or 1 in 1024.  Given these observations, is it likely that you are using the loaded die?

Well...  it depends.  What are the other models?  The probability for the same sequence with a fair die would be 1 in 6,0466,176.  On the other hand, if you suspect that the die might be loaded so that it rolls sixes 90% of the time, the observations fit much better with that model.

My first exposure to HMMs was in linguistics.  I built up two language models for classified advertisements -- one for Spanish and one for English.  By comparing the probabilities of any random classified ad, I could guess fairly easily whether an ad were English or Spanish.  But if it happened to be in French or Vietnamese, my tool would have failed miserably.  (On a side note, one of my friends faced with a similar problem for news stories used a simpler solution -- he counted the number of 'the's and contrasted that to the number of 'el's and 'la's.  I never heard of a single bad identification with his approach.  It goes to show, the sophisticated solution might not always be what is needed).

This raises some interesting questions for me in the context of computer viruses.  HMMs seem to be a compelling option for virus detection, but what do they compare against?  You can imagine a series of models built for different virus families, but what if the file is not a virus?  It does not seem realistic to build a model for 'all benign programs'.  Neither does it seem realistic to build a model for each type of benign program.

There is likely a clean, well-known solution.  I just don't know it yet.

Otherwise, life in Laval has been fun.  My wife has arrived, and we've started to explore the surrounding town together.  The Lavalloise seem to be a little shy about their town.  Compared to Rennes or some of the other larger towns, perhaps Laval is a little sleepy.  But somehow it is very cool to sit and have a glass of wine at the foot of an 800 year old castle.  Coming from the western United States, where 100 years seems like a long time, the history of Laval is amazing.

The Virus War

In ESIEA, I am doing research on metamorphic viruses.  It is a new area for me, so I have been reading up on lots of new material.  I am fascinated at some of the gambits and defenses that are happening in the war between virus writers and antivirus researchers.

In the past week, I have been experimenting with virus construction kits, octave (free version of matlab), and reading reams of papers on computer viruses, hidden Markov models, etc.  I feel like I am going in about 12 directions at once.  But as my master's thesis adviser once told me, "that's research".

A quick history of viruses...

The classic viruses were fairly easy to detect through a method known as "signature detection".  Essentially, virus scanners look for a bit pattern associated with a virus to identify a corrupted file.  This method is still the predominant one, but newer viruses are being designed to evade this method.

"Encrypted viruses" attempt to evade scanners by encrypting the body of the virus.  Typically, this would be done with a XOR operation, so that the same procedure can be used to both encrypt and decrypt the body of the virus.  By itself, this approach is not especially useful -- the virus scanner can still identify the signature of the encryption/decryption code.

"Polymorphic viruses" improve on encrypted viruses by mutating the decrypter function.  A simple version of the signature detection approach will then fail totally.  Except...  Modern scanners will decrypt the virus body, and then scan the virus.  (I am still a little fuzzy on how they know when to decrypt the virus body.)

But polymorphic viruses point the way to a far more interesting approach.  Rather than relying on encryption, "metamorphic viruses" mutate the body of the virus.  This strategy can evade signature detection approaches without relying on encryption.  (Interestingly, DRM systems are apparently exploring this technique to defy reverse engineering efforts).

Detecting metamorphic viruses is fairly challenging.  Fortunately, most of the metamorphic viruses today have not been particularly good.  But some are.  NGVCK (Next Generation Virus Construction Kit) was designed (apparently) as a proof of concept.  It produces harmless, but hard to detect viruses.  (Its last release was in 2002 -- virus scanners might have caught up to it these days).

Current research has been exploring statistical models, especially hidden Markov Models (HMM).  The results seem promising, but the battle is not over.  Some research suggests that attackers could tune the mutations to emulate benign files.  Virus scanners are then left with the unpleasant choice of rejecting benign files or accepting some malicious files (and probably some of both).

Anyway, it is an exciting new realm for me!

An American in Laval

After finishing up a fantastic summer at Mozilla, I hopped on board a plane to France to begin my 3 month odyssey abroad.  I was still exhausted from the all hands meeting at Mozilla.  I woke up at 4am to catch the shuttle to the airport, with a layover in Philly, followed by an hour shuttle from Charles de Gaulle to the train station at Montparnasse, followed by a 2 hour train ride to Laval, finally to arrive at my destination at about noon the following day.  I think I am just finally catching up on sleep now.

I have been in France for almost a week, and I've been overwhelmed by my reception.  The people here have been uniformly friendly, and have gone out of their way to make me feel welcome.  The town of Laval is lovely, and the food has been delicious.

The Saturday market in Laval was overwhelming.  In California, we have farmer's markets, but these are pitifully small compared to Laval.  There was fresh-baked bread, giant tubs of paella, seafood so fresh that it was literally trying to escape, and produce that has to be seen to be believed.  I think the produce section alone would be equivalent to 3 or 4 farmer's markets back home.  I think I will enjoy my time here.

So far, the biggest difference that I have noticed is that there is a sharp divide between work and play.  In the states, we buy huge cups of coffee and take them to go so that we can go back to work.  Half the time, 'work' might consist of Facebook and Farmville, but the pressure to be at our desks is very strong.

In France, cups of coffee are small, and no one gets them to go.  You sit and chat with friends, and when you are finished, you go back to work.  And then you work.  I'm not sure who comes out ahead in terms of production, but I am gaining an appreciation for the French approach.

Counting Tiger Vim Fu, Macro Monkey Vim Fu

In my travels, I have seen many masters of their text editors, but none to match the masters of the ancient and complex art of Vim Fu.  In contrast to the arts from The Land of The Darkened Sun or the Builders, the art of Vim Fu has great subtlety and variety.  If you were to see two different masters of Vim Fu, you might not even realize that they were practicing the same art.  And so, in this article, I attempt to chronicle several of the more common types of Vim Master that you may encounter.

The Counting Tigers have a straightforward, powerful style. With a near Rain Man like ability to count at a glance, practitioners of this style will frequently use commands like 5dd or 2y{.  To Counting Tigers, the world may be broken up into units and numbers.

The masters who follow the Path Of One Thousand Stars (though they use an equal number of question marks, pluses, dots, and other meta characters) eschew the straight-forward approach of the Counting Tigers, instead opting for the elegance of patterns.  Their forte will be commands like yt,.  And :%s/old/new/gc is deeply ingrained into their muscle memory.

The Masters of the Hidden Mark rely on mx and 'x extensively, whereas the Counting Tigers would remember the line number and type 166G.  The Visionaries rely extensively on visual mode, highlighting relevant sections and applying their commands to all the region.  Members of this clan frequently rely on column mode editing, an art that other masters often see as being of little value.

The Way of the Macro Monkey leads practitioners to define mini programs of character sequences, constantly creating ephemeral, custom scripts to achieve their goals.  The Pure take this craft to another level, constantly redefining the basic art of Vim Fu to the point that it is not recognizable by other Vim Fu practitioners.  Their .vim/plugin/ directories may have thousands of .vim files.  They can also be readily identified by their fierce, nearly fanatical refusal to use any other tool.

In great contrast to The Pure, there are a number of Wandering Masters.  While Vim is their first weapon of choice, they will use Eclipse, NetBeans, or any other tool that makes their task easier.  They eschew the more esoteric features of Vim Fu, instead seeking to synthesize features of other arts with the core basics of the Vi Path.

Here I feel I must make a special mention of the masters of the Viper Clan.  Though not proper Vim Fu masters, they seek to join the great art with techniques gleaned from the masters from the strange, twisted lands of Emacs.  Some consider their art an abomination.  Others hope that they will bring civility to Emacs.  Yet others prophesy of a chosen one that will at last bring true peace and unity to the House of Vim and the House of Emacs.

Lastly are those of "No Style".  Borrowing from the philosophies of Bruce Lee, these rare few strive to master all aspects of Vim Fu, and yet limiting themselves to no one approach.  These masters can do things with their Vim Fu that seem magical even to other Vim Fu masters.  While it is not meant for all of us to truly master the art of Vim Fu to this degree, by aspiring to perfection, and constantly extending our mastery of our craft, we can become better practitioners of the Noble Art.

Switching to LaTeX

I am giving up on Word.  After many years of use, I've gotten too irritated with its many quirks.  The final straw for me was when it inserted section breaks in the middle of my novel and would not let me delete them.

So instead, I am now using Aquamacs and LaTeX.  When I started the process of converting over my novel, I was not sure if it was really a good idea.  LaTeX is great for technical papers (especially greek-letter heavy ones that we are so fond of in programming languages research), but I was not sure if it would really offer much for a fiction writer.

Almost immediately I found the benefits.  Although there are some hassles, the precise control of formatting is fantastic, and that alone makes the trade very worth while.  Also, being able to use my vi key bindings with emacs viper mode has been a true delight.  And I can actually track my changes through source control.

It has also been interesting what I have not found useful.  I had expected that variables would be a useful feature.  For instance, I could make a command for a character's name, and then allow myself to change it easily if I thought of a better one.  I tried this for one of my characters briefly, but found it to be more of an irritation than anything else.

But there is one feature that I did not give much thought, and yet has me fully committed to a life using LaTeX for all of my non-technical documents.  And that is comments....

It seems so obvious now, but before I had a wide variety of text files and hand written note-cards; all of these have been collapsed into the .tex file itself.  I use simple line comments for things like 'fix up this section' with no worries about cluttering up the document for anyone I ask to review it.  For more complex notes, like outlines and lists of my characters, I make new commands.  When I set the command to do nothing, the notes are invisible.  But when I change it to output the text, I can share those details with anyone who wishes to review it.

LaTeX is not a program for everyone.  But if you are familiar with it, don't be afraid to use it for less technical documents.  Many of the same benefits will still carry over.

Narcissus addon available

Today I uploaded a plugin for Narcissus JavaScript.  Details can be found here.

It requires Firefox 4, so to test this out, you will need to build from source for the time being.  Longer term, the plan is to swap out the JS engine at will.  But, it is at least usable for experimenting currently.  Enjoy!

More Awesomeness: jstests.py

In working with Narcissus, I've uncovered a jewel of a resource in Mozilla's JavaScript unit tests.  The value of unit tests is well understood.  The most critical quality, however, is that they are maintained.  Mozilla has done an exemplary job of this.  The unit tests are constantly growing.  As I write this, there are just shy of 3000 unit tests.  All of them pass with the Tracemonkey engine.

The organization of the tests is well thought out.  Each version of JavaScript has its own test directory.  So if you don't care about certain batches of tests, it is easy to ignore them.

Unfortunately, as with many other things at Mozilla, there is an abundance of out of date documentaion.  The JS Test Library page refers to a Perl script, which appears to be more or less abandoned.  Instead, there is a python script 'jstests.py', written by Dave Mandelin in his copious amounts of free time.  Unfortunately, the old version of unit testing documentation comes up first when searching for it on Google.  Alas.

One thing that is sadly not emphasized is just how easy it is to plug in a different engine to use these tests.  In fact, I have a "shell" for Narcissus JS, written in 30-some lines of Python code.

To hook into the testing framework, you need a "shell" that supports two command line options:
1) -e, which interprets a string of JavaScript code.
2) -f, which handles a JavaScript file.
It must be possible to support multiple files and strings of code, though this is not a particularly difficult challenge.

Some tricks:

• The -m option will let you specify your own module of JS tests.  This is also a convenient way to insert a little JavaScript into the testing process to handle tests relying on non-standard JS features.
• The -x option allows you to ignore a list of tests.  Since Mozilla includes tests for special Firefox-only features, you will want to use this.
• The --failure-file option will write the failing unit tests to the specified file, useful as input for the -x option.

On another note, Narcissus now runs without special extensions in Firefox 4.  Huzzah!  Now to make it interact smoothly with the DOM...

This summer I am working at Mozilla. It is an awesome, awesome place to be. The people here are friendly and amazingly bright. Plus there is a dinosaur in the lobby. How can you beat that?

My work has been looking at Narcissus JavaScript. Narcissus is a JavaScript implementation written in JavaScript, and as such, it is an ideal vehicle for playing around with the language.

Narcissus relies on a few special extensions, which require a special build. It is not hard to build, but it is not all that well documented either. Here is the process that I use (in OSX):

  $hg clone http://hg.mozilla.org/tracemonkey/
  $cd tracemonkey/js/src/
  $autoconf213
  $./configure --enable-narcissus
  $make

With that, you are good to go. From the command line, ./js will give you a tracemonkey shell. Here, you can use evaluate(prog) to test out narcissus.

With some adjustments, Narcissus should run within a standard Firefox build. I am down to a handful of failing unit tests. After that, running Narcissus within a browser should be as simple as loading the Narcissus js code.

NetBeans -- The Least Terrible IDE

My favorite text editor is MacVim. It lays a nice Aqua interface over the top of the classic editor. It is not perfect. I could rant for hours on its shortcomings, but it is still a shorter rant than I have for Aquamacs and TextMate, and these are the only two other text editors that I find even usable.

But text editors and IDEs are not the same thing, despite the fact that they do many of the same tasks. When I start working with a Java or Scala project, I want the advanced functionality that a full-featured IDE offers. (And yes, I am aware of Eclim, but that does not really change the fact that an IDE has been brought into the process.)

I have played around with a number of IDEs over the years; I have yet to settle on one. Eclipse is the classic, with extensions and plugins for about everything. Now that IntelliJ has an open-source version, I have been spending a good deal of time with that. Nonetheless, NetBeans is still, in my opinion, the best.

It is not that it is the prettiest. In fact, it is by far the ugliest of the three on any other system besides OSX, and still not that pretty even then.

It is not that it has the best Scala plugin. While I have had more success with the NetBeans plugin than the Eclipse one, I have found IntelliJ's to be more full-featured (albeit a little more of a hassle to install, since the version offered through their plugin manager does not seem to work. But that is another rant...).

And while I love some of the plugins available (the jVi plugin in particular being the best vi emulator that I have seen for any IDE or editor), they still pale compared to the wealth offered by Eclipse.

No, these are pluses, but relatively minor points. The reason that I keep returning to NetBeans, in a word, is Ant.

There is a fantastic article by Jeff Atwood titled "F5 is not a build process". This comes close to the core of the issue: every project should have a build process that functions outside of the IDE. This is a problem I have seen even with some open-source projects. Instructions like "start up NetBeans" or "download the Eclipse plugin" should never appear for any tool that is not specific to an IDE.

Unlike Eclipse and IntelliJ, NetBeans does not have its own special build tools. It defers this to Ant (or yes, you can use Maven instead). As a result, the build process is available outside of NetBeans. In my experience, projects designed with the other IDEs only create a build process as an afterthought.

This is huge. It means that the IDE actually takes care of the build process for you, as is just and proper. It also, however, allows you some extra options when things go wonky.

I do some weird things with my IDE. Using Scala is one of the more tame uses, but even that can start causing problems when you combine it with Antlr or JavaCC. And God help you if you want to add unit tests into this mix. I have managed to confuse and break every IDE I have used, to one degree or another. (To be fair, IntelliJ has generally handled more weird stuff than the others).

The beautiful thing about NetBeans is that when it stumbles, it can still function. For one project, NetBeans gave me hundreds of spurious errors, but the build process still worked.

NetBean's generated ant files include a number of hooks where you can add in your own ant tasks, a feature that I have found invaluable. Furthermore, if something in NetBean's ant files is horribly broken, you can replace it with your own; not something that I would recommend, but it can save you in extreme cases.

And so, despite NetBeans numerous quirks and limitations, I still list it as my favorite IDE. And while I believe the NetBeans team will address the product's current limitations sooner or later, I doubt that either the IntelliJ or Eclipse teams will ever fix what I see as a fundamentally broken architectural decision.

What main method?

I've been looking for an excuse to learn more Scala for a while. Currently, I am working on implementing the TAPL interpreters in Scala.

I ran into a problem that took me a while to figure out. To illustrate, I'll give a simple example. Consider this main method:

def main(args: Array[String]) = {
var sum = 0
for (s <- args) { sum += s.toInt }
if (args.length > 0) {sum/args.length} else 0
}

This simply averages a list of numbers specified from the command line. But when I try to run it, I get this error:

Exception in thread "main" java.lang.NoSuchMethodError: main
Java Result: 1

With a very slight change, this will work:

def main(args: Array[String]) = {
var sum = 0
for (s <- args) { sum += s.toInt }
println(if (args.length > 0) {sum/args.length} else 0)
}

This just prints out the average, no complaints.

So what is the difference? Why does printing out the result make a difference? The answer requires a little understanding of the way that Scala works.

Scala relies on type inference. This means that it tries to determine the types of your variables (including methods) from analyzing your program. This is not the same as dynamically typed languages like Perl or Ruby -- the type checking is still done at compile time. The compiler is smart enough to infer the types of your variables from the context.

But there is a problem here. It also has implicit returns. The program will automatically return the last executed expression in the program. This has been a popular feature in many programming languages, including the Lisps, Ruby, and the various ML dialects (of which Scala itself is something of a shirttail cousin). Personally I despise this feature, but I am in the minority.

Combining these features with Java programmer expectations leaves a nice little trap. The main method must not return anything. (Or if you rather, its return type must be Unit). If we copy the second version into the interpreter, we will see this:


main: (Array[String])Unit

The second version works because it prints out the results -- println has a return type of Unit. However, the if statement in this example has a return type of Int. The first version of the program, although it looks nearly identical, has a different type:

main: (Array[String])Int

As a result, Scala looks for main: (Array[String])Unit but cannot find it and returns an error message. This is easy to fix:

def main(args: Array[String]): Unit = { ... }

Or better yet, we can just remove the equals sign from the original:


def main(args: Array[String]) { ... }

Interestingly, the original version actually catches a problem in my code; I am calculating an average, but then I do nothing with it. More often than not, if you see this error, you are probably doing something wrong -- why calculate a value for no reason?

Still, when you are learning a language, a stumbling block like this can be confusing. The error message is clear enough to anyone who understands Scala. My only suggestion would be to add a special case for main specifically. It is a slight hair in the implementation, but it can save clueless newbies (like me) a little frustration, and really... what other kind of method would be named "main"?

Building Hyype.net with Helma Object Publisher

Today, Hyype.net was launched. Hyype.net is a micro-blogging sports site. It is designed to encourage loud, boisterous, trash-talking about each others favorite teams. We have some ideas of where we'd like to expand to, but the first part is completed and live.

This blog entry is not about Hyype.net though. This is about building Hyype.net, and more specifically, about my experiences with Helma Object Publisher. Helma is a web-development framework using Mozilla's Rhino JavaScript as the server-side language.

Disclaimer: I am not an expert at using Helma. There may be better ways to organize things. Hopefully, some of the Helma guys will step in and point out where I did things the stupid way. This entry is about a newbie's experience learning the framework and building something useful with it.

Getting Started

After having struggled with installing a number of different frameworks, Helma was absolute bliss to get up and running. Inside of a minute I was able to download it, configure it, and get it running. At that time, I was presented with this:


It was a powerful feel-good moment. Not only was it easy to get up and running, but I was presented with something that was actually useful instead of some stupid splash page. This page also has links to a tutorial and documentation.

The tutorial was a great place to start, and after an hour of going through that, I felt like I knew what to do.

HOPped up on objects

At about this point, my romance with Helma began to sour. What exactly is a HopObject? Why is there 'Root' and 'Global'? Why the @#$@! do my macros work here, but not there?

The organization of a Helma project at first seems bizarre. Combined with some subtle differences in how functions work (Why doesn't renderSkin() work like this.renderSkin()?), I slowly developed the feeling that Helma was well-polished on the surface, but buggy and horribly organized. I compared it to PHP, which is about the dirtiest insult you can hurl at a Java-land framework.

And then, all of a sudden, I got it. The pieces clicked together, the organization made sense, and I was left reeling. I had to lie down for a minute until the world stopped spinning.

Here is the key thing you need to understand about Helma: it is an object-oriented framework.

That might not sound like a big point. After all, most frameworks today are built with object-oriented languages and overrun with objects. But that is not quite the same thing. Take Rails, for instance. It has a clean organization with model objects, controller objects, and views (which are probably objects too—I don't really know).

In contrast, each Helma application is one massive, amorphous object. It is like the difference between classes and prototypes. This is the type of organization that makes complete sense if you are thinking in a JavaScripty way.

One subtle distinction about JavaScript programs is that each one runs in the global object. It has no name, we never really see it, but it is there. (This was a crucial point when I was building JOMP—top-level functions are methods of the global object and need no special handling). Helma has just taken this idea and build a web framework on top of it. My master's thesis spends a great deal of time extolling the benefits of JavaScript's object system, so I was a little embarrassed that it took me so long to grock Helma's organization.

Each folder in Helma corresponds to some type of HopObject. These folders will hold the database configuration (if needed), the actions (more on those later), the methods, and the skins (html templates).

Global is the global object where all JavaScript objects live. Before I understood it properly, I had a fair amount of the skins stored there. Now, I have only a single macro. I'm not really sure I even need that. This is probably good form — global===bad, as Douglas Crockford would say.

Root is the single massive object that is the web application. This was the single most crowded directory for Hyype.net. Anything that did not clearly belong to a single object ended up here.

There is also HopObject, which serves as the parent object of all other Helma objects, including Root itself. I probably should have used this more, but this directory is empty for Hyype.net.

Every other directory corresponds to some object specific to your application. As it turns out, Hyype.net really only needed 3: Users, Profiles, and Thread.

For the rest of this article, remember that everything is owned by some object. A skin or an action belongs to some object—HopObjects are not passive lumps of data like they are in other webdev frameworks.

Persistence

The core use of objects in almost any web framework is to serve as containers for data. This is easy to do in Helma. It does take some configuration, but that is pretty minimal. Here is the User configuration for Hyype.net:

_db         = myDataSource
_table      = user
_id         = id

username    = username
password    = password
email       = email
verified    = verified
createdAt   = created_at
updatedAt   = updated_at

profile = object (Profile)
profile.foreign = user_id
profile.local = id

This is a little more work than it would be in Rails, but it is not too bad. It would be nice if Helma could generate a base config, but it would really only save a minute of work. Better yet would be if it could default to the same names as the DB fields. The associations are also a little trickier than in Rails, but not bad. The documentation is fantastic, however, so I was able to figure things out without too much trouble.

One note: In good Helma form, the database field names should have been in all caps. I ignored that, but it would have made the configuration a little clearer. With all lowercase, it is not as obvious which side is which.

HACking something together

Every action corresponds to a url. While there are different ways to do it, the standard approach (and the one I used) is to create a separate .hac file for each action. Here is a simple one:

res.data.title = "About Hyype.net";
res.data.selected = "about";

res.data.body = this.renderSkinAsString("about");

this.renderSkin("layout");

"res" stands for "response". The data property stores the variables that your template will need. The renderSkin methods will write out the templates.

I would have liked Rails' style layouts. If there is a built-in setup for this in Helma, I could not find it. Rolling my own was easy enough though. I did this by creating a skin cleverly named "layout". Within it, there is a body variable. Using "renderSkinAsString" I could fill out this spot and then render the layout itself.

Skins

My last project involved JavaServer Faces. Before that, it was a PHP project. They represent the two extremes. JSF is rigid and unfriendly (and, perhaps worst of all, not html). PHP lets you do whatever you like, even things you should never, never do.

Helma's skins take a nice middle ground. They don't allow arbitrary code, but they do allow you to call macros (special JavaScript functions). The result is that no code bleeds into the html. Some html did bleed into the code, however, but I managed to keep this relatively minimal.

One really cool thing about Helma's organization is that objects can render themselves. They can contain their own skins. For example, Thread has templates for being rendered as clips (displayed on the left rail), details (for a more complete view of the thread), and myClip (for the "My Hyype" page). Within the main page, each thread is rendered with 'str += hyype.renderSkinAsString(clipSkin);' The current thread is displayed on the center of the page with 'res.data.body = selectedThread.renderSkinAsString("details");'

It seems like an unusual way of organizing templates, but I found it very effective. Unlike objects in other frameworks, HopObjects know how to dress themselves.

Here is the template for Hyype.net's threads in clip view.

  <div class="bb-box"><div class="ctl"><div class="ctr"></div></div><div class="pad">
   <a href="<% rootPath %><% this.posterProfileHref %>"><img src="<% this.smallPic %>" alt="<% this.poster %>" title="<% this.poster %>" class="bb-img" /></a>
   Posted by<br />
   <a href="<% rootPath %><% this.posterProfileHref %>" class="lnk-green"><% this.poster %></a>
   in <a href="<% rootPath %>main?cat=<% this.categoryName %>" class="lnk-green"><% this.categoryName %></a><br />
   on <% this.postedOn %>
   <div class="comm"><a href="<% rootPath %>main?threadID=<% this.id %>&pageNum=<% request.pageNum %>" class="clip-title"><% this.title %></a></div>
   <% this.agreeDisagree %>
  </div><div class="cbl"><div class="cbr"></div></div></div>

Utilities

Helma offers a rich set of utilities as part of its core and a number of optional extensions. And if that is not enough for you, the Jala Project offers a bunch more modules. For the most part, these are well-documented and easy to use. The one time I had a problem, a note to the Helma mailing list quickly answered my question.

One problem spot I did run into was trying to include Java code in my project. For the most part, this is not needed. Rhino allows you to call Java. However, I wanted to get the MD5 hash of a string for comparing passwords. I could not call the methods I needed in Rhino, since it was ambiguous to Rhino which method I was calling.

I wrote the method I needed in Java, but there was no good place to store the class file. Instead, I put the class file in a jar and added it to the lib/ext/ directory. This was not the cleanest process in the world, especially since it is shared by all projects, but it did the job.

Deployment

Once we had the application ready to deploy, life got a touch more challenging. We went off the beaten path a bit by not using Antville. Things went more or less smoothly until we tested out uploading images.

The server crashed. We reconfigured a few things. It crashed again. After several hours of research and troubleshooting, we nailed down the problem to our version of Java. Specifically, it was using GNU's implementation, which ignored some of the flags we were passing it. I still have not totally forgiven GNU.

While I think Helma is blameless for this, it is one risk of using a framework that is not as widely used as PHP, JSF, etc. Fortunately, the community is friendly and active, which usually more than makes up for this.

Conclusion

Helma has gotten to be my favorite webdev framework. Given any choice in the matter, this is what I'd use for a new project. Helma seems to be in active development, and I'll be excited to see where it goes from here. If you have a new project coming up, give Helma a look.

Yes, We're Open

90% of the time when I need some work done on my car, I take it to Sears' Automotive Center. This is despite the fact that they are usually unable to fix anything more than the basics.

The main reason I keep going there is that it is incredibly convenient. When I have tire problems driving home on a Friday night, they are open. If I want to drop my car off on a Saturday and then walk to the movie theater in the same shopping mall, I can do it.

In contrast, the Ford dealership has generally done a better job of fixing more complicated problems. But they are only open 9-5, and they are in the middle of nowhere. I can only read outdated snowboarding magazines for so long. Also, they charge twice as much, so I take my car there only in desperation.

Convenience Matters

This should not come as a surprise. Unfortunately, the state of many software projects out there would suggest otherwise. Far too many of them have terrible installation instructions and horrid documentation.

If you want people to use your tools, put yourself in the position of your new users. What are the pain points? Where is this confusing? You won't catch all possible problems, but the more you do catch, the more users you will be able to win over.

I've been working on XMUltra lately. My initial efforts have largely been focused on just this part -- how to make it easy for a new user to get started. I have a ways to go yet, but I think the initial public launch had much better tips for getting started than the internal documentation.

Eclipse vs. Emacs

These two have very different strengths. Emacs is the more powerful of the too, or so the Emacs elite claim. (Not being an Emacs power-user myself, I have to take their word for it).

Eclipse is decidedly easier to get started with. For one, it uses the standard key bindings, so that ctrl-x, ctrl-c, ctrl-v, ctrl-s, etc. all work as you would expect. In addition, it is decidedly prettier. It has nice file open dialogs and more or less intuitive preferences settings.

Actually, Aquamacs comes pretty close here. It takes advantage of the fact that Macs have both a command and a ctrl key, so you can still use command-x, command-v, etc like you would in any other program, but use ctrl for emacs key sequences. It also has nice GUI interfaces overlaying the Gnu Emacs versions, so you can get decent looking file open dialogs (in addition to the regular Gnu emacs versions).

But Emacs and all of its touted power fails miserably in actually being able to transfer that power. Yes, I'm sure it is much easier to write an Emacs plugin than to write an Eclipse plugin, but that is only half of the story. After I write one, someone else has to install it, or nobody but me benefits from it.

In Eclipse, I have to type in a url into a package manager.

For Emacs, let's take a look at a couple of examples of installation instructions:

• JDEE includes steps like "Download the latest versions of Eric Ludlam's speedbar, eieio, and semantic bovinator packages and install them on your system, each in their own directory." Great.
• HtmlModeDeluxe: "First, you must install all four modes needed. The php-mode I tested so far and which seem to work well are Turadg Aleahmad's 1.0.2 and 1.0.4. As I had several reports during the last few weeks that the current mmm-mode cvs does NOT work, I now recomment to first try..."
• Emacs WebDev: "You will need the following packages: *psgml *mmm *generic-x *php-mode *css-mode *xxml *tidy"

Hmm... So do I have semantic bovinator installed on my system already? How do I tell? What version do I have? And, most importantly...

WHEN THE @#$%! DID I BECOME A FLIPPIN PACKAGE MANAGER!!!

This is the single biggest failure point of Emacs. There are some attempting to fix this, but one really, really, really needs to be made part of the Gnu Emacs code base.

A Tale of Two Frameworks

I'm a big fan of Rhino JavaScript. Enough so that I built a JSF-based web framework for it. I spent some time looking around at what other frameworks existed.

One of the first I found was Phobos. They have nice webcasts, and it looks pretty cool. But... For one, Java.net is painfully slow to load. Some might be turned off before they ever see the homepage. But if you have some patience, you might make is as far as the download page and see this in the installation instructions: "You need to have the NetBeans IDE installed."

Wow. So in one fell swoop, the Eclipse users have already been turned away. But maybe this is just for tool support. Perhaps this will work just fine as long as Glassfish is installed. And what does the website say? <cue the crickets>

In contrast, let's take a look at Helma. Here we have a nice, prominent download link. After downloading it and following the instructions, I had Helma running with a useful admin tool inside of a minute. It even has a default file-system persistence setup in case you don't have a database installed. The first experience is lots of happy juju.

So take a wild guess which framework I am using now.

Helma takes some getting used to, but it is worth the effort. The organization is very cool... but very different. While I did some head scratching before I figured it out, I made the effort because Helma had already won me over. I wanted to like Helma, and so I spent the time to find its cool features. Also, the documentation pointed out those features for me. (And seriously... If you are a Rhino fan, check out Helma -- I actually prefer it over Ruby on Rails.)

As Simple As Possible, But Not Simpler

One final note... Sometimes you cannot make things as simple as you would like. If you are building a complicated product, it may be that a complex installation is required. But challenge every pain point. Would a Rails-style sensible default be a wise move? The answer is usually yes.

Back to my mechanic... Before Sears, I went to a private mechanic. The work was fantastic, the prices were fair, and I went there despite the hassle of driving to and waiting in the middle of nowhere.

Unfortunately, he went out of business.