Posts about Programming

Node Knockout 2011 and Twends

toby ho — Mon, 29 Aug 2011 22:19:07 +0000

This past weekend I participated in the Node Knockout - a programming competition where teams of 1 - 4 members build something using node.js in 48 hours. I was very fortunate to have worked with @robfaraj on this, it was a fun couple of days! The end result is what we call ~~Twends~~ Twends: new location is now twends.us. Twends is a real-time Twitter trend visualization...app. It looks like this

First, you either select a trending topic or type in a term of your own, then you will see this word visualization based on a series of packed circles. The larger the circle and the redder it is, the more frequently the word is being said at the moment. All the while, birds are fly across the screen. Each bird on the screen represents a tweet that's been written, now...ish. Just sit back, relax and watch the pretty circles changing shape and the birds flying across your screen. By reading the words you can often learn interesting things. For example, I learned while reading Twends that Justin Beiber fans are called beliebers, while Taylor Swift fans are called swifies.

Oh! And here's a video of it in action

The Tech

Twends was built using client-side Javascript with the help of some great libraries and the Twitter REST API. d3.js is without a doubt the most valuable library(MVL). We used it to layout the circles as well as animate them from their old positions to the new positions each time new data comes in. We also used HTML5 Boilerplate and jQuery. On the server side, we used express and the jade templating language.

Graphics is done using SVG, which d3.js works very well with - although is independent of. IE8 and below do no support SVG so Twends do not work there. Twends could have worked on IE9 but it doesn't. IE9 supports SVG, so that was not the problem. The problem was that IE9 was blocking the JSONP requests I was making to the Twitter API due to this problem, of which the blame is half on Microsoft and half on Twitter. So, if you go to the site with IE, it will greet you with a friendly(?) message and prompt you to install Google Chrome Frame, yea!

The Aftermath

~~The competition is still being judged at the moment, so I don't know what's the aftermath just yet. I guess I'll update this part of the post another time.~~

Node knockout came to an end and the winners were victorious! Congrats to them. I am a little disappointed/surprised that one of my favorite entries: Pitchography didn't win anything, here's a demo of their awesome presentation tool. As for Twends, well we were decidedly better than half of the entries out there :) We also presented it at the Atlanta Javascript Users Group and got some good feedback. I've made the source code public on github as well.

Node Knockout was a great experience. The organizers did a superb job of allowing the contestants to focus on writing code rather than dealing with setting up servers or administrative stuff. I enjoyed everything from the planning stages where we threw around random ideas, to the actually coding and then to the showing it off to people and seeing their reactions and comments. If you love code, then I highly recommend that you try a hackathon like this one.

My JSConf Talk in Video

toby ho — Fri, 19 Aug 2011 15:51:16 +0000

The awesome folks behind JSConf just released a whole bunch of great talks in video form from the Portland event. Great chance to catch up on the talks whether you were there or not! They even put mine up there, which since this is my website, I am obliged to embed here. Wow I sound so very Chinese.

Embedded Content

Book Review: Python Testing Cookbook

toby ho — Tue, 02 Aug 2011 12:47:37 +0000

This is the second of 2 Python book reviews. The book is Python Test Cookbook.

This book is a very complete overview of all aspects of testing with Python. It starts out with simple unit tests, then moves on to nose, BDD, Doctest, acceptance testing, coverage, continuous integration, and smoke tests and load tests. The last chapter of the book gives you a number of advices for starting out with testing.

The Good

I enjoyed this book in that it gave me a birds-eye view of all the tools and techniques that are there for Python, many of which I didn't know about before. I also learn new things about things that I had already used before. Lastly, I liked the last chapter in which the author gives us his heart-felt experiences with testing. Although I don't agree with everything he says(running your tests once a day is way too few), there is a lot of good advice there as well as good stories.

The Bad

One thing that I don't like about this book is that it's too code heavy. Pages and pages of code are print onto the page with only interleaving explanations. I mostly skimmed those parts because reading code line-by-line like that is just too boring. Code is simply not meant to be read in a linear fashion. When you read code, you jump between different files rapidly in order to fill the in missing context as you try to gain an understanding of the code. An even better aid in reading code is the ability to execute the code and instrument it with print statements or stop it with the debugger and inspect the program state. So, in summary, code-heavy books just doesn't make sense. Perhaps instead of writing this as a book, we should just point the readers at a code repository?

The second problem is that not enough was done to motivate each technique that was used. The author didn't properly motivate why one should write unit tests at all. Or why use a given technique listed in the book. Only cursory introduction is given for each technique or tool, and then it was straight to the code. We dwelled too much time on the how and not on the why, which makes this book very unfriendly to beginners.

Summing Up

Overall, I would not recommend this book to beginners or people who are looking for an introduction to testing. I am on the fence about recommending it at all, but it might be helpful for people who are already familiar with test driven development but just haven't been doing it with Python.

Book Review: Python 3 Web Development - Beginners Guide

toby ho — Thu, 21 Jul 2011 23:54:14 +0000

I just finished reading Python 3 Web Development - Beginners Guide by Michel Anders and here is my book review.

Michel Anders is no doubt a brilliant programmer. He has managed to go through HTML, Javascript, jQuery, jQuery plugins(both using and writing), jQuery UI, CSS selectors, CSS 3, Python, CherryPy, SQL and SQLite, unit testing, writing your own ORM, and Python meta-classes all in one book! Which is the problem with this book. I doubt any one who qualifies as a beginner will be able to get any further than chapter 2. I believe the author made the mistake of not putting in enough thought about his audience when writing. But, before making my final remarks, I'll give it proper treatment by commenting on each of the chapters.

Chapter 1

Chapter 1 explained the authors reasons for choosing the tools that he did, and boy were there a lot of choices to be made. Here were the choices:

Web framework: CherryPy
Language: Python 3
Javascript Library: jQuery and jQuery UI
Database: SQLLite
ORM: None. He's rolling his own
Revision Control: svn

There are a lot of high level talk about the rationale of each choice and why he made them. It was a bit strange that he put the choice of using Python after the choice of using CherryPy. The first chapter is quite intimidating for beginners - there is really lots to consider when building a web app. He admits it too, but he puts in a lot of encouraging words to encourage the reader to stay focused.

Chapter 2

Chapter 2 went through the exercise of building a web-based spreadsheet application, a la Google Docs. The goal is ambitious, especially for a "hello world" program. I wondered how he was going to pull it off.

The Author covered CherryPy, jQuery/CSS selectors, a jQuery plugin called jEditable, wrote his own jQuery plugin that builds a spreadsheet which uses jEditable, CSS 3 and HTML all in this chapter. The amount of Javascript in this chapter really surprised me because in the who-is-this-book-for section, it says "introductory level knowledge of JavaScript might be useful, but is not strictly necessary". He simply tried to cram too many things in, explaining each technology only cursorily as he encounters them. I am pretty sure that he lost almost all readers in his audience with this chapter.

The Rest of the Chapters

Since we've already lost almost all readers I will place less emphasis on chapters 3-10. Basically, it is more of the same: trying to cram too much information into his readers' minds too quickly. These chapters whirl-wind tour the following:

Writing an authentication handler, and then later authorization within CherryPy
Using the file system as storage
Using SQLite to replace the file system as as storage
Using SQL directly to manipulate and query the database
Ajax and lots more of jQuery and jQuery UI
Unit testing in Python
Building a hand-rolled ORM, and then improving it using meta-classes, where the phrase "make your head explode" was used
Adding a CRUD interface to the system for his CRM, and then refining the CRM.

The further you get in the book, the larger blocks of code were put on the page, and the author just tries to talk through what is going on. I skimmed most of these chapters because when you are explaining code that is so dense, it is easier to read the code itself - which I often resorted to - than to read the prose.

Conclusion

Unfortunately, I cannot recommend this book. Reading from the book title, the audience is a Python programmer, perhaps beginner to intermediate, who is beginning in building web applications. This book will send this programmer away crying. I think perhaps a more fitting title for this book is "A Coding Cowboy's Journey In Web Development with Python 3".

Python Book Reviews Forthcoming

toby ho — Fri, 15 Jul 2011 10:38:09 +0000

The kind folks at PacktPub are letting me review a couple of Python books. Hooray! They are

Look for my reviews of them in the near future.

The Javascript Stacktrace Blog Post

toby ho — Wed, 08 Jun 2011 21:44:11 +0000

For the past week or so I did some research on how best to include scripts in Tutti's sandbox environment and still get the best stacktraces possible. I got sucked in deeper and deeper and it just got uglier and uglier. This post will attempt to summarize all the knowledge that I've uncovered.

What's a Stacktrace?

When I say stacktrace, I mean it in the context an exception being thrown in your code, and getting the information about where exactly in your code the exception was thrown. This is what a exception might look like

Error: Buck stops here.
at failInlined (http://localhost:8000/error_object/:16:11)
at catchError (http://localhost:8000/util.js:77:26)
at http://localhost:8000/error_object/:49:1

First, there is an error message. Then, the stacktrace. The stacktrace basically represents the state of the program's call stack at the point when the exception was thrown. The stacktrace also contains the location(file URL and line number) of each call in the stack, which is extremely useful for debugging.

So What's the Problem?

The problem, is that not all browsers give you informative stacktraces. Not only that, different browsers give you the information in different ways.

Most browsers today give you the stacktrace information in the Error object that was thrown. In Firefox, Chrome and Opera, for example, the Error object contains a stack property which is a string representation of the stacktrace, much like the example shown above.

// Get stacktrace on Firefox, Chrome and Opera

try{

   throw new Error('Buck stops here')

}catch(e){

   console.log(e.stack) // Yea!

In Safari, however, the stack property is non-existent: it does not give you the full stacktrace. Instead it gives you the location for the top stack frame only. This is gotten from the properties sourceURL and line.

// Get 1-level stacktrace on Safari

try{

  throw new Error('Buck stops here')

}catch(e){

  console.log(e.sourceURL + ': ' + e.line) // Meh

On IE, you are out of luck. There is no stacktrace information available on the Error object.

Going Deeper

Javascript can be executed in various different ways/techniques, and not all of them yield good stacktraces. My next objective was to see what the stacktraces look like for the different techniques

Technique 1: Script Tag Includes

The script tag include is probably the most common way to execute Javascript on a webpage.

<script src="myscript.js"></script>

Technique 2: Inlined Script Tag

Alternatively you can inlined Javascript code inside of the script tag.

<script>

console.log('Hello world')

</script>

Technique 3: Dynamically Inserted Script Tag

You can also create a script element dynamically in Javascript and load some Javascript into it.

var script = document.createElement('script')

if (<IE sucks and you are it>)

  script.text = jsCode

else

  script.appendChild(document.createTextNode(jsCode))

document.body.appendChild(script)

document.body.removeChild(script)

Technique 4: Eval

Use eval to execute the Javascript in Javascript. You probably want to global eval it.

// This code for global eval stolen from jQuery 1.6.1

(window.execScript || function(jsCode) {
	window["eval"].call(window, jsCode);
} )(data);

Technique 5: Data URL in the Script Tag

Using a data URL in a script tag is...I am not sure if anyone actually does this, and I am not entirely sure it's useful, but what the heck.

<script url="data:application/javascript,console.log('hello%20world')"></script>

Summary: Stacktrace Information From Error Object

Long story short, I tested the stacktrace output for each of these five techniques in different browsers. I summarized the results in this here table (that's right, geek out!)

Browser	Include				Inlined				Dynamic				Eval'ed				Data URL
Browser	LN	SR	ST	AG	LN	SR	ST	AG	LN	SR	ST	AG	LN	SR	ST	AG	LN	SR	ST	AG
IE 6 - 8																	Unsupported
IE 9
Safari 5	Y	Y			Y	Y			Y*	Y			Y				Y*	Y
Chrome 11	Y	Y	Y		Y	Y	Y		Y*	Y	Y		Y*	Y	Y		Y*	Y	Y
Firefox 3.6 - 4	Y	Y	Y	Y	Y	Y	Y	Y	Y*	Y	Y	Y	Y*	Y	Y	Y	Y*	Y	Y	Y
Opera 11	Y	Y	Y		Y*	Y	Y		Y*	Y	Y				Y		Y*	Y	Y

*: Line number may not be helpful
LN: Line numbers
SR: Source file URL
ST: Full stacktrace
AG: Call arguments in stacktrace
Include: Script tag with src set to a Javasrcipt file URL
Inlined: Script tag with inlined Javascript
Dynamic: Dynamically inserted script tag with inlined Javascript
Eval'ed: Global eval'ed Javascript
Data URL: Script tag with src set to a Data URL containing Javascript

A straightforward script tag include yields the best stacktrace information. As the asterisk(*) indicates, the dynamic script element, global eval, and data URL techniques can yield stacktraces with non-helpful line numbers. The data URL technique actually gives correct line numbers, it's just that all the newlines have been encoded and the entire source jammed onto a single line - so, I can't really say the line numbers are all that helpful. Strangly, Opera yields non-helpful line numbers even with the Javascript-inline-in-a-script tag technique.

So That's it? No IE?

There's no way to get stack information on IE, not unless you want to do the other thing, but you don't want to know about that.

What's that? You do want to know? No you do not! Just forget I said anything.

Okay! I'll tell you, but you are going to regret it.

window.onerror

On IE - and as it happens on Firefox and Chrome as well - you can register an onerror handler on the window object, like so

window.onerror = function(message, fileURL, lineNumber){

  log(message + ': ' + fileURL + ': ' + lineNumber)

The handler will be called anytime a thrown exception is uncaught: propagates all the way to the bottom of the call stack with no catchers. Catching the exception will cause the handler not to be called, unless you rethrow it, but even then, there are other complications. When the handler is called, there is no error object, only the error message, file URL, and the line number.

Bad News: IE

Because of various IE bugs, the fileURL and line number passed into the onerror handler isn't always accurate or helpful. After some research, I've isolated these bugs:

In IE 7 and below, if an exception is thrown from within the code of an included script, fileURL will erroneously be set to the URL of the current HTML document even though the lineNumber will be set to the correct line number from within the script.
In IE 8 and below, if an exception is thrown using the throw statement, (i.e. throw new Error() ) and is caught, and then re-thrown, and then propagates all the way up, the line number reported will be the the place it was re-thrown, rather than the place it was first thrown. This bug does not surface when the error is triggered otherwise, such as with type and reference errors.

IE 9 has both of these bugs fixed, even when it emulates older versions of the browser.

What Can You Throw

The "throw" statement lets you throw just about any old value - even numbers and strings! But this is not recommended as you won't be able to get any line numbers or stacktraces if the value thrown isn't an object - except for in IE, where you can get line numbers using onerror. In IE, throwing anything other than a pre-defined error object(Error, TypeError, etc) will result in the loss of a meaningful message in both the caught error object and in the onerror handler - where it will just say "Exception thrown and not caught".

So, the simplest recommendation here is to only throw the pre-defined Error object when you want to explicitly throw an error using the "throw" statement.

throw new Error('Buck stops here.')

or better yet, don't use the throw statement at all(this avoids the second IE bug mentioned earlier).

The DIY Stacktrace

The DIY stacktrace was pioneered by Eric Wendelin among others. It first looks up argument.callee to get the current function being executed, then it walks the call stack by accessing the function's caller attribute recursively. For example:

> function f(){

g()

> function g(){

h()

> function h(){

    console.log(arguments.callee.caller.caller)

> f()

function f(){

g()

In the example, we are able to access the function f from with the body of h's execution by walking two levels up the call stack using the caller attribute.

Pros and Cons of the DIY Stacktrace

The advantage of the DIY stacktrace is that it will work in any browser.

The disadvantages are

It will not get you file locations/line numbers.
It does not work with thrown errors - either implicitly or explicitly - because the building of the stacktrace needs to be explicited done as its own statement, say, printStackTrace, for example. So, you cannot have a stacktrace and an error thrown - you cannot have your cake and eat it too.

Conclusion

Sorry I don't have a better conclusion than: It's a great big mess. However, with the knowledge covered here, I think it is possible to employ some sophisticated hackery to recover at least some stacktrace info for each browser - the hardest case being IE. Such a technique - if made practical - would be useful for unit test frameworks and crash reporting.

References

Eric Wendelin's Javascript Stacktrace ( GitHub repo )
DIY Stacktrace - Helen Emerson
3 painful ways to obtain a stacktrace from Your Sway/CrashKit

Try Traceur

toby ho — Thu, 19 May 2011 15:12:51 +0000

I screwed around with Traceur last night and this morning. Then I built this thing called Try Traceur to make it easy for folk to screw around with it too. Try it! You know you want to.

JSConf Summary and My Codez

toby ho — Fri, 06 May 2011 22:25:03 +0000

JSConf 2011 Portland was truly amazing experience for me, personally. Not only did I meet some fantastic people in the community - some of whom I've long wanted to meet, I also got to show off my stuff, and it didn't totally suck. Also, add to that, Portland is a fantastic city! I want to thank Chris Williams again for motivating me, and everybody involved for such a great event!

My talk was on Tutti and most of its time was occupied by demos, so rather than posting the slides I thought I would just post the example code for the demos. I hope this is helpful to people!

Indent Your Code the Way You Want with Indentinator

toby ho — Sat, 26 Mar 2011 22:58:15 +0000

Indentinator is a script that inspects and changes the indentation in your source code. It is language-agnostic. "Indent your code the way you want."

Install

Assuming you have ruby and rubygem installed, you can

gem install Indentinator

to install it.

Usage

indentinator [-c INDENT_AMOUNT] <file> [<file2>...]

Examples

Using indentinator with a file argument will tell you what amount of indentation the file uses.

$ indentinator lib/myapp.rb 
lib/myapp.rb uses 2 spaces.

Using with the -c option will convert the file to the desired indentation amount, and print the result to stdout.

$ indentinator -c 4 lib/myapp.rb

The -r option will overwrite the original file with the output.

$ indentinator -rc 4 lib/myapp.rb

Make sure you have the file backed up or source controlled first! I am not responsible for any damages.

If you want to read the input from stdin, use the -i option

$ cat lib/myapp.rb | indentinator -ic 4

If you use a mixture of indentation sizes in your source file, indentinator may guess wrong, which will in turn affect the conversion output. To tell it explicitly the original indentation amount for it to use during the conversion process use the -a option

$ indentinator -a 4 -c lib/myapp.rb

Totally Use it Inside TextMate and Stuff

If you copied a snippet of code from the internets, and it uses a different indentation then you, it will be helpful to use indentinator from within TextMate. To do this, follow these steps:

Go to TextMate->Preferences...->Advanced->Shell Variables, and make sure the PATH variable contains the location of the indentinator executable.
Paste the code snippet into TextMate, select that snippet and hit Command-Option-R.
In the Command: text field type in the indentinator command to use, for example indentinator -ic 2, which will take the selected text as input (from stdin) and output it with an indentation of 2 spaces.
Click Execute.

Codez

Source code for Indentinator is distributed on GitHub. Submit me an issue if you have one, thanks ;)

String Difference in Ruby

toby ho — Fri, 25 Mar 2011 23:28:53 +0000

class String

  def -(other)
    self.index(other) == 0 ? self[other.size..self.size] : nil
  end
end

This snippet gives the Ruby String class a minus operator. Which works like

> 'abcde' - 'abc'

=> "de"

> '    ' - '  '
=> "  "

> 'abc' - 'de'
=> nil

Meh.

Smart Parameter Checks in Python

toby ho — Thu, 24 Mar 2011 22:27:29 +0000

While I really enjoy using Michael Foord's Mock library, there is one thing missing and I thought would be a fun project to add - which is the subject of this post.

Fun with Parameters

Python has the most sophisticated function parameter calling mechanics I've ever seen. Given a function f

def f(a, b):

    return a + b

You can either call it like normal(positional parameters)

f(1, 2)

or by keyword parameter

f(a=1, b=2)

With keyword parameters, the position doesn't matter anymore, so the above call is equivalent to

f(b=2, a=1)

You can even mix positional and keyword parameters, as long as you put all the positionals first

f(1, b=2)

So, all the ways I've listed of calling the function f with the parameter a as 1 and b as 2 will have the same functional behavior, given what we know of the signature of the function. So, it would be logical that the mock library would consider

f.assert_called_with(a=1, b=2)

a passing assertion even though in reality it was called as

f(1, 2)

but alas, it does not.

With that, it was my mission to make this happen!

Proof-of-Concept

First I created a prototype/proof-of-concept. See code here. Using

inspect.getargspec(self.func)

I was able to get all the info about the signature of a function I needed. When a function call actually comes in, my callable receives them as

def __call__(self, *args, **kwds):

at which point, I resolve all the parameters with the parameter list in the function signature(I called this "combining" the args)

First, by walking the actual positionals (the args array coming in) and cross referencing them with the ones in the function signature. After this is done, we basically converted the positionals into named form - as a dict.
Then merge everything in kwds into the same dict. So we have one unified dict with all the actually parameters and keyed by parameter name.
If there are any left over from the variable position args, they are stored in an extras list.

Now, with this information handy, we can actually do our own safe parameter checking, and we should! So when a call is made, I also check if all the required params have been provided, whether there are any unexpected params and so forth, and actually raise some TypeErrors like a real function if need be.

But the better part is what comes next - we can now implement the assert_called_with with these new semantics. When a call to assert_called_with is called, I "combine" the args coming into that call, and check if they match the previously made function call. It's that simple!

Working in Python-Mock

Well great. But we still can't use for the Mock library. So I did some more hacking on a branch of Foord's code. Not sure it's the cleanest code, but it now works if you want to try it out

hg clone https://airportyh-python-mock.googlecode.com/hg/ airportyh-python-mock

The way you use it with the Mock library is through mocksignature.

from mock import mocksignature

mocksignature wraps a function that you pass it and uses the signature of that function as a guide. Given the function f that we had previously, you can do

f = mocksignature(f)

now call it

f(1, 2)

and then checking the associated mock with a different call style

f.mock.assert_called_with(a=1, b=2)

should now pass.

Cheers!

Tuttiterm - Tutti for your Terminal

toby ho — Fri, 04 Mar 2011 07:23:16 +0000

I've created Tuttiterm - which allows you to control browsers via Tutti. See the project README. Here's a screenshot: