log.debug( Don )

Testing Javascript for the Browser

Tue, 04 Sep 2012 15:31:00 -0400

I experience a lot of pain trying to test my javascript for use in a browser. Some of this has been relieved by categorizing the flavor of code and testing each flavor differently. I can usually break down browser code flavors into the following:

Synchronous utility code
Interact with the DOM (read, add event listener, fire event, write)
Handle some event
Make some server request

If your code isn’t separating these things, it probably should. This is how I test each of these flavors:

This calls for standard unit testing, test input -> test output. These should make no XHR calls, select arbitrary nodes from the DOM at large, and fire no events unless that is ALL they do and its pretty clear from the function/class name they do so. If your test for this is unmanageable, odds are your function is doing too much.
These are basically view tests. I’d recommend using jsdom to mock the DOM during these test. Firing events should also be part of your integration tests as one event can override another, stop propagation early, or forget to stop event propagation without being caught here.
These are asynchronous tests. Similar to unit testing, you just pass in an event with test args and make sure it calls the right function stubs/events. I like how mocha handles async testing via a timeout. You can also use this timeout to optimize for speed.
Use sinon to stub XHR requests. If you use websockets, stub them as well (I haven’t found a sinon for websockets yet, but it should be easy enough). You can pass in test input, test the message passed to the xhr/websocket stub, and wala! Note: please don’t put logic in XHR callbacks. Instead, please fire and trigger an event. Testing the server response should be part of the server test code. I’d love to share the same request/response data for callback testing and server testing, but I haven’t done this before so I can’t say how it works out.

Internet3

Fri, 17 Aug 2012 14:59:22 -0400

network-accessible storage + ad hoc networks === Internet 3

There aren’t centralized server apps or client apps, but peer apps.
You store your peers data and they store yours.
You communicate directly with devices close to you.
Access to the network is free.
The internet is only used to ‘jump’ across other dense networks.
Most of your stuff is on the peer network.
If you live in a dense place like a city, no need for a cell plan, as you use less battery connecting to a nearby peer device rather than a tower 1/2 mile away.
The network doesn’t go down when the power does.
Some people are gateways to the wider internet and charge you to jump out— but you set your price and if someone else nearby offers it lower you use them instead.
You’ll never pay for a local phone plan again.
You can broadcast messages to just people nearby.
You can download apps from your peers or send them.
You can share music and videos to people next to you without any authorization.
When you are at a crowded event you get better signal, not worse.
You can cache pages looked at while browsing internet on your device so others can access it free or faster (but only if you want to).
Banks issue you codes that are equivalent to cash. You sign it and send it to the receiver with a description of what you are buying. They can’t cash it without verifying their own identities, yours, and the description.
Verifying identity will likely be the biggest industry, and may be municipal or for-profit driven.

It would be an interesting world and it’s possible now.

How HTTP servers/libs could work

Thu, 03 May 2012 16:17:57 -0400

This is more a brainstorming session than anything else. J2EE has been killing me of late and I’ve been trying to explain to coworkers how Node.js/Connect and Ruby/Rails work differently

HTTP Server

You should always be able to access the response and request.
You should always have access to a session identifier if it exists and be able to remove it.
Resource Mapping should consist of mapping urls to resources. A resource should be a static file or a controller. In order to support generic controllers, variables may be extractable from a URL. Static resources should be cached and listened to for changes!
There should be an easy way to listen and post to a web socket in addition to subscribing to open/close events
There should be a way to intercept a static file or controller request

The above rules should be the sole extent of an Http Server.

HTTP Utilities:

There should be an easy way to extract HTML Form data from a request.
There should be an easy way to extract POST data from a request.
There should be an easy way to add cookies to a response and retrieve cookies from a request.
There should be an easy way to urlize and de-urlize a string

Container Utilities:

There should be an easy way to extract JSON and XML from a String
There should be an easy way to stream bits in from a request and out to a response
There should be an easy way to Map a Dictionary of List of String to and from HTML form data
There should be an easy way to associate data with a session, and delete a session

"You always hear people say, “kids don’t like math!” Correction…kids..."

Fri, 06 Apr 2012 13:08:12 -0400

“You always hear people say, “kids don’t like math!” Correction…kids don’t like feeling dumb. People don’t like feeling dumb.”

- Bryan Meyer, http://www.doingmathematics.com/2/post/2012/04/putting-the-cart-before-the-horse.html

Apache Balancer Names are Case Insensitive

Mon, 02 Apr 2012 13:34:00 -0400

In apache 2.2.22,

 # yay 
 <Proxy balancer://yay stickysession=JSESSIONID failonstatus=500,501,502,503>
 BalancerMember http://webqa1.somewhere.com:8400/yay/ retry=10 route=webqa1
 BalancerMember http://webqa2.somewhere.com:8400/yay/ retry=10 route=webqa2
 </Proxy>
 ProxyPass /reslife/ balancer://yay/ 
 # YAY 
 <Proxy balancer://YAY stickysession=JSESSIONID failonstatus=500,501,502,503> 
 BalancerMember http://webqa1.somewhere.com:8400/YAY/ retry=10 route=webqa1 
 BalancerMember http://webqa2.somewhere.com:8400/YAY/ retry=10 route=webqa2 
 </Proxy> 
 ProxyPass /YAY/ balancer://YAY/

Will produce a balancer called “balancer://yay” with 4 workers mapped to both /YAY and /yay. Requests will randomly stick to one or the other. Fun bug/feature of the day.

"If Bitcoin ever went mainstream, Bitcoin’s broadcast-every-transaction-to-everybody..."

Mon, 26 Mar 2012 17:09:30 -0400

“If Bitcoin ever went mainstream, Bitcoin’s broadcast-every-transaction-to-everybody infrastructure may turn out to be a bottleneck.”

- Coding the Wheel, http://www.codingthewheel.com/archives/could-bitcoin-revolutionize-online-poker

"Twitter is often mislabeled as a social network when it’s actually more of a real-time..."

Sun, 25 Mar 2012 14:37:00 -0400

“Twitter is often mislabeled as a social network when it’s actually more of a real-time information network. Yes, people make connections, but they tend to connect based on shared interests and location above existing friendships. You don’t follow your friends from high school, or others with whom you have nothing in common; You follow people who have something to say. […] Twitter is the most expansive, real-time, searchable window to the world today.”

- Mat Honan, The case against Google

“Both of them recognized a cultural wrong, they envisioned...

Sun, 25 Mar 2012 10:23:21 -0400

“Both of them recognized a cultural wrong, they envisioned a world without that wrong, and they dedicated themselves to fighting for a principle.”

"The truth though, is that doing what you’ve been doing is going to get you what you’ve..."

Fri, 23 Mar 2012 15:48:15 -0400

“The truth though, is that doing what you’ve been doing is going to get you what you’ve been getting. If the narrative is getting in the way, if the archetypes you’ve been modeling and the worldview you’ve been nursing no longer match the culture, the economy or your goals, something’s got to give.”

- Seth Godin, http://sethgodin.typepad.com/seths_blog/2012/03/extending-the-narrative.html

OnBeforeUnload Ajax Browser Support

Fri, 23 Mar 2012 12:00:00 -0400

Unsupported

iPad1 Safari - Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3
Opera - Opera/9.80 (Macintosh; Intel Mac OS X 10.7.3; U; en) Presto/2.10.229 Version/11.61
iPad2 Safari - Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B176 Safari/7534.48.3
iPhone Safari - Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_5 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8L1 Safari/6533.18.5
iPhone Opera - Opera/9.80 (iPhone; Opera Mini/6.5.1.23995/27.1366; U; en) Presto/2.8.119 Version/11.10

Supported

Chromium - Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Ubuntu/11.10 Chromium/17.0.963.79 Chrome/17.0.963.79 Safari/535.11
Chrome - Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Firefox 11 - Mozilla/5.0 (Windows NT 6.1; rv:11.0) Gecko/20100101 Firefox/11.0
IE9 - Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
IE7 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3)
OSX 10.7 Safari - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/534.54.16 (KHTML, like Gecko) Version/5.1.4 Safari/534.54.16
Android browser - Mozilla/5.0 (Linux; U; Android 2.3.3; en-us; DROIDX Build/4.5.1_57_DX5-35) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1

Source: https://gist.github.com/2172114

"In this position I saw first hand the disconnect that currently exists between vocational/career..."

Thu, 22 Mar 2012 09:15:49 -0400

“In this position I saw first hand the disconnect that currently exists
between vocational/career tech education and traditional high schools—the
result being students following both paths are basically being cheated of a
well rounded education.”

- Maureen Carruthers on School Factory Boards

"i agree wholeheartedly with more or less everything except your last point. i think the word..."

Tue, 06 Mar 2012 18:37:16 -0500

“i agree wholeheartedly with more or less everything except your last point. i think the word ‘slut’ is always harmful because it’s lazy, and i think lazy language encourages lazy thinking and a lack of empathy, diminishing our ability to recognise others as real human beings. some people do hurtful or manipulative things, but perhaps those people have better selves that they haven’t yet figured out how to be. perhaps they are just vicious on purpose, but that doesn’t occur in a vacuum either.”

- Liz Turner, http://9gag.com/gag/3144092

Software Engineering for English Majors

Thu, 01 Mar 2012 19:27:00 -0500

I have a belief that English majors can make great software engineers. Useful traits of the typical English major that transfer over well to Software Engineering:

Can critically analyze
Know the power and importance of naming
Can spell
Are focused on people
Naturally think in terms of stories
Understand what it means to iteratively improve something
Can communicate intent

I’ve developed a curriculum I am using to teach a couple people now. Its a syntax-last approach. So far the progression is like this:

Organizing a how-to [Putting on shoes example]
Reducing a how-to to functions
Naming functions and stateful definitions
Reusing, replacing, and composing functions (refactoring as well)
Interfaces
Unit Tests
Side effects (methods vs. functions)
Properties and states [Shoes example]
Objects and message passing [List example]
Synchronous vs. asynchronous
Classes as Object templates
Inheritance and Mixins (and its limits) [ToDo List and Book checkout examples]
Exceptions
Mocking
Collections (Bags, Sets, Lists, Maps)
Sorting
Strings
Building programs
Dependency management
Conditionals
Loops
Finite State Machines
Regular Expressions
Stacks and Queues (special lists)
Streams
Packets and Protocols (and how the internet works)
Encryption
Many to Many, One to Many, Many to One, and One to One relationships
Key/Value and Document Databases
Map/Reduce
Singletons
Row/Relational Databases
Locking, Transactions, and Mutability
Design patterns (at least Bridge, Observer, Visitor, Composite, Decorator, Factory)
Model2, MVC, and other Architectural patterns
HTML, CSS, JSON, XML, and other famous acronyms
How to choose a language and framework
Metacircular evaluators and creating your own languages (DSLs)
Working with clients

It seems like a lot, but its really only about 10-15 classes. Many of the examples can just be extended and reused. My girlfriend Alice is my primary guinea pig, but I’m picking up a few more.

To be even more focused than English majors, I’m really trying to gear this towards librarians, many of whom will soon be out of jobs. Let me know if you want a more detailed write-up of my progress and lesson plans— I’ll publish more eventually.

On increasing the quality and quantity of virtual content

Sun, 29 Jan 2012 13:04:00 -0500

Nowadays in the industrialized world, survival is easy. We live in a world where we are paying more and more money for virtual objects rather than physical objects. A high percentage of the objects we buy exist merely to display and interact with these virtual objects.

However, this world is different than the “old” material world. Unlike with material objects, virtual objects can be duplicated without cost. This has led to the rise of the aggregators: big, wonderful companies who aggregate your data, music, information, etc. Aggregators such as Google, Yahoo, Apple make the bulk of the money the internet makes. However, each one is destroying its reason for existence; the way an aggregator makes more money is to reduce the cost of the content it aggregates to as close to zero as possible. Content creators make an ever slimmer slice of the pie. This negatively affects the quality and quantity of the content and eventually the aggregators destroy its market.

Content creators need to fight back. But they can’t in a digital world. Piracy is too easy, unstoppable. Once something is free, it can never be locked down again.

So right now we are turning the virtual world into a police state. A world where every digital transaction is read, analyzed and watched. The fight against piracy will always be there in this world as people copy virtual objects from the regulated world to their walled off gardens. To truly stop piracy, you’d have to give up all privacy, turning this world into a real police state.

I don’t think that’s what I want my future to be.

The only alternative I’ve seen is to change the funding model. You don’t sell instances of IP; you sell its existence. We create a digital world where IP is bought before it’s ever disbursed.

This is a hard world. Someone has to pay for digital content or it will never be created. You fight against the prisoner’s dilema; if I think the idea is good enough to fund, then the bloke next to me will fund it and I don’t need to. Any successful system will have to fight this.

It’s also a tough sell. Content creators are often like gamblers. If they win, they win big. If they lose, as they usually do, they just play again (“bad luck”). More content creators think they will win big than actually will. This new world changes them into salaried or contract workers. If they “win big” they don’t profit as highly. If they “lose”, they might get fired or not be able to pick up another job. There are ways to mitigate this: safety nets or bonus systems. Bonuses are possible: you’d just have to separate evaluation from investment.

In summary, I don’t trust the police state model. I understand the romantic appeal of winning big, but I don’t think its sustainable in this new digital world. If we hope to increase the quality and quantity of digital content, I think the world needs to start paying for it before it’s published.

My Javascript Toolchain

Tue, 17 Jan 2012 21:58:00 -0500

Editor: Textmate

Design Patterns: Underscore, Coffeescript

DOM: jQuery

Dependency Management: Require.js when needed

Testing: Mocha

Coverage: node-jscoverage?

Async: q or Step, haven’t decided yet

AOP: YouAreDaChef

Templates: Jade, Stylus, jQote2

HTTP: Express

MVVM: Knockout/Knockback/Backbone, Derby/Racer

NoSQL Rundown

Mon, 09 Jan 2012 19:04:30 -0500

This is more for my reference later, the MBA version of http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis.

If you want to make a _____ use _______

Wiki, CouchDB

Offline/Online App, CouchDB

Stateful Chat Server, Redis

Twitter clone, MongoDB

Online FPS, Membase

Payment system, Riak

Facebook, Neo4j

Logging, Cassandra

Under

Wed, 04 Jan 2012 09:11:00 -0500

James Hague has a post up at Planet Erlang that describes an keyword called ‘under’ in the programming language J.

F = A under B

is equivalent to

F = function() {
  B.inverse(A(B(this)));
};

Theoretically, this would be very useful for situations in which you do B, then do A, then undo B. This includes adding in frequency space (Fourier transforms), file I/O and database connections (open then close), and parsing (escape some values, do a function, then unescape).

Ruby supports a similar pattern to this by taking blocks (that would be equivalent to A). However, ruby lacks internal knowledge of an inverse function (or even if a function is inversible).

In Javascript, closures can operate to take functions blocks as well. Making a layer over Node.js file I/O that uses closures (rather than a callback) might save a lot of time and suffering in the future (especially since forgetting to undo certain actions is a very common production problem).

However, enough of theory. Lets look at the first example James shows, the magnitude of an array:

mag = sum under square

The definition I used for ‘under’ fits this example pretty well:

Array.prototype.mag = function() {
  square.inverse(sum(square(this)));
};

For this to work in JavaScript though, square() would need to square each entry in the array AND square.inverse() would need to take a scalar. So to be real inverses, they’d need more complex implementations. They could be implemented something like this:

var sum = function(a) {
  return _.reduce(a, function(m, n){ return m + n; });
};
var square = function(a) {
  return _.isArray(a) ? _.map(a, square) : (a * a);
};
square.inverse = function(a) {
  return _.isArray(a) ? _.map(a, sqrt) : Math.sqrt(a);
};

This is rather wordy. Here it is with Coffeescript:

F = (x) ->
  B.inverse A B x

sum = (a) -> _.reduce a, (m,n) -> m+n
square = (a) -> _.isArray(a) ? _.map a, (a) -> a^2 : a^2
square.inverse = (a) -> _.isArray(a) ? _.map a, (a) -> Math.sqrt(a) : Math.sqrt(a)

//define mag the 'hard' way
mag = (x) -> 
  square.inverse sum square x
//define using 'under'
mag = sum under square

So an under keyword in Coffeescript would save exactly 19 characters. I’m not sure this is common enough to warrant a new language feature. However, I do like the idea of explicitly stating that a function is an inverse though (and its quite testable).

Maybe a utility function would be enough:

mag = first(square)
  .andAtTheEnd(Math.sqrt)
  .andInTheMiddle (x) ->
    sum x

This might make more sense with a file I/O example:

translatefile = first(openFile)
  .andAtTheEnd(closeFile)
  .andInTheMiddle (openfile) ->
    translate openfile

Why I Love Aspect Oriented Programming

Tue, 03 Jan 2012 17:13:00 -0500

EDIT: Someone saying the same as me, only better— And he wrote an awesome Coffeescript AOP implementation.

I found AspectJ (through Spring) and immediately fell in love with the idea of Aspect Oriented Programming (AOP).

Say I have this method:

String getDescription(String code) {
  String descr = databaseLookup(code,
      someConstantOrClassVariableKeys);
  //Some cleanup of descr, maybe even appending of (code)
  descr = cleanUpDescr(descr);
  return descr;
}

I want to cache the result of some method. The only problem is that just when my program is in high demand, the client realizes they want to change some description and I can’t restart the app. Usually I’d have to instance and configure a cache somewhere, manage it, and create some hook so that cache can be cleared without restarting the app. If you do a lot of work you can boil this problem down to this:

Cache descrCache = someConfig();
String getDescription(String code) {
  if (descrCache.contains(code, someConstantOrClassVariableKeys))
    return descrCache.get(code, someConstantOrClassVariableKeys)
  String descr = databaseLookup(code,
      someConstantOrClassVariableKeys);
  //Some cleanup of descr, maybe even appending of (code)
  descr = cleanUpDescr(descr);
  if (descr != null)
    descrCache.put(descr, code, someConstantOrClassVariableKeys)
  return descr;
}

OK, I can deal with this a few times.

Now imagine I have to add validation to the method arguments. And security based on the user (another class variable) and some of the arguments. Oh, and I want to cache security. Suddenly, its a nightmare of a function and separating the purpose of the function from the added features is disturbingly hard.

This is when Aspect-oriented programming comes into play. I just wrap the method with some before or around interceptors and life is peachy. My original method code doesn’t change. It might look something like this when completed:

String getDescription(String code) {
  String descr = databaseLookup(code,
      someConstantOrClassVariableKeys);
  //Some cleanup of descr, maybe even appending of (code)
  descr = cleanUpDescr(descr);
  return descr;
}
//cache the description globally by caching the lookup and the cleanup separately
Cache dbCache = new SomeCache();
@aspect databaseLookup(ProxiedMethod m, String code, 
    Object someConstantOrClassVariableKeys) {
  if (dbCache.contains(code, someConstantOrClassVariableKeys))
    return dbCache.get(code, someConstantOrClassVariableKeys);
  String s = m.execute(code, someConstantOrClassVariableKeys);
  dbCache.put(s, code, someConstantOrClassVariableKeys);
  return s;
}
Cache cleanupDescrCache = new SomeCache();
@aspect cleanUpDescr(ProxiedMethod m, String descr) {
  if (cleanupDescrCache.contains(descr))
    return cleanupDescrCache.get(descr);
  String s = m.execute(descr);
  cleanupDescrCache.put(s, descr);
  return s;
}
//add validation
@aspect getDescription(ProxiedMethod m, String code) {
 if (isValid(code))
    return m.execute(code);
  return null;
}
//add security
@aspect getDescription(ProxiedMethod m, String code) {
  boolean isAllowed = doSecurity(code);
  if (isAllowed)
    return m.execute(code);
  throw new SecurityError();
}
//cache the security
Cache securityCache = new SomeCache();
@aspect doSecurity(ProxiedMethod m, String code) {
  if (securityCache.contains(code))
    return securityCache.get(code);
  boolean isAllowed = m.execute(code);
  securityCache.put(code, isAllowed);
  return s;
}

Its no where close to the right syntax and I am missing key implementation details, but the pattern is sound. At this point I could even take the cache logic out into a separate class that takes the ProxiedMethod as an argument and automatically instances a cache.

Doing AOP in javascript is as easy as replacing the method you want to call with the wrapper. Here’s a list of implementations with various syntaxes. I do enjoy the regex passing to wrap multiple functions in one go.

First Impressions of Several Node.js Web Frameworks

Mon, 26 Dec 2011 12:50:00 -0500

Connect - Rack for Node.js. Yay for easy middleware config.

Express - Sinatra is to Rack as Express is to Connect.

Zappa - Conventions for express and socket.io written in Coffeescript.

RailwayJS - Rails for Node without the community. I’d rather be on rails. If you care about async, see async-rails.

SocketStream - Webserver built on connect that wraps websockets quite nicely. If I wanted to have more than one entry point I’d have to do a lot of work (I’d love to have single page interface support or at least something like BBQ baked in). It looks like the initial page load is large and lazy loading is not supported. Great for small to medium single page apps with one entry point.

Derby - Closest thing to an MVVM that lets you ignore client/server lines yet (via racer). Connect-like route setup. Uses mustache-like HTML templates and Stylus for CSS templates. I’ll miss using Jade templates (slated for 0.3 release). To evaluate Derby vs. SocketStream I’d have to read/write a few apps, no opinions yet.

Flatiron - Its hard to talk about this framework from a high level because it has several independent components:

Director is a different approach to routing that more easily supports hierarchal pages.
Union is a middleware layer that replaces connect but still supports connect middleware. I’m not seeing a ton of value in this.
Plates is a templating engine that is very limited since it doesn’t support binding lists to templates (looping) or conditionals.
Resourceful is an object<->document mapper thats implementation-agnostic.
Broadway is a plugin format with an event-based lifecycle.

The code is clean and well-written, but I’m not seeing why I’d use this framework over the others mentioned here. That being said, there are many useful patterns to learn from in the flatiron stack.

Kibi - Cute and small (but not obscuring) client-side web framework. This is a great start for teaching web servers (code is 80 lines or so) with templating, url patterns, and jsonp solely driven by the client-side!

Let me know about any other interesting and/or quality frameworks.

Why I Bothered To Learn Node.js

Mon, 26 Dec 2011 11:02:00 -0500

So I’ve been spending my time lately jumping onto the node.js bandwagon. I did this for three reasons:

I think Javascript is the best language to teach basic programming concepts and be productive as early as possible. Prior to node, this was not the case. Node added dependency management, system level calls, and other programming concepts that made Javascript move out of the DOM-only arena and become a “real” programming language. Since I plan on teaching an intro programming class this Spring, I figured working with some server-side js would make me more qualified.
Coffeescript is a great distillation of js design patterns. As I have no problem with whitespace-dependent syntax, it’s beautiful for writing terse meaningful code (even if it can produce wordier js than handcoding).
I don’t want to write either the communication layer between the server & client or different validators for the server & client. Manually specifying url patterns for ajax services really needs to go out of style. I have not found a Node.js framework that does this for me yet, but I know it is possible— I’m just hoping someone has done (most of) the hard work for me.