This is the 14th in a series of posts leading up to Node.js Knockout on using mongodb with node-mongodb-native This post was written by Node Knockout judge and node-mongo-db-native author Christian Kvalheim.

In the first tutorial we targeted general usage of the database. But Mongo DB is much more than this. One of the additional very useful features is to act as a file storage system. This is accomplish in Mongo by having a file collection and a chunks collection where each document in the chunks collection makes up a Block of the file. In this tutorial we will look at how to use the GridFS functionality and what functions are available.

A simple example

Let’s dive straight into a simple example on how to write a file to the grid using the simplified Grid class.

var mongo = require('mongodb'),
  Db = mongo.Db,
  Grid = mongo.Grid;

Db.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
  if(err) return console.dir(err);

  var grid = new Grid(db, 'fs');    
  var buffer = new Buffer("Hello world");
  grid.put(buffer, {metadata:{category:'text'}, content_type: 'text'}, function(err, fileInfo) {
    if(!err) {
      console.log("Finished writing file to Mongo");
    }
  });
});

All right let’s dissect the example. The first thing you’ll notice is the statement

var grid = new Grid(db, 'fs');

Since GridFS is actually a special structure stored as collections you’ll notice that we are using the db connection that we used in the previous tutorial to operate on collections and documents. The second parameter 'fs' allows you to change the collections you want to store the data in. In this example the collections would be fs_files and fs_chunks.

Having a live grid instance we now go ahead and create some test data stored in a Buffer instance, although you can pass in a string instead. We then write our data to disk.

var buffer = new Buffer("Hello world");
grid.put.(buffer, {metadata:{category:'text'}, content_type: 'text'}, function(err, fileInfo) {
  if(!err) {
    console.log("Finished writing file to Mongo");
  }
});

Let’s deconstruct the call we just made. The put call will write the data you passed in as one or more chunks. The second parameter is a hash of options for the Grid class. In this case we wish to annotate the file we are writing to Mongo DB with some metadata and also specify a content type. Each file entry in GridFS has support for metadata documents which might be very useful if you are for example storing images in you Mongo DB and need to store all the data associated with the image.

One important thing is to take not that the put method return a document containing a _id, this is an ObjectID identifier that you’ll need to use if you wish to retrieve the file contents later.

Right so we have written out first file, let’s look at the other two simple functions supported by the Grid class.

the requires and and other initializing stuff omitted for brevity

var mongo = require('mongodb'),
  Db = mongo.Db,
  Grid = mongo.Grid;

Db.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
  if(err) return console.dir(err);

  var grid = new Grid(db, 'fs');    
  var buffer = new Buffer("Hello world");
  grid.put.(buffer, {metadata:{category:'text'}, content_type: 'text'}, function(err, fileInfo) {        
    grid.get(fileInfo._id, function(err, data) {
      console.log("Retrieved data: " + data.toString());
      grid.delete(fileInfo._id, function(err, result) {
      });        
    });
  });
});

Let’s have a look at the two operations get and delete

grid.get(fileInfo._id, function(err, data) {});

The get method takes an ObjectID as the first argument and as we can se in the code we are using the one provided in fileInfo._id. This will read all the chunks for the file and return it as a Buffer object.

The delete method also takes an ObjectID as the first argument but will delete the file entry and the chunks associated with the file in Mongo.

This api is the simplest one you can use to interact with GridFS but it’s not suitable for all kinds of files. One of it’s main drawbacks is you are trying to write large files to Mongo. This api will require you to read the entire file into memory when writing and reading from Mongo which most likely is not feasible if you have to store large files like Video or RAW Pictures. Luckily this is not the only way to work with GridFS. That’s not to say this api is not useful. If you are storing tons of small files the memory usage vs the simplicity might be a worthwhile tradeoff. Let’s dive into some of the more advanced ways of using GridFS.

Advanced GridFS or how not to run out of memory

As we just said controlling memory consumption for you file writing and reading is key if you want to scale up the application. That means not reading in entire files before either writing or reading from Mongo DB. The good news it’s supported. Let’s throw some code out there straight away and look at how to do chunk sized streaming writes and reads.

the requires and and other initializing stuff omitted for brevity

var fileId = new ObjectID();
var gridStore = new GridStore(db, fileId, "w", {root:'fs'});
gridStore.chunkSize = 1024 * 256;

gridStore.open(function(err, gridStore) {
 Step(
   function writeData() {
     var group = this.group();

     for(var i = 0; i < 1000000; i += 5000) {
       gridStore.write(new Buffer(5000), group());
     }
   },

   function doneWithWrite() {
     gridStore.close(function(err, result) {
       console.log("File has been written to GridFS");
     });
   }
 )
});

Before we jump into picking apart the code let’s look at

var gridStore = new GridStore(db, fileId, "w", {root:'fs'});

Notice the parameter "w" this is important. It tells the driver that you are planning to write a new file. The parameters you can use here are.

  • "r" - read only. This is the default mode
  • "w" - write in truncate mode. Existing data will be overwritten
  • "w+" - write in edit mode

Right so there is a fair bit to digest here. We are simulating writing a file that’s about 1MB big to Mongo DB using GridFS. To do this we are writing it in chunks of 5000 bytes. So to not live with a difficult callback setup we are using the Step library with its’ group functionality to ensure that we are notified when all of the writes are done. After all the writes are done Step will invoke the next function (or step) called doneWithWrite where we finish up by closing the file that flushes out any remaining data to Mongo DB and updates the file document.

As we are doing it in chunks of 5000 bytes we will notice that memory consumption is low. This is the trick to write large files to GridFS. In pieces. Also notice this line.

gridStore.chunkSize = 1024 * 256;

This allows you to adjust how big the chunks are in bytes that Mongo DB will write. You can tune the Chunk Size to your needs. If you need to write large files to GridFS it might be worthwhile to trade of memory for CPU by setting a larger Chunk Size.

Now let’s see how the actual streaming read works.

var gridStore = new GridStore(db, fileId, "r");
gridStore.open(function(err, gridStore) {
  var stream = gridStore.stream(true);

  stream.on("data", function(chunk) {
    console.log("Chunk of file data");
  });

  stream.on("end", function() {
    console.log("EOF of file");
  });

  stream.on("close", function() {
    console.log("Finished reading the file");
  });
});

Right let’s have a quick lock at the streaming functionality supplied with the driver (make sure you are using 0.9.6-12 or higher as there is a bug fix for custom chunksizes that you need)

var stream = gridStore.stream(true);

This opens a stream to our file, you can pass in a boolean parameter to tell the driver to close the file automatically when it reaches the end. This will fire the close event automatically. Otherwise you’ll have to handle cleanup when you receive the end event. Let’s have a look at the events supported.

  stream.on("data", function(chunk) {
    console.log("Chunk of file data");
  });

The data event is called for each chunk read. This means that it’s by the chunk size of the written file. So if you file is 1MB big and the file has chunkSize 256K then you’ll get 4 calls to the event handler for data. The chunk returned is a Buffer object.

  stream.on("end", function() {
    console.log("EOF of file");
  });

The end event is called when the driver reaches the end of data for the file.

  stream.on("close", function() {
    console.log("Finished reading the file");
  });

The close event is only called if you the autoclose parameter on the gridStore.stream method as shown above. If it’s false or not set handle cleanup of the streaming in the end event handler.

Right that’s it for writing to GridFS in an efficient Manner. I’ll outline some other useful function on the Gridstore object.

Other useful methods on the Gridstore object

There are some other methods that are useful

gridStore.writeFile(filename/filedescriptor, function(err fileInfo) {});

writeFile takes either a file name or a file descriptor and writes it to GridFS. It does this in chunks to ensure the Eventloop is not tied up.

gridStore.read(length, function(err, data) {});

read/readBuffer lets you read a #length number of bytes from the current position in the file.

gridStore.seek(position, seekLocation, function(err, gridStore) {});

seek lets you navigate the file to read from different positions inside the chunks. The seekLocation allows you to specify how to seek. It can be one of three values.

  • GridStore.IO_SEEK_SET Seek mode where the given length is absolute
  • GridStore.IO_SEEK_CUR Seek mode where the given length is an offset to the current read/write head
  • GridStore.IO_SEEK_END Seek mode where the given length is an offset to the end of the file

    GridStore.list(dbInstance, collectionName, {id:true}, function(err, files) {})

list lists all the files in the collection in GridFS. If you have a lot of files the current version will not work very well as it’s getting all files into memory first. You can have it return either the filenames or the ids for the files using option.

gridStore.unlink(function(err, result) {});

unlink deletes the file from Mongo DB, that’s to say all the file info and all the chunks.

This should be plenty to get you on your way building your first GridFS based application. As in the previous article the following links might be useful for you. Good luck and have fun.

Links and stuff

This is the 13th in a series of posts leading up to Node.js Knockout on using mongodb with node-mongodb-native This post was written by Node Knockout judge and node-mongo-db-native author Christian Kvalheim.

Mongo DB has rapidly grown to become a popular database for web applications and is a perfect fit for Node.JS applications, letting you write Javascript for the client, backend and database layer. Its schemaless nature is a better match to our constantly evolving data structures in web applications and the integrated support for location queries is a bonus that’s hard to ignore. Throw in replicasets for scaling and we are looking at really nice platform to grow your storage needs now and in the future.

Now to shamelessly plug my driver. It can be downloaded either using npm or fetched from the github repository. To install via npm do the following:

npm install mongodb

or go fetch it from github at https://github.com/christkv/node-mongodb-native

Once this business is taken care of, let’s move through the types available for the driver and then how to connect to your Mongo DB instance before facing the usage of some crud operations.

Mongo DB data types

So there is an important thing to keep in mind when working with Mongo DB and that is that there is a slight mapping difference between the types supported in Mongo DB and the native types in Javascript. Let’s have a look at the types supported out of the box and then how types are promoted by the driver to try to fit the native Javascript types as closely possible.

  • Float is a 8 byte and is directly convertible to the Javascript type Number
  • Double class a special class representing a float value, this is especially useful when using capped collections where you need to ensure your values are always floats.
  • Integers is a bit trickier due to the fact that Javascript represents all Numbers as 64 bit floats meaning that the maximum integer value is at a 53 bit. Mongo has two types for integers, a 32 bit and a 64 bit. The driver will try to fit the value into 32 bits if it can and promote it to 64 bits if it has to. Similarly it will deserialize attempting to fit it into 53 bits if it can. If it cannot it will return an instance of Long to avoid losing precession.
  • Long class a special class that lets you store 64 bit integers and also lets you operate on the 64 bits integers.
  • Date maps directly to a Javascript Date
  • RegEp maps directly to a Javascript RegExp
  • String maps directly to a Javascript String (encoded in utf8)
  • Binary class a special class that lets you store data in Mongo DB
  • Code class a special class that lets you store javascript functions in Mongo DB, can also provide a scope to run the method in
  • ObjectID class a special class that holds a MongoDB document identifier (the equivalent to a Primary key)
  • DbRef class a special class that lets you include a reference in a document pointing to another object
  • Symbol class a special class that lets you specify a symbol, not really relevant for javascript but for languages that supports the concept of symbols.

As we see the number type can be a little tricky due to the way integers are implemented in Javascript. The latest driver will do correct conversion up to 53 bits of complexity. If you need to handle big integers the recommendation is to use the Long class to operate on the numbers.

Getting that connection to the database

Let’s get around to setting up a connection with the Mongo DB database. Jumping straight into the code, let’s do a direct connection and then look at the code.

var mongo = require('mongodb'),
  Server = mongo.Server,
  Db = mongo.Db;

Db.connect("mongodb://localhost:27017/exampleDb", {auto_reconnect:true}, function(err, db) {
  if(!err) {
    console.log("We are connected");
  }
});

Let’s have a quick look at how the connection code works. The Db.connect method let’s use use a uri to connect to the Mongo database, where localhost:27017 is the server host and port and exampleDb the db we wish to connect to. After the url notice the hash containing the auto_reconnect key. Auto reconnect tells the driver to retry sending a command to the server if there is a failure during it’s execution.

Another useful option you can pass in is

poolSize, this allows you to control how many tcp connections are opened in parallel. The default value for this is 5 but you can set it as high as you want. The driver will use a round-robin strategy to dispatch and read from the tcp connection.

We are up and running with a connection to the database. Let’s move on and look at what collections are and how they work.

Mongo DB and Collections

Collections are the equivalent of tables in traditional databases and contain all your documents. A database can have many collections. So how do we go about defining and using collections? Well, there are a couple of methods that we can use. Let’s jump straight into code and then look at the code.

the requires and and other initializing stuff omitted for brevity

db.open(function(err, db) {
  if(err) { return console.dir(err); }

  db.collection('test', function(err, collection) {});

  db.collection('test', {w:1}, function(err, collection) {});

  db.createCollection('test', function(err, collection) {});

  db.createCollection('test', {w:1}, function(err, collection) {});

});

Three different ways of creating a collection object but slightly different in behavior. Let’s go through them and see what they do

db.collection('test', function(err, collection) {});

This function will not actually create a collection on the database until you actually insert the first document.

db.collection('test', {w:1}, function(err, collection) {});

Notice the {w:1} option. This option will make the driver check if the collection exists and issue an error if it does not.

db.createCollection('test', function(err, collection) {});

This command will create the collection on the Mongo DB database before returning the collection object. If the collection already exists it will ignore the creation of the collection.

db.createCollection('test', {w:1}, function(err, collection) {});

The {w:1} option will make the method return an error if the collection already exists.

With an open db connection and a collection defined we are ready to do some CRUD operation on the data.

And then there was CRUD

So let’s get dirty with the basic operations for Mongo DB. The Mongo DB wire protocol is built around 4 main operations insert/update/remove/query. Most operations on the database are actually queries with special json objects defining the operation on the database. But I’m getting ahead of myself. Let’s go back and look at insert first and do it with some code.

the requires and and other initializing stuff omitted for brevity

db.open(function(err, db) {
  if(err) { return console.dir(err); }

  var collection = db.collection('test');
  var doc1 = {'hello':'doc1'};
  var doc2 = {'hello':'doc2'};
  var lotsOfDocs = [{'hello':'doc3'}, {'hello':'doc4'}];

  collection.insert(doc1);

  collection.insert(doc2, {w:1}, function(err, result) {});

  collection.insert(lotsOfDocs, {w:1}, function(err, result) {});

});

A couple of variations on the theme of inserting a document as we can see. To understand why it’s important to understand how Mongo DB works during inserts of documents.

Mongo DB has asynchronous insert/update/remove operations. This means that when you issue an insert operation it’s a fire and forget operation where the database does not reply with the status of the insert operation. To retrieve the status of the operation you have to issue a query to retrieve the last error status of the connection. To make it simpler to the developer, the driver implements the {w:1} options so that this is done automatically when inserting the document. {w:1} becomes especially important when you do update or remove as otherwise it’s not possible to determine the number of documents modified or removed.

Now let’s go through the different types of inserts shown in the code above.

collection.insert(doc1);

Taking advantage of the async behavior and not needing confirmation about the persisting of the data to Mongo DB we just fire off the insert (we are doing live analytics, losing a couple of records does not matter).

collection.insert(doc2, {w:1}, function(err, result) {});

That document needs to stick. Using the {w:1} option ensure you get the error back if the document fails to insert correctly.

collection.insert(lotsOfDocs, {w:1}, function(err, result) {});

A batch insert of document with any errors being reported. This is much more efficient if you need to insert large batches of documents as you incur a lot less overhead.

Right, that’s the basics of inserts ironed out. We got some documents in there but want to update them as we need to change the content of a field. Let’s have a look at a simple example and then we will dive into how Mongo DB updates work and how to do them efficiently.

the requires and and other initializing stuff omitted for brevity

db.open(function(err, db) {
  if(err) { return console.dir(err); }

  var collection = db.collection('test');
  var doc = {mykey:1, fieldtoupdate:1};

  collection.insert(doc, {w:1}, function(err, result) {
    collection.update({mykey:1}, {$set:{fieldtoupdate:2}}, {w:1}, function(err, result) {});
  });

  var doc2 = {mykey:2, docs:[{doc1:1}]};

  collection.insert(doc2, {w:1}, function(err, result) {
    collection.update({mykey:2}, {$push:{docs:{doc2:1}}, {w:1}, function(err, result) {});
  });
});

Alright before we look at the code we want to understand how document updates work and how to do the efficiently. The most basic and less efficient way is to replace the whole document, this is not really the way to go if you want to change just a field in your document. Luckily Mongo DB provides a whole set of operations that let you modify just pieces of the document Atomic operations documentation. Basically outlined below.

  • $inc - increment a particular value by a certain amount
  • $set - set a particular value
  • $unset - delete a particular field (v1.3+)
  • $push - append a value to an array
  • $pushAll - append several values to an array
  • $addToSet - adds value to the array only if its not in the array already
  • $pop - removes the last element in an array
  • $pull - remove a value(s) from an existing array
  • $pullAll - remove several value(s) from an existing array
  • $rename - renames the field
  • $bit - bitwise operations

Now that the operations are outline let’s dig into the specific cases show in the code example.

collection.update({mykey:1}, {$set:{fieldtoupdate:2}}, {w:1}, function(err, result) {});

Right, so this update will look for the document that has a field mykey equal to 1 and apply an update to the field fieldtoupdate setting the value to 2. Since we are using the {w:1} option the result parameter in the callback will return the value 1 indicating that 1 document was modified by the update statement.

collection.update({mykey:2}, {$push:{docs:{doc2:1}}, {w:1}, function(err, result) {});

This updates adds another document to the field docs in the document identified by {mykey:2} using the atomic operation $push. This allows you to modify keep such structures as queues in Mongo DB.

Let’s have a look at the remove operation for the driver. As before let’s start with a piece of code.

the requires and and other initializing stuff omitted for brevity

db.open(function(err, db) {
  if(err) { return console.dir(err); }

  var collection = db.collection('test');
  var docs = [{mykey:1}, {mykey:2}, {mykey:3}];

  collection.insert(docs, {w:1}, function(err, result) {

    collection.remove({mykey:1});

    collection.remove({mykey:2}, {w:1}, function(err, result) {});

    collection.remove();
  });
});

Let’s examine the 3 remove variants and what they do.

collection.remove({mykey:1});

This leverages the fact that Mongo DB is asynchronous and that it does not return a result for insert/update/remove to allow for synchronous style execution. This particular remove query will remove the document where mykey equals 1.

collection.remove({mykey:2}, {w:1}, function(err, result) {});

This remove statement removes the document where mykey equals 2 but since we are using {w:1} it will back to Mongo DB to get the status of the remove operation and return the number of documents removed in the result variable.

collection.remove();

This last one will remove all documents in the collection.

Time to Query

Queries are of course a fundamental part of interacting with a database and Mongo DB is no exception. Fortunately for us it has a rich query interface with cursors and close to SQL concepts for slicing and dicing your datasets. To build queries we have lots of operators to choose from Mongo DB advanced queries. There are literarily tons of ways to search and ways to limit the query. Let’s look at some simple code for dealing with queries in different ways.

the requires and and other initializing stuff omitted for brevity

db.open(function(err, db) {
  if(err) { return console.dir(err); }

  var collection = db.collection('test');
  var docs = [{mykey:1}, {mykey:2}, {mykey:3}];

  collection.insert(docs, {w:1}, function(err, result) {

    collection.find().toArray(function(err, items) {});

    var stream = collection.find({mykey:{$ne:2}}).streamRecords();
    stream.on("data", function(item) {});
    stream.on("end", function() {});

    collection.findOne({mykey:1}, function(err, item) {});

  });
});

Before we start picking apart the code there is one thing that needs to be understood, the find method does not execute the actual query. It builds an instance of Cursor that you then use to retrieve the data. This lets you manage how you retrieve the data from Mongo DB and keeps state about your current Cursor state on Mongo DB. Now let’s pick apart the queries we have here and look at what they do.

collection.find().toArray(function(err, items) {});

This query will fetch all the document in the collection and return them as an array of items. Be careful with the function toArray as it might cause a lot of memory usage as it will instantiate all the document into memory before returning the final array of items. If you have a big resultset you could run into memory issues.

var stream = collection.find({mykey:{$ne:2}}).stream();
stream.on("data", function(item) {});
stream.on("end", function() {});

This is the preferred way if you have to retrieve a lot of data for streaming, as data is deserialized a data event is emitted. This keeps the resident memory usage low as the documents are streamed to you. Very useful if you are pushing documents out via websockets or some other streaming socket protocol. Once there is no more document the driver will emit the end event to notify the application that it’s done.

collection.findOne({mykey:1}, function(err, item) {});

This is special supported function to retrieve just one specific document bypassing the need for a cursor object.

That’s pretty much it for the quick intro on how to use the database. I have also included a list of links to where to go to find more information and also a sample crude location application I wrote using express JS and mongo DB.

Links and stuff

This is the 11th in a series of posts leading up to Node.js Knockout on debugging node processes using Bonsai.js. This post was written by Bonsai.js contributor Dustan Kasten of Skookum Digital Works.

Bonsai.js is a new graphics library that, unlike most graphic libraries currently in the wild, ships with an SVG renderer. Not only that, but it has an architecturally separate runner and renderer so that all the heavy lifting can happen in an iFrame, WebWorker, or node context.

Built by the amazing team at Uxebu, Bonsai is the HTML5 library that Pixelplant depends on for its Flash conversion. If you are familiar with Flash development or terminology you already have a grasp of how to create with bonsai.

Note: All of this code is available at https://github.com/iamdustan/bonsai-demos. To view demos, git clone git@github.com:iamdustan/bonsai-demos && npm start then browse to http://localhost:8080 in your favorite browser.

From Tree Level

A few things to note as you walk into the bonsai forest. Due to the architecturally separated runner and renderer, all of the code that draws to the bonsai stage is run in a sandbox. In this sandbox you have access to a large number of tools that bonsai creates for you (many examples listed below under Bonsai Tools).

The bonsai world revolves around the stage. The runner and renderer communicate through passing messages back and forth. This is for both user triggered actions (e.g. pointer events and clicks) or for things you want to trigger (say, pressing a Start button).

Bonsai Tools

These are just a few of the objects bonsai makes available to you in the runner context.

Simple shapes

  • Rect
  • Arc
  • Circle
  • Ellipse
  • Polgyon
  • Star

Assets

  • Audio
  • Bitmap
  • FontFamily
  • Movie
  • Video

And more…

Let The Fun Begin

Getting started with bonsai always seems to be the biggest hurdle. The documentation has recently been super-sized with an explanation of the execution environment that is required reading: Bonsai execution.

Through the following examples, you will have everything you need to know to get started making awesome.

Grab the Latest Bonsai

Releases happen very regularly with new features added and bugs smashed. The latest official release is always available on github or from the cdn. Additionally, if you are like me and feeling a bit edgier you could just build the latest copy from master yourself (requires java to run Closure Compiler):

cd /tmp; git clone git@github.com:Uxebu/bonsai && make build
cp ./dist/bonsai.js YOUR_DIRECTORY_HERE

A Simple Example

To start things off we’re going to do the smallest amount of boilerplate necessary and demonstrate a few of the methods and sugar that Bonsai provides.

// pass a function through as the code parameter

var element = document.getElementById('movie');
bonsai.run(movie, {
  code: runner,
  width: 600,
  height: 400
});

function runner () {
  var rect = new Rect(0, 0, 200, 200);
  rect
    .fill('random')
    .addTo(stage)
    .attr({
      x: stage.width - rect.attr('width'),
      y: stage.width - rect.attr('height')
    })
    .animate('0.5s', {
      x: 0,
      y: 0
    });
}

As you can see, our instance of a Rect has some helpful methods accompanying it, and don’t forget the lovely color, random. All displayable objects are an instance of a DisplayObject, aptly title.

Clearly, this method will not scale to complex code, so let’s break that off.

Separating Things Out

Let’s begin by moving that runner method into a file to call its own, following the Flash semantics, movie.js.

// movie.js
var rect = new Rect(0, 0, 200, 200);
rect
  .fill('random')
  .addTo(stage)
  .attr({
    x: stage.width - rect.attr('width'),
    y: stage.width - rect.attr('height')
  })
  .animate('0.5s', {
    x: 0,
    y: 0
  });


// app.js
var element = document.getElementById('movie');
bonsai.run(movie, {
  url: 'path/to/movie.js',
  plugins: [],
  width: 600,
  height: 400
});

Well, that was easy enough. Bonsai here is taking the file you specify under the url key and loading into the appropriate runner context.

Don’t Forget to Listen to Your Users

Let’s continue adding complexity and add a second file to handle user interactions. Bonsai pipes all user interaction into the runner context. This is imperative since the runner has no concept of things like pointer events or event.pageX

// ui.js
stage.on('message', handleMessage)
stage.on('pointerdown', handlePointerdown)
stage.on('keypress', handleKeypress)

function handleMessage(message) {
  if (message.type === 'Rect' && message.attr)
    new Rect(message.attr.x, message.attr.y, message.attr.w, message.attr.h)
      .attr(message.attr)
      .fill(message.attr.fill || 'random')
      .addTo(stage);
}

function handlePointerdown (e) {
  handleMessage({
    type: 'Rect',
    attr: {
      x: e.stageX-25,
      y: e.stageY-25,
      w: 50,
      h: 50
    }
  })
}

function handleKeypress (e) {
  stage.sendMessage('keypress', e.keyCode);
}


// app.js
var element = document.getElementById('movie');
var stage = bonsai.run(movie, {
  url: 'path/to/movie.js',
  plugins: ['path/to/ui.js'],
  width: 600,
  height: 400
});

stage.sendMessage({
  type: 'Rect',
  attr: {
    x: Math.random() * 100, y: Math.random() * 100,
    w: Math.random() * 100, h: Math.random() * 100
  }
});

stage.on('message:keypress', function (data) {
  console.log('Hey! Someone touched me at {keyCode}! - the
Keyboard'.replace(/{keyCode}/g, data));
});

But This Is Node Knockout

Oh right. I almost forgot. We’ve just been letting bonsai manage setting up the runner context in a WebWorker or iFrame so far. We need a way to run this on the server and connect our thousand friends to it. Very well, let’s get to it!

Up to this point we’ve been starting bonsai and passing the configuration object all at the same time. We will use a slightly different version this time where we first call setup passing in a Socket.io runner context.

Note: This demo uses a currently custom build of Bonsai that exposes a few internal utilities. Grab it here: https://github.com/uxebu/bonsai-server/blob/master/example/bonsai.js

// main.js

var movie = document.getElementById('movie')
var runnerContext = function (runnerUrl) {
  this.socket = io.connect(runnerUrl);
};

// some boilerplate to connext via socket.io
var proto = runnerContext.prototype = bonsai.tools.mixin({
  init: function () {
    var self = this;
    this.socket.on('message', function(msg) {
      self.emit('message', msg[0]);
    });
  },
  notify: function (message) {
    this.socket.emit('message', message);
  },
  notifyRunner: function (message) {
    this.socket.emit('message', message);
  },
  run: function (code) {
    this.notifyRunner({
      command: 'runScript',
      code: code
    });
  }
}, bonsai.EventEmitter);

proto.notifyRunnerAsync = proto.notifyRunner;

bonsai
  .setup({
    runnerContext: runnerContext,
    runnerUrl: 'http://localhost:3000
  })
  .run(movie, {
    width: 600,
    height: 600
  });


// movie.js
// this is read by and run on the server
// demo from http://demos.bonsaijs.org/demos/circles/index.html
var centerX = 250,
    centerY = 250,
    circles = 180,
    distance = 180,
    frames = 14,
    radiusMin = 10,
    radiusVar = 10;

var circle, random = Math.random;

for (var i = 0; i < circles; ++i) {
    var f = i / circles,
        x = centerX + distance * Math.sin(f*2*Math.PI),
        y = centerY + distance * -Math.cos(f*2*Math.PI),
        radius = random() * radiusVar + radiusMin;

    circle = new Circle(x, y, radius).
      attr({fillColor: 'random', fillGradient: bonsai.gradient.radial(['#FFFFFF88', '#FFFFFF00'])});

    circle.x = x;
    circle.y = y;

    stage.addChild(circle);
}

var c = stage.children();
stage.length(frames);
var spread = 80;
stage.on(0, function() {
  for (var i = 0, circle; (circle = c[i++]); ) {
    circle.animate(frames, {
      x: circle.x + spread * random() - spread / 2,
      y: circle.y + spread * random() - spread / 2
    }, {easing: 'sineInOut'});
  }
});


// server.js
var bonsai = require('bonsai');
var fs = require('fs');

var bonsaiCode = fs.readFileSync('./movie.js');
var socketRenderer = function (socket) {
  this.socket = socket;
};

var socket = require('socket.io').listen(4000);

socket.sockets.on('connection', function (socket) {
  var movie = bonsai.run(null, {
    code: bonsaiCode,
    plugins: []
  });

  movie.runnerContext.on('message', function () {
    socket.emit('message', arguments);
  });

  movie.on('message', function (msg) {
    movie.runnerContext.notifyRunner(msg);
  });

  socket.on('disconnect', function () {
    movie.destroy();
  });

});


// package.json
{
  name: "roger-rabbit",
  version: "0.0.0",
  main: "server.js",
  dependencies: {
    "bonsai": "git+ssh://git@github.com:uxebu/bonsai.git",
    "socket.io": "~0.9.10"
  }
}

Here we are using the node modules socket.io and bonsai, which we’re grabbing straight from the source. (Note: npm install bonsai will return a different module.) Socket.io is responsible for keeping the connection live and then just being the telephone wire transfering messages from the runner to the renderers.

Additional Resources

Bonsai is a young, but active project and community. Learn more, get involved, and stay connected.

I look forward to seeing what you create with Bonsai!

Teleportd is providing contestants unlimited access to their photo api for the competition. Click the link above to see how it works.

This is the 10th in a series of posts leading up to Node.js Knockout on debugging node processes using Node Inspector. This post was written by Node Knockout judge and Node Inspector author Danny Coates.

While node has a built in debugger, Node Inspector provides a pleasant graphical interface for debugging node programs.

Node Inspector is a debugger interface for node.js using the WebKit Web Inspector, the familiar javascript debugger from Safari and Chrome.

Install

With npm:

npm install -g node-inspector

Enable debug mode

To use node-inspector, enable debugging on the node you wish to debug. You can either start node with a debug flag like:

$ node --debug your/node/program.js

or, to pause your script on the first line:

$ node --debug-brk your/short/node/script.js

NOTE: make sure that the --debug flag comes before your/node/program.js or else you may see an EADDRINUSE error.

Or you can enable debugging on a node that is already running by sending it a signal:

  1. Get the PID of the node process using your favorite method. pgrep or ps -ef are good

    $ pgrep -l node
    2345 node your/node/server.js

  2. Send it the USR1 signal

    $ kill -s USR1 2345

Great! Now you’re ready to attach node-inspector.

Debugging

  1. start the inspector. I usually put it in the background

    $ node-inspector &

  2. open http://127.0.0.1:8080/debug?port=5858 in your favorite WebKit based browser

  3. you should now see the javascript source from node. If you don’t, click the scripts tab.

  4. select a script and set some breakpoints (far left line numbers) or simply add a debugger call in your code (node will break automatically on the call, just as V8 does).

  5. then watch the slightly outdated but hilarious screencasts

node-inspector works almost exactly like the web inspector in Safari and Chrome. Here’s a good overview of the UI.

FAQ

  1. I don’t see one of my script files in the file list.

    try refreshing the browser (F5 or ⌘-r or control-r)

  2. My script runs too fast to attach the debugger.

    add a debugger statement in your code where you want the debugger to stop

    or use --debug-brk to pause the script on the first line

  3. Can I debug remotely?

    Yes. node-inspector needs to run on the same machine as the node process, but your browser can be anywhere. Just make sure the firewall is open on 8080

  4. I got the ui in a weird state.

    when in doubt, refresh

This is the 8th in a series of posts leading up to Node.js Knockout on using restify. This post was written by Node Knockout judge and restify creator Mark Cavage.

restify is a node.js module purpose built to create REST web services in Node. restify makes lots of the hard problems of building such a service, like versioning, error handling and content-negotiation easier. It also provides built in DTrace probes that you get for free to quickly find out where your application’s performance problems lie. Lastly, it provides a robust client API that handles retry/backoff for you on failed connections, along with some other niceties.

Installation

Like everything else in node, install restify through npm.

npm install restify

The current stable version stream is 1.4.x.

Writing an Application

restify, on the surface, looks very similar to express, which itself looks similar to sinatra, which looks similar to J2EE’s Intercepting Filter, which I’m sure looks similar to something that came before it. Basically, you define your app by mapping URLs to a list of functions. For example, here’s an “echo” server in restify:

var restify = require('restify');

var server = restify.createServer();

server.get('/echo/:name', function (req, res, next) {
  res.send({name: req.params.name});
  next();
});

server.listen(8080, function () {
  console.log('%s listening at %s', server.name, server.url);
});

Fire up that server with node:

$ node echo.js
restify listening at http://0.0.0.0:8080

And then we can call it with curl (note these examples also assume you’ve installed Trent Mick’s jsontool, which is oh so awesome for working with JSON REST services):

$ curl -isS localhost:8080/echo/mark | json
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 15
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
Access-Control-Allow-Methods: GET
Access-Control-Expose-Headers: X-Api-Version, X-Request-Id, X-Response-Time
Connection: Keep-Alive
Content-MD5: YWrpZVss8txd0nkH4yISjA==
Date: Wed, 31 Oct 2012 03:00:37 GMT
Server: restify
X-Request-Id: 223ca389-5ee4-4376-89ea-c8a13d0af033
X-Response-Time: 0

{
  "name": "mark"
}

Cool. Note above we had a URL parameter (mark), and we just sent it back. By default, restiy assumes JSON services generally (you can override this to anything, but for NKO you probably want to stick with JSON).

Writing a “full” restify server

Beyond “hello world”, let’s take a look at a more “complete” echo server. Note here I’m also including bunyan to show how you can set up both debug logging and audit logging:

var bunyan = require('bunyan');
var restify = require('restify');

var log = bunyan.createLogger({
  name: 'my_restify_application',
  level: process.env.LOG_LEVEL || 'info',
  stream: process.stdout,
  serializers: bunyan.stdSerializers
});

var server = restify.createServer({
  log: log,
  name: 'my_restify_application'
});

server.use(restify.acceptParser(server.acceptable));
server.use(restify.authorizationParser());
server.use(restify.dateParser());
server.use(restify.queryParser());
server.use(restify.bodyParser());
server.use(restify.throttle({
  burst: 100,
  rate: 50,
  ip: true, // throttle based on source ip address
  overrides: {
    '127.0.0.1': {
      rate: 0, // unlimited
      burst: 0
    }
  }
}));
server.on('after', restify.auditLogger({ log: log }));

server.use(function authenticate(req, res, next) {
  // call redis or something here
  next();
});

// this one will be explained in the next section
server.use(function slowPoke(req, res, next) {
  setTimeout(next.bind(this), parseInt((process.env.SLEEP_TIME || 0), 10));
});

server.post('/echo/:name', function echoParms(req, res, next) {
  req.log.debug(req.params, 'echoParams: sending back all parameters');
  res.send(req.params);
  next();
});

server.listen(8080, function () {
  log.info('%s listening at %s', server.name, server.url);
});

Let’s fire this one up, and set the log level to debug (also, pipe the output to the bunyan formatter so we get “pretty printed” logs):

$ npm install bunyan
$ LOG_LEVEL=debug node echo.js | ./node_modules/.bin/bunyan
my_restify_application listening at http://0.0.0.0:8080

Now go ahead and invoke curl again (note this time we changed the method to a POST so we can send some data over):

$ curl -isS localhost:8080/echo/mark?q1=foo\&q2=bar \
       -X POST -H content-type:application/json \
       --data-binary '{"body_param": "baz"}' | json
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 56
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
Access-Control-Allow-Methods: POST
Access-Control-Expose-Headers: X-Api-Version, X-Request-Id, X-Response-Time
Connection: Keep-Alive
Content-MD5: J1dujES7TARwUfsRQNMDhw==
Date: Wed, 31 Oct 2012 03:17:23 GMT
Server: my_restify_application
X-Request-Id: 3a8404f5-c5be-45d0-a4be-38f17c9e60d1
X-Response-Time: 1

{
    "name": "mark",
    "q1": "foo",
    "q2": "bar",
    "body_param": "baz"
}

This time, we should also have seen the following information kick back from the server console:

[2012-10-31T03:17:23.853Z] DEBUG: mark/51296 on bluesnoop: echoParams: sending back all parameters (req_id=3a8404f5-c5be-45d0-a4be-38f17c9e60d1, route="POST /echo/:name", q1=foo, q2=bar, body_param=baz)
[2012-10-31T03:17:23.856Z]  INFO: my_restify_application/51296 on bluesnoop: POST /echo/:name handled: 200 (req_id=3a8404f5-c5be-45d0-a4be-38f17c9e60d1, 4ms, audit=true, remoteAddress=127.0.0.1, remotePort=50996, secure=false, _audit=true, req.version=*)
    POST /echo/mark?q1=foo&q2=bar HTTP/1.1
    user-agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
    host: localhost:8080
    accept: */*
    content-type: application/json
    content-length: 21
    --
    HTTP/1.1 200 OK
    content-type: application/json
    content-length: 56
    access-control-allow-origin: *
    access-control-allow-headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
    access-control-allow-methods: POST
    access-control-expose-headers: X-Api-Version, X-Request-Id, X-Response-Time
    connection: Keep-Alive
    content-md5: J1dujES7TARwUfsRQNMDhw==
    date: Wed, 31 Oct 2012 03:17:23 GMT
    server: my_restify_application
    x-request-id: 3a8404f5-c5be-45d0-a4be-38f17c9e60d1
    x-response-time: 4
    --
    route: {
      "name": "POST /echo/:name",
      "version": false
    }

Note that if we didn’t pipe the output to the bunyan formatter, we would have had logs in pure JSON, making it easier to mine later.

DTracing a restify application

Note: this section is only applicable if you are on Mac OS X or Joyent SmartOS. You’ll want 3 shell sessions open for this.

Ok, so above I added a slowPoke handler:

server.use(function slowPoke(req, res, next) {
  setTimeout(next.bind(this), parseInt((process.env.SLEEP_TIME || 0), 10));
});

Which notably lets us pass in an arbitrary amount of time to pause. Let’s one more time fire up the server but with a lag in it (also, let’s skip the bunyan formatter so you see what I mean about JSON logs):

$ SLEEP_TIME=500 node echo.js
{"name":"my_restify_application","hostname":"bluesnoop","pid":51387,"level":30,"msg":"my_restify_application listening at http://0.0.0.0:8080","time":"2012-10-31T03:29:01.627Z","v":0}

Great. Now in another terminal, start running loop of our curl commands (put it in () so you can ctrl-c sanely):

$ (while [ true ] ; do curl -isS localhost:8080/echo/mark?q1=foo\&q2=bar -X POST -H content-type:application/json --data-binary '{"body_param": "baz"}' ; done)

You should see it going pretty slowly, seeing as we have a sleep in there. Now, lets go find that sleep from DTrace. In a third window run this:

$ sudo dtrace -lP my_restify_application*
ID   PROVIDER            MODULE                          FUNCTION NAME
6168 my_restify_application52121            module                              func postechoname-start
6169 my_restify_application52121            module                              func postechoname-done
6170 my_restify_application52121            module                              func postechoname-parseAccept-start
6171 my_restify_application52121            module                              func postechoname-parseAccept-done
6172 my_restify_application52121            module                              func postechoname-parseAuthorization-start
6173 my_restify_application52121            module                              func postechoname-parseAuthorization-done
6174 my_restify_application52121            module                              func postechoname-parseDate-start
6175 my_restify_application52121            module                              func postechoname-parseDate-done
6176 my_restify_application52121            module                              func postechoname-parseQueryString-start
6177 my_restify_application52121            module                              func postechoname-parseQueryString-done
6178 my_restify_application52121            module                              func postechoname-parseBody-start
6179 my_restify_application52121            module                              func postechoname-parseBody-done
6180 my_restify_application52121            module                              func postechoname-rateLimit-start
6181 my_restify_application52121            module                              func postechoname-rateLimit-done
6182 my_restify_application52121            module                              func postechoname-authenticate-start
6183 my_restify_application52121            module                              func postechoname-authenticate-done
6184 my_restify_application52121            module                              func postechoname-slowPoke-start
6185 my_restify_application52121            module                              func postechoname-slowPoke-done
6186 my_restify_application52121            module                              func postechoname-echoParms-start
6187 my_restify_application52121            module                              func postechoname-echoParms-done

Note those are all of our handlers. Nice, huh? Now, without going into the details of DTrace, here’s a nice D script you can run that will show you latency broken down by handler:

#!/usr/sbin/dtrace -s
#pragma D option quiet

my_restify_application*:::postechoname-*-start
{
        tracker[arg0, substr(probename, 0, rindex(probename, "-"))] = timestamp;
}

my_restify_application*:::postechoname-*-done
/tracker[arg0, substr(probename, 0, rindex(probename, "-"))]/
{
        this->name = substr(probename, 0, rindex(probename, "-"));
        @[this->name] = quantize(((timestamp - tracker[arg0, this->name]) / 1000000));
        tracker[arg0, substr(probename, 0, rindex(probename, "-"))] = 0;
}

Go ahead and run that for a few seconds (let’s say 5-10):

$ chmod +x ./echo.d
$ sudo ./echo.d
^C

postechoname-authenticate
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-parseAccept
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-parseAuthorization
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-parseBody
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-parseDate
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-parseQueryString
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-rateLimit
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 |                                         0

postechoname-echoParms
value  ------------- Distribution ------------- count
-1 |                                         0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10
1 |                                         0

postechoname-slowPoke
value  ------------- Distribution ------------- count
128 |                                         0
256 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
512 |                                         0

And we see that our latency is sorted according to “worst”; at the bottom is our slowPoke handler, chewing up time while everything else is running < ms. With this it’s pretty obvious what you should go look at.

Wrap up

Obviouly real applications have a lot more moving parts than what was shown here, but hopefully you have a grasp on the basics of restify are. I’ve left out the details on lots of complex parts of it; like any dependency you’ll have to read the docs a bit more thoroughly to get into more advanced uses. But if you’re looking to build a well-behaved REST application, this should help get you going.

This is the 7th in a series of posts leading up to Node.js Knockout on using Nodemailer. This post was written by Node Knockout judge and Nodemailer creator Andris Reinman.

tl;dr — How to send an e-mail using Nodemailer

Install Nodemailer

> npm install nodemailer

Include the module in your script and create a reusable transport object

var nodemailer = require("nodemailer");

var smtpTransport = nodemailer.createTransport("SMTP",{
   service: "Gmail",
   auth: {
       user: "gmail.user@gmail.com",
       pass: "gmailpass"
   }
});

Send an e-mail using the connection object

smtpTransport.sendMail({
   from: "My Name <me@example.com>", // sender address
   to: "Your Name <you@example.com>", // comma separated list of receivers
   subject: "Hello ✔", // Subject line
   text: "Hello world ✔" // plaintext body
}, function(error, response){
   if(error){
       console.log(error);
   }else{
       console.log("Message sent: " + response.message);
   }
});

Send another e-mail without caring about the outcome (or add the callback function as well, like in the previous code block, if you do care)

smtpTransport.sendMail({
   from: "My Name <me@example.com>", // sender address
   to: "Your Name <you@example.com>", // comma separated list of receivers
   subject: "Hello ✔", // Subject line
   text: "Hello world ✔" // plaintext body
});

For additional details see Nodemailer documentation or read through this article.

Super brief introduction to e-mail technologies

There are several ways to send an e-mail but in the end it all comes down to the SMTP protocol (Simple Mail Transfer Protocol), since this is the method how e-mails are actually transferred in the interwebs. There have been some other propositions as well in the past, like ISO standardized X.400 for example but these never took off and thus don’t really matter for most.

The reason why you don’t just open a TCP connection to the recipients incoming SMTP server (the one that is found from the domains DNS MX record and listening on port 25), transmit the message and be done with it, is that in this way, most probably, your message ends up in the junk folder or gets lost entirely - for one, you are not a trusted sender, your sending volumes are way too low (what an irony). This behaviour is also not optimal, as your application would have to handle e-mail bounces, errors, greylisting etc. The better way instead is to turn to your Mail Submission Agent (MSA), which also speaks SMTP and for an exchange of your credentials, happily queues your message for delivery by a relaying Mail Transfer Agent (MTA), which in turn, also speaks SMTP. Often MSA and MTA are one and the same but in theory they are meant to perform somewhat different tasks.

SMTP is a textual protocol and looks a bit like this:

CLIENT: <initiates connection to the server>
SERVER: 220 bob the mailcat listening
CLIENT: HELO joe.the.sender
SERVER: 250 Hello, joe.the.sender, nice to meet you!
CLIENT: MAIL FROM:<me@example.com>
SERVER: 250 Ok, proceed with recipients
...

The numbers returned from the server are status codes, anything equal or above 400 is an error (HTTP anyone?)

While SMTP defines the transport of the envelope (from the sender’s computer to the recipients inboxes), the contents of the envelope is a completely different thing. Actually the two are so unrelated, you can set whatever you want to as the message contents (you can spoof “from” and “to” addresses in the mail header, you can set a future or prehistoric date for the message, you can use strange byte sequences or use an invalid format etc.) and it still gets delivered. Depending on the MSA, some of the data might be corrected though (eg. the prehistoric date of yours might be adjusted to a more recent value) but not all of it.

It’s exactly as with regular mail - you have an envelope that says where the letter is supposed end up and then you have the envelope contents that can contain a valid letter, addressed to the same person labeled on the envelope; or some money without an explanation; or some barb wire; or instead a letter addressed someone totally unrelated from the person on the label of the envelope. If you don’t have the envelope you can never be sure just based on the included letter from where the message originates and to whom it was sent.

Thats why when you look at a spam mail you often see that the recipient name and address fields are blank or invalid - the e-mail client program takes this information from the message contents and not from the envelope. Luckily most servers prepend some of the envelope information also to the mail header, so if you look at the raw source of the message, the first line, “Delivered-To” says which actual address was the final receiver. With several mail accounts ending up in the same e-mail client box, you might not be always able to tell it by yourself. That is also how BCC works - the name is on the envelope but not in the message.

The most simple e-mail message only consists of header and ascii content

From: me@example.com
To: you@example.com
Subject: Hello

How are you doing?

If you’ve ever done HTTP, then the structure should seem familiar. There’s a header block and after two consecutive line breaks (NB! all line breaks in the interwebs protocols use the form <CR><LF> (eg. the “windows style”) and not the <LF> (the “unix style”) form) comes the body of the message.

Nontrivial E-mail messages (the ones that want to use non-ascii characters or attachments or almost any other feature) are formatted by MIME which stands for Multipurpose Internet Mail Extensions. You can include multiple “views” of the same data for the message (ie. HTML view and plaintext view), add binary attachments, use different character sets and so on but eventually it is all compiled to a single ascii-only file before it is being sent as the e-mail. The compiled raw source was meant to be human readable on ancient 80 column terminals but with multipart messages (attachments, different “views” etc), it is not a pleasant exercise to browse through - all “extensions” are added on top of the original plain format, making up an onion-like structure. For example one horror child of such extending is the way to support long unicode filenames etc. for attachments (RFC2231) that uses urlencoding instead of mime encoding.

While most of the strings in e-mail headers are encoded in MIME encoded-word, like the following

Subject: =?us-ascii?Q?This_is_even_more_***fun***_isn't_it!?=

Then here’s an excerpt from the RFC2231 for splitting long header values:

Content-Type: application/x-stuff
   title*0*=us-ascii'en'This%20is%20even%20more%20
   title*1*=%2A%2A%2Afun%2A%2A%2A%20
   title*2="isn't it!"

No, it is not fun.

Y U NO USE SAME ENCODINGS FOR EVERYTHINGZ?

E-mail modules

To keep you away from the tedious task of mingling together standard compliant e-mail messages that would go through the wire, enter e-mail sending libraries. There are some for every programming language and for Node.js, there’s Nodemailer by yours truly or emailjs by eleith. These, of course, are not the only options, there used to be (now deprecated) node_mailer by Marak Squires and there still is pony by substack, there’s probably even more but for feature completeness I would definitely suggest using either Nodemailer or emailjs. As the author of Nodemailer I’m not going to describe the other modules but these are still worth checking out.

Installing Nodemailer

Installing Nodemailer follows the common path - you can install it from npm. Add a dependency to your package.json and run npm install

package.json contents:

...
"dependencies": {
   "nodemailer": "0.3.29"
},
...

NB! If you don’t have a package file yet, you can create one automatically by running npm init.

Now update your installation by running in the same directory the package.json resides.

npm install

If you are on Windows, you are going to see some error messages flying through while installing but this is normal and doesn’t affect how the module is going to work in the end. Nodemailer tries to drag iconv module to the party as an optional dependency and iconv needs compiling. The iconv module isn’t even used, since all the output and input strings for Nodemailer are always in UTF-8. I just haven’t found a good way to automatically exclude the iconv module since the other modules that share the same codebase (eg. MailParser) do need it.

And as the last part of the installation equation, include Nodemailer in your script file

var nodemailer = require("nodemailer");

Selecting mail transport method

While SMTP makes up the barebones of e-mail messaging, you don’t actually have to use it directly if your MSA accepts something else, for example HTTP POST request with e-mail file etc. Nodemailer includes 3 different mail transports and if this doesn’t suit you, you are welcomed to add your own.

  1. SMTP (obviously)
  2. sendmail to stream message contents to the stdin of sendmail command
  3. SES to post the message contents to Amazon SES REST API endpoint (SES also exposes SMTP endpoint but the used credentials are different in this case, see SES docs for details)

In this article we only look at using SMTP as it is the most important of them all.

Setting up SMTP

To use SMTP you need to have access to a MSA server. If you have a Gmail or Google Apps account, you can use Gmail SMTP for this but you can’t ride it far as there are strict usage limits (about 100 emails per day, unless you have a Google Apps Business account that has higher limits), besides you can only send mail from the same user you are authenticated as, you can’t change it to something else (actually you can change the address but it is displayed as a nasty “via authenticated.user@gmail.com” address, not good for your business if you want it to show just “billing@mycompany.com” instead). If you need to go any further, consider using Sendgrid or Postmark or Mailgun or Amazon SES or any other professional message delivery service. There’s a great discussion in Quora about the reasons behind pricing of these services. If you don’t want to send hundreds of millions messages, it is usually more feasible to pay one of the providers than building your own e-mail infrastructure. But getting started is totally free - you can use Gmail as a starting point and move on to a free trial or a developer account of another service before you need to pay a cent.

Lets say that for a starters you’re going to use Gmail as the SMTP service. Setting up Nodemailer is a piece of cake in this case, you just need to have your username and password available:

var smtpTransport = nodemailer.createTransport("SMTP",{
   service: "Gmail",
   auth: {
       user: "gmail.user@gmail.com",
       pass: "gmailpass"
   }
});

And that’s it, you now have a working wire for transmitting e-mails.

NB! If your mail configuration needs go beyond Gmail, you can find the full spectre of connection settings from the Nodemailer documentation.

The problem with SMTP is that you need to have plaintext passwords available somewhere for the app. In case of Gmail there are actually several other options as well. If you use 2 factor authentication, then you can generate application specific password for your mailing app without compromising your real password. There’s also a way to use OAuth1.0 and OAuth2.0 tokens with Gmail if you want to skip using passwords at all. See Nodemailer documentation for details.

Sending an e-mail

Now that we have the working wire, we can start sending e-mails. The simplest form of e-mail requires sender address, recipients address, subject line and message text which can be formatted like this:

var mailOptions = {
   from: "me@example.com", // sender address
   to: "you@example.com", // list of receivers
   subject: "Hello ✔", // Subject line
   text: "Hello world ✔" // plaintext body
}

If the e-mail structure has been composed, we can send the mail using the “transport wire” defined earlier

smtpTransport.sendMail(mailOptions, function(error, response){
   if(error){
       console.log(error);
   }else{
       console.log("Message sent: " + response.message);
   }
});

Nodemailer transport object accepts a large number of messages to be sent at once, you don’t have to wait for the first message to be delivered before you can proceed to the following. Only a limited count of connections are opened to the SMTP server (Nodemailer uses built-in connection pooling) at once and messages are queued automatically for delivery. Thus you can initiate sendMail just as well from a for loop, no need for a complex async setup or a callback hell.

for(var i=0; i<mailArray.length; i++){
   smtpTransport.sendMail(mailArray[i]);
}

Address formatting

Usually you want to include the name of the sender or receiver in addition to the plain address, or sometimes you want to send the e-mail to not just one but several recipients. Nodemailer recipient fields (“to”, “cc”, “bcc”) accept comma separated addresses for multiple recipients and the name can be formatted as well (remeber to enclose the name in double quotes if it contains a comma).

...,
from: "\"Name, User\" <user.name@gmail.com>",
to: "Receiver Name <receiver@gmail.com>, plain@example.com, \"Name, Another\" <another@gmail.com>"
...

And as with any other field, unicode is allowed - you can use non-ascii characters both for the names and for the domains, unicode e-mail domains are converted to the punycode form automatically.

Advanced options

Nodemailer supports a lot of advanced options, like DKIM signing or OAuth2.0 token generation and we are not going to cover all of these here, see Nodemailer docs instead for all the details about these features.

Using HTML and plaintext views

Usually the e-mail contains either HTML view (usually the default), plain text view or both. It is a good practice to include plaintext view in addition to the HTML view, so the less-able mail clients could fall back. Nodemailer can generate the plaintext view automatically, based on the HTML view but you can create it yourself if you want to (or skip it altogether, as only plaintext capable clients are pretty rare nowadays).

You can use both html and text views by specifying according properties of the e-mail object.

...,
text: "plaintext contents",
html: "<p>HTML contents</p>",
...

Keep in mind that one of the main differences between text and html views is that newlines are ignored in the html view but not in the plaintext view when rendering the contents.

You can generate the text view automatically when using generateTextFromHTML property

...,
html: "<p>HTML contents</p>",
generateTextFromHTML: true,
...

Or if you don’t want to include the text contents at all, use only html property

...,
html: "<p>HTML contents</p>",
...

Just whatever you do, do not use a plaintext view that only states “Your e-mail client is not capable of displaying this e-mail” - a lot of e-mail clients use the plaintext part (if it exists) as the preview for the message and html part for the actual viewing.

Adding attachments

Attachments are defined as an array of attached files. You can use a variety of data sources for the attachments (files on disk, strings, buffers, streams, even web urls) and for these examples we are using files on disk. See Nodemailer documentation for the other options.

The most simple form of attachments are file paths on disk. Attachment filename and content type are derived automatically in this case (although you can override these values if you want to).

...,
attachments: [
   {filePath: __dirname + "/attachment1.txt"},
   {filePath: __dirname + "/attachment2.txt"}
],
...

When using images as attachments, you can point to these images as embedded files for the HTML view. You need to define unique cid value for the attachment and use it as an URL.

...,
html: "<p>Embedded image: <img src='cid:image1'/></p>",
attachments:[
   {filePath: __dirname + "/image1.jpg", cid: "image1"}
],
...

In conclusion

Using an e-mail library like Nodemailer makes your life so much easier if you want to send an e-mail with Node.js. Sure, you can compose the message by yourself and transmit it using the standard Node.js net module (and if you’re really interested how the e-mail system works, then this is exactly the way you’re going to do it) but most probably you just need to send an e-mail to your customer and you don’t really care how it is delivered, as long as it makes it through and displays as intended.

If you want to bring your e-mail related application to the next level, you could also start receiving and parsing e-mail in addition to sending it. For creating your own SMTP MX server for accepting e-mail from the interwebs, you can use simpler SimpleSMTP by yours truly or full featured Haraka by baudehlo which is able to fight against spam as well. If you do not want to manage message exchange server by yourself but to use IMAP for connecting to an existing one, you can use node-imap by mscdex or inbox. Once you have received the e-mail (either through SMTP or IMAP), you can parse the raw source into a structurized object wih MailParser. I have also created a helper module mailuploader to parse an e-mail raw source and post the parsed object to a HTTP address as a regular multipart/form-data upload, so even a PHP script could easily handle it.

I hope that by reading this article you found out how easy it is to send e-mails using Nodemailer. And if you ever run into trouble with it, then you know where you can file an issue.

This is the 6th in a series of posts leading up to Node.js Knockout on using hapi. This post was written by Node Knockout judge and hapi contributor Wyatt Preul.

Hapi is a framework for rapidly building RESTful web services. Whether you are building a very simple set of RESTful services or a large scale, cache heavy, and secure set of services, hapi has you covered. Hapi will help get your server developed quickly with its wide range of configurable options.

Building a Products API

The following example will walk you through using hapi to build a RESTful set of services for creating and listing out products. To get started create a directory named ProductsAPI and add a package.json file to the directory that looks like the following.

{
    "name": "ProductsAPI",
    "version": "0.0.1",
    "main": "server",
    "engines": {
        "node": ">=0.8.0"
    },
    "dependencies": {
        "hapi": "0.8.x"
    },
    "private": "true"
}

Then run npm install.

Create a server.js file that will serve as the entry point for the service. Add the following contents to the server.js file.

var hapi = require('hapi');
var routes = require('./routes');

var config = { docs: true };
var http = new hapi.Server('0.0.0.0', 8080, config); // 8080 is the port to listen on

http.addRoutes(routes);

http.start();

In the server.js code above a new instance of the hapi server is started using the configuration specified in config.

By setting docs to true the documentation generator will be enabled. The documentation generator provides a set of pages that explain what endpoints are available and the requirements for those endpoints. The documentation generator will use the validation rules you will create for each route to construct appropriate documentation pages under the /docs path.

Hapi provides a function for adding a single route or an array of routes. In this example we are adding an array of routes from a routes module.

Go ahead and create a routes.js file, which will contain the route information and handlers. When defining the routes we will also be specifying validation requirements. Therefore, at the top of the file require hapi and assign its Types property to a local variable like below.

var Types = require('hapi').Types;

For this example three routes will be created. Below is the code you should use to add the routes. Add the following code to your routes.js file.

module.exports = [
    { method: 'GET', path: '/products', config: { handler: getProducts, query: { name: Types.String() } } },
    { method: 'GET', path: '/products/{id}', config: { handler: getProduct } },
    { method: 'POST', path: '/products', config: { handler: addProduct, payload: 'parse', schema: { name: Types.String().required().min(3) }, response: { id: Types.Number().required() } } }
];

The routes are exported as an array so that they can easily be included by the server implementation we added. For the products listing endpoint we are allowing a querystring parameter for name. When this querystring parameter exists then we will filter the products for those that have a matching name.

The second route is a very simple route that demonstrates how a parameter can become part of the path definition. This route will return a product matching the ID that’s requested.

In the last route, the one used for creating a product, you will notice that extra validation requirements are added, even those on the response body. The request body must contain a parameter for name that has a minimum of 3 characters and the response body must contain an ID to be validated.

Next add the handlers to the routes.js file.

function getProducts(request) {

    if (request.query.name) {
        request.reply(findProducts(request.query.name));
    }
    else {
        request.reply(products);
    }
}

function findProducts(name) {
    return products.filter(function(product) {
        return product.name.toLowerCase() === name.toLowerCase();
    });
}

function getProduct(request) {
    var product = products.filter(function(p) {
        return p.id == request.params.id;
    }).pop();

    request.reply(product);
}

function addProduct(request) {
    var product = {
        id: products[products.length - 1].id + 1,
        name: request.payload.name
    };

    products.push(product);

    request.reply.created('/products/' + product.id)({
        id: product.id
    });
}

As you can see in the handlers, hapi provides a simple way to add a response body by using the request.reply function. Also, in the instance when you have created an item you can use the request.reply.created function to send a 201 response.

Lastly, add a simple array to contain the products that the service will serve.

var products = [{
        id: 1,
        name: 'Guitar'
    },
    {
        id: 2,
        name: 'Banjo'
    }
];

Running the server

Go ahead and run npm start or node server.js to start the server. Now you can navigate to http://localhost:8080/docs to see the documentation for the routes. To see a list of the products navigate to http://locahost:8080/products. Below is a screenshot of what the response looks like.

Go ahead and append ?name=banjo to the URL to try searching for a product by name.

Use curl or a REST console to create a product. Make a POST request to the products endpoint with a name in the body. Using curl the command looks like: curl http://localhost:8080/products -d "name=test". Below is an example of the response headers from making a request to create a product.

Now if you navigate to the Location specified in the response headers you should see the product that you created.

Other features

There are a lot of different configuration features that you can add to the server. The extensive list can be found in the readme at https://github.com/walmartlabs/hapi/#server-configuration.

The built-in cache support has providers for mongo and redis. Setting up cache is as simple as passing cache: true as part of the server configuration.

Additionally, there are several configuration options available on a per route basis. The full list can be found at https://github.com/walmartlabs/hapi/#route-configuration. For example, caching expiration times can also be configured on a per route basis. Also, you can have per-route authentication settings.

Conclusion

By now you should have a decent understanding of what hapi has to offer. There are still many other features and options available to you when using hapi that is covered in the documentation. Please take a look at the github repository and feel free to provide any feedback you may have.

This is the 5th in a series of posts leading up to Node.js Knockout on using Mongoose. This post was written by Node Knockout judge and Mongoose maintainer Aaron Heckmann.

Getting started with Mongoose and Node

In this post we’ll talk about getting started with Mongoose, an object modeling tool for MongoDB and node.js.

Install

We’re going to assume that you have both MongoDB and npm installed for this post. Once you have those, you can install Mongoose:

$ npm install mongoose

Hurray! Now we can simply require mongoose like any other npm package.

var mongoose = require('mongoose');

Schema definition

Though MongoDB is a schema-less database we often want some level of control over what goes in and out of our database collections. We’re confident that we’re going to be the next Netflix so we’ll need a Movie schema and a Rating schema. Each Movie is allowed to have multiple Ratings.

var Schema = mongoose.Schema;
var RatingSchema = new Schema({
    stars    : { type: Number, required: true }
  , comment  : { type: String, trim: true }
  , createdAt: { type: Date, default: Date.now }
});

So far we’ve created a Rating schema with a stars property of type Number, a comment property of type String, and a createdAt property of type Date. Whenever we set the stars property it will automatically be cast as a Number. Note also that we specified required which means validation will fail if an attempt is made to save a rating without setting the number of stars. Likewise, whenever we set the comment property it will first be cast as a String before being set, and since whitespace around comments is very uncool, we use the built-in trim setter.

Now that we’re happy with our Rating model we’ll use it within our Movie model. Each movie should have name, director, year, and ratings properties.

var MovieSchema = new Schema({
    name    : { type: String, trim: true, index: true }
  , ratings : [RatingSchema]
  , director: Schema.ObjectId
  , year    : Number
});

Here we see that ratings is set to an array of Rating schemas. This means that we’ll be storing Ratings as subdocuments on each Movie document. A subdocument is simply a document nested within another.

You might have noticed the index option we added to the name property. This tells MongoDB to create an index on this field.

We’ve also defined director as an ObjectId. ObjectIds are the default primary key type MongoDB creates for you on each document. We’ll use this as a foreign key field, storing the document ObjectId of another imaginary Person document which we’ll leave out for brevity.

TIP: Note that we needed to declare the subdocument Rating schema before using it within our Movie schema definition for everything to work properly.

This is what a movie might look like within the mongo shell (included when you install MongoDB):

{ name: 'Inception',
  year: 2010,
  ratings:
   [ { stars: 8.9,
       comment: 'I fell asleep during this movie, and yeah, you\'ve heard this joke before' },
     { stars: 9.3 } ],
  director: ObjectId("4e4b4a8b73e1d576d6a1438e") }

Now that we’ve finished our schemas we’re ready to create our movie model.

var Movie = mongoose.model('Movie', MovieSchema);

And thats it! Everything is all set with the exception of being able to actually talk to MongoDB. So let’s create a connection.

mongoose.connect('mongodb://localhost/nodeknockout', function (err) {
  if (err) return handleErrorSomehow(err);
  // ok we're set
});

Now we’re ready to create a movie and save it.

var movie = new Movie({ name: "Frankenweenie", director: anObjectId, year: 2012 });

movie.save(function (err) {
  if (err) return console.error(err); // we should handle this
});

Oh, but what about adding ratings?

Movie.findOne({ name: "Frankenweenie" }).where("year").equals(2012).exec(function (err, movie) {
  if (err) // handle this

  // add a rating
  movie.ratings.push({ stars: 9.0, comment: "it made me happy" });
  movie.save(callback);
});

To look up our movie we used Model.findOne which accepts a where clause as its first argument. We also took advantage of the Query object returned by this method to add some more sugary filtering. Finally, we called the Query’s run method to execute it.

We didn’t have to do it this way, instead you could just pass all of your where params directly as the first argument like so:

Movie.findOne({ name: "Frankenweenie", year: 2012 }, callback);

Though the first example is more verbose it highlights some of the expressive flexibility provided by the Query object returned.

Here are a couple more ways we could write this query:

Movie.where('name', /^Frankenweenie/i).where('year', 2012).limit(1).exec(callback);

Movie.find({ name: "Frankenweenie", year: { $gt: 2011, $lt: 2013 }}, null, { limit: 1 }, callback);

This is all well and good but what if we look up movies by director and year a lot and need the query to be fast? First we’ll create a static method on our Movie model:

MovieSchema.statics.byNameAndYear = function (name, year, callback) {
  // NOTE: find() returns an array and may return multiple results
  return this.find({ name: name, year: year }, callback);
}

We’ll also add a compound index on these two fields to give us a performance boost:

MovieSchema.index({ name: 1, year: 1 });

For good measure we’ll add a movie instance method to conveniently look up the director:

MovieSchema.methods.findDirector = function (callback) {
  // Person is our imaginary Model we skipped for brevity
  return this.model('Person').findById(this.director, callback);
}

Putting it all together:

Movie.byNameAndYear("Frankenweenie", 2012, function (err, movies) {
  if (err) return console.error(err); // handle this
  var movie = movies[0];
  movie.findDirector(function (err, director) {
    if (err) ...
    // woot
  })
});

Thats it for this post. For more info check out mongoosejs.com, the mongoose plugins site, the github README, or the Mongoose test directory to see even more examples.

This is the 4th in a series of posts leading up to Node.js Knockout on how to use Socket.IO. This post was written by socket.io creator Guillermo Rauch.

Ready to rock the Node Knockout 3? Considering making a real-time app or game? Then you’re likely considering Socket.IO

Socket.IO makes realtime easy and cross-browser

If the web 2.0 was about AJAX requests and responses, the real-time web is about events.

With Socket.IO, you can emit events from the server to the client and vice-versa, at any time.

Consider the following example, in which we push data from the server to the client every 3 seconds. We’re going to get the latest tweets and render them on the client.

On the server side, we create a directory called example/ with the following package.json inside:

{
    "name": "example"
  , "version": "0.0.2"
  , "dependencies": {
        "socket.io": "~0.9.10"
      , "request": "~2.11.4"
    }
}

Then run npm install. That will get Socket.IO and a library to make http requests (to Twitter) very simple to write.

Create an app.js with the following:

var sio = require('socket.io')
  , http = require('http')
  , index = require('fs').readFileSync(__dirname + '/index.html')
  , request = require('request')

var app = http.createServer(function (req, res) {
      res.writeHead(200, { 'Content-Type': 'text/html' });
      res.end(index);
    })
  , io = sio.listen(app);

app.listen(3000);

So far, we initialized a HTTP server that serves the index file, and we attach Socket.IO to it. We then make it listen on port 3000.

We’re now gonna search twitter every 2 seconds to send tweets to the client. The clients don’t explicitly request it like in traditional Ajax, but we push it to them:

setInterval(function () {
  request({
    url: 'http://search.twitter.com/search.json?q=today'
  }, function (err, res, data) {
    var obj = JSON.parse(data);
    io.sockets.emit('tweets', obj.results.map(function (v) { return v.text; }));
  });
}, 2000);

Notice that I accessed the io.sockets property. That’s what we call the Manager of sockets, which also emits a connection event for individual people that connect:

io.sockets.on('connection', function (socket) {
  socket.broadcast.emit('someone connected');

  socket.on('some event', function () {
    console.log('I got an event');
  });
});

In this case, I’m making this socket broadcast to others that it connected. Broadcasting from the socket object means in this case we’re sending a message to everyone else, except for that particular socket.

Now that we’re done with the server part, we capture these events on the client and do some jQuery juggling. Create an index.html with this code:

<!doctype html>
<html>
  <head>
    <script src="/socket.io/socket.io.js"></script>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>

    <script>
      var socket = io.connect();

      $(function () {
        socket.on('tweets', function (tweets) {
          $.each(tweets, function (i, tweet) {
            $('#tweets').prepend($('<li>').text(tweet));
          });
        });

        socket.on('someone connected', function () {
          $('body').prepend('<p>Someone just connected!</p>');
        });

        socket.emit('some event');
      });
    </script>
  </head>
  <body>
    <h1>Tweets</h1>
    <ul id="tweets"></ul>
  </body>
</html>

Putting sockets in rooms

Many times you need to emit information to certain people and not others. Use socket.join:

io.sockets.on('connection', function (socket) {
  socket.join('fighters');
});

Then you can broadcast and emit to those people:

// to all sockets in a group
io.sockets.in('fighters').emit('something');

// from a socket to the rest of the sockets
socket.broadcast.to('fighters').emit('something else');

Tools for game makers.

Remember the broadcast property that I accessed above? That’s what we call a flag:

A useful one for game developers is called the volatile flag. If you ever have data that it’s not absolutely crucial that the client gets if the client has not finished receiving previous messages, you can make that volatile:

socket.volatile.emit('position', x, y);

If the player changes its x and y coordinates really quickly, and we’re not done sending a particular position, some packets will be dropped. This will make things faster in many scenarios.

Going beyond, with callbacks

On both ends (server and client side), you can pass a function to request a explicit data response. Consider an example where the chat client sets a nickname, but you want to validate whether it’s available or not.

On the client:

socket.emit('set nickname', $('#my-input').val(), function (available) {
  if (available) {
    alert('Nick is available. Congrats!');
  } else {
    alert('Nick not available');
  }
});

On the server:

io.sockets.on('connection', function (socket) {
  socket.on('set nickname', function (fn) {
    isNickNameAvailable(function (bool) {
      fn(bool);
    });
  });
});

Wrapping up

You can check out the example on GitHub. Clone the repo, then run npm install and node app and point your browser to http://localhost:3000!

These are all the tools in your toolbox. Go make great realtime apps!