This is the 13th in a series of posts leading up to Node.js Knockout on using mongodb with node-mongodb-native This post was written by Node Knockout judge and node-mongo-db-native author Christian Kvalheim.
Mongo DB has rapidly grown to become a popular database for web applications and is a perfect fit for Node.JS applications, letting you write Javascript for the client, backend and database layer. Its schemaless nature is a better match to our constantly evolving data structures in web applications and the integrated support for location queries is a bonus that’s hard to ignore. Throw in replicasets for scaling and we are looking at really nice platform to grow your storage needs now and in the future.
Now to shamelessly plug my driver. It can be downloaded either using npm or fetched from the github repository. To install via npm do the following:
npm install mongodb
or go fetch it from github at https://github.com/christkv/node-mongodb-native
Once this business is taken care of, let’s move through the types available for the driver and then how to connect to your Mongo DB instance before facing the usage of some crud operations.
So there is an important thing to keep in mind when working with Mongo DB and that is that there is a slight mapping difference between the types supported in Mongo DB and the native types in Javascript. Let’s have a look at the types supported out of the box and then how types are promoted by the driver to try to fit the native Javascript types as closely possible.
As we see the number type can be a little tricky due to the way integers are implemented in Javascript. The latest driver will do correct conversion up to 53 bits of complexity. If you need to handle big integers the recommendation is to use the Long class to operate on the numbers.
Let’s get around to setting up a connection with the Mongo DB database. Jumping straight into the code, let’s do a direct connection and then look at the code.
var mongo = require('mongodb'),
Server = mongo.Server,
Db = mongo.Db;
Db.connect("mongodb://localhost:27017/exampleDb", {auto_reconnect:true}, function(err, db) {
if(!err) {
console.log("We are connected");
}
});
Let’s have a quick look at how the connection code works. The Db.connect method let’s use use a uri to connect to the Mongo database, where localhost:27017 is the server host and port and exampleDb the db we wish to connect to. After the url notice the hash containing the auto_reconnect key. Auto reconnect tells the driver to retry sending a command to the server if there is a failure during it’s execution.
Another useful option you can pass in is
poolSize, this allows you to control how many tcp connections are opened in parallel. The default value for this is 5 but you can set it as high as you want. The driver will use a round-robin strategy to dispatch and read from the tcp connection.
We are up and running with a connection to the database. Let’s move on and look at what collections are and how they work.
Collections are the equivalent of tables in traditional databases and contain all your documents. A database can have many collections. So how do we go about defining and using collections? Well, there are a couple of methods that we can use. Let’s jump straight into code and then look at the code.
the requires and and other initializing stuff omitted for brevity
db.open(function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});
Three different ways of creating a collection object but slightly different in behavior. Let’s go through them and see what they do
db.collection('test', function(err, collection) {});
This function will not actually create a collection on the database until you actually insert the first document.
db.collection('test', {w:1}, function(err, collection) {});
Notice the {w:1} option. This option will make the driver check if the collection exists and issue an error if it does not.
db.createCollection('test', function(err, collection) {});
This command will create the collection on the Mongo DB database before returning the collection object. If the collection already exists it will ignore the creation of the collection.
db.createCollection('test', {w:1}, function(err, collection) {});
The {w:1} option will make the method return an error if the collection already exists.
With an open db connection and a collection defined we are ready to do some CRUD operation on the data.
So let’s get dirty with the basic operations for Mongo DB. The Mongo DB wire protocol is built around 4 main operations insert/update/remove/query. Most operations on the database are actually queries with special json objects defining the operation on the database. But I’m getting ahead of myself. Let’s go back and look at insert first and do it with some code.
the requires and and other initializing stuff omitted for brevity
db.open(function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var doc1 = {'hello':'doc1'};
var doc2 = {'hello':'doc2'};
var lotsOfDocs = [{'hello':'doc3'}, {'hello':'doc4'}];
collection.insert(doc1);
collection.insert(doc2, {w:1}, function(err, result) {});
collection.insert(lotsOfDocs, {w:1}, function(err, result) {});
});
A couple of variations on the theme of inserting a document as we can see. To understand why it’s important to understand how Mongo DB works during inserts of documents.
Mongo DB has asynchronous insert/update/remove operations. This means that when you issue an insert operation it’s a fire and forget operation where the database does not reply with the status of the insert operation. To retrieve the status of the operation you have to issue a query to retrieve the last error status of the connection. To make it simpler to the developer, the driver implements the {w:1} options so that this is done automatically when inserting the document. {w:1} becomes especially important when you do update or remove as otherwise it’s not possible to determine the number of documents modified or removed.
Now let’s go through the different types of inserts shown in the code above.
collection.insert(doc1);
Taking advantage of the async behavior and not needing confirmation about the persisting of the data to Mongo DB we just fire off the insert (we are doing live analytics, losing a couple of records does not matter).
collection.insert(doc2, {w:1}, function(err, result) {});
That document needs to stick. Using the {w:1} option ensure you get the error back if the document fails to insert correctly.
collection.insert(lotsOfDocs, {w:1}, function(err, result) {});
A batch insert of document with any errors being reported. This is much more efficient if you need to insert large batches of documents as you incur a lot less overhead.
Right, that’s the basics of inserts ironed out. We got some documents
in there but want to update them as we need to change the content of a
field. Let’s have a look at a simple example and then we will dive into
how Mongo DB updates work and how to do them efficiently.
the requires and and other initializing stuff omitted for brevity
db.open(function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var doc = {mykey:1, fieldtoupdate:1};
collection.insert(doc, {w:1}, function(err, result) {
collection.update({mykey:1}, {$set:{fieldtoupdate:2}}, {w:1}, function(err, result) {});
});
var doc2 = {mykey:2, docs:[{doc1:1}]};
collection.insert(doc2, {w:1}, function(err, result) {
collection.update({mykey:2}, {$push:{docs:{doc2:1}}, {w:1}, function(err, result) {});
});
});
Alright before we look at the code we want to understand how document updates work and how to do the efficiently. The most basic and less efficient way is to replace the whole document, this is not really the way to go if you want to change just a field in your document. Luckily Mongo DB provides a whole set of operations that let you modify just pieces of the document Atomic operations documentation. Basically outlined below.
Now that the operations are outline let’s dig into the specific cases show in the code example.
collection.update({mykey:1}, {$set:{fieldtoupdate:2}}, {w:1}, function(err, result) {});
Right, so this update will look for the document that has a field mykey equal to 1 and apply an update to the field fieldtoupdate setting the value to 2. Since we are using the {w:1} option the result parameter in the callback will return the value 1 indicating that 1 document was modified by the update statement.
collection.update({mykey:2}, {$push:{docs:{doc2:1}}, {w:1}, function(err, result) {});
This updates adds another document to the field docs in the document identified by {mykey:2} using the atomic operation $push. This allows you to modify keep such structures as queues in Mongo DB.
Let’s have a look at the remove operation for the driver. As before let’s start with a piece of code.
the requires and and other initializing stuff omitted for brevity
db.open(function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var docs = [{mykey:1}, {mykey:2}, {mykey:3}];
collection.insert(docs, {w:1}, function(err, result) {
collection.remove({mykey:1});
collection.remove({mykey:2}, {w:1}, function(err, result) {});
collection.remove();
});
});
Let’s examine the 3 remove variants and what they do.
collection.remove({mykey:1});
This leverages the fact that Mongo DB is asynchronous and that it does not return a result for insert/update/remove to allow for synchronous style execution. This particular remove query will remove the document where mykey equals 1.
collection.remove({mykey:2}, {w:1}, function(err, result) {});
This remove statement removes the document where mykey equals 2 but since we are using {w:1} it will back to Mongo DB to get the status of the remove operation and return the number of documents removed in the result variable.
collection.remove();
This last one will remove all documents in the collection.
Queries are of course a fundamental part of interacting with a database and Mongo DB is no exception. Fortunately for us it has a rich query interface with cursors and close to SQL concepts for slicing and dicing your datasets. To build queries we have lots of operators to choose from Mongo DB advanced queries. There are literarily tons of ways to search and ways to limit the query. Let’s look at some simple code for dealing with queries in different ways.
the requires and and other initializing stuff omitted for brevity
db.open(function(err, db) {
if(err) { return console.dir(err); }
var collection = db.collection('test');
var docs = [{mykey:1}, {mykey:2}, {mykey:3}];
collection.insert(docs, {w:1}, function(err, result) {
collection.find().toArray(function(err, items) {});
var stream = collection.find({mykey:{$ne:2}}).streamRecords();
stream.on("data", function(item) {});
stream.on("end", function() {});
collection.findOne({mykey:1}, function(err, item) {});
});
});
Before we start picking apart the code there is one thing that needs to be understood, the find method does not execute the actual query. It builds an instance of Cursor that you then use to retrieve the data. This lets you manage how you retrieve the data from Mongo DB and keeps state about your current Cursor state on Mongo DB. Now let’s pick apart the queries we have here and look at what they do.
collection.find().toArray(function(err, items) {});
This query will fetch all the document in the collection and return them as an array of items. Be careful with the function toArray as it might cause a lot of memory usage as it will instantiate all the document into memory before returning the final array of items. If you have a big resultset you could run into memory issues.
var stream = collection.find({mykey:{$ne:2}}).stream();
stream.on("data", function(item) {});
stream.on("end", function() {});
This is the preferred way if you have to retrieve a lot of data for streaming, as data is deserialized a data event is emitted. This keeps the resident memory usage low as the documents are streamed to you. Very useful if you are pushing documents out via websockets or some other streaming socket protocol. Once there is no more document the driver will emit the end event to notify the application that it’s done.
collection.findOne({mykey:1}, function(err, item) {});
This is special supported function to retrieve just one specific document bypassing the need for a cursor object.
That’s pretty much it for the quick intro on how to use the database. I have also included a list of links to where to go to find more information and also a sample crude location application I wrote using express JS and mongo DB.
This is the 12th in a series of posts leading up to Node.js Knockout, and covers using Ratchet.io in your node app.
Ratchet.io is an error monitoring, aggregation and intelligence service that has plugins for a wide variety of languages and frameworks including Rails, Django, PHP and Node.js.
Let’s create a simple server with some bugs to get started.
npm install ratchetio
npm install express
Most servers written in node use the Express.js framework which is built on top of Connect.js and makes it really easy to write and include custom middleware. Let’s include Ratchet’s error handling middleware and record our first error.
var express = require('express'),
ratchet = require('ratchetio');
var app = express();
app.use(app.router);
// Make sure this is below the app.router
app.use(ratchet.errorHandler('ACCESS_TOKEN'));
app.get('/', function(req, res) {
throw new Error('Hello World!');
});
console.log('browse to http://localhost:8080/ and then to your ' +
'ratchet.io project');
app.listen(8080);
If you haven’t created a project on Ratchet.io yet, head over to
https://ratchet.io/fasttrack/nk2012/
and create a free account. Then, replace the ACCESS_TOKEN in the code above
with your server-side access token.
Next, you should be able to run your server and see the following in your browser at http://localhost:8080:
And now on ratchet.io:
Recording errors is great but sometimes we just want to do the equivalent of
console.log() and have it stored in one place even when running on multiple
production servers.
Ratchet makes this dead simple. Just pass in the string to send over along with
an optional level (or it will default to error). Choices for level include debug, info, warning, error and critical.
ratchet.reportMessage('Why the heck is this not working??? ' + (typeof myVar), 'debug');
And if you want to send over some useful request context…
ratchet.reportMessage('Oooooh, is it because the cookie is missing?',
'debug', request);
UncaughtExceptionNode.js has a super-handy and potentially dangerous event that it will trigger when there’s an uncaught exception at the top-most level of your code. These are the types of things that will kill your server so it’s good to know when and why they happened.
Thankfully, node.js makes it super-simple to track these and so does Ratchet.
ratchet.handleUncaughtExceptions();
var ratchet = require('ratchetio');
ratchet.handleUncaughtExceptions('ACCESS_TOKEN');
var foo = bar();
It’s important to have full control over when error reporting happens. Ratchet provides a few different options for when to report errors and messages back to ratchet.io.
This is the default handler if one is not specified. It will create a Timer
function which will execute every N seconds, (where N is configurable using the handlerInterval option.)
e.g. Queue up any errors/messages and send to ratchet.io every 10 seconds
ratchet.init("ACCESS_TOKEN", {handler: "setInterval", handlerInterval: 10});
or
app.use(ratchet.errorHandler("ACCESS_TOKEN",
{handler: "setInterval", handlerInterval: 10});
This handler will schedule a callback which will send any queued up
errors/messages to ratchet.io on the next tick of the javascript runtime. This
is useful for ensuring that any callbacks passed into the handleError or
reportMessage functions is called asynchronously.
ratchet.init("ACCESS_TOKEN", {handler: "nextTick"});
ratchet.reportMessage("Record me asynchronously", "debug", function(ratchetErr) {
if (err) {
console.log("Problem sending message to ratchet: " + ratchetErr);
} else {
console.log("Recorded message to ratchet");
}
});
This handler will send any recorded messages or errors to ratchet as soon as they are processed. This is useful for debugging complex code and making sure that important errors are sent to ratchet.io as soon as possible.
ratchet.init("ACCESS_TOKEN", {handler: "inline"});
ratchet.reportMessage("DATABASE IS DOWN!", "critical", function(ratchetErr) {
if (err) {
console.log("Problem sending message to ratchet: " + ratchetErr);
} else {
console.log("Recorded message to ratchet... shutting down");
ratchet.shutdown();
process.exit(1);
}
});
The Ratchet notifier also allows you to switch handler types whenever you want. If you notice that the queue is building up or you need to start pushing errors to ratchet.io as soon as they’re available, it’s as simple as:
ratchet.changeHandler("inline");
or
ratchet.changeHandler("setInterval");
Under the covers, the Ratchet node notifier is completely asynchronous and provides the ability to pass around callbacks that are executed after any i/o occurs.
e.g. Trigger a callback to make sure a message was queued to be sent to ratchet.io:
ratchet.reportMessage("Something important", "info", function(ratchetErr) {
// ratchetErr will be null if the message was queued
});
e.g. Execute some code after ratchet is cleanly shutdown:
ratchet.shutdown(function(ratchetErr) {
// this function will be called after all queued items were sent
// to ratchet.io or an error occurred
});
That’s really all there is to it. The above instructions should cover 95% of the average use cases.
Happy Hacking!
We’re continually adding new features to ratchet.io and all of the notifiers. If you’re interested, take a look at the api.js code for some more functionality or at our API docs for even more.
Send us an email at support@ratchet.io or a pull request for bugfixes and/or new features!
This is the 11th in a series of posts leading up to Node.js Knockout on debugging node processes using Bonsai.js. This post was written by Bonsai.js contributor Dustan Kasten of Skookum Digital Works.
Bonsai.js is a new graphics library that, unlike most graphic libraries currently in the wild, ships with an SVG renderer. Not only that, but it has an architecturally separate runner and renderer so that all the heavy lifting can happen in an iFrame, WebWorker, or node context.
Built by the amazing team at Uxebu, Bonsai is the HTML5 library that Pixelplant depends on for its Flash conversion. If you are familiar with Flash development or terminology you already have a grasp of how to create with bonsai.
Note: All of this code is available at https://github.com/iamdustan/bonsai-demos. To view demos,
git clone git@github.com:iamdustan/bonsai-demos && npm startthen browse tohttp://localhost:8080in your favorite browser.
A few things to note as you walk into the bonsai forest. Due to the architecturally separated runner and renderer, all of the code that draws to the bonsai stage is run in a sandbox. In this sandbox you have access to a large number of tools that bonsai creates for you (many examples listed below under Bonsai Tools).
The bonsai world revolves around the stage. The runner and renderer communicate through passing messages back and forth. This is for both user triggered actions (e.g. pointer events and clicks) or for things you want to trigger (say, pressing a Start button).
These are just a few of the objects bonsai makes available to you in the runner context.
And more…
Getting started with bonsai always seems to be the biggest hurdle. The documentation has recently been super-sized with an explanation of the execution environment that is required reading: Bonsai execution.
Through the following examples, you will have everything you need to know to get started making awesome.
Releases happen very regularly with new features added and bugs smashed. The latest official release is always available on github or from the cdn. Additionally, if you are like me and feeling a bit edgier you could just build the latest copy from master yourself (requires java to run Closure Compiler):
cd /tmp; git clone git@github.com:Uxebu/bonsai && make build
cp ./dist/bonsai.js YOUR_DIRECTORY_HERE
To start things off we’re going to do the smallest amount of boilerplate necessary and demonstrate a few of the methods and sugar that Bonsai provides.
// pass a function through as the code parameter
var element = document.getElementById('movie');
bonsai.run(movie, {
code: runner,
width: 600,
height: 400
});
function runner () {
var rect = new Rect(0, 0, 200, 200);
rect
.fill('random')
.addTo(stage)
.attr({
x: stage.width - rect.attr('width'),
y: stage.width - rect.attr('height')
})
.animate('0.5s', {
x: 0,
y: 0
});
}
As you can see, our instance of a Rect has some helpful methods
accompanying it, and don’t forget the lovely color, random.
All displayable objects are an instance of a
DisplayObject, aptly
title.
Clearly, this method will not scale to complex code, so let’s break that off.
Let’s begin by moving that runner method into a file to call its
own, following the Flash semantics, movie.js.
// movie.js
var rect = new Rect(0, 0, 200, 200);
rect
.fill('random')
.addTo(stage)
.attr({
x: stage.width - rect.attr('width'),
y: stage.width - rect.attr('height')
})
.animate('0.5s', {
x: 0,
y: 0
});
// app.js
var element = document.getElementById('movie');
bonsai.run(movie, {
url: 'path/to/movie.js',
plugins: [],
width: 600,
height: 400
});
Well, that was easy enough. Bonsai here is taking the file you specify
under the url key and loading into the appropriate runner context.
Let’s continue adding complexity and add a second file to handle
user interactions. Bonsai pipes all user interaction into the runner
context. This is imperative since the runner has no concept of things
like pointer events or event.pageX
// ui.js
stage.on('message', handleMessage)
stage.on('pointerdown', handlePointerdown)
stage.on('keypress', handleKeypress)
function handleMessage(message) {
if (message.type === 'Rect' && message.attr)
new Rect(message.attr.x, message.attr.y, message.attr.w, message.attr.h)
.attr(message.attr)
.fill(message.attr.fill || 'random')
.addTo(stage);
}
function handlePointerdown (e) {
handleMessage({
type: 'Rect',
attr: {
x: e.stageX-25,
y: e.stageY-25,
w: 50,
h: 50
}
})
}
function handleKeypress (e) {
stage.sendMessage('keypress', e.keyCode);
}
// app.js
var element = document.getElementById('movie');
var stage = bonsai.run(movie, {
url: 'path/to/movie.js',
plugins: ['path/to/ui.js'],
width: 600,
height: 400
});
stage.sendMessage({
type: 'Rect',
attr: {
x: Math.random() * 100, y: Math.random() * 100,
w: Math.random() * 100, h: Math.random() * 100
}
});
stage.on('message:keypress', function (data) {
console.log('Hey! Someone touched me at {keyCode}! - the
Keyboard'.replace(/{keyCode}/g, data));
});
Oh right. I almost forgot. We’ve just been letting bonsai manage setting up the runner context in a WebWorker or iFrame so far. We need a way to run this on the server and connect our thousand friends to it. Very well, let’s get to it!
Up to this point we’ve been starting bonsai and passing the configuration object all at the same time. We will use a slightly different version this time where we first call setup passing in a Socket.io runner context.
Note: This demo uses a currently custom build of Bonsai that exposes a few internal utilities. Grab it here: https://github.com/uxebu/bonsai-server/blob/master/example/bonsai.js
// main.js
var movie = document.getElementById('movie')
var runnerContext = function (runnerUrl) {
this.socket = io.connect(runnerUrl);
};
// some boilerplate to connext via socket.io
var proto = runnerContext.prototype = bonsai.tools.mixin({
init: function () {
var self = this;
this.socket.on('message', function(msg) {
self.emit('message', msg[0]);
});
},
notify: function (message) {
this.socket.emit('message', message);
},
notifyRunner: function (message) {
this.socket.emit('message', message);
},
run: function (code) {
this.notifyRunner({
command: 'runScript',
code: code
});
}
}, bonsai.EventEmitter);
proto.notifyRunnerAsync = proto.notifyRunner;
bonsai
.setup({
runnerContext: runnerContext,
runnerUrl: 'http://localhost:3000
})
.run(movie, {
width: 600,
height: 600
});
// movie.js
// this is read by and run on the server
// demo from http://demos.bonsaijs.org/demos/circles/index.html
var centerX = 250,
centerY = 250,
circles = 180,
distance = 180,
frames = 14,
radiusMin = 10,
radiusVar = 10;
var circle, random = Math.random;
for (var i = 0; i < circles; ++i) {
var f = i / circles,
x = centerX + distance * Math.sin(f*2*Math.PI),
y = centerY + distance * -Math.cos(f*2*Math.PI),
radius = random() * radiusVar + radiusMin;
circle = new Circle(x, y, radius).
attr({fillColor: 'random', fillGradient: bonsai.gradient.radial(['#FFFFFF88', '#FFFFFF00'])});
circle.x = x;
circle.y = y;
stage.addChild(circle);
}
var c = stage.children();
stage.length(frames);
var spread = 80;
stage.on(0, function() {
for (var i = 0, circle; (circle = c[i++]); ) {
circle.animate(frames, {
x: circle.x + spread * random() - spread / 2,
y: circle.y + spread * random() - spread / 2
}, {easing: 'sineInOut'});
}
});
// server.js
var bonsai = require('bonsai');
var fs = require('fs');
var bonsaiCode = fs.readFileSync('./movie.js');
var socketRenderer = function (socket) {
this.socket = socket;
};
var socket = require('socket.io').listen(4000);
socket.sockets.on('connection', function (socket) {
var movie = bonsai.run(null, {
code: bonsaiCode,
plugins: []
});
movie.runnerContext.on('message', function () {
socket.emit('message', arguments);
});
movie.on('message', function (msg) {
movie.runnerContext.notifyRunner(msg);
});
socket.on('disconnect', function () {
movie.destroy();
});
});
// package.json
{
name: "roger-rabbit",
version: "0.0.0",
main: "server.js",
dependencies: {
"bonsai": "git+ssh://git@github.com:uxebu/bonsai.git",
"socket.io": "~0.9.10"
}
}
Here we are using the node modules socket.io and bonsai, which
we’re grabbing straight from the source. (Note: npm install bonsai
will return a different module.) Socket.io is responsible for keeping the
connection live and then just being the telephone wire transfering
messages from the runner to the renderers.
Bonsai is a young, but active project and community. Learn more, get involved, and stay connected.
I look forward to seeing what you create with Bonsai!
Find shared photos in real-time with Teleportd -
Teleportd is providing contestants unlimited access to their photo api for the competition. Click the link above to see how it works.
This is the 10th in a series of posts leading up to Node.js Knockout on debugging node processes using Node Inspector. This post was written by Node Knockout judge and Node Inspector author Danny Coates.
While node has a built in debugger, Node Inspector provides a pleasant graphical interface for debugging node programs.
Node Inspector is a debugger interface for node.js using the WebKit Web Inspector, the familiar javascript debugger from Safari and Chrome.
With npm:
npm install -g node-inspector
To use node-inspector, enable debugging on the node you wish to debug. You can either start node with a debug flag like:
$ node --debug your/node/program.js
or, to pause your script on the first line:
$ node --debug-brk your/short/node/script.js
NOTE: make sure that the --debug flag comes before your/node/program.js
or else you may see an EADDRINUSE error.
Or you can enable debugging on a node that is already running by sending it a signal:
Get the PID of the node process using your favorite method. pgrep
or ps -ef are good
$ pgrep -l node
2345 node your/node/server.js
Send it the USR1 signal
$ kill -s USR1 2345
Great! Now you’re ready to attach node-inspector.
start the inspector. I usually put it in the background
$ node-inspector &
open http://127.0.0.1:8080/debug?port=5858 in your favorite WebKit based browser
you should now see the javascript source from node. If you don’t, click the scripts tab.
select a script and set some breakpoints (far left line numbers) or simply
add a debugger call in your code (node will break automatically on the
call, just as V8 does).
then watch the slightly outdated but hilarious screencasts
node-inspector works almost exactly like the web inspector in Safari and
Chrome. Here’s a good overview of the UI.
I don’t see one of my script files in the file list.
try refreshing the browser (F5 or ⌘-r or control-r)
My script runs too fast to attach the debugger.
add a
debuggerstatement in your code where you want the debugger to stopor use
--debug-brkto pause the script on the first line
Can I debug remotely?
Yes. node-inspector needs to run on the same machine as the node process, but your browser can be anywhere. Just make sure the firewall is open on 8080
I got the ui in a weird state.
when in doubt, refresh
This is the 9th in a series of posts leading up to Node.js Knockout on using Passport. This post was written by Node Knockout judge and Passport creator Jared Hanson.
Passport is middleware for Node.js that makes implementing authentication quick and easy. Authentication can take a variety of forms, including a username and password, OpenID, OAuth, and BrowserID.
Many users prefer to sign in using an existing social network account, such as Facebook or Twitter. In this example, we’ll implement support for signing in with Twitter.
$ npm install express
$ npm install passport
$ npm install passport-twitter
$ npm install connect-ensure-login
We’ll be building on the simple Express application that TJ Holowaychuk built in an earlier post. Passport has a modular architecture that breaks authentication mechanisms into strategies (in this case Twitter) which are distributed separately, keeping the core lightweight. We’ll also use connect-ensure-login to protect authenticated routes.
The following three items need to be checked off to implement authentication.
We’ll cover each of them, starting from outside in: routes to middleware.
Let’s add a route to /account, which shows a person their account details.
app.get('/account',
ensureLoggedIn('/login'),
function(req, res) {
res.send('Hello ' + req.user.username);
});
In order to access this page, a user will need to be logged in. ensureLoggedIn
will verify that this is the case, and if not, redirect the user to the login
page. Which looks like:
app.get('/login',
function(req, res) {
res.send('<html><body><a href="/auth/twitter">Sign in with Twitter</a></body></html>');
});
Simple. The user can just click a link and sign in with Twitter. Which brings us to the next step in our implementation:
Twitter authentication uses OAuth, which means you’ll need to obtain a key and secret from Twitter. If you don’t have one yet, you’ll need to register your application with Twitter.
Once you have keys, configure the Twitter authentication strategy:
var TWITTER_CONSUMER_KEY = "INSERT_KEY_HERE";
var TWITTER_CONSUMER_SECRET = "INSERT_SECRET_HERE";
passport.use(new TwitterStrategy({
consumerKey: TWITTER_CONSUMER_KEY,
consumerSecret: TWITTER_CONSUMER_SECRET,
callbackURL: "http://127.0.0.1:3000/auth/twitter/callback"
},
function(token, tokenSecret, profile, done) {
// NOTE: You'll probably want to associate the Twitter profile with a
// user record in your application's DB.
var user = profile;
return done(null, user);
}
));
The function supplied to the strategy is known as a “verify callback”. Verify
callbacks receive credentials as arguments (in this case the token, tokenSecret
and profile), which are used to locate and return user records. The user
instance returned will be logged in and set on the request at req.user.
In most applications, you’ll want to associate the Twitter account with a user record in your application’s database. This allows you to also associate other accounts (such as Facebook) with the same user, allowing them to log in using either service. To keep this example simple, we will use the profile data directly, avoiding the need for database associations.
The OAuth protocol used by Twitter involves a two-step process using redirects to exchange and verify tokens. This is fairly complicated, but Passport middleware makes it easy. Just drop in the following routes:
app.get('/auth/twitter', passport.authenticate('twitter'));
app.get('/auth/twitter/callback', passport.authenticate('twitter', { successReturnToOrRedirect: '/', failureRedirect: '/login' }));
The first route will begin an OAuth transaction and redirect the user to Twitter. Once signed in, Twitter will redirect the user back to our application and Passport will return them to the original page they requested (or ‘/’).
Easy enough, but there’s still one more thing to do:
In order to keep track of the fact that a user has logged in, an application needs to implement support for sessions. Do that by using Express’ built-in cookie parser and session middleware, and initializing Passport.
app.use(express.cookieParser());
app.use(express.session({ secret: 'keyboard cat' }));
app.use(passport.initialize());
app.use(passport.session());
When a user logs in, the user record is stored in the session in order to maintain the logged-in state as she browses your site. Serialization and deserialization functions are supplied to control this process.
passport.serializeUser(function(user, done) {
done(null, user);
});
passport.deserializeUser(function(obj, done) {
done(null, obj);
});
As noted above, if you are creating user records in your own database, you can serialize just the user ID to minimize the amount of data stored in the session. For simplicity, the entire record is serialized in this example.
That’s it, people can now sign in with Twitter! The complete code for this example is available as a gist.
If you take authentication further, consult guide for in-depth details on how Passport operates. Additionally, if you need to implement API authentication, see Passport’s sister projects: OAuthorize and OAuth2orize
This is the 8th in a series of posts leading up to Node.js Knockout on using restify. This post was written by Node Knockout judge and restify creator Mark Cavage.
restify is a node.js module purpose built to create REST web services in Node. restify makes lots of the hard problems of building such a service, like versioning, error handling and content-negotiation easier. It also provides built in DTrace probes that you get for free to quickly find out where your application’s performance problems lie. Lastly, it provides a robust client API that handles retry/backoff for you on failed connections, along with some other niceties.
Like everything else in node, install restify through npm.
npm install restify
The current stable version stream is 1.4.x.
restify, on the surface, looks very similar to express, which itself looks similar to sinatra, which looks similar to J2EE’s Intercepting Filter, which I’m sure looks similar to something that came before it. Basically, you define your app by mapping URLs to a list of functions. For example, here’s an “echo” server in restify:
var restify = require('restify');
var server = restify.createServer();
server.get('/echo/:name', function (req, res, next) {
res.send({name: req.params.name});
next();
});
server.listen(8080, function () {
console.log('%s listening at %s', server.name, server.url);
});
Fire up that server with node:
$ node echo.js
restify listening at http://0.0.0.0:8080
And then we can call it with curl (note these examples also assume you’ve installed Trent Mick’s jsontool, which is oh so awesome for working with JSON REST services):
$ curl -isS localhost:8080/echo/mark | json
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 15
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
Access-Control-Allow-Methods: GET
Access-Control-Expose-Headers: X-Api-Version, X-Request-Id, X-Response-Time
Connection: Keep-Alive
Content-MD5: YWrpZVss8txd0nkH4yISjA==
Date: Wed, 31 Oct 2012 03:00:37 GMT
Server: restify
X-Request-Id: 223ca389-5ee4-4376-89ea-c8a13d0af033
X-Response-Time: 0
{
"name": "mark"
}
Cool. Note above we had a URL parameter (mark), and we just sent it back. By default, restiy assumes JSON services generally (you can override this to anything, but for NKO you probably want to stick with JSON).
Beyond “hello world”, let’s take a look at a more “complete” echo server. Note here I’m also including bunyan to show how you can set up both debug logging and audit logging:
var bunyan = require('bunyan');
var restify = require('restify');
var log = bunyan.createLogger({
name: 'my_restify_application',
level: process.env.LOG_LEVEL || 'info',
stream: process.stdout,
serializers: bunyan.stdSerializers
});
var server = restify.createServer({
log: log,
name: 'my_restify_application'
});
server.use(restify.acceptParser(server.acceptable));
server.use(restify.authorizationParser());
server.use(restify.dateParser());
server.use(restify.queryParser());
server.use(restify.bodyParser());
server.use(restify.throttle({
burst: 100,
rate: 50,
ip: true, // throttle based on source ip address
overrides: {
'127.0.0.1': {
rate: 0, // unlimited
burst: 0
}
}
}));
server.on('after', restify.auditLogger({ log: log }));
server.use(function authenticate(req, res, next) {
// call redis or something here
next();
});
// this one will be explained in the next section
server.use(function slowPoke(req, res, next) {
setTimeout(next.bind(this), parseInt((process.env.SLEEP_TIME || 0), 10));
});
server.post('/echo/:name', function echoParms(req, res, next) {
req.log.debug(req.params, 'echoParams: sending back all parameters');
res.send(req.params);
next();
});
server.listen(8080, function () {
log.info('%s listening at %s', server.name, server.url);
});
Let’s fire this one up, and set the log level to debug (also, pipe the output to the bunyan formatter so we get “pretty printed” logs):
$ npm install bunyan
$ LOG_LEVEL=debug node echo.js | ./node_modules/.bin/bunyan
my_restify_application listening at http://0.0.0.0:8080
Now go ahead and invoke curl again (note this time we changed the method to a POST so we can send some data over):
$ curl -isS localhost:8080/echo/mark?q1=foo\&q2=bar \
-X POST -H content-type:application/json \
--data-binary '{"body_param": "baz"}' | json
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 56
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
Access-Control-Allow-Methods: POST
Access-Control-Expose-Headers: X-Api-Version, X-Request-Id, X-Response-Time
Connection: Keep-Alive
Content-MD5: J1dujES7TARwUfsRQNMDhw==
Date: Wed, 31 Oct 2012 03:17:23 GMT
Server: my_restify_application
X-Request-Id: 3a8404f5-c5be-45d0-a4be-38f17c9e60d1
X-Response-Time: 1
{
"name": "mark",
"q1": "foo",
"q2": "bar",
"body_param": "baz"
}
This time, we should also have seen the following information kick back from the server console:
[2012-10-31T03:17:23.853Z] DEBUG: mark/51296 on bluesnoop: echoParams: sending back all parameters (req_id=3a8404f5-c5be-45d0-a4be-38f17c9e60d1, route="POST /echo/:name", q1=foo, q2=bar, body_param=baz)
[2012-10-31T03:17:23.856Z] INFO: my_restify_application/51296 on bluesnoop: POST /echo/:name handled: 200 (req_id=3a8404f5-c5be-45d0-a4be-38f17c9e60d1, 4ms, audit=true, remoteAddress=127.0.0.1, remotePort=50996, secure=false, _audit=true, req.version=*)
POST /echo/mark?q1=foo&q2=bar HTTP/1.1
user-agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
host: localhost:8080
accept: */*
content-type: application/json
content-length: 21
--
HTTP/1.1 200 OK
content-type: application/json
content-length: 56
access-control-allow-origin: *
access-control-allow-headers: Accept, Accept-Version, Content-Length, Content-MD5, Content-Type, Date, X-Api-Version
access-control-allow-methods: POST
access-control-expose-headers: X-Api-Version, X-Request-Id, X-Response-Time
connection: Keep-Alive
content-md5: J1dujES7TARwUfsRQNMDhw==
date: Wed, 31 Oct 2012 03:17:23 GMT
server: my_restify_application
x-request-id: 3a8404f5-c5be-45d0-a4be-38f17c9e60d1
x-response-time: 4
--
route: {
"name": "POST /echo/:name",
"version": false
}
Note that if we didn’t pipe the output to the bunyan formatter, we would have had logs in pure JSON, making it easier to mine later.
Note: this section is only applicable if you are on Mac OS X or Joyent SmartOS. You’ll want 3 shell sessions open for this.
Ok, so above I added a slowPoke handler:
server.use(function slowPoke(req, res, next) {
setTimeout(next.bind(this), parseInt((process.env.SLEEP_TIME || 0), 10));
});
Which notably lets us pass in an arbitrary amount of time to pause. Let’s one more time fire up the server but with a lag in it (also, let’s skip the bunyan formatter so you see what I mean about JSON logs):
$ SLEEP_TIME=500 node echo.js
{"name":"my_restify_application","hostname":"bluesnoop","pid":51387,"level":30,"msg":"my_restify_application listening at http://0.0.0.0:8080","time":"2012-10-31T03:29:01.627Z","v":0}
Great. Now in another terminal, start running loop of our curl commands (put it
in () so you can ctrl-c sanely):
$ (while [ true ] ; do curl -isS localhost:8080/echo/mark?q1=foo\&q2=bar -X POST -H content-type:application/json --data-binary '{"body_param": "baz"}' ; done)
You should see it going pretty slowly, seeing as we have a sleep in there. Now, lets go find that sleep from DTrace. In a third window run this:
$ sudo dtrace -lP my_restify_application*
ID PROVIDER MODULE FUNCTION NAME
6168 my_restify_application52121 module func postechoname-start
6169 my_restify_application52121 module func postechoname-done
6170 my_restify_application52121 module func postechoname-parseAccept-start
6171 my_restify_application52121 module func postechoname-parseAccept-done
6172 my_restify_application52121 module func postechoname-parseAuthorization-start
6173 my_restify_application52121 module func postechoname-parseAuthorization-done
6174 my_restify_application52121 module func postechoname-parseDate-start
6175 my_restify_application52121 module func postechoname-parseDate-done
6176 my_restify_application52121 module func postechoname-parseQueryString-start
6177 my_restify_application52121 module func postechoname-parseQueryString-done
6178 my_restify_application52121 module func postechoname-parseBody-start
6179 my_restify_application52121 module func postechoname-parseBody-done
6180 my_restify_application52121 module func postechoname-rateLimit-start
6181 my_restify_application52121 module func postechoname-rateLimit-done
6182 my_restify_application52121 module func postechoname-authenticate-start
6183 my_restify_application52121 module func postechoname-authenticate-done
6184 my_restify_application52121 module func postechoname-slowPoke-start
6185 my_restify_application52121 module func postechoname-slowPoke-done
6186 my_restify_application52121 module func postechoname-echoParms-start
6187 my_restify_application52121 module func postechoname-echoParms-done
Note those are all of our handlers. Nice, huh? Now, without going into the details of DTrace, here’s a nice D script you can run that will show you latency broken down by handler:
#!/usr/sbin/dtrace -s
#pragma D option quiet
my_restify_application*:::postechoname-*-start
{
tracker[arg0, substr(probename, 0, rindex(probename, "-"))] = timestamp;
}
my_restify_application*:::postechoname-*-done
/tracker[arg0, substr(probename, 0, rindex(probename, "-"))]/
{
this->name = substr(probename, 0, rindex(probename, "-"));
@[this->name] = quantize(((timestamp - tracker[arg0, this->name]) / 1000000));
tracker[arg0, substr(probename, 0, rindex(probename, "-"))] = 0;
}
Go ahead and run that for a few seconds (let’s say 5-10):
$ chmod +x ./echo.d
$ sudo ./echo.d
^C
postechoname-authenticate
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-parseAccept
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-parseAuthorization
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-parseBody
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-parseDate
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-parseQueryString
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-rateLimit
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
1 | 0
postechoname-echoParms
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 10
1 | 0
postechoname-slowPoke
value ------------- Distribution ------------- count
128 | 0
256 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 9
512 | 0
And we see that our latency is sorted according to “worst”; at the bottom
is our slowPoke handler, chewing up time while everything else is running
< ms. With this it’s pretty obvious what you should go look at.
Obviouly real applications have a lot more moving parts than what was shown here, but hopefully you have a grasp on the basics of restify are. I’ve left out the details on lots of complex parts of it; like any dependency you’ll have to read the docs a bit more thoroughly to get into more advanced uses. But if you’re looking to build a well-behaved REST application, this should help get you going.
This is the 7th in a series of posts leading up to Node.js Knockout on using Nodemailer. This post was written by Node Knockout judge and Nodemailer creator Andris Reinman.
Install Nodemailer
> npm install nodemailer
Include the module in your script and create a reusable transport object
var nodemailer = require("nodemailer");
var smtpTransport = nodemailer.createTransport("SMTP",{
service: "Gmail",
auth: {
user: "gmail.user@gmail.com",
pass: "gmailpass"
}
});
Send an e-mail using the connection object
smtpTransport.sendMail({
from: "My Name <me@example.com>", // sender address
to: "Your Name <you@example.com>", // comma separated list of receivers
subject: "Hello ✔", // Subject line
text: "Hello world ✔" // plaintext body
}, function(error, response){
if(error){
console.log(error);
}else{
console.log("Message sent: " + response.message);
}
});
Send another e-mail without caring about the outcome (or add the callback function as well, like in the previous code block, if you do care)
smtpTransport.sendMail({
from: "My Name <me@example.com>", // sender address
to: "Your Name <you@example.com>", // comma separated list of receivers
subject: "Hello ✔", // Subject line
text: "Hello world ✔" // plaintext body
});
For additional details see Nodemailer documentation or read through this article.
There are several ways to send an e-mail but in the end it all comes down to the SMTP protocol (Simple Mail Transfer Protocol), since this is the method how e-mails are actually transferred in the interwebs. There have been some other propositions as well in the past, like ISO standardized X.400 for example but these never took off and thus don’t really matter for most.
The reason why you don’t just open a TCP connection to the recipients incoming SMTP server (the one that is found from the domains DNS MX record and listening on port 25), transmit the message and be done with it, is that in this way, most probably, your message ends up in the junk folder or gets lost entirely - for one, you are not a trusted sender, your sending volumes are way too low (what an irony). This behaviour is also not optimal, as your application would have to handle e-mail bounces, errors, greylisting etc. The better way instead is to turn to your Mail Submission Agent (MSA), which also speaks SMTP and for an exchange of your credentials, happily queues your message for delivery by a relaying Mail Transfer Agent (MTA), which in turn, also speaks SMTP. Often MSA and MTA are one and the same but in theory they are meant to perform somewhat different tasks.
SMTP is a textual protocol and looks a bit like this:
CLIENT: <initiates connection to the server>
SERVER: 220 bob the mailcat listening
CLIENT: HELO joe.the.sender
SERVER: 250 Hello, joe.the.sender, nice to meet you!
CLIENT: MAIL FROM:<me@example.com>
SERVER: 250 Ok, proceed with recipients
...
The numbers returned from the server are status codes, anything equal or above 400 is an error (HTTP anyone?)
While SMTP defines the transport of the envelope (from the sender’s computer to the recipients inboxes), the contents of the envelope is a completely different thing. Actually the two are so unrelated, you can set whatever you want to as the message contents (you can spoof “from” and “to” addresses in the mail header, you can set a future or prehistoric date for the message, you can use strange byte sequences or use an invalid format etc.) and it still gets delivered. Depending on the MSA, some of the data might be corrected though (eg. the prehistoric date of yours might be adjusted to a more recent value) but not all of it.
It’s exactly as with regular mail - you have an envelope that says where the letter is supposed end up and then you have the envelope contents that can contain a valid letter, addressed to the same person labeled on the envelope; or some money without an explanation; or some barb wire; or instead a letter addressed someone totally unrelated from the person on the label of the envelope. If you don’t have the envelope you can never be sure just based on the included letter from where the message originates and to whom it was sent.
Thats why when you look at a spam mail you often see that the recipient name and address fields are blank or invalid - the e-mail client program takes this information from the message contents and not from the envelope. Luckily most servers prepend some of the envelope information also to the mail header, so if you look at the raw source of the message, the first line, “Delivered-To” says which actual address was the final receiver. With several mail accounts ending up in the same e-mail client box, you might not be always able to tell it by yourself. That is also how BCC works - the name is on the envelope but not in the message.
The most simple e-mail message only consists of header and ascii content
From: me@example.com
To: you@example.com
Subject: Hello
How are you doing?
If you’ve ever done HTTP, then the structure should seem familiar. There’s a header block and after two consecutive line breaks (NB! all line breaks in the interwebs protocols use the form <CR><LF> (eg. the “windows style”) and not the <LF> (the “unix style”) form) comes the body of the message.
Nontrivial E-mail messages (the ones that want to use non-ascii characters or attachments or almost any other feature) are formatted by MIME which stands for Multipurpose Internet Mail Extensions. You can include multiple “views” of the same data for the message (ie. HTML view and plaintext view), add binary attachments, use different character sets and so on but eventually it is all compiled to a single ascii-only file before it is being sent as the e-mail. The compiled raw source was meant to be human readable on ancient 80 column terminals but with multipart messages (attachments, different “views” etc), it is not a pleasant exercise to browse through - all “extensions” are added on top of the original plain format, making up an onion-like structure. For example one horror child of such extending is the way to support long unicode filenames etc. for attachments (RFC2231) that uses urlencoding instead of mime encoding.
While most of the strings in e-mail headers are encoded in MIME encoded-word, like the following
Subject: =?us-ascii?Q?This_is_even_more_***fun***_isn't_it!?=
Then here’s an excerpt from the RFC2231 for splitting long header values:
Content-Type: application/x-stuff
title*0*=us-ascii'en'This%20is%20even%20more%20
title*1*=%2A%2A%2Afun%2A%2A%2A%20
title*2="isn't it!"
No, it is not fun.
To keep you away from the tedious task of mingling together standard compliant e-mail messages that would go through the wire, enter e-mail sending libraries. There are some for every programming language and for Node.js, there’s Nodemailer by yours truly or emailjs by eleith. These, of course, are not the only options, there used to be (now deprecated) node_mailer by Marak Squires and there still is pony by substack, there’s probably even more but for feature completeness I would definitely suggest using either Nodemailer or emailjs. As the author of Nodemailer I’m not going to describe the other modules but these are still worth checking out.
Installing Nodemailer follows the common path - you can install it from npm. Add a dependency to your package.json and run npm install
package.json contents:
...
"dependencies": {
"nodemailer": "0.3.29"
},
...
NB! If you don’t have a package file yet, you can create one automatically by running npm init.
Now update your installation by running in the same directory the package.json resides.
npm install
If you are on Windows, you are going to see some error messages flying through while installing but this is normal and doesn’t affect how the module is going to work in the end. Nodemailer tries to drag iconv module to the party as an optional dependency and iconv needs compiling. The iconv module isn’t even used, since all the output and input strings for Nodemailer are always in UTF-8. I just haven’t found a good way to automatically exclude the iconv module since the other modules that share the same codebase (eg. MailParser) do need it.
And as the last part of the installation equation, include Nodemailer in your script file
var nodemailer = require("nodemailer");
While SMTP makes up the barebones of e-mail messaging, you don’t actually have to use it directly if your MSA accepts something else, for example HTTP POST request with e-mail file etc. Nodemailer includes 3 different mail transports and if this doesn’t suit you, you are welcomed to add your own.
sendmail commandIn this article we only look at using SMTP as it is the most important of them all.
To use SMTP you need to have access to a MSA server. If you have a Gmail or Google Apps account, you can use Gmail SMTP for this but you can’t ride it far as there are strict usage limits (about 100 emails per day, unless you have a Google Apps Business account that has higher limits), besides you can only send mail from the same user you are authenticated as, you can’t change it to something else (actually you can change the address but it is displayed as a nasty “via authenticated.user@gmail.com” address, not good for your business if you want it to show just “billing@mycompany.com” instead). If you need to go any further, consider using Sendgrid or Postmark or Mailgun or Amazon SES or any other professional message delivery service. There’s a great discussion in Quora about the reasons behind pricing of these services. If you don’t want to send hundreds of millions messages, it is usually more feasible to pay one of the providers than building your own e-mail infrastructure. But getting started is totally free - you can use Gmail as a starting point and move on to a free trial or a developer account of another service before you need to pay a cent.
Lets say that for a starters you’re going to use Gmail as the SMTP service. Setting up Nodemailer is a piece of cake in this case, you just need to have your username and password available:
var smtpTransport = nodemailer.createTransport("SMTP",{
service: "Gmail",
auth: {
user: "gmail.user@gmail.com",
pass: "gmailpass"
}
});
And that’s it, you now have a working wire for transmitting e-mails.
NB! If your mail configuration needs go beyond Gmail, you can find the full spectre of connection settings from the Nodemailer documentation.
The problem with SMTP is that you need to have plaintext passwords available somewhere for the app. In case of Gmail there are actually several other options as well. If you use 2 factor authentication, then you can generate application specific password for your mailing app without compromising your real password. There’s also a way to use OAuth1.0 and OAuth2.0 tokens with Gmail if you want to skip using passwords at all. See Nodemailer documentation for details.
Now that we have the working wire, we can start sending e-mails. The simplest form of e-mail requires sender address, recipients address, subject line and message text which can be formatted like this:
var mailOptions = {
from: "me@example.com", // sender address
to: "you@example.com", // list of receivers
subject: "Hello ✔", // Subject line
text: "Hello world ✔" // plaintext body
}
If the e-mail structure has been composed, we can send the mail using the “transport wire” defined earlier
smtpTransport.sendMail(mailOptions, function(error, response){
if(error){
console.log(error);
}else{
console.log("Message sent: " + response.message);
}
});
Nodemailer transport object accepts a large number of messages to be sent at once, you don’t have to wait for the first message to be delivered before you can proceed to the following. Only a limited count of connections are opened to the SMTP server (Nodemailer uses built-in connection pooling) at once and messages are queued automatically for delivery. Thus you can initiate
sendMailjust as well from aforloop, no need for a complex async setup or a callback hell.
for(var i=0; i<mailArray.length; i++){
smtpTransport.sendMail(mailArray[i]);
}
Usually you want to include the name of the sender or receiver in addition to the plain address, or sometimes you want to send the e-mail to not just one but several recipients. Nodemailer recipient fields (“to”, “cc”, “bcc”) accept comma separated addresses for multiple recipients and the name can be formatted as well (remeber to enclose the name in double quotes if it contains a comma).
...,
from: "\"Name, User\" <user.name@gmail.com>",
to: "Receiver Name <receiver@gmail.com>, plain@example.com, \"Name, Another\" <another@gmail.com>"
...
And as with any other field, unicode is allowed - you can use non-ascii characters both for the names and for the domains, unicode e-mail domains are converted to the punycode form automatically.
Nodemailer supports a lot of advanced options, like DKIM signing or OAuth2.0 token generation and we are not going to cover all of these here, see Nodemailer docs instead for all the details about these features.
Usually the e-mail contains either HTML view (usually the default), plain text view or both. It is a good practice to include plaintext view in addition to the HTML view, so the less-able mail clients could fall back. Nodemailer can generate the plaintext view automatically, based on the HTML view but you can create it yourself if you want to (or skip it altogether, as only plaintext capable clients are pretty rare nowadays).
You can use both html and text views by specifying according properties of the e-mail object.
...,
text: "plaintext contents",
html: "<p>HTML contents</p>",
...
Keep in mind that one of the main differences between text and html views is that newlines are ignored in the html view but not in the plaintext view when rendering the contents.
You can generate the text view automatically when using generateTextFromHTML property
...,
html: "<p>HTML contents</p>",
generateTextFromHTML: true,
...
Or if you don’t want to include the text contents at all, use only html property
...,
html: "<p>HTML contents</p>",
...
Just whatever you do, do not use a plaintext view that only states “Your e-mail client is not capable of displaying this e-mail” - a lot of e-mail clients use the plaintext part (if it exists) as the preview for the message and html part for the actual viewing.
Attachments are defined as an array of attached files. You can use a variety of data sources for the attachments (files on disk, strings, buffers, streams, even web urls) and for these examples we are using files on disk. See Nodemailer documentation for the other options.
The most simple form of attachments are file paths on disk. Attachment filename and content type are derived automatically in this case (although you can override these values if you want to).
...,
attachments: [
{filePath: __dirname + "/attachment1.txt"},
{filePath: __dirname + "/attachment2.txt"}
],
...
When using images as attachments, you can point to these images as embedded files for the HTML view. You need to define unique cid value for the attachment and use it as an URL.
...,
html: "<p>Embedded image: <img src='cid:image1'/></p>",
attachments:[
{filePath: __dirname + "/image1.jpg", cid: "image1"}
],
...
Using an e-mail library like Nodemailer makes your life so much easier if you want to send an e-mail with Node.js. Sure, you can compose the message by yourself and transmit it using the standard Node.js net module (and if you’re really interested how the e-mail system works, then this is exactly the way you’re going to do it) but most probably you just need to send an e-mail to your customer and you don’t really care how it is delivered, as long as it makes it through and displays as intended.
If you want to bring your e-mail related application to the next level, you could also start receiving and parsing e-mail in addition to sending it. For creating your own SMTP MX server for accepting e-mail from the interwebs, you can use simpler SimpleSMTP by yours truly or full featured Haraka by baudehlo which is able to fight against spam as well. If you do not want to manage message exchange server by yourself but to use IMAP for connecting to an existing one, you can use node-imap by mscdex or inbox. Once you have received the e-mail (either through SMTP or IMAP), you can parse the raw source into a structurized object wih MailParser. I have also created a helper module mailuploader to parse an e-mail raw source and post the parsed object to a HTTP address as a regular multipart/form-data upload, so even a PHP script could easily handle it.
I hope that by reading this article you found out how easy it is to send e-mails using Nodemailer. And if you ever run into trouble with it, then you know where you can file an issue.
This is the 6th in a series of posts leading up to Node.js Knockout on using hapi. This post was written by Node Knockout judge and hapi contributor Wyatt Preul.
Hapi is a framework for rapidly building RESTful web services. Whether you are building a very simple set of RESTful services or a large scale, cache heavy, and secure set of services, hapi has you covered. Hapi will help get your server developed quickly with its wide range of configurable options.
The following example will walk you through using hapi to build a RESTful set of services for creating and listing out products. To get started create a directory named ProductsAPI and add a package.json file to the directory that looks like the following.
{
"name": "ProductsAPI",
"version": "0.0.1",
"main": "server",
"engines": {
"node": ">=0.8.0"
},
"dependencies": {
"hapi": "0.8.x"
},
"private": "true"
}
Then run npm install.
Create a server.js file that will serve as the entry point for the service. Add the following contents to the server.js file.
var hapi = require('hapi');
var routes = require('./routes');
var config = { docs: true };
var http = new hapi.Server('0.0.0.0', 8080, config); // 8080 is the port to listen on
http.addRoutes(routes);
http.start();
In the server.js code above a new instance of the hapi server is started using the configuration specified in config.
By setting docs to true the documentation generator will be enabled. The documentation generator provides a set of pages that explain what endpoints are available and the requirements for those endpoints. The documentation generator will use the validation rules you will create for each route to construct appropriate documentation pages under the /docs path.
Hapi provides a function for adding a single route or an array of routes. In this example we are adding an array of routes from a routes module.
Go ahead and create a routes.js file, which will contain the route information and handlers. When defining the routes we will also be specifying validation requirements. Therefore, at the top of the file require hapi and assign its Types property to a local variable like below.
var Types = require('hapi').Types;
For this example three routes will be created. Below is the code you should use to add the routes. Add the following code to your routes.js file.
module.exports = [
{ method: 'GET', path: '/products', config: { handler: getProducts, query: { name: Types.String() } } },
{ method: 'GET', path: '/products/{id}', config: { handler: getProduct } },
{ method: 'POST', path: '/products', config: { handler: addProduct, payload: 'parse', schema: { name: Types.String().required().min(3) }, response: { id: Types.Number().required() } } }
];
The routes are exported as an array so that they can easily be included by the server implementation we added. For the products listing endpoint we are allowing a querystring parameter for name. When this querystring parameter exists then we will filter the products for those that have a matching name.
The second route is a very simple route that demonstrates how a parameter can become part of the path definition. This route will return a product matching the ID that’s requested.
In the last route, the one used for creating a product, you will notice that extra validation requirements are added, even those on the response body. The request body must contain a parameter for name that has a minimum of 3 characters and the response body must contain an ID to be validated.
Next add the handlers to the routes.js file.
function getProducts(request) {
if (request.query.name) {
request.reply(findProducts(request.query.name));
}
else {
request.reply(products);
}
}
function findProducts(name) {
return products.filter(function(product) {
return product.name.toLowerCase() === name.toLowerCase();
});
}
function getProduct(request) {
var product = products.filter(function(p) {
return p.id == request.params.id;
}).pop();
request.reply(product);
}
function addProduct(request) {
var product = {
id: products[products.length - 1].id + 1,
name: request.payload.name
};
products.push(product);
request.reply.created('/products/' + product.id)({
id: product.id
});
}
As you can see in the handlers, hapi provides a simple way to add a response body by using the request.reply function. Also, in the instance when you have created an item you can use the request.reply.created function to send a 201 response.
Lastly, add a simple array to contain the products that the service will serve.
var products = [{
id: 1,
name: 'Guitar'
},
{
id: 2,
name: 'Banjo'
}
];
Go ahead and run npm start or node server.js to start the server. Now you
can navigate to http://localhost:8080/docs to see the documentation for the
routes. To see a list of the products navigate to
http://locahost:8080/products. Below is a screenshot of what the response
looks like.
Go ahead and append ?name=banjo to the URL to try searching for a product by
name.
Use curl or a REST console to create a product. Make a POST request to the
products endpoint with a name in the body. Using curl the command looks like:
curl http://localhost:8080/products -d "name=test". Below is an example of
the response headers from making a request to create a product.
Now if you navigate to the Location specified in the response headers you should see the product that you created.
There are a lot of different configuration features that you can add to the server. The extensive list can be found in the readme at https://github.com/walmartlabs/hapi/#server-configuration.
The built-in cache support has providers for mongo and redis. Setting up cache is as simple as passing cache: true as part of the server configuration.
Additionally, there are several configuration options available on a per route basis. The full list can be found at https://github.com/walmartlabs/hapi/#route-configuration. For example, caching expiration times can also be configured on a per route basis. Also, you can have per-route authentication settings.
By now you should have a decent understanding of what hapi has to offer. There are still many other features and options available to you when using hapi that is covered in the documentation. Please take a look at the github repository and feel free to provide any feedback you may have.
This is the 5th in a series of posts leading up to Node.js Knockout on using Mongoose. This post was written by Node Knockout judge and Mongoose maintainer Aaron Heckmann.
In this post we’ll talk about getting started with Mongoose, an object modeling tool for MongoDB and node.js.
We’re going to assume that you have both MongoDB and npm installed for this post. Once you have those, you can install Mongoose:
$ npm install mongoose
Hurray! Now we can simply require mongoose like any other npm package.
var mongoose = require('mongoose');
Though MongoDB is a schema-less database we often want some level of control over what goes in and out of our database collections. We’re confident that we’re going to be the next Netflix so we’ll need a Movie schema and a Rating schema. Each Movie is allowed to have multiple Ratings.
var Schema = mongoose.Schema;
var RatingSchema = new Schema({
stars : { type: Number, required: true }
, comment : { type: String, trim: true }
, createdAt: { type: Date, default: Date.now }
});
So far we’ve created a Rating schema with a stars property of type Number, a comment property of type String, and a createdAt property of type Date. Whenever we set the stars property it will automatically be cast as a Number. Note also that we specified required which means validation will fail if an attempt is made to save a rating without setting the number of stars. Likewise, whenever we set the comment property it will first be cast as a String before being set, and since whitespace around comments is very uncool, we use the built-in trim setter.
Now that we’re happy with our Rating model we’ll use it within our Movie model. Each movie should have name, director, year, and ratings properties.
var MovieSchema = new Schema({
name : { type: String, trim: true, index: true }
, ratings : [RatingSchema]
, director: Schema.ObjectId
, year : Number
});
Here we see that ratings is set to an array of Rating schemas. This means that we’ll be storing Ratings as subdocuments on each Movie document. A subdocument is simply a document nested within another.
You might have noticed the index option we added to the name property. This tells MongoDB to create an index on this field.
We’ve also defined director as an ObjectId. ObjectIds are the default primary key type MongoDB creates for you on each document. We’ll use this as a foreign key field, storing the document ObjectId of another imaginary Person document which we’ll leave out for brevity.
TIP: Note that we needed to declare the subdocument Rating schema before using it within our Movie schema definition for everything to work properly.
This is what a movie might look like within the mongo shell (included when you install MongoDB):
{ name: 'Inception',
year: 2010,
ratings:
[ { stars: 8.9,
comment: 'I fell asleep during this movie, and yeah, you\'ve heard this joke before' },
{ stars: 9.3 } ],
director: ObjectId("4e4b4a8b73e1d576d6a1438e") }
Now that we’ve finished our schemas we’re ready to create our movie model.
var Movie = mongoose.model('Movie', MovieSchema);
And thats it! Everything is all set with the exception of being able to actually talk to MongoDB. So let’s create a connection.
mongoose.connect('mongodb://localhost/nodeknockout', function (err) {
if (err) return handleErrorSomehow(err);
// ok we're set
});
Now we’re ready to create a movie and save it.
var movie = new Movie({ name: "Frankenweenie", director: anObjectId, year: 2012 });
movie.save(function (err) {
if (err) return console.error(err); // we should handle this
});
Oh, but what about adding ratings?
Movie.findOne({ name: "Frankenweenie" }).where("year").equals(2012).exec(function (err, movie) {
if (err) // handle this
// add a rating
movie.ratings.push({ stars: 9.0, comment: "it made me happy" });
movie.save(callback);
});
To look up our movie we used Model.findOne which accepts a where clause as its first argument. We also took advantage of the Query object returned by this method to add some more sugary filtering. Finally, we called the Query’s run method to execute it.
We didn’t have to do it this way, instead you could just pass all of your where params directly as the first argument like so:
Movie.findOne({ name: "Frankenweenie", year: 2012 }, callback);
Though the first example is more verbose it highlights some of the expressive flexibility provided by the Query object returned.
Here are a couple more ways we could write this query:
Movie.where('name', /^Frankenweenie/i).where('year', 2012).limit(1).exec(callback);
Movie.find({ name: "Frankenweenie", year: { $gt: 2011, $lt: 2013 }}, null, { limit: 1 }, callback);
This is all well and good but what if we look up movies by director and year a lot and need the query to be fast? First we’ll create a static method on our Movie model:
MovieSchema.statics.byNameAndYear = function (name, year, callback) {
// NOTE: find() returns an array and may return multiple results
return this.find({ name: name, year: year }, callback);
}
We’ll also add a compound index on these two fields to give us a performance boost:
MovieSchema.index({ name: 1, year: 1 });
For good measure we’ll add a movie instance method to conveniently look up the director:
MovieSchema.methods.findDirector = function (callback) {
// Person is our imaginary Model we skipped for brevity
return this.model('Person').findById(this.director, callback);
}
Putting it all together:
Movie.byNameAndYear("Frankenweenie", 2012, function (err, movies) {
if (err) return console.error(err); // handle this
var movie = movies[0];
movie.findDirector(function (err, director) {
if (err) ...
// woot
})
});
Thats it for this post. For more info check out mongoosejs.com, the mongoose plugins site, the github README, or the Mongoose test directory to see even more examples.