This post is sponsored by Joyent, a Node Knockout 2013 Platinum sponsor.

Along with your team’s ubuntu instance, Joyent is also offering up to $130 worth* of access to Manta. If you haven’t heard of Manta yet, it’s Joyent’s object store with built in map/reduce. In other words “upload your data, then crunch on it in-place”.

Some examples of how you can use Manta:

  • Upload your server logs, do some clickstream analysis.
  • Upload photos, resize them “in place”.
  • Upload db dumps, do some data analytics, produce some graphs.

Did I mention that you can run almost anything in compute? All your favorite unix tools are already there (grep, sort, uniq, awk, etc) as well as popular scripting engines (ruby, python, etc.). Oh, and node of course!

This post will help get you started with Manta.

Signing up and getting your keys in place…

This section will help you get set up to use the Joyent Cloud, including Manta. This section will walk you through:

  1. Signing up for a Joyent Account
  2. Generating an SSH key
  3. Adding your SSH key to your Joyent account

First, sign up for the Joyent Cloud.

Access to the Joyent Cloud is done using SSH keys. If you already have an SSH key you use for other things (like the one for your ubuntu instance, for example), you can use the same key for accessing the Joyent Cloud. You can check if you’ve already generated an SSH key on a Unix-y system by:

$ cat .ssh/id_rsa.pub
ssh-rsa ...

If you don’t already have an SSH key generated, check out this great guide on github. Windows users can use this..

Once you’ve signed up and have your SSH key, you’re ready to add your SSH key to your Joyent account:

  1. Navigate to Joyent.com and Sign In using the link on the top/right of the page.
  2. Click on the “My Account” link at the top of the page.
  3. On The right-hand side of the page, click the “SSH Keys” link.
  4. Click the “Import Public Key” button.
  5. Copy/paste the contents of your public key file to the “Public Key” box. If you are using Mac OS X, this command will copy your public key to your paste buffer: cat .ssh/id_rsa.pub | pbcopy.
  6. Click “Add”.

Installing and using the Manta CLI

Joyent has put together a suite of command line tools for working with Manta. For example, mls lists a directory, mput puts a file to manta, and mjob manages compute jobs (there are others). This section shows you how to:

  1. Install the command line tools
  2. Set up your environment variables
  3. Your first mls!

Pro tip: If you want to install everything for Mac OS X in one go (node, npm, Manta tools, etc) use the Mac OS X installer.

Assuming that you’ve already installed node and npm, you can install the Manta CLI with:

$ sudo npm install manta -g

Since you’ve already set up your SSH key with Joyent, you only need to set up the following environment variables, replacing $JOYENT_CLOUD_USER_NAME with the username you used when signing up with Joyent.

$ export MANTA_URL=https://us-east.manta.joyent.com
$ export MANTA_USER=$JOYENT_CLOUD_USER_NAME
$ export MANTA_KEY_ID=$(ssh-keygen -l -f $HOME/.ssh/id_rsa.pub | awk '{print $2}')

Now you should be able to mls (~~ is shorthand for /$MANTA_USER):

$ mls ~~/
jobs/
public/
reports/
stor/

Some Manta basics

Those are the four top-level directories every user has in Manta.

Quickly:

  • jobs/ is where we store information on the jobs you’ve run (stdin, stdout, stderr, job status)

  • public/ is what the world has read access to. i.e. the Mac OS X installer mentioned above is hosted in Manta’s public directory.

  • reports/ detailed usage information of storage and compute.

  • stor/ read only for your account.

Command line tools available (each has help with --help):

  • mmkdir makes directories, mmkdir ~~/public/ebooks

  • mput puts files, mput -f dracula.txt ~~/public/ebooks/dracula.txt

  • mls lists directories, mls ~~/public/ebooks

  • mfind recursively finds files/dirs mfind -t o ~~/public

  • mrm delete files, mrm ~~/public/ebooks/dracula.txt

  • mrmdir delete (empty) directories, mrmdir ~~/public/ebooks. You can also mrm -r ~~/public/ebooks if you want to recursively remove a directory and everything in it.

  • mlogin “login” to a compute zone (drops you into a shell), giving you the environment you’ll have when running compute jobs, mlogin ~~/public/ebooks/dracula.txt

  • mjob manages jobs (create, get, list, share, etc.). For example, this will run a job to count the number of times vampire appears in Dracula, echo ~~/public/ebooks/dracula.txt | mjob create -o -m "grep -ci vampire"

For more examples of Manta usage see Joyent’s docs on Job Examples and Patterns.

Note that normal pricing applies to mlogin sessions since you are, literally, logged into a single instance of compute. If you left it running for 24 hours it’d end up costing ~ $3.50 (see Manta Pricing for details).

Accessing Manta from Node

This is for Node Knockout after all! The manta client is kept in the node-manta repository on github. The client can be found in [client.js (https://github.com/joyent/node-manta/blob/master/lib/client.js).

Don’t be scared though, usage is actually quite easy. Here’s a short example to list a directory. First, initiate a bunyan logger and the manta client:

var log = bunyan.createLogger({
    name: path.basename(process.argv[1]),
    level: (process.env.LOG_LEVEL || 'info'),
    stream: process.stdout
});
var client = manta.createBinClient({ 'log': log });

Then we’re going to ask the client to list a directory:

client.ls('/nfitch/stor', function (err, res) {

Most of the results (res above) are event emitters. The one for listing directories emits object and directory for each listing it receives from Manta.

res.on('object', function (o) {
    console.log(o.name);
});
res.on('directory', function (d) {
    console.log(d.name + '/');
});

Finally, when it is done, close the client:

res.once('end', function () {
    client.close();
});

You can check out the full example at nfitch/node-manta-demo.

To see how the client is used, I’d encourage you to go check out the source code for the m tools at node-manta/bin.

Other stuff to check out

If you have any questions, join us on #manta on freenode.

*Important details on the trial credit.

  1. You have to create a new account.
  2. the credit is broken across two monthly billing periods at up to USD$65/period. After that you’re responsible.
  3. Billing periods correspond to calendar months.
  4. Yes, we do require a credit card.
  5. we’re typically on Freenode IRC #manta and #joyent if you have questions.
Blog comments powered by Disqus