Messaging with MQTT and Javascript

I do a lot with messaging architectures, and because I work on embedded systems so much lately, my main broker protocol has been MQTT, used with Javascript. I learned something that surprised me this morning, even though it really shouldn’t have, given some thought.

MQTT is a common protocol used in IoT. It stands for “Message Queuing Telemetry Transport“, and the current specification for MQTT is 5.0. Common brokers include Eclipse Mosquitto, EMQX, and HiveMQ.

As its name implies, MQTT is designed to be super-light. It has some pretty nice features, including termination semantics, quality of service (guaranteed delivery), retention, and other facets, but the primary use is for fairly simple communication of small messages – the header has room for a “payload length” and it’s two bytes, so your maximum packet size is somewhere in the region of 64k bytes.

The thing about MQTT that got me was the subscription model. It would be convenient to have a handler per subscription:

// this code does not do what it seems to expect.
    (message) => { 
        console.log('hey, we got a message on rpc/1: ${message.toString()}'); 
    (message) => { 
        client.publish('rpc/1', `hello, ${message}`); 

As the comment says – who reads comments, right? – this code does not work. It looks like it should do something specific when a message comes in on rpc/1, and something different when a message comes in on rpc/2 – but it doesn’t. It actually solely establishes two subscriptions for the client connection and those “message handlers” will not be executed, ever, under normal circumstances.

MQTT clients in Javascript, using the mqtt and async-mqtt modules, use Javascript’s stream paradigm. Connections use on() to establish event handlers for events as they come through. Thus, a connection event emits a connect event, an incoming message emits a message event, and so on.

If you wanted to subscribe to two topics, as expressed above, you’d have a simpler subscription process:


This tells the client connection (and the broker) to handle any messages matching the topic name, including any wildcards you might want. You can have as many subscriptions on a single client as you like, although I imagine the brokers have rational limits.

That applies in the same way to message handlers. You add a message handler as a callback for the message event:

client.on('message', (topic, message) => {
        // assumes 'message' is a human-readable string!
        console.log(topic, message.toString());

Here’s the thing: you can have multiple message handlers, too. And they’ll get every message that the client is subscribed to.

If we’re subscribed to rpc/1 and rpc/2, that same message handler gets any message posted to either of those topics. It won’t get any other messages – presuming we haven’t added any more subscriptions – but it will get every message for those subscribed topics.

What’s more, if we add another message handler – via client.on('message', ...) again – every message handler will get every message, without discrimination.

If the handlers should need to handle only specific messages, then they each have to implement that functionality – filtering on topic, for example, or message content.

An alternative approach – and the one I think is more appropriate, within the limits of resource consumption – is to have multiple MQTT connections, each one with subscriptions that match a specific functionality.

In our first broken example, we have two topics, rpc/1 and rpc/2, where a message written to rpc/1 emits a sort of “hello world” message, and a message written to rpc/2 causes a message to be published to rpc/1.

If we’re preserving connections to the broker, our message handler would have to look something like this:

client.on('message', (topic, message)=>{
    if(topic.endsWith('1')) {
        console.log('hello', message.toString());
    if(topic.endsWith('2')) {
        client.publish('rpc/1', message.toString());

In environments where sockets are less expensive – i.e., we aren’t worried about counting how many sockets we use – we can be a lot more clear:

const helloClient=MQTT.connect('tcp:localhost:1883');
const sayHelloClient=MQTT.connect('tcp:localhost:1883');

helloClient.on('connect', ()=>{
helloClient.on('message', (topic, message)=>{
    console.log('hello', message.toString());

sayHelloClient.on('connect', ()=>{
sayHelloClient.on('message', (topic, message)=>{
    sayHelloClient.publish('rpc/1', message.toString());

In most message queueing libraries, you would set up a handler for incoming messages on each subscription, but MQTT is designed to be lighter than that. This shows a clean way to handle topic propagation in MQTT.

Playing with Meteor: Account Management is awesome

I started playing with Meteor – a Node.js and MongoDB framework – last week as a toy app to help some friends, and it is actually really nice. I don’t know yet how serious it is – I’ve only got maybe five hours playing with it – but one thing it has over almost every other framework I’ve played with lately: a trivial way to handle account and user management.

meteor add accounts-ui accounts-password alanning:roles

… and done. To provide a login component, you add an invocation to your HTML template:

{{> loginButtons}}

Role management isn’t quite as trivial; you have to set up the roles for the users, but checking for roles is pretty easy too:

{{#if currentUser}} 
    You're logged in.
    {{#if isInRole 'admin' 'default-group'}}
        You're in the admin group.
    You're not logged in.

Considering that this is one of the first things an application needs to set up, I have no idea why no other framework I’ve looked at has anything quite so trivial – or if it does, why it’s so hard to find. I’ve looked at multiple languages – Python, Ruby, Java – and multiple frameworks for most of those languages, and Meteor has them all beaten, hands down.

So for the first time I’m actually looking at using Javascript – via Node – seriously as a development platform for myself. It’s … interesting; my thought that Javascript just isn’t that great remains, and the Meteor interface to Mongo is actually quite constrained, but I think I can manage that.

The account management isn’t perfect – but it is good enough that you can actually get started with it, and focus on your app, while using what you need from the users collection.

In fact, some of the things about it are confusing – for example, account names seem to be populated (by registration) inconsistently – but these are problems that are probably caused by my lack of experience, rather than actual issues with the mechanisms themselves.

Well played, Node. Well played.

A quick first-contact tutorial for PencilBlue

Let’s see if we can figure out how to use PencilBlue. It looks interesting enough, and I like much about the approach they’re taking (“Eventually someone who’s not in love with the technology has to manage the site”); let’s see if we can hammer out some basic workflows.

The philosophy PencilBlue seems to prefer is “less is more,” and boy howdy, do they stick to that. My new site, after setting a banner, looks like this when it’s invoked:

PencilGlue's site, with my custom banner
PencilGlue’s site, with my custom banner

All right then! After I log in, it looks way different:

The site after I've logged in
The site after I’ve logged in

… except, no, it doesn’t really look different at all. The main difference is the “log out” button on the top right. If we select the farthest left icon, we end up being able to manage our profile (which allows us to change our password, our name, and our picture), but there’s still not a lot of information being thrown at us.

I actually could not find a way to get to any content administration from the landing page, whether I was logged in or not (and I’m the administrator on the site, so I should have access – and I do, as we’ll see.)

I can open the administration console by going to /admin – which is available only if I’m logged in, which is good. The admin console looks like this:

The administration console
The administration console

There’re a few things we can do here – set email preferences, see the configuration (as shown here), look at users, and check out the plugins (which include a few useful things, but nothing marvelous yet besides a mechanism by which you can import data from WordPress.)

Settings, mostly read-only
Settings, mostly read-only

What really interests us, though, is the content. Choosing the content dropdown gives us options for:

  • Navigation
  • Topics
  • Pages
  • Articles
  • Media
  • Comments
  • Custom objects

Let’s start with topics, because taxonomies should be determined fairly early (or so I think) – if we decide to add to the taxonomic structure, well, I don’t think anyone will blame us.

The topics screen is really pretty easy: It has an option for adding topics, searching for topics, and importing topics; it allows us to delete topics already created. This seems easy enough to use.

Our topics, which I'm not yet sure how to use
Our topics, which I’m not yet sure how to use

I’m not sure offhand what the difference is between a page and an article. In WordPress, a page is contained in a different structure than a post is (even though internally they’re almost identical); I’m guessing that this is similar, that articles represent information that is time-relevant, and pages represent static content for the site.

Let’s assume this is right, and create an “About” page, because everyone wants to know about this site.

Opening the pages menu option (under “content”) gives you the ability to manage your pages, and add new ones; we don’t have one, so let’s add one.

The content addition page has four headings of information for a given page: the page content, any media attached to it, topics, and SEO, which makes a lot of sense.

I’m going to add the page under the url “/page/about”, with a headline of “About this site,” and a subheading of “All the stuff you didn’t know you wanted to know.” I’m setting the publication date to now, because… just because, and I’m basically typing in an equivalent to lorem ipsum as the actual content.

At the bottom there’re two buttons: “Cancel” and “Save draft,” except “Save draft” also has an option (accessible through an up arrow) of “Save.” I’m not worried about drafting – either I’m a fantastic writer, or I’m not too concerned about the draft of this content, since it’s mostly throwaway stuff – so let’s choose “Save.”

Now, the pages … page says that it’s been published; saving is the same as publishing, based on the publication date. This is fairly important. It’s also not obvious to me, but then again, I’m probably used to more traditional publication cycles.

The problem now is: how do I find this page? The front page of the site looks exactly the same as it did. Maybe it’s time to add some navigation to my about page.

Navigation items can fit one of five different types: container, section, page, article, and link. We want to add a page, so let’s try that.

I entered “About” as the name, a short description, ignored the parent navigation item; off to “Content.” Here, it’s asking us to search for the right content. I happen to remember (because I wrote it down, you know) that the content I’m trying to include has “about” in the name, so I typed “about” and then selected “About this site,” since that was what the dropdown tried to show me.

Naturally, once I’d done that, I tried to save it, because saving is important.

Opening the site now shows me a marvelous change: I have an “About” link! It even works!


Well, that’s exciting. It’s time to see what else we can do. I’m going to create a page called “Node.js,” just because; it’s going to be like our other page, and mostly consist of content to throw away.

Now, I open my navigation panel, and add “Languages” as a container. After that, I add Node.js as a page, except instead of not having a parent navigation item, I’m choosing “Languages.” Opening the site now gives me “About” – a direct link – and “Languages,” with a dropdown including our new Node.JS page.

Except that doesn’t work! The “.js” makes it look like something it’s not; we need to use _js instead. After we fix that, the page works. We now have rudimentary, rather sparse content to go with our sparse site.

It’s time to blog, then. I’m adding two articles, called “Hello world” and “hello world, again”, both under the languages taxonomy. After saving, both show as being “published.”

And WHOA! Now we have two pieces, published in most-recent order (just like every other blog), on our home page!

It looks like a blog now.
It looks like a blog now.

Now we’re cooking with gas. I’m not sure what it does with long content – I was creating very short articles so I don’t know how well the system reduces front page content; I’m not yet sure how to use the taxonomies. But the site works – theoretically I could create a prettier (i.e., more busy) theme and publish a real, live site with PencilBlue – and it wasn’t even difficult to do.

You may be wondering where this magic site is; after all, resolves, but doesn’t necessarily work. This is not PencilBlue’s fault, at all; it’s because the VPS I use has limited resources, and it just doesn’t have the horsepower to keep everything up at the same time. I had to choose between applications; DeveloperStorm, being a simple demonstration site, just didn’t have enough priority for me to allocate resources. If I find a different container to run it on, I’ll point the IP there.

PencilBlue on Fedora 20

Tom Callaway (spot) mentioned a company local to the Raleigh area, publishing an open source CMS, called PencilBlue. PencilBlue is written with Node.js, a server environment leveraging Javascript.

You might wonder why I’m even considering trying a different platform. After all, I’m already a user of WordPress; I also work for a company that produces some killer middleware (that actually has commercial support).

Well, there are a few reasons why I’d be interested in kicking PencilBlue’s tires:

  • WordPress works, but as a CMS it’s somewhat lacking. Plus, it’s PHP – and I feel dirty even typing that. I have to admit, I like WordPress’ ecosystem; that’s why I use it in the first place… but I’m always hunting for the next thing.
  • WildFly is awesome; I really like working with it, and unlike Glassfish, you’re not hoping that Oracle will continue to support it. (Hint: they probably won’t, because Glassfish eats users from their cash cows. This is my opinion only.) However…
    • My VPS isn’t all that large, and while WildFly’s pretty light, it’s still going to be heavy for the virtual server I’m using.
    • The list of WildFly features my websites use is… very small. Not enough to justify the deployment on a public website for me. Your websites – especially if they’re not hobbyist things like mine – would not have this consideration; it’d likely be worth it for you to use WildFly.
  • I like to try new things – and PencilBlue gives me a chance to dip my toes in Node.js’ waters as well as checking out a new open source product. (Because open source is a Good Thing, you know?)

Installation of PencilBlue was very straightforward, with three steps:

Install PencilBlue

Open up a console on Fedora, and use the following commands:

sudo yum install npm nodejs git mongodb-server
sudo systemctl enable mongodb
sudo systemctl start mongodb

This installs git, Node.js, and MongoDB, and starts a MongoDB instance.

Change to where you want to install the application – let’s say /srv/develop/ for my test instance – and check out the PencilBlue repository; then let’s have Node.js install PencilBlue’s dependencies.

su - # change to a user with write access
mkdir /srv # if it doesn't already exist
cd /srv
mkdir logs
git clone develop
chown -R apache:apache /srv/*
cd develop
npm install
cp sample.config.json config.json

We’ll want to edit that config.json file: it has a value for "siteRoot" that we’ll want to change to our actual host, so I set the line to this:

"siteRoot": "",

We also might want to change the site port or otherwise firewall it off.

We actually have a site that we can use at this point, but we’re relying on manual startup and we can’t actually reach the system without specifying the (to-be-closed) port. What we need to do now is configure systemd to start our application automatically, and then configure nginx to forward to our application.

Configure systemd

systemd is how Fedora manages system services; what we need to do is write the service, and then tell systemd to manage it for us. (We already did this for MongoDB, when we prepared our system for PencilBlue; what we’re going to do is write the part that tells systemd how to manage our installation.)

It’s really simplicity itself. What I did was created a file, /usr/lib/systemd/system/developerstorm.service with the following contents:

ExecStart=/bin/node /srv/develop/pencilblue.js


I have nginx configured to run with the apache user and group; I made sure to set the directory ownership to apache, when I installed the PencilBlue app.

Believe it or not, we’re actually ready to start our application (and make sure it’s restarted if the server should reboot):

sudo systemctl enable developerstorm
sudo systemctl start developerstorm

If we haven’t closed off port 8080 to external access, we should be able to hit the application at this point.

Want to block port 8080? In Linux, use the following commands to block 8080 from external addresses while allowing the port forwarding to work:

sudo iptables -A INPUT -p tcp --dport 8080 -s -j ACCEPT
sudo iptables -A INPUT -p tcp --destination-port 8080 -j DROP
sudo service iptables save

Configure nginx

At last, it’s time to actually set up our site. First, let’s look at the site configuration file, which I have as /etc/nginx/sites-available/develop.local:

server {
    access_log /srv/develop/logs/access.log;
    error_log /srv/develop/logs/error.log;
    root /srv/www/develop/public_html;
    location / {
        proxy_pass http://localhost:8080/;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

I then linked this file to another configuration directory:

sudo mkdir /etc/nginx/sites-enabled # if it doesn't exist
sudo ln -s /etc/nginx/sites-available/develop.local /etc/nginx/sites-enabled/develop.local

Lastly, I altered /etc/nginx/conf. Look for include /etc/nginx/conf.d/*.conf; and add a line:

# Load modular configuration files from the /etc/nginx/conf.d directory.
# See
# for more information.
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

After we’ve done that, all we need to do is restart nginx:

sudo systemctl restart nginx

And we’re done! We should be able to hit our url (in my case,, set up our initial user, and start playing with PencilBlue.

I haven’t actually done anything with it yet – as of this writing, if you go to that URL it gives you a blank site – but it’s installed and running. Next I’ll see how it manages content.

Important Note

For one thing, the names and details have been changed – this setup works, but it’s not entirely secure or trustworthy. I’ll work on hardening it – this was mostly to get a server up and running to try it out.