Blog

At Solspace we work frequently building custom API integrations to connect our client's various cloud platforms to one another to meet various business needs. We usually build these on ExpressionEngine or Craft CMS. We use these platforms as a hub. They house the code that manages the business logic of connecting the other cloud systems together like Shopify, Stripe, Salesforce, NetSuite, MemberSuite, HubSpot, etc.

Recently we've been using Node JS to serve as this hub. We have a client who has multiple e-commerce stores running on Shopify. There is a separate fulfillment company who warehouses the physical products that are sold and also fulfills the orders when they go through. Shopify can't talk to this fulfillment company out of the box and the fulfillment company can't talk to Shopify out of the box. So a script was written using Node JS to glue the two systems together.

The script is more like a portal that our client can login to in order to do maintenance tasks on the system. For the most part though, the system just sits and runs all day and all night long. It checks a few directories on a connected server every minute, looking for changes. The fulfillment company automatically uploads CSV files that contain updates about orders or fulfillments or product inventories. The Node script detects these uploads and processes their content. When the processing takes place, the script connects to the Shopify API and updates stuff.

All goes along swimmingly until the script tries to process a CSV file with an error. All goes along fine until someone spills a burrito on the server and knocks out the script. It has to be restarted, but that only happens when someone notices the script isn't running. Guess who's in charge of noticing when the script has stopped running? Right, the customer - not my client, but the customer, the person who is trying to buy one or more physical products from the Shopify store.

So someone drops a Slurpee on the cable bundle in the data center. The machines short out. The customer, who had paid for their order with their credit card and who is expecting delivery in two days ends up getting nothing in the mail for five days. She calls my client, quite annoyed, completely uninterested in burrito or Slurpee excuses. She just wants her socks. That's it!

Now we need to do better. (In fairness we inherited this script from another developer.) We need to make this better. We need some monitoring. We need to be able to watch this poor little Node JS script and make sure that it doesn't go on vacation.

There are quite a few monitoring services out there, but we need something that my client will feel comfortable with. My client is not a web developer, that's why they hired me. My client wants to stay focused on socks, not internets. What she feels most comfortable with, in the internet realm, is email. Email!!??

"Email is the least stable of all internet protocols," you say. "You're right!" I say. But still, my client likes it. And honestly, let's keep it simple, stupid. If my client is used to receiving an email from the Node script every hour and then doesn't get that email for a couple of hours, she will notice. There will be plenty of time to dive in and check on pending or missed orders before a customer service issue pops up.

Simple. Nodemailer!

Node JS has many packages ready for install and use in your Node JS app. They can be found at NPM. Nodemailer is a handy little email tool. It stays out of the way. It seems happiest when you use it through Gmail, which is fine by me. We want to leave our own email accounts alone. So we added Nodemailer into our package.json file. We ran...

npm install

... from inside our script's directory. Then we created a new module in our modules folder. I called it 'Notify'. In that module we call in Nodemailer...

var Nodemailer = require('nodemailer'),
consolere = require('console-remote-client');  

...I'll explain what 'consolere' is in a minute. Actually here's the rest of the Notify module code since, if you're this far in, you may as well cut, paste and tweak...

/*jslint node: true, es5: true*/
(function () {
    "use strict";

    var Nodemailer = require('nodemailer'),
        consolere = require('console-remote-client');

    function Notify() {}

    function sendMail(to, subject, message) {
	    if (this.config.Notify.on === false) return false;

	    var smtpTransport	= Nodemailer.createTransport({
		    host: this.config.Notify.host,
		    port: this.config.Notify.port,
		    secure: this.config.Notify.secure, // use SSL
		    auth: {
			    user: this.config.Notify.username,
			    pass: this.config.Notify.password
		    }
	    });
	
	    to		= (to === null) ? this.config.Notify.to: to;
	    subject	= subject.replace('%%store%%', this.config.Notify.store);
	    message	= message.replace('%%store%%', this.config.Notify.store);
     
	    smtpTransport.sendMail({
    	    to: to,
    	    subject: subject,
    	    text: message,
    	    html: message
         }, function(error, info) {
		    if (error) {
			    return console.log(error);
		    }
		    console.log(info.response);
	    });
    }

    function sendPing(message) {
	    console.re.log(message);
    }

    function init(config, app, modules) {
	    this.config = config;
	    consolere.connect('console.re', '80', config.Notify.consolere);
    }

    module.exports = {
	    "init": init,
        "sendMail": sendMail,
        "sendPing": sendPing
    };
}());

You see I have some config references in there. In our main script file, the one that boots up this whole thing, I initialize the Notify module by calling it's init() method. I pass the config file values in there. I also allow the Notify module to parse the store name since this Node JS script manages more than one Shopify store.

Here's a snippet from the config file in case you want to connect to your own Gmail account one day...

"to": "yourgmail@gmail.com",
"store": "Sock Store Thing",
"host": "smtp.gmail.com",
"port": 465,
"secure": true,
"username": "yourgmail",
"password": "meowcatjiggityjig",

Ok enough tech. The important thing is we have a module that we can 'require' into our other modules and tell it to fire off emails, like this:

Notify.sendMail('you@gmail.com', 'The Script! She is a-runnin!', 'Proof of life. Yay!');

I know you're still squirming. "Can't you do better than monitoring through email?" you ask.

Yes of course we can, but remember, my client is most comfortable with email. You want a comfortable client. Ease them gradually into the rest of the internet. It's moving so fast, give them a break.

But yes, some overkill is called for here. Email is not good enough. What would be much better is if my client could just click on a url on some 3rd party website to see if the script was still alive. Wouldn't it be cool if my client could just see like a running, dynamic log file? At the command line we call this 'tail'. But wouldn't it be sick of the console.log() messages that our script is logging could be logged to the web somewhere that my client could see? Enter Console.re!

This is SOOO cool guys. So you can install Console.re (that's latin for console) into a web page using a script tag. You can also install the rascal into your Node JS app like...

npm install console-remote-client -g

or putting this in your package.js file...

"console-remote-client": "^0.4.x",

You can then require the module into your scripts and instead of doing this...

console.log('Things are running quite smoothly in the Node JS script right now.');

...do this...

consolere.connect('console.re', '80', 'sock-joint-and-all-that');
console.re.log('Things are running quite smoothly in the Node JS script right now.');

The output of that does a standard console.log() but also pushes that log event up to http://console.re/sock-joint-and-all-that. Of course I faked the name for this blog post. But you can watch any log events in your apps just by claiming a url on console.re.

What you see in console.re is a real time running capture or tail of your log events. Console.re does not maintain a history of your log, it only maintains what you allowed it to log into your browser window. Still, it's great! My client can click to the url and watch for a few minutes to see what's happening in the script. If it's running fine after a few log events are captured, she can relax and go back to her socks.

Yes there are other monitoring systems, but the simple-dumb stuff is usually the best.

Want to talk with us about your API integration problems? Holler!