Update 4: By instantiating the restify client (see controllers/messages.js) outside of the function and calling global.gc() after every request it seems the memory growth rate has been reduced a lot (~500KB per 10secs). Yet, the memory usage is still constantly growing.

Update3: Came across this post: /

It might be worth noting that I'm using HTTPS with Restify.

Update 2: Updated the code below to the current state. I've tried swapping out Restify with Express. Sadly this didn't make any difference. It seems that the api call at the end of the chain (restify -> mongodb -> external api) causes everything to retain to memory.

Update 1: I have replaced Mongoose with the standard MongoDB driver. Memory usage seems to grow less fast, yet the leak remains..

---

I've been working on trying to locate this leak for a couple of days now.

I'm running an API using Restify and Mongoose and for every API call I do at least one MongoDB lookup. I've got about 1-2k users that hit the API multiple times in a day.

What I have tried

• I've isolated my code to just using Restify and used ApacheBench to fire a huge amount of requests (100k+). The memory usage stays around 60MB during the test.
• I've isolated my code to just using Restify and Mongoose and tested it the same way as above. Memory usage stays around 80MB.
• I've tested the full production code locally using ApacheBench. Memory usage stays around 80MB.
• I've automatically dumped the heap on intervals. The biggest heap dump I had was 400MB. All I can see that there are tons of Strings and Arrays but I cannot clearly see a pattern in it.

So, what could be wrong?

I've done the above tests using just one API user. This means that Mongoose only grabs the same document over and over. The difference with production is that a lot of different users hit the API meaning mongoose gets a lot of different documents.

When I start the nodejs server the memory quickly grows to 100MB-200MB. It eventually stabilizes around 500MB. Could this mean that it leaks memory for every user? Once every user has visited the API it will stabilize?

I've included my code below which outlines the general structure of my API. I would love to know if there's a critical mistake in my code or any other approach to finding out what is causing the high memory usage.

Code

app.js

var restify = require('restify');
var MongoClient = require('mongodb').MongoClient;

// ... setup restify server and mongodb

require('./api/message')(server, db);

api/message.js

module.exports = function(server, db) {

    // Controllers used for retrieving accounts via MongoDB and munication with an external api
    var accountController = require('../controllers/accounts')(db);        
    var messageController = require('../controllers/messages')();

    // Restify bind to put
    server.put('/api/message', function(req, res, next) {
        // Token from body
        var token = req.body.token;

        // Get account by token
        accountController.getAccount(token, function(error, account) {

            // Send a message using external API
            messageController.sendMessage(token, account.email, function() {
                res.send(201, {});
                return next();
            });
        });
    });
};

controllers/accounts.js

module.exports = function(db) {

    // Gets account by a token
    function getAccount(token, callback) {
        var ObjectID = require('mongodb').ObjectID;

        var collection = db.collection('accounts');

        collection.findOne({
            token: token
        }, function(error, account) {

            if (error) {
                return callback(error);
            }

            if (account) {
                return callback('', account);
            }

            return callback('Account not found');
        });
    }
};

controllers/messages.js

module.exports = function() {

    function sendMessage(token, email, callback) {

        // Get a token used for external API
        getAccessToken(function() {}

            // ... Setup client

            // Do POST
            client.post('/external_api', values, function(err, req, res, obj) {
                return callback();
            });

        });
    }

    return {
        sendMessage: sendMessage
    };
};

Heap snapshot of suspected leak

Update 4: By instantiating the restify client (see controllers/messages.js) outside of the function and calling global.gc() after every request it seems the memory growth rate has been reduced a lot (~500KB per 10secs). Yet, the memory usage is still constantly growing.

Update3: Came across this post: https://journal.paul.querna/articles/2011/04/05/openssl-memory-use/

It might be worth noting that I'm using HTTPS with Restify.

Update 2: Updated the code below to the current state. I've tried swapping out Restify with Express. Sadly this didn't make any difference. It seems that the api call at the end of the chain (restify -> mongodb -> external api) causes everything to retain to memory.

Update 1: I have replaced Mongoose with the standard MongoDB driver. Memory usage seems to grow less fast, yet the leak remains..

---

I've been working on trying to locate this leak for a couple of days now.

I'm running an API using Restify and Mongoose and for every API call I do at least one MongoDB lookup. I've got about 1-2k users that hit the API multiple times in a day.

What I have tried

• I've isolated my code to just using Restify and used ApacheBench to fire a huge amount of requests (100k+). The memory usage stays around 60MB during the test.
• I've isolated my code to just using Restify and Mongoose and tested it the same way as above. Memory usage stays around 80MB.
• I've tested the full production code locally using ApacheBench. Memory usage stays around 80MB.
• I've automatically dumped the heap on intervals. The biggest heap dump I had was 400MB. All I can see that there are tons of Strings and Arrays but I cannot clearly see a pattern in it.

So, what could be wrong?

I've done the above tests using just one API user. This means that Mongoose only grabs the same document over and over. The difference with production is that a lot of different users hit the API meaning mongoose gets a lot of different documents.

When I start the nodejs server the memory quickly grows to 100MB-200MB. It eventually stabilizes around 500MB. Could this mean that it leaks memory for every user? Once every user has visited the API it will stabilize?

I've included my code below which outlines the general structure of my API. I would love to know if there's a critical mistake in my code or any other approach to finding out what is causing the high memory usage.

Code

app.js

var restify = require('restify');
var MongoClient = require('mongodb').MongoClient;

// ... setup restify server and mongodb

require('./api/message')(server, db);

api/message.js

module.exports = function(server, db) {

    // Controllers used for retrieving accounts via MongoDB and munication with an external api
    var accountController = require('../controllers/accounts')(db);        
    var messageController = require('../controllers/messages')();

    // Restify bind to put
    server.put('/api/message', function(req, res, next) {
        // Token from body
        var token = req.body.token;

        // Get account by token
        accountController.getAccount(token, function(error, account) {

            // Send a message using external API
            messageController.sendMessage(token, account.email, function() {
                res.send(201, {});
                return next();
            });
        });
    });
};

controllers/accounts.js

module.exports = function(db) {

    // Gets account by a token
    function getAccount(token, callback) {
        var ObjectID = require('mongodb').ObjectID;

        var collection = db.collection('accounts');

        collection.findOne({
            token: token
        }, function(error, account) {

            if (error) {
                return callback(error);
            }

            if (account) {
                return callback('', account);
            }

            return callback('Account not found');
        });
    }
};

controllers/messages.js

module.exports = function() {

    function sendMessage(token, email, callback) {

        // Get a token used for external API
        getAccessToken(function() {}

            // ... Setup client

            // Do POST
            client.post('/external_api', values, function(err, req, res, obj) {
                return callback();
            });

        });
    }

    return {
        sendMessage: sendMessage
    };
};

Heap snapshot of suspected leak

• javascript
• node.js
• memory-leaks
• mongoose
• restify

Share Improve this question Follow edited Mar 26, 2014 at 17:02 ndsc asked Mar 22, 2014 at 10:29 ndscndsc 1,21322 gold badges1414 silver badges2424 bronze badges 7

• in api/message.js is there a reason to call return next()? Not sure if this would help, but it seems a little redundant as you have already written the response. – Vasil Dininski Commented Mar 25, 2014 at 14:42
• I'd remend using a profiler to find out where the leak is ing from instead of manually searching by changing code. Take a look at: github./felixge/node-memory-leak-tutorial and / or github./node-inspector/node-inspector – Timothy Strimple Commented Mar 25, 2014 at 17:46
• 1 @Vasil I'm calling next in order to call an event in Restify to do work after the request is handled. Don't worry, I've tried disabling the event and not calling next(). Doesn't seem to make a difference. – ndsc Commented Mar 26, 2014 at 17:08
• @TimothyStrimple Thanks for the info. I've gathered some heapsnapshots prior to posting. I have trouble interpreting the snapshots though. I've included a recent snapshot at the bottom of the question. I suspect the leak to be located there. – ndsc Commented Mar 26, 2014 at 17:09
• 1 I don't know the libraries you're using, so can't tell how much the style of the code is dictated by their APIs. But the style of getAccount is asking for a leak. The callback that provides the account (and also the one for error) looks like it could easily lead to an indefinitely long chain of activations, where - in this case - each will reference the entire account collection from the DB call. I'd look for a way to provide the account as a true return value to let the stack unwind. – Gene Commented Mar 29, 2014 at 20:17

 |  Show 2 more ments

4 Answers 4

Sorted by: Reset to default 2

Might be a bug in getters, I got it when using virtuals or getters for mongoose schema https://github./LearnBoost/mongoose/issues/1565

It's actually normal to only see string and arrays, as most programs are largely based on them. The profiler that allow sorting by total object count are therefore not of much use as they many times give the same results for many different programs.

A better way to use the memory profiling of chrome is to take one snapshot for example after one user calls an API, and then a second heap snapshot after a second user called the API.

The profiler gives the possibility to pare two snapshots and see what is the difference between one and the other (see this tutorial), this will help understand why the memory grew in an unexpected way.

Objects are retained in memory because there is still a reference to them that prevents the object from being garbage collected.

So another way to try to use the profiler to find memory leaks is to look for an object that you believe should not be there and see what is it's retaining paths, and see if there are any unexpected paths.

Not sure whether this helps, but could you try to remove unnecessary returns?

api/message.js

        // Send a message using external API
        messageController.sendMessage(token, account.email, function() {
            res.send(201, {});
            next(); // remove 'return'
        });

controllers/accounts.js

module.exports = function(db) {

    // Gets account by a token
    function getAccount(token, callback) {
        var ObjectID = require('mongodb').ObjectID;

        var collection = db.collection('accounts');

        collection.findOne({
            token: token
        }, function(error, account) {

            if (error) {
                callback(error); // remove 'return'
            } else if (account) {
                callback('', account); // remove 'return'
            } else {
                callback('Account not found'); // remove 'return'
            }
        });
    }

    return { // I guess you missed to copy this onto the question.
        getAccount: getAccount
    };
};

controllers/messages.js

            // Do POST
            client.post('/external_api', values, function(err, req, res, obj) {
                callback(); // remove 'return'
            });

Your issue is in the getAccount mixed with how GC work's.

When you chain lots of function the GC only clears one at a time and the older something is on memory the less chances it has of being collected so on your get account you need at least that I can count 6 calls to global.gc() or auto executes before it can be collected by this time the GC assumes its something that it probably wont collect so it doesn't check it anyway.

collection{
   findOne{
      function(error, account){
         callback('', account)
            sendMessage(...)
               getAccessToken(){
                  Post
               }
            }
         }
      }
   }
}

as suggested by Gene remove this chaining.

PS: This is just a representation of how the GC works and depends on Implementation but you get the point.