I've been running into a really odd and seemingly random issue on our web app that I just can't seem to successfully debug. It runs fine for anywhere from 10 minutes to 6 hours, and then all of a sudden no remote requests to or from the server can be made, they just hang (this includes regular http and web socket requests). The odd thing is that going to the site regularly still works, until the OS file descriptor limit is reached and then http pletely crashes with all of the stalled connections.

There are no errors, though the following error is thrown when the issue begins (I assume this is a side-effect of whatever is going on rather than the cause).

TypeError: Cannot read property '0' of null
    at null.<anonymous> (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/collection.js:504:22)
    at args.(anonymous function) (/app/node_modules/strong-agent/lib/proxy.js:85:18)
    at g (events.js:175:14)
    at EventEmitter.emit (events.js:98:17)
    at Base.__executeAllServerSpecificErrorCallbacks (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/base.js:315:29)
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:273:22
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:370:11
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:352:28
    at _callback (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:670:5)
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/auth/mongodb_cr.js:47:13

I've tried raising the file descriptor limits and the global agent maxSockets with no affect on this behavior. There's no influx of traffic when this happens, and it happens equally as often during peak and off-peak times. The CPU usage consistently stays below 5% and doesn't have any perceptible changes leading up to or during the crash. The server also never drops below 1GB of free memory.

The stack: SmartOS cloud server (Joyent), Express, Socket.io, MongoDB and Redis.

I've been debugging this for several days and have pletely run out of ideas where to look. Hoping someone on SO has run into something similar or has different ideas of what can be tried or tested.

I've been running into a really odd and seemingly random issue on our web app that I just can't seem to successfully debug. It runs fine for anywhere from 10 minutes to 6 hours, and then all of a sudden no remote requests to or from the server can be made, they just hang (this includes regular http and web socket requests). The odd thing is that going to the site regularly still works, until the OS file descriptor limit is reached and then http pletely crashes with all of the stalled connections.

There are no errors, though the following error is thrown when the issue begins (I assume this is a side-effect of whatever is going on rather than the cause).

TypeError: Cannot read property '0' of null
    at null.<anonymous> (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/collection.js:504:22)
    at args.(anonymous function) (/app/node_modules/strong-agent/lib/proxy.js:85:18)
    at g (events.js:175:14)
    at EventEmitter.emit (events.js:98:17)
    at Base.__executeAllServerSpecificErrorCallbacks (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/base.js:315:29)
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:273:22
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:370:11
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/connection/repl_set/ha.js:352:28
    at _callback (/app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/db.js:670:5)
    at /app/node_modules/mongojs/node_modules/mongodb/lib/mongodb/auth/mongodb_cr.js:47:13

I've tried raising the file descriptor limits and the global agent maxSockets with no affect on this behavior. There's no influx of traffic when this happens, and it happens equally as often during peak and off-peak times. The CPU usage consistently stays below 5% and doesn't have any perceptible changes leading up to or during the crash. The server also never drops below 1GB of free memory.

The stack: SmartOS cloud server (Joyent), Express, Socket.io, MongoDB and Redis.

I've been debugging this for several days and have pletely run out of ideas where to look. Hoping someone on SO has run into something similar or has different ideas of what can be tried or tested.

• javascript
• node.js
• unix
• express
• socket.io

Share Improve this question Follow asked Sep 12, 2013 at 17:02 James SimpsonJames Simpson 13.7k2626 gold badges8585 silver badges111111 bronze badges 12

• is this for an array? is the array empty? if not, have you tried to access array[index - 1]? – aug2uag Commented Sep 12, 2013 at 17:13
• That error message is not from my code, it is ing from the MongoDB native driver for Node.js. I don't think this is the source of the issue, but I thought I'd include it for pleteness. – James Simpson Commented Sep 12, 2013 at 17:42
• Can you regularly check the number of open file descriptors, and see if it slowly and constantly increases? If so, you have a fd leak somewhere. Likely not directly in node, since it will time out sockets. – Nitzan Shaked Commented Sep 12, 2013 at 19:34
• @NitzanShaked It seems to stay fairly level and then all of a sudden goes through the roof. My assumption is that it goes through the roof as a result of the real issue, but I may be wrong about that. – James Simpson Commented Sep 12, 2013 at 19:41
• OK, so next debugging step: write a small script that prints the number of open file descriptors per process, and when the total starts going thru the roof try to see which process is to blame. If it's node, I would first look at any modules which are wrappers around native (read: c) code. – Nitzan Shaked Commented Sep 12, 2013 at 19:48

 |  Show 7 more ments

1 Answer 1

Sorted by: Reset to default 9

After countless hours of debugging and more debugging, I finally found the culprit. An error was being thrown inside of several different mongojs callbacks, which appears to have bubbled up and blocked the connections from closing. Over time, this got to a tipping point and connections started hanging until the file descriptor limit was reached.

The error turned out to be in the Now.js node module (which has been abandoned). If there is anyone out there that is running into this issue using Now.js, I forked it and patched the bug. You you can find the mit here: https://github./goldfire/now/mit/b5bd54f8950602f752a710c606be6754b759cab2.

The way I found this bug was to attach an error listener to the DB object:

var db = require('mongojs').connect('...', ['collection']);
db.client.on('error', function(err){
  console.log(err.stack);
});