I'm using Request library (gzip enabled) and it crash with this error.

It seems this error occur at this line,

response.body = response.body.toString(self.encoding)

Scope containing that line,

response.body = Buffer.concat(buffers, bufferLength)
if (self.encoding !== null) {
 response.body = response.body.toString(self.encoding)
}

Full Stacktrace,

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.utf8Slice (<anonymous>)
    at Buffer.toString (buffer.js:797:17)
    at Request.<anonymous> (/home/proj/node_modules/request/request.js:1128:39)
    at Request.emit (events.js:315:20)
    at IningMessage.<anonymous> (/home/proj/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:421:28)
    at IningMessage.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_STRING_TOO_LONG'
}

What's the best way to convert the buffer to String when it's longer than 1073741799 characters?

I'm using Request library (gzip enabled) and it crash with this error.

It seems this error occur at this line,

response.body = response.body.toString(self.encoding)

Scope containing that line,

response.body = Buffer.concat(buffers, bufferLength)
if (self.encoding !== null) {
 response.body = response.body.toString(self.encoding)
}

Full Stacktrace,

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Buffer.utf8Slice (<anonymous>)
    at Buffer.toString (buffer.js:797:17)
    at Request.<anonymous> (/home/proj/node_modules/request/request.js:1128:39)
    at Request.emit (events.js:315:20)
    at IningMessage.<anonymous> (/home/proj/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:421:28)
    at IningMessage.emit (events.js:327:22)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ERR_STRING_TOO_LONG'
}

What's the best way to convert the buffer to String when it's longer than 1073741799 characters?

• javascript

Share Improve this question Follow asked Mar 23, 2021 at 23:16 jeffbRTCjeffbRTC 2,0691616 silver badges3838 bronze badges 10

• 1 That's within the ballpark of the length of Tolstoy's War and Peace. What exactly are we talkin bout here? Why are you trying to create a 1~2 MB string? – Jared Smith Commented Mar 23, 2021 at 23:18
• 1 Javascript isn't equipped to really handle strings of that size. The error could not be clearer, actually. This is not the tool for the job. – somethinghere Commented Mar 23, 2021 at 23:23
• @JaredSmith I don't but the request library does. I fetch websites you know and that size is mon but it unable to move forward. – jeffbRTC Commented Mar 23, 2021 at 23:43
• @somethinghere It's NodeJS. It should be able to handle strings of that size. – jeffbRTC Commented Mar 23, 2021 at 23:44
• But I'm open to skipping this string as-well. I just don't know what should I do to skip it. – jeffbRTC Commented Mar 23, 2021 at 23:46

 |  Show 5 more ments

1 Answer 1

Sorted by: Reset to default 3

It seems I ran into a crawling trap while crawling the websites. The page that request fetched is almost over 2GB in size.

NodeJS's Buffer.toString has a limit of 1 GB and this content is way over that.

The authors of request library has forgotten to put a catch block inside the toString operation.

The fix is to add the catch block like below in request.js file's line 1130,

  if (self.encoding !== null) {
    try {
      response.body = response.body.toString(self.encoding)
    } catch(e) {
      self.emit('error', e)
    }
  }