javascript - Insane amount of RAM usage with BullMQ + fetch - Stack Overflow

I have a small app that runs several requests(node-fetch) with in a queue with bullMQ.And i have 3 wor

I have a small app that runs several requests(node-fetch) with in a queue with bullMQ.

And i have 3 worker threads (if i dont use this main event loop gets blocked and my whole app freezes.) (This is an another issue why main loop gets blocked with a few fetch requests ?)

queue A , queue B and queue C

all queues have same configuration

new Queue('a-queue', {
connection: {
    url: process.env.REDIS_URL,
    enableReadyCheck: true,
},
defaultJobOptions: {
    attempts: 5,
    removeOnComplete: true,
    removeOnFail: false,
},

worker for the queue (a,b,c all same):

const processor = pathToFileURL(__dirname + '/a_processor.js')
new Worker('a-queue', processor, {
    connection: {
        url: process.env.REDIS_URL,
        enableReadyCheck: true,
    },
    concurrency: 1,
    useWorkerThreads: true,
})

And a_processor.js is sending some requests with fetch and returns the values.

Also i have a setting for the global concurrency limit in index.js (start of the express app):

await queue.setGlobalConcurrency(5)

Upto this points it seems like it works as i expected (bullmq dashboard blue ones are active yellow ones are waiting):

But after like a minute redis instance is crashing due high memory consumption:

Memory usage is like skyrocket when i run this app.And what is doing is just sending fetch requests (10-15 in paralel) and these requests are not like responsing mbs of data its only few kb of json each time. There is nothing much but this consumes so much memory idk how.Or someting else causing that idk.

Here's the request method:

const headerGenerator = new HeaderGenerator()
const headers = headerGenerator.getHeaders()

const response = await fetch(`${domain}`, {
    headers: headers,
    agent: agentDcIp,
    signal: AbortSignal.timeout(15000) as any,
})

const body = await response.text()

return {
    body: body,
    ok: response.ok,
    status: response.status,
}

Any idea why this is happening and how to prevent it.

发布者:admin,转转请注明出处:http://www.yc00.com/questions/1745632985a4637220.html

相关推荐

发表回复

评论列表(0条)

  • 暂无评论

联系我们

400-800-8888

在线咨询: QQ交谈

邮件:admin@example.com

工作时间:周一至周五,9:30-18:30,节假日休息

关注微信