What I want to do: I have CPU intensive computation that I want to offload to a worker process so as not to block my main server.

Without ProcessPoolExecutor my code looks like this:

my_model = model_init()

def handler(args):
    return my_model.process(args)

How do I utilize ProcessPoolExecutor?

consider the function that creates a ProcessPoolExecutor:

concurrent.futures.ProcessPoolExecutor(max_workers=1, initializer=init_function, initargs=(model_name))

say I have:

def init_function():
   model = model_init()

The core of the problem is how to "persist" the model created in the init_function? Later on I want to submit tasks to the ProcessPoolExecutor and be able to use the model created in the init_function to process the task.

Its difficult to explain - the issue is that ProcessPoolExecutor only knows about two functions the initializer and the function I will provide in submit. I don't know how to pass state between these functions. How can the function that is argument to submit access a variable that was initialized when initializer was called?

The only example I found is here and that does not go over this issue.

What I want to do: I have CPU intensive computation that I want to offload to a worker process so as not to block my main server.

Without ProcessPoolExecutor my code looks like this:

my_model = model_init()

def handler(args):
    return my_model.process(args)

How do I utilize ProcessPoolExecutor?

consider the function that creates a ProcessPoolExecutor:

concurrent.futures.ProcessPoolExecutor(max_workers=1, initializer=init_function, initargs=(model_name))

say I have:

def init_function():
   model = model_init()

The core of the problem is how to "persist" the model created in the init_function? Later on I want to submit tasks to the ProcessPoolExecutor and be able to use the model created in the init_function to process the task.

Its difficult to explain - the issue is that ProcessPoolExecutor only knows about two functions the initializer and the function I will provide in submit. I don't know how to pass state between these functions. How can the function that is argument to submit access a variable that was initialized when initializer was called?

The only example I found is here and that does not go over this issue.

• python

Share Improve this question Follow asked Mar 13 at 23:55 morpheusmorpheus 20.4k2929 gold badges110110 silver badges188188 bronze badges 1

• Does this model have a mutable state? Specifically, will the worker modify it? – Lukasz Tracewski Commented Mar 14 at 6:44

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

This is generally a place where it makes sense to use a global variable. The global variable will be used only within the child process, so you don't need to worry about different processes using the same value and tripping over each other.

Try:

def init_func(*args):
    global my_model
    my_model = model_init(*args)  # or do whatever is appropriate with the args

def process_func(*args)
    my_model.process(*args)