I am using Mpi4py with OpenMPI on Ubuntu, and I also need this to run on RedHat. I am trying to run compute_ldos() for a relatively large set of parameters, and am trying to use as all available cores on my node to make this faster. Right now I am calling compute_ldos on a single thread for each parameter combination (embarrassingly parallel I believe it's called?). compute_ldos will use all available processes, and I need to use multiple threads to reduce memory usage. However, I lose efficiency if I assign to many to a single compute_ldos call. How do I assign multiple processes to each worker process?

def worker_process(distances, rank):
    """Worker function executed by each MPI process."""
    return [compute_ldos(dist[0], dist[1], dist[2], rank) for dist in distances]

def main():
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()
    size = comm.Get_size()
    start = 50
    end = 100
    res = [20, 10, 5, 5]
    num_angles = 10
    
    distances = []
    points = np.linspace(start, end, len(res) + 1)
    
    for idx, point in enumerate(points[:-1]):
        distances.extend(np.linspace(point, points[idx + 1], res[idx])[:-1])

    distances.append(end)
    
    angle_sample = np.arange(0, 360, num_angles / 360)
    
    distances = [[x, angle, False] for x in distances for angle in angle_sample]
    distances.append([0, 0, True])

    # Distribute distances across MPI ranks
    distances_split = np.array_split(distances, size)[rank]

    setup_stdout(rank)

    # Each MPI process runs its own worker function
    procs = worker_process(distances_split, rank)

compute_ldos calls a meep function to run an FDTD simulation.