I am learning OpenMPI and trying to merge childComm and parentComm into a single intraComm by using MPI_Intercomm_merge after calling MPI_Comm_spawn, but the parent and child processes are both stuck at MPI_Intercomm_merge. Here is my program (test.c):

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    // get parent communicator
    MPI_Comm parentcomm, mergedcomm;
    MPI_Comm_get_parent(&parentcomm);

    if (parentcomm == MPI_COMM_NULL) {
        // spawn child process
        MPI_Comm intercomm;
        MPI_Comm_spawn("./test", MPI_ARGV_NULL, 1, MPI_INFO_NULL, 0, MPI_COMM_SELF, 
                      &intercomm, MPI_ERRCODES_IGNORE);
        
        // merge communicator
        printf("Parent process: Merging communicator\n");
        MPI_Intercomm_merge(intercomm, 1, &mergedcomm);
        printf("Parent process: Merge done\n");

    } else {
        // merge communicator
        printf("Child process: Merging communicator\n");
        MPI_Intercomm_merge(parentcomm, 0, &mergedcomm);
        printf("Child process: Merge done\n");

        MPI_Comm_free(&parentcomm);
    }

    MPI_Comm_free(&mergedcomm);
    MPI_Finalize();
    return 0;
} 

and here is the output:

$ mpirun -np 1 ./test
Parent process: Merging communicator
Child process: Merging communicator
(stuck here)

Can someone help me find the issue in my program?

I observed that the code I found online is similar to mine, which makes it difficult for me to identify the issue in my program. I tried reinstalling OpenMPI, but the problem still persisted.

I used send and recv for communication between the child and parent, and it also worked successfully.

I also attempted to use MPI_Comm_split to divide MPI_COMM_WORLD into two separate communicators, then used MPI_Intercomm_create to establish inter-communicator communication, and finally called MPI_Intercomm_merge. This approach worked, so I suspect that the issue is not with the OpenMPI library.

I am learning OpenMPI and trying to merge childComm and parentComm into a single intraComm by using MPI_Intercomm_merge after calling MPI_Comm_spawn, but the parent and child processes are both stuck at MPI_Intercomm_merge. Here is my program (test.c):

#include <mpi.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);

    // get parent communicator
    MPI_Comm parentcomm, mergedcomm;
    MPI_Comm_get_parent(&parentcomm);

    if (parentcomm == MPI_COMM_NULL) {
        // spawn child process
        MPI_Comm intercomm;
        MPI_Comm_spawn("./test", MPI_ARGV_NULL, 1, MPI_INFO_NULL, 0, MPI_COMM_SELF, 
                      &intercomm, MPI_ERRCODES_IGNORE);
        
        // merge communicator
        printf("Parent process: Merging communicator\n");
        MPI_Intercomm_merge(intercomm, 1, &mergedcomm);
        printf("Parent process: Merge done\n");

    } else {
        // merge communicator
        printf("Child process: Merging communicator\n");
        MPI_Intercomm_merge(parentcomm, 0, &mergedcomm);
        printf("Child process: Merge done\n");

        MPI_Comm_free(&parentcomm);
    }

    MPI_Comm_free(&mergedcomm);
    MPI_Finalize();
    return 0;
} 

and here is the output:

$ mpirun -np 1 ./test
Parent process: Merging communicator
Child process: Merging communicator
(stuck here)

Can someone help me find the issue in my program?

I observed that the code I found online is similar to mine, which makes it difficult for me to identify the issue in my program. I tried reinstalling OpenMPI, but the problem still persisted.

I used send and recv for communication between the child and parent, and it also worked successfully.

I also attempted to use MPI_Comm_split to divide MPI_COMM_WORLD into two separate communicators, then used MPI_Intercomm_create to establish inter-communicator communication, and finally called MPI_Intercomm_merge. This approach worked, so I suspect that the issue is not with the OpenMPI library.

• c
• mpi

Share Improve this question Follow asked Mar 27 at 7:15 Leonel ChenLeonel Chen 3111 silver badge33 bronze badges 10

• the code works for me. Which Open MPI version are you running? – Gilles Gouaillardet Commented Mar 27 at 8:11
• I notice you use MPI_ERRCODES_IGNORE on the spawn-call. That might mask a bug. – julaine Commented Mar 27 at 9:25
• hmm, I found no obvious mistakes in the code like misordered arguments, and your usage of spawn/merge seems to match the documentation (though I have no experience with these functions). Please do post an answer when you find one. – julaine Commented Mar 27 at 9:48
• I noted spawn hangs if the binary cannot be found. Are you spawning on the same node? What if you use the absolute path? – Gilles Gouaillardet Commented Mar 27 at 9:50
• 1 try mpirun --mca pml ob1 -np 1 ./test if it still hangs, try mpirun --mca pml ob1 --mca btl tcp,self -np 1 ./test and if it still hangs, pick a network interface that is known to be working and has no firewall on it (e.g. eth0) and then mpirun --mca pml ob1 --mca btl tcp,self --mca btl_tcp_if_include eth0 -np 1 ./test – Gilles Gouaillardet Commented Mar 29 at 7:15

 |  Show 5 more comments

2 Answers 2

Sorted by: Reset to default 2

Thank you @Gilles Gouaillardet, for providing the solution in comments. I will try to explain what is happening under the hood and how to observe it in the log files.

Summary: Your code for spawning a child process and merging the intercommunicator works fine and the problem appears when you use MPI_Comm_spawn followed by MPI_Intercomm_merge. Also, as you might have observed, you wont face any issue while using point-to-point and collective operations. This is due to an issue with the network interface that Open MPI automatically selects for establishing the parent–child connection.

As @Gilles Gouaillardet pointed out, if it hangs with the default settings, try specifying the interface that is known to work. This ensures both the parent and child communicate over the same interface, resolving the hang during the merge.

What’s Happening Under the Hood:

When you run your program without any additional MCA (Modular Component Architecture) parameters, Open MPI’s TCP Byte Transfer Layer (BTL) examines all available network interfaces. By default, it excludes the loopback interface (127.0.0.1) (and some other interfaces- run command ompi_info --all | grep btl_tcp ) but may still pick up multiple interfaces.

You can increase the verbosity level to see which interfaces are used. For example, run:

mpirun --mca btl_base_verbose 100 -np 1 ./test

In the verbose logs, you will see entries like:

btl: tcp: Using interface: ...
...
btl:tcp: Found match: 127.0.0.1 (lo0)
...
btl: tcp: Using interface: en0   // (IPv4: 192.168.x.x, IPv6: fdxx:xxxx)

The key issue occurs during the exchange of connection information. One process is advertising an address from one interface (e.g., en0), while the other ends up using a different one (for example, en4 ). You may see log lines such as:

btl: tcp: Match incoming connection ... with locally known IP 10.x.x.x failed (iface 2/3)!

This mismatch in expected IP addresses prevents the parent and child from correctly completing the merge operation, causing the MPI_Intercomm_merge to hang.

In a typical MPI run with all processes started together (e.g., for collectives or point-to-point communication), the automatic interface selection is usually consistent among processes. However, with MPI_Comm_spawn, the child process is started dynamically and might pick a different default interface than the parent. This is why your spawn-based merge hangs while collectives do not.

As Gilles Gouaillardet mentioned in comments, forcing the use of a single, known-good interface ensures that both the parent and the spawned child agree on the network addresses used for MPI communications. For example, if you know that en0 is configured correctly and has no firewall issues, run your program as follows:

mpirun --mca btl_tcp_if_include en0 -np 1 ./test

This MCA parameter forces Open MPI’s TCP BTL to use only the “en0” interface. As a result, the IP address advertised during the connection exchange is consistent between parent and child, allowing MPI_Intercomm_merge to complete successfully.

Sorry for posting the solution so late. And thanks to @j23 and @Gilles Gouaillardet helping me with the answer.

I found the OpenMPI documentation, which suggested using ompi_info --param btl tcp to search for TCP-related parameters.

$ ompi_info --param btl tcp
         MCA btl: tcp (MCA v2.1.0, API v3.3.0, Component v5.0.7)
     MCA btl tcp: ---------------------------------------------------
     MCA btl tcp: parameter "btl_tcp_if_include" (current value: "",
                  data source: default, level: 1 user/basic, type:
                  string)
                  Comma-delimited list of devices and/or CIDR
                  notation of networks to use for MPI communication
                  (e.g., "eth0,192.168.0.0/16").  Mutually exclusive
                  with btl_tcp_if_exclude.
     MCA btl tcp: parameter "btl_tcp_if_exclude" (current value:
                  "127.0.0.1/8,sppp", data source: default, level: 1
                  user/basic, type: string)
                  Comma-delimited list of devices and/or CIDR
                  notation of networks to NOT use for MPI
                  communication -- all devices not matching these
                  specifications will be used (e.g.,
                  "eth0,192.168.0.0/16").  If set to a non-default
                  value, it is mutually exclusive with
                  btl_tcp_if_include.
     MCA btl tcp: parameter "btl_tcp_progress_thread" (current value:
                  "0", data source: default, level: 1 user/basic,
                  type: int)

In my case, my processes attempt to communicate with each other over any available network, including the inappropriate network docker0.

Adding --mca btl_tcp_if_include <proper network> or --mca btl_tcp_if_exclude docker0 both solved the problem.