I am writing a simulation in C, and want to output the data during runtime to .h5 files using the HDF5 library. The following code is used to output the files:

void output_data(int t, int n_output) {
    char filename[32];
    sprintf(filename, "data_%d.h5", n_output);
    hid_t hdf5_fp = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

    hid_t group_id = H5Gcreate2(hdf5_fp, "/data", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

    hsize_t dim_1d[1] = {1};
    hid_t dataspace_id_1d = H5Screate_simple(1, dim_1d, NULL);
    hsize_t dims[2] = {NX, NY};
    hid_t dataspace_id = H5Screate_simple(2, dims, NULL);

    hid_t dataset_id;
    dataset_id = H5Dcreate2(group_id, "time", H5T_NATIVE_INT32, dataspace_id_1d, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
    H5Dwrite(dataset_id, H5T_NATIVE_INT32, H5S_ALL, H5S_ALL, H5P_DEFAULT, &t);
    
    // Output of other data

    H5Sclose(dataspace_id_1d);
    H5Sclose(dataspace_id);
    H5Dclose(dataset_id);
    H5Gclose(group_id);
    H5Fclose(hdf5_fp);
}

I link statically with the hdf5 library using:

gcc main.c -o main -std=c17 -O3 -Wall -Wextra -I/usr/local/hdf5/include -L/usr/local/hdf5/lib -l:libhdf5.a -lm -lz

And I could also link with the shared library using:

gcc main.c -o main -std=c17 -O3 -Wall -Wextra -I/usr/local/hdf5/include -L/usr/local/hdf5/lib -Wl,-rpath /usr/local/hdf5/lib -l:libhdf5.so -lm -lz

If I want to read out the .h5 files during runtime using the h5py library in python, I get the following error:

BlockingIOError: [Errno 11] Unable to synchronously open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')

And if I abort the simulation during runtime, I get the following error when reading out the .h5 files:

OSError: Unable to synchronously open file (bad object header version number)

As far as I know, I am closing the .h5 files correctly in the C code. What could be causing these problems?

I am writing a simulation in C, and want to output the data during runtime to .h5 files using the HDF5 library. The following code is used to output the files:

void output_data(int t, int n_output) {
    char filename[32];
    sprintf(filename, "data_%d.h5", n_output);
    hid_t hdf5_fp = H5Fcreate(filename, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

    hid_t group_id = H5Gcreate2(hdf5_fp, "/data", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

    hsize_t dim_1d[1] = {1};
    hid_t dataspace_id_1d = H5Screate_simple(1, dim_1d, NULL);
    hsize_t dims[2] = {NX, NY};
    hid_t dataspace_id = H5Screate_simple(2, dims, NULL);

    hid_t dataset_id;
    dataset_id = H5Dcreate2(group_id, "time", H5T_NATIVE_INT32, dataspace_id_1d, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
    H5Dwrite(dataset_id, H5T_NATIVE_INT32, H5S_ALL, H5S_ALL, H5P_DEFAULT, &t);
    
    // Output of other data

    H5Sclose(dataspace_id_1d);
    H5Sclose(dataspace_id);
    H5Dclose(dataset_id);
    H5Gclose(group_id);
    H5Fclose(hdf5_fp);
}

I link statically with the hdf5 library using:

gcc main.c -o main -std=c17 -O3 -Wall -Wextra -I/usr/local/hdf5/include -L/usr/local/hdf5/lib -l:libhdf5.a -lm -lz

And I could also link with the shared library using:

gcc main.c -o main -std=c17 -O3 -Wall -Wextra -I/usr/local/hdf5/include -L/usr/local/hdf5/lib -Wl,-rpath /usr/local/hdf5/lib -l:libhdf5.so -lm -lz

If I want to read out the .h5 files during runtime using the h5py library in python, I get the following error:

BlockingIOError: [Errno 11] Unable to synchronously open file (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')

And if I abort the simulation during runtime, I get the following error when reading out the .h5 files:

OSError: Unable to synchronously open file (bad object header version number)

As far as I know, I am closing the .h5 files correctly in the C code. What could be causing these problems?

• python
• c
• hdf5

Share Improve this question Follow asked Nov 20, 2024 at 19:10 DaanDaan 3955 bronze badges 4

• 1 The return codes of the hdf5 functions are checked and prove not to be the problem, but I am not interested in checking them during runtime when performing actual simulations. – Daan Commented Nov 20, 2024 at 22:18
• You are describing 2 different errors. The first h5py error message is a file locking problem. What file mode is used to open the file with h5py? By default, HDF5 is SWMR - Single Write, Multiple Read. When your simulation program has the file open for writing, you have to open in Read mode with other applications. Read this answer for a complete explanation of file locking issues: Locking of HDF files using h5py The second error (from the abort operation) is most likely due to improperly closing the file. – kcw78 Commented Nov 21, 2024 at 14:33
• The .h5 files can be read out when the C program is allowed to run until the end. To read out the data with Python, the following line was used: with h5py.File(data_files[i], 'r') as data_file:. I have used this Python code with other code bases written in C that output .h5 files, and the problems of opening files during runtime of the C program or when the program is aborted do not occur. – Daan Commented Nov 21, 2024 at 18:11
• All that said, you clearly have a file lock issue. You can eliminate the h5py file access mode as the cause. Now you have to work thru the others. After the simulation finished did you try h5clear to reset the lock? Since this works for your other codes you must have a subtle difference. Different behavior == something is different. Time to break out the debugger and profiler and step thru the file creation process to find the root cause. – kcw78 Commented Nov 21, 2024 at 21:53

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Both issues of being unable to open files during runtime and corrupted files when aborting the program can be fixed by adding H5Fflush(hdf5_fp, H5F_SCOPE_GLOBAL); before the close functions. Found this with the help of ChatGPT, thanks Stack Overflow!