2.1.1-beta (revision 4705)
|
This is a short example of how to use the OTF2 writing interface with MPI. This example is available as source code in the file otf2_mpi_writer_example.c .
We start with inclusion of some standard headers.
And than include the MPI and OTF2 header.
Now prepare the inclusion of the <otf2/OTF2_MPI_Collectives.h> header. As it is a header-only interface, it needs some information about the used MPI environment. In particular the MPI datatypes which match the C99 types uint64_t
and int64_t
. In case you have a MPI 3.0 conforming MPI implementation you can skip this. If not, provide #define's for the following macros prior the #include statement. In this example, we assume a LP64 platform.
After this preparatory step, we can include the <otf2/OTF2_MPI_Collectives.h> header.
We use MPI_Wtime
to get timestamps for our events but need to convert the seconds to an integral value. We use a nano second resolution.
Define a pre and post flush callback. If no memory is left in OTF2's internal memory buffer or the writer handle is closed a memory buffer flushing routine is triggered. The pre flush callback is triggered right before a buffer flush. It needs to return either OTF2_FLUSH to flush the recorded data to a file or OTF2_NO_FLUSH to suppress flushing data to a file. The post flush callback is triggered right after a memory buffer flush. It has to return a current timestamp which is recorded to mark the time spend in a buffer flush. The callbacks are passed via a struct to OTF2.
Now everything is prepared to begin with the main program.
First initialize the MPI environment and query the size and rank.
Create new archive handle.
Set the previously defined flush callbacks.
Now we provide the OTF2 archive object the MPI collectives. As all ranks in MPI_COMM_WORLD
write into the archive, we use this communicator as the global one. We set the local communicator to MPI_COMM_NULL
, as we don't care about file optimization here.
Now we can create the event files. Though physical files aren't created yet.
Each rank now requests an event writer with its rank number as the location id.
We note the start time in each rank, this is later used to determine the global epoch.
Write an enter and a leave record for region 0 to the local event writer.
We also record a MPI_Barrier
in the trace. For this we generate an event before we do the MPI call.
Now we can do the MPI_Barrier
call.
After we passed the MPI_Barrier
. we can note the end of the collective operation inside the event stream.
Finally we leave the region again with the leave region.
The event recording is now done, note the end time in each rank.
Now close the event writer, before closing the event files collectively.
After we wrote all of the events we close the event files again.
We now collect all of the epoch_start
and epoch_end
timestamps by calculating the minimum and maximize and provide these to the root rank.
Only the root rank will write the global definitions, thus only he requests a writer object from the archive.
We need to define the clock used for this trace and the overall timestamp range.
Now we can start writing the referenced definitions, starting with the strings.
Write definition for the code region which was just entered and left to the global definition writer.
Write the system tree to the global definition writer.
For each rank we define a new location group and one location. We provide also a unique string for each location group.
The last step is to define the MPI communicator. This is a three-step process. First we define that this trace actually recorded in the MPI paradigm and enumerate all locations which participate in this paradigm. As we used the MPI ranks directly as the location id, the array with the locations is the identity.
Now we can define sub-groups of the previously defined list of communication. locations. For MPI_COMM_WORLD
this is the whole group here. Note the these sub-groups are created by using indices into the list of communication locations, and not by enumerating location ids again. But in this example the sub-group is the identity again.
Finally we can write the definition of the MPI_COMM_WORLD
communicator. This finalizes the writing of the global definitions and we can also close the writer object.
All the other ranks wait inside this barrier so that root can write the global definitions.
At the end, close the archive, finalize the MPI environment, and exit.
To compile your program use a command like the following. Note that we need to activate the C99 standard explicitly for GCC.
Now you can link your program with: