Usage in reading mode - MPI example

This is a example of how to use the OTF2 reading interface with MPI. It shows how to define and register callbacks and how to use the provided MPI collective callbacks to read all events of a given OTF2 archive in parallel. This example is available as source code in the file otf2_mpi_reader_example.c .

We start with inclusion of some standard headers.

#include <stdlib.h>
#include <stdio.h>
#include <inttypes.h>

And than include the MPI and OTF2 header.

#include <mpi.h>

#include <otf2/otf2.h>

Now prepare the inclusion of the <otf2/OTF2_MPI_Collectives.h> header. As it is a header-only interface, it needs some information about the used MPI environment. In particular the MPI datatypes which match the C99 types uint64_t and int64_t. In case you have a MPI 3.0 conforming MPI implementation you can skip this. If not, provide #define's for the following macros prior the #include statement. In this example, we assume a LP64 platform.

#if MPI_VERSION < 3
#define OTF2_MPI_UINT64_T MPI_UNSIGNED_LONG
#define OTF2_MPI_INT64_T  MPI_LONG
#endif

After this preparatory step, we can include the <otf2/OTF2_MPI_Collectives.h> header.

#include <otf2/OTF2_MPI_Collectives.h>

The following section until describing main is the same as in the Usage in reading mode - a simple example.

Define an event callback for entering and leaving a region.

static OTF2_CallbackCode
Enter_print( OTF2_LocationRef    location,
             OTF2_TimeStamp      time,
             void*               userData,
             OTF2_AttributeList* attributes,
             OTF2_RegionRef      region )
{
    printf( "Entering region %u at location %" PRIu64 " at time %" PRIu64 ".\n",
            region, location, time );

    return OTF2_CALLBACK_SUCCESS;
}

static OTF2_CallbackCode
Leave_print( OTF2_LocationRef    location,
             OTF2_TimeStamp      time,
             void*               userData,
             OTF2_AttributeList* attributes,
             OTF2_RegionRef      region )
{
    printf( "Leaving region %u at location %" PRIu64 " at time %" PRIu64 ".\n",
            region, location, time );

    return OTF2_CALLBACK_SUCCESS;
}

The global definition file provides all location IDs that are included in the OTF2 trace archive. When reading the global definitions these location IDs must be collected and stored by the user. Probably, the easiest way to do that is to use a C++ container.

struct vector
{
    size_t   capacity;
    size_t   size;
    uint64_t members[];
};

static OTF2_CallbackCode
GlobDefLocation_Register( void*                 userData,
                          OTF2_LocationRef      location,
                          OTF2_StringRef        name,
                          OTF2_LocationType     locationType,
                          uint64_t              numberOfEvents,
                          OTF2_LocationGroupRef locationGroup )
{
    struct vector* locations = userData;

    if ( locations->size == locations->capacity )
    {
        return OTF2_CALLBACK_INTERRUPT;
    }

    locations->members[ locations->size++ ] = location;

    return OTF2_CALLBACK_SUCCESS;
}

Now everything is prepared to begin with the main program.

int
main( int    argc,
      char** argv )
{

First initialize the MPI environment and query the size and rank.

    MPI_Init( &argc, &argv );
    int size;
    MPI_Comm_size( MPI_COMM_WORLD, &size );
    int rank;
    MPI_Comm_rank( MPI_COMM_WORLD, &rank );

Create a new reader handle. The path to the OTF2 anchor file must be provided as argument.

    OTF2_Reader* reader = OTF2_Reader_Open( "ArchivePath/ArchiveName.otf2" );

Now we provide the OTF2 reader object the MPI collectives.

    OTF2_MPI_Reader_SetCollectiveCallbacks( reader, MPI_COMM_WORLD );

OTF2 provides an API to query the number of locations prior reading the global definitions. We use this to pre-allocate the storage for all locations.

    uint64_t number_of_locations;
    OTF2_Reader_GetNumberOfLocations( reader,
                                      &number_of_locations );
    struct vector* locations = malloc( sizeof( *locations )
                                       + number_of_locations
                                       * sizeof( *locations->members ) );
    locations->capacity = number_of_locations;
    locations->size     = 0;

All ranks need to read the global definitions to know the list of locations in the trace. Get a global definition reader with the above reader handle as argument.

    OTF2_GlobalDefReader* global_def_reader = OTF2_Reader_GetGlobalDefReader( reader );

Register the above defined global definition callbacks. All other definition callbacks will be deactivated. And instruct the reader to pass the locations object to each call of the callbacks.

    OTF2_GlobalDefReaderCallbacks* global_def_callbacks = OTF2_GlobalDefReaderCallbacks_New();
    OTF2_GlobalDefReaderCallbacks_SetLocationCallback( global_def_callbacks,
                                                       &GlobDefLocation_Register );
    OTF2_Reader_RegisterGlobalDefCallbacks( reader,
                                            global_def_reader,
                                            global_def_callbacks,
                                            locations );
    OTF2_GlobalDefReaderCallbacks_Delete( global_def_callbacks );

Read all global definitions. Every time a location definition is read, the previously registered callback is triggered. In definitions_read the number of read definitions is returned.

    uint64_t definitions_read = 0;
    OTF2_Reader_ReadAllGlobalDefinitions( reader,
                                          global_def_reader,
                                          &definitions_read );

After reading all global definitions all location IDs are stored in the generic container ListOfLocations. After that, the locations that are supposed to be read are selected. We distribute the locations round-robin to all ranks in MPI_COMM_WORLD. We need also to remember, whether this rank actually reads any locations.

    uint64_t number_of_locations_to_read = 0;
    for ( size_t i = 0; i < locations->size; i++ )
    {
        if ( locations->members[ i ] % size != rank )
        {
            continue;
        }
        number_of_locations_to_read++;
        OTF2_Reader_SelectLocation( reader, locations->members[ i ] );
    }

When the locations are selected the according event and definition files can be opened. Note that the local definition files are optional, thus we need to remember the success of this call.

    bool successful_open_def_files =
        OTF2_Reader_OpenDefFiles( reader ) == OTF2_SUCCESS;
    OTF2_Reader_OpenEvtFiles( reader );

When the files are opened the event and definition reader handle can be requested. We distribute the locations round-robin to all ranks in MPI_COMM_WORLD. To apply mapping tables stored in the local definitions, the local definitions must be read. Though the existence of these files are optional. The call to OTF2_Reader_GetEvtReader is mandatory, but the result is unused.

    for ( size_t i = 0; i < locations->size; i++ )
    {
        if ( locations->members[ i ] % size != rank )
        {
            continue;
        }

        if ( successful_open_def_files )
        {
            OTF2_DefReader* def_reader =
                OTF2_Reader_GetDefReader( reader, locations->members[ i ] );
            if ( def_reader )
            {
                uint64_t def_reads = 0;
                OTF2_Reader_ReadAllLocalDefinitions( reader,
                                                     def_reader,
                                                     &def_reads );
                OTF2_Reader_CloseDefReader( reader, def_reader );
            }
        }
        OTF2_EvtReader* evt_reader =
            OTF2_Reader_GetEvtReader( reader, locations->members[ i ] );
    }

The definition files can now be closed, if it was successfully opened in the first place.

    if ( successful_open_def_files )
    {
        OTF2_Reader_CloseDefFiles( reader );
    }

Only these ranks which actually read events for locations, can now open a new global event reader. This global reader automatically contains all previously opened local event readers.

    if ( number_of_locations_to_read > 0 )
    {
        OTF2_GlobalEvtReader* global_evt_reader = OTF2_Reader_GetGlobalEvtReader( reader );

Register the above defined global event callbacks. All other global event callbacks will be deactivated.

        OTF2_GlobalEvtReaderCallbacks* event_callbacks = OTF2_GlobalEvtReaderCallbacks_New();
        OTF2_GlobalEvtReaderCallbacks_SetEnterCallback( event_callbacks,
                                                        &Enter_print );
        OTF2_GlobalEvtReaderCallbacks_SetLeaveCallback( event_callbacks,
                                                        &Leave_print );
        OTF2_Reader_RegisterGlobalEvtCallbacks( reader,
                                                global_evt_reader,
                                                event_callbacks,
                                                NULL );
        OTF2_GlobalEvtReaderCallbacks_Delete( event_callbacks );

Read all events in the OTF2 archive. The events are automatically ordered by the time they occurred in the trace. Every time an enter or leave event is read, the previously registered callbacks are triggered. In events_read the number of read events is returned.

        uint64_t events_read = 0;
        OTF2_Reader_ReadAllGlobalEvents( reader,
                                         global_evt_reader,
                                         &events_read );

The global event reader can now be closed and the event files too.

        OTF2_Reader_CloseGlobalEvtReader( reader, global_evt_reader );

As the call to OTF2_Reader_CloseEvtFiles is a collective operation all ranks need to call this, not only those which read events.

    }
    OTF2_Reader_CloseEvtFiles( reader );

At the end, close the reader and exit. All opened event and definition readers are closed automatically. Free resources and finalize the MPI environment.

    OTF2_Reader_Close( reader );

    free( locations );

    MPI_Finalize();

    return EXIT_SUCCESS;
}

To compile your program use a command like the following. Note that we need to activate the C99 standard explicitly for GCC.

    mpicc -std=c99 `otf2-config --cflags` \
            -c otf2_mpi_reader_example.c \
            -o otf2_mpi_reader_example.o

Now you can link your program with:

    mpicc otf2_mpi_reader_example.o \
            `otf2-config --ldflags` \
            `otf2-config --libs` \
            -o otf2_mpi_reader_example