DNDSR 0.1.0.dev1+gcd065ad
Distributed Numeric Data Structure for CFV
Loading...
Searching...
No Matches
MPI.hpp File Reference

MPI wrappers: MPIInfo, collective operations, type mapping, CommStrategy. More...

#include <vector>
#include <fstream>
#include <mutex>
#include "Defines.hpp"
#include "mpi.h"
Include dependency graph for MPI.hpp:
This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Classes

struct  DNDS::has_static_CommMult< T, typename >
 here are some reasons to upgrade to C++20... More...
 
struct  DNDS::has_static_CommMult< T, std::void_t< decltype(T::CommMult())> >
 
struct  DNDS::has_static_CommType< T, typename >
 SFINAE trait detecting a static CommType member in T. More...
 
struct  DNDS::has_static_CommType< T, std::void_t< decltype(T::CommType())> >
 
struct  DNDS::MPIInfo
 Lightweight bundle of an MPI communicator and the calling rank's coordinates. More...
 
class  DNDS::MPI::ResourceRecycler
 Singleton that tracks and releases long-lived MPI resources at MPI_Finalize time. More...
 
struct  DNDS::MPITypePairHolder
 RAII vector of (count, MPI_Datatype) pairs that frees every committed datatype when destroyed. More...
 
struct  DNDS::MPIReqHolder
 RAII vector of MPI_Requests that frees each non-null handle when destroyed. More...
 
class  DNDS::MPIBufferHandler
 Process-singleton managing the buffer attached to MPI for MPI_Bsend (buffered sends). More...
 
class  DNDS::MPI::CommStrategy
 Process-wide singleton that selects how ArrayTransformer packs and waits for MPI messages. More...
 

Namespaces

namespace  DNDS
 the host side operators are provided as implemented
 
namespace  DNDS::MPI
 
namespace  DNDS::Debug
 

Macros

#define MAX_MPI_int   INT32_MAX
 
#define MAX_MPI_Aint   INT64_MAX
 
#define DNDS_MPI_InsertCheck(mpi, info)    InsertCheck(mpi, info, __FUNCTION__, __FILE__, __LINE__)
 Debug helper: barrier + print "CHECK <info> RANK r @ fn @ file:line".
 
#define DNDS_assert_info_mpi(expr, mpi, info)
 Collective-aware variant of DNDS_assert_info: every rank calls into an MPI barrier before aborting, so logs are not interleaved.
 
#define MPIBufferHandler_REPORT_CHANGE
 

Typedefs

using DNDS::MPI_int = int
 MPI counterpart type for MPI_int (= C int). Used for counts and ranks in MPI calls.
 
using DNDS::MPI_index = MPI_Aint
 MPI-compatible address/offset type (= MPI_Aint, 64-bit on all supported platforms). Used by the hindexed datatype machinery.
 
using DNDS::tMPI_sizeVec = std::vector< MPI_int >
 Vector of MPI counts.
 
using DNDS::tMPI_intVec = tMPI_sizeVec
 Alias for tMPI_sizeVec; used where the name "int vec" reads better.
 
using DNDS::tMPI_indexVec = std::vector< MPI_index >
 Vector of MPI_Aint byte-offsets for hindexed datatypes.
 
using DNDS::tMPI_AintVec = tMPI_indexVec
 Alias for tMPI_indexVec to match MPI_Aint terminology.
 
using DNDS::tMPI_statVec = std::vector< MPI_Status >
 Vector of MPI_Status, for MPI_Waitall / MPI_Testall.
 
using DNDS::tMPI_reqVec = std::vector< MPI_Request >
 Vector of MPI_Request, for persistent / nonblocking calls.
 
using DNDS::tMPI_typePairVec = std::vector< std::pair< MPI_int, MPI_Datatype > >
 
using DNDS::tpMPITypePairHolder = ssp< MPITypePairHolder >
 Shared-pointer alias to MPITypePairHolder.
 

Functions

template<class Tbasic >
constexpr MPI_Datatype DNDS::__DNDSToMPITypeInt ()
 Map a DNDS integer type size to an MPI signed-integer datatype.
 
template<class Tbasic >
constexpr MPI_Datatype DNDS::__DNDSToMPITypeFloat ()
 Map a DNDS floating-point type size to an MPI datatype.
 
template<class T >
std::pair< MPI_Datatype, MPI_intDNDS::BasicType_To_MPIIntType_Custom ()
 Dispatch to a user-provided CommPair / CommMult+ CommType pair on T.
 
template<class T >
std::pair< MPI_Datatype, MPI_intDNDS::BasicType_To_MPIIntType ()
 Deduce an (MPI_Datatype, count) pair that represents a T value.
 
MPI_int DNDS::MPIWorldSize ()
 Convenience: MPI_Comm_size(MPI_COMM_WORLD).
 
MPI_int DNDS::MPIWorldRank ()
 Convenience: MPI_Comm_rank(MPI_COMM_WORLD).
 
std::string DNDS::getTimeStamp (const MPIInfo &mpi)
 Format a human-readable timestamp using the calling rank as context.
 
bool DNDS::Debug::IsDebugged ()
 Whether the current process is running under a debugger. Implemented via /proc/self/status TracerPid on Linux.
 
void DNDS::Debug::MPIDebugHold (const MPIInfo &mpi)
 If isDebugging is set, block every rank in a busy-wait loop so the user can attach a debugger and inspect state. Exit by setting isDebugging = false in the debugger.
 
void DNDS::assert_false_info_mpi (const char *expr, const char *file, int line, const std::string &info, const DNDS::MPIInfo &mpi)
 MPI-aware assertion-failure reporter.
 
int DNDS::MPI::GetMPIThreadLevel ()
 Return the MPI thread-support level the current process was initialised with.
 
MPI_int DNDS::MPI::Init_thread (int *argc, char ***argv)
 Initialise MPI with thread support, honouring the DNDS_DISABLE_ASYNC_MPI environment override.
 
int DNDS::MPI::Finalize ()
 Release DNDSR-registered MPI resources then call MPI_Finalize.
 
MPI_int DNDS::MPI::Bcast (void *buf, MPI_int num, MPI_Datatype type, MPI_int source_rank, MPI_Comm comm)
 dumb wrapper
 
MPI_int DNDS::MPI::Alltoall (void *send, MPI_int sendNum, MPI_Datatype typeSend, void *recv, MPI_int recvNum, MPI_Datatype typeRecv, MPI_Comm comm)
 Wrapper over MPI_Alltoall (fixed per-peer count).
 
MPI_int DNDS::MPI::Alltoallv (void *send, MPI_int *sendSizes, MPI_int *sendStarts, MPI_Datatype sendType, void *recv, MPI_int *recvSizes, MPI_int *recvStarts, MPI_Datatype recvType, MPI_Comm comm)
 Wrapper over MPI_Alltoallv (variable per-peer counts + displacements).
 
MPI_int DNDS::MPI::Allreduce (const void *sendbuf, void *recvbuf, MPI_int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
 Wrapper over MPI_Allreduce.
 
MPI_int DNDS::MPI::Scan (const void *sendbuf, void *recvbuf, MPI_int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
 Wrapper over MPI_Scan (inclusive prefix reduction).
 
MPI_int DNDS::MPI::Allgather (const void *sendbuf, MPI_int sendcount, MPI_Datatype sendtype, void *recvbuf, MPI_int recvcount, MPI_Datatype recvtype, MPI_Comm comm)
 Wrapper over MPI_Allgather.
 
MPI_int DNDS::MPI::Barrier (MPI_Comm comm)
 Wrapper over MPI_Barrier.
 
MPI_int DNDS::MPI::BarrierLazy (MPI_Comm comm, uint64_t checkNanoSecs)
 Polling barrier that sleeps checkNanoSecs ns between MPI_Test calls. Reduces CPU spin when many ranks wait unevenly.
 
MPI_int DNDS::MPI::WaitallLazy (MPI_int count, MPI_Request *reqs, MPI_Status *statuses, uint64_t checkNanoSecs=10000000)
 Like WaitallAuto but sleeps checkNanoSecs ns between polls.
 
MPI_int DNDS::MPI::WaitallAuto (MPI_int count, MPI_Request *reqs, MPI_Status *statuses)
 Wait on an array of requests, choosing between MPI_Waitall and the lazy-poll variant based on CommStrategy settings.
 
void DNDS::MPI::AllreduceOneReal (real &v, MPI_Op op, const MPIInfo &mpi)
 Single-scalar Allreduce helper for reals (in-place, count = 1).
 
void DNDS::MPI::AllreduceOneIndex (index &v, MPI_Op op, const MPIInfo &mpi)
 Single-scalar Allreduce helper for indices (in-place, count = 1).
 
template<class F >
void DNDS::MPISerialDo (const MPIInfo &mpi, F f)
 Execute f on each rank serially, in rank order.
 
bool DNDS::MPI::isCudaAware ()
 Runtime probe: is the current MPI implementation configured with CUDA-aware support? Affects whether arrays are transferred on-device or via the host round-trip.
 
void DNDS::InsertCheck (const MPIInfo &mpi, const std::string &info="", const std::string &FUNCTION="", const std::string &FILE="", int LINE=-1)
 Barrier + annotated print used by DNDS_MPI_InsertCheck.
 

Variables

const MPI_Datatype DNDS::DNDS_MPI_INDEX = __DNDSToMPITypeInt<index>()
 MPI datatype matching index (= MPI_INT64_T).
 
const MPI_Datatype DNDS::DNDS_MPI_REAL = __DNDSToMPITypeFloat<real>()
 MPI datatype matching real (= MPI_REAL8).
 

Detailed Description

MPI wrappers: MPIInfo, collective operations, type mapping, CommStrategy.

Unit Test Coverage (test_MPI.cpp, MPI np=1,2,4)
  • MPIInfo: constructor, setWorld, operator==, field validation
  • MPIWorldSize, MPIWorldRank
  • Allreduce (MPI_SUM, MPI_MAX for real/index), AllreduceOneReal, AllreduceOneIndex
  • Scan (MPI_SUM), Allgather, Bcast, Barrier, Alltoall
  • BasicType_To_MPIIntType (scalar, std::array, Eigen::Matrix)
  • CommStrategy: get/set HIndexed/InSituPack
Not Yet Tested
  • Alltoallv, WaitallLazy, WaitallAuto, BarrierLazy
  • MPIBufferHandler, MPITypePairHolder, MPIReqHolder (tested indirectly via ArrayTransformer)
  • MPI::ResourceRecycler, MPISerialDo, InsertCheck
  • Sub-communicators, CommStrategy functional impact on ArrayTransformer

Definition in file MPI.hpp.

Macro Definition Documentation

◆ DNDS_assert_info_mpi

#define DNDS_assert_info_mpi (   expr,
  mpi,
  info 
)
Value:
((static_cast<bool>(expr)) \
? void(0) \
: ::DNDS::assert_false_info_mpi(#expr, __FILE__, __LINE__, info, mpi))
void assert_false_info_mpi(const char *expr, const char *file, int line, const std::string &info, const DNDS::MPIInfo &mpi)
MPI-aware assertion-failure reporter.
Definition MPI.cpp:95

Collective-aware variant of DNDS_assert_info: every rank calls into an MPI barrier before aborting, so logs are not interleaved.

Definition at line 458 of file MPI.hpp.

◆ DNDS_MPI_InsertCheck

#define DNDS_MPI_InsertCheck (   mpi,
  info 
)     InsertCheck(mpi, info, __FUNCTION__, __FILE__, __LINE__)

Debug helper: barrier + print "CHECK <info> RANK r @ fn @ file:line".

Compiled out when either NDEBUG or NINSERT is defined. Used to trace parallel execution during development; see InsertCheck.

Definition at line 320 of file MPI.hpp.

◆ MAX_MPI_Aint

#define MAX_MPI_Aint   INT64_MAX

Definition at line 48 of file MPI.hpp.

◆ MAX_MPI_int

#define MAX_MPI_int   INT32_MAX

Definition at line 47 of file MPI.hpp.

◆ MPIBufferHandler_REPORT_CHANGE

#define MPIBufferHandler_REPORT_CHANGE

Definition at line 544 of file MPI.hpp.