Abstract interface for reading/writing scalars, vectors, and byte arrays. More...

#include <SerializerBase.hpp>

Inheritance diagram for DNDS::Serializer::SerializerBase:

Collaboration diagram for DNDS::Serializer::SerializerBase:

Public Member Functions
virtual	~SerializerBase ()

virtual void	OpenFile (const std::string &fName, bool read)=0
	Open a backing file (H5 file or JSON file depending on subclass).

virtual void	CloseFile ()=0
	Close the backing file, flushing buffers.

virtual void	CreatePath (const std::string &p)=0
	Create a sub-path (H5 group / JSON object) at the current location.

virtual void	GoToPath (const std::string &p)=0
	Navigate to an existing path. Supports `/` -separated segments.

virtual bool	IsPerRank ()=0
	Whether this serializer is per-rank (JSON-file-per-rank) or collective (shared H5 file). Controls which API entry points are used.

virtual std::string	GetCurrentPath ()=0
	String form of the current path.

virtual std::set< std::string >	ListCurrentPath ()=0
	Names of direct children of the current path.

virtual int	GetMPIRank ()=0
	Rank index cached by the serializer (relevant for collective I/O).

virtual int	GetMPISize ()=0
	Rank count cached by the serializer.

virtual const MPIInfo &	getMPI ()=0
	MPI context the serializer was opened with.

virtual void	WriteInt (const std::string &name, int v)=0
	Write a scalar int under `name` at the current path.

virtual void	WriteIndex (const std::string &name, index v)=0
	Write a scalar index under `name`.

virtual void	WriteReal (const std::string &name, real v)=0
	Write a scalar real under `name`.

virtual void	WriteString (const std::string &name, const std::string &v)=0
	Write a UTF-8 string under `name`.

virtual void	WriteIndexVector (const std::string &name, const std::vector< index > &v, ArrayGlobalOffset offset)=0
	Write an index vector (collective for H5). `offset` carries the distribution mode (ArrayGlobalOffset_Parts, explicit offset, etc.).

virtual void	WriteRowsizeVector (const std::string &name, const std::vector< rowsize > &v, ArrayGlobalOffset offset)=0
	Write a rowsize vector (collective for H5).

virtual void	WriteRealVector (const std::string &name, const std::vector< real > &v, ArrayGlobalOffset offset)=0
	Write a real vector (collective for H5).

virtual void	WriteSharedIndexVector (const std::string &name, const ssp< host_device_vector< index > > &v, ArrayGlobalOffset offset)=0
	Write a shared index vector; deduplicated across multiple writes that share the same `shared_ptr`.

virtual void	WriteSharedRowsizeVector (const std::string &name, const ssp< host_device_vector< rowsize > > &v, ArrayGlobalOffset offset)=0
	Write a shared rowsize vector; deduplicated across multiple writes.

virtual void	WriteUint8Array (const std::string &name, const uint8_t *data, index size, ArrayGlobalOffset offset)=0
	Write a raw byte buffer under `name`. `offset.isDist()` = true means the caller provides the exact per-rank slab; otherwise the buffer is treated according to the offset's sentinel.

virtual void	WriteIndexVectorPerRank (const std::string &name, const std::vector< index > &v)=0
	Write a per-rank index vector (replicated name, independent values).

virtual void	ReadInt (const std::string &name, int &v)=0
	Read a scalar int into `v`.

virtual void	ReadIndex (const std::string &name, index &v)=0
	Read a scalar index into `v`.

virtual void	ReadReal (const std::string &name, real &v)=0
	Read a scalar real into `v`.

virtual void	ReadString (const std::string &name, std::string &v)=0
	Read a UTF-8 string into `v`.

virtual void	ReadUint8Array (const std::string &name, uint8_t *data, index &size, ArrayGlobalOffset &offset)=0
	Two-pass byte array read.


virtual void	ReadIndexVector (const std::string &name, std::vector< index > &v, ArrayGlobalOffset &offset)=0

virtual void	ReadRowsizeVector (const std::string &name, std::vector< rowsize > &v, ArrayGlobalOffset &offset)=0

virtual void	ReadRealVector (const std::string &name, std::vector< real > &v, ArrayGlobalOffset &offset)=0

virtual void	ReadSharedIndexVector (const std::string &name, ssp< host_device_vector< index > > &v, ArrayGlobalOffset &offset)=0

virtual void	ReadSharedRowsizeVector (const std::string &name, ssp< host_device_vector< rowsize > > &v, ArrayGlobalOffset &offset)=0

Protected Member Functions
template<class T >
bool	dedupLookup (const ssp< T > &v, std::string &outPath)
	Check if a shared pointer was already written; if so return its path.

template<class T >
void	dedupRegister (const ssp< T > &v, const std::string &path)
	Register a shared pointer after writing its data.

void	dedupClear ()
	Clear all dedup state (call on CloseFile).

Protected Attributes
std::map< void *, std::pair< std::shared_ptr< void >, std::string > >	ptr_2_pth

std::map< std::string, void * >	pth_2_ssp
	Reverse map for read-side dedup: path -> raw pointer to the ssp local variable.

Detailed Description

Abstract interface for reading/writing scalars, vectors, and byte arrays.

Collective semantics (SerializerH5)

For the HDF5 implementation, all Read/Write vector and array methods are MPI-collective: every rank must call the same method in the same order, even if a rank reads/writes 0 elements. Failing to participate causes a hang because HDF5 collective I/O synchronizes across the communicator.

Zero-size reads

Some ranks may legitimately have 0 elements to read (e.g., when nGlobal < nRanks in an even-split read). The Read*Vector and ReadShared*Vector implementations handle this by passing a dummy non-null pointer to the internal HDF5 read when the output container is empty, so that std::vector<>::data() returning nullptr does not cause the rank to skip the collective H5Dread call.

Warning: ReadUint8Array uses an explicit two-pass pattern: the caller first calls with data == nullptr to query the size, then calls again with a buffer. When the queried size is 0, the caller must still pass a non-null data pointer on the second call so that the collective H5Dread is not skipped. Use a stack dummy:
uint8_t dummy;

ser->ReadUint8Array(name, bufferSize == 0 ? &dummy : buf, bufferSize, offset);

Definition at line 153 of file SerializerBase.hpp.

Constructor & Destructor Documentation

◆ ~SerializerBase()

DNDS::Serializer::SerializerBase::~SerializerBase ( )

virtualdefault

Member Function Documentation

◆ CloseFile()

virtual void DNDS::Serializer::SerializerBase::CloseFile ( )

pure virtual

Close the backing file, flushing buffers.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ CreatePath()

virtual void DNDS::Serializer::SerializerBase::CreatePath ( const std::string & p )

pure virtual

Create a sub-path (H5 group / JSON object) at the current location.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ dedupClear()

void DNDS::Serializer::SerializerBase::dedupClear ( )

inlineprotected

Clear all dedup state (call on CloseFile).

Definition at line 185 of file SerializerBase.hpp.

Here is the caller graph for this function:

◆ dedupLookup()

template<class T >

bool DNDS::Serializer::SerializerBase::dedupLookup	(	const ssp< T > &	v,
		std::string &	outPath
	)

inlineprotected

Check if a shared pointer was already written; if so return its path.

Definition at line 166 of file SerializerBase.hpp.

Here is the caller graph for this function:

◆ dedupRegister()

template<class T >

void DNDS::Serializer::SerializerBase::dedupRegister	(	const ssp< T > &	v,
		const std::string &	path
	)

inlineprotected

Register a shared pointer after writing its data.

Definition at line 179 of file SerializerBase.hpp.

Here is the caller graph for this function:

◆ GetCurrentPath()

virtual std::string DNDS::Serializer::SerializerBase::GetCurrentPath ( )

pure virtual

String form of the current path.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ getMPI()

virtual const MPIInfo & DNDS::Serializer::SerializerBase::getMPI ( )

pure virtual

MPI context the serializer was opened with.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ GetMPIRank()

virtual int DNDS::Serializer::SerializerBase::GetMPIRank ( )

pure virtual

Rank index cached by the serializer (relevant for collective I/O).

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ GetMPISize()

virtual int DNDS::Serializer::SerializerBase::GetMPISize ( )

pure virtual

Rank count cached by the serializer.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ GoToPath()

virtual void DNDS::Serializer::SerializerBase::GoToPath ( const std::string & p )

pure virtual

Navigate to an existing path. Supports / -separated segments.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ IsPerRank()

virtual bool DNDS::Serializer::SerializerBase::IsPerRank ( )

pure virtual

Whether this serializer is per-rank (JSON-file-per-rank) or collective (shared H5 file). Controls which API entry points are used.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ListCurrentPath()

virtual std::set< std::string > DNDS::Serializer::SerializerBase::ListCurrentPath ( )

pure virtual

Names of direct children of the current path.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ OpenFile()

virtual void DNDS::Serializer::SerializerBase::OpenFile	(	const std::string &	fName,
		bool	read
	)

pure virtual

Open a backing file (H5 file or JSON file depending on subclass).

Parameters

read	`true` for reading, `false` for writing.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadIndex()

virtual void DNDS::Serializer::SerializerBase::ReadIndex	(	const std::string &	name,
		index &	v
	)

pure virtual

Read a scalar index into v.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadIndexVector()

virtual void DNDS::Serializer::SerializerBase::ReadIndexVector	(	const std::string &	name,
		std::vector< index > &	v,
		ArrayGlobalOffset &	offset
	)

pure virtual

Read methods resize the output container and populate it. Internally these use a two-pass HDF5 pattern (size query, then data read). Both passes are collective. When the local size is 0, a dummy non-null pointer is passed to the second pass so the rank participates in H5Dread.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadInt()

virtual void DNDS::Serializer::SerializerBase::ReadInt	(	const std::string &	name,
		int &	v
	)

pure virtual

Read a scalar int into v.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadReal()

virtual void DNDS::Serializer::SerializerBase::ReadReal	(	const std::string &	name,
		real &	v
	)

pure virtual

Read a scalar real into v.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadRealVector()

virtual void DNDS::Serializer::SerializerBase::ReadRealVector	(	const std::string &	name,
		std::vector< real > &	v,
		ArrayGlobalOffset &	offset
	)

pure virtual

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadRowsizeVector()

virtual void DNDS::Serializer::SerializerBase::ReadRowsizeVector	(	const std::string &	name,
		std::vector< rowsize > &	v,
		ArrayGlobalOffset &	offset
	)

pure virtual

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadSharedIndexVector()

virtual void DNDS::Serializer::SerializerBase::ReadSharedIndexVector	(	const std::string &	name,
		ssp< host_device_vector< index > > &	v,
		ArrayGlobalOffset &	offset
	)

pure virtual

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadSharedRowsizeVector()

virtual void DNDS::Serializer::SerializerBase::ReadSharedRowsizeVector	(	const std::string &	name,
		ssp< host_device_vector< rowsize > > &	v,
		ArrayGlobalOffset &	offset
	)

pure virtual

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadString()

virtual void DNDS::Serializer::SerializerBase::ReadString	(	const std::string &	name,
		std::string &	v
	)

pure virtual

Read a UTF-8 string into v.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ ReadUint8Array()

virtual void DNDS::Serializer::SerializerBase::ReadUint8Array	(	const std::string &	name,
		uint8_t *	data,
		index &	size,
		ArrayGlobalOffset &	offset
	)

pure virtual

Two-pass byte array read.

Pass 1 (data == nullptr): queries the local element count into size and resolves offset. Pass 2 (data != nullptr): reads size bytes into data.

Warning: When size is 0 after pass 1, the caller must still pass a non-null data pointer on pass 2 so the rank participates in the collective HDF5 read. Use a stack dummy (uint8_t dummy; ... &dummy).

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteIndex()

virtual void DNDS::Serializer::SerializerBase::WriteIndex	(	const std::string &	name,
		index	v
	)

pure virtual

Write a scalar index under name.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteIndexVector()

virtual void DNDS::Serializer::SerializerBase::WriteIndexVector	(	const std::string &	name,
		const std::vector< index > &	v,
		ArrayGlobalOffset	offset
	)

pure virtual

Write an index vector (collective for H5). offset carries the distribution mode (ArrayGlobalOffset_Parts, explicit offset, etc.).

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteIndexVectorPerRank()

virtual void DNDS::Serializer::SerializerBase::WriteIndexVectorPerRank	(	const std::string &	name,
		const std::vector< index > &	v
	)

pure virtual

Write a per-rank index vector (replicated name, independent values).

Every rank writes its own vector under name; in the H5 case each rank's slab is placed in a separate dataset.

Parameters

name	Dataset name (identical on every rank).
v	Rank-local vector; size may differ between ranks.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteInt()

virtual void DNDS::Serializer::SerializerBase::WriteInt	(	const std::string &	name,
		int	v
	)

pure virtual

Write a scalar int under name at the current path.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteReal()

virtual void DNDS::Serializer::SerializerBase::WriteReal	(	const std::string &	name,
		real	v
	)

pure virtual

Write a scalar real under name.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteRealVector()

virtual void DNDS::Serializer::SerializerBase::WriteRealVector	(	const std::string &	name,
		const std::vector< real > &	v,
		ArrayGlobalOffset	offset
	)

pure virtual

Write a real vector (collective for H5).

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteRowsizeVector()

virtual void DNDS::Serializer::SerializerBase::WriteRowsizeVector	(	const std::string &	name,
		const std::vector< rowsize > &	v,
		ArrayGlobalOffset	offset
	)

pure virtual

Write a rowsize vector (collective for H5).

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteSharedIndexVector()

virtual void DNDS::Serializer::SerializerBase::WriteSharedIndexVector	(	const std::string &	name,
		const ssp< host_device_vector< index > > &	v,
		ArrayGlobalOffset	offset
	)

pure virtual

Write a shared index vector; deduplicated across multiple writes that share the same shared_ptr.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteSharedRowsizeVector()

virtual void DNDS::Serializer::SerializerBase::WriteSharedRowsizeVector	(	const std::string &	name,
		const ssp< host_device_vector< rowsize > > &	v,
		ArrayGlobalOffset	offset
	)

pure virtual

Write a shared rowsize vector; deduplicated across multiple writes.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteString()

virtual void DNDS::Serializer::SerializerBase::WriteString	(	const std::string &	name,
		const std::string &	v
	)

pure virtual

Write a UTF-8 string under name.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

◆ WriteUint8Array()

virtual void DNDS::Serializer::SerializerBase::WriteUint8Array	(	const std::string &	name,
		const uint8_t *	data,
		index	size,
		ArrayGlobalOffset	offset
	)

pure virtual

Write a raw byte buffer under name. offset.isDist() = true means the caller provides the exact per-rank slab; otherwise the buffer is treated according to the offset's sentinel.

Implemented in DNDS::Serializer::SerializerH5, and DNDS::Serializer::SerializerJSON.

Member Data Documentation

◆ pth_2_ssp

std::map<std::string, void *> DNDS::Serializer::SerializerBase::pth_2_ssp

protected

Reverse map for read-side dedup: path -> raw pointer to the ssp local variable.

Definition at line 162 of file SerializerBase.hpp.

◆ ptr_2_pth

std::map<void *, std::pair<std::shared_ptr<void>, std::string> > DNDS::Serializer::SerializerBase::ptr_2_pth

protected

Shared-pointer deduplication map: raw pointer -> (kept-alive shared_ptr, H5/JSON path). By storing the shared_ptr itself we prevent the pointed-to memory from being freed and its address reused, which would cause false dedup hits.

Definition at line 159 of file SerializerBase.hpp.

The documentation for this class was generated from the following files:

/home/runner/work/DNDSR/DNDSR/src/DNDS/SerializerBase.hpp
/home/runner/work/DNDSR/DNDSR/src/DNDS/SerializerBase.cpp

Public Member Functions

Protected Member Functions

Protected Attributes

Detailed Description

Collective semantics (SerializerH5)

Zero-size reads

Constructor & Destructor Documentation

◆ ~SerializerBase()

Member Function Documentation

◆ CloseFile()

◆ CreatePath()

◆ dedupClear()

◆ dedupLookup()

◆ dedupRegister()

◆ GetCurrentPath()

◆ getMPI()

◆ GetMPIRank()

◆ GetMPISize()

◆ GoToPath()

◆ IsPerRank()

◆ ListCurrentPath()

◆ OpenFile()

◆ ReadIndex()

◆ ReadIndexVector()

◆ ReadInt()

◆ ReadReal()

◆ ReadRealVector()

◆ ReadRowsizeVector()

◆ ReadSharedIndexVector()

◆ ReadSharedRowsizeVector()

◆ ReadString()

◆ ReadUint8Array()

◆ WriteIndex()

◆ WriteIndexVector()

◆ WriteIndexVectorPerRank()

◆ WriteInt()

◆ WriteReal()

◆ WriteRealVector()

◆ WriteRowsizeVector()

◆ WriteSharedIndexVector()

◆ WriteSharedRowsizeVector()

◆ WriteString()

◆ WriteUint8Array()

Member Data Documentation

◆ pth_2_ssp

◆ ptr_2_pth