The DataLayout class describes the data stored inside a DataLayoutContentBlock. More...

#include <DataLayout.h>

Inheritance diagram for vrs::DataLayout:

Classes
class	IndexEntry
	Describes where the data of a variable size DataPiece is in the varData_ buffer. More...

Public Member Functions
DataLayout &	operator= (const DataLayout &)=delete

	DataLayout (const DataLayout &)=delete

virtual	~DataLayout ()

ContentBlock	getContentBlock () const

vector< int8_t > &	getFixedData ()

vector< int8_t > &	getVarData ()

size_t	getFixedDataSizeNeeded () const

size_t	getVarDataSizeFromIndex () const

size_t	getVarDataSizeNeeded () const

void	collectVariableDataAndUpdateIndex ()

void	collectVariableDataAndUpdateIndex (void *destination)

void	getRawData (vector< int8_t > &outRawData) const

void	stageCurrentValues ()

bool	copyClonedDataPieceValues (const DataLayout &originalLayout)

bool	mapLayout (DataLayout &targetLayout)

bool	isMapped () const

bool	hasAllRequiredPieces () const

void	requireAllPieces ()
	Mark all the fields of the layout as required.

void	printLayout (ostream &out, const string &indent="") const

void	printLayoutCompact (ostream &out, const string &indent="") const

string	asJson (JsonFormatProfile profile) const

string	asJson (const JsonFormatProfileSpec &profile=JsonFormatProfileSpec()) const

string	getListOfPiecesSpec () const
	Get a text list of fields, types & names, one per line. Useful for tests.

bool	isSame (const DataLayout &otherLayout) const

template<class T >
const DataPieceValue< T > *	findDataPieceValue (const string &label) const

template<class T >
DataPieceValue< T > *	findDataPieceValue (const string &label)

template<class T >
const DataPieceArray< T > *	findDataPieceArray (const string &label, size_t arraySize) const

template<class T >
DataPieceArray< T > *	findDataPieceArray (const string &label, size_t arraySize)

template<class T >
const DataPieceVector< T > *	findDataPieceVector (const string &label) const

template<class T >
DataPieceVector< T > *	findDataPieceVector (const string &label)

template<class T >
const DataPieceStringMap< T > *	findDataPieceStringMap (const string &label) const

template<class T >
DataPieceStringMap< T > *	findDataPieceStringMap (const string &label)

const DataPieceString *	findDataPieceString (const string &label) const

DataPieceString *	findDataPieceString (const string &label)

void	forEachDataPiece (const std::function< void(const DataPiece *)> &, DataPieceType type=DataPieceType::Undefined) const

void	forEachDataPiece (const std::function< void(DataPiece *)> &, DataPieceType type=DataPieceType::Undefined)
	Same as above, but as a non-const version.

bool	isVarDataIndexValid () const

size_t	getDeclaredFixedDataPiecesCount () const

size_t	getDeclaredVarDataPiecesCount () const

size_t	getAvailableFixedDataPiecesCount () const

size_t	getAvailableVarDataPiecesCount () const

Static Public Member Functions
static unique_ptr< DataLayout >	makeFromJson (const string &json)

Static Public Attributes
static const size_t	kNotFound = numeric_limits<size_t>::max()
	Special OffsetAndLength offset value marking that a piece of data isn't available.

static const size_t	kVariableSize = numeric_limits<size_t>::max() - 1
	Special value used for a DataPiece size, telling that that DataPiece has a variable size.

Protected Member Functions
template<class T >
T *	getFixedData (size_t offset, size_t size)

const IndexEntry *	getVarSizeIndex () const

IndexEntry *	getVarSizeIndex ()

template<class T >
T *	getVarData (size_t varPieceIndex, size_t &outCount)

void	initLayout ()

void	serialize (JsonWrapper &rj, const JsonFormatProfileSpec &profile) const

Static Protected Member Functions
static bool	mapPieces (const vector< DataPiece * > &searchPieces, const vector< DataPiece * > &givenPieces)

Protected Attributes
vector< DataPiece * >	fixedSizePieces_
	Ordered fixed-size DataPieces.

vector< DataPiece * >	varSizePieces_
	Ordered variable-size DataPieces.

vector< int8_t >	fixedData_
	Buffer to hold fixed-size pieces, and the index of var size pieces (if any).

size_t	fixedDataSizeNeeded_ {}
	Byte count for all the fixed size pieces + var size index.

vector< int8_t >	varData_
	Buffer holding variable-size pieces, after they've been collected, or read from disk.

bool	hasAllRequiredPieces_ {true}
	Tells all the required pieces have been mapped successfully.

DataLayout *	mappedDataLayout_ {}
	DataLayout this layout has been mapped to, if any.

Friends
class	internal::DataLayouter

class	DataPiece

template<class T >
class	DataPieceValue

template<class T >
class	DataPieceArray

template<class T >
class	DataPieceVector

template<class T >
class	DataPieceStringMap

class	DataPieceString

Detailed Description

The DataLayout class describes the data stored inside a DataLayoutContentBlock.

A DataLayout object is usually constructed using AutoDataLayout and AutoDataLayoutEnd helpers. This method allows the easy & safe definition of a DataLayout in the form of a struct. The member variables of an AutoDataLayout allow the easy access of the individual pieces of content that make up a DataLayoutContentBlock, both for writing & reading.

Note that though DataLayout objects may look like structs, constructing them is relatively expensive, and creating AutoDataLayout objects involve a synchronisation lock. Therefore, avoid creating & destroying using short lived AutoDataLayout stack variables. Instead, prefer allocating the DataLayout objects you will need frequently as member variables of other classes that are long lived.

A DataLayout object can generate a json representation of itself, that is stored by VRS as an internal tag of a VRS record stream. This json data will be used when reading the VRS file to interpret the data of the corresponding DataLayoutContentBlock. The instantiation of a DataLayout from a json message is meant for exclusive use by VRS itself, when reading a file. VRS is very careful to only do this json expensive deserialization once per type.

The key feature of DataLayout is the separation of the layout description saved once per stream, from the actual payload in each record, which can be minimal, containing only binary data. The DataLayout sections of records are not stored in json format.

Most methods of this class are meant for exclusive use by VRS itself, even if they are public.

Anatomy of a DataLayout

A DataLayout is an ordered collection of individual pieces of data, all deriving from the DataPiece abstract class. Classes deriving from DataPiece fall in two categories: Fixed-size DataPiece objects, and variable-size DataPiece objects.

Fixed-size DataPiece objects, use the same number of bytes, no matter the actual value:

single values of the supported POD types.
fixed size arrays of the supported POD types.

DataLayout supports the following POD types:

native C/C++ types, like int8, int16, int32, int64, signed or unsigned, float & double.
Bool, which is a replacement for bool, because vector<bool> is not an STL container, see https://stackoverflow.com/questions/17794569/why-is-vectorbool-not-a-stl-container
2, 3 & 4 D points, 3 & 4 D matrices, of types int32, float & double, for instance: Point2Df: a 2 dimension point using float. Matrix4Dd: a 4 dimension matrix using double.

DataPieceValue<T> and DataPieceArray<T> are the two fixed size DataPiece template classes.

Note: You can not use DataLayout template types with your own POD types, because:

DataLayout needs a factory able to instantiate them, even when your code isn't around,
it would defeat the purpose of handling format changes, as any change made to your POD type would have to be versioned & managed by hand.

Variable-size DataPiece objects use a number of bytes that depends on their actual value:

DataPieceVector<T>: a vector of any of the supported POD types, or std::string.
DataPieceStringMap<T>: a string map of any of the supported POD types, or std::string.
DataPieceString: an ASCII or utf8 text string.

Internal representation (what you don't need to know unless you are working on DataLayout)

Because there is a known fixed number of fixed-size DataPiece objects, each of known size, we can pre-allocate a buffer to hold the data values of all those DataPiece objects. Fixed-size DataPiece objects reference their data using an offset and the size of their data. All read & write value operations on fixed-size DataPiece objects use that buffer, so that we can directly read from and write to disk that part of a DataLayout's data. That buffer is the first section of the fixedData_ byte vector.

For variable-size DataPiece objects, because the DataPiece data has an unforeseeable size, data reads & writes are not handled the same as for fixed-size DataPiece objects. Variable-size DataPiece objects don't have setter methods, but "stage" methods, that set the value in a field of the DataPiece object itself. The "stage" methods also allows to read the staged values. Once the value of all the variable-size DataPiece objects are staged, the staged values can be collected in a buffer, the varData_ byte vector. To interpret a variable size buffer read from disk, we need an index, to know where each piece starts, and their size. That's the variable size index. The size of that index is known ahead of time, since there is one index entry per variable-size DataPiece, and we know up-front how many we have. The index is stored in the fixedData_ vector, after the data of fixed-size DataPiece values. The values of the index are set when the variable size data is collected. When reading the value of a variable-size DataPiece, the varData_ buffer is used, if present.

To write a DataLayout section, the 3 following operations are needed:

collect the variable-size DataPiece values. That operations allocates the varData_ buffer, and updates the var data index in the fixedData_ buffer.
write the fixedData_ buffer.
write the varData_ buffer.

Therefore, to read & write the data of a DataLayout from & to disk, there is no encoding of the data, unlike with json, which guaranties the bit accuracy of your data. That can be critically important when using float & double numbers.

Of course, all of this is the responsibility of VRS itself.

Constructor & Destructor Documentation

◆ ~DataLayout()

vrs::DataLayout::~DataLayout ( )

virtualdefault

DataLayout has no virtual method, but it is used in containers, and some of its derived classes have important clean-up work to do in their destructor. Therefore, DataLayout requires a virtual destructor.

Member Function Documentation

◆ asJson() [1/2]

string vrs::DataLayout::asJson ( const JsonFormatProfileSpec & profile = JsonFormatProfileSpec() ) const

Generate json representation of this layout, using a profile telling which fields to include, and other presentation options.

Parameters

profile Formatting profile.

Returns: json text string.

◆ asJson() [2/2]

string vrs::DataLayout::asJson ( JsonFormatProfile profile ) const

Generate json representation of this layout, using a profile telling which fields to include, and other presentation options.

Parameters

profile Formatting profile.

Returns: json text string.

◆ collectVariableDataAndUpdateIndex() [1/2]

void vrs::DataLayout::collectVariableDataAndUpdateIndex ( )

Collect all the variable-size fields in the varData_ buffer.

◆ collectVariableDataAndUpdateIndex() [2/2]

void vrs::DataLayout::collectVariableDataAndUpdateIndex ( void * destination )

Collect variable-size data to the buffer specified.

Parameters

destination Pointer to the buffer. The buffer must be at least getVarDataSizeNeeded() large.

◆ copyClonedDataPieceValues()

bool vrs::DataLayout::copyClonedDataPieceValues ( const DataLayout & originalLayout )

When a layout was cloned from another layout - this must be true -, this method will copy or stage all the data piece values from the original layout, so edits can be made and a new record generated with the DataLayout, using the values from originalLayout when possible.

Parameters

originalLayout cloned datalayout from which DataPiece values need to be copied.

Returns: True if the copy was successful, false if the layout isn't a clone of originalLayout.

◆ findDataPieceArray()

template<typename T >

const DataPieceArray< T > * vrs::DataLayout::findDataPieceArray	(	const string &	label,
		size_t	arraySize
	)		const

Find a field of type DataPieceArray<T> by name.

Parameters

label Text name of the field (not the variable name).

Returns: Pointer to the DataPiece found, or nullptr.

◆ findDataPieceString()

const DataPieceString * vrs::DataLayout::findDataPieceString ( const string & label ) const

Find a field of type DataPieceString by name.

Parameters

label Text name of the field (not the variable name).

Returns: Pointer to the DataPiece found, or nullptr.

◆ findDataPieceStringMap()

template<typename T >

template const DataPieceStringMap< string > * vrs::DataLayout::findDataPieceStringMap< string > ( const string & label ) const

Find a field of type DataPieceStringMap<T> by name.

Parameters

label Text name of the field (not the variable name).

Returns: Pointer to the DataPiece found, or nullptr.

◆ findDataPieceValue()

template<typename T >

const DataPieceValue< T > * vrs::DataLayout::findDataPieceValue ( const string & label ) const

Find a field of type DataPieceValue<T> by name.

Parameters

label Text name of the field (not the variable name).

Returns: Pointer to the DataPiece found, or nullptr.

◆ findDataPieceVector()

template<typename T >

template const DataPieceVector< string > * vrs::DataLayout::findDataPieceVector< string > ( const string & label ) const

Find a field of type DataPieceVector<T> by name.

Parameters

label Text name of the field (not the variable name).

Returns: Pointer to the DataPiece found, or nullptr.

◆ forEachDataPiece()

void vrs::DataLayout::forEachDataPiece	(	const std::function< void(const DataPiece *)> &	,
		DataPieceType	type = `DataPieceType::Undefined`
	)		const

Iterate over the different data piece elements of a DataLayout.

Parameters

callback	a function to call for each element found.
type	filter to select only an element type, of UNDEFINED for no filtering.

◆ getAvailableFixedDataPiecesCount()

size_t vrs::DataLayout::getAvailableFixedDataPiecesCount ( ) const

For debugging: get the number of fixed size data pieces available. When a datalayout is mapped on another one (when reading from disk), some pieces might not be available, because the datalayout read from disk might not have them.

Returns: The number of fixed size data pieces available in this datalayout.

◆ getAvailableVarDataPiecesCount()

size_t vrs::DataLayout::getAvailableVarDataPiecesCount ( ) const

For debugging: get the number of variable size data pieces available. When a datalayout is mapped on another one (when reading from disk), some pieces might not be available, because the datalayout read from disk might not have them.

Returns: The number of variable size data pieces available in this datalayout.

◆ getContentBlock()

ContentBlock vrs::DataLayout::getContentBlock ( ) const

Returns: ContentBlock object to build a RecordFormat definition.

◆ getDeclaredFixedDataPiecesCount()

size_t vrs::DataLayout::getDeclaredFixedDataPiecesCount ( ) const

inline

For debugging: get the number of fixed size data pieces declared.

Returns: The number of fixed size data pieces declared in this datalayout.

◆ getDeclaredVarDataPiecesCount()

size_t vrs::DataLayout::getDeclaredVarDataPiecesCount ( ) const

inline

For debugging: get the number of variable size data pieces declared.

Returns: The number of variable size data pieces declared in this datalayout.

◆ getFixedData() [1/2]

vector< int8_t > & vrs::DataLayout::getFixedData ( )

inline

To access the buffer for fixed-size DataPieces. No need for a const version.

◆ getFixedData() [2/2]

template<class T >

T * vrs::DataLayout::getFixedData	(	size_t	offset,
		size_t	size
	)

inlineprotected

Get pointer to a section of raw memory, by offset & size.

Parameters

offset	Offset of the field in the fixed-size fields buffer.
size	Byte count size of the field.

Returns: Pointer, or nullptr if the whole field won't fit in the buffer.

◆ getFixedDataSizeNeeded()

size_t vrs::DataLayout::getFixedDataSizeNeeded ( ) const

inline

Size required to fit all fixed size data.

◆ getRawData()

void vrs::DataLayout::getRawData ( vector< int8_t > & outRawData ) const

Get this layout's raw data. Assumes no data needs to be collected (nothing was "staged"). In particular, can be used in RecordFormatStreamPlayer's callback to copy the layout verbatim.

◆ getVarData() [1/2]

vector< int8_t > & vrs::DataLayout::getVarData ( )

inline

To access the buffer for variable-size DataPieces. No need for a const version.

◆ getVarData() [2/2]

template<class T >

T * vrs::DataLayout::getVarData	(	size_t	varPieceIndex,
		size_t &	outCount
	)

inlineprotected

Get pointer & size of a variable-size field's data.

Parameters

varPieceIndex	Index of the variable-size piece.
outCount	Reference to set to the size in bytes of the data.

Returns: pointer to the variable-size data found, or nullptr.

◆ getVarDataSizeFromIndex()

size_t vrs::DataLayout::getVarDataSizeFromIndex ( ) const

Retrieve size of the var size pieces from the index.

Only really meaningful after variable-size pieces:

have been collected, or
the layout's data has been read from disk, or
after the layout has been mapped onto another layout.
Returns
Byte count.

◆ getVarDataSizeNeeded()

size_t vrs::DataLayout::getVarDataSizeNeeded ( ) const

Returns: Size to fit the variable-size data written to its variable-size DataPieces fields. Won't match varData_'s size, unless collectVariableDataAndUpdateIndex() was called.

◆ getVarSizeIndex() [1/2]

DataLayout::IndexEntry * vrs::DataLayout::getVarSizeIndex ( )

protected

Get the var size index.

Returns: The var-size index, non-const version.

◆ getVarSizeIndex() [2/2]

const DataLayout::IndexEntry * vrs::DataLayout::getVarSizeIndex ( ) const

protected

Get the var size index.

Returns: The var-size index, const version.

◆ hasAllRequiredPieces()

bool vrs::DataLayout::hasAllRequiredPieces ( ) const

inline

Returns: True if the layout is mapped to another layout, and all the fields marked required have been successfully mapped onto a field of the target layout. Also returns True if the layout isn't mapped to another layout but has been initialized successfully by dataLayoutEnd(), initLayout(), and, for subclasses of AutoDataLayout, by the customary extra AutoDataLayoutEnd field added to the end of every AutoDataLayout.

◆ initLayout()

void vrs::DataLayout::initLayout ( )

protected

After construction of a DataLayout, initializes/resets buffers to hold the DataPiece objects that have been registered for this DataLayout. This is generally, done automatically by VRS, but this might be useful for client code to manage a DataLayout creation manually. In common practice, this is probably only needed by the VRS code itself.

◆ isMapped()

bool vrs::DataLayout::isMapped ( ) const

inline

Returns: True if the layout is mapped to another layout, in which case read values will come from the fields of the mapped layout.

◆ isSame()

bool vrs::DataLayout::isSame ( const DataLayout & otherLayout ) const

Compare two layouts, and tells if all the pieces are in the same order, with the same properties (name, type, tags, etc). Does not compare actual values!

Parameters

otherLayout Layout to compare to.

Returns: True if the layouts are equivalent.

◆ isVarDataIndexValid()

bool vrs::DataLayout::isVarDataIndexValid ( ) const

For debugging: validate that the index for the variable size data looks valid. If you think you need this in production code, please contact the VRS team...

Returns: True if the index for the variable size data looks valid.

◆ makeFromJson()

unique_ptr< DataLayout > vrs::DataLayout::makeFromJson ( const string & json )

static

Create a DataLayout from a json description generated with asJson().

Parameters

json	json datalayout description.

Returns: A DataLayout, or nullptr in case of error.

◆ mapLayout()

bool vrs::DataLayout::mapLayout ( DataLayout & targetLayout )

Map the data pieces of this layout to that of another layout, field by field. Note that this call is fairly expensive, and it is normally not necessary to call directly. In particular, in the callback RecordFormatStreamPlayer::onDataLayoutRead(), calling mapLayout() directly can usually be avoided by using the methods RecordFormatStreamPlayer::getExpectedLayout() and RecordFormatStreamPlayer::getLegacyLayout(), which will cache the mapping for you.

Parameters

targetLayout Other DataLayout to map each DataPiece to. This is usually the layout that contains a record's actual data.

Returns: True if all required DataPieces were found. When returning false, all possible matches are still made, and a false return value doesn't mean that the mapping "failed". It only means that some (or all) the DataPieces that have been explicitly marked as required have not been found in the target layout. This result value is cached and returned by hasAllRequiredPieces(). Again, DataPieces are NOT marked required by default, so you must call DataPiece::setRequired(true) for each DataPiece you require, or call DataLayout::requireAllPieces() to require them all in one shot. A good place to do this is probably your DataLayout's constructor itself.

◆ printLayout()

void vrs::DataLayout::printLayout	(	ostream &	out,
		const string &	indent = `""`
	)		const

Print the fields of this DataLayout, showing all known details & values. If the layout is mapped, the values are that of the mapped fields. If a field of a mapped layout wasn't successfully mapped, no value is shown.

Parameters

out	Ostream to write to.
indent	Prefix to each output line, for nested presentation.

◆ printLayoutCompact()

void vrs::DataLayout::printLayoutCompact	(	ostream &	out,
		const string &	indent = `""`
	)		const

Print the values of the fields of this datalayout, in a compact form.

Parameters

out	Ostream to write to.
indent	Prefix to each output line, for nested presentation.

◆ serialize()

void vrs::DataLayout::serialize	(	JsonWrapper &	rj,
		const JsonFormatProfileSpec &	profile
	)		const

protected

Export the DataLayout as json, using a specific profile.

Parameters

jsonWrapper	Wrapper around a json type (to isolate any 3rd party library dependency).
profile	Profile describing what information needs to be exported as json.

◆ stageCurrentValues()

void vrs::DataLayout::stageCurrentValues ( )

Take the values of the variable size fields and stage them. This is basically the opposite of collectVariableDataAndUpdateIndex(). This is useful if you have read a DataLayout from disk, and want to edit some fields to write an edited version of the DataLayout.

The documentation for this class was generated from the following files:

vrs/DataLayout.h
vrs/DataLayout.cpp

Classes

Public Member Functions

Static Public Member Functions

Static Public Attributes

Protected Member Functions

Static Protected Member Functions

Protected Attributes

Friends

Detailed Description

Anatomy of a DataLayout

Constructor & Destructor Documentation

◆ ~DataLayout()

Member Function Documentation

◆ asJson() [1/2]

◆ asJson() [2/2]

◆ collectVariableDataAndUpdateIndex() [1/2]

◆ collectVariableDataAndUpdateIndex() [2/2]

◆ copyClonedDataPieceValues()

◆ findDataPieceArray()

◆ findDataPieceString()

◆ findDataPieceStringMap()

◆ findDataPieceValue()

◆ findDataPieceVector()

◆ forEachDataPiece()

◆ getAvailableFixedDataPiecesCount()

◆ getAvailableVarDataPiecesCount()

◆ getContentBlock()

◆ getDeclaredFixedDataPiecesCount()

◆ getDeclaredVarDataPiecesCount()

◆ getFixedData() [1/2]

◆ getFixedData() [2/2]

◆ getFixedDataSizeNeeded()

◆ getRawData()

◆ getVarData() [1/2]

◆ getVarData() [2/2]

◆ getVarDataSizeFromIndex()

◆ getVarDataSizeNeeded()

◆ getVarSizeIndex() [1/2]

◆ getVarSizeIndex() [2/2]

◆ hasAllRequiredPieces()

◆ initLayout()

◆ isMapped()

◆ isSame()

◆ isVarDataIndexValid()

◆ makeFromJson()

◆ mapLayout()

◆ printLayout()

◆ printLayoutCompact()

◆ serialize()

◆ stageCurrentValues()