Record Format
Record Format Version
Each record in a stream has its own format version number, which is a uint32_t
value. However, because records belong to a single stream and each has a record type (Configuration, State, or Data), format version numbers are only meaningful within that stream and for that record type. You do not need to worry about format version collisions between streams and record types.
Before RecordFormat
was available, record format versioning was critical, because it was the only information about how the record's data was formatted. In the StreamPlayer
callbacks you received when reading a file, you were responsible for interpreting every byte of data and you also had to manually manage all data format changes. Since record data formats were not self-described within the file, each time you needed to add, remove, or change a field, you had to change the format version, and handle a growing number of format versions explicitly in your code. This was unmanageable. Moreover, it was not possible to write standard tools that could interpret that was stored in records, show the images they might contains, or any other metadata.
RecordFormat
and DataLayout
were designed to solve these challenges, and since, record format version changes are very rarely needed. RecordFormat
structures records as a succession of typed content blocks, embedding descriptions, including DataLayout
definitions, in the stream itself. VRS uses these embedded descriptions to interpret records, calculate content block boundaries using DataLayout Conventions, and send parsed content blocks to RecordFormatStreamPlayer
callbacks when reading a VRS file. With RecordFormat
and the DataLayout Conventions, it is now possible to write generic tools like vrsplayer, that can let you explore what's in a VRS file without any prior knowledge of the use case in which the file was recorded.
RecordFormat
Use RecordFormat
to describe records as a sequence of typed content blocks. This structure applies to configuration, state, and data records alike.
ContentBlock
The content block types are: image
, audio
, datalayout
, and custom
. VRS saves RecordFormat
definitions as a string that is generated and parsed for you, but which was designed to be very expressive and compact. Content block descriptions may contain additional information, specific to the content type. Here are some examples of single content block RecordFormat
definitions:
image
image/png
image/jpg
image/jxl
image/raw
image/raw/640x480/pixel=grb8
image/raw/640x480/pixel=grey8/stride=648
image/custom_codec/codec=my_experiment
image/video
image/video/codec=H.264
audio
audio/pcm
audio/pcm/uint24be/rate=32000/channels=1
datalayout
datalayout/size=48
custom
custom/format=my_own_payload_format
custom/size=160
image
and audio
content blocks are pretty much what you expect when you read their text description. datalayout
blocks contain structured metadata information. custom
content blocks are blocks of raw data, which format is known only to you, and which you are responsible for interpreting.
You can assemble as many content blocks as you like in a record, which might look like this:
datalayout+image/raw
datalayout+datalayout+audio/pcm
Again, these text descriptions are generated and parsed for you, so you don't need to worry about their syntax.
The RecordFormat
and DataLayout
descriptions of a stream's records are stored in the VRS tags of the stream. You will only see these text descriptions when you are using tools to dump a stream's VRS tags. VRS tags are associated with each stream for VRS internal usage, and are kept separate from the user stream tags.
In practice, the majority of the records used in VRS today use one of the following record formats:
datalayout
: for records containing a single metadata content block, which is typical of configuration records.datalayout+image/raw
: for records containing some image specific metadata and the raw pixel data of an image.datalayout+image/jpg
anddatalayout+image/video
: for records containing some image specific metadata and compressed image data.
Datalayout Content Blocks
Datalayout content blocks, commonly referred to as datalayouts, are DataLayout
objects that hold a collection of DataPieceXXX
objects, which are containers of POD values and strings. If you have never seen a DataLayout
definition, look at the MyDataLayout
definition in the DataLayout
Examples section below.
DataLayout
are typically struct
objects containing a series of DataPieceXXX
member variables, that each have their own type and text label. The supported DataPieceXXX
types are:
DataPieceValue
, a single value of POD type T
:
- Type:
template <class T> DataPieceValue<T>;
- Example:
DataPieceValue<int32_t> exposure{"exposure"};
DataPieceEnum
, a single value of enum ENUM_TYPE
with the underlying type POD_TYPE
:
- Type:
template <typename ENUM_TYPE, typename POD_TYPE> DataPieceEnum<ENUM_TYPE, POD_TYPE>
- Example:
DataPieceEnum<PixelFormat, uint32_t> pixelFormat{"pixel_format"};
DataPieceArray
, a fixed size array of values of POD type T
:
- Type:
template <class T> DataPieceArray<T>;
- Example:
DataPieceArray<float> calibration{"calibration", 25};
DataPieceVector
, a vector of values of type T
, which size may change for each record:
- Type:
template <class T> DataPieceVector<T>
- Example:
DataPieceVector<int8_t> udpPayload{"udp_payload"};
DataPieceStringMap
, the equivalent of std::map<std::string, T>
:
- Type:
template <class T> DataPieceStringMap<T>
- Example:
DataPieceStringMap<Point2Di> labelledPoints{"labelled_points"};
DataPieceString
, a std::string
value:
- Type:
DataPieceString
- Example:
DataPieceString message{"message"};
Template class T
can be any of these built-in POD types:
- Boolean (use
vrs::Bool
) - Signed or unsigned integer (8, 16, 32, or 64 bits)
- 32 bit float
- 64 bit double
Template class T
can also be any of these vector types (using float
, double
or int32_t
for coordinates):
- 2, 3, or 4D points
- 3 or 4D matrices
std::string
can be used with DataPieceVector<T>
and DataPieceStringMap<T>
, but cannot be used with the other template types.
Always use <cstdint>
definitions. Never use native platform dependent types like short
, int
, long
, or size_t
, because their actual size will vary depending on the architecture or the compiler configuration.
DataLayout
Format Resilience
DataLayout
objects are structs, so it is very simple to add, remove, and reorder DataPieceXXX
fields. But datalayouts definitions are very resilient to definition changes, so that even when making such changes, newer code can read older files, and older code can read newer files.
Datalayouts format resilience is possible, because each DataPieceXXX
object is identified by a unique combination of these elements:
DataPiece
type- Label
- Template class
T
, except forDataPieceString
This unique combination is critical to providing datalayout forward/backward compatibility, without worrying about the actual placement of the data. If you change the type or the label of a field, you will change its signature, and it won't be recognized in older files. The modified field will look like a new field, and the data from older files will no longer be accessible using the updated definition.
DataLayout
definitions do not support other types of containers or nested containers, because that would make it impossible to guarantee forward/backward compatibility. However, it is possible to use repeated and nested structs, using DataLayoutStruct
, as shown in the second example below.
In some situations, such as when you need to save space, it is desirable to store some fields only in some conditions. The OptionalDataPieces
template makes it easy to specify and control if a group of fields should be saved or not, but the choice must be made once for the whole file.
If you need more freedom, you can use a free form container such as JSON in a DataPieceString
field. If you have binary data, you can use a DataPieceVector<uint8_t>
field.
We recommend that you use lowercase snake_case as your naming convention for labels. This will limit problems if these names are used as keys in a Python dictionary, in particular when using pyvrs to create or read datalayouts.
DataLayout
Examples
- Example 1: standard case
- Example 2: nested definitions
- Example 3: optional definitions
Here is a sample DataLayout
definition:
struct MyDataLayout : public AutoDataLayout {
// Fixed size pieces: std::string is NOT supported as a template type.
DataPieceValue<double> exposureTime{"exposure_time"};
DataPieceValue<uint64_t> frameCounter{"frame_counter"};
DataPieceValue<float> cameraTemperature{"camera_temperature"};
DataPieceEnum<PixelFormat, uint32_t> pixelFormat{"pixel_format"};
DataPieceArray<Matrix3Dd> arrayOfMatrix3Dd{"matrices", 3}; // array size = 3
// Variable size pieces: std::string is supported as a template type.
DataPieceVector<Point3Df> vectorOfPoint3Df{"points"};
DataPieceVector<string> vectorOfString{"strings"};
DataPieceString description{"description"}; // Any string. Could be json.
DataPieceStringMap<Matrix4Dd> aStringMatrixMap{"some_string_to_matrix4d_map"};
DataPieceStringMap<string> aStringStringMap{"some_string_to_string_map"};
AutoDataLayoutEnd endLayout;
};
Notice that this struct must derive from AutoDataLayout
, and finish with an AutoDataLayoutEnd
field. This is required to make the DataLayout
magic happen. Under the hood, the DataPieceXXX
constructors will register themselves to the enclosing AutoDataLayout
. As we will generally only create a single DataLayout
instance, the overhead is minimal and does not matter. Also, notice that each field has a unique label.
This option is not commonly needed.
It is possible to define structs that can be nested in a DataLayout definition. For example:
struct Pose : public DataLayoutStruct {
DATA_LAYOUT_STRUCT(Pose) // repeat the name of the struct
DataPieceVector<vrs::Matrix4Dd> orientation{"orientation"};
DataPieceVector<vrs::Matrix3Dd> translation{"translation"};
};
struct MyDataLayout : public AutoDataLayout {
Pose leftHand{"left_hand"};
Pose rightHand{"right_hand"};
AutoDataLayoutEnd endLayout;
};
The name of each field in the DataLayoutStruct
is prepended by the name of the DataLayoutStruct
itself, with a ‘/
’ added to make it look like a path. This also makes it unique at the datalayout level.
Effectively, the declaration above creates the same DataPiece
fields and the same datalayout definition as the datalayout definition below, which requires different member variable names to avoid conflicts at the struct level:
struct MyDataLayout: public AutoDataLayout {
DataPieceVector<vrs::Matrix4Dd> leftHandOrientation{"left_hand/orientation"};
DataPieceVector<vrs::Matrix3Dd> leftHandTranslation{"left_hand/translation"};
DataPieceVector<vrs::Matrix4Dd> rightHandOrientation{"right_hand/orientation"};
DataPieceVector<vrs::Matrix3Dd> rightHandTranslation{"right_hand/translation"};
AutoDataLayoutEnd endLayout;
};
It is possible to nest a DataLayoutStruct
within other DataLayoutStruct
definitions as often as makes sense. The resulting DataPiece
fields will have labels similarly constructed, with deeper nesting. However, it is not possible to use DataLayoutStruct
definitions in template containers.
This option is only very rarely needed.
You can define fields that are used only when some recording conditions are met, or with some devices. This helps to save space, and makes the records less ambiguous, since they will only show these fields if they were actually used while recording.
For example:
/// Sample sensor not always available on all devices
struct TemperatureData {
DataPieceValue<float> cameraTemperature{"camera_temperature"};
};
struct MyDataLayout : public AutoDataLayout {
MyDataLayout(bool allocateOptionalFields = false)
: optionalFields(allocateOptionalFields) {}
DataPieceValue<double> exposureTime{"exposure_time"};
DataPieceValue<uint64_t> frameCounter{"frame_counter"};
const OptionalDataPieces<TemperatureData> optionalTemperature;
AutoDataLayoutEnd endLayout;
};
When recording a file, you need to decide upfront, at runtime, whether the optional fields are needed for this recording, and then select the appropriate constructor. This is because the optional fields must be allocated during the datalayout construction.
When reading a file, you can try to use the appropriate constructor, or you can always include the optional fields and test if data is present by checking the isAvailable()
method for each optional field.
Image, Audio, and Custom Content Blocks
Image, audio, and custom content blocks directly contain their payload, and no additional metadata. In some cases, such as for image/jpg
or image/png
data, no other information is needed to interpret the data. In other cases, such with images/raw
images, which are raw pixel buffers, image dimensions, pixel format and possibly stride information are required to know how to interpret the image content block. If that information never changes, it may provided directly in the RecordFormat
definition, otherwise, it might need to be provided in a configuration record, or in the data records themselves, using what we call the Datalayout Conventions.
- Image Content Block Examples
- Audio Content Block Examples
- Custom Content Block Examples
ContentBlock(ContentType::IMAGE); // Image content block, without any detail
ContentBlock(ContentType::JPG); // A jpeg image
ContentBlock(ContentType::JPG, 640, 480); // A 640x480 jpeg image
ContentBlock(ContentType::RAW); // A raw pixel buffer image
ContentBlock(PixelFormat::GREY8, 640, 480); // A raw pixel buffer image, 640x480 large, with 8 bit greyscale pixels.
Please refer to the Image Support section for more details on how to manage image content blocks.
ContentBlock(ContentType::AUDIO); // Audio content block, without any detail
ContentBlock(AudioFormat::PCM); // PCM audio data
ContentBlock(AudioSampleFormat::S16_LE, 2, 48000); // PCM audio data, int16 little endian samples, 2 channels, 48 kHz
Audio blocks are analog to image blocks, and are handled the same way.
ContentBlock(ContentType::CUSTOM); // No details at all
ContentBlock(ContentType::CUSTOM, 256); // 256 bytes custom content block
CustomContentBlock("my_thing", 48); // 48 bytes custom content block in the format "my_thing"
If they are not the last content block in the record, custom content blocks may need to have their size provided using the Datalayout conventions.
Registering your RecordFormat
and DataLayout
Definitions
When you create a Recordable
object to record a stream, you need to register the RecordFormat
for its records. For example:
// Assuming your recordable has a member variable declared like so:
MyDataLayout config_;
// in your Recordable's constructor, call:
addRecordFormat(
Record::Type::CONFIGURATION, // record types are defined separately
kConfigurationRecordFormatVersion, // only change when the RecordFormat changes
config_.getContentBlock(), // RecordFormat definition: a single datalayout block
{&config_}); // DataLayout definition for the first block
Here is an example of a record that contains a datalayout block, followed by an image block (“datalayout+image/raw
”):
// Assuming your recordable has a member variable declared like so:
MyDataLayoutForDataRecords data_;
// in your Recordable's constructor, call:
addRecordFormat(
Record::Type::DATA, // record types are defined separately
kDataRecordFormatVersion, // only change when RecordFormat changes
data_.getContentBlock() + ContentBlock(ImageFormat::RAW), // RecordFormat definition
{&data_}); // DataLayout definition for the first block, nothing for the image block
Each record has a record format version number. Each RecordFormat
, its record format version number, and its DataLayout
definitions are tied to a particular stream. Therefore, it is possible to have records using different RecordFormat
/DataLayout
definitions within a particular stream, by using different record format version numbers.
DataLayout
definitions fully describe what is stored in a datalayout content block. So, you can freely change DataLayout
definitions without changing the record format version.
Reading Records
To read records described using RecordFormat
conventions, attach a RecordFormatStreamPlayer
to your RecordFileReader
. Then, hook code to whichever of these virtual methods is appropriate for your records:
onDataLayoutRead()
onImageRead()
onAudioRead()
onCustomBlockRead()
You will get one callback per content block, until one of the callbacks returns false
, signaling that the end of the record should not be decoded.
Reading a Datalayout
When reading a datalayout
content block, you will get an onDataLayoutRead
callback in your RecordFormatStreamPlayer
object, with the datalayout already loaded. In the onDataLayoutRead
callback, you will want to handle records differently, depending on their record type.
For each record type, you will have a specific DataLayout
definition, describing the latest version of the datalayout you are using. But you cannot know if that definition matches what was read, since the file could be using an older or newer version of the datalayout definition. Use the getExpectedLayout<MyDataLayout>
API to get a DataLayout
instance of the type your code is looking for. You can then access each of its fields safely, with the caveat that each field may or may not find actual data in the datalayout that was read from disk.
Each data field is mapped according to its data type and label only. So, you do not need to worry whether fields have been added, removed, or moved. Mapping is cached per file/stream/type. So, after the first record is mapped, mapping is extremely cheap, and fields are read in constant time, no matter how complicated the datalayouts are.
When debugging, use DataLayout::printLayout(std::cout)
to print the incoming datalayout. This will show the field names, their type, and their value, as they are in the record read.
class MyCameraStreamPlayer : public RecordFormatStreamPlayer {
bool onDataLayoutRead(const CurrentRecord& record, size_t blockIndex, DataLayout& data) override {
switch (record.recordType) {
case Record::Type::CONFIGURATION: {
MyCameraConfigRecordDataLayout& myConfig =
getExpectedLayout<MyCameraConfigRecordDataLayout>(data, blockIndex);
// use the data...
myConfig.cameraRole.get(); // access the data...
} break;
case Record::Type::DATA: {
// Here are the fields written & expected in the latest version
MyCameraDataRecordDataLayout& myData =
getExpectedLayout<MyCameraDataRecordDataLayout>(data, blockIndex);
// use the data...
myData.cameraTemperature.get();
// Rare case: access field that were removed or renamed
// e.g., frame_counter's type was changed: fetch the old version if necessary
uint64_t frameCounter = 0;
if (myData.frameCounter.isAvailable()) {
frameCounter = myData.frameCounter.get();
} else {
// MyCameraLegacyFields contains removed fields definitions
MyCameraLegacyFields& legacyData =
getLegacyLayout<MyCameraLegacyFields>(data, blockIndex);
frameCounter = myConversionLogic(legacyData.frameCounter.get());
}
} break;
default:
assert(false); // should not happen, but you want to know if it does!
break;
}
return true; // read next content blocks, if any
}
Datalayout Conventions
Datalayout Conventions are a set of names and types that VRS uses to find missing RecordFormat
specifications, such as the resolution and pixel format, if they are missing in the definition of an “image/raw”
content block. Datalayout Conventions can also be used to specify the size of a content block when it is ambiguous. Refer to the source header <vrs/DataLayoutConventions.h>
to see the actual Datalayout Conventions.
In the examples above, you can determine the size of the datalayout blocks by looking at the actual DataLayout
definition. However, that only works if only fixed type pieces are used. When only fixed type pieces are used, the datalayout size is constant no matter what the content is. Look again at the definition of MyDataLayout
above to see the difference between fixed size pieces and variable size pieces.
When only fixed size pieces are used, the getContentBlock()
API generates "datalayout/size=XXX"
, with XXX
being the number of bytes. If the datalayout contains any variable size pieces, the size of the datalayout can change from record to record, and the getContentBlock()
API will return "datalayout"
.
If any variable size pieces are present, the datalayout will include an index, which has a fixed size. The index's size depends only on the number of variable size pieces declared, not on their actual values. This index makes it possible for VRS to determine the overall size of the DataLayout
in two successive reads. The first read includes the data for all the fixed size pieces and the index for the variable size pieces. The added sizes found in the variable size index tells the total size of the variable size pieces, which VRS can now read with a second file read call. Therefore, VRS can always read a DataLayout
block, because we can always determine its actual size.
In the second RecordFormat
example above, we have a datalayout block followed by an image block (“datalayout+image/raw”
). Since the image block is the last content block of the record, and VRS knows the overall size of the record, and how to figure out the size of the datalayout, we can see that all the remaining bytes must belong to the “image/raw”
block. However, this is not sufficient to interpret the image pixel data. This is when we need the Datalayout Conventions.
When working with a device such as a camera, typically, during the hardware initialization/setup, before the data collection begins, the software stack will configure the camera to function in a particular mode, which includes parameters such as resolution, color mode, exposure mode, and frame rate. These parameters will never change unless the configuration of the camera is changed, which is extremely rare in practice. These parameters all belong to a configuration record and can easily be saved in a datalayout block.
In a more advanced system, a camera’s resolution and color mode may change for each frame, as when driven by a computer vision algorithm or some other heuristic. When you save only a sub-region of a whole image (the way Portal does when it tracks a target and crops the image received from the sensor), the crop size of the image might change in every frame. In such cases, the image parameters should not be placed in a configuration record. Those parameters should be specified in the datalayout block preceding the image block.
VRS uses the following heuristics:
-
Search each datalayout block before the ambiguous block, in the same record, in reverse content block order. If the
RecordFormat
is