Class to abstract VRS file system operations, to enable support for alternate storage methods, in particular network/cloud storage implementations. More...

#include <FileHandler.h>

Inheritance diagram for vrs::FileHandler:

Classes
struct	CacheStats
	Stats for cache. More...

Public Types
using	CacheStatsCallbackFunction = std::function< void(const CacheStats &stats)>

Public Member Functions
virtual unique_ptr< FileHandler >	makeNew () const =0

virtual const string &	getFileHandlerName () const =0

virtual const string &	getWriteFileHandlerName () const

virtual int	open (const string &filePath)

virtual int	openSpec (const FileSpec &fileSpec)=0

int	delegateOpen (const FileSpec &fileSpec, unique_ptr< FileHandler > &outNewDelegate) override

virtual bool	isOpened () const =0

virtual int64_t	getTotalSize () const =0

virtual int	close ()=0

virtual int	skipForward (int64_t offset)=0

virtual int	setPos (int64_t offset)=0

virtual bool	isAvailableOrPrefetch (MAYBE_UNUSED size_t length)

virtual int	read (void *buffer, size_t length)=0

template<typename T , std::enable_if_t< std::is_trivially_copyable< T >::value, int > = 0>
int	read (T &object)
	Helper to read trivially copyable objects, in a chunk aware way.

virtual size_t	getLastRWSize () const =0

virtual bool	isReadOnly () const

virtual vector< std::pair< string, int64_t > >	getFileChunks () const =0

virtual void	forgetFurtherChunks (int64_t maxSize)=0
	Call this method to forget any chunk beyond this file size.

virtual int	getLastError () const =0

virtual bool	isEof () const =0

virtual int64_t	getPos () const =0

virtual int64_t	getChunkPos () const =0

virtual int	getChunkRange (int64_t &outChunkOffset, int64_t &outChunkSize) const =0

virtual bool	setCachingStrategy (CachingStrategy)

virtual CachingStrategy	getCachingStrategy () const

virtual bool	prefetchReadSequence (MAYBE_UNUSED const vector< std::pair< size_t, size_t > > &sequence, MAYBE_UNUSED bool clearSequence=true)

virtual bool	setStatsCallback (const CacheStatsCallbackFunction &)

virtual bool	purgeCache ()

bool	isFileHandlerMatch (const FileSpec &fileSpec) const

virtual bool	isRemoteFileSystem () const =0

virtual bool	showProgress () const
	Tell if the file handler is probably slow, and extra progress information might be useful.

Public Member Functions inherited from vrs::FileDelegator
virtual int	parseUri (FileSpec &inOutFileSpec, MAYBE_UNUSED size_t colonIndex) const

Static Public Member Functions
static unique_ptr< FileHandler >	makeOpen (const string &filePath)
	Open a file in read-only mode. Returns an open file handler, or nullptr on error.

static unique_ptr< FileHandler >	makeOpen (const FileSpec &fileSpec)

Detailed Description

Class to abstract VRS file system operations, to enable support for alternate storage methods, in particular network/cloud storage implementations.

For simplicity, in this documentation, we will references "files", but they might be one or more data blobs on a network storage.

VRS file users probably only need to use RecordFileReader & RecordFileWriter, but they have the option to use FileHandler directly to access files stored on remote file systems, same as VRS. Use FileHandlerFactory::delegateOpen() to find the proper FileHandler implementation and open a file. FileHandler only exposes read operations, because it's the most implementation, while WriteFileHandler extends FileHandler for write operations. Both are abstract classes.

'int' return values are status codes: 0 means success, while other values are error codes, which can always be converted to a human readable string using vrs::errorCodeToMessage(code). File sizes and offset are specified using int64_t, which is equivalent to the POSIX behavior. Byte counts use size_t.

Member Function Documentation

◆ close()

virtual int vrs::FileHandler::close ( )

pure virtual

Close the file & free all the held resources, even if an error occurs.

Returns: A status code for first error while closing, or 0, meaning success.

Implemented in vrs::DiskFileT< FileChunk >, and vrs::AtomicDiskFile.

◆ delegateOpen()

int vrs::FileHandler::delegateOpen	(	const FileSpec &	fileSpec,
		unique_ptr< FileHandler > &	outNewDelegate
	)

overridevirtual

Open a file, while giving the opportunity to the FileHandler to delegate the file operations to another FileHandler. With this method, a FileHandler might decide that another FileHandler is the right one to open a file, after inspecting the spec, parsing of the path, or lookup.

Parameters

fileSpec	file specification.
outNewDelegate	If provided, might be a fallback FileHandler to use. On exit, may be set to a different FileHandler than the current object, if the current FileHandler was not ultimately the right one to handle the provided path, or cleared if the current FileHandler should be used to continue accessing the file.

Returns: A status code, 0 meaning success. Use errorCodeToString() to get an error description.

Implements vrs::FileDelegator.

◆ forgetFurtherChunks()

virtual void vrs::FileHandler::forgetFurtherChunks ( int64_t maxSize )

pure virtual

Call this method to forget any chunk beyond this file size.

Implemented in vrs::DiskFileT< FileChunk >.

◆ getCachingStrategy()

virtual CachingStrategy vrs::FileHandler::getCachingStrategy ( ) const

inlinevirtual

Get caching strategy.

Returns: Caching strategy.

◆ getChunkPos()

virtual int64_t vrs::FileHandler::getChunkPos ( ) const

pure virtual

Get position in the current chunk.

Returns: The current position in the current chunk.

Implemented in vrs::DiskFileT< FileChunk >.

◆ getChunkRange()

virtual int vrs::FileHandler::getChunkRange	(	int64_t &	outChunkOffset,
		int64_t &	outChunkSize
	)		const

pure virtual

Get range of the current chunk.

Parameters

outChunkOffset	index of the first byte of the chunk.
outChunkSize	number of bytes in the chunk.

Returns: A status of 0 if the request succeeded, or some error code (no file is open...)

Implemented in vrs::DiskFileT< FileChunk >.

◆ getFileChunks()

virtual vector< std::pair< string, int64_t > > vrs::FileHandler::getFileChunks ( ) const

pure virtual

Get the list of chunks, path + size.

Returns: A succession of path-size pairs.

Implemented in vrs::DiskFileT< FileChunk >.

◆ getLastError()

virtual int vrs::FileHandler::getLastError ( ) const

pure virtual

Get the last error code.

Returns: A status code, 0 meaning success.

Implemented in vrs::DiskFileT< FileChunk >.

◆ getLastRWSize()

virtual size_t vrs::FileHandler::getLastRWSize ( ) const

pure virtual

Get the number of bytes actually moved during the last read or write operation.

Returns: The number of bytes last read or written during the last read or write call.

Implemented in vrs::DiskFileT< FileChunk >.

◆ getPos()

virtual int64_t vrs::FileHandler::getPos ( ) const

pure virtual

Get the absolute position in the file, in a chunk aware way.

Returns: The absolute position in the file, including all previous chunks.

Implemented in vrs::DiskFileT< FileChunk >.

◆ getTotalSize()

virtual int64_t vrs::FileHandler::getTotalSize ( ) const

pure virtual

Get the total size of all the chunks considered.

Returns: The total size of the open file, or 0.

Implemented in vrs::DiskFileT< FileChunk >.

◆ isAvailableOrPrefetch()

virtual bool vrs::FileHandler::isAvailableOrPrefetch ( MAYBE_UNUSED size_t length )

inlinevirtual

Check if a number of bytes are available for immediate return (e.g. on disk or in-cache)

Parameters

length the number of bytes to check availability of.

Returns: true if available, false if unavailable (e.g. requiring a network fetch)

◆ isEof()

virtual bool vrs::FileHandler::isEof ( ) const

pure virtual

Tell if we are at the end of the last chunk.

Returns: True if the read/write pointer is past the last byte of the file.

Implemented in vrs::DiskFileT< FileChunk >.

◆ isOpened()

virtual bool vrs::FileHandler::isOpened ( ) const

pure virtual

Tell if a file is actually open.

Returns: True if a file is currently open.

Implemented in vrs::DiskFileT< FileChunk >.

◆ isReadOnly()

bool vrs::FileHandler::isReadOnly ( ) const

virtual

Find out if the file is currently open in read-only mode.

Returns: True if the file is currently open in read-only mode. Undefined if no file is open.

Reimplemented in vrs::DiskFileT< FileChunk >, and vrs::WriteFileHandler.

◆ isRemoteFileSystem()

virtual bool vrs::FileHandler::isRemoteFileSystem ( ) const

pure virtual

Tell if the file handler is handling remote data. Readers might need caching. Writers might not support modifying written data (and require a split head).

Implemented in vrs::DiskFileT< FileChunk >.

◆ makeNew()

virtual unique_ptr< FileHandler > vrs::FileHandler::makeNew ( ) const

pure virtual

Make a new instance of the concrete class implementing this interface in its default state, no matter what this object's state is, so that we can access more files using the same method.

Returns: A new object of the concrete type, ready to be used to open a new file.

Implemented in vrs::DiskFileT< FileChunk >.

◆ open()

int vrs::FileHandler::open ( const string & filePath )

virtual

Open a file in read-only mode.

Parameters

filePath a disk path, or anything that the particular module recognizes.

Returns: A status code, 0 meaning success.

◆ openSpec()

virtual int vrs::FileHandler::openSpec ( const FileSpec & fileSpec )

pure virtual

Open a file in read-only mode.

Parameters

fileSpec a file spec supported by this file handler.

Returns: A status code, 0 meaning success.

Implemented in vrs::DiskFileT< FileChunk >.

◆ prefetchReadSequence()

virtual bool vrs::FileHandler::prefetchReadSequence	(	MAYBE_UNUSED const vector< std::pair< size_t, size_t > > &	sequence,
		MAYBE_UNUSED bool	clearSequence = `true`
	)

inlinevirtual

Tell what read operations are going to happen, so that, if the file handler supports it, data can be cached ahead of time.

Parameters

sequence	a series of (file_offset, length), ordered by anticipated request order. Read request must not happen exactly as described: each segment may be read in multiple successive requests section or entire segments may be skipped entirely Warning: If a read request is made out of order (backward), or outside the sequence, the predictive cache may be disabled, in part or entirely.
clearSequence	Flag on whether to cancel any pre-existing custom read sequence upon caching starts.

Returns: True if the file handler support custom read sequences.

◆ purgeCache()

virtual bool vrs::FileHandler::purgeCache ( )

inlinevirtual

Purge read cache buffer, if any. Sets the caching strategy to Passive, and clears any pending read sequence.

Returns: True if the read caches were cleared (or there were none to begin with).

◆ read()

virtual int vrs::FileHandler::read	(	void *	buffer,
		size_t	length
	)

pure virtual

Read a number of bytes, in a chunk aware way. If fewer than length bytes can be read, an error code is returned, then use getLastRWSize() to know how many bytes were really read. If there are too few remaining bytes in the current chunk, then the new chunk is opened and read, until enough data can be read.

Parameters

buffer	a buffer to the bytes to write.
length	the number of bytes to write.

Returns: A status code, 0 meaning success and length bytes were successfully read.

Implemented in vrs::DiskFileT< FileChunk >.

◆ setCachingStrategy()

virtual bool vrs::FileHandler::setCachingStrategy ( CachingStrategy )

inlinevirtual

Set caching strategy.

Parameters

CachingStrategy Caching strategy desired.

Returns: True if the caching strategy was set. False if the file handler doesn't support the requested strategy, or any particular strategy.

◆ setPos()

virtual int vrs::FileHandler::setPos ( int64_t offset )

pure virtual

Set the file position at an arbitrary position, in a chunk aware way.

Parameters

offset the absolute position to jump to, which may be forward or backward.

Returns: A status code, 0 meaning success.

Implemented in vrs::DiskFileT< FileChunk >.

◆ skipForward()

virtual int vrs::FileHandler::skipForward ( int64_t offset )

pure virtual

Skip a number of bytes further in the file, in a chunk aware way.

Parameters

offset the number of bytes to skip.

Returns: A status code, 0 meaning success.

Implemented in vrs::DiskFileT< FileChunk >.

The documentation for this class was generated from the following files:

vrs/FileHandler.h
vrs/FileHandler.cpp

Classes

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Function Documentation

◆ close()

◆ delegateOpen()

◆ forgetFurtherChunks()

◆ getCachingStrategy()

◆ getChunkPos()

◆ getChunkRange()

◆ getFileChunks()

◆ getLastError()

◆ getLastRWSize()

◆ getPos()

◆ getTotalSize()

◆ isAvailableOrPrefetch()

◆ isEof()

◆ isOpened()

◆ isReadOnly()

◆ isRemoteFileSystem()

◆ makeNew()

◆ open()

◆ openSpec()

◆ prefetchReadSequence()

◆ purgeCache()

◆ read()

◆ setCachingStrategy()

◆ setPos()

◆ skipForward()