Ocean
Ocean::Tracking::Point::HomographyTracker Class Reference

This class implements a homography-based tracker for planar backgrounds. More...

Data Structures

class  KeyFrame
 This class composes several parameters necessary for a key-frame. More...
 

Public Member Functions

 HomographyTracker ()
 Creates a new tracker object. More...
 
bool resetRegion (const AnyCamera &camera, const Box2 &region)
 Sets a new region of interest (or resets an existing region of interest). More...
 
bool resetRegion (const AnyCamera &camera, const Box2 &region, const Quaternion &cameraOrientation, const Vector3 &planeNormal, HomogenousMatrix4 *pose=nullptr, HomogenousMatrix4 *plane=nullptr)
 Sets a new region of interest (or resets an existing region of interest). More...
 
bool determineHomography (const AnyCamera &camera, const Frame &yFrame, SquareMatrix3 &homography, HomogenousMatrix4 *pose, const Quaternion &cameraOrientation=Quaternion(false), Worker *worker=nullptr)
 Determines the homography between the current frame and the initial frame. More...
 
void reset ()
 Resets the homography tracker. More...
 

Protected Types

enum  HomographyQuality { HQ_FAILED , HQ_MODERATE , HQ_GOOD }
 Definition of homography qualities. More...
 
typedef std::vector< Vectors2Vectors2Pyramid
 Definition of a pyramid of 2D image points, mainly a vector with image points located in individual pyramid frames. More...
 
typedef std::vector< Vectors3Vectors3Pyramid
 Definition of a pyramid of 3D object points, mainly a vector with object points visible in individual pyramid frames. More...
 

Static Protected Member Functions

static bool addNewFeaturePointsToPyramid (const CV::FramePyramid &yFramePyramid, Vectors2Pyramid &pointsPyramid, Vectors2Pyramid &initialPointsPyramid, const Box2 &region, const SquareMatrix3 &homography, const unsigned int minimalFeatures=30u, const unsigned int maximalFeatures=60u, Worker *worker=nullptr)
 Adds new feature points to all pyramid layers (at least to all desired layers e.g., layer 0 and 2) if the layers do not contain enough feature points already. More...
 
static bool addNewFeaturePointsToPyramidLayer (const CV::FramePyramid &yFramePyramid, Vectors2Pyramid &pointsPyramid, const Triangles2 &layerSubRegionTriangles, const unsigned int layer, const unsigned int featureThreshold=25u, const unsigned int minimalFeatures=30u, const unsigned int maximalFeatures=60u, Worker *worker=nullptr)
 Adds new feature points to one specific pyramid layer if the layer does not contain enough feature points already. More...
 
static bool trackPoints (const CV::FramePyramid &yPreviousFramePyramid, const CV::FramePyramid &yCurrentFramePyramid, Vectors2Pyramid &previousPointsPyramid, Vectors2Pyramid &currentPointsPyramid, const unsigned int startLayer, const float maximalOffsetPercent, Indices32 &validTrackedPointIndices, const SquareMatrix3 &roughHomography=SquareMatrix3(false), const unsigned int coarsestLayerRadius=2u, const unsigned int subPixelIterations=2u, Worker *worker=nullptr)
 
static HomographyQuality determineHomographyWithPyramid (const AnyCamera &camera, const Plane3 &plane, const CV::FramePyramid &yPreviousFramePyramid, const CV::FramePyramid &yCurrentFramePyramid, Vectors2Pyramid &previousPointsPyramid, Vectors2Pyramid &currentPointsPyramid, Vectors2Pyramid &initialPointsPyramid, const SquareMatrix3 &previousHomography, const Box2 &region, SquareMatrix3 &homography, HomogenousMatrix4 *pose, const SquareMatrix3 &predictedLocalHomography, const Quaternion &initialCameraOrientation, const Quaternion &currentCameraOrientation, RandomGenerator &randomGenerator, const float explicitMaximalOffsetPercent=-1.0f, Worker *worker=nullptr)
 Determines the homography between two consecutive camera frames based on known feature points located in the previous frame (pyramid). More...
 
static bool isRegionVisible (const Quaternion &wRi, const Quaternion &wRc, const Scalar maximalAngle=Numeric::deg2rad(50))
 Returns whether the region of interest is visible based on a simple angle threshold. More...
 
static bool isRegionVisible (const AnyCamera &camera, const SquareMatrix3 &globalHomography, const Box2 &initialRegion)
 Returns whether the region of interest is visible based on the known homography for the current frame. More...
 
static bool isHomographyPlausible (const SquareMatrix3 &pHi, const SquareMatrix3 &cHi, const Box2 &initialRegion, const Scalar maximalAngleChange=Numeric::deg2rad(5))
 Returns whether a given homography is plausible in the context of a previous homography. More...
 
static bool isPosePlausible (const HomogenousMatrix4 &currentPose, const Quaternion &initialCameraOrientation, const Quaternion &currentCameraOrientation, const Scalar maximalAngle=Numeric::deg2rad(3))
 Returns whether a given 6DOF camera pose if plausible based on e.g., an IMU-based camera orientation. More...
 

Protected Attributes

CV::FramePyramid previousFramePyramid_
 The frame pyramid of the previous frame. More...
 
CV::FramePyramid currentFramePyramid_
 The frame pyramid of the current frame. More...
 
RandomGenerator randomGenerator_
 The random generator object. More...
 
Vectors2Pyramid initialPointsPyramid_
 The image points located in the coordinate system of the initial frame (the frame in which the tracking region has been defined). More...
 
Vectors2Pyramid previousPointsPyramid_
 The image points located in 'previousFramePyramid_'. More...
 
Quaternion initialCameraOrientation_
 The orientation of the initial camera frame, (wRi), if known. More...
 
Quaternion previousCameraOrientation_
 The orientation of the camera matching with the previous camera frame, defined w.r.t. the world coordinate system (wTp). More...
 
SquareMatrix3 globalHomography_
 The homography transforming initial image points to current image points: currentImagePoint = globalHomography_ * initialImagePoint, (globalHomography_ = cHi). More...
 
Quaternion globalCameraOrientation_
 The orientation of the camera frame for the last valid homography 'globalHomography_' (wTl), if known. More...
 
Box2 region_
 The tracking region (the region of interest) located on the plane, defined in the coordinate system of the initial frame. More...
 
Plane3 plane_
 The 3D plane on which all feature points will be located, if known. More...
 
KeyFrame keyFrames_ [numberKeyFrames_]
 The two key frames which will be used during re-initialization. More...
 
bool needsReInitialization_ = false
 True, if the tracker needs to be re-initialized. More...
 

Static Protected Attributes

static constexpr size_t numberKeyFrames_ = 2
 

Detailed Description

This class implements a homography-based tracker for planar backgrounds.

The tracker can be started with a region of interest located on a planar background, while the entire surrounding area around the region may also be used for tracking.

Member Typedef Documentation

◆ Vectors2Pyramid

Definition of a pyramid of 2D image points, mainly a vector with image points located in individual pyramid frames.

◆ Vectors3Pyramid

Definition of a pyramid of 3D object points, mainly a vector with object points visible in individual pyramid frames.

Member Enumeration Documentation

◆ HomographyQuality

Definition of homography qualities.

Enumerator
HQ_FAILED 

The homography is invalid.

HQ_MODERATE 

The homography has a moderate quality, good enough for visualization, but not good enough to e.g., use the frame as key-frame.

HQ_GOOD 

The homography has a good quality so that the corresponding frame e.g., can be used as key-frame.

Constructor & Destructor Documentation

◆ HomographyTracker()

Ocean::Tracking::Point::HomographyTracker::HomographyTracker ( )
inline

Creates a new tracker object.

Member Function Documentation

◆ addNewFeaturePointsToPyramid()

static bool Ocean::Tracking::Point::HomographyTracker::addNewFeaturePointsToPyramid ( const CV::FramePyramid yFramePyramid,
Vectors2Pyramid pointsPyramid,
Vectors2Pyramid initialPointsPyramid,
const Box2 region,
const SquareMatrix3 homography,
const unsigned int  minimalFeatures = 30u,
const unsigned int  maximalFeatures = 60u,
Worker worker = nullptr 
)
staticprotected

Adds new feature points to all pyramid layers (at least to all desired layers e.g., layer 0 and 2) if the layers do not contain enough feature points already.

Parameters
yFramePyramidThe frame pyramid of the frame for which the new feature points will be determined, must be valid
pointsPyramidThe pyramid of already existing feature points for the given frame (individual points for individual pyramid layers)
initialPointsPyramidThe pyramid of feature points defined in the initial camera frame corresponding with the points in 'pointsPyramid'
homographyThe homography transforming points defined in the initial camera frame to points defined in the given camera frame (pyramid), must be valid, (currentPoint = homography * initialPoint)
regionThe region of interest defined in the initial camera frame, this region will be used prioritize feature points in this area, can be invalid if no region of interest is known
minimalFeaturesThe minimal number of feature points each pyramid layer should contain, with range [1, infinity)
maximalFeaturesThe maximal number of feature points each pyramid layer should contain, with range [minimalFeatures, infinity)
workerOptional worker object to distribute the computation
Returns
True, if succeeded
See also
addNewFeaturePointsToPyramidLayer().

◆ addNewFeaturePointsToPyramidLayer()

static bool Ocean::Tracking::Point::HomographyTracker::addNewFeaturePointsToPyramidLayer ( const CV::FramePyramid yFramePyramid,
Vectors2Pyramid pointsPyramid,
const Triangles2 layerSubRegionTriangles,
const unsigned int  layer,
const unsigned int  featureThreshold = 25u,
const unsigned int  minimalFeatures = 30u,
const unsigned int  maximalFeatures = 60u,
Worker worker = nullptr 
)
staticprotected

Adds new feature points to one specific pyramid layer if the layer does not contain enough feature points already.

This function is supposed to be called out of 'addNewFeaturePointsToPyramid()'

Parameters
yFramePyramidThe frame pyramid of the frame for which the new feature points will be determined, must be valid
pointsPyramidThe pyramid of already existing feature points (individual points for individual pyramid layers)
layerSubRegionTrianglesThe triangles defined a sub-region within the specified layer in which more feature points will be determined compared to the surrounding area, no triangle to avoid the usage of a hot spot
layerThe pyramid layer for which the new feature points will be added, with range [0, yFramePyramid.size() - 1]
featureThresholdThe strength threshold each new feature point must exceed to count as new feature candidate, with range [0, 255]
minimalFeaturesThe minimal number of feature points each pyramid layer should contain, with range [1, infinity)
maximalFeaturesThe maximal number of feature points (the ideal number of feature points) the pyramid layer should contain, in case the layer contains more points already, no additional point is added, with range [minimalFeatures, infinity)
workerOptional worker object to distribute the computation
Returns
True, if succeeded
See also
addNewFeaturePointsToPyramid().

◆ determineHomography()

bool Ocean::Tracking::Point::HomographyTracker::determineHomography ( const AnyCamera camera,
const Frame yFrame,
SquareMatrix3 homography,
HomogenousMatrix4 pose,
const Quaternion cameraOrientation = Quaternion(false),
Worker worker = nullptr 
)

Determines the homography between the current frame and the initial frame.

The initial frame is the frame given after the region of interest has been (re-)set.

Parameters
cameraThe camera profile defining the projection, must be valid
yFrameThe current image frame for which the homography will be determined (matching with the camera profile), must have a pixel format FORMAT_Y8, must be valid
homographyThe resulting homography transforming a point defined in the initial frame to a point defined in the current frame (currentPoint = homograph * initialPoint)
poseOptional resulting 6DOF camera pose in case the tracker has been reset with a known 3D plane normal, otherwise an invalid pose
cameraOrientationThe orientation of the camera matching with the given (current) camera frame, defined w.r.t. the world coordinate system (wTc)
workerOptional worker object to distribute the computation
Returns
True, if succeeded

◆ determineHomographyWithPyramid()

static HomographyQuality Ocean::Tracking::Point::HomographyTracker::determineHomographyWithPyramid ( const AnyCamera camera,
const Plane3 plane,
const CV::FramePyramid yPreviousFramePyramid,
const CV::FramePyramid yCurrentFramePyramid,
Vectors2Pyramid previousPointsPyramid,
Vectors2Pyramid currentPointsPyramid,
Vectors2Pyramid initialPointsPyramid,
const SquareMatrix3 previousHomography,
const Box2 region,
SquareMatrix3 homography,
HomogenousMatrix4 pose,
const SquareMatrix3 predictedLocalHomography,
const Quaternion initialCameraOrientation,
const Quaternion currentCameraOrientation,
RandomGenerator randomGenerator,
const float  explicitMaximalOffsetPercent = -1.0f,
Worker worker = nullptr 
)
staticprotected

Determines the homography between two consecutive camera frames based on known feature points located in the previous frame (pyramid).

Parameters
cameraThe camera profile defining the projection, must be valid
planeThe 3D plane on which the image points (the corresponding 3D object points) are located, must be valid
yPreviousFramePyramidThe frame pyramid of the previous camera frame, must have a pixel format FORMAT_Y8, must be valid
yCurrentFramePyramidThe frame pyramid of the current camera frame, must have a pixel format FORMAT_Y8, must be valid
previousPointsPyramidThe pyramid of known points located in the previous camera frame, these points will be used for homography estimation, points which could not be used for tracking will be removed during homography estimation, must be valid
currentPointsPyramidThe resulting pyramid of points located in the current camera frame, one point of each point in the pyramid of points for the previous camera frame 'previousPointsPyramid'
initialPointsPyramidThe pyramid of known points located in the initial camera frame, one point of each point in the pyramid of points for the previous camera frame 'previousPointsPyramid'
previousHomographyThe known homography transforming points defined in the initial camera frame to points defined in the previous camera frame (defined for the finest pyramid layer), must be valid, (previousPoint = previousHomography * initialPoint)
regionThe region of interest within the camera frame, must be valid
homographyThe resulting homography transforming points defined in the initial camera frame to points defined in the current camera frame (defined for the finest pyramid layer), (currentPoint = previousHomography * initialPoint)
poseOptional resulting 6DOF camera pose in case the tracker has been reset with a known 3D plane normal, otherwise an invalid pose
predictedLocalHomographyOptional known prediction of the homography transformation points defined in the previous camera frame to points defined in the current camera frame, a null matrix otherwise, (currentPoint = predictedLocalHomography * previousPoint)
initialCameraOrientationThe orientation of the camera frame for the initial frame, (wRi), if known
currentCameraOrientationThe orientation of the camera frame for the current frame, (wRc), if known
randomGeneratorRandom generator to be used
explicitMaximalOffsetPercentAn explicit threshold defining the maximal offset between two corresponding feature points, defined in relation to the maximal frame size, with range [0, 1], -1 to use internal thresholds
workerOptional worker object to distribute the computation
Returns
True, if succeeded

◆ isHomographyPlausible()

static bool Ocean::Tracking::Point::HomographyTracker::isHomographyPlausible ( const SquareMatrix3 pHi,
const SquareMatrix3 cHi,
const Box2 initialRegion,
const Scalar  maximalAngleChange = Numeric::deg2rad(5) 
)
staticprotected

Returns whether a given homography is plausible in the context of a previous homography.

Based on a region of interest, the homography is rated based on the angular difference within the transformed region of interest.
In case, the angle between two neighboring edges of the ROI exceeds a given threshold, the homography is rated is invalid.

Parameters
pHiThe homography transforming points defined in the initial camera frame to points defined in the previous camera frame (previousPoint = pHi * initialPoint), must be valid
cHiThe homography transforming points defined in the initial camera frame to points defined in the current camera frame (currentPoint = cHi * initialPoint), must be valid
initialRegionThe region of interest defined in the initial camera frame, must be valid
maximalAngleChangeThe maximal allowed change of inner angles of the ROI so that the homography counts as plausible, in radian, with range [0, PI_4]
Returns
True, if so

◆ isPosePlausible()

static bool Ocean::Tracking::Point::HomographyTracker::isPosePlausible ( const HomogenousMatrix4 currentPose,
const Quaternion initialCameraOrientation,
const Quaternion currentCameraOrientation,
const Scalar  maximalAngle = Numeric::deg2rad(3) 
)
staticprotected

Returns whether a given 6DOF camera pose if plausible based on e.g., an IMU-based camera orientation.

Parameters
currentPoseThe current 6DOF camera pose to be checked, must be valid
initialCameraOrientationThe orientation of the initial camera frame, (wTi), if known
currentCameraOrientationThe orientation of the current camera frame, (wTc), if known
maximalAngleThe maximal angle between the orientation of the given 6DOF camera pose and the IMU-based camera orientation so that the pose counts as plausible, in radian, with range [0, PI_2]
Returns
True, if so

◆ isRegionVisible() [1/2]

static bool Ocean::Tracking::Point::HomographyTracker::isRegionVisible ( const AnyCamera camera,
const SquareMatrix3 globalHomography,
const Box2 initialRegion 
)
staticprotected

Returns whether the region of interest is visible based on the known homography for the current frame.

Parameters
cameraThe camera profile defining the projection, must be valid
globalHomographyThe homography transforming initial image points to current image points: currentImagePoint = globalHomography_ * initialImagePoint, (globalHomography_ = cHi)
initialRegionThe initial region of interest, must be valid
Returns
True, if so

◆ isRegionVisible() [2/2]

static bool Ocean::Tracking::Point::HomographyTracker::isRegionVisible ( const Quaternion wRi,
const Quaternion wRc,
const Scalar  maximalAngle = Numeric::deg2rad(50) 
)
staticprotected

Returns whether the region of interest is visible based on a simple angle threshold.

The function determines the angle between viewing direction (the z-axis) for the current frame and the initial frame.
In case, the viewing angle exceeds a specified threshold, the region is rated to be invisible.
If one or both provided camera orientation are unknown, the region is rated to be visible.

Parameters
wRiThe rotation of the camera matching with the initial camera frame, can be invalid if unknown
wRcThe rotation of the camera matching with the current camera frame, can be invalid if unknown
maximalAngleThe maximal angle between both viewing directions, in radian, with range [0, PI_2]

◆ reset()

void Ocean::Tracking::Point::HomographyTracker::reset ( )
inline

Resets the homography tracker.

◆ resetRegion() [1/2]

bool Ocean::Tracking::Point::HomographyTracker::resetRegion ( const AnyCamera camera,
const Box2 region 
)

Sets a new region of interest (or resets an existing region of interest).

The tracker will be set to a new initial state and any resulting homography will be defined in relation to this initial state.

Parameters
cameraThe camera profile defining the projection, must be valid
regionThe region of interest to be set, defined in the coordinate system of the image frames used for tracking, must be valid
Returns
True, if succeeded

◆ resetRegion() [2/2]

bool Ocean::Tracking::Point::HomographyTracker::resetRegion ( const AnyCamera camera,
const Box2 region,
const Quaternion cameraOrientation,
const Vector3 planeNormal,
HomogenousMatrix4 pose = nullptr,
HomogenousMatrix4 plane = nullptr 
)

Sets a new region of interest (or resets an existing region of interest).

The tracker will be set to a new initial state and any resulting homography will be defined in relation to this initial state.

Parameters
cameraThe camera profile defining the projection, must be valid
regionThe region of interest to be set, defined in the coordinate system of the image frames used for tracking, must be valid
cameraOrientationThe orientation of the camera matching with the given (current) camera frame, defined w.r.t. the world coordinate system (wTc), must be valid
planeNormalNormal the planar background at the region of interest, defined w.r.t. the world coordinate system (wN), must be valid
poseOptional resulting 6DOF camera pose for the initial camera frame, transforming points defined in the coordinate system of the camera to points defined in the world coordinate system (wTc)
planeOptional resulting transformation transforming points defined in the coordinate system of the plane to points defined in the coordinate system of the world coordinate system (wTp)
Returns
True, if succeeded

◆ trackPoints()

static bool Ocean::Tracking::Point::HomographyTracker::trackPoints ( const CV::FramePyramid yPreviousFramePyramid,
const CV::FramePyramid yCurrentFramePyramid,
Vectors2Pyramid previousPointsPyramid,
Vectors2Pyramid currentPointsPyramid,
const unsigned int  startLayer,
const float  maximalOffsetPercent,
Indices32 validTrackedPointIndices,
const SquareMatrix3 roughHomography = SquareMatrix3(false),
const unsigned int  coarsestLayerRadius = 2u,
const unsigned int  subPixelIterations = 2u,
Worker worker = nullptr 
)
staticprotected
Parameters
yPreviousFramePyramidThe pyramid image of the previous frame with pixel format FORMAT_Y8 (1 channel, uint8_t), must be valid
yCurrentFramePyramidThe pyramid image of the current frame with same pixel format as 'yPreviousFramePyramid', must be valid
previousPointsPyramidThe image points located in the pyramid image of the previous frame, which will be tracked to the current frame, (with different points for different pyramid layers)
currentPointsPyramidThe resulting image points located in the pyramid image of the current frame, one point for each point in the corresponding previous image layer
startLayerThe layer within the pyramid image of the previous frame from which the image points will be tracked, with range [0, yPreviousFramePyramid.layers() - 1]
maximalOffsetPercentThe maximal pixel offset between corresponding image points defined in percentage in relation to the max(layerWidth, layerHeight), with range [0, 1]
validTrackedPointIndicesThe resulting indices of all previous image points which could be tracked reliably
roughHomographyOptional homography providing a rough guess for the location of the points in the current frame, defined for the finest pyramid layer (finestCurrentPoint = roughHomography * finestPreviousPoint), a null matrix otherwise
coarsestLayerRadiusThe search radius on the coarsest pyramid layer used for tracking, with range [1, infinity)
subPixelIterationsThe number of sub-pixel iterations that will be applied, each iteration doubles the sub-pixel accuracy, with range [1, infinity)
workerOptional worker object to distribute the computation
Returns
True, if succeeded

Field Documentation

◆ currentFramePyramid_

CV::FramePyramid Ocean::Tracking::Point::HomographyTracker::currentFramePyramid_
protected

The frame pyramid of the current frame.

◆ globalCameraOrientation_

Quaternion Ocean::Tracking::Point::HomographyTracker::globalCameraOrientation_
protected

The orientation of the camera frame for the last valid homography 'globalHomography_' (wTl), if known.

◆ globalHomography_

SquareMatrix3 Ocean::Tracking::Point::HomographyTracker::globalHomography_
protected

The homography transforming initial image points to current image points: currentImagePoint = globalHomography_ * initialImagePoint, (globalHomography_ = cHi).

◆ initialCameraOrientation_

Quaternion Ocean::Tracking::Point::HomographyTracker::initialCameraOrientation_
protected

The orientation of the initial camera frame, (wRi), if known.

◆ initialPointsPyramid_

Vectors2Pyramid Ocean::Tracking::Point::HomographyTracker::initialPointsPyramid_
protected

The image points located in the coordinate system of the initial frame (the frame in which the tracking region has been defined).

◆ keyFrames_

KeyFrame Ocean::Tracking::Point::HomographyTracker::keyFrames_[numberKeyFrames_]
protected

The two key frames which will be used during re-initialization.

◆ needsReInitialization_

bool Ocean::Tracking::Point::HomographyTracker::needsReInitialization_ = false
protected

True, if the tracker needs to be re-initialized.

◆ numberKeyFrames_

constexpr size_t Ocean::Tracking::Point::HomographyTracker::numberKeyFrames_ = 2
staticconstexprprotected

◆ plane_

Plane3 Ocean::Tracking::Point::HomographyTracker::plane_
protected

The 3D plane on which all feature points will be located, if known.

◆ previousCameraOrientation_

Quaternion Ocean::Tracking::Point::HomographyTracker::previousCameraOrientation_
protected

The orientation of the camera matching with the previous camera frame, defined w.r.t. the world coordinate system (wTp).

◆ previousFramePyramid_

CV::FramePyramid Ocean::Tracking::Point::HomographyTracker::previousFramePyramid_
protected

The frame pyramid of the previous frame.

◆ previousPointsPyramid_

Vectors2Pyramid Ocean::Tracking::Point::HomographyTracker::previousPointsPyramid_
protected

The image points located in 'previousFramePyramid_'.

◆ randomGenerator_

RandomGenerator Ocean::Tracking::Point::HomographyTracker::randomGenerator_
protected

The random generator object.

◆ region_

Box2 Ocean::Tracking::Point::HomographyTracker::region_
protected

The tracking region (the region of interest) located on the plane, defined in the coordinate system of the initial frame.


The documentation for this class was generated from the following file: