Ocean
|
This class implements a homography-based tracker for planar backgrounds. More...
Data Structures | |
class | KeyFrame |
This class composes several parameters necessary for a key-frame. More... | |
Public Member Functions | |
HomographyTracker () | |
Creates a new tracker object. More... | |
bool | resetRegion (const AnyCamera &camera, const Box2 ®ion) |
Sets a new region of interest (or resets an existing region of interest). More... | |
bool | resetRegion (const AnyCamera &camera, const Box2 ®ion, const Quaternion &cameraOrientation, const Vector3 &planeNormal, HomogenousMatrix4 *pose=nullptr, HomogenousMatrix4 *plane=nullptr) |
Sets a new region of interest (or resets an existing region of interest). More... | |
bool | determineHomography (const AnyCamera &camera, const Frame &yFrame, SquareMatrix3 &homography, HomogenousMatrix4 *pose, const Quaternion &cameraOrientation=Quaternion(false), Worker *worker=nullptr) |
Determines the homography between the current frame and the initial frame. More... | |
void | reset () |
Resets the homography tracker. More... | |
Protected Types | |
enum | HomographyQuality { HQ_FAILED , HQ_MODERATE , HQ_GOOD } |
Definition of homography qualities. More... | |
typedef std::vector< Vectors2 > | Vectors2Pyramid |
Definition of a pyramid of 2D image points, mainly a vector with image points located in individual pyramid frames. More... | |
typedef std::vector< Vectors3 > | Vectors3Pyramid |
Definition of a pyramid of 3D object points, mainly a vector with object points visible in individual pyramid frames. More... | |
Static Protected Member Functions | |
static bool | addNewFeaturePointsToPyramid (const CV::FramePyramid &yFramePyramid, Vectors2Pyramid &pointsPyramid, Vectors2Pyramid &initialPointsPyramid, const Box2 ®ion, const SquareMatrix3 &homography, const unsigned int minimalFeatures=30u, const unsigned int maximalFeatures=60u, Worker *worker=nullptr) |
Adds new feature points to all pyramid layers (at least to all desired layers e.g., layer 0 and 2) if the layers do not contain enough feature points already. More... | |
static bool | addNewFeaturePointsToPyramidLayer (const CV::FramePyramid &yFramePyramid, Vectors2Pyramid &pointsPyramid, const Triangles2 &layerSubRegionTriangles, const unsigned int layer, const unsigned int featureThreshold=25u, const unsigned int minimalFeatures=30u, const unsigned int maximalFeatures=60u, Worker *worker=nullptr) |
Adds new feature points to one specific pyramid layer if the layer does not contain enough feature points already. More... | |
static bool | trackPoints (const CV::FramePyramid &yPreviousFramePyramid, const CV::FramePyramid &yCurrentFramePyramid, Vectors2Pyramid &previousPointsPyramid, Vectors2Pyramid ¤tPointsPyramid, const unsigned int startLayer, const float maximalOffsetPercent, Indices32 &validTrackedPointIndices, const SquareMatrix3 &roughHomography=SquareMatrix3(false), const unsigned int coarsestLayerRadius=2u, const unsigned int subPixelIterations=2u, Worker *worker=nullptr) |
static HomographyQuality | determineHomographyWithPyramid (const AnyCamera &camera, const Plane3 &plane, const CV::FramePyramid &yPreviousFramePyramid, const CV::FramePyramid &yCurrentFramePyramid, Vectors2Pyramid &previousPointsPyramid, Vectors2Pyramid ¤tPointsPyramid, Vectors2Pyramid &initialPointsPyramid, const SquareMatrix3 &previousHomography, const Box2 ®ion, SquareMatrix3 &homography, HomogenousMatrix4 *pose, const SquareMatrix3 &predictedLocalHomography, const Quaternion &initialCameraOrientation, const Quaternion ¤tCameraOrientation, RandomGenerator &randomGenerator, const float explicitMaximalOffsetPercent=-1.0f, Worker *worker=nullptr) |
Determines the homography between two consecutive camera frames based on known feature points located in the previous frame (pyramid). More... | |
static bool | isRegionVisible (const Quaternion &wRi, const Quaternion &wRc, const Scalar maximalAngle=Numeric::deg2rad(50)) |
Returns whether the region of interest is visible based on a simple angle threshold. More... | |
static bool | isRegionVisible (const AnyCamera &camera, const SquareMatrix3 &globalHomography, const Box2 &initialRegion) |
Returns whether the region of interest is visible based on the known homography for the current frame. More... | |
static bool | isHomographyPlausible (const SquareMatrix3 &pHi, const SquareMatrix3 &cHi, const Box2 &initialRegion, const Scalar maximalAngleChange=Numeric::deg2rad(5)) |
Returns whether a given homography is plausible in the context of a previous homography. More... | |
static bool | isPosePlausible (const HomogenousMatrix4 ¤tPose, const Quaternion &initialCameraOrientation, const Quaternion ¤tCameraOrientation, const Scalar maximalAngle=Numeric::deg2rad(3)) |
Returns whether a given 6DOF camera pose if plausible based on e.g., an IMU-based camera orientation. More... | |
Protected Attributes | |
CV::FramePyramid | previousFramePyramid_ |
The frame pyramid of the previous frame. More... | |
CV::FramePyramid | currentFramePyramid_ |
The frame pyramid of the current frame. More... | |
RandomGenerator | randomGenerator_ |
The random generator object. More... | |
Vectors2Pyramid | initialPointsPyramid_ |
The image points located in the coordinate system of the initial frame (the frame in which the tracking region has been defined). More... | |
Vectors2Pyramid | previousPointsPyramid_ |
The image points located in 'previousFramePyramid_'. More... | |
Quaternion | initialCameraOrientation_ |
The orientation of the initial camera frame, (wRi), if known. More... | |
Quaternion | previousCameraOrientation_ |
The orientation of the camera matching with the previous camera frame, defined w.r.t. the world coordinate system (wTp). More... | |
SquareMatrix3 | globalHomography_ |
The homography transforming initial image points to current image points: currentImagePoint = globalHomography_ * initialImagePoint, (globalHomography_ = cHi). More... | |
Quaternion | globalCameraOrientation_ |
The orientation of the camera frame for the last valid homography 'globalHomography_' (wTl), if known. More... | |
Box2 | region_ |
The tracking region (the region of interest) located on the plane, defined in the coordinate system of the initial frame. More... | |
Plane3 | plane_ |
The 3D plane on which all feature points will be located, if known. More... | |
KeyFrame | keyFrames_ [numberKeyFrames_] |
The two key frames which will be used during re-initialization. More... | |
bool | needsReInitialization_ = false |
True, if the tracker needs to be re-initialized. More... | |
Static Protected Attributes | |
static constexpr size_t | numberKeyFrames_ = 2 |
This class implements a homography-based tracker for planar backgrounds.
The tracker can be started with a region of interest located on a planar background, while the entire surrounding area around the region may also be used for tracking.
|
protected |
Definition of a pyramid of 2D image points, mainly a vector with image points located in individual pyramid frames.
|
protected |
Definition of a pyramid of 3D object points, mainly a vector with object points visible in individual pyramid frames.
|
protected |
Definition of homography qualities.
|
inline |
Creates a new tracker object.
|
staticprotected |
Adds new feature points to all pyramid layers (at least to all desired layers e.g., layer 0 and 2) if the layers do not contain enough feature points already.
yFramePyramid | The frame pyramid of the frame for which the new feature points will be determined, must be valid |
pointsPyramid | The pyramid of already existing feature points for the given frame (individual points for individual pyramid layers) |
initialPointsPyramid | The pyramid of feature points defined in the initial camera frame corresponding with the points in 'pointsPyramid' |
homography | The homography transforming points defined in the initial camera frame to points defined in the given camera frame (pyramid), must be valid, (currentPoint = homography * initialPoint) |
region | The region of interest defined in the initial camera frame, this region will be used prioritize feature points in this area, can be invalid if no region of interest is known |
minimalFeatures | The minimal number of feature points each pyramid layer should contain, with range [1, infinity) |
maximalFeatures | The maximal number of feature points each pyramid layer should contain, with range [minimalFeatures, infinity) |
worker | Optional worker object to distribute the computation |
|
staticprotected |
Adds new feature points to one specific pyramid layer if the layer does not contain enough feature points already.
This function is supposed to be called out of 'addNewFeaturePointsToPyramid()'
yFramePyramid | The frame pyramid of the frame for which the new feature points will be determined, must be valid |
pointsPyramid | The pyramid of already existing feature points (individual points for individual pyramid layers) |
layerSubRegionTriangles | The triangles defined a sub-region within the specified layer in which more feature points will be determined compared to the surrounding area, no triangle to avoid the usage of a hot spot |
layer | The pyramid layer for which the new feature points will be added, with range [0, yFramePyramid.size() - 1] |
featureThreshold | The strength threshold each new feature point must exceed to count as new feature candidate, with range [0, 255] |
minimalFeatures | The minimal number of feature points each pyramid layer should contain, with range [1, infinity) |
maximalFeatures | The maximal number of feature points (the ideal number of feature points) the pyramid layer should contain, in case the layer contains more points already, no additional point is added, with range [minimalFeatures, infinity) |
worker | Optional worker object to distribute the computation |
bool Ocean::Tracking::Point::HomographyTracker::determineHomography | ( | const AnyCamera & | camera, |
const Frame & | yFrame, | ||
SquareMatrix3 & | homography, | ||
HomogenousMatrix4 * | pose, | ||
const Quaternion & | cameraOrientation = Quaternion(false) , |
||
Worker * | worker = nullptr |
||
) |
Determines the homography between the current frame and the initial frame.
The initial frame is the frame given after the region of interest has been (re-)set.
camera | The camera profile defining the projection, must be valid |
yFrame | The current image frame for which the homography will be determined (matching with the camera profile), must have a pixel format FORMAT_Y8, must be valid |
homography | The resulting homography transforming a point defined in the initial frame to a point defined in the current frame (currentPoint = homograph * initialPoint) |
pose | Optional resulting 6DOF camera pose in case the tracker has been reset with a known 3D plane normal, otherwise an invalid pose |
cameraOrientation | The orientation of the camera matching with the given (current) camera frame, defined w.r.t. the world coordinate system (wTc) |
worker | Optional worker object to distribute the computation |
|
staticprotected |
Determines the homography between two consecutive camera frames based on known feature points located in the previous frame (pyramid).
camera | The camera profile defining the projection, must be valid |
plane | The 3D plane on which the image points (the corresponding 3D object points) are located, must be valid |
yPreviousFramePyramid | The frame pyramid of the previous camera frame, must have a pixel format FORMAT_Y8, must be valid |
yCurrentFramePyramid | The frame pyramid of the current camera frame, must have a pixel format FORMAT_Y8, must be valid |
previousPointsPyramid | The pyramid of known points located in the previous camera frame, these points will be used for homography estimation, points which could not be used for tracking will be removed during homography estimation, must be valid |
currentPointsPyramid | The resulting pyramid of points located in the current camera frame, one point of each point in the pyramid of points for the previous camera frame 'previousPointsPyramid' |
initialPointsPyramid | The pyramid of known points located in the initial camera frame, one point of each point in the pyramid of points for the previous camera frame 'previousPointsPyramid' |
previousHomography | The known homography transforming points defined in the initial camera frame to points defined in the previous camera frame (defined for the finest pyramid layer), must be valid, (previousPoint = previousHomography * initialPoint) |
region | The region of interest within the camera frame, must be valid |
homography | The resulting homography transforming points defined in the initial camera frame to points defined in the current camera frame (defined for the finest pyramid layer), (currentPoint = previousHomography * initialPoint) |
pose | Optional resulting 6DOF camera pose in case the tracker has been reset with a known 3D plane normal, otherwise an invalid pose |
predictedLocalHomography | Optional known prediction of the homography transformation points defined in the previous camera frame to points defined in the current camera frame, a null matrix otherwise, (currentPoint = predictedLocalHomography * previousPoint) |
initialCameraOrientation | The orientation of the camera frame for the initial frame, (wRi), if known |
currentCameraOrientation | The orientation of the camera frame for the current frame, (wRc), if known |
randomGenerator | Random generator to be used |
explicitMaximalOffsetPercent | An explicit threshold defining the maximal offset between two corresponding feature points, defined in relation to the maximal frame size, with range [0, 1], -1 to use internal thresholds |
worker | Optional worker object to distribute the computation |
|
staticprotected |
Returns whether a given homography is plausible in the context of a previous homography.
Based on a region of interest, the homography is rated based on the angular difference within the transformed region of interest.
In case, the angle between two neighboring edges of the ROI exceeds a given threshold, the homography is rated is invalid.
pHi | The homography transforming points defined in the initial camera frame to points defined in the previous camera frame (previousPoint = pHi * initialPoint), must be valid |
cHi | The homography transforming points defined in the initial camera frame to points defined in the current camera frame (currentPoint = cHi * initialPoint), must be valid |
initialRegion | The region of interest defined in the initial camera frame, must be valid |
maximalAngleChange | The maximal allowed change of inner angles of the ROI so that the homography counts as plausible, in radian, with range [0, PI_4] |
|
staticprotected |
Returns whether a given 6DOF camera pose if plausible based on e.g., an IMU-based camera orientation.
currentPose | The current 6DOF camera pose to be checked, must be valid |
initialCameraOrientation | The orientation of the initial camera frame, (wTi), if known |
currentCameraOrientation | The orientation of the current camera frame, (wTc), if known |
maximalAngle | The maximal angle between the orientation of the given 6DOF camera pose and the IMU-based camera orientation so that the pose counts as plausible, in radian, with range [0, PI_2] |
|
staticprotected |
Returns whether the region of interest is visible based on the known homography for the current frame.
camera | The camera profile defining the projection, must be valid |
globalHomography | The homography transforming initial image points to current image points: currentImagePoint = globalHomography_ * initialImagePoint, (globalHomography_ = cHi) |
initialRegion | The initial region of interest, must be valid |
|
staticprotected |
Returns whether the region of interest is visible based on a simple angle threshold.
The function determines the angle between viewing direction (the z-axis) for the current frame and the initial frame.
In case, the viewing angle exceeds a specified threshold, the region is rated to be invisible.
If one or both provided camera orientation are unknown, the region is rated to be visible.
wRi | The rotation of the camera matching with the initial camera frame, can be invalid if unknown |
wRc | The rotation of the camera matching with the current camera frame, can be invalid if unknown |
maximalAngle | The maximal angle between both viewing directions, in radian, with range [0, PI_2] |
|
inline |
Resets the homography tracker.
bool Ocean::Tracking::Point::HomographyTracker::resetRegion | ( | const AnyCamera & | camera, |
const Box2 & | region | ||
) |
Sets a new region of interest (or resets an existing region of interest).
The tracker will be set to a new initial state and any resulting homography will be defined in relation to this initial state.
camera | The camera profile defining the projection, must be valid |
region | The region of interest to be set, defined in the coordinate system of the image frames used for tracking, must be valid |
bool Ocean::Tracking::Point::HomographyTracker::resetRegion | ( | const AnyCamera & | camera, |
const Box2 & | region, | ||
const Quaternion & | cameraOrientation, | ||
const Vector3 & | planeNormal, | ||
HomogenousMatrix4 * | pose = nullptr , |
||
HomogenousMatrix4 * | plane = nullptr |
||
) |
Sets a new region of interest (or resets an existing region of interest).
The tracker will be set to a new initial state and any resulting homography will be defined in relation to this initial state.
camera | The camera profile defining the projection, must be valid |
region | The region of interest to be set, defined in the coordinate system of the image frames used for tracking, must be valid |
cameraOrientation | The orientation of the camera matching with the given (current) camera frame, defined w.r.t. the world coordinate system (wTc), must be valid |
planeNormal | Normal the planar background at the region of interest, defined w.r.t. the world coordinate system (wN), must be valid |
pose | Optional resulting 6DOF camera pose for the initial camera frame, transforming points defined in the coordinate system of the camera to points defined in the world coordinate system (wTc) |
plane | Optional resulting transformation transforming points defined in the coordinate system of the plane to points defined in the coordinate system of the world coordinate system (wTp) |
|
staticprotected |
yPreviousFramePyramid | The pyramid image of the previous frame with pixel format FORMAT_Y8 (1 channel, uint8_t), must be valid |
yCurrentFramePyramid | The pyramid image of the current frame with same pixel format as 'yPreviousFramePyramid', must be valid |
previousPointsPyramid | The image points located in the pyramid image of the previous frame, which will be tracked to the current frame, (with different points for different pyramid layers) |
currentPointsPyramid | The resulting image points located in the pyramid image of the current frame, one point for each point in the corresponding previous image layer |
startLayer | The layer within the pyramid image of the previous frame from which the image points will be tracked, with range [0, yPreviousFramePyramid.layers() - 1] |
maximalOffsetPercent | The maximal pixel offset between corresponding image points defined in percentage in relation to the max(layerWidth, layerHeight), with range [0, 1] |
validTrackedPointIndices | The resulting indices of all previous image points which could be tracked reliably |
roughHomography | Optional homography providing a rough guess for the location of the points in the current frame, defined for the finest pyramid layer (finestCurrentPoint = roughHomography * finestPreviousPoint), a null matrix otherwise |
coarsestLayerRadius | The search radius on the coarsest pyramid layer used for tracking, with range [1, infinity) |
subPixelIterations | The number of sub-pixel iterations that will be applied, each iteration doubles the sub-pixel accuracy, with range [1, infinity) |
worker | Optional worker object to distribute the computation |
|
protected |
The frame pyramid of the current frame.
|
protected |
The orientation of the camera frame for the last valid homography 'globalHomography_' (wTl), if known.
|
protected |
The homography transforming initial image points to current image points: currentImagePoint = globalHomography_ * initialImagePoint, (globalHomography_ = cHi).
|
protected |
The orientation of the initial camera frame, (wRi), if known.
|
protected |
The image points located in the coordinate system of the initial frame (the frame in which the tracking region has been defined).
|
protected |
The two key frames which will be used during re-initialization.
|
protected |
True, if the tracker needs to be re-initialized.
|
staticconstexprprotected |
|
protected |
The 3D plane on which all feature points will be located, if known.
|
protected |
The orientation of the camera matching with the previous camera frame, defined w.r.t. the world coordinate system (wTp).
|
protected |
The frame pyramid of the previous frame.
|
protected |
The image points located in 'previousFramePyramid_'.
|
protected |
The random generator object.
|
protected |
The tracking region (the region of interest) located on the plane, defined in the coordinate system of the initial frame.