![Page 1: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/1.jpg)
Computer Vision 2 – Lecture 2 Template-based Tracking (18.04.2016)
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
[email protected], [email protected]
RWTH Aachen University, Computer Vision Group
http://www.vision.rwth-aachen.de
![Page 2: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/2.jpg)
2
Content of the Lecture
• Single-Object Tracking Background modeling
Template based tracking
Color based tracking
Contour based tracking
Tracking by online classification
Tracking-by-detection
• Bayesian Filtering
• Multi-Object Tracking
• Visual Odometry
• Visual SLAM & 3D Reconstruction
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 3: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/3.jpg)
3
Today: Template based Tracking
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Image source: Robert Collins, Shi & Tomasi
![Page 4: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/4.jpg)
4
Topics of This Lecture
• Recap: Lucas-Kanade Optical Flow Brightness Constancy constraint
LK flow estimation
Coarse-to-fine estimation
• Feature Tracking KLT feature tracking
• Template Tracking LK derivation for templates
Warping functions
General LK image registration
• Applications
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 5: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/5.jpg)
5
Recap: Estimating Optical Flow
• Optical Flow Given two subsequent frames, estimate the apparent motion field u(x,y)
and v(x,y) between them.
• Key assumptions Brightness constancy: projection of the same point looks the same in
every frame.
Small motion: points do not move very far.
Spatial coherence: points move like their neighbors.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
I(x,y,t–1) I(x,y,t)
Slide credit: Svetlana Lazebnik
![Page 6: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/6.jpg)
6
Recap: The Brightness Constancy Constraint
• Brightness Constancy Equation:
• Linearizing the right hand side using Taylor expansion:
• Hence,
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
I(x,y,t–1) I(x,y,t)
),()1,,( ),,(),( tyxyx vyuxItyxI
),(),(),,()1,,( yxvIyxuItyxItyxI yx
0x y tI u I v I
Temporal derivative Spatial derivatives
Slide credit: Svetlana Lazebnik
![Page 7: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/7.jpg)
13
Recap: Solving the Aperture Problem
• How to get more equations for a pixel?
• Spatial coherence constraint Pretend the pixel’s neighbors have the same (u,v).
If we use a 5£5 window, that gives us 25 equations per pixel
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
B. Lucas and T. Kanade. An iterative image registration technique with an application to
stereo vision. In Proc. IJCAI’81, pp. 674–679, 1981.
Slide credit: Svetlana Lazebnik
![Page 8: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/8.jpg)
14
Recap: Solving the Aperture Problem
• Least squares problem:
• Minimum least squares solution given by solution of
(The summations are over all pixels in the K£K window)
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide credit: Svetlana Lazebnik
![Page 9: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/9.jpg)
15
Recap: Conditions for Solvability
• Optimal (u, v) satisfies Lucas-Kanade equation
• When is this solvable?
ATA should be invertible.
ATA entries should not be too small (noise).
ATA should be well-conditioned.
Looking for cases where A has two large eigenvalues
(i.e., corners and highly textured areas).
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide credit: Svetlana Lazebnik
![Page 10: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/10.jpg)
16
Recap: Iterative LK Refinement
1. Estimate velocity at each pixel using one iteration of LK
estimation.
2. Warp one image toward the other using the estimated flow
field. (Easier said than done)
3. Refine estimate by repeating the process.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide adapted from Steve Seitz
![Page 11: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/11.jpg)
17
Recap: Iterative LK Refinement
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
x x0
Initial guess:
Estimate:
estimate
update
(using d for displacement here instead of u)
Slide credit: Steve Seitz
![Page 12: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/12.jpg)
18
Recap: Iterative LK Refinement
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
x x0 x x0
estimate
update Initial guess:
Estimate:
(using d for displacement here instead of u)
Slide credit: Steve Seitz
![Page 13: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/13.jpg)
19
Recap: Iterative LK Refinement
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
x x0
Initial guess:
Estimate:
estimate
update
(using d for displacement here instead of u)
Slide credit: Steve Seitz
![Page 14: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/14.jpg)
20
Recap: Iterative LK Refinement
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
(using d for displacement here instead of u)
Slide credit: Steve Seitz
![Page 15: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/15.jpg)
21
Problem Case: Large Motions
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide credit: Svetlana Lazebnik
![Page 16: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/16.jpg)
22
Temporal Aliasing
• Temporal aliasing causes ambiguities in optical flow because
images can have many pixels with the same intensity.
• I.e., how do we know which ‘correspondence’ is correct?
• To overcome aliasing: coarse-to-fine estimation.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Nearest match is
correct (no aliasing)
Nearest match is
incorrect (aliasing)
actual shift
estimated shift
Slide credit: Steve Seitz
![Page 17: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/17.jpg)
23
Idea: Reduce the Resolution!
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide credit: Svetlana Lazebnik
![Page 18: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/18.jpg)
24
Recap: Coarse-to-fine Optical Flow Estimation
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Gaussian pyramid of image 1 Gaussian pyramid of image 2
Image 2 Image 1 u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Slide credit: Steve Seitz
![Page 19: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/19.jpg)
25
Recap: Coarse-to-fine Optical Flow Estimation
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Gaussian pyramid of image 1 Gaussian pyramid of image 2
Image 2 Image 1
Slide credit: Steve Seitz
Run iterative LK
Run iterative LK
Warp & upsample
.
.
.
![Page 20: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/20.jpg)
26
Topics of This Lecture
• Recap: Lucas-Kanade Optical Flow Brightness Constancy constraint
LK flow estimation
Coarse-to-fine estimation
• Feature Tracking KLT feature tracking
• Template Tracking LK derivation for templates
Warping functions
General LK image registration
• Applications
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 21: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/21.jpg)
27
KLT Feature Tracking
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
http://www.cs.unc.edu/~ssinha/Research/GPU_KLT/
![Page 22: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/22.jpg)
28
Shi-Tomasi Feature Tracker
• Idea Find good features using eigenvalues of second-moment matrix
Key idea: “good” features to track are the ones that can be tracked
reliably.
• Frame-to-frame tracking Track with LK and a pure translation motion model.
More robust for small displacements, can be estimated from smaller
neighborhoods (e.g., 5£5 pixels).
• Checking consistency of tracks Affine registration to the first observed feature instance.
Affine model is more accurate for larger displacements.
Comparing to the first frame helps to minimize drift.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Slide credit: Svetlana Lazebnik
![Page 23: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/23.jpg)
29
Tracking Example
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Slide credit: Svetlana Lazebnik
![Page 24: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/24.jpg)
30
Real-Time GPU Implementations
• This basic feature tracking framework (Lucas-Kanade + Shi-
Tomasi) is commonly referred to as “KLT tracking”. Used as preprocessing step for many applications
Lends itself to easy parallelization
• Very fast GPU implementations available, e.g., C. Zach, D. Gallup, J.-M. Frahm,
Fast Gain-Adaptive KLT tracking on the GPU.
In CVGPU’08 Workshop, Anchorage, USA, 2008
216 fps with automatic gain adaptation
260 fps without gain adaptation
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
http://www.cs.unc.edu/~ssinha/Research/GPU_KLT/
http://www.inf.ethz.ch/personal/chzach/opensource.html
![Page 25: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/25.jpg)
31
Topics of This Lecture
• Recap: Lucas-Kanade Optical Flow Brightness Constancy constraint
LK flow estimation
Coarse-to-fine estimation
• Feature Tracking KLT feature tracking
• Template Tracking LK derivation for templates
Warping functions
General LK image registration
• Applications
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 26: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/26.jpg)
32
Lucas-Kanade Template Tracking
• Traditional LK Typically run on small, corner-like features (e.g., 5£5 patches) to
compute optical flow (! KLT).
However, there is no reason why we can’t use the same approach on a
larger window around the tracked object.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
80£50 pixels
Slide credit: Robert Collins
![Page 27: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/27.jpg)
33
Basic LK Derivation for Templates
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
E(u; v) =Px
£I(x + u; y + v)¡ T(x; y)
¤2
Template model
Current frame
(u,v) = hypothesized
location of template
in current frame
Slide credit: Robert Collins
![Page 28: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/28.jpg)
34
Basic LK Derivation for Templates
• Taylor expansion
• Taking partial derivatives
• Equation in matrix form
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
E(u; v) =Px
£I(x + u; y + v)¡ T(x; y)
¤2
¼Px
£I(x; y) + uIx(x; y) + vIy(x; y)¡T(x; y)
¤2
=Px
£uIx(x; y) + vIy(x; y) + D(x; y)
¤2with
@E@u
= 2Px
£uIx(x; y) + vIy(x; y) + D(x; y)
¤Ix(x; y)
!= 0
@E@v
= 2Px
£uIx(x; y) + vIy(x; y) + D(x; y)
¤Iy(x; y)
!= 0
X
x
·I2x IxIy
IxIy I2y
·̧u
v
¸=X
x
·IxD
IyD
¸
D = I ¡T
Solve via
least-squares
Slide credit: Robert Collins
![Page 29: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/29.jpg)
35
One Problem With This...
• Problematic Assumption Assumption of constant flow (pure translation) for all pixels in a larger
window is unreasonable for long periods of time.
• However... We can easily generalize the LK approach to other 2D parametric
motion models (like affine or projective) by introducing a “warp” function
W with parameters p.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
E(p) =Px
£I(W([x; y];p))¡ T([x; y])
¤2
E(u; v) =Px
£I(x + u; y + v)¡ T(x; y)
¤2
Slide credit: Robert Collins
![Page 30: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/30.jpg)
36
Geometric Image Warping
• The warp W(x; p) describes the geometric relationship
between two images
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Transformed Image Input Image
x’ x
x0 =W(x;p) =
·Wx(x;p)
Wy(x;p)
¸
Parameters of the warp
Slide credit: Jinxiang Chai
![Page 31: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/31.jpg)
37
Example Warping Functions
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
3D rotation
3 unknowns
Slide credit: Steve Seitz
![Page 32: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/32.jpg)
38
Example Warping Functions
• Translation
• Affine
• Perspective
Note: Other parametrizations are possible; the above ones are just
particularly convenient here.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide credit: Jinxiang Chai
W([x; y];p) =
·x + p1x + p3y + p5y + p2x + p4y + p6
¸=
·1 + p1 p3 p5
p2 1 + p4 p6
2̧4
x
y
1
35
W([x; y];p) =
·x + p1y + p2
¸=
·1 0 p10 1 p2
¸24
x
y
1
35
W([x; y];p) =1
p7x + p8y + 1
·x + p1x + p3y + p5y + p2x + p4y + p6
¸
![Page 33: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/33.jpg)
39
General LK Image Registration
• Goal
Find the warping parameters p that minimize the sum-of-squares
intensity difference between the template image and the warped input
image.
• LK formulation Formulate this as an optimization problem
We assume that an initial estimate of p is known and iteratively solve
for increments to the parameters ¢p:
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
argminp
X
x
£I(W(x;p))¡ T (x)
¤2
argmin¢p
X
x
£I(W(x;p+ ¢p))¡ T(x)
¤2
![Page 34: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/34.jpg)
40
Step-by-Step Derivation
• Key to the derivation
Taylor expansion around ¢p
Using pixel coordinates x = [x,y]
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
I(W(x;p+ ¢p)) ¼ I(W(x;p)) +rI@W
@p¢p+O(¢p2)
I(W([x; y];p+ ¢p)) ¼ I(W([x; y]; p1; : : : ; pn))
+
·@I
@x
@Wx
@p1+
@I
@y
@Wy
@p1
¸
p1
¢p1
+
·@I
@x
@Wx
@p2+
@I
@y
@Wy
@p2
¸
p1
¢p2
+ : : :
+
·@I
@x
@Wx
@pn+
@I
@y
@Wy
@pn
¸
pn
¢pn
Slide credit: Robert Collins
![Page 35: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/35.jpg)
41
Step-by-Step Derivation
• Rewriting this in matrix notation
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
I(W([x; y];p+ ¢p)) ¼ I(W([x; y]; p1; : : : ; pn))
+h@I@x
@I@y
i "@Wx
@p1@Wy
@p1
#
p1
¢p1
+h@I@x
@I@y
i "@Wx
@p2@Wy
@p2
#
p2
¢p2
+ : : :
+h@I@x
@I@y
i "@Wx
@pn@Wy
@pn
#
pn
¢pn
Slide credit: Robert Collins
![Page 36: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/36.jpg)
42
Step-by-Step Derivation
• And further collecting the derivative terms
Written in matrix form
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
I(W([x; y];p+ ¢p)) ¼ I(W([x; y]; p1; : : : ; pn))
+h@I@x
@I@y
i24@Wx
@p1
@Wx
@p2: : : @Wx
@pn
@Wy
@p1
@Wy
@p2: : :
@Wy
@pn
35
26664
¢p1¢p2
...
¢pn
37775
Gradient Jacobian Increment
parameters
to solve for
I(W(x;p+ ¢p)) ¼ I(W(x;p)) +rI@W
@p¢p
Slide credit: Robert Collins
![Page 37: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/37.jpg)
43
Example: Jacobian of Affine Warp
• General equation of Jacobian
• Affine warp function (6 parameters)
• Result
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
W([x; y];p) =
·1 + p1 p3 p5
p2 1 + p4 p6
2̧4
x
y
1
35
@W
@p=
24@Wx
@p1
@Wx
@p2: : : @Wx
@pn
@Wy
@p1
@Wy
@p2: : :
@Wy
@pn
35
@W
@p=
@
·x + p1x + p3y + p5p2x + y + p4y + p6
¸
@p
=
·x 0 y 0 1 0
0 x 0 y 0 1
¸
Slide credit: Robert Collins
![Page 38: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/38.jpg)
44
Minimizing the Registration Error
• Optimization function after Taylor expansion
• Minimizing this function How?
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
argmin¢p
X
x
hI(W(x;p)) +rI @W
@p¢p¡ T (x)
i2
![Page 39: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/39.jpg)
45
Minimizing the Registration Error
• Optimization function after Taylor expansion
• Minimizing this function Taking the partial derivative and setting it to zero
Closed-form solution for ¢p (Gauss-Newton):
where H is the Hessian
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
argmin¢p
X
x
hI(W(x;p)) +rI @W
@p¢p¡ T (x)
i2
2X
x
hrI @W
@p
iThI(W(x;p)) +rI @W
@p¢p¡ T (x)
i!= 0
¢p =H¡1X
x
hrI @W
@p
iT£T (x)¡ I(W(x;p))
¤
H =Px
hrI @W
@p
iThrI @W
@p
i
![Page 40: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/40.jpg)
46
Summary: LK Algorithm
• Iterate Warp I to obtain I(W([x, y]; p))
Compute the error image T([x, y]) – I(W([x, y]; p))
Warp the gradient rI with W([x, y]; p)
Evaluate at ([x, y]; p) (Jacobian)
Compute steepest descent images
Compute Hessian matrix
Compute
Compute
Update the parameters p à p + ¢p
• Until ¢p magnitude is negligible
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
H =Px
hrI @W
@p
iThrI @W
@p
i
¢p =H¡1Px
hrI @W
@p
iT£T ([x; y])¡ I(W([x; y];p))
¤
Px
hrI @W
@p
iT£T ([x; y])¡ I(W([x; y];p))
¤
![Page 41: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/41.jpg)
47
LK Algorithm Visualization
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
[S. Baker, I. Matthews, IJCV’04]
![Page 42: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/42.jpg)
48
Discussion LK Alignment
• Pros All pixels get used in matching
Can get sub-pixel accuracy
(important for good mosaicking)
Fast and simple algorithm
Applicable to Optical Flow estimation, stereo disparity estimation,
parametric motion tracking, etc.
• Cons Prone to local minima.
Relatively small movement.
Good initialization necessary
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 43: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/43.jpg)
49
Side Note
• LK Registration needs a good initialization Taylor expansion corresponds to a
linearization around the initial
position p.
This linearization is only valid in a
small neighborhood around p.
• When tracking templates... We typically use the previous frame’s result as initialization.
The higher the frame rate, the smaller the warp will be.
This means we get better results and need fewer LK iterations.
Tracking becomes easier (and faster!) with higher frame rates.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
x x0
![Page 44: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/44.jpg)
50
Discussion
• Beyond 2D Tracking/Registration So far, we focused on registration between 2D images.
The same ideas can be used when performing registration between a
3D model and the 2D image (model-based tracking).
The approach can also be extended for dealing with articulated objects
and for tracking in subspaces.
We will come back to this in later lectures when we talk about
model-based 3D tracking...
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
Slide credit: Jinxiang Chai
![Page 45: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/45.jpg)
51
Topics of This Lecture
• Recap: Lucas-Kanade Optical Flow Brightness Constancy constraint
LK flow estimation
Coarse-to-fine estimation
• Feature Tracking KLT feature tracking
• Template Tracking LK derivation for templates
Warping functions
General LK image registration
• Applications
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 46: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/46.jpg)
52
Example of a More Complex Warping Function
• Encode geometric constraints into region tracking
• Constrained homography transformation model Translation parallel to the ground plane
Rotation around the ground plane normal
W(x) = Wobj P Wt Wα Q x
Input for high-level tracker with car steering model.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler
![Page 47: Computer Vision 2 Lecture 2 - RWTH Aachen University · Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking Prof. Dr. Bastian Leibe, Dr. Jörg Stückler B. Lucas and](https://reader030.vdokument.com/reader030/viewer/2022040612/5edd3f08ad6a402d6668456c/html5/thumbnails/47.jpg)
54
References and Further Reading
• The original paper by Lucas & Kanade B. Lucas and T. Kanade. An iterative image registration technique with
an application to stereo vision. In Proc. IJCAI, pp. 674–679, 1981.
• A more recent paper giving a better explanation S. Baker, I. Matthews. Lucas-Kanade 20 Years On: A Unifying
Framework. In IJCV, Vol. 56(3), pp. 221-255, 2004.
• The original KLT paper by Shi & Tomasi J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Lecture: Computer Vision 2 (SS 2016) – Template-based Tracking
Prof. Dr. Bastian Leibe, Dr. Jörg Stückler