Interest Operators Find “interesting” pieces of the image

Interest Operators

• Find “interesting” pieces of the image

• e.g. corners, salient regions
• Focus attention of algorithms
• Speed up computation

• Many possible uses in matching/recognition

• Search
• Object recognition
• Image alignment & stitching
• Stereo
• Tracking
• …

Interest points

Local invariant photometric descriptors

History - Matching

• 1. Matching based on correlation alone

• 2. Matching based on geometric primitives

• e.g. line segments

• Not very discriminating (why?)

• Solution : matching with interest points & correlation

• [ A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry,

• Z. Zhang, R. Deriche, O. Faugeras and Q. Luong,

• Artificial Intelligence 1995 ]

Approach

• Extraction of interest points with the Harris detector

• Comparison of points with cross-correlation

• Verification with the fundamental matrix

Harris detector

Harris detector

Harris detector

Cross-correlation matching

Global constraints

Summary of the approach

• Very good results in the presence of occlusion and clutter

• local information
• discriminant greyvalue information
• robust estimation of the global relation between images
• works well for limited view point changes

• Solution for more general view point changes

• wide baseline matching (different viewpoint, scale and rotation)
• local invariant descriptors based on greyvalue information

Invariant Features

• Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002

Approach

• 1) Extraction of interest points (characteristic locations)

• 2) Computation of local descriptors (rotational invariants)

• 3) Determining correspondences

• 4) Selection of similar images

Harris detector

Autocorrelation

Autocorrelation

Background: Moravec Corner Detector

Shortcomings of Moravec Operator

• Only tries 4 shifts. We’d like to consider “all” shifts.

• Uses a discrete rectangular window. We’d like to use a smooth circular (or later elliptical) window.

• Uses a simple min function. We’d like to characterize variation with respect to direction.

Harris detector

Harris detector

Harris detector

Harris detection

• Auto-correlation matrix

• captures the structure of the local neighborhood
• measure based on eigenvalues of M
• 2 strong eigenvalues => interest point
• 1 strong eigenvalue => contour
• 0 eigenvalue => uniform region

• Interest point detection

• threshold on the eigenvalues
• local maximum for localization

Some Details from the Harris Paper

• Corner strength R = Det(M) – k Tr(M)2

• Let  and  be the two eigenvalues

• Tr(M) =  + 

• Det(M) = 

• R is positive for corners, - for edges, and small for flat regions

• Select corner pixels that are 8-way local maxima

Determining correspondences

Distance Measures

• We can use the sum-square difference of the values of the pixels in a square neighborhood about the points being compared.

Some Matching Results

Some Matching Results

Summary of the approach

• Basic feature matching = Harris Corners & Correlation

• Very good results in the presence of occlusion and clutter

• local information
• discriminant greyvalue information
• invariance to image rotation and illumination

• Not invariance to scale and affine changes

• Solution for more general view point changes

• local invariant descriptors to scale and rotation
• extraction of invariant points and regions

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Rotation/Scale Invariance

Invariant Features

• Local image descriptors that are invariant (unchanged) under image transformations

Canonical Frames

Canonical Frames

Multi-Scale Oriented Patches

Multi-Scale Oriented Patches

• Sample scaled, oriented patch

Multi-Scale Oriented Patches

• Sample scaled, oriented patch

• 8x8 patch, sampled at 5 x scale

Multi-Scale Oriented Patches

• Sample scaled, oriented patch

• 8x8 patch, sampled at 5 x scale

• Bias/gain normalised

• I’ = (I – )/

Matching Interest Points: Summary

• Harris corners / correlation

• Extract and match repeatable image features
• Robust to clutter and occlusion
• BUT not invariant to scale and rotation

• Multi-Scale Oriented Patches

• Corners detected at multiple scales
• Descriptors oriented using local gradient
• Also, sample a blurred image patch
• Invariant to scale and rotation