11. Motion Tracking

In this lab exercise you will look at normalized cross correlation for motion tracking. For the theory we refer to the lecture notes.

11.1. Tracking an Object

In the following video you see a ball following the characteristic parabolic path.

The Python code that follows downloads the video for you but in case you have a need to do it yourself you can. The video can be downloaded by right clicking on the video and selecting “Save video as …”.

The following piece of code will play the video in a matplotlib figure. Note that matplotlib is not meant for this type of display. The rendering pipeline in matplotlib is quite complex taking care of different axes in windows, different scalings etc etc. So don’t expect real time performance.

#
# mp4 video player
#
# (i wouldn't use it to watch 'Guardians of the Galaxy')
#
from os.path import isfile
import wget
import imageio
from pprint import pprint

plt.ion()

videofile = 'ball2.mp4'
if not isfile(videofile):
    print('Downloading the video')
    wget.download('http://staff.science.uva.nl/R.vandenBoomgaard/IPCV20162017/_static/ball2.mp4')
reader = imageio.get_reader(videofile)

pprint(reader.get_meta_data())

img = None
txt = None
prev_im = None
for i in range(40,400,4):
    im = reader.get_data(i)
    if img is None:
        img = plt.imshow(im)
    else:
        img.set_data(im)
   
    plt.pause(0.01)
    plt.draw()
    if prev_im is not None:
        print((im==prev_im).all())
    
    prev_im = im.copy()

Reading video’s is not a trivial task. There are many different video formats (encodings) and several file formats (yes these are two different things…). OpenCV can read mp4 video files but the OpenCV distributions for Linux do not always have the necessary code compiled. For this reason i have downloaded and installed imageio. This package needs ffmpeg or libav installed. I have ffmpeg installed.

11.1.1. Exercise 1

Write a tracker using the normalized cross correlation that is available in skimage. In skimage it is called match_template, its documentation can be found here.

  1. Manually select a template in frame 40 enclosing the ball.
  2. Use that template to find the ball in subsequent frames.
  3. With a marker (the scatter command in matplotlib) mark the position of the ball.

The end result should be an image showing the last frame and all markers drawn showing the patch the ball followed in its flight.

11.1.2. Exercise 2

The template matcher from skimage is also based on the article J.P. Lewis, “Fast Normalized Cross-Correlation”, just like the text in the lecture notes. Although the resulting expressions are a bit different (the results the same if all is well…).

In this exercise you have to implement match_template yourself based on the formula’s in the lecture notes.

11.2. Egomotion: Simulating an Optical Mouse

An optical mouse contains a camera that looks at the surface it is moving on. The resolution of the camera is quite low but because of that the frame rate is quite high. Therefore when moving the mouse, frame i and frame i+1 have a significant overlap in area that is depicted.

The video below shows what a mouse sees when moving over a table surface (this is a simulation!)

11.2.1. Exercise 3

You have to write a program that calculates what the path is that the mouse travelled. Assume that in the first frame the center of the image is at position (0,0). Pick a subset of image in frame 0 and use that as a template to look for in frame 1. The position where you found that in frame 1 is indicative of the movement of the mouse. Then you select a template in frame 1 and look in frame 2 where it has moved to. Etc, etc.

The result is a list of (dx,dy) tuples indicating the movement from frame i to frame i+1.

Assuming a starting position you can calculate the list of positions (x,y) for all frames. Make a plot of the path. At every 10th frame you have to plot the frame index at the associated position (this helps us in determining whether you have found the right path).

In this exercise you can either use your own cross correlation template matcher or the one from skimage.