Camera Stabilization Paint Trail

We have implemented video trails to further help body detection. These trails are meant to retain the background as our image moves out of the ROI as a direct result of camera stabilization. In the previous post, we have camera stabilization without the video trails, in the following two videos, we have implemented video trails. The first video has the optical flow vectors showing. The second video doesn't have the optical flow vectors. You can see at various points of the video when the frame is moving out of the ROI, the background information is retained.

Optical Flow and Padding. Possible Shot Boundary Detection

Result of Optical Flow Program:
-Red vectors for optical flow vectors
-Green vectors for median optical flow vector


Input 1



Ouput 1





Input 2


Output 2



Output 2.1








We have also been able to zero pad the whole video, however we had some trouble outputting it to a video so here is a screenshot:










We've also noticed that we might be able to detect shot boundaries with precision. For every frame where we calculate the median optical flow vector, we noticed that at every actual shot boundary there is a large jump of the median optical flow vector values. So we are going to investigate this precision more and compare it to the shot detect software that we were previously using. We then will decide what would be the best approach to the problem. Here is an example of the output of the median vectors:

#### -1,2 ####
#### -1,1 ####
#### -1,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,3 ####
#### -1,3 ####
#### -1,2 ####
#### -1,1 ####
#### -1,1 ####
#### -1,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,1 ####
#### -1,1 ####
#### 0,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,2 ####
#### -1,1 ####
#### -1,1 ####
#### 0,1 ####
#### 0,1 ####
#### 842150451,842150451 #### //Corresponded to an actual shot boundary
#### -3,0 ####
#### -2,0 ####
#### -1,0 ####
#### 0,0 ####
#### -1,1 ####
#### -1,1 ####
#### -1,1 ####
#### -1,0 ####
#### -1,1 ####
#### -3,1 ####
#### -2,1 ####
#### -2,1 ####
#### -2,1 ####
#### -3,1 ####
#### -2,1 ####
#### -1,1 ####
#### -1,1 ####
#### -1,1 ####
#### 0,0 ####
#### 0,1 ####
#### 0,1 ####
#### 0,1 ####
#### 0,1 ####
#### 0,0 ####
#### -1,0 ####
#### -1,0 ####
#### -1,0 ####
#### -1,0 ####
#### -2,0 ####
#### -2,0 ####
#### -1,0 ####
#### -2,0 ####
#### -2,0 ####
#### -1,1 ####
#### 0,1 ####
#### 0,1 ####
#### -1,1 ####
#### -1,1 ####
#### -1,2 ####
#### -2,2 ####
#### -3,1 ####
#### -3,0 ####
#### -5,0 ####
#### -3,0 ####
#### -2,0 ####
#### 0,0 ####
#### -1,0 ####
#### -1,0 ####
#### 0,1 ####
#### 0,1 ####
#### 0,0 ####
#### 0,0 ####
#### 0,0 ####
#### 0,0 ####
#### -1,1 ####
#### -1,0 ####
#### -1,0 ####
#### -2,0 ####
#### -1,0 ####
#### 0,1 ####
#### 0,1 ####
#### 0,1 ####
#### 1,1 ####
#### 1,1 ####
#### 1,1 ####
#### 1,0 ####
#### 0,2 ####
#### 0,1 ####
#### 0,1 ####
#### -1,2 ####
#### -1,1 ####
#### -1,2 ####
#### -2,1 ####
#### -2,0 ####
#### -3,0 ####
#### -2,0 ####
#### -2,0 ####
#### -3,0 ####
#### -3,-1 ####
#### -3,0 ####
#### -2,1 ####
#### -4,2 ####
#### -2,2 ####
#### -2,2 ####
#### -4,2 ####
#### -3,2 ####
#### -5,2 ####
#### -3,1 ####
#### -3,0 ####
#### -4,0 ####
#### -5,1 ####
#### -3,1 ####
#### -3,1 ####
#### -3,1 ####
#### -2,2 ####
#### -3,2 ####
#### -1,2 ####
#### -1,1 ####
#### -2,1 ####
#### -3,0 ####
#### -6,0 ####
#### -6,0 ####
#### -5,0 ####
#### -5,0 ####
#### -4,0 ####
#### -6,1 ####
#### -3,1 ####
#### -2,1 ####
#### -2,1 ####
#### -2,1 ####
#### -4,2 ####
#### -4,1 ####
#### -4,1 ####
#### -5,2 ####
#### -10,3 ####
#### -5,1 ####
#### -4,1 ####
#### -4,1 ####
#### -3,1 ####
#### -1,1 ####
#### 0,2 ####
#### -1,1 ####
#### -1,1 ####
#### -2,1 ####
#### -1,1 ####
#### -2,1 ####
#### 842150451,842150451 #### //Corresponded to an actual shot boundary
#### -4,2 ####
#### -3,1 ####
#### 0,0 ####
#### 3,-2 ####
#### 2,-2 ####
#### 1,-1 ####
#### 0,0 ####
#### 5,3 ####
#### 2,2 ####
#### 1,2 ####
#### -1,3 ####
#### -1,1 ####

Shot Detect Performance

We are still having problems with compiling the Shot Detect software. For now, we will use the shot detect executable to get an idea of how well it performs. We ran shot detect on 4 videos from our dataset. Eventually we will implement this performance check on our entire Dataset. The results are as follows...

We define Error as (Undetected Shots + False Positives) / Total # of Shots

Song 1 -
52 shots
Undetected Shots - 5
False Positives - 0
Error - .096

Song 2 -
57 shots
Undetected Shots - 5
False Positives - 0
Error - .087

Song 3 -
49 shots
Undetected Shots - 2
False Positives - 4
Error - .122

Song 4 -
27 shots
Undetected Shots - 10
False Positives - 2
Error - .444

The nature of the first three songs is such that the shot detection is very obvious to the human eye. Song 4 isn't so obvious, i.e. at times, there are effects like dissolve and fade between shots. These sorts of transition between shots are not detected by the shot detect program.

Plan for Moving Forward...

Our project is two fold: 1. implement shot boundary detection, 2. implement human-dancer detection.

1. We are using free shotdetect software by Johan Mathe.
http://shotdetect.nonutc.fr/

This software takes AVI files as input and outputs two things of interest to us:
- XML file containing result of the analysis.
- Thumbs of the first and last image of the detected scene

We have an executable shot detect, and we also have the source code for the software. We are looking to change the shot detect software so that it may input one of two things:
- AVI files for each shot.
- If this proves too hard to accomplish, we would like shotdetect to output a text file containing results of the analysis. This text file will then be used as input fcutter, an open source command line program, that will cut our full song-dance sequence into shots.

2. Implementing Human Detection.
For human body detection, we are making use of Amy Alexander's SVEN - Surveillance Video Entertainment Network.
http://deprogramming.us/sven/index.html

One of the problems with SVEN is that it only works well if the background is stable. We will look to change this feature so that human detection is possible even with moving backgrounds.