ScNat
In Cam Audio Lag

Subframe audio video sync measurement

I designed and realized an apparatus to measure the synchronisation between the in cam audio stream and the video stream as recorded on the camera memory card to assess the performance of syncing programs.

In any NLE software, one can verify that sound is in sync with the images, stepping frame by frame. But how to measure synchronization correctness within a precision better than a frame duration? This method successfully determined the sync error of some consumer models within a 2 milliseconds precision:

• Testing a Panasonic Lumix G85, sound leads by 3 ± 2 ms (for 32 ms frames)
• Testing a FUJIFILM X-E1, sound leads by 24 ± 2 ms (for 42 ms frames)

Experimental Setup

A mechanical slate is used to generates events filmed by the camera. The measurement must be done multiple times in order to evaluate its reproducibility, hence a 30 RPM rotating arm is used as a slate and recorded for a minute or two, generating some 30-60 clicking events, all processed to generate valid statistical distributions.

No sound is produced: an electrical contact sends for each turn a 100 mV pulse directly into the camera microphone input. Before running the experiment, a calibration static shot is done to record the rotating arm reference position, exactly at the angular position where the electrical click is occurring.

Principle of operation

Successive arm positions are used for a linear regression to interpolate the precise time when the arm passes through the reference position, this time (called video time, $T_{v}$) is compared to the time detected for the sound pulse in the audio recording ($T_{a}$, audio time). Ideally those time should coincide, and the sync error

should be zero, or at the very least, much less than a frame duration. If $t_{a} \lt t_{v}$, the audio click happens before the arm reaches the reference position so the audio is leading. Hence cameras with lagging audio have a positive $\Delta t$ and leading audio will have a $\Delta t \lt 0$.

Automated image and sound processing

About ten frames are used for each pass (five frames before the click and five after) and for each frame the angular arm position has to be measured with the best available spatial precision (at pixel level). Manually the whole process would be tedious so the video frames and sound track are analyzed with a python script. To simplify image processing, high contrast videos are obtained in complete obscurity with a lightly lit LED fixed on the rotating arm. This LED produces an arc of circle due to motion blur. At this low light, the camera sets the maximum shutter exposition length: one full frame (almost a 360 degrees shutter angle) and successive arcs almost touch each other.

The steps are listed below (after manually finding the $(x,y)$ reference position of the LED in the ‘calibration’ static sequence):

• Find the time $T_{a}$ at which in the sound track the (first) click occurs
• With this time $T_{a}$, load in memory some frames around that instant
• For each loaded image, detect the $(x,y)$ position of the arc extremity
• With this set of extremities, determine the centre of rotation.
• Convert each $(x,y)$ positions in their corresponding angular position $\theta$ (with the click reference angle set as origin $\theta=0$)
• Using frame numbers and the clip beginning as $t_{v}=0$, for each angle $\theta$ calculate the corresponding video time $t_{v}$
• With this set of points $(\theta, t_{v})$ fit a straight line (angular velocity is constant)
• The x-intercept of the fitted line is $T_{v}$ the time of the click determined with video frames.
• Compute and store the synchronization error $\Delta t = T_{a} - T_{v}$
• Wash, rinse and repeat until the end of the clip.