Adaptive Framerates Or How Software Video Players Can Maximize The User Experience

Here is an article I originally wrote for the boxee forum:

I would like to address a feature commonly ignored by all media players I have come across so far. Let's talk about adaptive speed. Regardless if you are living in the USA (NTSC) or Europe (PAL) - you will suffer from video jittering sooner or later. (more after the link)

Let me describe the issue here a bit closer.

Imagine a landscape being filmed while the camera pans slowly from the left side to the right side. If you are watching this scene on a DVD using a standalone DVD Player everything should be smooth and without any jittering. Now if you use this DVD and try to playback that one using a PC you will notice the playback not to be that smooth anymore. So, what is happening here?

Once the DVD Player has determined the frame rate of the source video material it will also set the video output refresh rate to match that speed. For instance, If we are dealing with NTSC material which would be at ~30fps the TV will be advised to use ~60Hz. Each frame in the video source will be displayed on 2 frames rendered on your screen. So, everything is fine and in sync here.

Now, if you head over to your PC and configure your monitor to use the refresh rate of 75 Hz for instance, that very same video would map one frame in the video material to 2.5 Frames on the screen. Depending on the player software used you will notice one of the following artifacts:

  1. The player software does not wait until the frame being rendered on the monitor is finished (VSync) while the advance from the 2nd to the 3rd frame of the source video takes place. What you will see now is called tearing - the image seems to be tearing apart. The player basically mixes two video frames on the screen, one at the upper part of the image and one at the lower.
  2. The player software is aware of the tearing issue and waits until the graphics card has finished rendering the image on the screen before it advances to the next video frame. Here you will notice jittering; the pan will not be smooth and will jump irregularly instead of playing back smoothly. This happens because you cannot divide the 75 frames the screen will display per seconds by the 30 frames per second of the source video without coming up with a decimal fraction. So there is no way to fit the original video into whole frames rendered on the monitor of an equal duration.

Of course this is only one real world example. There are PLENTY of combinations. Usually you will encounter the following frame rates in video files:

  • 23,976fps (Film)
  • 24fps (Film)
  • 29,97 (NTSC TV)
  • 25fps (PAL TV)

Output refresh-rates:

  • ~59.94 Hz (NTSC TV)
  • 50 Hz (PAL TV)
  • 48 Hz (Film, newer HDTVs do that)

Depending on your TV the actual refresh rate might be a multiple of one of these.

So, why should you care and what can be done about it?

Well, If your goal is to get your users the best experience possible - and just be better than all the other software video players out there - you should provide some solutions to this problems.

This is where things can get very messy. Basically it boils down to 2 options which can be used (and perhaps combined) to provide jitter and tearing free playback:

a)      Alter the playback speed of the video and optionally pitch-shift / resample the audio
b)     
Change the output refresh rate according to the video material

Back then, when I was using Media Portal on my HTPC, I decided to fix the output rate to 50 Hz and use ReClock to do what a) refers to. It worked just fine for 24fps material which was sped up to 25fps and most of the 29,97 TV material wasn't really at ~30fps but used the 3:2 pulldown (interlacing the frames using a predefined pattern) technique to get a source video at 23,976 pumped up to 29,76. That way the video would look "OK" on a NTSC TV. This was usually done during the playback and not during the mastering of the video material, so we can still access the "real" frame rate and use that for playback by ignoring the advice in the video file to use a pulldown.

ReClock wasn't exactly stable and 100% reliable - not to mention the development being dead for over than 2 years. But it was OK for a CRT device with a TV-Out capable graphics card. Those will usually not allow any arbitrary output refresh rates other than standard PAL or NTSC. Let's not go into Powerstrip and custom VGA -> RGB Scart adapters here please.

But let's go back to the current situation.

Thankfully the home entertainment setups are moving towards HDTVs very quickly. By getting rid of the TV Encoder chips on the graphics card we can use pretty much very output resolution and refresh rate the TV-Set will support.

So this finally opens the opportunity to a media center software to provide as excellent video playback quality as the standalone devices do - such as my beloved Popcorn Hour. Ok, I must admit I need to change the output refresh rate each time I choose to watch a movie with a different frame rate, but once I did that the viewing experience is awesome.

Until this very date, media center software will provide this feature - at least it does not the last time I checked, which was a couple of months ago. I've been looking at Media Portal, MythTV, Geexbox, XBMC, Freevo ... you name it.

So, please media center programmers! Take this issue into consideration. I know this might be not an easy task; you will have to talk to the graphics hardware directly and implement different routines to alter the output signal based on the hardware / operating system used and go crazy about frame rate detection in the video files. Throw some MPEG2 Parsing for pulldown flag detection and maybe even some über-fancy inverse telecine functionality into it for a good measure.  

Let me know if I can clarify some parts of this article, English is not my native language and some sentence constructs might not make any sense to you. Furthermore I have mastered a DVD including the panning-scene described at the beginning of this article using various frame rates. I used that to test-drive the Meda Center and Media Player solutions. Let me know if I should pass you the download link to that.

References: