Tuesday, March 29, 2011

Sneak Preview: 3D Video Support in the Silverlight Media Framework

I'm excited to announce that the upcoming 2.5 release of the Silverlight Media Framework (also known as the Media Platform Player Framework) will include support for playback of stereoscopic 3D video content. This powerful new functionality will allow developers to easily integrate 3D video content into their player experiences, and will allow dynamically changing, enabling and disabling 3D mode based on an end-user's preferences.

Content providers can now provide a single encode of a stream, which the player can then dynamically display in a range of different 3D modes or standard 2D mode based on the user's preferences (and hardware/3D glasses availability). This dynamism provides a powerful tool for online broadcast of 3D content in that it does not leave any users out in the cold, unable to consume the content. You can now reach your entire user base, with 3D as an optional feature, with only a single video source!
Here's a sneak preview of what will be included, as well as some background on how it all works:

What's Included
  • Anaglyph Stereoscopic 3D video plugin, packaged with the SMF 2.5 source code
    • Support for Anaglyph 3D, Greyscale 3D, Left eye only modes
  • NVidia 3DVision Active Shutter Stereoscopic 3D plugin available for separate download
  • Support for 3rd party Stereoscopic 3D plugins
    • Tools to build your own Stereoscopic 3d implementation
  • Simple implementation of 3D support for consuming applications:
    • A new "S3DProperties" property has been added to the PlaylistItem class
    • Simply set up the appropriate S3D Properties, add a reference to the S3D plugin you'd like to use, and 3D will work automatically!
  • Stereoscopic 3D supported for:
    • Progressive Download (WMV, MP4)
    • Smooth Streaming
  • Four new sample applications will be included in the Samples project to demonstrate how to use the Anaglyph 3D plugin:
    • Simple Anaglyph 3D example
    • Multi-mode Anaglyph 3D example (toggling 3D on and off, changing modes)
    • 3D support via HTML playlists
    • 3D support via JavaScript playlists
  • Full documentation will be available on http://smf.codeplex.com when SMF 2.5 is released
Building Your Own S3D SMF-based Video Player
The documentation to be released with SMF 2.5 will include full instructions on how to include 3D video content in your player, complete with sample applications. For a peek at how easy the setup will be, here's a quick how-to video: http://ecn.channel9.msdn.com/o9/content/smf/howto/v2/videos/smf3d.wmv
The code generated during this video will also be available on Codeplex on the release date.

Writing your own S3D Plugin
The SMF 2.5 documentation will also include details on how to build a Stereoscopic 3D plugin of your own that will integrate with the new I3DPlugin interface.

How Does Stereoscopic 3D Work? 
Three of the most widely used 3D video display technologies are passive complementary color anaglyph (just referred to as "anaglyph" in this post), passive polarized, and active liquid crystal shutter, each with pros and cons. Let's take a quick look at all three, and discuss how they work in the Silverlight Media Framework:

The anaglyph approach to 3D uses two complementary colors as a filter for each eye. Most often, red and cyan are the colors used, and these are the colors currently supported by the SMF's Anaglyph S3D plugin. An anaglyph image (below) is displayed such that the left eye will view only red colors in the spectrum, and the right eye will view only blue and green colors (which combine to make cyan) to create the 3D effect.

The SMF's anaglyph plugin creates an anaglyph image by taking a base video source with the left eye and right eye images side by side or top and bottom, and uses Silverlight pixel shaders to apply color filters to the images and superimpose them together to form a single image. We based the red-cyan anaglyph formula on Peter Wimmer's anaglyph forumula – we used the ITU BT.709 color matrix coefficients instead of the ITU BT.601 coefficients used by Peter.
  • No special hardware is required to display the image
  • The glasses are very cheap (paper glasses can be purchased for less than 50 cents online)
  • The technology has very wide reach, and makes it the most common 3D display technology for images on the internet.
The major disadvantages to anaglyph display are around the quality of the image being viewed:
  • The left and right images are displayed superimposed on top of each other.
  • Due to the color filters, color resolution is lost. One technique to counter the color loss is to display a grayscale or monochrome anaglyph image: a colorless image suffers no color loss. The anaglyph plugin in the SMF supports a grayscale option – the pixel shaders have a modified algorithm that will turn a video stream with color images into a monochrome image before applying the red-cyan color filter.


You probably experienced polarized 3D the last time you saw a 3D movie in a theater. Linear polarization (as used by IMAX), and circular polarization (as used by RealD) work in a similar manner: the left and right eye images are superimposed through polarizing filters. In linear polarization, the polarized lenses in the glasses are oriented 90 degrees offset from each other (one vertically, and one horizontally), and in circular polarization one of the lenses is mounted in reverse (the advantage being that you can tilt your head and still maintain left/right separation).

The Silverlight Media Framework does not currently have support for polarized 3D, since most users don't have interlaced polarized displays (LCD computer monitors such as this one). However, 3rd party developers are free to create a 3D plugin for interlaced polarized displays for the SMF. The approach would involve using pixel shaders to convert the frame compatible images into interlaced images.

  • The color loss with anaglyph is eliminated using polarization, and the technology works well on the "silver screen".
  • This achieves a superior image quality vs. anaglyph, though there is still resolution loss.
  • The glasses are more expensive than anaglyph glasses, and special projection hardware is required to make this work.
  • This limits the versatility and reach of this technology, particularly with online video!
  • Polarization only works with silver screen projection (such as in a theater) or interlaced polarized digital displays. Silver screens are inconvenient for home use and interlaced polarized displays are very rare.
Active Shutter

The anaglyph and polarized approaches are considered "passive" because to the glasses themselves don't have any active pieces. In contrast, Active LCD shutter technology uses powered glasses that filter each eye differently at different times. NVidia's 3D Vision solution uses a 120Hz monitor to present the left and right eye images on alternating frames, while still maintaining a full 60Hz signal per eye. The glasses shut out the image to the left and right eye every 1/120th of a second, syncing with the image on the monitor via an IR transmitter.
NVidia worked with Vertigo to create an active shutter 3D Vision plugin for the Silverlight Media Framework, which will be available on NVidia's site.
  • Full resolution per eye, since the images are not superimposed.
    • This allows full 1080p HD images to be viewed without any resolution loss, creating great looking video.
  • No restrictions on viewing angle
  • Works on any PC with a recent (~5 years old or younger) NVidia GPU and 120Hz monitor and an NVIDIA 3D Vision IR emitter
  • Powered glasses are more expensive than polarized or anaglyph
Personally, I have to say that the NVidia Active Shutter technology is the most stunning 3D experience I've viewed so far – 3D video looks so much better in full resolution HD!

Sample 3D Video Content
Finally, Microsoft and NVidia have made the following sample video content available for developers:

Have fun
3D is exploding onto consumer devices like the Nintendo 3DS, and I expect to see a big jump in 3D video online over the upcoming years. I'm very much looking forward to seeing the applications consumers of the Silverlight Media Framework create using this new addition to the SMF!

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.