Help <a href="http://plumi.org">Plumi</a> reach the promised land of Plone3 That was the former catchline, but it should certainly be changed. A little bit.

About

There is at least two products to integrate video content into Plone sites : Plumi and Plone4Artists. We will try to understand how they work, what they can do and what they cannot, and pinpoint the things which are major showstoppers for real-world use.

Plumi under the hood

Plumi provides a new content type, ATVideo. When the user creates a ATVideo element, she has to provide the video file (in any supported format, like MPEG, AVI...) and a few metadata (production date, length, and a lot of optional informations about de video content). The uploaded video file lands in a directory which is "watched" by an external process, Indytube. As soon as Indytube notices a new file, it tries to transcode it, using ffmpeg2theora and/or mencoder. The resulting file(s) are put in another directory, which itself should be served by a static HTTP server. Alongside with those encoded files, small HTML snippets are created. Those HTML snippets contain markup to embed a video player : either Flash-based Flowplayer (to play FLV files), or Java-based Cortado (to play Theora files). The ATVideo default view will include those HTML snippet on-the-fly.

Plone4Artists

(This section needs to be completed and improved, since P4A Video support was not thoroughly tested during the sprint)

Plone4Artists takes a different approach : the user uploads regular files to her Plone site ; and P4A provides adapters to enrich those files with views and metadata.

Integrating Plone and video content : issues and challenges

To help everyone to grasp the problems which have to be dealt with when tackling video content integration, let's do a quick parallel with picture integration.

Here are the required steps to put a picture on a web site (Plone or anything else) :

1. Create your source image (take a picture with a camera, draw it on a sheet of paper and scan it, etc.)
2. Convert it to an appropriate file format (e.g. PNG for small icons and sketches ; JPEG for other things) and, if required, scale and crop it to the appropriate size
   
3. Upload the file to your web site, and store it
   
4. Integrate the picture (that is : arrange things so it will display on the right place on the right page)
5. Serve the content (so that your visitors' browsers can download and display the image)
6. Enhance your visitors' experience, providing keywords, metadata and thumbnails (those features can be totally irrelevant in some cases : for instance, icons generally don't need metadata and thumbnails ; but photo galleries will require all of them)
   

 Now, compare each step with the corresponding action with video content :

1. Create your video content and acquire it to some digital file format (DV, AVI, MPEG, whatever) - Plone won't help you with that, nor will any CMS ;-)
2. Convert the video to a format better suited to web operation (heated debate can occur there ; so let's say we want at least FLV and Theora) - This is done by Indytube after the file upload, but could be greatly improved
3. Upload the video to your web site and store it - this is much more complicated than image upload, for two reasons : first, it is much, much longer than image upload, since the files are much, much bigger ; and then, Zope has a limit of 4 concurrent threads, meaning that 4 simultaneous uploads will hog your server and block regular page requests ; also storage in ZODB is out of question for performance and maintainability reasons
   
4. Integrate the video - this will generally make use of a Flash-based player, or another embedding technology, since no common browser is able to play video files natively
5. Stream the video to your visitors - this was not a problem with images, but raises some nasty performance issues for video files, some of which were already mentioned for upload
   
6. Enhance visitor experience with thumbnails (snapshots), low- and high- bitrate versions of the content, as well as metadata and keywords

Note : steps 2 and 3 could actually be swapped, since it is not uncommon to do the conversion (or transcoding) on the server. Transcoding locally leads to shorter upload times (since you upload a low-bandwidth version of your video file), but it can be quite complicated to setup ffmpeg, mencoder, codecs, and all that stuff ; so a lot of people will be thankful if we allow server-side transcoding.

Major showstoppers and how to address them

Now that we have described the main workflow for video content, let's pinpoint the areas where we clearly need some improvements to get an acceptable solution.

File upload

Problem : when uploading a file to Zope, the user does not get any feedback, and she hogs one of the 4 precious threads. That means that if you upload 4 videos simultaneously, your Zope will stop answering to other requests until the end of the uploads.

Solutions :

• Patch Zope core to allow more threads.
  Pros : sounds great.
  Cons : sounds a great deal of work, too.
• Use an external file upload method, based on existing software (like ProFTP or mod_dav).
  Pros : can be über-efficient ; can be outside of Zope (on another computer if needed).
  Cons : needs extra software ; needs some development to implement Zope authentication ; complicated to use for your average Joe user.
• Put a reverse proxy in front of Zope, to handle (some) HTTP POST requests.
  Pros : mostly transparent for the user ; can be (quite) efficient
  Cons : needs extra software ; needs additional development - or not !

JNut offered his httpwizard daemon, which does exactly the 3rd solution, using EventPush and KSS to provide additional visual feedback to the user.

We need to :

• simplify httpwizard installation and configuration, so it can be deployed easily for any Plone site (we can use a buildout recipe for that) ;
• polish some rough edges while we're at it ;
• understand how to integrate it with our own content types (if we chose to go that way).
  

File serving

Problem : similar to the file upload problem, due to Zope thread limitation.

Solutions :

• Arrange things so that video content is stored as plain files, served by any static HTTP server.
  Pros : very easy to setup, very efficient.
  Cons : no authentication ; every content is available to any user.
• Use a general-purpose web server (or a home-made one), allowing to hook an access checking function before actually serving the content. The access checking function should relay the authentication cookie to the Plone site to check whether the user as the required access rights (roles) to access the content.
  Pros : not too difficult to setup, quite efficient.
  Cons : need some development, a bit harder to setup.

The first solution would defeat all access checking ; so we prefer thinking about the 2nd one. Many variants are possible ; just to list a few :

• use a "prg" RewriteMap in Apache, to send URL + cookies to a small rewrite script which will delegate access checking to Zope ;
• implement a custom auth handler for Apache ;
• implement a custom HTTP server to do the job ;
• cache authentication information (so that the call to Zope is done only when starting the streaming, and not on subsequent seeks, if any) ;
  
• ...

It's useful to note that this component is not essential to get a working solution : using external fileserving looks the same from the Zope+Plone point of view, and authentication can be added later without changes to the Zope products themselves.

Metadata handling

Problem : there is either too much, or too few, metadata.

When one uses Plumi, with its ATVideo content type, she has to specify a lot of mandatory fields, which will just annoy some users.

When one uses P4A, it adapts the default File content type, so you basically have no metadata.

We can arbitrarily split metadata in a few parts :

• fundamental metadata, which will always be mandatory ; should there be anything apart from the title ?
• technical metadata, which should be automatically infered (using tools like hachoir, or libavcodec/libavformat) ; like resolution, frames per second, duration, average bitrate, number of audio tracks, codec, etc. this kind of metadata is linked to the file itself
  
• "human generated" metadata, like description, production date, copyright informations, etc. ; each application will need a different set of metadata, so we need flexibility here ; also, some fields won't need a translation (like production date), some will need .po-based translation (like "genre") and some will need a full-blown translation engine (like "synopsis")

One appealing idea was to use a folderish item, in which we would store each version (different bitrates etc.) as different a file ; the metadata would then be held in a separate object (allowing to use different objects for different languages).