Aggregating video feeds

The major new feature of OpenPackage Video 5.x-3.8 is integration with Feed Element Mapper (feedapi_mapper).

The development of this functionality was sponsored by Hudson Street Media using our bounty system:
http://openpackage.biz/bounty/15
http://openpackage.biz/bounty/14

Feed Element Mapper is an add-on module for FeedAPI that maps a feed's elements such as tags or (in this case) videos to taxonomy or CCK fields on your site.

This enables your site to aggregate videos from other sites.

I have set up various video feeds on our site as a demonstration. Click on the 'Feed item' links to view the videos.

The following instructions assume that you have already setup a content type with a op_video field and that your site has been configured to transcode. Your php installation needs to have cURL support.

1. Install feedapi and feedapi_mapper on your site.
2. Download SimplePie and copy the file simplepie.inc from the archive to feedapi/parser_simplepie. It is necessary to use SimplePie parser, as Common syndication parser does not support enclosures.
3. Enable the following modules:
   • FeedAPI Mapper
   • FeedAPI Node Views
   • FeedAPI
   • FeedAPI Node
   • SimplePie parser
4. Check admin/logs/status to ensure that SimplePie has been installed correctly.
5. Go to node/add/feedapi-node. Scroll down and enter your video feed url, e.g. http://www.lullabot.com/videocast/feed.
6. Do not select 'Update existing feed items' as this will cause all the videos in the feed to be retranscoded every time the feed changes.
7. Click on 'Processors' and select your node type that has an op_video field.
8. Click 'Submit' and then click on the 'Map' tab.
9. Scroll down to 'options->enclosures->video' and select the op_video option.
10. Scroll down to the bottom and click 'Update', then click the 'Refresh' tab.
11. Click on the 'Feed items' link - the videos have not yet been transcoded.
12. Run cron.php / wait for it to run and the videos will appear.

FFmpeg info

To transcode your videos locally on your server, OpenPackage Video uses FFmpeg. This is free software, like Drupal.

If you are using the transcoder at openpackage.biz, then you do not require FFmpeg.

Some of the technology in FFmpeg is covered by software patents. Most countries do not recognise software patents, the notable exception being the USA. Check out this video.

If you use FFmpeg for commercial purposes in a country that does recognise software patents you are exposing yourself to the risk of litigation. Here is what FFmpeg have to say on the matter.

However, if you are a private user using FFmpeg for your own personal purposes, then there is nothing to worry about.

If you are using OpenPackage Video in a commercial manner in a country that recognises software patents, we recommend you outsource the transcoding to us. This frees you from the risk of litigation and also offers the advantage of not having to worry about FFmpeg maxing out your web server. Also, it is not usually possible to have FFmpeg installed in a basic web hosting package.

Installation

These instructions are specific to unix systems like Linux. For FFmpeg under Microsoft Windows see this site. We, of course, recommend you use OpenPackage Video (and Drupal in general) on a unix system.

The important thing when installing FFmpeg is ensuring that it supports the codecs you require. The FFmpeg packaged with your distro may have poor codec support (e.g. Debian / Ubuntu's FFmpeg doesn't support mp3).

To ensure you have all the codecs you require we recommend you compile the latest version of FFmpeg.

You need subversion installed. Use your distro's package manager to install it. Then checkout FFmpeg from its repository:

svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
cd ffmpeg

You then need to configure FFmpeg. Below is the configure command we recommend. However, it may detect that you do not have all the required libraries in place. In this case, either install them with your distro's package manager or remove them from the configure command. If you are using Fedora, you may wish to try Livna.

./configure --prefix=/usr --enable-static --enable-shared --enable-gpl --enable-postproc --enable-swscale --enable-pthreads --enable-liba52 --enable-libamr-nb --enable-libamr-wb --enable-libfaac --enable-libfaad --enable-libmp3lame --enable-libx264 --disable-ffserver --disable-ffplay --disable-debug --disable-network --enable-nonfree --enable-avfilter --enable-avfilter-lavf --disable-devices --disable-vhook

You then need to build FFmpeg and install it:

make
sudo make install

Smart transcoding

OpenPackage Video uses a technique we call 'smart transcoding'. This involves interrogating the file for information and utilising this information to improve the quality of the output.

• Transcoding always reduces quality to some extent because compression artifacts are cumulative. For both the audio and video streams, there are conditions that when met, allow the stream to 'passthrough' without being transcoded.
  • Audio stream
    • target codec is the same as source
    • target bitrate is greater than or equal to source
    • target channel count is greater than or equal to source
    • target sample rate is greater than or equal to source
  • Video stream
    • target codec is the same as source
    • target bitrate is greater than or equal to source
    • target width is greater than or equal to source
    • target frame rate is greater than or equal to source
• If video transcoding does occur, some modifications are made to the transcoding parameters to improve the result:
  • Videos are never upscaled - this would only reduce quality and bitrate value.
  • The frame rate is never increased (unless the codec demands it) - this only serves to make the video less smooth and can also reduce bitrate value.

Using our remote transcoding service from a shared web host

Video transcoding at a shared web host presents a few challenges due to resource usage restrictions put in place by any sane/competent host. Even if you were able to compile FFmpeg on the server, you'd likely exceed CPU limits for your user account if you tried to use it. Alternatively, when using our remote transcoding services, you'll likely encounter limitations on the execution time of your scripts (i.e. Drupal's cron.php) and allowable file upload sizes.

Pair Networks

Here are some notes from one of our users about getting our remote transcoding service working on a Pair Networks shared server:

• Pair's relevant usage policies:
  • http://www.pair.com/policies/resource.html
  • http://www.pair.com/policies/cgiresource.html
• For security reasons, i'm running PHP under Pair's implementation of php-cgiwrap:
  http://www.pair.com/support/knowledge_base/authoring_development/system_...
• To transfer the videos to and from openpackage.biz for the transcoding, you need to adjust your php.ini settings per http://www.radinks.com/upload/config.php
• with php-cgiwrap, i can customize my own php.ini file in
  ~/public_html/cgi-bin/ (hypothetically, you could do this via .htaccess, but i didn't test that)
• Running cron once a video is queued for transcoding:
  • running cron from CLI fails in less than a minute with: Killed: 9
  • with Pair's default php.ini settings, cron would die in the web browser and dump a long list of mysql errors to the screen after running for a few minutes
  • even if the php.ini settings were increased ridiculously high, cron would die in the web browser with an Apache 500 error after running a minute or two longer than it did with more restrictive php.ini settings (presumably due to the Apache timeout directive)
• in the end, after trial and error, i found that the largest file i could transfer for transcoding was just under 30MB.
• i adjusted my php.ini settings (roughly) accordingly:
  • max_execution_time = 600
  • max_input_time = 600 (added to Pair's default php.ini)
  • upload_max_filesize = 60M
  • post_max_size = 60M (added to Pair's default php.ini)