Setting Up Adaptive Streaming with Nginx

Recently, I’m working out a system to smoothly stream live events for an organization. That is pretty new to me and, after a bunch of research, found that Nginx with the RTMP module seems to be a good choice. There are many difficulties when setting all this up and after several days of testing, I found a good setting that is worth a post.

Setup Nginx and RTMP module

First, let’s get Nginx set up. In order to use the RTMP module, we need to compile that as an Nginx module. It would look something like this:

# Installing requirements in Ubuntu/Debian
apt-get install git gcc make libaio1 libpcre3-dev openssl libssl-dev ffmpeg -y

# Installing the same thing in RHEL/CentOS
yum install git gcc make libaio libaio-devel openssl libssl-devel pcre-devel ffmpeg -y

# Download nginx and nginx-rtmp-module
wget http://nginx.org/download/nginx-1.9.4.tar.gz
git clone https://github.com/arut/nginx-rtmp-module.git

# Compile nginx with nginx-rtmp and libaio
tar zxvf nginx-1.9.4.tar.gz

./configure --prefix=/usr/local/nginx --with-file-aio --add-module=/path/to/nginx-rtmp/
make
make install

# Link nginx
ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginx

nginx # Start Nginx
nginx -s stop # Stop Nginx

After all things are done, check whether nginx is compiled properly.

Capture

If you can see that Nginx RTMP is included, you can go to the next step. Before we proceed to configuring Nginx for live streaming, we should confirm what kind of resolution we should provide for live streams and how much hardware power you have.

Prerequisites

For converting live streams into several streams for adaptive streaming, you need to make sure your server have enough CPU for such workload. Otherwise, the live stream will suffer from continuous delays and/or server becomes unresponsive. I have spawn some EC2 c3.large and c3.xlarge instances, test with them and I found out their optimized CPU can handle such workload with ease. Something that also worth mention is about the I/O limits of the disks. If possible, store the HLS fragments generated to an high-speed SSD helps maintain smooth streaming experiences.

ec2CPU Usage when using an EC2 c3.xlarge instance.

Then, you also need to think about what kind of resolutions you will be offering for adaptive streaming. Generally about 4-5 variants are good enough to provide great loading speeds for different network speeds and devices. Here’s my recommended list of variants used for live streaming:

  1. 240p Low Definition stream at 288kbps
  2. 480p Standard Definition stream at 448kbps
  3. 540p Standard Definition stream at 1152kbps
  4. 720p High Definition stream at 2048kbps
  5. Source resolution, source bitrate

Configuring nginx for live streaming

Here is my own nginx.conf with comments that you can have references on.

worker_processes  auto;
events {
    # Allows up to 1024 connections, can be adjusted
    worker_connections  1024;
}

# RTMP configuration
rtmp {
    server {
        listen 1935; # Listen on standard RTMP port
        chunk_size 4000; 
        
        # This application is to accept incoming stream
        application live {
            live on; # Allows live input
            
            # Once receive stream, transcode for adaptive streaming
            # This single ffmpeg command takes the input and transforms
            # the source into 4 different streams with different bitrate
            # and quality. P.S. The scaling done here respects the aspect
            # ratio of the input.
            exec ffmpeg -i rtmp://localhost/$app/$name -async 1 -vsync -1
                        -c:v libx264 -c:a libvo_aacenc -b:v 256k -b:a 32k -vf "scale=480:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_low
                        -c:v libx264 -c:a libvo_aacenc -b:v 768k -b:a 96k -vf "scale=720:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_mid
                        -c:v libx264 -c:a libvo_aacenc -b:v 1024k -b:a 128k -vf "scale=960:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_high
                        -c:v libx264 -c:a libvo_aacenc -b:v 1920k -b:a 128k -vf "scale=1280:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_hd720
                        -c copy -f flv rtmp://localhost/show/$name_src;
        }
        
        # This application is for splitting the stream into HLS fragments
        application show {
            live on; # Allows live input from above
            hls on; # Enable HTTP Live Streaming
            
            # Pointing this to an SSD is better as this involves lots of IO
            hls_path /mnt/hls/;
            
            # Instruct clients to adjust resolution according to bandwidth
            hls_variant _low BANDWIDTH=288000; # Low bitrate, sub-SD resolution
            hls_variant _mid BANDWIDTH=448000; # Medium bitrate, SD resolution
            hls_variant _high BANDWIDTH=1152000; # High bitrate, higher-than-SD resolution
            hls_variant _hd720 BANDWIDTH=2048000; # High bitrate, HD 720p resolution
            hls_variant _src BANDWIDTH=4096000; # Source bitrate, source resolution
        }
    }
}

http {
    # See http://licson.net/post/optimizing-nginx-for-large-file-delivery/ for more detail
    # This optimizes the server for HLS fragment delivery
    sendfile off;
    tcp_nopush on;
    aio on;
    directio 512;
    
    # HTTP server required to serve the player and HLS fragments
    server {
        listen 80;
        
        location / {
            root /path/to/web_player/;
        }
        
        location /hls {
            types {
                application/vnd.apple.mpegurl m3u8;
            }
            
            root /mnt/;
            add_header Cache-Control no-cache; # Prevent caching of HLS fragments
            add_header Access-Control-Allow-Origin *; # Allow web player to access our playlist
        }
    }
}
Then, configure your live encoder to use these settings to stream into the server:
  • RTMP Endpoint: rtmp://yourserver/live/
  • RTMP Stream Name: [Whatever name you like]
Finally, configure your player for live playback. The HLS URL would look like this:
http://yourserver/hls/[The stream name above].m3u8

Recommended encoder settings for live events

If you can adjust the encoder, the following settings can help to gain better experiences.

  • Full HD Resolution (1920×1080) is recommended
  • H.264 Main profile, with target bitrate of 4000Kbps, maximum 6000Kbps
  • 25fps, 2 second keyframe interval
  • AAC audio at 128Kbps, 44.1kHz sample rate

And that’s all! I hope you can enjoy doing live events with these techniques.

Stream videos to HTML5 video container using HTTP & PHP

Sometimes we need to feed videos dynamically from the server-side. If you’re feeding the video to a HTML5 <video> element, you may find that the video progress controls freezes and users cannot move it in any ways. (Thought this situation only happens in some browsers like Chrome and Firefox, the user experience hurts a lot.)

With a bit of investigation, I found out that Chrome requested the video with an HTTP range request which, the server-side handle it incorrectly and Chrome falls back to progressive downloading the video. With the little PHP script I wrote below, the server-side can now handle the HTTP range requests normally and the progress controls no longer freezes! (There’s one more benefit: fast forward and backward works much smoother in large video files.)

<?php
// Clears the cache and prevent unwanted output
ob_clean();
@ini_set('error_reporting', E_ALL & ~ E_NOTICE);
@apache_setenv('no-gzip', 1);
@ini_set('zlib.output_compression', 'Off');

$file = "/path/to/your/media/file"; // The media file's location
$mime = "application/octet-stream"; // The MIME type of the file, this should be replaced with your own.
$size = filesize($file); // The size of the file

// Send the content type header
header('Content-type: ' . $mime);

// Check if it's a HTTP range request
if(isset($_SERVER['HTTP_RANGE'])){
    // Parse the range header to get the byte offset
    $ranges = array_map(
        'intval', // Parse the parts into integer
        explode(
            '-', // The range separator
            substr($_SERVER['HTTP_RANGE'], 6) // Skip the `bytes=` part of the header
        )
    );

    // If the last range param is empty, it means the EOF (End of File)
    if(!$ranges[1]){
        $ranges[1] = $size - 1;
    }

    // Send the appropriate headers
    header('HTTP/1.1 206 Partial Content');
    header('Accept-Ranges: bytes');
    header('Content-Length: ' . ($ranges[1] - $ranges[0])); // The size of the range

    // Send the ranges we offered
    header(
        sprintf(
            'Content-Range: bytes %d-%d/%d', // The header format
            $ranges[0], // The start range
            $ranges[1], // The end range
            $size // Total size of the file
        )
    );

    // It's time to output the file
    $f = fopen($file, 'rb'); // Open the file in binary mode
    $chunkSize = 8192; // The size of each chunk to output

    // Seek to the requested start range
    fseek($f, $ranges[0]);

    // Start outputting the data
    while(true){
        // Check if we have outputted all the data requested
        if(ftell($f) >= $ranges[1]){
            break;
        }

        // Output the data
        echo fread($f, $chunkSize);

        // Flush the buffer immediately
        @ob_flush();
        flush();
    }
}
else {
    // It's not a range request, output the file anyway
    header('Content-Length: ' . $size);

    // Read the file
    @readfile($file);

    // and flush the buffer
    @ob_flush();
    flush();
}

Audio visualizations using Web Audio API

Have you ever seen some visual effects when playing music with your favourite music player? (Here’s some example photos.)

effects-2

effects-1

effects-3

This is just… great. But how they do this kind of effects? We need to understand how sound is made first. Sound is actually the vibrations of air (and more generally, objects) and the change in air pressure is recorded using a microphone. Everytime when you play a music, the speakers generates the vibrations using the change in air pressures recorded by microphone.

We can make use of these data to make visual effects from sounds. In this tutorial, I’ll teach you how to create a visualization like this:

web-audio-visuals-demo

In this demonstration, we’ll be able to drop audio files in and play the music with visuals. OK! Let’s start!

First, we need to set up the HTML for our visuals. It’ll be pretty simple as we only need a canvas.

<canvas id="canvas" style="width: 100%; height: 100%;"></canvas>

Then, it’s time to code some JavaScript!

// Init. all the variables we need
var audio, source, analyser, canvas, ctx;

// Logs messages
function clog(str) {
    if (window.console && console.log) console.log(str);
}

// This is an empty event handler
// thatprevents the event from happening
function noop(e) {
    e.stopPropagation();
    e.preventDefault();
}

// This is the function used to
// handle the drop of the audio files.
function drop(e) {
    // Prevents the default action of the event
    // In this case, open an audio file
    e.stopPropagation();
    e.preventDefault();

    // Creates a file reader
    var reader = new FileReader();

    // What should we do when the reader is ready?
    reader.onload = function(e) {
        // Decode the audio file
        // using the Web Audio API
        if (audio.decodeAudioData) {
            audio.decodeAudioData(e.target.result, function(b) {
                source.buffer = b;

                // Play the audio file
                source.noteOn(0);
            }, function(e) {
                clog(e);
                alert('Audio not playable or not supported.');
            });
        }
        else {
            source.buffer = audio.createBuffer(e.target.result, true);

            // Play the audio file
            source.noteOn(0);
        }
    }

    // Read the dropped file
    reader.readAsArrayBuffer(e.dataTransfer.files[0]);    
}

// The function that actually
// draws the visuals
function draw(e) {
    // Get the audio data
    var inp = e.inputBuffer.getChannelData(0);
    var out = e.outputBuffer.getChannelData(0);

    // Start drawing the visuals
    // We first clear the canvas first
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    ctx.strokeStyle = '#6633FF';
    ctx.beginPath();
    ctx.moveTo(0, canvas.height / 2);

    // Loop through the data and plot them
    for (var i = 0; i < inp.length; i++) {
        out[i] = 0;
        i == 0 ? ctx.moveTo(0, canvas.height / 2 + inp[i] * canvas.height / 2) : ctx.lineTo(canvas.width * i / inp.length, canvas.height / 2 + inp[i] * canvas.height / 2);
    }

    // Draw it!
    ctx.stroke();
}

window.onload = function() {
    // Init. the audio context
    // and the canvas
    audio = new(window.AudioContext || window.webkitAudioContext)();
    canvas = document.querySelector('#canvas');
    ctx = canvas.getContext('2d');
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    // Listen for file drop
    // We also need to prevent the default actions of
    // other related events
    canvas.addEventListener('dragover', noop, false);
    canvas.addEventListener('drop', drop, false);
    document.addEventListener('dragover', noop, false);
    document.addEventListener('drop', drop, false);

    // Create the audio source
    // and the analyser that
    // actually gets the audio data
    source = audio.createBufferSource();
    analyser = audio.createScriptProcessor(1024, 1, 1);
    analyser.onaudioprocess = draw;
    source.connect(analyser);
    analyser.connect(audio.destination);
    source.connect(audio.destination);
    source.loop = true;
    source.gain.value = 1;
};

// Handle window resize
function resize() {
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;
}

// Add the resize listener
window.addEventListener('resize', resize, false);

OK, it’s finished. I’ve prepared a live demo for all you guys. Enjoy! 🙂

Create real-time updates using Server-sent Events and libSSE

When I first learn node.js, I learned how to use socket.io, a node.js module for real-time communication between the client and server. However, when I’m going back coding PHP for a while, I started to figure out only few PHP libraries offered this functions and the techniques they uses are old (AJAX Streaming, Forever <iframe>, etc.). One day I found a new technique called Server-sent Events and it’s pretty easy to implant. I have an idea of creating an interface for Server-sent Events in order to speed up development and make it easier to use. libSSE is the library I’ve create just for this and it’s event-based which means data is sent only when an event happens. It also has some utility classes for communicate with other PHP scripts.

Here’s the library: https://github.com/licson0729/libSSE-php

It’s pretty easy to use. Here’s an example.

<?php
require_once('./src/libsse.php');//include the library

//create the event handler
//every event must inherit the base event class SSEEvent
class YourEventHandler extends SSEEvent {
    public function update(){
        //Here's the place to send data
        return time();
    }
    public function check(){
        //Here's the place to check when the data needs update
        return time() % 20 === 0;
    }
}

$sse = new SSE();//create a libSSE instance
//register your event handler
$sse->addEventListener(
    'event_name', //The name of the event
    new YourEventHandler() //The event handler defined above
);
$sse->start();//start the event loop

On the client-side you need to set up an EventSource object to listen event from server.

var sse = new EventSource('path/to/your/sse/script.php');
sse.addEventListener('event_name',function(e){
    var data = e.data;
    //handle your data here
},false);

Remember that Server-sent Events is still a new standard and not all browsers supports it. However, polyfills are available for them. Hope you like it and find it useful.