Setting Up Adaptive Streaming with Nginx

Recently, I’m working out a system to smoothly stream live events for an organization. That is pretty new to me and, after a bunch of research, found that Nginx with the RTMP module seems to be a good choice. There are many difficulties when setting all this up and after several days of testing, I found a good setting that is worth a post.

Setup Nginx and RTMP module

First, let’s get Nginx set up. In order to use the RTMP module, we need to compile that as an Nginx module. It would look something like this:

# Installing requirements in Ubuntu/Debian
apt-get install git gcc make libaio1 libpcre3-dev openssl libssl-dev ffmpeg -y

# Installing the same thing in RHEL/CentOS
yum install git gcc make libaio libaio-devel openssl libssl-devel pcre-devel ffmpeg -y

# Download nginx and nginx-rtmp-module
wget http://nginx.org/download/nginx-1.9.4.tar.gz
git clone https://github.com/arut/nginx-rtmp-module.git

# Compile nginx with nginx-rtmp and libaio
tar zxvf nginx-1.9.4.tar.gz

./configure --prefix=/usr/local/nginx --with-file-aio --add-module=/path/to/nginx-rtmp/
make
make install

# Link nginx
ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginx

nginx # Start Nginx
nginx -s stop # Stop Nginx

After all things are done, check whether nginx is compiled properly.

Capture

If you can see that Nginx RTMP is included, you can go to the next step. Before we proceed to configuring Nginx for live streaming, we should confirm what kind of resolution we should provide for live streams and how much hardware power you have.

Prerequisites

For converting live streams into several streams for adaptive streaming, you need to make sure your server have enough CPU for such workload. Otherwise, the live stream will suffer from continuous delays and/or server becomes unresponsive. I have spawn some EC2 c3.large and c3.xlarge instances, test with them and I found out their optimized CPU can handle such workload with ease. Something that also worth mention is about the I/O limits of the disks. If possible, store the HLS fragments generated to an high-speed SSD helps maintain smooth streaming experiences.

ec2CPU Usage when using an EC2 c3.xlarge instance.

Then, you also need to think about what kind of resolutions you will be offering for adaptive streaming. Generally about 4-5 variants are good enough to provide great loading speeds for different network speeds and devices. Here’s my recommended list of variants used for live streaming:

  1. 240p Low Definition stream at 288kbps
  2. 480p Standard Definition stream at 448kbps
  3. 540p Standard Definition stream at 1152kbps
  4. 720p High Definition stream at 2048kbps
  5. Source resolution, source bitrate

Configuring nginx for live streaming

Here is my own nginx.conf with comments that you can have references on.

worker_processes  auto;
events {
    # Allows up to 1024 connections, can be adjusted
    worker_connections  1024;
}

# RTMP configuration
rtmp {
    server {
        listen 1935; # Listen on standard RTMP port
        chunk_size 4000; 
        
        # This application is to accept incoming stream
        application live {
            live on; # Allows live input
            
            # Once receive stream, transcode for adaptive streaming
            # This single ffmpeg command takes the input and transforms
            # the source into 4 different streams with different bitrate
            # and quality. P.S. The scaling done here respects the aspect
            # ratio of the input.
            exec ffmpeg -i rtmp://localhost/$app/$name -async 1 -vsync -1
                        -c:v libx264 -c:a libvo_aacenc -b:v 256k -b:a 32k -vf "scale=480:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_low
                        -c:v libx264 -c:a libvo_aacenc -b:v 768k -b:a 96k -vf "scale=720:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_mid
                        -c:v libx264 -c:a libvo_aacenc -b:v 1024k -b:a 128k -vf "scale=960:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_high
                        -c:v libx264 -c:a libvo_aacenc -b:v 1920k -b:a 128k -vf "scale=1280:trunc(ow/a/2)*2" -tune zerolatency -preset veryfast -crf 23 -f flv rtmp://localhost/show/$name_hd720
                        -c copy -f flv rtmp://localhost/show/$name_src;
        }
        
        # This application is for splitting the stream into HLS fragments
        application show {
            live on; # Allows live input from above
            hls on; # Enable HTTP Live Streaming
            
            # Pointing this to an SSD is better as this involves lots of IO
            hls_path /mnt/hls/;
            
            # Instruct clients to adjust resolution according to bandwidth
            hls_variant _low BANDWIDTH=288000; # Low bitrate, sub-SD resolution
            hls_variant _mid BANDWIDTH=448000; # Medium bitrate, SD resolution
            hls_variant _high BANDWIDTH=1152000; # High bitrate, higher-than-SD resolution
            hls_variant _hd720 BANDWIDTH=2048000; # High bitrate, HD 720p resolution
            hls_variant _src BANDWIDTH=4096000; # Source bitrate, source resolution
        }
    }
}

http {
    # See http://licson.net/post/optimizing-nginx-for-large-file-delivery/ for more detail
    # This optimizes the server for HLS fragment delivery
    sendfile off;
    tcp_nopush on;
    aio on;
    directio 512;
    
    # HTTP server required to serve the player and HLS fragments
    server {
        listen 80;
        
        location / {
            root /path/to/web_player/;
        }
        
        location /hls {
            types {
                application/vnd.apple.mpegurl m3u8;
            }
            
            root /mnt/;
            add_header Cache-Control no-cache; # Prevent caching of HLS fragments
            add_header Access-Control-Allow-Origin *; # Allow web player to access our playlist
        }
    }
}
Then, configure your live encoder to use these settings to stream into the server:
  • RTMP Endpoint: rtmp://yourserver/live/
  • RTMP Stream Name: [Whatever name you like]
Finally, configure your player for live playback. The HLS URL would look like this:
http://yourserver/hls/[The stream name above].m3u8

Recommended encoder settings for live events

If you can adjust the encoder, the following settings can help to gain better experiences.

  • Full HD Resolution (1920×1080) is recommended
  • H.264 Main profile, with target bitrate of 4000Kbps, maximum 6000Kbps
  • 25fps, 2 second keyframe interval
  • AAC audio at 128Kbps, 44.1kHz sample rate

And that’s all! I hope you can enjoy doing live events with these techniques.

Optimizing Nginx for (large) file delivery

Some times ago, I have a need to host some big files for open download. At first, I think Nginx will perform pretty well without muck configuration. In reality, there are complaints about slow and interrupted downloads which is quite annoying.

I ended up using Xender for PC to transfer the files, but after digging the Nginx docs, I did find some nice changes that can fix these problems and produce a high throughput. Here’s my tweaks made to the nginx.conf file:

  1. Turn off sendfileThe Linux sendfile call is known to have throughput degradation when in high load. Disabling it helps to keep a higher throughput at high load. Also, when serving large files with sendfile, there are no ways to control readahead.
  2. Enable TCP nopushTCP nopush fills the TCP packet to its maximum size before sending. This can help increase throughput if you’re serving large files.
  3. Use Nginx’s directio to load fileUsing directio can help improving performance by skipping a bunch of steps happened in the kernel when reading files, thus speed up the throughput.
  4. Enable the use of libaio for optimal performancelibaio allows asynchronous I/O to be done in kernel, which results in faster read and write speed. However, it needs libaio to be installed and re-compiling your Nginx in order to have it supported. I used the following flow to recompiling Nginx with aio support.
    # Install libaio on RHEL/CentOS
    yum install libaio libaio-devel -y
    
    wget http://nginx.org/download/nginx-1.9.4.zip
    unzip -q nginx-1.9.4.zip
    cd nginx-1.9.4
    
    # Configure Nginx according to your needs, but it should also include
    # --with-file-aio in order to use libaio
    ./comnfigure --with-file-aio
    make

The complete nginx.conf should look like this:

http {
    ...
    sendfile off;
    tcp_nopush on;
    aio on;
    directio 512; # required if you're using Linux and uses aio
    ...
}

There are also some lower-level tewaks like mounting your disks with noatime flags and use ext4/xfs when serving files.

Merry Christmas!

Hi! It’s almost Christmas, and it’s time to say Merry Christmas to everyone out here. This year, I really want to thank so many people for their help and support. Although some of my friends are going elsewhere to study and my mother’s having some illness, with the help of different people, I’m feeling much better now.

Finally, I recorded a nice song (“Joy to the World”) for this merciful moment, enjoy! And, Merry Christmas to everyone and have a nice new year!

[Music Sharing] How To Train Your Dragon Soundtrack

My blog is mainly about programming but I want to tell you: I <3 music also! This time, I want to share you a nice soundtrack called “How To Train Your Dragon”. Let’s enjoy!

What is a router explained

(This article is written for my classmates.)

As everyone knows, the internet is extremely big and it seamlessly connects every people, in the world. But there’s one question: how can the internet knows where to pass the data around? The answer to this problem is the router. The router, by definition, is a device that forwards data between computer networks. To understand what a router actually does, we need to have a basic understanding to the structures of the internet. Continue reading “What is a router explained”