RTMP Tutorial - nginx build

As I described in my introduction page I wanted to live stream output from my GoPro, but as I do not have thousands of followers on YouTube and I avoid Facebook like the plague it left me with few other easy and affordable options.
So I did it myself.
In this section of my nginx, rtmp, oauth live streaming server setup I describe building nginx to inlcude the missing pieces. For this solution to work nginx needs to be able to generate SHA1 signatures and be able to create base 64 strings.
Ben Wilber’s blog describes that the needed modules include:
- ngx_devel_kit
- set-misc-nginx-module
- ngx_http_substitutions_filter_module
Section 2: Dealing with the missing nginx pieces
For this section I will not be explaining everything in grubby detail as there are enough tutorials on the web that try to do that. So if I make an off hand comment of “needed to use ldconfig to add /usr/local/lib” I kind of expect you know what that means, or at least can figure it out and why.
What I do include though is a script that I use to on my Ubuntu server that downloads nginx with its components then builds and installs it to the normal system locations. As such, make not to have nginx already installed, otherwise the script will stomp all over it and your next update will be thoroughly confused.
If you did have nginx on an Ubuntu server and typed “nginx -v” and “nginx -V” you might get the following responses:
nginx version: nginx/1.15.9 (Ubuntu)
and
nginx version: nginx/1.15.9 (Ubuntu) built with OpenSSL 1.1.1b 26 Feb 2019 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fdebug-prefix-map=/build/nginx-Kfupup/nginx-1.15.9=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-compat --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-threads --with-http_addition_module --with-http_flv_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_mp4_module --with-http_perl_module=dynamic --with-http_random_index_module --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-stream=dynamic --with-stream_ssl_module --with-stream_ssl_preread_module --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-headers-more-filter --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-auth-pam --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-cache-purge --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-dav-ext --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-ndk --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-echo --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-fancyindex --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/nchan --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-lua --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/rtmp --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-uploadprogress --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-upstream-fair --add-dynamic-module=/build/nginx-Kfupup/nginx-1.15.9/debian/modules/http-subs-filter
respectively.
The second section lists the arguments and modules built into the nginx server, which likely do not include the needed http-set-misc module.
After making my nginx server build those commands on my server return:
nginx version: nginx/1.17.3 (Ubuntu)
and
nginx version: nginx/1.17.3 (Ubuntu) built by gcc 8.3.0 (Ubuntu 8.3.0-6ubuntu1) built with OpenSSL 1.1.1c 28 May 2019 TLS SNI support enabled configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2 -Wimplicit-fallthrough=0 -Wno-error=date-time' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/usr/local/lib -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --user=nginx --group=nginx --build=Ubuntu --builddir=build --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-file-aio --with-http_degradation_module --with-pcre=../pcre-8.43 --with-pcre-jit --with-threads --with-http_addition_module --with-http_dav_module --with-http_auth_request_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-mail --with-mail_ssl_module --with-http_mp4_module --with-http_perl_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-stream --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_xslt_module --with-zlib=../zlib-1.2.11 --with-openssl=../openssl-1.1.1c --with-openssl-opt=no-nextprotoneg --add-module=/opt/nginx/nginx-modules/http-headers-more-filter --add-module=/opt/nginx/nginx-modules/http-ndk --add-module=/opt/nginx/nginx-modules/rtmp --add-module=/opt/nginx/nginx-modules/http-subs-filter --add-module=/opt/nginx/nginx-modules/http-set-misc --add-module=/opt/nginx/nginx-modules/nchan --add-module=/opt/nginx/nginx-modules/http-echo --add-module=/opt/nginx/nginx-modules/http-auth-pam --add-module=/opt/nginx/nginx-modules/http-cache-purge --add-module=/opt/nginx/nginx-modules/http-fancyindex --add-module=/opt/nginx/nginx-modules/http-uploadprogress --add-module=/opt/nginx/nginx-modules/http-upstream-fair --add-module=/opt/nginx/nginx-modules/nginx-ts-module --add-module=/opt/nginx/nginx-modules/http-dav-ext --with-compat --with-poll_module --with-select_module
What I sought to ensure was that the nginx I built supported at least the same modules as the stock Ubuntu release, plus the added modules needed for my project. That’s where the additional components such as “–add-module=/opt/nginx/nginx-modules/http-set-misc” come from.
Another goal I set out to ensure was that my builld of nginx installs all its default files in the same locations that the official Ubuntu release uses. That way my configs remain the same and in the same place, as well as (most) libraries and support files are in the default locations. One exception was openssl, for which I needed ldconfig to add the missing /usr/local/lib folder.
Of course there are a number of core operating system kits that need to be installed for the build to work, such as build-essential, git and perl. I am sure there are more, like php, that need to be present, so watch for missing component errors from the script and install whatever appears needed to get the build to work.
The script I use is below. Basically I put all the build pieces in /opt/nginx and create the kit and components in that location. The script has a few hard coded parameters that can be changed as needed, such as the nginx version, pcre version, zlib version and openssl version. The other nginx modules that I use are checked out and updated from their source git repositories. So if a new version of nginx is available, update the version value in the script. The same is true for openssl and the others.
One thing I noticed is that the nginx modules tend to have different project names than the expected folders that show up in the original nginx config. For example the openresty/headers-more-nginx-module module appears in the ubuntu nginx release as http-headers-more-filter. As such the script extraxcts the modules in an expected named folder. Some are the same, like nchan, but many folders have a different name.
My build script:
#!/bin/bash ROOT=/opt/nginx MODULES_DIR=${ROOT}/nginx-modules NGINX_VERSION=1.17.3 PCRE_VERSION=8.43 ZLIB_VERSION=1.2.11 OPENSSL_VERSION=1.1.1c # Make modules folder if [ ! -d ${MODULES_DIR} ] then mkdir ${MODULES_DIR} fi # pcre kit if [ ! -f "${ROOT}/pcre-${PCRE_VERSION}.tar.gz" ] then cd ${ROOT} rm -rf pcre* wget https://ftp.pcre.org/pub/pcre/pcre-${PCRE_VERSION}.tar.gz tar -zxf pcre-${PCRE_VERSION}.tar.gz cd ${ROOT}/pcre-${PCRE_VERSION} ./configure make make install fi # zlib kit if [ ! -f "${ROOT}/zlib-${ZLIB_VERSION}.tar.gz" ] then cd ${ROOT} rm -rf zlib* wget http://zlib.net/zlib-${ZLIB_VERSION}.tar.gz tar -zxf zlib-${ZLIB_VERSION}.tar.gz cd ${ROOT}/zlib-${ZLIB_VERSION} ./configure make make install fi # openssl kit if [ ! -f "${ROOT}/openssl-${OPENSSL_VERSION}.tar.gz" ] then cd ${ROOT} rm -rf openssl* wget http://www.openssl.org/source/openssl-${OPENSSL_VERSION}.tar.gz tar -zxf openssl-${OPENSSL_VERSION}.tar.gz cd ${ROOT}/openssl-${OPENSSL_VERSION} ./config make sudo make install fi # Install modules function fetch_module() { MOD=$1 NAME=$2 if [ ! -d "${MODULES_DIR}/${NAME}" ] then git clone ${MOD} ${MODULES_DIR}/${NAME} else pushd ${MODULES_DIR}/${NAME} 2>&1 > /dev/null git clean -fxd git pull popd 2>&1 > /dev/null fi } fetch_module https://github.com/openresty/headers-more-nginx-module http-headers-more-filter fetch_module https://github.com/arut/nginx-rtmp-module.git rtmp fetch_module https://github.com/arut/nginx-ts-module.git nginx-ts-module fetch_module https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git http-subs-filter fetch_module https://github.com/openresty/set-misc-nginx-module.git http-set-misc fetch_module https://github.com/slact/nchan.git nchan fetch_module https://github.com/openresty/echo-nginx-module.git http-echo fetch_module https://github.com/sto/ngx_http_auth_pam_module.git http-auth-pam fetch_module https://github.com/FRiCKLE/ngx_cache_purge.git http-cache-purge fetch_module https://github.com/aperezdc/ngx-fancyindex.git http-fancyindex fetch_module https://github.com/masterzen/nginx-upload-progress-module.git http-uploadprogress fetch_module https://github.com/itoffshore/nginx-upstream-fair.git http-upstream-fair fetch_module https://github.com/simplresty/ngx_devel_kit.git http-ndk fetch_module https://github.com/arut/nginx-dav-ext-module.git http-dav-ext # nginx kit cd ${ROOT} if [ ! -f "${ROOT}/nginx-${NGINX_VERSION}.tar.gz" ] then wget http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz fi rm -rf nginx-${NGINX_VERSION} tar xf nginx-${NGINX_VERSION}.tar.gz cd ${ROOT}/nginx-${NGINX_VERSION} ./configure \ --with-cc-opt="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2 -Wimplicit-fallthrough=0 -Wno-error=date-time" \ --with-ld-opt="-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/usr/local/lib -fPIC" \ --prefix=/usr/share/nginx \ --conf-path=/etc/nginx/nginx.conf \ --http-log-path=/var/log/nginx/access.log \ --error-log-path=/var/log/nginx/error.log \ --lock-path=/var/lock/nginx.lock \ --pid-path=/run/nginx.pid \ --user=nginx \ --group=nginx \ --build=Ubuntu \ --builddir=build \ --modules-path=/usr/lib/nginx/modules \ --http-client-body-temp-path=/var/lib/nginx/body \ --http-fastcgi-temp-path=/var/lib/nginx/fastcgi \ --http-proxy-temp-path=/var/lib/nginx/proxy \ --http-scgi-temp-path=/var/lib/nginx/scgi \ --http-uwsgi-temp-path=/var/lib/nginx/uwsgi \ --with-compat \ --with-debug \ --with-file-aio \ --with-http_degradation_module \ --with-pcre=../pcre-${PCRE_VERSION} \ --with-pcre-jit \ --with-threads \ --with-http_addition_module \ --with-http_dav_module \ --with-http_auth_request_module \ --with-http_flv_module \ --with-http_geoip_module \ --with-http_gunzip_module \ --with-http_gzip_static_module \ --with-http_image_filter_module \ --with-mail \ --with-mail_ssl_module \ --with-http_mp4_module \ --with-http_perl_module \ --with-http_random_index_module \ --with-http_realip_module \ --with-http_secure_link_module \ --with-http_slice_module \ --with-http_ssl_module \ --with-stream \ --with-stream_geoip_module \ --with-stream_realip_module \ --with-stream_ssl_module \ --with-stream_ssl_preread_module \ --with-http_stub_status_module \ --with-http_sub_module \ --with-http_v2_module \ --with-http_xslt_module \ --with-zlib=../zlib-${ZLIB_VERSION} \ --with-openssl=../openssl-${OPENSSL_VERSION} \ --with-openssl-opt=no-nextprotoneg \ --with-poll_module \ --with-select_module \ --add-module=${MODULES_DIR}/http-headers-more-filter \ --add-module=${MODULES_DIR}/http-ndk \ --add-module=${MODULES_DIR}/rtmp \ --add-module=${MODULES_DIR}/http-subs-filter \ --add-module=${MODULES_DIR}/http-set-misc \ --add-module=${MODULES_DIR}/nchan \ --add-module=${MODULES_DIR}/http-echo \ --add-module=${MODULES_DIR}/http-auth-pam \ --add-module=${MODULES_DIR}/http-cache-purge \ --add-module=${MODULES_DIR}/http-fancyindex \ --add-module=${MODULES_DIR}/http-uploadprogress \ --add-module=${MODULES_DIR}/http-upstream-fair \ --add-module=${MODULES_DIR}/nginx-ts-module \ --add-module=${MODULES_DIR}/http-dav-ext make make install service nginx stop # just in case killall -9 nginx cp build/nginx /usr/sbin rm -f /var/log/nginx/* service nginx start
With that I end up with an updated nginx using current components (as opposed to old and busted components) that supports the modules needed for my project. Of course it should be pointed out that unless you really trust what you have built it is adviseable not to run this in an exposed production server. The nginx executable and its components are only as good as the most recent check-ins and releases, as well as your development environment. It is possible that something in the newer releases of the source code does not work properly, so you’ve been warned.
Hopefully though the script can be helpful for someone.
- Introduction: What this project is about
- Section 1: The nginx config and how it controls access
- Section 2: Dealing with the missing nginx pieces <== YOU ARE HERE
- Section 3: Designing an application to glue it together
- Section 4: Integrating OAuth authentication using Nextcloud (Coming Soon)
- Section 5: Session storage with Redis (cool Enterprise scaling option) (Coming Soon)
- Section 6: Streaming with my GoPro or with ffmpeg (Coming Soon)
In the next section I will discuss my python application that glues it all together.