Live broadcast protocol

RTMP(Real Time Messaging Protocol)

brief introduction

Time Messaging Protocol, Real time messaging protocol

RTMP yes Adobe The company is Flash Audio between player and server , Open protocol for video and data transmission development .

agreement : Long connection TCP

principle : Data per moment , Forward as soon as you receive it

delay :1~3 second

advantage

* High real-time performance : Generally, it can be done 3 Seconds .
* Support encryption :rtmpe and rtmps Is the encryption protocol .
* High stability : stay PC On the platform flash The most stable way to play is rtmp, If you do CDN Or large and medium-sized cluster distribution , It is necessary to choose a stable protocol .
* This protocol is generally supported by mainstream coders .
shortcoming

* The protocol is complex : Developers are tired of writing , Efficiency is not good either .
* Cache trouble : Stream protocol is not convenient for caching .
RTSP(Real Time Streaming Protocol)

brief introduction

Real time streaming protocol (Real Time Streaming Protocol,RTSP) Is a network application protocol , Designed for the use of entertainment and communication systems , To control streaming media
The server . This protocol is used to create and control media sessions between terminals . Client publishing of media server VCR command , For example, play , Recording and pausing , To facilitate real-time control from server to client ( video on demand ) Or from client to server ( Voice recording ) Media streaming for .

Time Streaming Protocol, Real time streaming protocol )

advantage

●  Low latency , It can be done in general 500ms.

●  Good bandwidth , High aging rate .

●  Double speed playback , It is mainly the function provided during playback .

●  Precise control , Select any playback point .

shortcoming

●  The implementation of server is complex .

●  Weak proxy server : The quantity is small , Less optimization .

●  No router firewall penetration

●  Tube flow separation : need 1-3 Channels .

HLS(HTTP Live Streaming)

brief introduction

HTTP Live Streaming( The abbreviation is HLS) It's an apple company
<https://zh.wikipedia.org/wiki/%E8%8B%B9%E6%9E%9C%E5%85%AC%E5%8F%B8> Based on HTTP
<https://zh.wikipedia.org/wiki/HTTP> Streaming media for
<https://zh.wikipedia.org/wiki/%E6%B5%81%E5%AA%92%E4%BD%93> Network transmission protocol
<https://zh.wikipedia.org/wiki/%E7%BD%91%E7%BB%9C%E4%BC%A0%E8%BE%93%E5%8D%8F%E8%AE%AE>
. It's apple QuickTime X
<https://zh.wikipedia.org/w/index.php?title=QuickTime_X&action=edit&redlink=1> and
iPhone <https://zh.wikipedia.org/wiki/IPhone>
Part of a software system . It works by dividing the whole flow into small ones HTTP File to download , Download only a few at a time . When the media stream is playing , Clients can choose to download the same resources at different rates from many different standby sources , Allow streaming media sessions to adapt to different data rates . When starting a streaming media session , The client will download a
extended M3U (m3u8)
<https://zh.wikipedia.org/w/index.php?title=Extended_M3U&action=edit&redlink=1> 
playlist
<https://zh.wikipedia.org/w/index.php?title=Playlist&action=edit&redlink=1>
file , Used to find available media streams .

agreement : Short link HTTP

principle : Gather data for a period of time , generate ts Slice file , to update m3u8

delay :>10 second

advantage

●  Good performance : and http equally .

●  pierce through a wall : and http equally .

●  Native support is good :iOS Support perfect ,Android The support is poor .PC/flash There are also various as Plug in support HLS.

shortcoming

●  Poor real-time performance : And ts The length of the slice is related to the length of the slice , about 3 Slice length time delay , Basically HLS Delay in 10 More than seconds .

●  File fragmentation : If distributed HLS, Low bit stream , The slice is small , Can cause too many file fragments

Streaming media server

Main introduction

* NGINX-rtmp
* Node-media-server
* livego
* SRS
NGINX-rtmp

brief introduction : be based on NGINX Streaming media server for

Support agreement :RTMP/HLS/MPEG-DASH

Support platform :Linux/FreeBSD/MacOS/Windows

Official website :https://github.com/arut/nginx-rtmp-module
<https://github.com/arut/nginx-rtmp-module>

If it is windows platform , The compiled files can be downloaded directly

https://github.com/illuspas/nginx-rtmp-win32
<https://github.com/illuspas/nginx-rtmp-win32>

Node-media-server

brief introduction :A Node.js implementation of RTMP/HTTP-FLV/WS-FLV/HLS/DASH/MP4 Media Server

Support agreement :RTMP/HTTP-FLV/WS-FLV/HLS/DASH

Support platform :Windows/Linux/Unix

Official website :https://github.com/illuspas/Node-Media-Server/

Deploy a streaming service in one minute


https://github.com/illuspas/Node-Media-Server/wiki/%E4%B8%80%E5%88%86%E9%92%9F%E9%83%A8%E7%BD%B2%E4%B8%80%E4%B8%AA%E6%B5%81%E5%AA%92%E4%BD%93%E6%9C%8D%E5%8A%A1

<https://github.com/illuspas/Node-Media-Server/wiki/%E4%B8%80%E5%88%86%E9%92%9F%E9%83%A8%E7%BD%B2%E4%B8%80%E4%B8%AA%E6%B5%81%E5%AA%92%E4%BD%93%E6%9C%8D%E5%8A%A1>

livego

brief introduction : pure Go Write live server

Support agreement :RTMP/AMF/HLS/HTTP-FLV

Support platform :Windows/Linux/MacOS

Official website :https://github.com/gwuhaolin/livego <https://github.com/gwuhaolin/livego>

SRS

brief introduction :SRS Positioning is an operation level Internet live broadcast server cluster , The pursuit of better conceptual integrity and the simplest implementation of code .

Support agreement :RTMP/RTSP/HSL/HTTP-FLV

Support platform :Linux/MacOS

Official website :https://github.com/ossrs/srs

This function is relatively powerful , The documents on the official website are very detailed , Go to the official website for details

If you want to windows experience , You can download the version modified by netizens

https://github.com/illuspas/srs-win32

Factors leading to delay

Network Delay


Network delay here refers to the collection from the host , Play to the audience , Time difference between . The time taken by the host segment to encode the video is not considered here , And the time for the audience to watch and decode the video , Only the delay in network transmission is considered . For example, the network delay in the figure below :



in addition , Logical interaction is also involved in the process of data transmission , For example, packet retransmission and confirmation , And some logic on the cache , It will increase a lot on this basis .


So let's simply estimate the approximate network delay . as everyone knows , The speed of light in vacuum is about 300,000km/s, In other media, the speed of light will be greatly reduced , So in ordinary optical fibers , In engineering, it is generally believed that the transmission speed is 200,000km/s. In reality , You can refer to the following :


Route

distance (km)

Round trip delay (ms)


Beijing to Shanghai

1,200

12


Beijing to New York

11,000

110


equatorial circumference

40,000

400


So , There are few nodes , In case of good network conditions , Then the network delay correspondence is also the minimum , Add a certain amount of cache , The delay can be controlled at 1s~2s about . But there are many nodes , In case of poor network , The network delay will increase correspondingly , Empirically, delays can be achieved 15s above .

Network jitter


Network jitter , Is the arrival order of packets , The interval is inconsistent with the time when it is sent out . for instance , send out 100 Packets , Interval per packet 1s issue . Results No 27 Packets encountered network congestion during transmission , Make a bag 27 Not immediately 26 Arrived , It's delayed until 87 Only in the back . On the air , The effect of this jitter is actually the same as packet loss . Because you can't play the content in the order you receive it , Otherwise, it will cause distortion .

Network jitter , The playback delay will increase correspondingly . If the network jitter is large , It will cause jamming and other phenomena .



As shown in the figure above , Anchor t3 and t5 Issued packages , Respectively in t3' and t5' arrive , But the intermediate delay increases , Network jitter occurs . As a result, the delay of watching video will be increased .

Network packet loss

CDN Used in live broadcast RTMP,HLS,HTTP
FLV And so on TCP On the basis of .TCP A very important feature is reliability , That is, there will be no data loss . To ensure reliability ,TCP There are 3 Second handshake , See the figure below . First, the client sends a connection request to the server , After the server agrees , The client will confirm the connection . This is it. 3 Second handshake . next , The client starts sending data , Send a batch of data at a time , Get the server's “ received ” After confirmation , Continue to send next batch .TCP In order to ensure the transmission , There will be an automatic retransmission mechanism . If packet loss occurs during transmission , Did not receive the “ received ” signal , The lost packets will be retransmitted automatically , Until the timeout .




Because the network condition of the Internet is changing , And the network condition of the host is uncontrollable . So when the packet loss rate in the network starts to increase , Retransmission will cause the delay to increase , Even lead to repeated attempts to reconnection , This does not cache effectively , In serious cases, it will lead to the audience can not watch the video .

RTMP Cumulative delay

This delay only occurs in RTMP In the agreement

RTMP One weakness is cumulative error , as a result of RTMP be based on TCP, and TCP No packet loss ,( Possible packet loss , But use it TCP To avoid retransmission mechanism ).

So when the network state is bad , The server caches the package , Resulting in cumulative delays ;

When the network is ready , Send it to the client together .

The solution to this is , When the client buffer is large , We're disconnected .

GOP-Cache

Image group (Group of pictures,GOP)

The principle is complicated , We don't have to play tricks here , It can be specific Google

chart

RTMP delay



Picture from :https://www.processon.com/view/56ebb341e4b01c9aeb5f137f

After you look at the picture above, you will probably understand ,rtmp The delay of is and gop Hooked .

that , Why? rtsp The delay will be more than rtmp Low ? because , It's precisely controlled , You can jump out of this gop Group as unit control .

Video basic properties

Frame rate (frame rate)

Frame refers to a single picture in the stream , Frame rate refers to the number of frames in the code stream per unit time , Company fps(frame per
second). The video we see is made up of frames , That is, the composition of pictures ; If there's a picture that runs past in a second 24 Zhang , The frame rate is 24 frame / second (fps); only 8 Zhang , The frame rate is 8 frame / second (fps); The higher the frame rate, the higher the frame rate , The more fluid the picture is , To a certain extent, of course , It doesn't need to be too big , On the one hand, the naked eye can not see the difference between hair , On the other hand, computers can't bear it !



 

Common frame rate

film :23.976fps

television (PAL):25fps

television (NTSC):29.97fps

CRT monitor :60Hz-85Hz

Liquid crystal display :60Hz-75Hz

3D monitor : 120Hz

resolving power (resolution)

Resolution refers to the size of the picture , The higher the resolution , The larger the image .

General , When the bit rate is fixed , The higher the resolution , The worse the video quality is .

There are two kinds of resolution: image resolution and display resolution , Image resolution refers to the size of the image , Display resolution refers to screen resolution .


For video , There are some fixed size resolution standards , as D1(720×576),4CIF(704×576),VGA(640×480),SVGA(800×600),VXGA(1600×1200) etc. , Later, for the fixed proportion (16:9) The picture of , The resolution is expressed by vertical height plus scanning , as 720P(1280×720, progressive scanning ),1080P(1920×1080, progressive scanning ),1080I(1920×1080, Interlaced scanning ), Then back , Use horizontal pixel description , as 2K(2048×1536 or 2560×1440 or 2560×1600),4K(4096×2160 or 3840×2160),8K(7680×4320).

Bit rate (bit rate)

The bit rate is the bit rate , Data bits generated per unit time , Company bps(bit per
second),1Mbps=1024kbps=1048576bps. General , When the resolution is fixed , The higher the bit rate , The better the video quality .

(* be careful bps And B/s The difference between ,1B/s=8bps.)


usually ,720P The bit rate of the bitstream is 2~4Mbps about ,1080P The bit rate of the bitstream is 4~8Mbps about , For users, it is broadband requirements , To see 720P Video needs to be at least 2M Bandwidth of , To see 1080P Video needs to be at least 4M Bandwidth of , Under the current domestic network environment , Unequal uplink and downlink rates , If the anchor wants to use HD video , Also make sure that the uplink bandwidth of the host is sufficient .



Coding method

Usually we think about it , How to make video files smaller , How about making the video clearer ? The answer lies in the bit rate , The bit rate is processed , It's called coding , Generally there are CBR/ABR/VBR Three ways .



 

CBR Full name constant bitrate, It means fixed bit rate , It's in the whole video , The bit rate is fixed ;

VBR Full name Variable
Bitrate, It means variable bit rate . Actually, in the video , The richer the picture content is , The higher the required bit rate , The image quality will be better , The content of the screen is not very rich, using high bit rate is a waste , Therefore, the variable bit rate is automatically adjusted according to the richness of the picture content , At the same time, the imaging quality is guaranteed , Make files smaller . So make the video file smaller , Clearer content , use VBR Coding method of , And set scientific values , Is the key to solving the problem .

ABR Full name average bitrate , It means average bit rate , Is a kind of between CBR and VBR Trade off between .

Codec  Video coding



among ,MPEG-1 The encoding of the VCD,MPEG-2 Code for DVD, After MPEG-4 It is used for more and more extensive platforms , there MPEG-4 It's in the legend MP4;



What's more , There is also the International Union of telex and video (ITU) Another dominant coding method , be called H.26X series , After and after MPEG Cooperation made it possible H.264 Coding method , It has become the most popular coding method .



Container format Video packaging format

reference resources :https://en.wikipedia.org/wiki/Comparison_of_video_container_formats

List of main package formats


name

Launch mechanism

Streaming media

Video coding supported

Supported audio coding

Current application fields


AVI

Microsoft Inc.

I won't support it

Almost all formats

Almost all formats

BT Download movie


MP4

MPEG

support

MPEG-2, MPEG-4, H.264, H.263 etc.

AAC, MPEG-1 Layers I, II, III, AC-3 etc.

Internet video website


TS

MPEG

support

MPEG-1, MPEG-2, MPEG-4, H.264

MPEG-1 Layers I, II, III, AAC,

IPTV, digital television


FLV

Adobe Inc.

support

Sorenson, VP6, H.264

MP3, ADPCM, Linear PCM, AAC etc.

Internet video website


MKV

CoreCodec Inc.

support

Almost all formats

Almost all formats

Internet video website


RMVB

Real Networks Inc.

support

RealVideo 8, 9, 10

AAC, Cook Codec, RealAudio Lossless

BT Download movie

 

How to reduce the delay time

reference resources :

FFEMPEG aspect

wiki:StreamingGuide

https://trac.ffmpeg.org/wiki/StreamingGuide
<https://trac.ffmpeg.org/wiki/StreamingGuide>

Low-Latency Live Streaming your Desktop using ffmpeg

http://fomori.org/blog/?p=1213 <http://fomori.org/blog/?p=1213>

ffmpeg Test and setting optimization of transcoding delay based on

https://blog.csdn.net/fireroll/article/details/51902018
<https://blog.csdn.net/fireroll/article/details/51902018>

FFmpeg Streaming delay 10 Second problem record

https://blog.csdn.net/vanjoge/article/details/79545490
<https://blog.csdn.net/vanjoge/article/details/79545490>

ffplay Parameter setting of video source delay

https://blog.csdn.net/cai6811376/article/details/52637158
<https://blog.csdn.net/cai6811376/article/details/52637158>

Web Live series 4——ffmpeg Real time streaming +nginx Load balancing reduces live broadcast delay _1

https://blog.csdn.net/zzhang_12/article/details/79798027
<https://blog.csdn.net/zzhang_12/article/details/79798027>

ffmpeg Pull current rtmp Solution of audio real time data delay

https://blog.csdn.net/a1317338022/article/details/78226834
<https://blog.csdn.net/a1317338022/article/details/78226834>

How to achieve this 1080P Delay less than 500ms Real time ultra clear live transmission technology based on


http://windrunnerlihuan.com/2016/09/18/%E5%A6%82%E4%BD%95%E5%AE%9E%E7%8E%B01080P%E5%BB%B6%E8%BF%9F%E4%BD%8E%E4%BA%8E500ms%E7%9A%84%E5%AE%9E%E6%97%B6%E8%B6%85%E6%B8%85%E7%9B%B4%E6%92%AD%E4%BC%A0%E8%BE%93%E6%8A%80%E6%9C%AF/

<http://windrunnerlihuan.com/2016/09/18/%E5%A6%82%E4%BD%95%E5%AE%9E%E7%8E%B01080P%E5%BB%B6%E8%BF%9F%E4%BD%8E%E4%BA%8E500ms%E7%9A%84%E5%AE%9E%E6%97%B6%E8%B6%85%E6%B8%85%E7%9B%B4%E6%92%AD%E4%BC%A0%E8%BE%93%E6%8A%80%E6%9C%AF/>

FFMPEG Parameter description

reference resources :

https://blog.csdn.net/leixiaohua1020/article/details/12751349
<https://blog.csdn.net/leixiaohua1020/article/details/12751349>

https://trac.ffmpeg.org/wiki/Encode/H.264
<https://trac.ffmpeg.org/wiki/Encode/H.264>

main parameter

* -i—— Set input file name .
* -f—— Set output format .
* -y—— If the output file already exists, the file is overwritten .
* -fs—— The conversion ends when the specified file size is exceeded .
* -t—— Specifies the duration of the output file , In seconds .
* -ss—— Convert from specified time , In seconds .
* -t from -ss Time begins to change ( as -ss 00:00:01.00 -t 00:00:10.00 From 00:00:01.00 Start to 00:00:11.00).
* -title—— Set title .
* -timestamp—— Set timestamp .
* -vsync—— Increase or decrease Frame Synchronize audio and video .
* -c—— Specifies the encoding of the output file .
* -metadata—— Change the metadata of the output file
<https://zh.wikipedia.org/wiki/%E5%85%83%E6%95%B0%E6%8D%AE>.
* -help—— View help information .
Video parameters

* -b:v—— Set video traffic , The default is 200Kbit/ second .( Please refer to the notes below )
* -r—— Set frame rate value , The default is 25.
* -s—— Set the width and height of the screen .
* -aspect—— Set the scale of the screen .
* -vn—— Do not process video , Used when processing sound only .
* -vcodec( -c:v )—— Set up video codec , If it is not set, use and 输入文件相同之编解码器.
声音参数

* -b:a——设置每Channel(最近的SVN <https://zh.wikipedia.org/wiki/Subversion>
版为所有Channel的总合)的流量.(单位请引用下方注意事项)
* -ar——设置采样率.
* -ac——设置声音的Channel数.
* -acodec ( -c:a ) ——设置声音编解码器,未设置时与视频相同,使用与输入文件相同之编解码器.
* -an——不处理声音,于仅针对视频做处理时使用.
* -vol——设置音量大小,256为标准音量.(要设置成两倍音量时则输入512,依此类推.)
注意事项

* 以-b:v及-b:a首选项流量时,根据使用的ffmpeg版本,须注意单位会有kbits/sec与bits/sec的不同.(可用ffmpeg
-h显示说明来确认单位.)
例如,单位为bits/sec的情况时,欲指定流量64kbps时需输入 -b:a 64k;单位为kbits/sec的情况时则需输入 -b:a 64.

*
以-acodec及-vcodec所指定的编解码器名称,会根据使用的ffmpeg版本而有所不同.例如使用AAC编解码器时,会有输入aac与libfaac的情况.此外,编解码器有分为仅供解码时使用与仅供编码时使用,因此一定要利用ffmpeg
-formats确认输入的编解码器是否能运作.
直播中常见参数

-preset

按编码速度的内置参数,内置的编码设置有:

* ultrafast
* superfast
* veryfast
* faster
* fast
* medium
* slow
* slower
* veryslow
* placebo
按快到慢排列,默认为medium级,当然越慢效果越好,但是体积会增大,一般最求高质量用veryslow就够了.

-tune

按不同的源或者不用的应用场景的内置参数

* film:电影,真人类型
* animation:动画
* grain:需要保留大量的grain时用
* stillimage:静态图像编码时使用
* psnr:为提高psnr做了优化的参数
* ssim:为提高ssim做了优化的参数
* fastdecode:可以快速解码的参数
* zerolatency:零延迟,用在需要非常低的延迟的情况下,比如电视电话会议的编码
在命令行中输入:ffmpeg -h encoder=libx264

参考:

https://my.oschina.net/u/3733374/blog/1581609
<https://my.oschina.net/u/3733374/blog/1581609>

https://www.zhihu.com/question/23651189
<https://www.zhihu.com/question/23651189>

https://blog.csdn.net/leixiaohua1020/article/details/18893769
<https://blog.csdn.net/leixiaohua1020/article/details/18893769>

https://sdk.cn/news/5556 <https://sdk.cn/news/5556>