For audio part , Mainly from the following parts to achieve .

1, Start of audio playback , stop it , suspend , Implementation of buffer write interface

2,ffmpeg Audio decoder on and audio decoding

3,ffmpeg Audio resampling standardizes the output format of audio

4, Synchronization of audio and video playback by multithreading and buffer queue

One , Implementation of start and stop interface for audio playback

First, we create an audio playback class XAudioPlay, Use singleton mode , Provide an interface , stay XAudioPlay.cpp Re implements this interface in , The statement is as follows .
#pragma once class XAudioPlay { public: XAudioPlay *Get();// Singleton mode virtual bool
Start()=0;// start-up virtual void Play(bool isplay)=0;// suspend virtual bool Write(const
char *data,int datasize) = 0;// Write audio to virtual void Stop()=0;// stop it virtual int
GetFree() = 0;// Get remaining space virtual ~XAudioPlay(); int sampleRate = 48000;// Sample rate int
sampleSize = 16;// Sample size int channel = 2;/// Number of channels protected: XAudioPlay(); };
Yes, start , stop it , suspend , The buffer write function is defined as follows .
#include "XAudioPlay.h" #include <QAudioOutput> #include<QMutex> class
CXAudioPlay :public XAudioPlay { public: QAudioOutput *output = NULL; QIODevice
*io = NULL; QMutex mutex; void Stop() { mutex.lock(); if
(output)// Open for AudioOutput { output->stop(); delete output; output = NULL; io =
NULL; } mutex.unlock(); } // First set the format and parameters of playback bool Start() { Stop(); mutex.lock();
QAudioOutput *out;// Play audio QAudioFormat fmt;// Set audio output format
fmt.setSampleRate(48000);//1 Second audio sampling rate fmt.setSampleSize(16);// Size of sound sample
fmt.setChannelCount(2);// Vocal tract fmt.setCodec("audio/pcm");// Decoding format
fmt.setByteOrder(QAudioFormat::LittleEndian);
fmt.setSampleType(QAudioFormat::UnSignedInt);// Set audio type output = new
QAudioOutput(fmt); io = output->start();// Play begins mutex.unlock(); return true; }
void Play(bool isplay) { mutex.lock(); if (!output) { mutex.unlock(); return; }
if (isplay) { output->resume();// Resume playback } else { output->suspend();// Pause playback }
mutex.unlock(); } int GetFree() { mutex.lock(); if (!output) { mutex.unlock();
return 0; } int free = output->bytesFree();// Remaining space mutex.unlock(); return free;
} bool Write(const char *data, int datasize) { mutex.lock(); if (io)
io->write(data, datasize);// Writes the acquired audio to the buffer mutex.unlock(); return true; } };
XAudioPlay::XAudioPlay() { } XAudioPlay::~XAudioPlay() { } XAudioPlay *
XAudioPlay::Get() { static CXAudioPlay ap; return ≈ }
Here, when we write to the buffer , We have to make sure that the buffer has space for us to write , So here it is GetFree() Function to detect the current buffer size .

Two ,ffmpeg Audio decoder on and audio decoding

We were in XFFmpeg.cpp in Open Video time , We only deal with video , Audio is not processed , Now we are Open For audio processing , Turn on the audio decoder first , stay .h Defined in
audioStream// Audio stream int sampleRate = 48000;// Sample rate int sampleSize = 16;// Sample size int
channel = 2;/// Number of channels
stay XFFmpeg.cpp Add the judgment audio stream after the judgment video stream as follows :
else if (enc->codec_type == AVMEDIA_TYPE_AUDIO)// If no audio stream { audioStream = i;// Audio stream
AVCodec *codec = avcodec_find_decoder(enc->codec_id);// Find decoder if
(avcodec_open2(enc, codec, NULL) < 0) { mutex.unlock(); return 0; }
this->sampleRate = enc->sample_rate;// Sample rate this->channel = enc->channels;// Number of channels
switch (enc->sample_fmt)// Sample size { case AV_SAMPLE_FMT_S16://signed 16 bits
this->sampleSize = 16; break; case AV_SAMPLE_FMT_S32://signed 32 bits
this->sampleSize = 32; default: break; } printf("audio sample rate:%d sample
size:%d chanle:%d\n",this->sampleRate,this->sampleSize,this->channel); }

Now let's think about audio decoding , stay XFFmpeg.cpp Of Decode() In the function of , Yes avcodec_send_packet() Function is used to send packets , It uses the same interface to send audio and video data , Then there is a problem , Can audio and video be stored together after decoding , Because we need to do audio later , The video needs to be synchronized , In other words, audio and video need to be played in multiple threads , There will be some problems when playing synchronously . So here's another piece of memory for the audio . First, in the .h Add the statement of storing audio after decoding , Now there's another problem , Before, our play was based on video , The playback speed can be controlled by video , After adding audio , Our video can still adapt to audio playback , But the audio should not adapt to the video , Because once you control the audio playback speed , May lead to audio distortion . stay XFFmpeg..h We've saved one pts, So here is the audio time after decoding , We deal with this pts, Don't change it for video , So how does video keep up with audio , Here we need to rewrite AVFrame
* XFFmpeg::Decode(const AVPacket *pkt), Previously, we returned the decoded video frame , Now we go back to the display time .
int XFFmpeg::Decode(const AVPacket *pkt) { mutex.lock(); if (!ic)// If the video is not turned on {
mutex.unlock(); return NULL; } if (yuv == NULL)// Object space for decoding { yuv =
av_frame_alloc(); } if (pcm == NULL) { pcm = av_frame_alloc(); } AVFrame *frame
= yuv;// At this time frame Is the decoded video stream if (pkt->stream_index == audioStream)// If there is no audio { frame =
pcm;// here frame Is the decoded audio stream } int re =
avcodec_send_packet(ic->streams[pkt->stream_index]->codec, pkt);// Read before sending pkt if
(re != 0) { mutex.unlock(); return NULL; } re =
avcodec_receive_frame(ic->streams[pkt->stream_index]->codec,
frame);// decode pkt Post deposit yuv in if (re != 0) { mutex.unlock(); return NULL; } qDebug() <<
"pts=" << frame->pts; mutex.unlock(); int p =
frame->pts*r2d(ic->streams[pkt->stream_index]->time_base);// Display time of current decoding if
(pkt->stream_index == audioStream)// Set for audio streaming pts this->pts = p; return p; }
Three ,ffmpeg Audio resampling standardizes the output format of audio , And video playback

There are many similarities between audio resampling and video conversion , We are here .h In China int ToPCM(char *out);// Resampling of audio , as well as SwrContext *aCtx =
NULL;// Audio resampling context , stay .cpp Is defined as follows
int XFFmpeg::ToPCM(char *out) { mutex.lock(); if (!ic || !pcm ||
!out)// File not open , Decoder not on , No data { mutex.unlock(); return 0; } AVCodecContext *ctx =
ic->streams[audioStream]->codec;// Audio decoder context if (aCtx == NULL) { aCtx =
swr_alloc();// initialization swr_alloc_set_opts(aCtx,ctx->channel_layout,
AV_SAMPLE_FMT_S16, ctx->sample_rate, ctx->channels, ctx->sample_fmt,
ctx->sample_rate, 0,0 ); swr_init(aCtx); } uint8_t *data[1]; data[0] = (uint8_t
*)out; // Resampling process of audio int len = swr_convert(aCtx, data, 10000, (const uint8_t
**)pcm->data, pcm->nb_samples ); if (len <= 0) { mutex.unlock(); return 0; }
int outsize = av_samples_get_buffer_size(NULL, ctx->channels, pcm->nb_samples,
AV_SAMPLE_FMT_S16, 0); mutex.unlock(); return outsize; }

stay ffmpeg It's used inside av_sample_get_buffer_size To calculate the number of bytes occupied by audio . Now let's play audio and video , First, in the aginexplay.cpp Medium open The sampling rate of audio resampling in function , Number of channels , Set the sample size , We use the data from the original video here , Then start the resampling of audio and video . as follows .
void agineXplay::open() { QString name = QFileDialog::getOpenFileName(this,
QString::fromLocal8Bit(" Select video file "));// Open video file if (name.isEmpty()) return;
this->setWindowTitle(name);// Set the title of the window int totalMs =
XFFmpeg::Get()->Open(name.toLocal8Bit());// Get video total time if (totalMs <= 0)// Not opened successfully {
QMessageBox::information(this, "err", "file open failed!");// Pop up error window return; }
XAudioPlay::Get()->sampleRate = XFFmpeg::Get()->sampleRate;
XAudioPlay::Get()->channel = XFFmpeg::Get()->channel;
XAudioPlay::Get()->sampleSize = 16; XAudioPlay::Get()->Start();// Start audio char
buf[1024] = { 0 };// Total storage time int min = (totalMs) / 60; int sec = (totalMs) % 60;
sprintf(buf, "%03d:%02d", min, sec);// Deposit buf in
ui.totaltime->setText(buf);// Display in the interface play(); }
, After setting the format of audio resampling , Now we need to achieve the effect of playing audio , Considering that the playing requirement of audio is higher than that of video per frame , We are here XVideoThread.cpp Make the following changes .
void XVideoThread::run() { char out[10000] = {0}; while (!isexit)// The thread did not exit { if
(!XFFmpeg::Get()->isPlay)// If suspended , Don't deal with it { msleep(10); continue; } int free =
XAudioPlay::Get()->GetFree();// The size of the buffer if (free < 10000) { msleep(1);
continue; } AVPacket pkt = XFFmpeg::Get()->Read(); if (pkt.size <= 0)// Video not open {
msleep(10); continue; } if (pkt.stream_index == XFFmpeg::Get()->audioStream) {
XFFmpeg::Get()->Decode(&pkt);// Decode audio av_packet_unref(&pkt);// release pkt package int len =
XFFmpeg::Get()->ToPCM(out);// Resampling audio XAudioPlay::Get()->Write(out, len);// Write audio
continue; } XFFmpeg::Get()->Decode(&pkt);// Frame decoding video av_packet_unref(&pkt); } }
One of them is here GetFree() Function is to ensure that when we write audio , At this time, its buffer can also store the frame audio , Avoid errors in subsequent audio reading .

Four , Synchronization of audio and video playback by multithreading and buffer queue


The above will not cause any problems when I play the local video , But when we play the network video, the audio and video may be out of sync , The phenomenon of scattered sound , Basically, video can't keep up with audio , Sometimes the audio can't keep up with the video , In this way, we can not usually only reduce the video frame to reduce the video playback . How do we synchronize audio and video here , One way is to use a List Linked list , Each time AVPacket Put the video frame in front of the audio , When we read the audio decode , At the same time List All the video frames in the list are decoded and played , Ensure the audio and video playback , How to deal with it ? First of all, we need to decode the audio and video Pts, Before in XFFMpeg.cpp One of them Decode() function , We've modified it and returned it after decoding pts, So here we are XVideoThread.cpp When read for audio is , Record it pts, Reprocessing list In linked list AVPacket To decode the video frame , The specific process is described in detail below .
#include "XVideoThread.h" #include "XFFmpeg.h" #include "XAudioPlay.h"
#include <list> using namespace std; static list<AVPacket> videos;// It is used to store the video frame before decoding
bool isexit = false;// The thread did not exit static int apts = -1;// Audio pts
XVideoThread::XVideoThread() { } XVideoThread::~XVideoThread() { } void
XVideoThread::run() { char out[10000] = {0}; while (!isexit)// The thread did not exit { if
(!XFFmpeg::Get()->isPlay)// If suspended , Don't deal with it { msleep(10); continue; } while
(videos.size()>0)// determine list Is there any AVpacket package { AVPacket pack =
videos.front();// Take it out every time list First in AVPack package int pts =
XFFmpeg::Get()->GetPts(&pack);// Get the pts if (pts > apts)// If the video package is larger than the audio package pts, end {
break; } XFFmpeg::Get()->Decode(&pack);// Frame decoding video
av_packet_unref(&pack);// Clean up the AVPacket package videos.pop_front();// from list Delete from linked list } int
free = XAudioPlay::Get()->GetFree();// The size of the buffer if (free < 10000) { msleep(1);
continue; } AVPacket pkt = XFFmpeg::Get()->Read(); if (pkt.size <= 0)// Video not open {
msleep(10); continue; } if (pkt.stream_index == XFFmpeg::Get()->audioStream) {
apts = XFFmpeg::Get()->Decode(&pkt);// Decode audio av_packet_unref(&pkt);// release pkt package S int
len = XFFmpeg::Get()->ToPCM(out);// Resampling audio XAudioPlay::Get()->Write(out,
len);// Write audio continue; } videos.push_back(pkt); } }
 

At this time, whether it is local video or network video, we can complete the synchronization of audio and video , So far, the audio processing has been completed , The whole audio and video play is over , The contents involved are as follows :


FFMpeg,Qt With QOpenGL Drawing video ( The back is useful OpenGL adopt GPU Code links for indirectly drawing videos ), Multithreading to read audio and video , Decoding and audio resampling , The transcoding rendering of the video is in the VideoWidget.cpp Implementation in .

design sketch :



Effect picture code link :https://download.csdn.net/download/hfuu1504011020/10672140
<https://download.csdn.net/download/hfuu1504011020/10672140>