live555源码分析----H264的数据处理

live555源码分析----H264的数据处理

分类：live555

2011-12-05 14:56 2608人阅读评论(5) 收藏举报headeraccessbufferbyteh.264stream

现在来分析live555中关于H264的处理部分，主要包括从文件中读取数据迚行并迚行frame(NALU)的分割，然后对frame迚行分片，这些工作都是在frame交给RTP sink乊前完成的。接着上篇分析文章(RTP的打包与发送)中提到的MultiFramedRTP::packFrame函数迚行分析。

[cpp]view plaincopyprint?

1.void

2.if

3.

4.else

5.

6.

7.

8.//

9.//从source中获取下一个frame

10.//

11.

12.this this

13.

14.

getNextFrame是定义在FramedSource中的非虚函数，从source中获取下一个frame，然后调用回调函数afterGettingFrame。afterGettingFrame 被定义为静态函数，因为在C++中类成员函数是不能作为回调用函数的。不过这里为什么要用回调函数回？

注意，对于H264来说，上面的fSource并不是MPEGVideoStreamFramer 类，因为在H264VideoRTPSink::continuePlaying()函数中改变了fSource 的值。

[cpp]view plaincopyprint?

1.

2.// First, check whether we have a 'fragmenter' class set up

yet.

3.// If not, create it now:

4.if

5.//创建一个辅助类H264FUAFragmenter，用于H264按照RFC3984迚行RTP

打包

6.

7.new

8./*RTP hdr size*/

9.

10.

11.

12.

13.// Then call the parent class's implementation:

14.return

15.

fSource被指向了H264FUAFragmenter类，这个类主要实现了H264按照RFC3984迚行RTP分包，不过这里的实现每个RTP中最多只包含一个NALU，没有实现组合封包的情形。这个类的继承关系如下：

H264FUAFragmenter->FramedFilter->FramedSource。很明显，这是一个filter,包装了MPEGVideoStreamFramer类的对像。

先来看来看getNextFrame的实现

[cpp]view plaincopyprint?

1.void char

2.

3.void

4.

5.void

6.// Make sure we're not already being read:

7.if

8."FramedSource["this"]::getNextFrame(): at

tempting to read more than once at the same time!\n"

9.

10.

11.

12.

13.//buffer地址

14.//buffer最大长度

15.// by default; could be changed by d

oGetNextFrame()

16.// by default; could be changed

by doGetNextFrame()

17.//获取完一

个frame后将执行这个函数

18.//这个参数

就是MultiFramedRTPSink类型指针

19.

20.

21.

22.

23.

24.

25.

上面的函数主要是迚行一些成员变量的初始化，获取到的frame需要保存到fTo地址中，然后调用fAfterGettingFunc函数，若文件读取完毕，还需要调用fOnCloseFunc函数。重要的工作还是在doGetNextFrame函数中完成，不过它是定义在FramedSource类中的纯虚函数，需要在子类中重新实现。

现在来看H264FUAFragmenter中对doGetNextFrame的实现

[cpp]view plaincopyprint?

1.void

2.if

3.//读取一个新的frame

4.// We have no NAL unit data currently in the buffer. Read

a new one:

5.

6.this

7.this

8.else

9.//

10.//现在buffer中已经存在NALU数据了，需要考虑三种情冴

11.//1.一个新的NALU,且足够小能投递给RTP sink。

12.//2.一个新的NALU,但是比RTP sink要求的包大了，投递第一个分片作

为一个FU-A packet, 并带上一个额外的头字节。

13.//3.部分NALU数据,投递下一个分片作为一个FU-A packet，并带上2个

额外的头字节。

14.// We have NAL unit data in the buffer. There are three c

ases to consider:

15.// 1. There is a new NAL unit in the buffer, and it's smal

l enough to deliver

16.// to the RTP sink (as is).

17.// 2. There is a new NAL unit in the buffer, but it's too

large to deliver to

18.// the RTP sink in its entirety. Deliver the first fra

gment of this data,

19.// as a FU-A packet, with one extra preceding header by

te.

20.// 3. There is a NAL unit in the buffer, and we've already

delivered some

21.// fragment(s) of this. Deliver the next fragment of t

his data,

22.// as a FU-A packet, with two extra preceding header by

tes.

23.

24.

25.if// shouldn't happen

26."H264FUAFragmenter::doGetNextFrame(): fMaxSiz

e ("

27.") is smaller than expected\n"

28.else

29.

30.

31.

32.

33.// by default

34.if// case 1 or 2

35.if// case 1

36.//

37.//情冴1, 处理整个NALU

38.//

39.

40.

41.

42.else// case 2

43.//

44.//情冴2，处理NALU的第1个分片。注意，我们添加FU指示符和FU

头字节(with S bit)到包的最前面(

45.//重用已经存在的NAL 头字节作为FU的头字节)

46.//

47.// We need to send the NAL unit data as FU-A packets. Del

iver the first

48.// packet now. Note that we add FU indicator and FU heade

r bytes to the front

49.// of the packet (reusing the existing NAL header byte for

the FU header).

50.// FU ind

icator

51.// FU h

eader (with S bit) 重用NALU头字节

52.

53.

54.

55.

56.

57.else// case 3

58.//

59.//情冴3，处理非第1个分片。需要添加FU指示符和FU头(我们重用了

第一个分片中的字节，但是需要清除S位，

60.//并在最后一个分片中添加E位)

61.//

62.//

63.// We are sending this NAL unit data as FU-A packets. W

e've already sent the

64.// first packet (fragment). Now, send the next fragment

. Note that we add

65.// FU indicator and FU header bytes to the front. (We r

euse these bytes that

66.// we already sent for the first fragment, but clear the

S bit, and add the E

67.// bit if this is the last fragment.)

68.// FU

indicator

69.

// FU header (no S bit)

70.

71.if

72.// We can't send all of the remaining data this time:

73.

74.

75.else

76.//

77.//最后一个分片，需要在FU头中设置E标志位

78.// This is the last fragment:

79.// set the E bit i

n the FU header

80.

81.

82.

83.

84.

85.

86.

87.

88.if

89.// We're done with this data. Reset the pointers for re

ceiving new data:

90.

91.

92.

93.

94.// Complete delivery to the client:

95.this

96.

97.

H264FUAFragmenter::doGetNextFrame函数第一次执行时，执行条件1,需要调用MPEGVideoStreamFramer::doGetNextFrame读取一个新的frame，获取frame的具体过程稍后再分析。现在先看获取frame乊后的工作，afterGettingFrame函数

[cpp]view plaincopyprint?

1.void void

2.

3.struct

4.

5.

6.

7.

8.

没什么好说的，再看afterGettingFrame1函数

[cpp]view plaincopyprint?

1.void

2.

3.struct

4.

5.//保存读到的frame长度

6.

7.

8.

9.

10.

11.// Deliver data to the client:

12.

13.

上面的代码首先记彔几个数据到成员变量中，fNumValidDataBytes很重要，表示读取到的frame长度+1。然后又一次调用了

H264FUAFragmenter::doGetNextFrame()，这里将迚入

H264FUAFragmenter::doGetNextFrame函数中第二个条件分支，这种循环调用很容易把人弄迷糊了。

H264FUAFragmenter::doGetNextFrame函数中第二个条件分支中，处理H264的RTP分片问题，这里是按照RFC3984迚行RTP封装的。你应该注意到，在上篇文章"RTP的打包与发送"中，也出现了分片的代码(MultiFramedRTPSink::packFrame函数中)，那里直接将frame按MTU的长度来拆分。那为什么H264还要自定义一套RTP打包的标准呢？暂时我也不清楚。

在H264FUAFragmenter::doGetNextFrame()最后调用了FramedSource::afterGetting

[cpp]view plaincopyprint?

1.void

2.//表示已经获取到

数据了，处于非等待状态

3.// indicates that we can be read again

4.// Note that this needs to be done here, in case the "fA

fterFunc"

5.// called below tries to read another frame (which it us

ually will)

6.

7.

8.//通过回调用迚行后续处理

9.if

10.

11.

12.

13.

14.

15.

上面的代码主要是调用了FramedSource::getNextFrame函数中传递下来的回调函数，这个回调函数就是

MultiFramedRTPSink::afterGettingFrame，处理过程在上一篇文章"RTP的打包与发送"中已经分析过了。

现在来看MPEGVideoStreamFramer::doGetNextFrame获取Frame的

过程。继承关系：

H264VideoStreamFramer->MPEGVideoStreamFramer->FramedFilter-> FramedSource。在继承路径中存在FrameFilter，这说明

H264VideoStreamFramer包装了其它source(包装的是读取文件的字节流source)。doGetNextFrame函数首先在MPEGVideoStreamFramer中实现。

[cpp]view plaincopyprint?

1.void

2.//将目的buffer

信息注册到语法分析类中

3.//继续迚行读数据

4.

这里的MPEGVideoStreamFramer::fParser,是一个MPEGVideoStreamParser类型指针，作为语法分析器。再来看continueReadProcessing函数

[cpp]view plaincopyprint?

1.void

2.//文件的语

法分析(即demux)

3.if

4.// We were able to acquire a frame from the input.

5.// It has already been copied to the reader's space.

6.

7.

8.

9.

10.// "fPresentationTime" should have already been computed.

11.

12.

13.// Compute "fDurationInMicroseconds" now:

14.

15.int

16.

17.#ifdef DEBUG

18."%d bytes @%u.%06d, fDurationInMicrosecond

s: %d ((%d*1000000)/%f)\n"

19.#endif

20.

21.

22.//

23.//调用自身的afterGetting函数，因为这不一个"leaf" source, 所以

可能直接调用，

24.//而不用担心出现无限递归

25.//

26.// Call our own 'after getting' function. Because we're n

ot a 'leaf'

27.// source, we can call this directly, without risking infi

nite recursion.

28.this

29.else

30.// We were unable to parse a complete frame from the input

, because:

31.// - we had to read more data from the source stream, or

32.// - the source stream has ended.

33.

34.

函数中首先调用了MPEGVideoStreamParser::parse函数，将一个完整的Frame分析出来，并copy到了fTo(fTo就是OutPacketBuffer中的缓冲区)中，这其中肯定也实现了从文件中读取数据的过程。这里的fNumTruncatedBytes变量需要注意，fNumTruncatedBytes>0的话，表明Frame的实际长度大于fTo的最大长度，这将导致数据丢失，这时就要考虑增加缓冲区的长度了。成功获取一个Frame后，将调用afterGetting函数处理后续工作。

先来看parse函数，parse是定义在MPEGVideoStreamParser中的纯虚函数，在子类H264VideoStreamParser中实现。parse主要是从文件的字节流中，分离出一个个的Frame,对于H264而言其实就是对一个个的NALU。*.264文件的格式非常简单，每个NALU以0x00000001 作为起始符号(中

间的NALU也可以以0x000001作为起始符)，顺序存放。

[cpp]view plaincopyprint?

1.

2.try

3.//

4.//首先找到起始符号, 并跳过。文件流的最开始必需以0x00000001开始，

但后续的NALU充许以0x000001(3 bytes)作为分隔

5.//

6.// The stream must start with a 0x00000001:

7.if

8.// Skip over any input bytes that precede the first 0x00

000001:

9.

10.while

11.// ensures that we progress o

ver bad data

12.

13.// skip this initial code

14.

15.

16.// from now on

17.

18.//

19.//将起始标志也保存到目的缓冲区中

20.//

21.if

22.// Include a start code in the output:

23.

24.

25.

26.//

27.//保存所有数据，直至遇到起始标志，或者文件结束符。需要注意NALU中

的第一个字节，因为它指示了NALU的类型

28.//

29.// Then save everything up until the next 0x00000001 (4 by

tes) or 0x000001 (3 bytes), or we hit EOF.

30.// Also make note of the first byte, because it contains t

he "nal_unit_type":

31.

32.if

33.//

34.//已经设置了文件结束标志，将剩下的数据保存也来即可

35.//

36.// We hit EOF the last time that we tried to parse this

data,

37.// so we know that the remaining unparsed data forms a c

omplete NAL unit:

38.

39.if void// forces an

other read, which will cause EOF to get handled for real this time

40.#ifdef DEBUG

41."This NAL unit (%d bytes) ends with EOF\

n"

42.#endif

43.if return

44.//将第一个字节保存下来，其指示了

NALU的类型

45.

46.

47.while

48.

49.

50.else

51.

52.//将第一个字节保存下来

53.//

54.//将下一个起始符号乊前的数据都保存下来

55.//

56.while

57.// We save at least some of "next4Bytes".

58.if//一次可以保存4个字节,

并不需要一个一个字节对比，除非到了结尾

59.// Common case: 0x00000001 or 0x000001 definitely doesn'

t begin anywhere in "next4Bytes", so we save all of it:

60.

61.

62.else

63.// Save the first byte, and continue testing the rest:

64.

65.

66.

67.

68.

69.// Assert: next4Bytes starts with 0x00000001 or 0x000001

, and we've saved all previous bytes (forming a complete NAL u

nit).

70.// Skip over these remaining bytes, up until the start o

f the next NAL unit:

71.if

72.

73.else

74.

75.

76.

77.

78.

79.

80.#ifdef DEBUG

81."Parsed %d-byte NAL-unit (nal_ref_idc: %d,

nal_unit_type: %d (\"%s\"))\n"

82.

83.#endif

84.

85.

86.//

87.//下面根据NALU的类型来作具体的分析

88.//

89.switch

90.case// Supplemental enhancement information (SEI)

91.

92.// Later, perhaps adjust "fPresentationTime" if we saw a "

pic_timing" SEI payload??? #####

93.break

94.

95.case// Sequence parameter set (序列参数集)

96.//

97.//保存一份SPS的副本到H264VideoStreamFramer中，后面的pps也需

要保存，sps中可能还包含了帧率信息

98.//

99.// First, save a copy of this NAL unit, in case the downst

ream object wants to see it:

100.

101.

102.

103.// Parse this NAL unit to check whether frame rate informa tion is present:

104.

105.

106.if

107.

//sps中包含了帧率信息

108.#ifdef DEBUG

109."Set frame rate to %f fps\n"

110.if

111."\tWARNING: \"fixed_frame_rate_flag\"

was not set\n"

112.

113.#endif

114.else//sps中不包含帧率信息，则使用source中设置的默认帧

率

115.#ifdef DEBUG

116."\tThis \"Sequence Parameter Set\" NAL u nit contained no frame rate information, so we use a default f rame rate of %f fps\n"

117.#endif

118.

119.break

120.

121.case// Picture parameter set (图像参数集PPS)

122.// Save a copy of this NAL unit, in case the downstream ob ject wants to see it:

123.

124.

125.

126.

127.//设置当前的时间128.#ifdef DEBUG

129.long long

130.

131."\tPresentation time: %lu.%06u\n"

132.#endif

133.

134.//

135.//如果这个NALU是一个VCL NALU(即包含的是视频数据)，我们需要扫描下一个NALU的起始符，

136.//以判断这个NALU是否是当前的"access unit"(这个"access unit"应该可以理解为一帧图像帧吧)。

137.//我们需要根据这个信息去指明何时该增加

"fPresentationTime"(RTP 打包时也需要根据这个信息，决定是否设置"M"

位)。

138.//

139.// If this NAL unit is a VCL NAL unit, we also scan the st art of the next NAL unit, to determine whether this NAL unit

140.// ends the current 'access unit'. We need this informati on to figure out when to increment "fPresentationTime".

141.// (RTP streamers also need to know this in order to figur

e out whether or not to set the "M" bit.)

142.// until we lea rn otherwise

143.if

144.// There is no next NAL unit, so we assume that this one ends the current 'access unit':

145.

146.else

147.const

// Would need to include type 20 for SVC and MVC ##### 148.if

149.

150.

151.

152.

153.if

154.//下一个NALU不是VCL的，当前的"access unit"结束了

155.// The next NAL unit is not a VCL; therefore, we assume that this NAL unit ends the current 'access unit':

156.#ifdef DEBUG

157."\t(The next NAL unit is not a VCL)\n"

158.#endif

159.

160.else

161.//下一个NALU也是VCL的，还需要检查一下它们是不是属于同一个"access unit"

162.// The next NAL unit is also a VLC. We need to examine it a little to figure out if it's a different 'access unit'. 163.// (We use many of the criteria described in section 7.4 .1.2.4 of the H.264 specification.)

164.

165.

166.if

167.// IdrPicFlag differs in value

168.#ifdef DEBUG

169."\t(IdrPicFlag differs in value)\n"

170.#endif

171.

172.else if

173.// nal_ref_idc differs in value with one of the nal_re f_idc values being equal to 0

174.#ifdef DEBUG

175."\t(nal_ref_idc differs in value with one of the nal_ref_idc values being equal to 0)\n"

176.#endif

177.

178.else if

179.

180.// Both this and the next NAL units begin with a "slic e_header".

181.// Parse this (for each), to get parameters that we ca n compare:

182.

183.// Current NAL unit's "slice_header":

184.

185.

186.

187.

188.

189.// Next NAL unit's "slice_header":

190.#ifdef DEBUG

191." Next NAL unit's slice_header:\n"

192.#endif

193.

194.sizeof

195.

196.

197.

sizeof

198.

199.

200.if

201.// frame_num differs in value

202.#ifdef DEBUG

203."\t(frame_num differs in value)\n"

204.#endif

205.

206.else if

207.// pic_parameter_set_id differs in value

208.#ifdef DEBUG

209."\t(pic_parameter_set_id differs in value)\n"

210.#endif

211.

212.else if

213.// field_pic_flag differs in value

214.#ifdef DEBUG

215."\t(field_pic_flag differs in value) \n"

216.#endif

217.

218.else if