2009年11月16日 星期一

H.264 RTP Streaming

根據RFC3984以RTP 封裝H.264 raw data來作video streaming.

1.H.264 raw data
以00 00 01 或 00 00 00 01作為開頭(Start Code),接著是8 bit NALU
NALU的format
      +---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+

F : forbidden zero bit, 一定為0
NRI : nal_ref_idc, 表示資料的重要性, 00為最不重要
.
Type :nal_unit_type, H.264只定義1~23的範圍

一個H.264 raw data看起來像這樣
00 00 00 01 09 30 ......

2.RTP header
因為一個H.264 video frame資料的大小往往會在數k bytes到數十K bytes,
在傳送封包時就會將資料切割分別封裝,也因此需要加入一些額外的參數讓
接收端可以正確組合被分割的video frame.這也是RFC3984最主要的目的.

RTP header 中有三個參數要注意
timestamp : 以90KHz作為基準,以30 fps為例,timestamp遞增 90000 / 30.
實務上是以payload實際間隔時間作計算.同一個video frame的
            分割資料timestamp是相同的
sequence : 每個RTP封包sequence number都遞增.
mark bit : RTP封包封裝的是最後一個分割的video frame時mark bit 為 1.

2.Payload format
1~23 : Single NAL unit packet.

RFC3984使用了H.264 NALU中未定義的type 24~29 (相當於增加H.264 nal_unit_type定義)

24 : STAP-A 單一時間組合
25 : STAP-B 單一時間組合
26 : MTAP16 多個時間組合
27 : MTAP32 多個時間組合
28 : FU-A 分割資料
29 : FU-B 分割資料

比較常見的是28,29.

3.Single NAL unit packet
當資料少於MTU的大小就用此方式封裝.
H.264 raw data foramt 為 [Start code][NALU][Raw Data]
封裝時去掉Start Code即可.Format 如下
[RTP Header][NALU][Raw Date]

4.FU-A (Fragmentation unit)
當資料大於MTU以此方式分割
H.264 raw data foramt 為 [Start code][NALU][Raw Data]
去掉[Start code],[NALU],以不超過MTU大小分割[Raw Data]
以NALU產生FU indicator, FU Header.

RFC 3984

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 14. RTP payload format for FU-A

The FU indicator octet has the following format:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+

The FU header has the following format:

+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|R| Type |
+---------------+

[FU indicator] = (NALU & 0x60) | 28;
[FU Header] = (start << 7) | (end << 6) | (NALU & 0x1f);

format如下

[RTP Header][FU indicator][FU header][Raw Data Part 0]
[RTP Header][FU indicator][FU header][Raw Data Part 1]
[RTP Header][FU indicator][FU header][Raw Data Part 2]
...

5.FU-B (Fragmentation unit)

RFC 3984

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | DON |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 15. RTP payload format for FU-B

format如下
[RTP Header][FU indicator][FU header][DON][Raw Data Part 0]
[RTP Header][FU indicator][FU header][DON][Raw Data Part 1]
[RTP Header][FU indicator][FU header][DON][Raw Data Part 2]
...

6.Encoding sample code with TI DSP

/* Write the encoded frame to the network */
if (Buffer_getNumBytesUsed(hOutBuf)) {
struct iovec data[3];
rtp_hdr_t rtp;
rtp.version = 2;
rtp.p = 0;
rtp.x = 0;
rtp.cc = 0;
rtp.m = 0;
rtp.pt = 96;

rtp.seq = htons( sequence );
rtp.ts = htonl( timestamp );

rtp.ssrc = 10;

data[0].iov_base = &rtp;
data[0].iov_len = sizeof(rtp_hdr_t);

unsigned char *ptr = (unsigned char *)Buffer_getUserPtr(hOutBuf);
size_t len = Buffer_getNumBytesUsed(hOutBuf);
size_t mtu = DEFAULT_MTU;

/* Skip NAL Start Code 00 00 00 01 */
unsigned char nalType = ptr[4] & 0x1f;
//printf("Processing Buffer with NAL TYPE=%d\n", nalType);

if(len < (mtu - sizeof(rtp_hdr_t))) {
/* Remove NAL Start Code 00 00 00 01 */
data[1].iov_base = (void *)ptr + 4;
data[1].iov_len = len - 4;

//printf("NAL Unit fit in one packet size=%d\n", len);
/* only set the marker bit on packets containing access units */
if (IS_ACCESS_UNIT (nalType))
rtp.m = 1;

writev(sfd, data, 2);

sequence++;
} else {
int start = 1, end = 0;
/* We keep 2 bytes for FU indicator and FU Header */
unsigned payload_len = mtu - sizeof(rtp_hdr_t) - 2;
unsigned char nalHeader = ptr[4];

/* Remove NAL Start Code 00 00 00 01 and NAL Type*/
ptr += 5;
len -= 5;

//printf("Using FU-A fragmentation for data size=%d\n", len);

while(end == 0) {
unsigned char fu[2];
payload_len = len < payload_len ? len : payload_len;
if (payload_len == len)
end = 1;

if (IS_ACCESS_UNIT (nalType))
rtp.m = end;

/* FU indicator */
fu[0] = (nalHeader & 0x60) | 28;
/* FU Header */
fu[1] = (start << 7) | (end << 6) | (nalHeader & 0x1f);

rtp.seq = htons( sequence );

data[1].iov_base = fu;
data[1].iov_len = 2;

data[2].iov_base = (void *)ptr;
data[2].iov_len = payload_len;

writev(sfd, data, 3);
start = 0;

ptr += payload_len;
len -= payload_len;

sequence++;
}
}
if (Time_delta(hTimestamp, &time) < 0) {
printf("Failed to get timer delta\n");
goto cleanup;
}

//printf("Read time: %uus\n", (Uns)time);
timestamp += (unsigned int)(0.09 * time); /*15 fps : 90000 / 15*/
}


2009年11月3日 星期二

F3F 20091018 龍蟠



OMAP3530 DSP Howto ( Really works by DSPLINK)

搞了一段很長的時間終於把DSP功能弄起來了,該是回頭紀錄一下"眉角"了.

1.Before touch anything
目前有兩種方式可以控制DSP,一個是TI DVSDK提供的DSPLINK,另一個則是DSPBRIDGE.
以我從google搜尋之後的了解DSPBRIDGE目前是沒有繼續maintain.
DSPLINK是TI比較建議的方式.

所搭配的media library為
DSPLINK + gstreamer
DSPBRIDGE + openmax + gstreamer
http://felipec.wordpress.com/2009/10/13/new-project-gst-dsp-with-beagleboard-demo-image/
http://felipec.wordpress.com/2008/12/12/gst-openmax-demo-on-the-beagleboard/

這邊我選擇DSPLINK控制DSP

2.Software

dvsdk_3_00_02_44
gst-ti-plugin-full-1.00.02
gst-omapfb

Build and install to target root file system (過程就省略了)

可參考以下作法
http://ossie.wireless.vt.edu/trac/wiki/BeagleBoard_CodecEngine

安裝目錄
DVSDK : /opt/dvsdk
gstreamer : /opt

3.Setup
DSPLINK透過share memory的方式跟DSP交換資料所以有另一個driver叫cmemk.ko負責建立memory pool. 但這塊memory必須脫離kernel的掌控因此必須要空出一塊memory讓cmemk.ko使用.至於大小要看實際memory size跟DSP所需要的size來決定.

http://pixhawk.ethz.ch/wiki/tutorials/omap/dsplink/memorymap

依照以上說明修改以下檔案

dsplink_1_61_03/packages/dsplink/config/all/CFG_OMAP3530_SHMEM.c
dsplink_1_61_03/packages/dsplink/dsp/inc/DspBios/5.XX/OMAP3530/dsplink-omap3530-base.tci
dsplink_1_61_03/packages/dsplink/config/all/CFG_OMAP3530_SHMEM.c

我的target board有256MB DDR,因此記憶體配置如下

# 0x40200000    60 KB   CMEM (internal memory, see comment below)
# 0x80000000    200 MB   Linux
# 0x8C900000    16 MB   CMEM (external memory)
# 0x8D900000     2 MB   DSPLINK (MEM)
# 0x8DB00000     unused

kernel boot command 加上 mem=200M

For kernel 2.6.27

optargs=init=/init omapfb.video_mode=1280x1024MR-16@60 vram=12M omapfb.vram=4M,4M,4M omapfb.debug=1 omap-dss.def_disp=lcd omap-dss.debug=1 mem=200M

For kernel 2.6.29

setenv optargs 'init=/init omapdss.def_disp=dvi omapfb.mode=dvi:1280x1024MR-16@60 vram=12M omapfb.vram=0:4M,1:4M,2:4M mem=200M'

memory pool配置如下

insmod cmemk.ko phys_start=0x8C900000 phys_end=0x8D900000 pools=20x4096,10x131072,2x1048576,1x5250000,4x829440,2x691200

gstreamer需要設定target上的環境變數好讓系統找的到執行檔及plug-in

export PATH=$PATH:/opt/gstreamer/bin
export GST_PLUGIN_PATH=/opt/gstreamer/lib/gstreamer-0.10
export LD_LIBRARY_PATH=/opt/gstreamer/lib

3.Get DSP work

載入drivers

#cd /opt/dvsdk
#./loadmodules.sh
CMEMK module: built on Oct 30 2009 at 20:23:52
  Reference Linux version 2.6.28
  File /home/gigijoe/OMAP3530/dvsdk_3_00_02_44/linuxutils_2_24_02/packages/ti/sdo/linuxutils/cmem/src/module/cmemk.c
ioremap_nocache(0x8c900000, 16777216)=0xcf000000
allocated heap buffer 0xcf000000 of size 0x32c000
cmem initialized 6 pools between 0x8c900000 and 0x8d900000
DSPLINK Module (1.61.03) created on Date: Oct 30 2009 Time: 20:20:31

# lsmod
lpm_omap3530 8276 0 - Live 0xbf02c000
dsplinkk 114052 1 lpm_omap3530, Live 0xbf00b000
cmemk 24428 0 - Live 0xbf000000

測試omapfb, 應該會在畫面上看到彩色條紋.

gst-launch videotestsrc num-buffers=1000 ! omapfbsink

播放TI h.264 raw data sample

gst-launch --gst-debug-level=1 filesrc location="/media/dcim/100andro/ntsc.264" ! typefind ! TIViddec2 ! omapfbsink

Play avi file

gst-launch --gst-debug-no-color --gst-debug=TI*:2 filesrc location=/media/dcim/100andro/aris.mpeg ! typefind ! qtdemux name=demux demux.audio_00 ! queue max-size-buffers=8000 max-size-time=0 max-size-bytes=0 ! typefind ! TIAuddec1 ! audioconvert ! osssink demux.video_00 ! typefind ! TIViddec2 ! omapfbsink

Play mpeg4 file

gst-launch --gst-debug-no-color --gst-debug=TI*:2 filesrc location=/media/dcim/100andro/KenBlock-TopGear.mp4 ! typefind ! qtdemux name=demux demux.audio_00 ! queue max-size-buffers=8000 max-size-time=0 max-size-bytes=0 ! typefind ! TIAuddec1 ! audioconvert ! osssink demux.video_00 ! typefind ! TIViddec2 ! omapfbsink

Play H.264 mpeg4 file without mp3 audio

gst-launch filesrc location=/media/dcim/100andro/sany0014.mp4  ! typefind ! qtdemux name=demux  demux.video_00 ! typefind ! TIViddec2 ! omapfbsink

4.Debug

http://tiexpressdsp.com/wiki/index.php?title=Debugging_DSPLink_using_SET_FAILURE_REASON_prints

http://tiexpressdsp.com/index.php/Enabling_trace_in_DSPLink

一開始有一段時間kernel-2.6.29 DSP_init總是失敗

# ./messagegpp ./message.out 1000
========== Sample Application : Failure [0x8000800b] in [0x401] at line 522
MESSAGE ==========
Entered MESSFailure [0x8000802d] in [0x401] at line 544
AGE_Create ()
Failure [0x8000802d] in [0x401] at line 544
Failure [0x8000802d] in [0x401] at line 544
ISR_Install:445
request_irq 28
request_irq failed with error: -16
Failure [0x80008008] in [0x502] at line 459
Failure [0x80008008] in [0x80a] at line 824
Failure [0x80008008] in [0x80a] at line 1061
Failure [0x80008008] in [0x801] at line 597
Failure [0x80008008] in [0x701] at line 370
 DSP_init status [0x80008008]
Assertion failed ((isrObj!= NULL) && (ISR_InstalledIsrs [isrObj->dspId][isrObj->irq] == isrObj)). File : /home/gigijoe/OMAP3530/dvsdk_3_00_02_44/dsplink_1_61_03/packages/dsplink/gp
p/src/../../gpp/src/osal/Linux/2.6.18/isr.c Line : 507
Failure [0x80008000] in [0x502] at line 515
Failure [0x80008008] in [0x300] at line 476
Failure [0x80008008] in [0x300] at line 563
PROC_attach () failed. Status = [0x80008008]
PROC_setup () failed. Status = [0x80008008]
Leaving MESSAGE_Create ()
Entered MESSAGE_Delete ()
Assertion failed (IS_VALID_MSGQ (msgqQueue)). File : msgq.c Line : 484
MSGQ_release () failed. Status = [0x8000800b]
Assertion failed (IS_VALID_MSGQ (msgqQueue)). File : msgq.c Line : 335
Leaving MESSAGE_Delete ()
====================================================

Trace DSPLINK driver發現request_irq 28 居然失敗了.很明顯IRQ被佔據了.
看看到底是誰
cat /proc/interrupts

           CPU0
 11:          0        INTC  prcm
 12:          1        INTC  DMA
 18:          0        INTC  sr1
 19:          0        INTC  sr2
 24:          0        INTC  omap-iommu.1, Omap 3 Camera ISP
 25:          1        INTC  OMAP DSS
 28:          0        INTC  omap-iommu.2
 56:        347        INTC  i2c_omap
 61:          0        INTC  i2c_omap
 72:          1        INTC  serial idle
 73:          1        INTC  serial idle
 74:         94        INTC  serial idle, serial
 77:          0        INTC  ehci_hcd:usb2
 83:          0        INTC  mmc0
 86:         14        INTC  mmc1
 92:          0        INTC  musb_hdrc
 93:          0        INTC  musb_hdrc
 95:        641        INTC  gp timer
160:          0        GPIO  mmc1
167:          0        GPIO  user
181:          8        GPIO  eth0
378:          0     twl4030  twl4030_usb
379:          0     twl4030  rtc0
384:          0     twl4030  mmc0

原來是iommu啊,OK,調整一下kernel config

Disable
Device Drivers->Multimedia Devices->Video Capture Adapters->OMAP 3 Camera Support
System Type->TI OMAP Implemenation->IOMMU Support

這樣就OK囉

其實還有一些疑慮沒有解決.
IOMMU 以 Shared IRQ 方式 request_irq而DSPLINK Driver request_irq 並不是.
因此造成問題,DSPLINK Driver是否應該以Shared IRQ方式request_irq呢?

Problem :

# gst-launch --gst-debug=TI*:2 v4l2src always_copy=FALSE num-buffers=100 ! video
/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! TIVidenc1 codecName=h264enc
 engineName=codecServer contiguousInputFrame=FALSE genTimeStamps=FALSE ! filesin
k location=/tmp/enc.264

(gst-launch-0.10:636): GLib-WARNING **: getpwuid_r(): failed due to unknown user id (0)
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:01.011901868   636    0x9ebb0 WARN             TIVidenc1 gsttividenc1.c:1432:gst_tividenc1_codec_start: error: failed to open codec engine "codecServer"

ERROR: from element /GstPipeline:pipeline0/GstTIVidenc1:tividenc10: failed to open codec engine "codecServer"

Additional debug info:
gsttividenc1.c(1432): gst_tividenc1_codec_start (): /GstPipeline:pipeline0/GstTIVidenc1:tividenc10
Execution ended after 382354731 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:01.016387951   636    0x9ebb0 WARN             TIVidenc1 gsttividenc1.c:1601:gst_tividenc1_encode_thread: error: failed to start codec

gst-launch-0.10: BufTab.c:440: BufTab_getNumBufs: Assertion `hBufTab' failed.
Aborted

重新compile ti-gstreamer就好了...

Problem:

ERROR: from element /GstPipeline:pipeline0/GstTIViddec2:tividdec20: Failed to determine target board

這個問題是因為所使用的OMAP3530 board不是TI support的.
自己修改來support吧
Get board name first
# cat /proc/cpuinfo
Processor       : ARMv7 Processor rev 3 (v7l)
BogoMIPS        : 499.92
Features        : swp half thumb fastmult vfp edsp thumbee neon vfpv3
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x1
CPU part        : 0xc08
CPU revision    : 3

Hardware        : OMAP3 Thunder Board
Revision        : 0020
Serial          : 0000000000000000

So, the board name is OMAP3 Thunder Board

Edit dmai_2_05_00_04/packages/ti/sdo/dmai/linux/Cpu.c

...

    }
    else if ((strcmp(valBuf,"OMAP3EVM Board") == 0) ||
            (strcmp(valBuf, "OMAP3 EVM") == 0) ||
            (strcmp(valBuf, "OMAP3 Beagle Board") == 0) ||
            (strcmp(valBuf, "OMAP3 Thunder Board") == 0)) {  <<-- Add here
            *device = Cpu_Device_OMAP3530;
    }
    else {
        Dmai_err0("Unknown Cpu Type!\n");
        return Dmai_EFAIL;
    }

    return Dmai_EOK;
}

...

Then rebuild again ...