bugBino - Bugs: bug #33368, Add support for hardware...

 
 

bug #33368: Add support for hardware accelerated video decoding

Submitted by:  Martin Lambers <marlam>
Submitted on:  Sun 22 May 2011 08:51:42 AM UTC  
Votes:  51  
 
Severity: 1 - WishStatus: Wont Fix
Privacy: PublicAssigned to: None
Open/Closed: Closed

Add a New Comment(Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

(Jump to the original submission Jump to the original submission)

Sat 03 Jun 2017 12:56:40 PM UTC, comment #16:

When last I looked, there was no documentation whatsoever on how to implement this on top of FFmpeg, and everything was a system- and hardware dependent mess. I have no plans to deal with that.

Martin Lambers <marlam>
Project Administrator
Sun 06 Jan 2013 06:39:05 PM UTC, comment #15:

This experimental patch will limit video decoding to 1 thread if crystalhd is available, other crystalhd users please test and report, vdpau users please adapt:

diff --git a/src/media_object.cpp b/src/media_object.cpp
index 84e1cd6..896f2a2 100644
--- a/src/media_object.cpp
+++ b/src/media_object.cpp
@@ -876,8 +876,8 @@ void media_object::open(const std::string &url, const device_request &dev_reques
{
_ffmpeg->format_ctx->streams[i]->discard = AVDISCARD_ALL; // ignore by default; user must activate streams
AVCodecContext *codec_ctx = _ffmpeg->format_ctx->streams[i]->codec;
- AVCodec *codec = (codec_ctx->codec_id == CODEC_ID_TEXT
- ? NULL : avcodec_find_decoder(codec_ctx->codec_id));
+ AVCodec *codec = (codec_ctx->codec_id == CODEC_ID_TEXT ? NULL : codec_ctx->codec_id == CODEC_ID_H264 ?
+ avcodec_find_decoder_by_name ( "h264_crystalhd" ) : avcodec_find_decoder(codec_ctx->codec_id));
// XXX: Sometimes the reported width and height for a video stream change after avcodec_open(),
// but the original values seem to be correct. This seems to happen mostly with 1920x1080 video
// that later is reported as 1920x1088, which results in a gray bar displayed at the bottom of
@@ -889,7 +889,12 @@ void media_object::open(const std::string &url, const device_request &dev_reques
{
// Activate multithreaded decoding. This must be done before opening the codec; see
// http://lists.gnu.org/archive/html/bino-list/2011-08/msg00019.html
- codec_ctx->thread_count = video_decoding_threads();
+ // h.264 Hardware Codecs like vdpau don't support >1 decoding threads, even if they claim like crystalhd,
+ // support may be broken or may interfere with openGL/DRI GPU output system m/futexex and no crystalhd
+ // support in and not every user or distro is using ffmpeg-mt libraries, and we don't need 2 >= SSE2
+ // cpu cores for decoding anyway if hardware codec offloading is available,
+ // vdpau users change to or add "h264_vdpau".
+ codec_ctx->thread_count = (codec->name == "h264_crystalhd" ? 1 : video_decoding_threads());
// Set CODEC_FLAG_EMU_EDGE in the same situations in which ffplay sets it.
// I don't know what exactly this does, but it is necessary to fix the problem
// described in this thread: http://lists.nongnu.org/archive/html/bino-list/2012-02/msg00039.html

Anonymous
Sun 06 Jan 2013 05:12:48 PM UTC, comment #14:

Tweaking the -threads option in mplayer I found out

bino: [inf] Using up to 2 threads for decoding.

and CAP_DR1

maybe the problem triggering the futex latency showed up in Latencytop with h264_crystalhd:

bino-chd: [wrn] Video: delay 1.018 seconds/24.4078 frames; dropping next frame.
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 205529

bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 199
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] sps_id out of range
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 428
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 454
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 454
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] brainfart cropping not supported, cropping disabled (right: 2, bottom: -1)
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 459

bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x4de27e0] missing picture in access unit with size 153190
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x4de27e0] missing picture in access unit with size 59836
bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x4de27e0] missing picture in access unit with size 59727

bino-chd: [err] FFmpeg: [h264_crystalhd @ 0x1a08800] missing picture in access unit with size 144341
bino-chd: [wrn] FFmpeg: [h264_crystalhd @ 0x1a08800] CrystalHD: Input buffer full

I try to limit Bino to one decoding thread, so ideally one cpu core is used for h.264 decoding offloaded to crystalhd and the other core for 3D SBS -> Anaglyph processing.

The broadcom driver now runs fine on Linux 3.2 with mplayer, has got issues with bleeding egde kernels 3.6, 3.7.

Still no reports from vdpau nividia and intel va-api GPU accelerator users?
Please try the decoder force patch above with "h264_vdpau" or the va-api decoder and report, thank You :-)

Anonymous
Tue 01 Jan 2013 05:00:26 PM UTC, comment #13:

Analyzing Bino reniced -20 with Latencytop I got slow motion movie instead of black picture with 1080p/i and it shows up that it consumes up to 50% cpu for decoding with "Application requested delay" using (hrtimer) sleep(), some mutex and is throttling the GPU waiting for something and Memory usage grows up to 30%,

but that cannot be caused by the crystalhd decoder, only 5% latency and 5ms average processing time measured with Latencytop.

Can anyone confirm this using h264_vdpau instead of h264_crystalhd in my patch above, please? Pentium M with ssse3 support but only a i915 GPU here.
You need to rebuid Your kernel with latency measuring debug support for this.

So according to my experiences while Bino's 3D code is very fast, mplayer's decoding is twice as fast (with gl output), so there seems to be a slowpath bottleneck in the decoder -> GPU path

which has to be fixed first before introducing hardware codec/accelerator support.

Bino devs please have a look at the corresponding mplayer codec -> opengl code and maybe in Media Player Classic - Home Cinema (MPC-HC) open source windows player, which runs at <10% cpu usage by decoding running crystalhd on a intel core2duo i965GM system but this system has a Linux pci-E driver issue with the BCM70012 I've to fix first or have to wait for the ordered BCM70015
# lspci -vv -s 03:00.0 |grep DevSta
DevSta: CorrErr+ UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
before I can test Bino on that machine.

Anonymous
Sun 30 Dec 2012 04:03:57 PM UTC, comment #12:

Here's a little experimental patch which makes bino choose crystalhd hardware dekoder or h264_vdpau to change for vdpau people:

diff --git a/src/media_object.cpp b/src/media_object.cpp
index 84e1cd6..c6e52cc 100644
--- a/src/media_object.cpp
+++ b/src/media_object.cpp
@@ -876,8 +876,8 @@ void media_object::open(const std::string &url, const device_request &dev_reques
{
_ffmpeg->format_ctx->streams[i]->discard = AVDISCARD_ALL; // ignore by default; user must activate streams
AVCodecContext *codec_ctx = _ffmpeg->format_ctx->streams[i]->codec;
- AVCodec *codec = (codec_ctx->codec_id == CODEC_ID_TEXT
- ? NULL : avcodec_find_decoder(codec_ctx->codec_id));
+ AVCodec *codec = (codec_ctx->codec_id == CODEC_ID_TEXT ? NULL : codec_ctx->codec_id == CODEC_ID_H264 ?
+ avcodec_find_decoder_by_name ( "h264_crystalhd" ) : avcodec_find_decoder(codec_ctx->codec_id));
// XXX: Sometimes the reported width and height for a video stream change after avcodec_open(),
// but the original values seem to be correct. This seems to happen mostly with 1920x1080 video
// that later is reported as 1920x1088, which results in a gray bar displayed at the bottom of

For me I've rejected a vaapi oder vdpau solution, since both APIs are X-Server dependant and there're still many framebuffer and embedded systems usecases.

Audiodecoding works now with libavcodec54 from ffmpeg 1.0.1, was a build config mismatch.

Anonymous
Sat 29 Dec 2012 04:14:24 AM UTC, comment #11:

... linux-media devlist.

Anonymous
Sat 29 Dec 2012 04:13:26 AM UTC, comment #10:

Major crystalhd driver problems occured which have to be fixed first, diverting to -unavailable-

Anonymous
Fri 28 Dec 2012 09:27:12 PM UTC, comment #9:

Correction: 70012 cannot handle 1080p according to specs.

Anonymous
Fri 28 Dec 2012 12:54:01 AM UTC, comment #8:

Tried with a 720p(?) media it's not getting better, maybe the problem is, the BCM70012 can't handle interlaced media:

15987 schorpp 20 0 375m 143m 28m S 22,4 16,4 0:20.43 bino
2954 root 20 0 163m 15m 9848 S 17,8 1,7 57:20.79 Xorg

Dec 28 01:44:17 vdr1 kernel: [81866.889214] crystalhd 0000:02:00.0: list_index:0 rx[433] rxtot[5026] Y:2 UV:0 Int:8 YDnSz:0 UVDnSz:0
Dec 28 01:44:17 vdr1 kernel: [81867.031843] crystalhd 0000:02:00.0: MISSING 4 PICTURES
Dec 28 01:44:17 vdr1 kernel: [81867.103139] crystalhd 0000:02:00.0: MISSING 4 PICTURES
Dec 28 01:44:17 vdr1 kernel: [81867.170201] crystalhd 0000:02:00.0: MISSING 4 PICTURES
Dec 28 01:44:17 vdr1 kernel: [81867.178990] crystalhd 0000:02:00.0: list_index:0 rx[434] rxtot[5042] Y:2 UV:0 Int:8 YDnSz:0 UVDnSz:0
Dec 28 01:44:17 vdr1 kernel: [81867.248240] crystalhd 0000:02:00.0: MISSING 2 PICTURES
Dec 28 01:44:17 vdr1 kernel: [81867.255573] crystalhd 0000:02:00.0: list_index:0 rx[435] rxtot[5044] Y:2 UV:0 Int:8 YDnSz:0 UVDnSz:0
Dec 28 01:44:19 vdr1 kernel: [81869.324293] crystalhd 0000:02:00.0: FETCH TIMEOUT
Dec 28 01:44:20 vdr1 kernel: [81870.174278] crystalhd 0000:02:00.0: Closing user[0] handle via ioctl with mode 1417200
Dec 28 01:44:20 vdr1 kernel: [81870.575815] crystalhd 0000:02:00.0: Opening new user[0] handle
Dec 28 01:44:23 vdr1 kernel: [81873.099530] start_capture: pause_th:12, resume_th:5
Dec 28 01:44:23 vdr1 kernel: [81873.306895] crystalhd 0000:02:00.0: [FMT CH] PIB:0 0 420 2 500 2d0 500 0 9 0
Dec 28 01:44:23 vdr1 kernel: [81873.415749] crystalhd 0000:02:00.0: MISSING 3 PICTURES
Dec 28 01:44:24 vdr1 kernel: [81873.888316] crystalhd 0000:02:00.0: Closing user[0] handle via ioctl with mode 1417200
Dec 28 01:44:24 vdr1 kernel: [81874.194675] crystalhd 0000:02:00.0: Opening new user[0] handle
Dec 28 01:44:26 vdr1 kernel: [81876.705679] start_capture: pause_th:12, resume_th:5
Dec 28 01:44:37 vdr1 vdr: [15683] frontend 2/0 lost lock on channel 76, tp 212480
Dec 28 01:44:37 vdr1 vdr: [15683] frontend 2/0 regained lock on channel 76, tp 212480
Dec 28 01:44:59 vdr1 kernel: [81909.012564] crystalhd 0000:02:00.0: Closing user[0] handle with mode 1417200

Anonymous
Fri 28 Dec 2012 12:17:21 AM UTC, comment #7:

For those with older slow machines, the BCM(9)70015 with scaling support to downscale HD SBS 3D to 720p and below before feeding bino maybe recommendable, ordered, testing in 2+ weeks.

Anonymous
Wed 26 Dec 2012 03:02:29 PM UTC, comment #6:

Just build ffmpeg with the software h.264 software codec disabled (--disable-decoder='h264,h264_vdpau,h264_vda' for mplayer -vc ffh264crystalhd ) ,
this will make bino using Your hardware codec:

$ bino -L debug
bino: could not connect to socket
bino: Datei oder Verzeichnis nicht gefunden
bino: [dbg] Ausnahme: Kann LIRC nicht initialisieren.
bino: [wrn] Kann LIRC nicht initialisieren.
bino: [dbg] 1 OpenAL devices available:
bino: [dbg] OpenAL Soft
bino: [dbg] LibASS: Raster: FreeType 2.4.9
bino: [dbg] LibASS: Shaper: FriBidi 0.19.2 (SIMPLE)
bino: [dbg] LibASS: Initialized
bino: [dbg] FFmpeg: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x9b914a0] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
bino: [dbg] FFmpeg: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x9b914a0] ISO: File Type Major Brand: isom
bino: [dbg] FFmpeg: [mov,mp4,m4a,3gp,3g2,mj2 @ 0x9b914a0] File position before avformat_find_stream_info() is 1825382689
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] CrystalHD Init for h264_crystalhd
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] CrystalHD: starting up
Running DIL (3.22.0) Version
DtsDeviceOpen: Opening HW in mode 0
Clock set to 180
Enable single threaded mode
Setting Color Mode to 1
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] CrystalHD: Init complete.
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] CrystalHD: decode_frame
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] CrystalHD: parser picture type 3
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] input "pts": 100000
bino: [dbg] FFmpeg: [h264_crystalhd @ 0x9b97f40] CrystalHD: Filling pipeline.
...

This will work with mplayer1+2 too, and saves the work to configure every frontend with -vc h264_vdpau, be cause the legacy h264 sw codec may take still precedence.

You need a recent libav version anyway because crystalhd maybe not the only codec flagged as experimental in libavcodec and so is only to enable by ffmpeg exe with -strict -2 in older libavcodec53, this method is done faster and more reliable than changing every frontend app for

Anonymous
Sat 22 Dec 2012 06:30:20 PM UTC, comment #5:

Codec acceleration is the codec library's job according to ISO OSI and not the job of frontend players like Bino, and it's already in:

# apt-cache depends libavcodec53
libavcodec53
Hängt ab von: libaacplus2
Hängt ab von: libavutil51
Hängt ab von: libc6
Hängt ab von: libcrystalhd3 <------------------
Hängt ab von: libdirac-encoder0
Hängt ab von: libfaac0
Hängt ab von: libgsm1
Hängt ab von: libmp3lame0
Hängt ab von: libopencore-amrnb0
Hängt ab von: libopencore-amrwb0
Hängt ab von: libopenjpeg2
Hängt ab von: libschroedinger-1.0-0
Hängt ab von: libspeex1
Hängt ab von: libtheora0
Hängt ab von: libva1 <--------------------
Hängt ab von: libvo-aacenc0
Hängt ab von: libvo-amrwbenc0
Hängt ab von: libvorbis0a
Hängt ab von: libvorbisenc2
Hängt ab von: libvpx1
Hängt ab von: libx264-124
Hängt ab von: libxvidcore4
Hängt ab von: zlib1g
Hängt ab von (vorher): multiarch-support
Kollidiert mit: libavcodec-extra-53
Kollidiert mit: <libavcodeccvs51>
Ersetzt: <libavcodeccvs51>
#
# apt-cache policy libavcodec53
libavcodec53:
Installiert: 7:0.10.3-dmo1
Installationskandidat: 7:0.10.3-dmo1
Versionstabelle:

      • 7:0.10.3-dmo1 0

500 http://debian.netcologne.de/debian-multimedia.org/ unstable/main i386 Packages

vdpau solutions are IMHO not recommended for open source environments due to issues of tainting proprietary Nvidia kernel drivers for full (proprietary) functionalities and problematic licensing.

Anonymous
Thu 30 Aug 2012 02:42:41 PM UTC, comment #4:

"The currently used software decoding is sometimes too slow, especially for full HD stereoscopic content.

With the existing FFmpeg decoder, the solution would be to tell FFmpeg to use VAAPI for hardware accelerated video decoding if available, and then tell VAAPI that we want the decoded video in an OpenGL-accessible form.

This is all possible, but unfortunately there is very little documentation.

See also:
http://www.freedesktop.org/wiki/Software/vaapi
http://www.splitted-desktop.com/~gbeauchesne/hwdecode-demos/" Martin Lambers

Searching at the FFMpeg libraries, I could find that it does HW decode via VAAPI. It does that throught the libavcodec (that is used in VLC,Mplayer,Kdenlive,ffdshow,Avidemux,Mencoder, etc)...

See more details at:
http://ffmpeg.org/doxygen/0.6/group__VAAPI__Decoding.html
http://ffmpeg.org/doxygen/0.6/group__Decoder.html

"Hi,

On Thu, 20 Aug 2009, Max Müller wrote:

> Is there anywhere a little example or a howto how libav is used
> together with libva? There seems to be (many?) differences about the
> usage from libav if hardware acceleration is required.


A very quick answer:
<http://www.splitted-desktop.com/~gbeauchesne/hwdecode-demos/>

- Look for ffmpeg files
- You have to initialize struct vaapi_context in your player
- You have to write AVCodecContext.{get_format,get_buffer,release_buffer}
+ MPlayer -> libvo/vo_vaapi.c to help further

This displays a static image. I will try to elaborate more tomorrow.

Regards,
Gwenole." (Gwenole Beauchesne schrieb)

André Madureira <andremadureira>
Thu 30 Aug 2012 01:12:09 PM UTC, comment #3:

Hello,

I think this a very important feature to be added because it really makes the difference between running or not a FULLHD 3D H-SBS video. Although I have a Quad Core system (8GB DDR3 RAM, Intel Quad Core 2.66 Ghz, nVidia Geforce 9600 GT 512 DDR3 VRAM), I can't handle smoothly FullHD Half-SBS 3D movies with BINO (this also happens with FullHD 2D movies) (this problem doesn't happen when I use 2D movie players like VLC, SMPLAYER or XBMC).

Therefore, I think that one good solution to this problem is using Hardware Acelleration (VDPAU for NVIDIA, VAAPI for Intel/ATI). Additionally, when I use SMplayer or VLC, I'm not using VDPAU (Nvidia HW driver), but FFMpeg-MT (as does BINO). And, like I said earlier, these decoding frame skipping and color issues don't happen in these programs. So, maybe Bino is not correctly implementing FFMpeg-MT to decode the video.

(VDPAU with FFMpeg Performance Test) As a test, I changed FFMpeg decoding from CPU to GPU VDPAU in SMplayer to check how it works and the result was not quite good at all (there were problems in the rendering like colors misrendered and block frame skipped). So, I think that FFMpeg is quite not good when using VDPAU HW. To assure that this is true, I used XBMC with VDPAU Active and there were no problems at all, the video was smooth and the processor use was low (5% - 10%). As a conclusion, as XBMC don't use FFMpeg-MT as decoder, I can verify that it does use a proper decoding system, and the XBMC Decoding System is better programmed to use VDPAU HW than the FFMpeg-MT.

So, The hardware acelleration is a very important feature, and should be implemented as soon as possible. Maybe, Bino could use GStreamer decoding packages to better achieve HW with VAAPI and VDPAU than using FFMpeg.

Thanks for your attention,

André Madureira <andremadureira>
Tue 06 Mar 2012 05:02:11 PM UTC, comment #2:

Sorry, wrong application. Stupid bug tracker :-(.

Tony Houghton <realh>
Tue 06 Mar 2012 04:48:59 PM UTC, comment #1:

I think the main problem is in the scaling, not the decoding. I've got an Acer Aspire Revo which is ION 2 with an Atom D525 I think (dual core, 1.83GHz). It can just about play low resolution Youtube videos in the browser window, but if I try to go fullscreen it crawls, even with only 360p (or even 240p I think) to decode. Adobe's supposed VDPAU support never seems to work so I was disappointed to find that gnash has this same scaling bug.

Perhaps the plugin interface doesn't allow for any acceleration, but surely in fullscreen mode gnash can use its own window and therefore at least use Xv or OpenGL for hardware scaling?

How about using gstreamer for the entire process from source to rendering? That can support VDPAU and VAAPI.

Otherwise, great job, gnash looks like it's now really a viable alternative to Adobe's non-free plugin if you have a decent CPU.

Tony Houghton <realh>
Sun 22 May 2011 08:51:42 AM UTC, original submission:

The currently used software decoding is sometimes too slow, especially for full HD stereoscopic content.

With the existing FFmpeg decoder, the solution would be to tell FFmpeg to use VAAPI for hardware accelerated video decoding if available, and then tell VAAPI that we want the decoded video in an OpenGL-accessible form.

This is all possible, but unfortunately there is very little documentation.

See also:
http://www.freedesktop.org/wiki/Software/vaapi
http://www.splitted-desktop.com/~gbeauchesne/hwdecode-demos/

Martin Lambers <marlam>
Project Administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach File(s):
   
   
Comment:
   

Attached Files
file #27154:  crystalhd.c.54~53.bpo.patch added by None (8KiB - text/x-diff - libavcodec54 crystalhd backport patch to 53 to avoid audio breakage of bino using 54, details on the mailing list.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -unavailable- added by andremadureira (Posted a comment)
  • -unavailable- added by andremadureira (Voted in favor of this item)
  • -unavailable- added by marlam (Submitted the item)
  •  

    Do you think this task is very important?
    If so, you can click here to add your encouragement to it.
    This task has 51 encouragements so far.

    Only logged-in users can vote.

     

    Please enter the title of George Orwell's famous dystopian book (it's a date):

     

     

    Follow 7 latest changes.

    Date Changed By Updated Field Previous Value => Replaced By
    Sat 03 Jun 2017 12:56:40 PM UTCmarlamStatusNone=>Wont Fix
      Open/ClosedOpen=>Closed
    Thu 27 Dec 2012 06:24:09 PM UTCNoneAttached File-=>Added crystalhd.c.54~53.bpo.patch, #27154
    Thu 30 Aug 2012 12:49:48 PM UTCandremadureiraCarbon-Copy-=>Added andremadureira
    Tue 06 Mar 2012 05:02:45 PM UTCrealhCarbon-CopyRemoved 87404=>-
    Tue 06 Mar 2012 05:00:31 PM UTCrealhCarbon-CopyRemoved realh=>-
    Tue 06 Mar 2012 04:48:59 PM UTCrealhCarbon-Copy-=>Added realh

    Back to the top


    Powered by Savane 3.1-cleanup1