ericssonresearch / openwebrtc-gst-plugins Goto Github PK

this project includes an implementation of the SCReAM algorithm, but it is not mentioned at the top README.md so is quite difficult to find.

(the SCReAM code seems behind in development, but that is a separate issue)

Please mention SCReAM on front README.md to ease discovering its existence.

E.g. this (first line from the C++ library implementation of SCReAM:

SCReAM (Self-Clocked Rate Adaptation for Multimedia) is a congestion control algorithm devised mainly for Video.

This would be needed for using videoconvert and videoscale properly on iOS, as it's impossible to allocate executable memory on iOS nowadays for Apple reasons.

https://bugzilla.gnome.org/show_bug.cgi?id=742843 is about this issue too. Basically orcc can output NEON assembly at build time already, but there are some problems with getting that to run and also with outputting something for orc programs that contain unsupported opcodes for the specific target.

Ideally we would change orcc to output backup C code plus inline assembly (or in a separate file is ok too), and then at runtime it selects which one to use. As that is still possible on iOS.

Hi, I would try androivideo libs for streaming embedded cameras but I'm to sure which is the right way to use it.
I run ndk-build in openwebrtc-gst-plugins/sys/androidvideo/video-source/jni/ setting GLIB_GST_PATH in Android.mk to the dir where I extracted openwebrtc-android-armv7-0.3.0.zip.
This is the message from ndk-build: http://pastebin.com/0ZycmwY2

Any suggestions?
Thanks.

Hi,

I am using screamqueue for testing purpose. The implementation for the sender and for the receiver part can be found respectively at: scream_sender and at scream_receiver.

For network simulation I use linux network namespaces, tc and qdisc for bandwidth changing. The base RTT is 200ms. I migrated the rtcp feedback setup and other code necessary for scream implementation from the current openwebrtc code (owr_transport_agent object)

Meanwhile I am testing it I have struggled the problem of insufficient MAX_TX_PACKETS due to the refresh interval of incoming_packets in the src_loop function, lead the program into a segfault after a certain amount of time and warning. I increased the MAX_TX_PACKETS value to 2000, but its still seems to be insufficient for a network channel with 3000kbit/s bandwidth.

Did I missed something in my implementation or do I need to change something?

videorepair considers the video repaired once an intra for a single slice has been received. This can result in e.g. 3/4 of the video being broken if using 4 slices.

Just a tracker issue to remind us of doing that sooner or later. Upstream GStreamer bug report: https://bugzilla.gnome.org/show_bug.cgi?id=744582

With test-data-channel test application:

(lt-test-data-channel:19162): GStreamer-WARNING **: gstpad.c:4188:gst_pad_push_data:<sctpenc_1:src> Got data flow before stream-start event
[and follow up errors from that one]

We build openwebrtc-gst-plugins in WebKit project with jhbuild
and got the following build error:

In file included from gstvideorepair.c:30:
In file included from ./gstvideorepair.h:29:
In file included from /home/webkit/WebKit/WebKitBuild/DependenciesEFL/Root/include/gstreamer-1.0/gst/gst.h:88:
/home/webkit/WebKit/WebKitBuild/DependenciesEFL/Root/include/gstreamer-1.0/gst/gstcompat.h:54:38: error: equality comparison with extraneous parentheses [-Werror,-Wparentheses-equality]
if (((((GstPad_)(pad))->direction) == GST_PAD_SRC))
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
/home/webkit/WebKit/WebKitBuild/DependenciesEFL/Root/include/gstreamer-1.0/gst/gstcompat.h:54:38: note: remove extraneous parentheses around the comparison to silence this warning
if (((((GstPad_)(pad))->direction) == GST_PAD_SRC))
~ ^ ~
/home/webkit/WebKit/WebKitBuild/DependenciesEFL/Root/include/gstreamer-1.0/gst/gstcompat.h:54:38: note: use '=' to turn this equality comparison into an assignment
if (((((GstPad*)(pad))->direction) == GST_PAD_SRC))
^~
=
1 error generated.

line54: if (GST_PAD_IS_SRC (pad))

Of course this warning can be silenced in gstreamer with fixing this line, but
"if GST_PAD_IS_SRC (pad)" would be very ugly. Additionally Wparentheses-equality
isn't a useful warning at all, it would be better to simply suppress it.

I'm going to suppress this warning in WebKit - https://bugs.webkit.org/show_bug.cgi?id=145837,
but it would be great if we can fix it in the upstream too.

Just a tracker issue to remind us of doing that sooner or later. Upstream GStreamer bug report: https://bugzilla.gnome.org/show_bug.cgi?id=737786

These are some broad thoughts I had on the erdtls plugins and their usage. Also including some comments by @Rugvip (who can correct me if I got these wrong).

1. Clients should use ErDtlsAgent as the object to be set on erdtls* elements In my opinion, this is better than a global hash table (and @Rugvip disagrees, as he finds the current mechanism to be easier to use by clients).
2. ErDtlsAgent should not expose API to provide its context, but instead should offer API to instantiate an ErDtlsConnection (this makes the tight coupling between the two clear, without having internal API exposing implementation details (i.e. the SSL context). In the long term, ErDtlsAgent and ErDtlsConnections can become abstract classes from which specific implementations can be derived (we have openssl now, we could add gnutls).
3. erdtlsenc and erdtlsdec should be merged into a single erdtlsconnection element. The two elements cannot be used independently of the other, and thus there should be a single element to manage both directions of the connection. The element would have always pads for the DTLS send and receive side, and request pads for applications to access send/receive plaintext data (request pads since they may not always explicitly be interested in both directions).
4. erdtlssrtpenc and erdtlssertpdec should be merged into a singl erdtlssrtpbin. Once the erdtlsenc/dec pair are merged, the SRTP bins on top of them will also need to be merged since the two can't exist independently, and share a single DTLS connection. Some complexity is added by the fact that RTCP may or may not be muxed, and this must be handled appropriately. I've illustrated what the bins might look like at: http://arunraghavan.net/downloads/misc/dtls-refactor.txt
5. Evaluate using the srtp-encode/decoder interface on rtpbin with the erdtlssrtpbin -- this might make it easier to handle DTLS/SRTP on the application side, but does need some thought for all the various special cases (optional presence of DTLS, RTCP muxing, etc.). Moreover, making the DTLS data available, as well as making sure both send and receive channels are created even in a unidirectional stream might pose some additional challenges.

This project includes an implementation of the SCReAM - awesome!!!

It seems, however, that the implementation is somewhat behind the C++ library implementation at https://github.com/EricssonResearch/scream. I do understand that this is highly experimental code, and imagine that the main focus is therefore on development and consumption by others is not a high priority (if a priority/possibility at all). In any case, I suggest to add a (brief, at least) documentation on the status of this module - and if in a usable state then a concrete example.

Thanks for the exciting work you are doing in this field!

Hi guys, i'm fighting with that to understand why i get this error everytime i try to compile for android.

see below the LIBS variable info from config.log

LIBS='-lxml2 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../gstreamer/lib -lgstreamer-1.0 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../glib/lib -lglib-2.0 -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lgstbase-1.0 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../gst-plugins-base/lib -lgstvideo-1.0 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../glib/lib -lgio-2.0 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../orc-0.4.22/lib -lorc-0.4 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../gstreamer/lib -lgstreamer-1.0 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../glib/lib -lglib-2.0 -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../libffi/lib -lffi -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../libusrsctp/lib -lusrsctp -L/Users/michael.tourneur/openwebrtc/scripts/dependencies/out/arm-linux-androideabi/openwebrtc-gst-plugins/../openssl-1.0.2/lib -lssl -lcrypto -lintl -liconv -lz'

as u can see the path to lib usrsctp if on the path, i update the code today with the new update but still has the error EricssonResearch/openwebrtc-build-scripts@3bacd35

any idea?

In case of the ercolorspace plugin, the sink/source pad capabilities are currently:

• Input (sink pad): BGRA, RGBA, I420, NV21, NV21.
• Output (source pad): BGRA, RGBA, I420.

The following transformations are currently implemented:

• NV12 -> BGRA, RGBA, I420
• NV21 -> BGRA, RGBA, I420
• I420 -> BGRA, RGBA

The following transformation are possibly missing and causing issues:

• BGRA -> (BGRA), RGBA, I420
• RGBA -> (RGBA), BGRA, I420
• I420 -> (I420)

Testing the current OpenWebRTC + Bowser master built with Cerbero on an iPad Mini 1 with iOS 8.1.3 yields the following error:

==== Error message start ====
Error in element source-video-convert-1.
Error: Internal GStreamer error: code not implemented.  Please file a bug at http://bugzilla.gnome.org/enter_bug.cgi?product=GStreamer.
Debugging info: src/gstercolorspace.c(340): gst_ercolorspace_transform_frame (): /GstPipeline:transport-agent-1/GstBin:source-bin-1/GstERColorspace:source-video-convert-1:
colorspace conversion failed: unsupported formats
==== Error message stop ====
==== Error message start ====
Error in element source-1.
Error: Internal data flow error.
Debugging info: gstbasesrc.c(2943): void gst_base_src_loop(GstPad *) (): /GstPipeline:transport-agent-1/GstBin:source-bin-1/GstInterVideoSrc:source-1:
streaming task paused, reason not-negotiated (-4)
==== Error message stop ====
==== Error message start ====
Error in element source-queue-1.
Error: Internal data flow error.
Debugging info: gstqueue.c(873): gboolean gst_queue_handle_sink_event(GstPad *, GstObject *, GstEvent *) (): /GstPipeline:transport-agent-1/GstBin:source-bin-1/GstQueue:source-queue-1:
streaming task paused, reason not-negotiated (-4)
==== Error message stop ====

Also see EricssonResearch/openwebrtc#168