IMSDroid video not working on HTC EVO about imsdroid HOT 68 CLOSED

The Wireshark traces are attached for H263, H263+ and H264 calls.

I have started to modify IMSDroid to support CIF video size. Perhaps I don't 
have all information about your network architecture because all responses sent 
from your client don't respect the specifications (RFC 3264 and 
draft-ietf-sipping-sip-offeranswer-12).

1. H263
  - IMSDroid offers H.263 (34) with QCIF=2
  - The response sent from your contains H.263(34) with QCIF=1;CIF=1, ... and H.263+ (96).
  ==> MUST not add new codec in it's offer
2. H.263+
  - IMSDroid offers H.263+(103) with CIF=2;QCIF=2
  - The response sent from your client contains H263+(96) and H263(34)
  ==> Two errors: MUST not add H263 and the payload type of H.263+ is incorrect
3. H264
  - IMSDroid offers H264 Base Profile 1.0 (104)
  - The response sent from your client contains H264 (96) with a profile-level-id=42000d
  ==> Two errors: The payload type is incorrect and the level does not match.

Do you have any idea why all responses are incorrect? 

[deleted comment]

The latest version (1.0.234) should allow decoding any video size. Tests have 
been done with Eyebeam. 
For encoding size, you can now choose the preferred size. For more info: 
http://code.google.com/p/imsdroid/wiki/Quick_Start#Video_Quality.

Thanks for the update. Will try it.
In the meantime, can you help figure out why the webcam not working on my HTC 
EVO? I looked at logs and it complains about "Invalid picture size requested: 
352x288".

This is very strange as the previous version only support QCIF (176x144) 
encoding (Bandwidth level was useless). As you have eclipse the best way is to 
run the application in debug mode and set a breakpoint in the prepare() 
function (VideoProducer.java) to see which size is requested by the native 
function. You can also change the bandwidth level (low<->medium) to switch from 
QCIF<->CIF to see if the problem persist.
Have you managed to get it to work with another SIP phone?

No sorry I confused you. I was using the latest version of your code to test 
with the bandwidth level set to high.
I also ran test calls to Bria and IMSDroid. The logs say it is failing at the 
setParameters() method when it is trying to open the default webcam.

08-09 14:24:21.783: DEBUG/QualcommCameraHardware(65): requested preview size 
768 x 432
08-09 14:24:21.783: DEBUG/QualcommCameraHardware(65): requested picture size 
352 x 288
08-09 14:24:21.783: ERROR/QualcommCameraHardware(65): Invalid picture size 
requested: 352x288
08-09 14:24:21.783: ERROR/QualcommCameraHardware(65): virtual android::status_t 
android::QualcommCameraHardware::set
Parameters(const android::CameraParameters&): 2945, rc = -22

I changed the setPictureSize() method in VideoProducer.java to 1024 x 768, then 
it was able to open the webcam and got video to going. But the call would only 
last for 10 secs or so and IMSDroid then crashed.

It's normal to crash. You should also change the frame allocation size.
From: float capacity = (float)(width*height)*1.5f/* (3/2) */;
To: float capacity = (float)(1024*768)*1.5f/* (3/2) */;

That fixed the 30 secs in-call crash. Now it is crashing during call hang up.
Also, heavy delay is occurring on the inbound video shown on the screen. Looks 
like 1024x768 is a bit too much? Do you know how to set the picture size to CIF 
without the Android platform complaining about invalid picture size? 

I confirm that 1024x768 is Toooo much for a mobile. You have the delay because 
the CPU overhead. You should only have delay when you start sending video. Is 
it right?
I'm looking into android source code to understand why this fail. According to 
your log the camera supports CIF (see line 243).
You can also use adb tool to see the "CPU usage".

I printed the webcam parameters out:
preview-size-values=1280x720,800x480,768x432,720x480,640x480,576x432,480x320,400
x240,384x288,352x288,320x240,272x272,240x240,240x160,176x144;
picture-size-values=3264x2448,3264x1952,2592x1952,2592x1552,2048x1536,2048x1216,
1600x1200,1280x960,1280x768,1024x768,640x480,640x384,512x384,400x400,272x272;
Now I am testing some of these combinations.

I tested with Bria and my initial finding is that everytime there will be some 
delay no matter what. I used setPreviewSize(176, 144), setPictureSize(272 , 
272) and bandwidth set to low for my testing.

Which SIP client you have tested with that gave you the least delay and best 
performance?

Very strange. I have done my tests with linphone, eyeabeam, xlite and ekiga.
The delay is on the remote or local video?
Which codec are you using? Do you have the same issue with all codecs?
What is the average CPU usage (with and without outgoing video)?

Its happening on the local video every time I did the "Send Video". CPU stats 
below:
User 62%, System 17%, IOW 0%, IRQ 1%
User 197 + Nice 0 + Sys 55 + Idle 56 + IOW 0 + IRQ 0 + SIRQ 6 = 314

  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 3728  63% S    23 195164K  29560K  fg app_8    org.doubango.imsdroid
  113  12% S    67 288736K  52964K  fg system   system_server
   65   1% S    13  43832K   5708K  fg media    /system/bin/mediaserver

I tested with H.263.

Without outgoing video, the CPU stats:
User 43%, System 40%, IOW 0%, IRQ 0%
User 136 + Nice 0 + Sys 125 + Idle 48 + IOW 0 + IRQ 0 + SIRQ 3 = 312

  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 3728  39% S    23 196300K  33248K  fg app_8    org.doubango.imsdroid
  113  28% S    67 288736K  52780K  fg system   system_server
  288   9% S     1      0K      0K  fg root     dhd_dpc
 3788   2% R     1    940K    416K  fg shell    top
   65   1% S    13  43832K   5708K  fg media    /system/bin/mediaserver
   76   1% S     8  10468K   1428K  fg root     /system/bin/iqd

Which audio codec is associated to the session?

Could you also disable fullscreen mode (Options->General) and give me CPU info 
(With and without encoding).

Without encoding:
User 46%, System 24%, IOW 0%, IRQ 1%
User 145 + Nice 0 + Sys 76 + Idle 86 + IOW 0 + IRQ 0 + SIRQ 5 = 312

  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 4533  41% S    24 196836K  30488K  fg app_8    org.doubango.imsdroid
  113  12% S    67 288544K  51752K  fg system   system_server
  288  12% S     1      0K      0K  fg root     dhd_dpc
   65   3% S    13  43832K   5772K  fg media    /system/bin/mediaserver

With encoding:
User 62%, System 33%, IOW 0%, IRQ 3%
User 279 + Nice 0 + Sys 148 + Idle 0 + IOW 0 + IRQ 0 + SIRQ 17 = 444

  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 4533  56% S    24 200804K  33400K  fg app_8    org.doubango.imsdroid
   65  11% S    17  52584K   6220K  fg media    /system/bin/mediaserver
  113   9% S    67 289568K  51756K  fg system   system_server


The application crashed everytime I used "send Video". You can take a look at 
the attached crash.log.

The crash happened during call release.

[deleted comment]

This is with fullscreen disabled?
On a HTC Hero (528MHz) I have respectively 27% and 48%. I don't know how you 
can have 41% and 50% on EVO(1GHz).
FYI: http://code.google.com/p/imsdroid/wiki/Video_Performance_Issues

For the crash try to comment the body of "surfaceDestroyed(SurfaceHolder 
holder)" function to see if it continue to happen.

I also tested it on HTC Hero
Full-screen mode: disabled
bandwidth: Low
audio codec: G.711u
video codec: H.263
setPictureSize(272, 272)
Test call made to: CounterPath Bria 2.5

Without encoding
User 44%, System 46%, IOW 0%, IRQ 0%
  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 2408  38% S    21 145180K  24384K  fg app_74   org.doubango.imsdroid

With encoding
User 70%, System 28%, IOW 0%, IRQ 0%
  PID CPU% S  #THR     VSS     RSS PCY UID      Name
 2408  66% S    21 149132K  26008K  fg app_74   org.doubango.imsdroid

Perhaps you can tell me the setup you got when you tested it on your HTC HERO 
with EyeBeam? Did you have to change the code to set the 272x272 picture size? 
I used a Sprint HTC HERO.

Which surfaceDestroyed() function you want me to comment out, VideoProducer or 
ScreenPresence?

you don't need to change the picture size. QCIF is already supported by Hero.
Try the test with QCIF and tell me what is the result?
According to Android source code, all devices should support QCIF and CIF. The 
problem with EVO is very strange. Video resizing (FFMpeg swscale()) is one of 
the function which use most CPU cycles.

The function to comment is from VideoProducer.java.

I tried imsdroid-1.0.234.apk on the HTC Hero. I set bandwidth to low, which 
means QCIF. I also disabled full-screen. When I tested it with EyeBeam, I got 
only one-way video because it complained about "Invalid picture size requested: 
176x144" when I did "Send Video". How did you get it to work without changing 
the picture size? I am lost here.

No idea. I'm using HTC Hero with Android 1.5.

Mine is Android-2.1. HTC EVO is Android-2.2. So that might be it?

In the comments 22 and 26: I said "HTC Hero" and not "HTC EVO". 
It was about CPU usage. You cannot consume more CPU on EVO(1GHz) than 
Hero(528Hz). Even on an old G1 H.263 call does take 66%. Only H.263 and Theora 
should take more than 50%.

[deleted comment]

Correction: Even on an old G1, H.263 call doesn't take 66%. Only H.264 and 
Theora should take more than 50%.

I also encounter this problem of web cam grab on my HTC EVO with Android 2.2. 
Did you finally solve this problem ?

@didier.chabanol
No but whglee is on the right direction. He has found that the problem come 
from the requested picture sizes (QCIF or CIF) which are not supported by HTC 
EVO. The problem could be resolved with getSupportedPictureSizes() function 
(http://developer.android.com/reference/android/hardware/Camera.Parameters.html#
getSupportedPictureSizes%28%29) available starting API level 5.

I am playing around in eclipse as well on an EVO running 2.1 -- not much 
programming experience though.  Whglee how are you creating those log files?

amwalters,
you need Android SDK tools (adb, ddms) installed on your development machine to 
capture those logs from the EVO.

Yep, I have them both and have everything running.  It just looked much more 
imsdroid specific than the logcat output filtered by pid.

bossiel,
do you mind to run a logcat from your Hero when making a video call? I would 
like to compare the webcam params on Android 1.5 against my Android 2.1 Hero.
thanks.

See attached.
As I said in comment 34. If you are targeting Android 2.0 or later you can use 
getSupportedPictureSizes() to get supported sizes. Another possible 
optimization is to check if your camera support YUV40SP. If yes, this will 
avoid to convert the picture from NV21 before encoding ().
On Android 2.0 and later you have many possibilities (AMR Cortex, 
CallbackBuffer, Picture Format, ...) to optimize the encoding process.
In all cases, H.263 should not take 66%.

I found out why there is a delay on the inbound video everytime I added the 
outgoing video leg. It was the build target. IMSDroid built on API level 3 
(Android 1.5) will not have the delay problem. If it is built on API level 7 or 
8 (Android 2.1 or 2.2), I will see delay. Looks like those optimization you 
mentioned are needed for Android 2.2 and later.

Also, if I set the bandwidth to high, I will see heavy delay on the inbound 
video when outbound video is on.

Regarding picture size, getSupportedPictureSizes() must be used to determine 
the supported size for a given device and IMSDroid can set it accordingly. The 
HTC Hero (2.1) and EVO (2.2) I have don't offer standard QCIF or CIF picture 
size. But, I can use whatever available size returned from the get size method 
and it works for me.

The problem is that getSupportedPictureSizes() is only supported starting API 
level 5 while we target API Level 3 and later.
Have you tried to use 
addCallBackBuffer(http://developer.android.com/reference/android/hardware/Camera
.html#addCallbackBuffer%28byte[]%29) instead of onPreviewFrame()?

Have you resolved the crash(when hangup)?

I haven't looked at the crash during call release just yet. I am still trying 
to figure out how to use addCallBackBuffer(). I am looking at 
http://code.google.com/p/android/issues/detail?id=2794 comment 22 on how that 
is being implemented. If you have a chance could you give me a hand, or some 
tips? You know better than I do on the code. Then I can spend time to look at 
the crash.

Should I implement addCallBackBuffer() on both VideoProducer and VideoConsumer, 
or more places?

For the crash, I have pushed a patch here: 
http://code.google.com/p/imsdroid/issues/detail?id=31
Could you tell if it works for you?

No you only need it for the Producer. It will allow you to use a single buffer 
instead of a buffer-per-frame. The problem with onPreviewFrame is that it will 
create, marshal and destroy a buffer for each _data[] callback. If you look at 
the log you will see a lot of messages about the GC.
addCallbackBuffer is used in conjunction with setPreviewCallbackWithBuffer. 
Each time you provide the buffer the callback should be called when the buffer 
is full.

Your patch does fix the crash.

bossiel,
can you explain what onPreviewFrame() does for VideoProducer? Looks like it is 
doing some buffer management.
If I use addCallbackBuffer and setPreviewCallbackWithBuffer, whatever code in 
onPreviewFrame I should not need to keep, am I right? Also how many callback 
buffers would you recommend?

I have implemented addCallbackBuffer and setPreviewCallbackWithBuffer to 
VideoProducer but video call is failing. No video is sending out. 

I got the addCallbackBuffer and setPreviewCallbackWithBuffer to work on calls 
with video working both ways. But the delay on the inbound video stream using 
CIF is still horrible. QCIF works great. I am testing it with Bria and only 
H.263 codec enabled. Please take a look at the attached file and let me know if 
I am doing it correctly. I just can't believe HTC EVO, with all that CPU speed, 
cannot support 2 way CIF video calls.

Why do you need to add three callback buffers?
1024*768 is too big, you should choose something close to CIF or re-size the 
buffer before sending it to the native world . Which are the supported picture 
sizes?
I have added new version (revision 264) with full ARMv5TE optimizations. This 
version should use less CPU cycles. As you are using Android 2.1 you can 
rebuild FFmpeg with ARMv6 optimizations and x264 with ARM Cortex opt.
Could you also send the trace (SIP + RTP)?

[deleted comment]

[deleted comment]

I have 3 callback buffers to see if that helped the delay, it didn't.
It doesn't matter if I use 1024x768 or else. I had tried 400x400 and heavy 
delay on inbound video still resulted.
The supported picture sizes on my HTC EVO: 
3264x2448,3264x1952,2592x1952,2592x1552,2048x1536,2048x1216,1600x1200,1280x960,1
280x768,1024x768,640x480,640x384,512x384,400x400,272x272
Where do I rebuild FFMpeg from and how to do it? I have not seen any official 
FFMpeg porting to Android.

Wireshark trace attached for a call to Bria with picture size set to 400x400. 
Callback buffer size is also based on 400x400 as well.

Have you tried with the revision 264?
I will send you "libtinyWRAP.so" with AMRv6 optimization.

I updated my source tree to the latest from SVN.
The latest did not help softened up the delay on CIF video calls. that was done 
without the addCallbackBuffer implementation.
Worst yet, when I plugged my the addCallbackBuffer calls, the CIF video call 
scenario stopped working. When I pressed "Send Video", it sent H.263 RTP 
packets from 1-2 secs and then it stopped sending. But QCIF video call works 
though.
What could possibly be the bottleneck for CIF video calls? I don't think its 
the CPU. Its more likely resource management. 

Very strange. Unfortunately I cannot test it on a 2.1 device.
Do you have the problem without addCallbackBuffer?
Have you tried with a picture size power of 2 (e.g. 512x512) instead of 400x400?
Do you have warnings or errors in the logs?
BTW, I think have I identified the problem about the delay and if all is OK the 
fix will be part of the next release.

Without using addCallbackBuffer, both QCIF & CIF video call works fine, only 
that there are heavy delay when in CIF mode.
It will be great news if you could get the delay issue fixed. Looking forward 
to the new release.

Hello whglee, We did test with the latest SVN trunk code on an HTC EVO with 
Android 2.2. It looks like when we perform video call the webcam is still not 
grabbed. Is your code already integrated in the trunk ?

I will not integrate the source code as it will only work on Android 2.1 and 
later.
The problem is that the HTC camera does not support standard QCIF and CIF 
sizes. It's easy to solve the problem: See comment 8 and 9 but instead of 
1024x768 use 272x272.
The supported picture sizes are listed at comment 54.

We performed new test with whglee VideoProducer.java the webcam is now grabbed. 
There are a couple of optimization to perform like being able to display 
correctly CIF and QCIF received video, we may work together on that ?
Our tests worked in h263 and h263+, does this version is able to encode h264 
flow ?

Is there a problem on displaying QCIF or CIF? If you are using the attached 
"VideoProducer.java" you should change the video size to something reasonable 
(e.g. 272x272).
I'm open to all suggestions. All ideas are welcome.
Yes H.264 is supported.

I also have another issue on my HTC EVO. The camera capturing of myself is 
rotated by 90 degree. As a result, remote side see the video rotated by 90 
degree as well. Take a look at the screen capture and notice the self view down 
at the bottom left. Any ideas?

Have pushed pre-release version which should support front facing camera on HTC 
EVO.
It's available here: 
http://imsdroid.googlecode.com/svn/trunk/debug/imsdroid-1.0.pre.apk
Appreciate any feedback.

I did test with version v1.0.285 the front camera is grabbed but there's one 
limitation :

  -The image grabed by the front camera is in the right direction only when the phone is placed vertically while when the phone is place vertically the image received is flattened. It's quiet difficult to use it.

I also noticed that h264 codec consum too much CPU in high quality and when I 
decrease the quality to low then the application crash.

- How many CPU is consumed? Which process takes more CPU, encoding or decoding?
- Is it possible to have a screenshot of the image?
- For the crash you can send the log using "Log collector" from the market

You should know that the Android video system is the worst one I have never 
seen.
There are two big design errors (even more) on the video system:
1- The preview callback will always create, marshal (C->Java) and destroy each 
picture. This will consume both CPU and memory. It's amazing!
2- Both the capture+preview callback and the canvas drawing operations are 
called on the main event loop. I let you imagine what happen when use canvas 
operations while capturing at high resolution.
Because of the issue 2 you will have delay on the display for high quality mode 
even if the CPU usage is at 2%.

I have started developing on Android 4 months ago and my conclusion is that 
it's far (very far) from being a good OS compared to iOS or Symbian.

This beta version should resolve all 
problems:http://imsdroid.googlecode.com/svn/trunk/debug/imsdroid-1.0.324-armv7-a
.apk.

You must have a phone with at least ARMv7-a (Cortex-a8) processor.

Issues:
 - 90° rotation on HTC EVO (Options->General->Flip video)
 - H.264 Delay for CIF video stream
 - H.264 codec should consume -50% CPU compared to the online version
 -...