Comments (10)
pilgun
commented on August 21, 2024
Hi Fabrice,
It's a good sign that acvtool works from the terminal.
There could be a few issues here.
First, it would be great to test your scripts on a single app that worked in terminal (second log mentions another app). Appium instrumentation probably conflicts with acvtool instrumentation.
Secondly, instrumentation process sometimes may break due to some glitches in emulator when used intensively. It could be helpful to take a look into adb logcat for instrumentation errors and fatal errors.
Finally, could you kindly try another version https://github.com/pilgun/acvcut - it contains more up to date acvtool version. I got rid there from the "wait until the coverage file is saved" logic.
Please let me know if you find Appium working well for you together with acvtool/acvcut.
from acvtool.
fabriceyhc
commented on August 21, 2024
Thanks for the quick response @pilgun!
I've copied the acvcut version of smiler and replaced it in the acvtool repo and pip installed again to reflect the updates. I confirmed that it was still working by running the following commands from the terminal:
emulator @avd0 > /dev/null 2>&1 &
acv instrument android_apps/photoai.apk -f
acv install ~/acvtool/acvtool_working_dir/instr_photoai.apk
acv start com.soumio.mikes.photoai -q
acv stop com.soumio.mikes.photoai
acv report com.soumio.mikes.photoai -p ~/acvtool/acvtool_working_dir/metadata/photoai.pickle
I also tested on a single "photoai.apk" that had previously worked, which reduces the likelihood of the particular apk being an issue and the likelihood of overly intensive emulator use.
Unfortunately, the same issue is present.
(amlt) fabrice@fabrice-ThinkPad-P1-Gen-4i:~/Documents/GitHub/AndroidMLTesting/app_tester$ pytest -s -k "photoai"
[AMLT] starting appium...
[AMLT] starting emulator...
======================================================================================== test session starts ========================================================================================
platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/fabrice/Documents/GitHub/AndroidMLTesting/app_tester
collected 4 items / 3 deselected / 1 selected
tests/test_photoai.py [AMLT] instrumenting new apk...
[AMLT] amlt_py2: acv instrument -f /home/fabrice/Documents/GitHub/AndroidMLTesting/app_tester/android_apps/photoai.apk
decompiled com.soumio.mikes.photoai
parsing /home/fabrice/acvtool/acvtool_working_dir/apktool/com.soumio.mikes.photoai/smali...
saving instrumented smali: /home/fabrice/acvtool/acvtool_working_dir/apktool/com.soumio.mikes.photoai/smali...
pickle file saved: /home/fabrice/acvtool/acvtool_working_dir/metadata/photoai.pickle
instrumented
built
apk instrumented: /home/fabrice/acvtool/acvtool_working_dir/instr_photoai.apk
package name: com.soumio.mikes.photoai
[AMLT] uninstalling existing apk...
Success
[AMLT] installing instrumented apk...
[AMLT] amlt_py2: acv install /home/fabrice/acvtool/acvtool_working_dir/instr_photoai.apk
installing instr_photoai.apk
Performing Streamed Install
Success
[AMLT] start coverage monitoring...
[AMLT] amlt_py2: acv start com.soumio.mikes.photoai -q
.[AMLT] stop coverage monitoring...
[AMLT] amlt_py2: acv stop com.soumio.mikes.photoai -t 10
finish testing
Broadcasting: Intent { act=tool.acv.finishtesting }
Broadcast completed: result=0
0
coverage files at /mnt/sdcard/com.soumio.mikes.photoai:
[AMLT] generating coverage report...
[AMLT] amlt_py2: acv report com.soumio.mikes.photoai -p ~/acvtool/acvtool_working_dir/metadata/photoai.pickle -o ./coverage_reports
Traceback (most recent call last):
File "/home/fabrice/miniconda3/envs/amlt_py2/bin/acv", line 11, in <module>
load_entry_point('acvtool', 'console_scripts', 'acv')()
File "/home/fabrice/Documents/GitHub/acvtool/acvtool.py", line 217, in main
run_actions(parser, args)
File "/home/fabrice/Documents/GitHub/acvtool/acvtool.py", line 88, in run_actions
granularity=args.granularity)
File "/home/fabrice/Documents/GitHub/acvtool/smiler/reporter.py", line 32, in generate
smiler.get_execution_results(package, ec_dir, images_dir)
File "/home/fabrice/Documents/GitHub/acvtool/smiler/smiler.py", line 76, in get_execution_results
Run acvtool with \'-start\' argument to produce coverage.".format(package_name))
Exception: No coverage or crash report files have been detected on the device for com.soumio.mikes.photoai package.
Run acvtool with '-start' argument to produce coverage.
ERROR conda.cli.main_run:execute(49): `conda run acv report com.soumio.mikes.photoai -p /home/fabrice/acvtool/acvtool_working_dir/metadata/photoai.pickle -o ./coverage_reports` failed. (See above for error)
[AMLT] stopping emulator...
OK: killing emulator, bye bye
OK
[AMLT] stopping appium...
from acvtool.
pilgun
commented on August 21, 2024
I guess it is still the problem with Appium. Acvtool and Appium can't work together because both of them instrument the app. I wish I could put my hands on it one day #6
from acvtool.
fabriceyhc
commented on August 21, 2024
It might be something with appium but it has worked together before. The first code block in the original post shows the full testing workflow using appium and acvtool together - it just isn't stable for some reason. I'm also only using appium for the driver to automatically navigate the apps and reach targeted testing activities - no appium-related coverage instrumentation.
from acvtool.
pilgun
commented on August 21, 2024
ok, let's think appium is fine.
From the log it is seen that acvtool does not generate coverage files: "coverage files at /mnt/sdcard/com.soumio.mikes.photoai:" no files here
this means that either instrumentation did not start correctly (acv start) or it accidentally crashed in the process before coverage files were pulled. Coverage files also may not be saved if read/write permissions were not activated after installing the app, but these should be run automatically by acvtool.
The instrumentation issue or a crash to be seen in the logcat for example with the command:
"adb logcat --pid=`adb shell pidof -s com.soumio.mikes.photoai`"
to be monitored after acv start command or after launching the app
from acvtool.
fabriceyhc
commented on August 21, 2024
So it looks like hardly anything is being saved to the logs during acv start when appium is used to exercise the app vs when I exercise it manually through the pure terminal approach.
Here's the logcat when running with appium (no coverage generated):
--------- beginning of main
10-21 15:39:22.056 3803 3803 W art : Unexpected CPU variant for X86 using defaults: x86
10-21 15:39:22.069 3803 3803 D AcvInstrumentation: ------------------------------------------------------------
10-21 15:39:22.069 3803 3803 D AcvInstrumentation: onCreate: main ---------
10-21 15:39:22.069 3803 3803 D AcvInstrumentation: onCreate: mkdirs: false ---------
10-21 15:39:22.069 3803 3803 D AcvInstrumentation: onCreate end: main ---------
and here it is when running the process from terminal and manually exercising the app (coverage generated as expected):
--------- beginning of main
10-21 16:42:10.072 12810 12810 W art : Unexpected CPU variant for X86 using defaults: x86
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: ------------------------------------------------------------
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: onCreate: main ---------
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: onCreate: mkdirs: false ---------
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: onCreate end: main ---------
10-21 16:42:23.269 12810 12810 W art : Before Android 4.1, method android.graphics.PorterDuffColorFilter android.support.graphics.drawable.VectorDrawableCompat.updateTintFilter(android.graphics.PorterDuffColorFilter, android.content.res.ColorStateList, android.graphics.PorterDuff$Mode) would have incorrectly overridden the package-private method in android.graphics.drawable.Drawable
10-21 16:42:23.287 12810 12810 D : HostConnection::get() New Host Connection established 0x9cbc8180, tid 12810
10-21 16:42:23.334 12810 12844 D : HostConnection::get() New Host Connection established 0x9cbc8d00, tid 12844
10-21 16:42:23.348 12810 12844 I OpenGLRenderer: Initialized EGL, version 1.4
10-21 16:42:23.349 12810 12844 D OpenGLRenderer: Swap behavior 1
10-21 16:42:23.349 12810 12844 W OpenGLRenderer: Failed to choose config with EGL_SWAP_BEHAVIOR_PRESERVED, retrying without...
10-21 16:42:23.349 12810 12844 D OpenGLRenderer: Swap behavior 0
10-21 16:42:23.364 12810 12844 D EGL_emulation: eglCreateContext: 0xac2850c0: maj 3 min 1 rcv 4
10-21 16:42:23.376 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:23.377 12810 12844 E eglCodecCommon: glUtilsParamSize: unknow param 0x000082da
10-21 16:42:23.377 12810 12844 E eglCodecCommon: glUtilsParamSize: unknow param 0x000082da
10-21 16:42:23.404 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:24.482 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:26.754 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:28.726 12810 12810 I Choreographer: Skipped 35 frames! The application may be doing too much work on its main thread.
10-21 16:42:28.748 12810 12810 D skia : onFlyCompress
10-21 16:42:28.836 12810 12810 D skia : onFlyCompress
10-21 16:42:28.841 12810 12815 I art : Do partial code cache collection, code=29KB, data=25KB
10-21 16:42:28.841 12810 12815 I art : After code cache collection, code=23KB, data=22KB
10-21 16:42:28.841 12810 12815 I art : Increasing code cache capacity to 128KB
10-21 16:42:28.903 12810 12810 D skia : onFlyCompress
10-21 16:42:28.964 12810 12810 D skia : onFlyCompress
10-21 16:42:29.031 12810 12810 D skia : onFlyCompress
10-21 16:42:29.095 12810 12810 D skia : onFlyCompress
10-21 16:42:29.158 12810 12810 D skia : onFlyCompress
10-21 16:42:29.219 12810 12810 D skia : onFlyCompress
10-21 16:42:29.286 12810 12810 D skia : onFlyCompress
10-21 16:42:29.345 12810 12810 D skia : onFlyCompress
10-21 16:42:29.407 12810 12810 D skia : onFlyCompress
10-21 16:42:29.464 12810 12810 D skia : onFlyCompress
10-21 16:42:29.523 12810 12810 D skia : onFlyCompress
10-21 16:42:29.584 12810 12810 D skia : onFlyCompress
10-21 16:42:29.647 12810 12810 D skia : onFlyCompress
10-21 16:42:29.706 12810 12810 D skia : onFlyCompress
10-21 16:42:29.762 12810 12810 D skia : onFlyCompress
10-21 16:42:29.826 12810 12810 D skia : onFlyCompress
10-21 16:42:29.887 12810 12810 D skia : onFlyCompress
10-21 16:42:29.953 12810 12810 D skia : onFlyCompress
10-21 16:42:30.015 12810 12810 D skia : onFlyCompress
10-21 16:42:30.080 12810 12810 D skia : onFlyCompress
10-21 16:42:30.140 12810 12810 D skia : onFlyCompress
10-21 16:42:30.202 12810 12810 D skia : onFlyCompress
10-21 16:42:30.258 12810 12810 D skia : onFlyCompress
10-21 16:42:30.265 12810 12815 I art : Do partial code cache collection, code=48KB, data=46KB
10-21 16:42:30.265 12810 12815 I art : After code cache collection, code=48KB, data=46KB
10-21 16:42:30.265 12810 12815 I art : Increasing code cache capacity to 256KB
10-21 16:42:30.324 12810 12810 D skia : onFlyCompress
10-21 16:42:30.382 12810 12810 D skia : onFlyCompress
10-21 16:42:30.440 12810 12810 D skia : onFlyCompress
10-21 16:42:30.502 12810 12810 D skia : onFlyCompress
10-21 16:42:30.560 12810 12810 D skia : onFlyCompress
10-21 16:42:30.622 12810 12810 D skia : onFlyCompress
10-21 16:42:30.682 12810 12810 D skia : onFlyCompress
10-21 16:42:30.746 12810 12810 D skia : onFlyCompress
10-21 16:42:30.806 12810 12810 D skia : onFlyCompress
10-21 16:42:30.870 12810 12810 D skia : onFlyCompress
10-21 16:42:30.929 12810 12810 D skia : onFlyCompress
10-21 16:42:30.996 12810 12810 D skia : onFlyCompress
10-21 16:42:31.056 12810 12810 D skia : onFlyCompress
10-21 16:42:31.121 12810 12810 D skia : onFlyCompress
10-21 16:42:31.176 12810 12810 D skia : onFlyCompress
10-21 16:42:31.245 12810 12810 D skia : onFlyCompress
10-21 16:42:31.307 12810 12810 D skia : onFlyCompress
10-21 16:42:31.373 12810 12810 D skia : onFlyCompress
10-21 16:42:31.433 12810 12810 D skia : onFlyCompress
10-21 16:42:31.503 12810 12810 D skia : onFlyCompress
10-21 16:42:31.562 12810 12810 D skia : onFlyCompress
10-21 16:42:31.623 12810 12810 D skia : onFlyCompress
10-21 16:42:31.689 12810 12810 D skia : onFlyCompress
10-21 16:42:31.754 12810 12810 D skia : onFlyCompress
10-21 16:42:31.814 12810 12810 D skia : onFlyCompress
10-21 16:42:31.878 12810 12810 D skia : onFlyCompress
10-21 16:42:31.940 12810 12810 D skia : onFlyCompress
10-21 16:42:32.000 12810 12810 D skia : onFlyCompress
10-21 16:42:32.063 12810 12810 D skia : onFlyCompress
10-21 16:42:32.125 12810 12810 D skia : onFlyCompress
10-21 16:42:32.183 12810 12810 D skia : onFlyCompress
10-21 16:42:32.246 12810 12810 D skia : onFlyCompress
10-21 16:42:32.304 12810 12810 D skia : onFlyCompress
10-21 16:42:32.363 12810 12810 D skia : onFlyCompress
10-21 16:42:32.423 12810 12810 D skia : onFlyCompress
10-21 16:42:32.480 12810 12810 D skia : onFlyCompress
10-21 16:42:32.545 12810 12810 D skia : onFlyCompress
10-21 16:42:32.605 12810 12810 D skia : onFlyCompress
10-21 16:42:32.668 12810 12810 D skia : onFlyCompress
10-21 16:42:32.726 12810 12810 D skia : onFlyCompress
10-21 16:42:32.793 12810 12810 D skia : onFlyCompress
10-21 16:42:33.044 12810 12810 D skia : onFlyCompress
10-21 16:42:33.179 12810 12810 D skia : onFlyCompress
10-21 16:42:33.244 12810 12810 D skia : onFlyCompress
10-21 16:42:33.304 12810 12810 D skia : onFlyCompress
10-21 16:42:33.366 12810 12810 D skia : onFlyCompress
10-21 16:42:33.426 12810 12810 D skia : onFlyCompress
10-21 16:42:33.484 12810 12810 D skia : onFlyCompress
10-21 16:42:33.547 12810 12810 D skia : onFlyCompress
10-21 16:42:33.606 12810 12810 D skia : onFlyCompress
10-21 16:42:33.668 12810 12810 D skia : onFlyCompress
10-21 16:42:33.725 12810 12810 D skia : onFlyCompress
10-21 16:42:33.787 12810 12810 D skia : onFlyCompress
10-21 16:42:33.853 12810 12810 D skia : onFlyCompress
10-21 16:42:33.909 12810 12810 D skia : onFlyCompress
10-21 16:42:33.969 12810 12810 D skia : onFlyCompress
10-21 16:42:34.028 12810 12810 D skia : onFlyCompress
10-21 16:42:34.089 12810 12810 D skia : onFlyCompress
10-21 16:42:34.151 12810 12810 D skia : onFlyCompress
10-21 16:42:34.216 12810 12810 D skia : onFlyCompress
10-21 16:42:34.286 12810 12810 D skia : onFlyCompress
10-21 16:42:34.357 12810 12810 D skia : onFlyCompress
10-21 16:42:34.427 12810 12810 D skia : onFlyCompress
10-21 16:42:34.514 12810 12810 D skia : onFlyCompress
10-21 16:42:34.596 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:34.600 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:34.600 12810 12810 D skia : onFlyCompress
10-21 16:42:34.692 12810 12844 D EGL_emulation: eglMakeCurrent: 0xac2850c0: ver 3 1 (tinfo 0xac2831a0)
10-21 16:42:34.694 12810 12810 W System.err: java.lang.RuntimeException: getParameters failed (empty parameters)
10-21 16:42:34.694 12810 12810 W System.err: at android.hardware.Camera.native_getParameters(Native Method)
10-21 16:42:34.694 12810 12810 W System.err: at android.hardware.Camera.getParameters(Camera.java:1896)
10-21 16:42:34.694 12810 12810 W System.err: at com.soumio.mikes.photoai.LiveFeed$4$1.onPreviewFrame(LiveFeed.java)
10-21 16:42:34.694 12810 12810 W System.err: at android.hardware.Camera$EventHandler.handleMessage(Camera.java:1110)
10-21 16:42:34.694 12810 12810 W System.err: at android.os.Handler.dispatchMessage(Handler.java:102)
10-21 16:42:34.694 12810 12810 W System.err: at android.os.Looper.loop(Looper.java:154)
10-21 16:42:34.694 12810 12810 W System.err: at android.app.ActivityThread.main(ActivityThread.java:6119)
10-21 16:42:34.694 12810 12810 W System.err: at java.lang.reflect.Method.invoke(Native Method)
10-21 16:42:34.694 12810 12810 W System.err: at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
10-21 16:42:34.694 12810 12810 W System.err: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)
10-21 16:42:34.694 12810 12810 E Camera : Error 2
10-21 16:42:46.272 12810 12810 D AcvInstrumentation: BroadcastReceiver: onReceive: main ---------
10-21 16:42:46.280 12810 12810 D AcvInstrumentation: BroadcastReceiver: onReceive: coverage_1666395766272.ec ---------
from acvtool.
pilgun
commented on August 21, 2024
AcvInstrumentation: onCreate shows that everything works as expected. The package directory /mnt/sdcard/com.soumio.mikes.photoai was already there.
But then acv stop is expected. The error may happen during the testing or at the acv stop event. Is there any FATAL ERROR or any "INSTRUMENTATION_RESULT: shortMsg=Process crashed." in the first Appium adb logcat?
You can clean logs adb logcat -c before testing and then save logs into a file adb logcat > logcat.txt
from acvtool.
fabriceyhc
commented on August 21, 2024
The two logcats in the previous post were copy+pasted from a logfile.txt file from different runs and the logs were deleted in between the runs. The main difference I see between these two is that the first one lacks the final part about the BroadcastReceiver.
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: ------------------------------------------------------------
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: onCreate: main ---------
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: onCreate: mkdirs: false ---------
10-21 16:42:10.084 12810 12810 D AcvInstrumentation: onCreate end: main ---------
# other stuff happening....
# ...
# ..
# .
10-21 16:42:46.272 12810 12810 D AcvInstrumentation: BroadcastReceiver: onReceive: main ---------
10-21 16:42:46.280 12810 12810 D AcvInstrumentation: BroadcastReceiver: onReceive: coverage_1666395766272.ec ---------
Using Appium: logfile.txt
Manual Testing: logfile.txt
Also, I remember seeing this error ("INSTRUMENTATION_RESULT: shortMsg=Process crashed.) during development but it stopped appearing a few hours ago. Haven't even seen it in the logcat.
from acvtool.
pilgun
commented on August 21, 2024
It's hard to say without logs. It looks to me it is related to how Appium operates. For example, if it kills the app before starting testing routines, then there would be no instrumentation process working at the time when acv stop is run.
from acvtool.
fabriceyhc
commented on August 21, 2024
@pilgun , you're brilliant!
Inspired by your thought, I ended up moving around the acv start and acv stop commands into the actual appium test scripts. Running acv start after the app has already launched causes it to close for some reason, but I just added a driver.activate_app(PACKAGE_NAME) call.
At this point, the app starts back up --> appium steps execute --> run acv stop --> exit test --> acv report = IT WORKS EVERY TIME!!!
I just want to reiterate that you're wonderful and thank you for all the quick help!
from acvtool.
Related Issues (20)
- acv crashes while generating report HOT 1
- Coverage report covers only Android API packages HOT 1
- 'acv' not found HOT 1
- Value Error HOT 5
- acv start HOT 7
- acv install returns: INSTALL_FAILED_DEXOPT
- Test Execution crashes when using instrumented apk HOT 1
- Error when path to APK contains space HOT 3
- Adding .ec files but get less reported coverage HOT 11
- Did you support for aab? HOT 2
- Permission denied on going to /mnt/sdcard/* HOT 5
- acv start process crashed HOT 1
- How to integrate with Sapienz? HOT 1
- Error when Instrumenting an apk HOT 3
- acv instrument doesn't work HOT 9
- how to timeout after a certain amount of time. HOT 3
- Error instrumenting apk HOT 2
- Crash or failed instrumentation HOT 1
- What's the diff between coverage generated by Jacoco and ACVTool? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from acvtool.