This repo is an anonymous repository for reviewing the paper Android United.
The Android Eval evaluation set and evaluation code used in the paper have already been released. Since the training data and evaluation images are still undergoing privacy and harmful information checks, the training data, evaluation images, and training result checkpoints are temporarily unavailable. We promise to make all training data and checkpoints public after the paper is accepted.
A visual example is provided in the visual_example
folder.
The evaluation environment will be set in a specific time and space. The current time is set to: 2024.5.10 12:00, and the location is: a station near Stanford University in San Francisco. Every time it starts running, it will reposition to the above time and space.
We offer two testing methods: AVD on Mac (arm64) and Docker on Linux (x86_64).
If you use AVD on Mac (arm64), please refer to here to set up the environment.
If you use Docker on Linux (x86_64), please refer to here to set up the environment.
To test, run:
python eval.py -n test_name -c your path to config.yaml
The specific output of each question is saved under ./logs/evaluation/test_name
, and the evaluation results are saved
in the output
folder.
If you only want to run a few questions for testing, you can refer to:
python eval.py -n test_name -c your path to config.yaml --task_id taskid_1,taskid_2,taskid_3
If you are using Docker for testing, we support parallel testing. Please note that you need to confirm in advance that you have sufficient memory and storage. Each concurrent session takes up approximately 6G of memory and 9G of storage space.
python eval.py -n test_name -c your path to config.yaml -p 3
The corresponding task_id for each question can be found in evaluation/config
.
Use the following code to generate a single evaluation result:
python generate_result.py --input_dir ./logs/evaluation/ --output_dir ./logs/evaluation/ --output_excel ./logs/evaluation/test_name.xlsx --key "your glm4 key"
You need to fill in your glm4 key in the --key parameter, which can be applied for on the glm4 official website. generate_result.py will generate an Excel file of all test results under --input_ir, containing detailed results for each question.
The Agent
class has been predefined in the agent/
folder, with implementations for the OpenAI interface based on
oneapi and the currently deployed GLM interface. If you need to add a base model, you need to:
- Create a new Python file under
agent/
, and refer toagent/model/OpenAIAgent
to implement your model call by inheriting theAgent
class. Theact
function input is already organized in the OpenAI message format, and the output is a string. If the input format of the corresponding model is different from OpenAI, you can refer to theformat_history
function inglm_model
for modifications. Theprompt_to_message
method modifies the current round of prompts and image inputs (if any) to the current model's single-round format, which can refer to the standard OpenAI format provided byOpenAIAgent
. - Import your new class in
agent/__init__.py
. - Replace the content under
agent
in the config file used bytext_only_auto_test.py
with:
agent:
name: Your Agent Module Name
args:
max_new_tokens: 512
Make sure the name matches your implemented class name, and the content under args
will be passed to your
class's init
function.
During the process of writing a new task, it is equally important to write and use the code to determine if your code is correct through actual running results. Therefore, please follow the steps below to ensure each new task is error-free.
- Write your task. Tasks include yaml files, evaluation methods, and corresponding mobile app installation.
- The task's yaml file should refer to other existing files under
evaluation/config
and must includetask_id
,task
,metric_type
, andmetric_func
.adb_query
is only used when the results need to be queried using adb commands. Althoughcategory
is not yet in use, it is strongly recommended to add it. - The evaluation method needs to inherit the
evaluation/task/SingleTask
class. After each recorded operation, thejudge
function will be executed, and its return value is a dict:{"judge_page": bool, "1": bool, ..., "complete": bool}
. The code will record the judgment result of the last page wherejudge_page
isTrue
, andcomplete
should only be set toTrue
if all judgment points are correct. If it's a task that compares return values, thecheck_answer
method has already been implemented. Modifyfinal_ground_truth
to the standard answer before calling this function. - Refer to other tasks, import all evaluation methods in
evaluation/app_name/__init__.py
into thefunction_map
class. - To ensure the model can execute the launch command correctly, add the app name and corresponding package name
in
templates/packages/apps_dict
. The package name can be obtained by executingadb -s {device} shell dumpsys window | grep mCurrentFocus | awk -F '/' '{print $1}' | awk '{print $NF}'
.
- The task's yaml file should refer to other existing files under
- Execute your task using at least the most advanced agent and generate evaluation results. If necessary, quickly complete the correct operation during model operation intervals to ensure that the recorded operation can capture the correct result page between two model operations to test if your code can complete the detection task.
- Use the
tools/check_result_multiprocess.py
function to generate screenshots of each step. Focus on checking whether the screenshots of correct model operations are indeed judged as correct.
If you want to define a mobile snapshot different from the android eval snapshot, you need to follow these steps:
- Download related docker files from the link: https://drive.google.com/file/d/1xpPEzVof5hrt5bQY6BHm_4Uoyq5mJQNb/view?usp=drive_link
- Extract the file, enter the extracted folder, and then run:
docker build -t android_eval_no_avd:latest .
- Configure your AVD snapshot on an x86_64 machine (it is recommended to configure it directly using Android Studio). Note that the default installed Android AVD type is:
RUN /bin/bash -c "source /root/.bashrc && yes | sdkmanager 'platform-tools' 'emulator' 'system-images;android-33;google_apis;x86_64'"
RUN /bin/bash -c "source /root/.bashrc && yes | sdkmanager 'build-tools;33.0.0'"
RUN /bin/bash -c "source /root/.bashrc && yes | sdkmanager 'platforms;android-33'"
If you want to configure the AVD for a different version, please modify the specific version number installed in the Dockerfile. Note that the version number must be strictly consistent, otherwise, the installed image will not be able to read the existing cache.
- You can use the following code to generate the AVD image used in the docker:
python tools/modify_mobile_to_docker.py
--avd_dir /Path/to/your/.android/avd
--device_name your device name
--save_dir /Path/to/your/save/avd
Alternatively, you can modify it as follows:
Find your .avd folder and .ini file through Android Studio -> Virtual Devices Manager -> Right-click -> Show on Disk, and make the following modifications:
In Pixel_7_Pro_API_33.ini, modify path and path.rel to the following paths:
avd.ini.encoding=UTF-8
path=/root/.android/avd/device name.avd
path.rel=avd/device name.avd
target=android-33
In Pixel_7_Pro_API_33.avd/config.ini, modify the following paths:
...
image.sysdir.1 = system-images/android-33/google_apis/x86_64/
...
skin.path = /root/.android/skins/pixel_7_pro
...
Keep the other contents unchanged.
- Start an image and copy your .avd folder and .ini file into the image:
docker run -it android_eval_no_avd:latest /bin/bash
docker cp /path/to/your/device name.avd container_id:/root/.android/avd
docker cp /path/to/your/device name.ini container_id:/root/.android/avd
After completing the above, you can execute the following in the image:
emulator -avd device name -no-window -no-audio -no-snapshot-save
Verify whether the installation is successful.