Hey

I saw that we are able to set a bunch of different setups for the device we are aiming to test on.
Do you plan to add in the near future the feature of shard values?

Thanks a lot for the plugin :)

I have no use for creating buckets manually (Test Lab does it for free if not specified) but a default name is used if firebase_test_lab_results_bucket is null, is there a way to omit the option entirely ?

fastlane-plugin-firebase_test_lab_android (1.1.0)

I tried this fastlane plugin, but processing didn't complete.
This is the log I tried on bitrise. Timeout occurs, because the processing is not completed.

.....
[16:46:49]: ▸ 16:46:49 Stopped logcat recording.
[16:46:49]: ▸ 16:46:49 Done. Test time = 102 (secs)
[16:47:01]: ▸ 16:47:01 Starting results processing. Attempt: 1
[16:49:14]: ▸ 16:49:14 Completed results processing. Time taken = 132 (secs)
[16:49:14]: ▸ 16:49:14 Test is Finished
[16:49:14]: ▸ Robo testing complete.
[16:49:14]: ▸ More details are available at [https://console.firebase.google.com/project/locoguide-android-test/testlab/histories/bh.7839fbe1ad175410/matrices/5612529911411096723].
[16:49:14]: ▸ API [toolresults.googleapis.com] not enabled on project
[16:49:14]: ▸ [321805086667]. Would you like to enable and retry (this will take a
timeout after 5400 seconds

Do I need to set anything configure?

  lane :roboTest do
    firebase_test_lab_android(
      project_id: "XXXX",
      gcloud_service_key_file: "XXXX.json"
      type: "robo",
      devices: [
        {
          model: "walleye",
          version: "28",
          locale: "en_US",
          orientation: "portrait"
        },
      ],
      app_apk: "app/build/outputs/apk/product/debug/app-product-debug.apk",
      timeout: "90s",
      firebase_test_lab_results_bucket: "XXXXXXX",
      extra_options: "--robo-directives mail=MAIL,password=PASS,ignore:layout_forgot_password="
    )
  end

With download_dir option, the plugin set test result files public-read, which permits non-authorized users to access the files.

Generally test result doesn't include sensitive information but it would be nice if the plugin has option that prevents files from setting public-read ACL.

Thanks for this awesome fastlane plugin!

Just a small feature request,
I use the official gcloud github action for setting up the gcloud credentials, so it would be nice if there was a default option for using default credentials instead of providing with the gcloud_service_key_file

src:
https://github.com/GoogleCloudPlatform/github-actions/tree/master/setup-gcloud

​lane​ ​:test​ ​do
​  ​take_screenshot​(
​      ​project_id​: ​"project-id"​,
​      ​gcloud_service_key_file​: ​"fastlane/client-secret.json"​,
​      ​devices​:
​          ​[
​              ​{
​                  ​model​: ​"shamu"​,
​                  ​version​: ​"22"​,
​                  ​locale​: ​"ja_JP"​,
​                  ​orientation​: ​"portrait"
​              ​}
​          ​]​,
​      ​app_apk​: ​"apk/app.apk"​,
​      ​app_test_apk​: ​"apk/test.apk"​,
​      ​firebase_test_lab_results_bucket​: ​"firebase_cats_test_bucket"​,
​      ​download_dir​: ​".screenshot"
​  ​)

​  ​pr_number​ ​=​ ​ENV​[​"CI_PULL_REQUEST"​]​ != ​nil​ ? ​ENV​[​"CI_PULL_REQUEST"​]​[​/(?<=https:​/​/​github.com​/​cats-oss​/​android​/​pull​/​)(.*)/​]​ : ​nil​\

​  ​compare_snapshot​(
​      ​gcloud_service_key_file​: ​"fastlane/client-secret.json"​,
​      ​snapshot_bucket​: ​"firebase_cats_test_bucket_compare"​,
​      ​working_dir​: ​".snapshot_test"​,
​      ​screenshot_dir​: ​".screenshot/shamu-22-ja_JP-portrait"​,
​      ​github_owner​: ​"cats-oss"​,
​      ​github_repository​: ​"android"​,
​      ​github_pr_number​: ​pr_number​,
​      ​github_api_token​: ​ENV​[​"DANGER_GITHUB_API_TOKEN"​]
​  ​)

​  ​upload_snapshot​(
​      ​gcloud_service_key_file​: ​"fastlane/client-secret.json"​,
​      ​snapshot_bucket​: ​"firebase_cats_test_bucket_upload"​,
​      ​working_dir​: ​".snapshot_test"​,

- ​: ​".screenshot/shamu-22-ja_JP-portrait"

​  ​)

​  ​screenshot_notifier​(
​      ​gcloud_service_key_file​: ​"fastlane/client-secret.json"​,
​      ​screenshot_bucket​: ​"firebase_cats_test_bucket_screenshot"​,
​      ​screenshot_dir​: ​".screenshot"​,
​      ​github_owner​: ​"cats-oss"​,
​      ​github_repository​: ​"android"​,
​      ​github_pr_number​: ​pr_number​,
​      ​github_api_token​: ​ENV​[​"DANGER_GITHUB_API_TOKEN"​]​,
​      ​image_length​: ​100

• • # - ​  ​)

Hi,

I'm having trouble downloading the test results of a test with sharding.

It looks that the URL to download the results does not take into account the folder structure with sharding.

 Exit status of command 'gsutil -m cp -r gs://any_path/any_test/Pixel2-29-es_ES-portrait .firebaseTestLabResults' was 1 instead of 0.
 CommandException: No URLs matched: gs://any_path/any_test/Pixel2-29-es_ES-portrait
 CommandException: 1 file/object could not be transferred.

When using sharding, the folder structure is gs://any_path/any_test/Pixel2-29-es_ES-portrait .firebaseTestLabResults-shard_#.

It might be better downloading the test results on the root folder or iterating through all shards folders.

I use this plugin and it is very well. However I cannot configure or it could be a bug: the job not fail on tests failure. I have such logs:

[13:04:48]: ▸ Test [matrix-xyz] has been created in the Google Cloud.
[13:04:48]: ▸ Firebase Test Lab will execute your instrumentation test on 2 device(s).
[13:04:48]: ▸ Creating individual test executions...
[13:05:00]: ▸ ...........................................................done.
[13:05:00]: ▸ Test results will be streamed to [https://console.firebase.google.com/project/xyz/testlab/histories/bh.xyz/matrices/xyz].
[13:11:02]: ▸ 13:11:02 Test matrix status: Finished:2
[13:11:02]: ▸ Instrumentation testing complete.
[13:11:02]: ▸ More details are available at [https://console.firebase.google.com/project/xyz/testlab/histories/bh.xyz/matrices/xyz].
[13:11:04]: Test status: [{"axis_value"=>"Nexus6P-23-pl_PL-portrait", "outcome"=>"Failed", "test_details"=>"1 test cases failed, 1 passed"}, {"axis_value"=>"Pixel2-28-en_US-portrait", "outcome"=>"Failed", "test_details"=>"1 test cases failed, 1 passed"}]
[13:11:04]: Finishing...

In logs I see tests failed, but the result of job is: passed. How can I configure this plugin, to fail job when tests not passed?

My lane"

lane :firebase_instrumentation_test do

        apkPath = "path"

    firebase_test_lab_android(
      project_id: "fitqbeapp",                                    # Your Firebase project name.
      gcloud_service_key_file: "#{ENV['GCP_KEY']}",         # File path containing the gcloud auth key.
      # gcloud_components_channel: "alpha",                            # If you use gcloud component channel option (alpha/beta).
      type: "instrumentation",                                                   # Optional: Test type (robo/instrumentation).
      devices: [                                                      # Devices
        {
          model: "Nexus6P",
          version: "23",
          locale: "pl_PL",
          orientation: "portrait"
        },
        {
          model: "Pixel2",
          version: "28"
        }
      ],
      app_apk: apkPath,                                       # The path for your android app apk.
      app_test_apk: "/app-debug-androidTest.apk",       # The path for your android instrumentation test apk.
      console_log_file_name: "fastlane/console_output.log",
      timeout: "30m",

    )
  end