Setting up a regression test suite for meeting performance goals about simple-android HOT 4 CLOSED

Looks great, a few questions to understand the approach better:

* Are real world app performance tests called benchmark tests?

Benchmarking is typically used in the industry to test performance of libraries, frameworks, etc. Just following that convention here.

* Under setting up benchmark data, `INSERT AND UPDATE` are going to happen regardless, because both of these are a huge part. Why is syncing again not ideal? The dataset will run all over again after syncing and that is a problem, why?

Syncing data from Sandbox will take a lot of time (~30m), so ideally we don't want to do it over and over again for every test. Backing up and restoring the data between test runs will be faster.

* If we backup and restore the dataset with no sync, are we sure we'll never receive a regression with syncing after data update because with the current approach this use case will not be included

In the sense? The regression is something we track not over a single test run, but over multiple test runs. We would want to see degraded performance of a query over a couple of days along with a significant difference in query performance before we label it as a regression. I am not sure if you were referring to something else.

from simple-android.

The proposal is looking good @vinaysshenoy . Having a base class for benchmark tests seems the easiest option, compared to having a lint rule for it.

from simple-android.

Looks great, a few questions to understand the approach better:

• Are real world app performance tests called benchmark tests?
• Under setting up benchmark data, INSERT AND UPDATE are going to happen regardless, because both of these are a huge part. Why is syncing again not ideal? The dataset will run all over again after syncing and that is a problem, why?
• If we backup and restore the dataset with no sync, are we sure we'll never receive a regression with syncing after data update because with the current approach this use case will not be included

from simple-android.

In the sense? The regression is something we track not over a single test run, but over multiple test runs. We would want to see degraded performance of a query over a couple of days along with a significant difference in query performance before we label it as a regression. I am not sure if you were referring to something else.

The second answer clarifies it, thanks.

from simple-android.