ryos-io / rhino Goto Github PK

Environment variables can be used as test user source.

The RMU features consist of:

• Scheduling load tests,
• Management of load test projects,
• Management of load patterns, that is to be applied to test instances,
• Management of test users,
• Monitoring and archiving load test scenarios,
• Creating and managing load test notifications,
• LDAP integration.

Send the name of the retention policy to Influx.

Better status log output into stdout while tests are running, including:

• number of successful requests
• number of failed requests

Non-reporting HttpSpec. In some cases, like in preparation step, the DSL is to be muted so that the framework is not be able to report any measurement.

As of 1.0 Rhino support only classpath files for user credentials. HashiCorp's Vault is broadly used secure server to manage credentials in Enterprises.

except from average values, more statistics like percentiles, medians, etc.

The experimental implementation of InfluxDB integration is done via over-HTTP by using InfluxDB Java library, that is not reliable but may exist as an option in the framework for testing. For production use cases, rhino should support streaming services integration between load generators and timeseries database.

The main flux needs to implement onErrorResume by returning an empty Mono. In error cases, the framework must report the error as well as the error rate in its output.

The field injections must inject proxies instead of values, so every access return a new value within the specs.

mapper to map the http spec's result into the entity.

Create a Rhino hello world example project.

As the project is started off as a prototype, we need to write more tests and the lack of tests deteriorates the confidence.

The load patterns as in ramp-up phase, or throughout the load testing run needs to be synced by a central orchestration component, that is DLO, distributed load orchestrator.

Add tests for input validations including

• pinging the influx db to verify the influx db configurations.
• pinging the Grafana.

Placeholder for Vault Service.

Current implementation yields to an integer, which does not make any sense.

Placeholder ticket to be groomed.

Load DSL is to describe the load test in reactive approach. The goal in release 1.1.0 is to provide a simple (I mean, really simple one) Http DSL.

Replace all stdouts with logger.

Timeout configurations for Async-Http-Client in properties file.

As of 1.1.0 simulation metrics is collected by instances of Measurement type, that is the object which records the execution time in scenario methods. The concept of measurement is to provide developers with a handy toolset with which the load test developers can set the measurement points in their scenario methods on their own. The concept works very fine with variety of load testing scenarios. However, whenever developers need to write load test for web services over HTTP, the salt request time from the time point once the request is first fired without taking the queue time into account is to be reported.

Moreover, the blocking fashion of scenario implementation comes with a well-known cost, that is inefficiency due to thread contention, starving threads, etc. Furthermore, if the scenario implementation is blocking one, the worker threads which run scenarios will also get blocked, so the entire load testing framework get halted.

From all drawbacks of blocking architecture listed above, a new architecture is to be initiated taking reactive principles into consideration, i.e pull vs push.

Proposal

• The scenarios is not to be run by single threads assigned from a threadpool, instead the scenarios is to return a specification describing how to run the load test. The framework itself will then materialise the specification provided. Materialised specs will subscribe themselves in load publishers.
• Blocking test execution strategy must exist along with the reactive runner. The developers are subject to select the runner strategy by providing @Runner(clazz=Runner Implementation) annotation.
• Rhino affords its own non-blocking HTTP client based on Reactor-Netty.
• In addition to metrics collected by Rhino's Measurement facility, we still need to measure the actual turn-around time transparent to the developers.

@Runner(clazz=ReactiveRunner.class)
public class ReactiveSimulation {

}

The challenge will be to subscribe the HTTP requests with the Flux load generator in reactive way. To cope with this, the scenario methods should return a specification of load generation, not the implementation itself. The framework will then materialize the specification into reactive implementation:

@Simulation
@Runner(clazz=ReactiveRunner.class)
public class ReactiveSimulation {
    @TestSpec
    public Spec loadSpec(Measurement measurement) {
        return new HttpSpec
            .get()
            .queryParams()
            .headers()
            .toSpec();
    }
}

Async processing is to be monitored through monitoring APIs, e.g:

waitUntil(session -> 
    http("monitor")
        .target("/monitor")
        .get()
        .saveTo("result"),
        result -> result.getStatusCode() == 200, 
        1 mins, 10 retries)

The measurement is to be done from the first monitoring request till the last attempt.

Operation signature is as follows:

waitUntil(Function<Spec, UserSession> specFunction ,
        Function<HttpResult, Boolean> specFunction,
        Duration delay, // Delay between two attempts
        Integer maxRetry // max attempt
)

The framework still uses Jersey Client to make Http requests for its internal workflows. Replace the Jersey client with async HTTP client in Auth workflows.

Aggregate the metrics, if client-side aggregation is enabled before they are sent to collectors.

Make the Influx DB batching configurable in props.

• Rhino is to be able to create dashboard with the simulation id given right before every load test load in case of Grafana integration is enabled.
• Grafana integration is configurable in rhino.properties.

The framework needs to support Basic Auth besides Access Token/OAuth.

The scenarios within a simulation are to be run sequentially.

1.0.0 supports CSV user sources, with classpath:// prefix.
"file:///" is to be added to define paths to the user source from outside of the classpath.

Dockerhub account bagdemir/rhino:latest

Instead of writing out to the sim log, use a logger like log4j to facilitate the rotation feature.

The load testing user which is streamed through the reactive pipeline and the user injected by provider, are to be different so that the multi-user tests can be carried out. As a matter of fact that the user repository is the same one as the user provider, it is likely that the same user is injected into the injection point is currently active in the pipeline in which case that multi user testing cannot be run.

Ramp-up load before the load generator reaches its maximum load generation capacity.

The ramp-up can be configured by annotation: @RampUp(injectUser=10, every=10)

map() function is to map the result of preceding spec into some other value.

To be able to test the DSL, a configuration for single run is needed, e.g
Spec.start(times(1))

The user session must be cleaned up after it completes its life-cycle.