Evaluate tree service performance about neofs-node HOT 7 OPEN

Test description

Tree service was used as a separate unit (no sharding, no other background processes), and network communications were mocked; as a loader, I have added some k6 scripts and extensions:
https://github.com/nspcc-dev/neofs-node/tree/feat/tree-cmd
https://github.com/nspcc-dev/xk6-neofs/tree/feat/tree-loader

This tree service config was used (sync_interval was either 5 mins (2-3 times per test) or turned off, see below):

tree:
  enabled: true
  cache_size: 15
  replication_worker_count: 1000
  replication_channel_capacity: 1000
  replication_timeout: 5s
  sync_interval: 5m # depends on tests

pilorama:
  max_batch_delay: 5ms
  max_batch_size: 100
  path: /tmp/tree.test

Every run had a "target" rate, meaning k6 tried to iterate exactly that number of operations per every second. Network deadline was 5s so the max number of VUs (virtual users, a k6 term) was 5 times bigger than a "target" RPS multiplied by 5, so it had to be from 0 to N*5 number of request-in-progress at a time where N is a "target" RPS.
Two types of load were presented: the first one was kinda "system" load (ADD tree operation, 10 requests for every test) that wrote to a "system" tree (multipart upload, lock operation, etc for S3 GW) and the second one was "user" load (ADD_BY_PATH tree operation, variable number of operations in every test) that wrote to a "version" tree (regular object PUT for S3 GW). Every tree node had 7 meta fields and a unique path.
Also, there were two types of tests in terms of tree synchronization: background sync does so much bad things to results so I decided to turn it off, although in general, that is not possible in real cases since this is a mechanism that restores missing operation in local logs on any normal and unexpected downtimes (#2161, #2165). Every "sync on" load was done for 15 mins., every "sync off" for 30 mins.

Results

1000 RPS, NO sync, 30 15 min load

Avg: 13.70ms
Errs: NO

summary1000_NOsync.html.pdf

1000 RPS, 5 min sync, 15 min load

Avg: 13.39 13.43ms
Errs: NO

summary1000_5minsync.html.pdf

5000 2500 RPS, NO sync, 30 15 min load

Avg: 2094.34 19.47ms
Errs: YES, <1% NO

summary2500_NOsync.html.pdf

5000 2500 RPS, 5 min sync, 15 min load

Avg: 1202.18 19.79ms
Errs: YES, <1% NO

summary2500_5minsync.html.pdf

7500 5000 RPS, NO sync, 30 15 min load

Avg: 4550.52 1081.33ms
Errs: YES, 84% NO

summary5000_NOsync.html.pdf

7500 5000 RPS, 5 min sync, 15 min load

Avg: 4082.70 1193.95ms
Errs: YES, 67% NO

summary5000_5minsync.html.pdf

from neofs-node.

Oh, 2.2.0 version of the reporter is broken. Numbers are actual but are placed in the wrong fields... Will rerun.

UPD: done.

from neofs-node.