Mega script to install/configure/run ovn-scale-test using browbeat on a simulated cluster deployed with ovn-fake-multinode.
NOTE: This script is designed to be used on test machines only. It performs disruptive changes to the machines it is run on (e.g., create insecure docker registries, cleanup existing docker containers).
-
TESTER: One machine to run the tests which needs to be able to SSH paswordless (preferably as
root
) to all other machines in the topology. Performs the following:- prepares the test enviroment: clone the specified versions of
OVS
andOVN
and build theovn-fake-multinode
image to be used by theOVN
nodes. - provisions all other
OVN
nodes with the required software packages and with the correct version ofovn-fake-multinode
to run simulated/fakeOVN
chassis. - runs a docker registry where the
ovn-fake-multinode
(i.e.,ovn/ovn-multi-node
) image is pushed and from which all otherOVN
nodes will pull the image. - runs
browbeat
which in turns runsovn-scale-test
(rally-ovs
) scenarios.
- prepares the test enviroment: clone the specified versions of
-
OVN-CENTRAL: One machine to run the
ovn-central
container(s) which runovn-northd
and theNorthbound
andSouthbound
databases. -
OVN-WORKER-NODE(s): Machines to run
ovn-netlab
container(s), each of which will simulate anOVN
chassis.
The initial provisioning for all the nodes is performed by the do.sh install
command. The simulated OVN
chassis containers and central container are
spawned by the rally-ovs
scenarios themselves.
NOTE: ovn-fake-multinode
assumes that all nodes (OVN-CENTRAL and
OVN-WORKER-NODEs) have an additional Ethernet interface connected to a
single L2 switch. This interface will be used for traffic to/from the
Northbound
and Southbound
databases and for tunneled traffic.
NOTE: there's no restriction regarding physical machine roles so for debugging issues the TESTER, OVN-CENTRAL and OVN-WORKER-NODEs can all be the same physical machine in which case there's no need for the secondary Ethernet interface to exist.
- TESTER:
host01.mydomain.com
- OVN-CENTRAL:
host02.mydomain.com
- OVN-WORKER-NODEs:
host03.mydomain.com
host04.mydomain.com
OVN-CENTRAL and OVN-WORKER-NODEs all have Ethernet interface eno1
connected to a physical switch in a separate VLAN, as untagged interfaces.
dnf install -y git ansible
dnf install -y grubby
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
reboot
dnf install -y grubby
grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
reboot
cd
git clone https://github.com/dceara/ovn-heater.git
A sample file written for the deployment described above is available at
physical-deployments/physical-deployment.yml
.
The file should contain the following mandatory sections and fields:
registry-node
: the hostname (or IP) of the node that will store the docker private registry. In usual cases this is should be the TESTER machine.internal-iface
: the name of the Ethernet interface used by the underlay (DB and tunnel traffic). This can be overridden per node if needed.central-node
:name
: the hostname (or IP) of the node that will runovn-central
(ovn-northd
and databases).
worker-nodes
:- the list of worker node hostnames (or IPs). If needed, worker nodes can be further customized using the per-node optional fields described below.
Global optional fields:
user
: the username to be used when connecting from the tester node. Default:root
.prefix
: a string (no constraints) that will be used to prefix container names for all containers that runOVN
fake chassis. For example,prefix: ovn-test
will generate container names of the formovn-test-X-Y
whereX
is the unique part of the worker hostname andY
is the worker node local index. Default:ovn-scale
.max-containers
: the maximum number of containers allowed to run on one host. Default: 100.
In case some of the physical machines in the setup have different
capabilities (e.g, could host more containers, or use a different ethernet
interface), the following per-node fields can be used to customize the
deployment. Except for fake-nodes
which is valid only in the context of
worker nodes, all others are valid both for the central-node
and also for
worker-nodes
:
user
: the username to be used when connecting from the tester node.internal-iface
: the name of the Ethernet interface used for DB and tunnel traffic. This overrides theinternal-iface
global configuration.fake-nodes
: the maximum number of containers allowed to run on this host. If not specified, the value ofmax-containers
from the global section is used instead.
This generates a runtime
directory and a runtime/hosts
ansible inventory
and installs all test components on all required nodes. Then it registers
the deployment with rally-ovs
.
cd ~/ovn-heater
./do.sh install && ./do.sh rally-deploy
By default OVS/OVN binaries are built with
CFLAGS="-g -O2 -fno-omit-frame-pointer"
.
For more aggressive optimizations we can use the EXTRA_OPTIMIZE
env variable
when installing the runtime. E.g.:
cd ~/ovn-heater
EXTRA_OPTIMIZE=yes ./do.sh install && ./do.sh rally-deploy
cd ~/ovn-heater
rm -rf runtime
Assuming we need to run a private OVS branch from a fork and a different OVN branch from another fork we can:
cd ~/ovn-heater
OVS_REPO=https://github.com/dceara/ovs OVS_BRANCH=tmp-branch OVN_REPO=https://github.com/dceara/ovn OVN_BRANCH=tmp-branch-2 ./do.sh install && ./do.sh rally-deploy
cd ~/ovn-heater
rm -rf runtime
Run the installation with rpm packages parameters specified:
cd ~/ovn-heater
RPM_SELINUX=$rpm_url_openvswitch-selinux-extra-policy RPM_OVS=$rpm_url_openvswitch RPM_OVN_COMMON=$rpm_url_ovn RPM_OVN_HOST=$rpm_url_ovn-host RPM_OVN_CENTRAL=$rpm_url_ovn-central ./do.sh install && ./do.sh rally-deploy
If the physical topology has changed then update
physical-deployment/physical-deployment.yml
to reflect the new physical
deployment.
Then deregister the old rally-ovs deployment:
cd ~/ovn-heater
./do.sh rally-undeploy
And generate the new ansible inventory and register the deployment:
cd ~/ovn-heater
./do.sh generate && ./do.sh rally-deploy
Browbeat and rally-ovs scenarios are defined in browbeat-scenarios/
. YAML
files are browbeat scenarios which in turn will run the JSON rally-ovs
tests.
cd ~/ovn-heater
./do.sh browbeat-run <browbeat-scenario> <results-dir>
This executes <browbeat-scenario>
on the registered deployment. Current
scenarios also cleanup the environment, i.e., remove all docker containers
from all physical nodes. NOTE: If the environment needs to be explictly
cleaned up, we can also execute before running the scenario:
cd ~/ovn-heater
./do.sh init
The results will be stored in test_results/<results-dir>
. The results
consist of:
- a
config
file where remote urls and SHA/branch-name of all test components (browbeat, rally, ovn-scale-test, ovs, ovn) are stored. - an
installer-log
where the ouptut of the./do.sh install
command is stored. - browbeat logs
- rally-ovs logs
- html reports
- a copy of the
hosts
ansible inventory used for the test. - OVN docker container logs (i.e., ovn-northd, ovn-controller, ovs-vswitchd, ovsdb-server logs).
- physical nodes journal files.
- perf sampling results if enabled
cd ~/ovn-heater
./do.sh browbeat-run browbeat-scenarios/switch-per-node-low-scale.yml test-low-scale
This will run browbeat-scenarios/osh_workload_incremental.json
which will
perform two iterations, each of which:
- brings up a fake
OVN
node. - configures a logical switch for the node and connects it to the
cluster-router
. - configures a logical switch port ("mgmt port") and binds it to an OVS
internal interface on the fake
OVN
node. - configures a logical gateway router on the node and sets up NAT to allow communication to the outside.
- moves the OVS internal interface to a separate network namespace and configures its networking.
- waits until ping from the new network namespace to the "outside" works through the local gateway router.
Results will be stored in ~ovn-heater/test_results/test-low-scale/
:
config
: remote urls and SHA/branch-names of components used by the test.hosts
: the autogenerated ansible host inventory.logs
: the OVN container logs and journal files from each physical node.20200212-083337.report
(timestamp will change): browbeat report overview.20200212-083337/rally/plugin-workloads/all-rally-run-0.html
: HTML report of the test run.20200212-083337/rally/plugin-workloads/switch-per-node-workload/20200212-083337-browbeat-switch-per-node-workload-1-1-iteration-0.log
: rally-ovs logs of the test run.
Instruct ovn-heater about the first and last iteration of perf record analysis
providing ext_cmd_start
and ext_cmd_stop
in switch-per-node-100.yml
farm_nodes: 100
ports_per_network: 100
ext_cmd_start: 50
ext_cmd_stop: 90
cluster_cmd_path: /root/ovn-heater/runtime/ovn-fake-multinode
Results will be stored in ~ovn-heater/test_results/test-100-<date>/logs/<node>-perf/
for each
container.
This simulates bringing up 30 OVN
nodes and binding 1000 pods (logical
ports), distributed across nodes. The test also configures port_groups,
address_sets and ACLs for all pods simulating a network policy that would
allow traffic.
The test is meant to check ovn-northd
performance so each iteration is
considered successful if ovn-northd
populated the
Southbound DB, and the logical port is up (up
is set to true
).
cd ~/ovn-heater
./do.sh browbeat-run browbeat-scenarios/switch-per-node-30-node-1000-pods.yml test-scale-pods
If needed, the number of simulated nodes and total number of simulated pods
can be tweaked, by changing the following fields in
browbeat-scenarios/switch-per-node-30-node-1000-pods.yml
:
farm_nodes: <node-count>
ports_per_network: <total-number-of-pods>
port_wait_type: "none"
port_internal_vm: "False"
By default tests configure NB/SB ovsdb-servers to run in clustered mode
(RAFT). If instead tests should be run in standalone mode then the browbeat
scenarios must be adapted by adding the ovn_cluster_db = False
configuration
and the deployment should be regenerated as follows:
cd ~/ovn-heater
./do.sh rally-undeploy
CLUSTERED_DB=False ./do.sh generate
./do.sh rally-deploy
echo ' ovn_cluster_db: "False"' >> browbeat-scenarios/switch-per-node-low-scale.yml
./do.sh browbeat-run <browbeat-scenario> <results-dir>