Background

Unit tests in exasol_integration_test_docker_environment.test should not be part of the implementation package.

Acceptance Criteria

1. Move exasol_integration_test_docker_environment.test into separate package
2. Also remove exasol_integration_test_docker_environment.test.utils.py (it's legacy and not used, replace by exasol_integration_test_docker_environment.testing.utils.py)

Situation

Currently docker container only exposes two ports (JDBC and BucketFS). As a user I would like to open additional ports, for example, for debugger.

Acceptance Criteria

• Added another parameter for additional mapped ports (possibly comma separated)
• Additional ports are mapped to the host
• Added tests

Situation

• Bazel creates during a build symlinks to the result of the build, but also to the source directory. This symlink to the source directory is a symlink loop (circular symlink).
• If you build the exaudfclient locally with bazel for testing and then try to build a flavor, EXASLCT is caught in an infinite loop which is caused by this symlink loop.
• EXASLCT is accumulating more and more memory and fails at some point with an out of memory error.

Workaround

• Use bazel clean to remove the symlinks or use the build configs --config no-symlinks or --config symlinks-to-tmp which are defined in the .bazelrc of the exaudfclient

Solution

• We could exclude the bazel symlinks from the builds with exasol/script-languages-release#143
• Or, a proper solution would be the detection of the symlink loop. You might be able to do this by collecting the already seen inodes of symlinks to directories and if you encounter one twice, throw an exception.
• Alternative, count the depths of your recursion and break after a configurable depth

The fix needs to be done in the https://github.com/exasol/integration-test-docker-environment/blob/master/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py

Check number of path segments at

Background

The docker build tasks can be (and are) used independently of the CLI. Thus, it makes sense to move them into it's own repository. (The script-languages-container-tool can depend upon only this new repository then, because it does not use the CLI of ITDE).

Acceptance Criteria

Create a new repository and move the package exasol_integration_test_docker_environment.lib there.

Situation:

• currently, we create the docker network twice, first to let docker generate the random subnet and the second time to create the docker network with a fixed IPAM config. For some reason, the first docker network didn't work, unfortunately it wasn't documented.

To-do:

• Revisit this code and try it without double creation and document why it doesn't work or remove double creation if not necessary

Background

In order to be compliant to integration team infrastructure, the changelog files names need to be in the format of:
changes_a.b.c.md, but currently they have the format: changes-.a.b.c.md

Acceptance Criteria

Change all changelog files and links to the correct format.

Background:

• We need specific ExaConf templates to set up the docker-db with specific configurations
• We currently use a preset of templates with a limited number of changeable parameters
• However, sometimes it could be useful for the user to provide a custom ExaConf

Situation:

• We run currently all tests for all docker-db versions which needs a lot of time
• Usually, it is enough to run a subset of the tests for all versions and run all tests only for the default version

Task:

• Run all tests only for master and on request [run all tests] in commit message
• Fail, if not all test were run to have a build breaker, such that we need to run all tests for all version before merging

Situation:

• Currently, this project is used as a submodule for the script-languages and is directly integrated into the source tree there. Because of this, we need to use relative module paths in this project, until we use proper packaging.
• In #28 were accidentally absolute module path introduced at some points

Tasks:

• Replace all absolute module paths with relative module paths, except for the test case files

BucketFS: root path 'EXAClusterOS/ScriptLanguages-standard-exasol-7.0.0-20200803'' does not exist in bucket 'default' of bucketfs 'bfsdefault' (Session: 1682442116722851840) (-1) (SQLExecDirectW)")

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 224, in run
task_generator = self.run_task()
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/docker_image_analyze_task.py", line 143, in run_task
.generate_image_hash(image_info_of_dependencies)
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/build_context_hasher.py", line 18, in generate_image_hash
hash_of_build_context = self._generate_build_context_hash()
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/build_context_hasher.py", line 33, in _generate_build_context_hash
hash_of_build_context = files_directories_list_hasher.hash(files_directories_to_hash)
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py", line 72, in hash
file_or_directory, collected_directories, collected_files)
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py", line 135, in collect_files_and_directories
numCharacters += sum([len(d[1]) for d in new_directories])
File "/usr/local/lib/python3.6/dist-packages/exasol_integration_test_docker_environment/lib/docker/images/create/utils/file_directory_list_hasher.py", line 135, in
numCharacters += sum([len(d[1]) for d in new_directories])
IndexError: string index out of range

Situation:

Exception can get printed multiple times in the final exception summary if a task is the child task of multiple other tasks

Fix:

1. We could use a set to collect the exceptions messages to remove duplicates
2. We could remember in the task if we already visited the task and collected its exception

Background

We plan to create a new release, for that we need to update the changelog-0.3.0.md

Acceptance Criteria

New changelog-0.3.0.md with correct content.

Background:

• We need specific ExaConf templates to set up the docker-db with specific configurations
• Currently, we need one template file per docker-db version
• However, most versions share the same template which is currently done by symlinks

Task:

• To reduce maintenance effort and reduce the time until a new docker-db can be tested we should use a fallback mechanism which use the template for the latest available ExaConf template which is compatible to the Exasol Minor version

Situation:

Currently, we only release source packages. With poetry, we can create proper python packages.

Acceptance criteria:

[ ] The python packages created by poetry get uploaded as release artifacts during a Github release
[ ] Github Actions which creates the release and uploads the packages gets triggered by creating a tag
[ ] There will be two different release types:

• a pre-release, that allows other projects to test the new release
• a normal release

Situation

The following new docker-db versions are available and don't have configs, yet

• 6.1.12
• 6.1.13
• 6.2.10
• 6.2.11
• 7.0.2
• 7.0.3

Acceptance Criteria

• Add configs for new docker-db versions
• Add new docker-db versions to CI
• Set the default version to 7.0.3

Exasol 6.0 and 6.1 reached their end of life, for that reason, we remove the support for them. This also saves us CI resources.
https://www.exasol.com/portal/display/DOWNLOAD/Exasol+Life+Cycle#ExasolLifeCycle-ExasolLifeCyclePolicy

Background:

• sometimes we want to execute logical identical tasks (this means the values of all significant parameters are equal) multiple times in the same python interpreter, for example for tests or in case the result is not deterministic, ...
• However, luigi is original a workflow management system doesn't want to rerun tasks if their input parameters have the same values
• Usually, we don't need the reruns in the same invocation

Possible solution

• Because we only need the reruns between different invocations of luigi, we could introduce a new significant parameter job_id which is usually inherited from the calling tasks or given by the creator of the root task
• The inheritance from the calling tasks allows an easy pass-through of the job_id through the task hierarchy
• Letting the creator give the root task the job_id, lets the creator decide which behavior he needs.
• An example for this can be found in https://github.com/exasol/integration-test-docker-environment/pull/77/files#diff-d42a1e87a1734995ecb539bd44cc24dc5c44338255d6ef596796502c8ca6fe31
• Alternatives are described here: https://stackoverflow.com/questions/34613296/how-to-reset-luigi-task-status
• However, using explicit job_ids has the advantage that your logs and outputs for the old run still exist after a rerun
• Also, this could replace in many case the job_id_config which is a global variable with many drawbacks.

Background

• New versions of the docker-db were released https://github.com/exasol/docker-db/releases
• We need to add the docker_db_config_template's for these versions and add them to the CI https://github.com/exasol/integration-test-docker-environment/tree/master/docker_db_config_template

Acceptance Criteria

• Added docker_db_config_template for these versions
• Add these versions to the CI matrix https://github.com/exasol/integration-test-docker-environment/blob/master/.github/workflows/main.yml
• Update default docker-db version in the CLI

Situation:

Cleanup methods can be called multiple times when the task is a child task of multiple other tasks.

Fix:

One solution could be to remember if as task already got cleaned up and return directly from the cleanup function if this is the case. For example, in the class BaseTask.

Situation:

• A test environment existed
• test_environment_reuse was active
• requirements file for the test_container was changed
• The test_container image got build but not restarted

Remarks:

• Probably, the SpawnTestContainer class doesn't check if the test_container image was newly build

The following DockerDB versions are currently missing:

• 6.1.10
• 6.1.11
• 6.2.7
• 6.2.8
• 6.2.9
• 7.0.0

Background

• docker-db 7.1.0 was released a while ago
• We need to use image 7.1.0-d1, because the image 7.1.0 didn't extract the built-in language container, see https://github.com/exasol/docker-db/issues/64

Acceptance Criteria

• Config for docker-db 7.1.0 with language container is available
• Default image version is changed to 7.1.0-d1

Situation:

• User might start the environment from inside a docker container which itself is already in a specific docker network
• It that case this container might not be able to access host port forward, the host bridge or the specifically generated docker network for the environment

To-do:

• Add a new command-line parameter which can be specified multiple times to allow the addition of multiple docker networks
• Pass the command-line parameter to SpawnTestEnvironment
• Add the networks to the docker API call to start the container

• Changelog
• Check version in pyproject.toml

Background:

• test/utils.py is used by the script-languages-container-tool for testing
• however, we removed the test directory from the package to not include the tests
• as such we need to separate the test/utils.py into a different module

Situation:

We often get bug reports with only the exceptions, however the exceptions often are not helpful because an external command failed, such as a Docker command or a command line tool, like tar.

Fix:

• Write the stdout and stderr into file in the output directory of the job
• Add the tail of the outputs to the exception.
• Add instructions how to report the problem.
  • print the path where the user can find the output directory of the job
• Maybe package .build_output directory for the job to an archive

Background

• Sometimes, we need to deploy the integration-test-docker-environment on a virtual machine to access the database from outside of the virtual machine. In that case, we can't use a docker-network, because we only would forward a few ports to the virtual-machine. However, the export in pyexasol or r-exasol opens another connection besides the normal database connection which we can't forward, because it happens on a random port.

Acceptance Criteria

• Add option to start docker-db and test-container on host network

Background

It turned out that Luigi treats dynamic dependencies differently from static dependencies, and parameters which are not set can cause unexpected errors when transforming a dependency to a dynamic one.
This was the case of

--external-exasol-db-port
--external-exasol-bucketfs-port

which are forwarded in cli.commands.run_db_test() to the TestContainer, but actually not set, and then eventually used as an Luigi IntParameter.

Acceptance Criteria

Not setting these parameter not correctly must not result in any unexpected behavior. Best way is to initialize these parameter correctly in cli.commands.run_db_test()

• The luigi base classes are used by several different use cases, by the docker build system, by the ITDE and by the SLCT
• For that reason, it makes sense to extract it into its own Repo

Situation:

Currently, we use the default docker runtime for the creation of the containers. For some data science testing, it might be interesting to use nvidia-docker as a runtime to make GPU available in the database.

Todo:

1. Add a command-line option for the runtime to SpawnTestEnvironment CLI Command
2. Add the docker config option for the runtime (we need this value for the creation of all containers)
3. Think about how to test it. Is it possible to install nvidia-docker without Nvidia driver?

Logs of level Info (and probably below) not working during dependency graph creation phase, with other words logs which are written within BaseTask.register_required().
Note: Logs of level Error do work.

Need to add a config file as described here

Error:

2021-03-24T07:12:49.7887300Z ##[group]Run vagrant ssh -c "cd /script-languages-release; ./exaslct build --flavor-path flavors/standard-EXASOL-7.0.0/ --goal udf_client_deps"

2021-03-24T07:12:49.7888810Z �[36;1mvagrant ssh -c "cd /script-languages-release; ./exaslct build --flavor-path flavors/standard-EXASOL-7.0.0/ --goal udf_client_deps"�[0m

2021-03-24T07:14:09.1595850Z Traceback (most recent call last):
2021-03-24T07:14:09.1597480Z   File "/script-languages-release/exaslct_src/exaslct.py", line 25, in <module>
2021-03-24T07:14:09.1598070Z     cli()
2021-03-24T07:14:09.1599360Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/click/core.py", line 829, in __call__
2021-03-24T07:14:09.1601130Z     return self.main(*args, **kwargs)
2021-03-24T07:14:09.1602540Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/click/core.py", line 782, in main
2021-03-24T07:14:09.1603480Z     rv = self.invoke(ctx)
2021-03-24T07:14:09.1604910Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
2021-03-24T07:14:09.1611870Z     return _process_result(sub_ctx.command.invoke(sub_ctx))
2021-03-24T07:14:09.1613640Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
2021-03-24T07:14:09.1614710Z     return ctx.invoke(self.callback, **ctx.params)
2021-03-24T07:14:09.1616290Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/click/core.py", line 610, in invoke
2021-03-24T07:14:09.1617250Z     return callback(*args, **kwargs)
2021-03-24T07:14:09.1618310Z   File "/script-languages-release/exaslct_src/exaslct/cli/commands/build.py", line 74, in build
2021-03-24T07:14:09.1619160Z     success, task = run_task(task_creator, workers, task_dependencies_dot_file)
2021-03-24T07:14:09.1621220Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/cli/common.py", line 93, in run_task
2021-03-24T07:14:09.1622330Z     task = task_creator()
2021-03-24T07:14:09.1623410Z   File "/script-languages-release/exaslct_src/exaslct/cli/commands/build.py", line 73, in <lambda>
2021-03-24T07:14:09.1624380Z     task_creator = lambda: DockerBuild(flavor_paths=list(flavor_path), goals=list(goal), shortcut_build=shortcut_build)
2021-03-24T07:14:09.1626010Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/luigi/task_register.py", line 98, in __call__
2021-03-24T07:14:09.1626980Z     h[k] = instantiate()
2021-03-24T07:14:09.1628360Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/luigi/task_register.py", line 79, in instantiate
2021-03-24T07:14:09.1629440Z     return super(Register, cls).__call__(*args, **kwargs)
2021-03-24T07:14:09.1631060Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/flavor_task.py", line 14, in __init__
2021-03-24T07:14:09.1632180Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1634110Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/stoppable_base_task.py", line 15, in __init__
2021-03-24T07:14:09.1635310Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1636990Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/timeable_base_task.py", line 10, in __init__
2021-03-24T07:14:09.1638140Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1639680Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 85, in __init__
2021-03-24T07:14:09.1640810Z     self.register_required()
2021-03-24T07:14:09.1641960Z   File "/script-languages-release/exaslct_src/exaslct/lib/tasks/build/docker_build.py", line 15, in register_required
2021-03-24T07:14:09.1642930Z     tasks = self.create_tasks_for_flavors_with_common_params(DockerFlavorBuild)
2021-03-24T07:14:09.1644890Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/flavor_task.py", line 21, in create_tasks_for_flavors_with_common_params
2021-03-24T07:14:09.1646450Z     for flavor_path in self.flavor_paths}
2021-03-24T07:14:09.1648100Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/flavor_task.py", line 21, in <dictcomp>
2021-03-24T07:14:09.1649280Z     for flavor_path in self.flavor_paths}
2021-03-24T07:14:09.1650970Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/flavor_task.py", line 25, in _create_task_for_with_common_params
2021-03-24T07:14:09.1652270Z     task = self.create_child_task_with_common_params(cls, **params)
2021-03-24T07:14:09.1654020Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 302, in create_child_task_with_common_params
2021-03-24T07:14:09.1655220Z     return task_class(**params)
2021-03-24T07:14:09.1656600Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/luigi/task_register.py", line 98, in __call__
2021-03-24T07:14:09.1657540Z     h[k] = instantiate()
2021-03-24T07:14:09.1659250Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/luigi/task_register.py", line 79, in instantiate
2021-03-24T07:14:09.1660340Z     return super(Register, cls).__call__(*args, **kwargs)
2021-03-24T07:14:09.1662000Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/flavor_task.py", line 33, in __init__
2021-03-24T07:14:09.1663190Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1664780Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/docker_base_task.py", line 9, in __init__
2021-03-24T07:14:09.1665910Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1667560Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/docker/images/create/docker_build_base.py", line 18, in __init__
2021-03-24T07:14:09.1668730Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1670320Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/stoppable_base_task.py", line 15, in __init__
2021-03-24T07:14:09.1671450Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1673150Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/timeable_base_task.py", line 10, in __init__
2021-03-24T07:14:09.1674300Z     super().__init__(*args, **kwargs)
2021-03-24T07:14:09.1675990Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/base/base_task.py", line 85, in __init__
2021-03-24T07:14:09.1677110Z     self.register_required()
2021-03-24T07:14:09.1678830Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/docker/images/create/docker_build_base.py", line 33, in register_required
2021-03-24T07:14:09.1680300Z     self.analyze_tasks_futures = self.register_dependencies(self.create_analyze_tasks())
2021-03-24T07:14:09.1682250Z   File "/home/vagrant/.local/share/virtualenvs/script-languages-release-igq1tTpT/lib/python3.6/site-packages/exasol_integration_test_docker_environment/lib/docker/images/create/docker_build_base.py", line 47, in create_analyze_tasks
2021-03-24T07:14:09.1683630Z     difference = goals.difference(self.available_goals)
2021-03-24T07:14:09.1684640Z AttributeError: 'tuple' object has no attribute 'difference'
2021-03-24T07:14:09.2440090Z ##[error]Process completed with exit code 1.

Fix

Use set(goals) instead goals which is a set for computing the difference

• Introduce poetry as project description and build system
• Remove all relative imports
• Convert poetry's pyproject.toml to Pipfile for starting with pipenv.
• Convert poetry's pyproject.toml to setup.py to allow installation from github for test purposes

Situation:

• Sometimes, it would be useful to pass through special devices such as GPU to the docker containers.

To-do:

1. Add command-line parameter which can be specified multiple times
2. Pass the command-line parameter the SpawnTestEnvironment
3. Extend the docker API call with the device option
   https://docker-py.readthedocs.io/en/stable/containers.html#docker.models.containers.ContainerCollection.run
   Option: devices

Situation:

Currently, the user needs to manually clean up the environment after the usage. The manual cleanup requires the extraction of names of the docker resources from the configs and then issuing manual docker commands to remove them.

Todo:

Add a CLI command which automatically removes all docker resources associated to an environment name.

Remarks:

We already have a way to remove the environment when the setup fails via the cleanup methods of the corresponding Spawn-Tasks. The question is, can we reuse it. It is probably a good idea to use the cleanup methods because then creation and cleanup are near together in the code which makes it easier to keep them in sync. The main issue with this is probably, that some of the Spawn-Tasks only get created during the runtime and during the task graph construction. We may be able to work around it, by extracting the cleanup methods from the dynamically created Spawn-Tasks.

Newly released version since last update:

• 6.2.12-d1
• 6.2.13-d1
• 6.1.14-d1
• 7.0.4
• 7.0.5
• 7.0.6

Background

Version code was not increased during last release (0.3.0).

Acceptance Criteria

Correct version set in pyproject.toml. We decided to jump to v0.3.1.

Situation:

Currently, we don't release the packages on pypi. However, pypi is more or less the standard..

Tasks:

• Register at pypi
• Setup credentials as secrets on Github
• Create Github Action workflow to publish the packages via poetry