In the code in the abm folder (connector.py and server.py) there are a lot of comments and variables referring to arrow flight.
I understand that the arrow flight module was used as a starting point, but these comments and variables should probably be fixed.

In fybrik, the module service name is changed due to: Modify the convention of release name generation

and thus, the README_mysql.md under fybrik directory and the test_module.sh script needs to be updated as well:

https://github.com/fybrik/airbyte-module/blob/main/fybrik/README_mysql.md

Add a readiness probe to the deployment, similarly to fybrik/arrow-flight-module#220

Helm chart version in Chart.yaml is not updated upon release.
This version number is actually been ignored and only the tag value in values.yaml is been used.

@cdoron
Some Airbyte connectors, like S3, will require that secrets be passed to them. Currently, the only way this could be done is to put the secret keyword and value in the Asset yaml where they will eventually end up in a mounted file (/etc/conf/conf.yaml) This is of course a security hole. Instead, the secrets - keyword as expected by the Airbyte connector and values - either be stored in Vault or in a Secrets asset and the module should then extract these, append them to the other values specified in Asset yaml and pass these as configuration parameters to the airbyte connector.

Add TLS support:

• run the server with TLS enabled.
• make the connection between the server and vault using TLS

It currently assumes katalog catalog

a release version of module.yaml is applied in the test:

CMD="kubectl apply -f https://github.com/fybrik/airbyte-module/releases/download/v$moduleVersion/module.yaml -n fybrik-system
"

while the fybrik resources applied are from the master version of the module.

a possible fix is to always test the master version.
otherwise, the test should be adjusted to support testing release and master version of the module.

When we deployed airbyte-module on OpenShift we have faced with a number of issues:

• The security context was not accepted by the OpenShift
• In order to run dind container, the pod has privilege: true property, which was not acceptable by OpenShift.
• Without the privilege: true property, the docker demon container cannot run.

Fybrik is soon to be updated to API v1beta1 (fybrik/fybrik#1651) thus the resource in this repo should be updated as well.
consider adding a compatibility matrix to the README.md.

The file https://github.com/fybrik/airbyte-module/tests/test_fybrik.sh uses the old fybrik taxonomy commands:

go run main.go taxonomy compile --out custom-taxonomy.json --base charts/fybrik/files/taxonomy/taxonomy.json$AIRBYTE_FYBRIK_TEST/fybrik-taxonomy-customize.yaml

As expected, it fails,

Error: unknown flag: --out
Usage:

Flags:
      --config string   config file (default is $HOME/.fybrik.yaml)
  -h, --help            help for fybrik

Error: unknown flag: --out

The default image namespace is fybrik, but the ABM helm uses cdoron .

A test should be added for the usage of reading / writing to mysql database.
The test can be run without Fybrik.

The current connection is defined in asset.yaml as follows:

connection:
      name: file
      file:
        connector: "airbyte/source-file"
        dataset_name: userdata
        format: parquet
        url: "https://github.com/Teradata/kylo/raw/master/samples/sample-data/parquet/userdata2.parquet"
        provider:
          storage: HTTPS

A number of issues:

1. Name "file" is misleading. Files can be local or remote, and not necessarily all connectors supporting one type, support another. Thus, I suggest changing the name. "remote"/"https" - suggestions are welcome.
2. "connector" property is an implementation detail, not a part of the connection. It should be determined in module charts.
3. "provider" is redundant since "https" appears in the url, unless I am missing here something.
4. "dataset_name" is 1. redundant and 2. incorrect since it appears in the url as userdata2.
5. format is not a part of the connection, it appears under dataFormat of the asset.
   I suggest having something like this:

connection:
      name: remote
      remote:
        protocol: HTTPS
        host: "github.com"
        folder: "Teradata/kylo/raw/master/samples/sample-data/parquet"
        file: "userdata2.parquet"

or

connection:
      name: https
      https:
        url: "https://github.com/Teradata/kylo/raw/master/samples/sample-data/parquet/userdata2.parquet"

The connection should be defined in Fybrik in a custom layer pkg/storage/layers/connection.yaml
@cdoron @Mohammad-nassar10 @revit13 @simanadler

Without normalization, the table are in airbyte format. Thus normalization process should be done to transform the data to the expected form.

Reference: https://docs.airbyte.com/understanding-airbyte/basic-normalization/