This module configures a Dremio cluster for data access in Fybrik. The module can either deploy a new Dremio cluster or use an existing one. The module first adds a S3 bucket to Dremio's catalog, and then promote to an Iceberg dataset. It then registers a governed virtual dataset based on the governance policy.
Ensure that you have the following:
- Helm 3.3 or greater must be installed and configured on your machine.
- Kubectl 1.18 or newer must be installed on your machine.
- Access to a Kubernetes cluster such as Kind as a cluster administrator.
Install Fybrik v1.1 using the Quick Start, without the section of Install modules
.
You can install a Dremio cluster using the following command:
helm install <chart-name> charts/dremio-module/charts/dremio-cluster
In dremio-module.yaml
can you specify the host and port of an already existing Dremio cluster. If Dremio is deployed using the previous step, you can set the Dremio parameters in dremio-module.yaml
as the following:
dremio.host: "dremio-client.<namespace of the Dremio chart>.svc.cluster.local"
dremio.port: "9047"
Alternatively, you can ask fybrik to deploy a new Dremio cluster. To that end, set the Dremio parameters in dremio-module.yaml
as the following:
dremio.host: "dremio-client.fybrik-blueprints.svc.cluster.local"
dremio.port: "9047"
dremio.enabled: "true"
Either way, apply the fybrik module using the following command:
kubectl apply -f dremio-module.yaml -n fybrik-system
TBD
kubectl create namespace fybrik-sample
kubectl config set-context --current --namespace=fybrik-sample
Replace the values of endpoint
, bucket
, and object_key
in sample/asset-iceberg.yaml
file according to your created asset. Then, add the asset to the internal catalog using the following command:
kubectl apply -f sample/asset-iceberg.yaml
The asset has been marked as a finance
data and the column _c1
has been marked with PII
tag.
First, create a K8S secret for the credentials for accessing the Iceberg table. Assuming the credentials are stored in as the environment variables ACCESS_KEY
and SECRET_KEY
respectivley, this can be done by:
kubectl create secret generic iceberg-dataset --from-literal=access_key=${ACCESS_KEY} --from-literal=secret_key=${SECRET_KEY}
You should also create a secret for accessing the Dremio cluster:
kubectl apply -f sample/secret-dremio.yaml
Register a policy. The example policy removes columns tagged as PII
from datasets marked as finance
.
kubectl -n fybrik-system create configmap sample-policy --from-file=sample/sample-policy.rego
kubectl -n fybrik-system label configmap sample-policy openpolicyagent.org/policy=rego
while [[ $(kubectl get cm sample-policy -n fybrik-system -o 'jsonpath={.metadata.annotations.openpolicyagent\.org/policy-status}') != '{"status":"ok"}' ]]; do echo "waiting for policy to be applied" && sleep 5; done
The following fybrikapplication
deploys a Dremio cluster (if specificed so by the Dremio-module) and configures it via a k8s job, which registers the Iceberg asset in Dremio and applies the policy to create a virtual dataset.
kubectl apply -f sample/fybrikapplication.yaml
Wait for the fybrikapplication
to be ready (could take a few minutes):
while [[ ($(kubectl get fybrikapplication fybrik-iceberg-sample -o 'jsonpath={.status.ready}') != "true") || ($(kubectl get jobs fybrik-iceberg-sample-fybrik-sample-dremio-module -n fybrik-blueprints -o 'jsonpath={.status.conditions[0].type}') != "Complete") ]]; do echo "waiting for FybrikApplication" && sleep 5; done
Use port-forward to access Dremio
kubectl port-forward svc/dremio-client -n <ns-of-Dremio> 9047:9047 &
You can access Dremio via the browser on http://localhost:9047/
, use the following credentials:
"name": "newUser",
"password": "testpassword123"
You can enter into the Space-api
space then select the sample-iceberg-vds
virtual dataset that was created by the module accoring to the polices.
You can also query the data set using the sample/query_sample.py
, for instance:
python sample/query.py --query '{"sql": "SELECT _c0 FROM \"Space-api\".\"sample-iceberg-vds\""}'
- Stop kubectl port-forward processes (e.g., using
pkill kubectl
) - Delete the
fybrikapplication
:kubectl delete -f sample_assets/fybrikapplication.yaml
- Delete the
fybrik-sample
namespace:kubectl delete namespace fybrik-sample
- Delete the policy created in the
fybrik-system
namespace:NS="fybrik-system"; kubectl -n $NS get configmap | awk '/sample/{print $1}' | xargs kubectl delete -n $NS configmap