• For Argolis Account

  1. Use "admin@" account
  2. Create a new Argolis project for this lab
  3. Make sure "admin@" user has the below privileges
     • Owner
     • ServiceAccountTokenCreator
     • Organization Admin
  4. Make sure you have enough of disk space(1.5 GB - 2 GB) in your gCloud shell for the terraform setup

• For Non-Argolis Account

  1. Create a GCP Project 1.1. Create a new GCP project and follow the guidelines here.

     1.2. The project must belong to the same VPC Service Control perimeter as the data destined to be in the lake. Refer to this link to use or add Dataplex to VPC-SC.

  2. Make sure both data and Dataplex regions are available in one of the Dataplex supported regions

  3. Organization Policies: The org policies should be set to below:

     • "compute.requireOsLogin" : false,
     • "compute.disableSerialPortLogging" : false,
     • "compute.requireShieldedVm" : false
     • "compute.vmCanIpForward" : true,
     • "compute.vmExternalIpAccess" : true,
     • "compute.restrictVpcPeering" : true
     • "compute.trustedImageProjects" : true,
     • "iam.disableCrossProjectServiceAccountUsage" :false #Only required when you want to set up in a separate project to your data project

  4. Enable Private Google Access on the network you plan to use with Dataplex Data Quality tasks. If you don't specify a network or sub-network when creating the Dataplex Data Quality task, Dataplex will use the default subnet, and you will need to enable Private Google Access for the default subnet.

  5. Make sure you have the appropriate Dataplex quotas

     ## dataplex.googleapis.com/zones in region:us-central1 should be at least 20
     ## dataplex.googleapis.com/lakes in region:us-central1 should be at least 5
     
     

     You can view these settings at https://console.cloud.google.com/iam-admin/quotas and then enter the filters - Metric:dataplex.googleapis.com/zones OR Metric:dataplex.googleapis.com/lakes region:us-central1

  6. Make sure you have enough of disk space(1.5 GB - 2 GB) for the terraform setup

1. 

2. Select the appropriate project. Make sure you are in the right project before you proceed.

3. Install the below python libraries

   pip3 install google-cloud-storage
   pip3 install numpy 
   pip3 install faker_credit_score

4. Declare variable

• In cloud shell, declare the following variables after substituting with yours.

• For Argolis, use fully qualified corporate email address - [email protected] otherwise use your fully qualified email address (e.g. [email protected]) as the USERNAME

  echo "export USERNAME=your-email" >> ~/.profile
  echo "export PROJECT_ID=$(gcloud config get-value project)" >> ~/.profile

  

5. To get the currently logged in email address, run: 'gcloud auth list as' below:

   gcloud auth list
   Credentialed Accounts
   
   ACTIVE: *
   ACCOUNT: [email protected] or admin@(for Argolis)

6. Clone this repository in Cloud Shell

   git clone https://github.com/mansim07/dataplex-labs.git

7. Trigger the terraform script to setup the infrastructure

   cd ~/dataplex-labs/setup/
   source ~/.profile
   bash deploy-helper.sh ${PROJECT_ID} ${USERNAME}

   The script will take about 30-40 minutes to finish.

8. Validate the Dataplex lakes and zones are created with the right number of asserts as shown below. Go to Dataplex -> Manage

   

9. Go to Composer… Then Environments… Click on -composer link..then click on 'Environment Variables'

We have a series of labs designed to get hands-on-experience with Dataplex concepts. Please refer to each of the lab specific README for more information on the labs.

Create HMS and attach it to the lake. Follow the instructions here Create multiple personas/roles in Cloud Indentity and play around with the security policies Become more creative and share ideas Don't forget post-survey and feedback

Please make sure you clean up your environment

gcloud projects delete ${PROJECT_ID}