Command to run ReIdentification From BigQuery pipeline gives error about dlp-dataflow-deidentification HOT 4 CLOSED

Thank you for confirming Mehran! I updated the README to include the dataset.

from dlp-dataflow-deidentification.

Hi Mehran,

We fixed the problem with auto-sharding in batch pipelines - could you please give it a try.

Sergei

from dlp-dataflow-deidentification.

Hi Sergei,

I did a new build with your latest fixes, and was able to run the the REID pipeline in batch mode successfully, The REID-pipeline did indeed re-identify the data!
Many thanks for your help!

The final Gradle command I ran in batch mode looks like this:
(To run in batch mode I just took away these two parameters from the command --streaming and --enableStreamingEngine, not sure if this is the right way, but it worked:-)

gradle run -DmainClass=com.google.swarm.tokenization.DLPTextToBigQueryStreamingV2 -Pargs=" \
--region=<region> \
--project=<project_id> \
--tempLocation=gs://<bucket>/temp \
--numWorkers=1 \
--maxNumWorkers=2 \
--runner=DataflowRunner \
--tableRef=<project_id>:<dataset>.<table> \
--dataset=<dataset>
--topic=projects/<project_id>/topics/<name> \
--autoscalingAlgorithm=THROUGHPUT_BASED \
--workerMachineType=n1-standard-1 \
--deidentifyTemplateName=projects/<project_id>/locations/<location>/deidentifyTemplates/<name> \
--DLPMethod=REID \
--keyRange=1024 \
--queryPath=gs://<bucket>/<query.sql> \
--DLPParent=projects/<project_id>/locations/<location>"

Maybe you should update the Gradle command in the example on the front-page of the repo where you add the option --dataset=<dataset> to avoid the Exception in thread "main" java.lang.NullPointerException: Null datasetId error, as mentioned in my first comment.

/Mehran

from dlp-dataflow-deidentification.

Hi,
I have used the same de-identification template for the re-identification also and ran the above command with the similar parameters. I have found out that the output on the pub-sub is not re-identified data but the same de-identified data being written to the pub-sub. Am I making any mistake with the parameters?
gradle run -DmainClass=com.google.swarm.tokenization.DLPTextToBigQueryStreamingV2 -Pargs="
--region=asia-south1
--project=
--gcpTempLocation=gs:///temp
--tempLocation=gs:///temp
--numWorkers=4
--maxNumWorkers=10
--runner=DataflowRunner
--tableRef=:workerDetails.WorkerDetails
--dataset=workerDetails
--topic=projects//topics/dlp-reid-final
--autoscalingAlgorithm=THROUGHPUT_BASED
--workerMachineType=n1-standard-1
--deidentifyTemplateName=projects//locations/asia-south1/deidentifyTemplates/dlp-for-worker-details
--DLPMethod=REID
--keyRange=1024
--queryPath=gs:///reid_query.sql
--DLPParent=projects//locations/asia-south1"

from dlp-dataflow-deidentification.