DatastoreIO accepts single namespace. about dataflowjavasdk HOT 6 CLOSED

Can you say more about what you specifically want to do? You want to query all entities in any namespace?

from dataflowjavasdk.

I want to process the Entities from all the namespace. For example., I have an Employee entity and i would like to add a new field to the entity and value as a String for all namespaces.

from dataflowjavasdk.

I don't think that this is supported by the Datastore API – you cannot read from multiple namespaces at the same time. Since the main purpose of Datastore namespaces is for isolation, they discourage and disallow using multiple namespaces in the same job.

(However, I'm reaching out to them to confirm.)

from dataflowjavasdk.

Datastore team has confirmed, modifying entities in multiple namespaces is strongly discouraged.

However, it should be easy to programmatically generate multiple Dataflow pipelines, each of which affects only one namespace.

Please re-open if we can provide more assistance.

Thanks!

from dataflowjavasdk.

Request to give an example on multiple dataflow pipelines.
It will be great if you can extend the WordCount datastore example to generate multiple dataflow piplelines, so that datastore entities in mulitple namespaces can be processed.

from dataflowjavasdk.

I would like to reopen. I have more than 5000 namespaces. Not sure, how to do the multiple pipelines.
Any example will be appreciated.

from dataflowjavasdk.