firestore-bigquery-export fs-bq-import-collection doesn't allow {widecard} path about extensions HOT 24 CLOSED

@kienner-philippe and @garciapd, I was able to get around this by making my own copy of the script and modifying the query to use collectionGroups.

Changed this:
https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/scripts/import/src/index.ts#L188

To this:

    let query = firebase
      .firestore()
      .collectionGroup(sourceCollectionPath)
      .limit(batch);

Added these to my package.json:

"devDependencies": {
    "@firebaseextensions/firestore-bigquery-change-tracker": "^1.0.0",
    "@google-cloud/bigquery": "^4.4.0",
    "inquirer": "^7.0.0",
}

Then I just run my local version of the script using ts-node with something like this:

GOOGLE_APPLICATION_CREDENTIALS="./secrets/service-account-key.json" \
    npx ts-node ./bin/firestore-bigquery-export-backfill.ts  # my local version of the file

when the script asks:

What is the path of the the Cloud Firestore Collection you would like to import from?

Give it the sub-collection name for the collection group query.
posts

from extensions.

please support sub collections!!

from extensions.

This is a great tool but definitely need a way to be able to do wildcards and collection groups would be good to. I have a small dataset so for now I'm going to cheat and just run updates on my existing records.

from extensions.

Hey, is this feature still not being worked on? I have a fix for it and can open a pull request if it's not been implemented yet.

from extensions.

I don't think that will work, @crablab. Wildcards are not supported in Firestore collection queries. You would have to target each collection individually.

from extensions.

Thanks for the feedback. This is not yet supported because we do not yet have a way to translate wild card path into a Firestore query. Wildcard are supported in triggers by default.

A low-hanging fruit might be adding support for collection group in the import script. Will that solve your use case?

from extensions.

@osheari1 If you have a fix for it then you're more than welcome to open a PR. I closed the issue because wildcard paths are not yet supported as a Firestore query, and a work around solution was merged in.

from extensions.

wild card is one thing, unable to import sub collection is a bigger issue. I wasn't able to implement the hacking script method, could you please support sub collections?

from extensions.

@russellwheatley thanks! Took care of it the way you recommended:)

from extensions.

Thanks. I realized this after looking at the code and the way imports were being made. I was able to work around this problem by using CollectionGroup and importing subcollections. For me that was extremely necessary.

from extensions.

A low-hanging fruit might be adding support for collection group in the import script. Will that solve your use case?

That would be great!

from extensions.

Same issue here with sub-collections.
As I have only few "chatid"s I thought I can just run the import script for each of my chatid by entering
chats/myChatId/posts as a collection path.
Same error message.
So, how could I import a sub-collection?

Hope I was clear...
Thanks in advance for your help.

from extensions.

Hi, any suggestion to import a sub-collection? like the proposed one chats/myChatId/posts??

from extensions.

Thanks @MattReidArnold !! it does work ;)

from extensions.

@kienner-philippe and @garciapd, I was able to get around this by making my own copy of the script and modifying the query to use collectionGroups.
......
Give it the sub-collection name for the collection group query.
posts

I don't find this file in my node modules folder
https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/scripts/import/src/index.ts#L188
where can I found it?

from extensions.

solve it by
"install-fs-bq-import-collection": "mkdir /.npm-global && npm config set prefix '/.npm-global' && export PATH=~/.npm-global/bin:$PATH && source /.profile && npm i -g @firebaseextensions/fs-bq-import-collection && NPM_CONFIG_PREFIX=/.npm-global",
"import-collection": "sudo /home/mohamed/.npm-global/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/index.js"

install-fs-bq-import-collection

let query = firebase
  .firestore()
  .collectionGroup(sourceCollectionPath)
  .limit(batch);

npm run import-collection

** note: change home/mohamed/.npm-global to your username

from extensions.

Hey @osheari1, do you have the fix? It would be great if you could share with us. =D

from extensions.

Can someone explain how to implement this workaround for importing/back filling sub collections using google cloud shell please?

from extensions.

@russellwheatley : What's the work around that you say was merged in?

from extensions.

Hi @Infinitism, I was mistaken, I thought support for collectionGroup queries was already implemented. I've created a PR to support collectionGroupqueries, although it's not certain when or if it will be merged. If you're really keen, I'd suggest cloning the repository, and running the script locally using the changes from this PR #354.

from extensions.

This is potentially tangential, but it doesn't appear that the tool supports exporting all the collections (unless I'm doing something wrong!). I've tried *, / and simply leaving the field blank, but this doesn't seem to be covered in the documentation, and thus I assume it is not supported?

from extensions.

This seems like a supremely necessary capability of this extension/import script.

Thanks for the workaround @MattReidArnold

from extensions.

@russellwheatley think we can close this out now that we've merged and published #354 ?

from extensions.

@jhuleatt Yes, agreed.

from extensions.