Running firestore-bigquery-export deposits private key into document data about extensions HOT 7 CLOSED

Hey @tbillington thanks for the report.

I've just imported a fresh dataset myself and I don't see any IAM related private keys. Would you be able to take a screenshot of where the data is (hiding any private info)? Do you mean the keys are within the exported data from Firestore, or are they visible within the BigQuery console?

from extensions.

They are in the data in the XXXX_raw_changelog and XXXX_raw_latest (the view over changelog)

This has happened in 3 of the 5 collections I've connected through the extension in firebase. Two of the collections just have their document data (which I assume is the expected default behaviour).

from extensions.

Thanks for posting that. I'll do some more investigation and get back to you.

from extensions.

Hi @tbillington, I've tried to reproduce this issue without any luck. Here is the line of code that sets the data property that is eventually stringified for BigQuery import: https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/scripts/import/src/index.ts#L207

As you can see it calls snapshot.data() which is the correct Firestore call to make if you want just the data from a Firestore record. What you are seeing in your sessions_raw_latest table is a snapshot which has not had its data() method called.

Could you please take a look at the experience property in your Firestore sessions collection and let me know if you see a _firestore object contained within, please? Thank you.

from extensions.

@russellwheatley Yes, I see a _firestore object. It has fields like _settings and _projectId.

from extensions.

@tbillington Ok, good to know we've found the issue! Somewhere in your codebase, you're storing the snapshot on an experience property. You ought to change it to snapshot.data() as referenced here

from extensions.

Ah, ok. This stuff can be easy to miss in JS land. Would have much preferred something like this crash/throw when it tried to be saved. Don't know if there's anything on your end you can do though.

Regardless, I appreciate you getting back on this 😃 cheers

from extensions.