firestore-bigquery-export failing about extensions HOT 22 CLOSED

Looks like a Windows permission issue. I'll get a local Window environment running and give it a go.

from extensions.

@Ehesp Much appreciated. It would be good to know if it actually works so I can continue to try and resolve. I have 2 other computers including a Mac that I can try on but will take a bit of time setting up.

Among other things I tried:
node v8 and v10 (currently on v10).
I tried using task manager to make sure no other versions on node were running and locking it.
I deleted and reinstalled all my npm packages several times.

from extensions.

I am still having this issue with Mac. No problem uploading schema files and having data populate against those, but I can't get any other result for import than Error unlinking journal file '/Users/.... ... Error: ENOENT: no such file or directory, unlink '/Users/....

`? What is your Firebase project ID? ideliver-cloud
? What is the path of the the Cloud Firestore Collection you would like to import from? (This may, or may not, be the same Collection for which you p
lan to mirror changes.) idriver-app\events\incoming
? What is the ID of the BigQuery dataset that you would like to use? (A dataset will be created if it doesn't already exist) ideliver_app
? What is the identifying prefix of the BigQuery table that you would like to import to? (A table will be created if one doesn't already exist) incom
ing
? How many documents should the import stream into BigQuery at once? 300
Importing data from Cloud Firestore Collection: idriver-app\events\incoming, to BigQuery Dataset: ideliver_app, Table: incoming_raw_changelog
Error unlinking journal file /Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog after successful import: Error: ENOENT: no such file or directory, unlink '/Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------`

Any help would be greatly appreciated

from extensions.

Ok, in that case I'll re-open for investigation.

from extensions.

Hey @cabljac!

I am currently having a issue to import export the data! I tried the techniques above, but I could not solve my problem.

I have all the permissions required - I am the owner of the project. But, I have not found any solution that works to me

My OS is an Ubuntu 22.10 and usually I do not have this kind of issue

from extensions.

Any update on this? I would like to know if it is working for others using Windows. If so then I can justify spending some more time trying to get it working.

from extensions.

Same error here, any workaround?

from extensions.

@chenlevy Not yet. I am still waiting to see if anyone has an answer. Others have said it works on a Mac so I may try that.

from extensions.

@madi8229 I think I figured it out, the problem is with colons in the cursor pointer file name mentioned in the script. Windows file names shouldn't contain colons. I've changed it to underscores and run it locally and it worked perfectly.

#131 Also opened a pull request

from extensions.

Fixed via #131

from extensions.

I am having the same error on a Windows machine

from extensions.

Hello from 2021, I am sad to say this is still broken on MacOS. Exact same error.

from extensions.

Hi @Gibbo3771.

Can you let me know the following is related? #589.

We are looking at further improvements for handling file paths for the export functions

from extensions.

I will have a look but I have also noticed there is a guard here that checks if the cursor file exists before doing anything, but further down (where it unlinks) right here there is no guard.

So the error people are most likely seeing is because the import was successful (and most likely 0 records) and this line is trying to unlink a file that simply doesn't exist, and never will (because 0 records).

I think I have a problem with my gcloud credentials or IAM role that is preventing it from importing the data, and this error has just blinded me to that fact.

EDIT: I am aware there is a try/catch there, but spitting out the entire error right after printing a "success" message in a catch block (?) is a bit weird.

from extensions.

@dackers86 to answer your queston, it looks to be unrelated.

I think I have a permission issue and there is no error handling for it, I would have thought the google sdk would throw if you tried to do something without the correct IAM roles, and there is no try/catch wrapped around the write call to bigquery, shown here.

So right now I have zero clue why my import is reporting 0 records.

from extensions.

Hi @Gibbo3771

I've been so far unable to recreate. We are looking to release better file sanitizing for paths as mentioned earlier.

Could this be a potential test post release? #589

from extensions.

Closing until further feedback provided. The above PR could be a potential fix for this.

from extensions.

I fixed it with the command below.
But be careful if you do it.

unset FIRESTORE_EMULATOR_HOST
unset GCLOUD_PROJECT

from extensions.

In my case on Mac OS setting permissions for current user on the .npm/_npx folder helped

sudo chown -R USER /Users/USER/.npm/_npx/ 

Replace USER with your local username

from extensions.

I got this issue on a Mac, when i had GOOGLE_APPLICATION_CREDENTIALS set to something.

I managed to fix it by doing unset GOOGLE_APPLICATION_CREDENTIALS

Still not clear to me what's going on...

from extensions.

If anyone is getting this issue and is unable to fix it with the above techniques, please reply to this thread and i will reopen. Otherwise I think we could do with better error handling in this situation perhaps.

from extensions.

Getting this as well - repro'd on two different OSX machines (14.1.2). Node v18.19.0.

However, it also doesn't seem to matter. My data is getting copied just fine. I just had to use the right syntax to get a a nested collection with group queries.

from extensions.