codediodeio / firestore-migrator Goto Github PK

I'm attempting to run the Options to export the data from Firestore, but I'm getting "Unexpected token export" when I attempt to run it from Node command line.

Did I miss a step?

$ npm run build

[email protected] build D:\cjexpress\firestore-migrator
tsc

node_modules/google-auth-library/build/src/auth/oauth2client.d.ts(298,55): error TS1039: Initializers are not allowed in ambient contexts.
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] build: tsc
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Environment
OS : windows 10
Node: node 12.3.1

hi, is it possible to import a collection (eg in json data format) and have the import "recognise" that the collection also has sub-collections (and so those sub collections get created for you too) ?

Now that this package has a bunch of features, we should make the binary avail from the command line without the need to build it. npx.

npx fire-migrate someFile ...

I received this error just now as I was waiting for an upload to complete.

fire-migrate import --chunk 500 clubs.json clubs

Is there any way to allow a larger file upload / get around this deadline exception ?

I have this following issue when running npm run build (version node : 12.4.0)

node_modules/google-auth-library/build/src/auth/oauth2client.d.ts:298:55 - error TS1039: Initializers are not allowed in ambient contexts.

298     protected static readonly GOOGLE_TOKEN_INFO_URL = "https://oauth2.googleapis.com/tokeninfo";

Hi,

I build successfully after install, update, then link, and bin starts ok.
The file I load is a 1Gb CSV.

I get that error in the stacktrace:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

I've tried to modify the package.json

  "scripts": {
    "build": "node --max-old-space-size=8192 node_modules/typescript/bin/tsc",
    "build-link": "node --max-old-space-size=8192 node_modules/typescript/bin/tsc && npm link",
    "migrate": "node --max-old-space-size=8192 dist/index.js"
  },

But no success. Any idea where I should modify the max old space size ?

While running the command "npm install" I'm facing following issue

[email protected] install D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc
node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp ERR! Tried to download(403): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-win32-x64-unknown.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v64 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp ERR! Tried to download(undefined): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.1/node-v64-win32-x64-unknown.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v64 ABI, unknown) (falling back to source compile with node-gyp)
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationW
arning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
WWarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation

Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationW
arning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
WWarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation

WWarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation

WWarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation

Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationW
arning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationW
arning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationW
arning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
Warning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationW
arning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation
WWarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilationarning: unrecognized setting VCCLCompilerTool/MultiProcessorCompilation

Traceback (most recent call last):
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 16, in
sys.exit(gyp.script_main())
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp_init_.py", line 545, in script_main
return main(sys.argv[1:])
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp_init_.py", line 538, in main
return gyp_main(args)
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp_init_.py", line 523, in gyp_main
generator.GenerateOutput(flat_list, targets, data, params)
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2025, in GenerateOutput
version=msvs_version)
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\MSVSNew.py", line 213, in init
self.Write()
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\MSVSNew.py", line 340, in Write
f.close()
File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\common.py", line 393, in close
os.rename(self.tmp_path, filename)
WindowsError: [Error 183] Cannot create a file when that file already exists
gyp ERR! configure error
gyp ERR! stack Error: gyp failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:345:16)
gyp ERR! stack at ChildProcess.emit (events.js:189:13)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
gyp ERR! System Windows_NT 10.0.17763
gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "configure" "--fallback-to-build" "--library=static_library" "--module=D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown\grpc_node.node" "--module_name=grpc_node" "--module_path=D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown"
gyp ERR! cwd D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc
gyp ERR! node -v v10.15.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --library=static_library --module=D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown\grpc_node.node --module_name=grpc_node --module_path=D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown' (1)
node-pre-gyp ERR! stack at ChildProcess. (D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\node_modules\node-pre-gyp\lib\util\compile.js:83:29)
node-pre-gyp ERR! stack at ChildProcess.emit (events.js:189:13)
node-pre-gyp ERR! stack at maybeClose (internal/child_process.js:970:16)
node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
node-pre-gyp ERR! System Windows_NT 10.0.17763
node-pre-gyp ERR! command "C:\Program Files\nodejs\node.exe" "D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\node_modules\node-pre-gyp\bin\node-pre-gyp" "install" "--fallback-to-build" "--library=static_library"
node-pre-gyp ERR! cwd D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc
node-pre-gyp ERR! node -v v10.15.1
node-pre-gyp ERR! node-pre-gyp -v v0.6.39
node-pre-gyp ERR! not ok
Failed to execute 'C:\Program Files\nodejs\node.exe C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js configure --fallback-to-build --library=static_library --module=D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown\grpc_node.node --module_name=grpc_node --module_path=D:\firestoremigrator\firestore-migrator\node_modules\google-gax\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown' (1)
npm WARN [email protected] No repository field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-pre-gyp install --fallback-to-build --library=static_library
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\user\AppData\Roaming\npm-cache_logs\2019-03-13T12_14_20_499Z-debug.log

var admin = require("firebase-admin");

var serviceAccount = require("path/to/serviceAccountKey.json");

admin.initializeApp({
credential: admin.credential.cert(serviceAccount),
databaseURL: 'https://<DATABASE_NAME>.firebaseio.com'
});

I have an error in my credentials.json file. "Expected a JSON object, array or literal.json(0)"
I copied it directly from the Admin SDK configuration snippet.

Where am I going wrong?

exporting a Date/Timestamp Field from Firebase lead to e.g. an {"start":"2018-08-26T12:00:00.000Z"}
and vice versa importing this into firebase marks this field as string ...
does the script excepts any other date pattern ? or is this a bug ?

I am getting the following error when I try to run any fire-migrate cmd in the cli....

error:
fire-migrate : The term 'fire-migrate' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name,
or if a path was included, verify that the path is correct and try again.
At line:1 char:1

• fire-migrate import --dry-run test.json myCollection

•   + CategoryInfo          : ObjectNotFound: (fire-migrate:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
  

when exporting fire-migrate export users-posts.json users I get a file, when I want to import fire-migrate i users-posts.json users. the same file I get Unhandled promise rejection (rejection id: 1): TypeError: Object.entries is not a function

There is an issue when exporting documents that contain Reference fields. Each reference that gets exported downloads the entire Firestore database information with private keys and other nonsense that isn't needed.

The Firestore Reference fields should just export with the reference path not the entire firestore system information.

Has anyone else experienced this? How can I fix this?

Im no sure about this, but I had to run npm link first to get fire-migrate command working

When exporting a sub-collection they end up at the root level of the export, not inside of the parent document.

Before:
const subCollections = await getCollections(subCollPaths); _.assign(doc, subCollections);

Suggestion:

const subCollections = await getCollections(subCollPaths); for(let col in subCollections) { doc[snap.id][col] = subCollections[col]; }

When we import empty strings, the importer imports the value as a number type 0. Any workarounds for this besides leaving out the field completely?

With a vanilla install, I had the same issue running npm run build that others have had in various tickets. "Initializing static variables in ambient context..."

Ran npm update and all is well. Thinking this might be a good addition to the README.

I uploaded many documents to from an excel table to my production firestore using fire-migrate 👍.
It worked fine.

I now, want to have the same dataset in the firestore emulator. I then tried to export and import this dataset using gcloud cli but was not successful (see https://stackoverflow.com/q/73364863/1344369).

I wonder if I can just upload from my excel table to my emulator using fire-migrate. But I don't know how.

Imagine you have a huge spreadsheet with many unnecessary fields or unwanted data types, you could define an object that would convert the data to the proper Firestore structure and set defaults for missing vals.

export const item = new Schema({
  name: { type: 'string', default: 'hello' }
  location: { type: 'geopoint' }
  date: { type: 'timestamp', default: new Date() } 
})

Then you run a command like fire-migrate items.csv items --schema item

Hello,

I have around 350k documents in 1 of my collections and I am trying to export this data out of firestore...However, I run into the below error. Is there a way to get batch export? I tried --chunk, but it does not work for exports...

Any other solution? Please advice..

Error:

Failure: { Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
at Object.exports.createStatusError (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\common.js:91:15)
at ClientReadableStream._emitStatusIfDone (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:233:26)
at ClientReadableStream._receiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:211:8)
at Object.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1272:15)
at InterceptingListener._callNext (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:618:8)
at C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1029:24
code: 4,
metadata: Metadata { _internal_repr: {} },
details: 'Deadline Exceeded' }
PS C:\Users\JD\Desktop\Learn\practice\firestore-migrator> fire-migrate export --no-subcolls allquotes.json quotes
Getting selected collections...
Failure: { Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
at Object.exports.createStatusError (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\common.js:91:15)
at ClientReadableStream._emitStatusIfDone (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:233:26)
at ClientReadableStream._receiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client.js:211:8)
at Object.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1272:15)
at InterceptingListener._callNext (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:568:42)
at InterceptingListener.onReceiveStatus (C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:618:8)
at C:\Users\JD\Desktop\Learn\practice\firestore-migrator\node_modules\grpc\src\client_interceptors.js:1029:24
code: 4,
metadata: Metadata { _internal_repr: {} },
details: 'Deadline Exceeded' }

I have the following issue:
want to import this test csv to one of my collections:

number,title,max,language
1,Anna,Húrelmélető,hu
2,Peti,Teve,hu
3,Béla,Bélapátfalva,en
4,Sári,Mária,esp
5,Teknősbéka,Sándor,en
6,Mátyás,Fogkefe Kecskével,hu
7,Kecske,Mézeskalács,en

and the result is i have no accents.

the command im trying to run is:
fire-migrate import --id max --merge .\tesztsheet2.csv test3

can you help me with this?

thanks!

Hey guys,
Trying to import a collection from a JSON file that happens to be in this format:

{
    "manufacturer": "hello",
    "name": "world",
    "grading": "8",
    "status": "completed",
    "submitted_on": "{\"type\":\"timestamp\",\"data\":\"10/4/18\"}",
    "submitted_by": "fox mulder",
    "approved_for": [
      "group-2",
      "group-3",
      "group-4"
    ]
  },

For some reasons grading gets imported as a number.
Am I Missing something?

Thanks

Does it intended?

The FieldHandler seems never use BooleanFH when I'm trying to console log, it always stops at NumberFH and result was converting boolean value to integer type instead

Unrelated, creating a new issue. Give us a sample of your missing.csv to try and replicate the issue. The "--id u" option implies your csv needs a column with header "u" containing your doc IDs. Another hiccup might be type conversion, are the IDs meant to be strings/numbers?

Originally posted by @basementaspirations in #13 (comment)

hello there,

So, I am using the the fire-migrate import command and assign my own doc id.

fire-migrate import --id u missing.csv Articles

now when I try to search the data for a certain ID using an IOS app, it cannot find it. When I try to add the article through my app, it refreshes this record that got uploaded and updates the doc ID. after this step, the app is able to locate the article. There seems to be some issue with how doc ID is written during the upload process.

I have a video that I can show which shows the record update process in firestore. I really cannot see what changed, but it clearly seems to update the doc ID apart from some few document fields which is understandable.

Hello, and thank you for this app, helped me a lot! I am new to firestore and I was able to import my data from xlsx but date fields '2020-01-01' are written to firestore as strings how can I change that to write those as timestamp?
Thanks in advance

Hello,
Can we upload multiple csv files if they have created the index file? I used your repo example and tried to add to my firestore. Within the "csv-indexed" folder. I ran the following command

fire-migrate import --merge test.INDEX.csv myCollection

How ever, only the first file is written to the database and not the rest....any other way to automatically add one after the other?

I have about 450 rows each in 10 different files.

As per SDB on #slack/pro-members, Firestore seems to have a 500 row limit on batch imports.

We'll have to allow chunking/splitting the import.

Followed all the steps and keep getting this error when I run 'fire-migrate'

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'lodash.js'

Any ideas?

I'm using Node.js v18.15.0

This looks like a cool and useful utility.

If you are using JSON instead of CSV or XLS files though, and if you want to specify the document IDs instead of having them auto-generated, it would seem more elegant to use a JSON object structured as a dictionary of sub-objects, where the top level keys become document IDs instead of a JSON array and then needing to use the --id [id] switch.

Adding support for JSON objects and keeping the existing functionality for arrays would give the user flexibility to choose which approach to take.

npm link appears to be missing from the install instructions. Without it, there is no firestore-migrate executable.

Followed the readme directions - caused this error. I found a few similar issues elsewhere, where updating TS, or creating a files section in the tsconfig.ts will solve the issue, however, these solutions did not work in my situation. Does anyone know what this would cause? TS Version: 4.0.3

This happens after npm run build

node_modules/@types/node/index.d.ts:167:11 - error TS2300: Duplicate identifier 'IteratorResult'.

167 interface IteratorResult { }
~~~~~~~~~~~~~~

node_modules/typescript/lib/lib.es2015.iterable.d.ts:41:6
41 type IteratorResult<T, TReturn = any> = IteratorYieldResult | IteratorReturnResult;
~~~~~~~~~~~~~~
'IteratorResult' was also declared here.

node_modules/typescript/lib/lib.es2015.iterable.d.ts:41:6 - error TS2300: Duplicate identifier 'IteratorResult'.

41 type IteratorResult<T, TReturn = any> = IteratorYieldResult | IteratorReturnResult;
~~~~~~~~~~~~~~

node_modules/@types/node/index.d.ts:167:11
167 interface IteratorResult { }
~~~~~~~~~~~~~~
'IteratorResult' was also declared here.

Improve docs with all available options.

Hello everyone,
This is probably a noob question. When i type:
npm run build
I got this :

[email protected] build C:\Users\gogv\Desktop\firestore-migrator-master
tsc

And then the terminal reload, nothing else happen and i'm no able to launch a fire-migrate command.. (command not found)

Could you let me know what I'm supposed to do ?
Thank you in advance

Have a nice day

When I try to run an import, I get the error Error: Cannot find module 'excel-as-json'

I have had no luck installing this module.

I would also note that I've had to jump through a lot of hoops to get this CLI tool running, which used to work like a charm.

We should also include the ability to export as CSV or xlsx.

The data was uploading fine a week ago? Not seeing anything empty missing in the spreadsheet?

See output below..

Matthews-MacBook-Pro:firestore-migrator matt$ fire-migrate import --omit-empty-fields --dry-run homePageNewCats.xlsx homePageNewCats
Reading XLSX with options { sheet: '1', isColOriented: false, omitEmtpyFields: true }
XLSX read complete.
homePageNewCats/pmTrsnIgxgyx3mgNuacy
(node:68731) UnhandledPromiseRejectionWarning: Error: Argument at index 0 should not be empty.
    at new FieldPath (/Users/matt/Documents/GitHub/firestore-migrator/node_modules/@google-cloud/firestore/src/path.js:473:15)
    at Function.fromObject (/Users/matt/Documents/GitHub/firestore-migrator/node_modules/@google-cloud/firestore/src/document.js:1152:23)
    at WriteBatch.set (/Users/matt/Documents/GitHub/firestore-migrator/node_modules/@google-cloud/firestore/src/write-batch.js:292:41)
    at /Users/matt/Documents/GitHub/firestore-migrator/dist/importCollection/index.js:109:48
    at step (/Users/matt/Documents/GitHub/firestore-migrator/dist/importCollection/index.js:32:23)
    at Object.next (/Users/matt/Documents/GitHub/firestore-migrator/dist/importCollection/index.js:13:53)
    at /Users/matt/Documents/GitHub/firestore-migrator/dist/importCollection/index.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/matt/Documents/GitHub/firestore-migrator/dist/importCollection/index.js:3:12)
    at batchSet (/Users/matt/Documents/GitHub/firestore-migrator/dist/importCollection/index.js:101:12)
(node:68731) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:68731) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

hi there, first a huge thank you for this great migrator lib. Has been massive time saver.

Is it possible to use firestore-migrator to do a batch delete of a collection, prior to running it for an upload ?

or is there a flag i can set so that if i re-run an upload to a collection, it will overwrite, not append the data (documents) to the collection ?