firebaseextended / firebase-import Goto Github PK

Node module for importing large data into Firebase.

License: MIT License

JavaScript 100.00%

firebase-import's Introduction

Firebase-Import

Firebase-import is a helper utility for importing large JSON files into your Firebase Realtime Database. It breaks the JSON into smaller chunks and uploads them individually through the Firebase API.

This utility is designed and tested for imports of files up to 400MB.

Installing

Install the firebase-import module globally:

$ npm install -g firebase-import

or install it locally and add it to your path:

$ npm install firebase-import
$ export PATH=$PATH:`npm bin`

Obtaining Service account file

Using your Google account, login to Firebase console: https://console.firebase.google.com/

See Example below

Project Settings ->
Service Accounts ->
Generate new private key

Usage

$ firebase-import
Usage: firebase-import

Options:
  --database_url     Firebase database URL (e.g. https://databaseName.firebaseio.com).   [required]
  --path             Database path (e.g. /products).                                     [required]
  --json             The JSON file to import.                                            [required]
  --merge            Write the top-level children without overwriting the whole parent.
  --force            Don't prompt before overwriting data.
  --service_account  Path to a JSON file with your service account credentials.

Example

$ firebase-import --database_url https://test.firebaseio-demo.com --path / --json test.json --service_account /path/to/your/service_account.json
All data at https://test.firebaseio-demo.com/ will be overwritten.
Press <enter> to proceed, Ctrl-C to abort.

Reading /Users/michael/tmp/test.json... (may take a minute)
Preparing JSON for import... (may take a minute)
Importing [=================================================] 100% (9431/9431)
Import completed.

Or an example of merging the contents of test.json with what's already in Firebase:

$ firebase-import --database_url https://test.firebaseio-demo.com --path / --json test.json --merge --service_account /path/to/your/service_account.json
Each top-level child in test.json will be written under https://test.firebaseio-demo.com/.
If a child already exists, it will be overwritten.
Press <enter> to proceed, Ctrl-C to abort.

Reading /Users/michael/tmp/test.json... (may take a minute)
Preparing JSON for import... (may take a minute)
Importing [=================================================] 100% (9431/9431)
Import completed.

firebase-import's People

Contributors

Stargazers

Watchers

Forkers

ferlyz05 ajhurliman lietmotiv chixcancode apgiorgi jonesfish narwhalgo joejoeknight 1312343 2947721120 malkafly alexboorman mcexco tomazy niilante emomoh schwer leminhtuan2015 davidstalane mscalora theworldtable feedingonyourflesh everythingumayneed neuralnoise madbarua ykankaya mattlk13 forkme7 jeremip11 siege-nnn andyhaas web20opensource jeremyplease nvthuong1996 38438-38438-org todun munyola x-frontkit jcalan8907 youssef-sourour puppup420247-org fahrulmustari6 pphak75 addtoevent gomezo76 capnv12 alexeynovichenko oxcarga el-amoura isabella232 jaehyunjung83 thomasabbink jkugker05 addingama maybui trizow jozef-javorsky-dodo

firebase-import's Issues

Issue with firebase-import.js:51:22

I'm probably using this wrong, but I get this:

Error: Firebase.child failed: First argument must be a non-empty string and can't contain ".", "#", "$", "[", or "]".
    at Error (<anonymous>)
    at Ha (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:12:204)
    at H.J.F (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:132:213)
    at main (/usr/local/lib/node_modules/firebase-import/bin/firebase-import.js:51:22)
    at Object.<anonymous> (/usr/local/lib/node_modules/firebase-import/bin/firebase-import.js:236:1)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)

Please update ReadMe with where to get the service account json

https://console.cloud.google.com/iam-admin/serviceaccounts

Hard time importing 20 mb json file

I'm uploading a json file that is right around 20mb and it's loading at a snails pace. I also tried using the firebase-streaming-import tool and for whatever reason that couldn't handle a 1.7 mb json file. It was just stuck at 'starting at'. What can I do to improve the upload speed of firebase-import and is there anything I can do to fix firebase-streaming-import?

My import is currently at (164/48513) and it's been running for 15 minutes.

Thanks!

usage of require limits file size of json

If you use the "require" to read the .json file, it will limit the size of the json file. A streamed json parser, like JSONStream might improve the capability to read large files. Granted, this approach would not solve the memory usage issue for those files once they are memory resident, but it would give a little more headroom.

How to enable permissions

Hello, just wondering how to enable the permissions since I see this message:
FIREBASE WARNING: The Firebase database 'abc-app-default-rtdb' has been disabled by a database owner.
Thank you

Failed to connect to Firebase

All worked on summer 2022. November 2022 doesn't work.
I'm trying to merge a small json and receive: "Failed to connect to Firebase."

publish last version on npm

Hi,

It looks like the master branch has a package.json file that has a version 0.0.0 https://github.com/firebase/firebase-import/blob/master/package.json#L4

I think this needs to be 2.0.1 and the package needs to be published on npm.

Currently if you install npm install -g firebase-import you don't get the latest version that is in master

Allow use of default service account

Example use case

Running firebase-import inside a Cloudbuild step
Default service account has appropriate role on Firebase RTDB
In the current implementation we will need to securely load a copy of the default service account credentials which are already available

Proposed solution

Given I'm running the CLI in cloudbuild
When the Service account path is undefined
Then use the default service account e.g

 admin.initializeApp({
    credential: argv.service_account ? admin.credential.cert(argv.service_account) : admin.credential.applicationDefault(),
    databaseURL: argv.database_url
  });

Despite setting --path to /, it points to /C%3A/Program%20Files/Git.

This is my input and what I get out:

$ firebase-import --database_url https://projectName.firebaseio.com --path / --json allTest.json

The 'serviceAccount' property specified in the first argument to initializeApp() is deprecated and will be removed in the next major version. You should instead use the 'firebase-admin' package. See https://firebase.google.com/docs/admin/setup for details on how to get started.
All data at https://projectName.firebaseio.com/C%3A/Program%20Files/Git will be overwritten.
Press <enter> to proceed, Ctrl-C to abort.

I hit enter, it proceeds apparently normally, but when I check my database none of the new data in my json file has been uploaded; my old data is untouched. Why's it adding that extra ProgramFiles/Git to the path?

Update to include streaming support for larger JSON files

The README suggests using firebase-streaming-import for files larger than 250MB. My database that I'd like to import is well over that threshold (6GB+), and with the speed of ~30seconds/MB, that ends up being somewhere around 50+ hours (and has to be run twice). Also, firebase-streaming-import hasn't been updated in almost 3 years.

I'm sure this could be improved, and should be included as a part of this package using node streams (I'd be surprised if I was the only 250MB+ user...).

--merge flag has unexpected behavior

Perhaps it's with my json structure, or not understanding parent and child, but I run firebase-import with 1.json

$ cat 1.json
{"parent": {"child_a" : 1 }}
$ firebase-import --database_url https://my_db.firebaseio.com/ --path / --json 1.json --service_account auth.json

which gives me this:

then I run with this (notice the --merge flag)

$ cat 2.json 
{"parent": {"child_b" : 2 }}
$ firebase-import --database_url https://my_db.firebaseio.com/ --path / --json 2.json --service_account auth.json --merge

and my first json is deleted and only the second is set.

I am using [email protected]

Shouldn't merge be doing an update, and not clearing the parent node? I am using geofire, which means all my keys need to be under the same parent node. But my JSON files are quite large, already at the limit of firebase-import, so I would like to not combine the JSON files into one. In the end I need a structure like this, but from multiple JSON files

{"parent" : {"child_a" : 1, "child_b" : 2}

Not working behind a proxy

I cannot use firebase-import behind a proxy.
I have try to set environment variable HTTP_PROXY but it don't change anything.

Is there any workaround ?

Thanks

Importing firebase user accounts

Hello,

For an app I'm building, I need to create close to 50k user accounts on my firebase. I'm using the email & password Authentication.

I have the list of user accounts in json format ready with a random password generated.

My question is, can I use this tool to import my user accounts ?

Kind regards,

Florian

How to Generate PushKeys for data in Json file

I have json data as follows:
I want this to add under a node like /courses in database:

{
"description" : "test course",
"title" : "New Course Title",
"url" : "https://courseurl.com"
}

But on import, It does not generate PushKeys like it does when we invoke .push({obj}) method of Firebase.
SO how to generate PushKey on import for data?

Not overwriting top-level child i.e. --append

Can we have an option to --append that doesn't overwrite the top-level child?

Error: WRITE_TOO_BIG

Error: WRITE_TOO_BIG: Data to write exceeds the maximum size that can be modified with a single request.

Json size is 291.8 MB

Passing modern (2021) database URL is not accepted (for example with another region, but also TLD is new – firebasedatabase.app)

firebase-import --database_url https://foo-bar-default-rtdb.europe-west1.firebasedatabase.app --path / --json foo.json --service_account scripts/foo-bar-admin.json

causes

Error: FIREBASE FATAL ERROR: Cannot parse Firebase url. Please use https://<YOUR FIREBASE>.firebaseio.com

But attempting to pass this database URL in any other form fails from the firebase side:

[2021-03-18T17:52:51.311Z]  @firebase/database: FIREBASE WARNING: Namespace foo-bar-default-rtdb lives in a different region. Please change your database URL to https://foo-bar-default-rtdb.europe-west1.firebasedatabase.app (https://foo-bar-default-rtdb.firebaseio.com)

Can't handle large imports

Throws FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory for large files (greater than 256MB).

sync stops working when path is blown away with firebase-import

I'm using firebase-import here:
https://www.dnwdebt.com/app#/demo/2500

To refresh the demo data every 30 minutes.

I notice that most times the sync stops when data is refreshed from json backups.
It requires the user to refresh the page.

IMHO this is a bug with firebase. $ref points to a path and doesn't receive any events that data was recreated and the client should be resynced. It seems the delete events occurred but not the create.

Error Javascript heap out of memory

Brief: While uploading a 400mb file using this tool, I get Javascript heap overflow error.

Command used:

firebase-import --database_url https://database.firebaseio.com/ --path / --json C:\User\Desktop\file.json --service_account C:\Users\service_account.json

Complete Error log:

<--- Last few GCs ---> [4052:000002065D718290] 155635 ms: Mark-sweep 1375.3 (1399.6) -> 1375.3 (1399.6) MB, 3254.4 / 0.0 ms (average mu = 0.091, current mu = 0.000) last resort GC in old space requested [4052:000002065D718290] 158828 ms: Mark-sweep 1375.3 (1399.6) -> 1375.3 (1399.6) MB, 3192.5 / 0.0 ms (average mu = 0.047, current mu = 0.000) last resort GC in old space requested <--- JS stacktrace ---> ==== JS stack trace ========================================= 0: ExitFrame [pc: 000000079E2DC5C1] Security context: 0x00d110a08ce9 1: push [000002490F19CC39](this=0x031861c9b181 <JSArray[1956331]>,0x02b158d904a1 ) 2: chunkInternal [000000C0F32CEA41] [C:\Users\firebase-import.js:~179] [pc=000000079E5D300A](this=0x001a0279ad11 ,ref=0x031861c9b1a1 <Reference map = 000002D... FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: 00007FF78778121A v8::internal::GCIdleTimeHandler::GCIdleTimeHandler+4810 2: 00007FF78775A5B6 node::MakeCallback+4518 3: 00007FF78775AFA0 node_module_register+2160 4: 00007FF7879EB3EE v8::internal::FatalProcessOutOfMemory+846 5: 00007FF7879EB31F v8::internal::FatalProcessOutOfMemory+639 6: 00007FF787F29304 v8::internal::Heap::MaxHeapGrowingFactor+11476 7: 00007FF787F27098 v8::internal::Heap::MaxHeapGrowingFactor+2664 8: 00007FF787AC9C08 v8::internal::Factory::AllocateRawArray+56 9: 00007FF787ACA582 v8::internal::Factory::NewFixedArrayWithFiller+66 10: 00007FF787B3F91C v8::internal::HashTablev8::internal::NumberDictionary,v8::internal::NumberDictionaryShape::KeyAt+43820 11: 00007FF787B3F59D v8::internal::HashTablev8::internal::NumberDictionary,v8::internal::NumberDictionaryShape::KeyAt+42925 12: 00007FF787B33396 v8::internal::HashTablev8::internal::NumberDictionary,v8::internal::NumberDictionaryShape::EntryToIndex+71382 13: 00007FF787D061CE v8::internal::SharedFunctionInfo::SetScript+23662 14: 00007FF787CE8CA3 v8::internal::JSReceiver::class_name+20931 15: 00007FF787AB9FD1 v8::internal::wasm::WasmCodeManager::LookupCode+18929 16: 00007FF787ABD068 v8::internal::wasm::WasmCodeManager::LookupCode+31368 17: 000000079E2DC5C1

file smaller than 400mb but can't import at "/" path, failed: write_too_big

Hello I have problem to import firebase realtime database (360mb), When i put export.json to root of my database i got this error (below) but when i choose a path other than "/" like "--path /test" it work, I don't understand why it doesn't work at the root.

`Alexandre:Documents alex$ firebase-import --database_url https://url.firebaseio.com/ --path / --json export.json --service_account /path/to/key/my_key.json 

All data at https://url.firebaseio.com/ will be overwritten.
Press <enter> to proceed, Ctrl-C to abort.
Reading /Users/alex/Documents/export.json... (may take a minute)

Preparing JSON for import... (may take a minute)
[2019-07-09T18:53:23.902Z]  @firebase/database: FIREBASE WARNING: set at / failed: write_too_big 
[2019-07-09T18:53:23.907Z]  @firebase/database: FIREBASE WARNING: Exception was thrown by user callback. Error: WRITE_TOO_BIG: Data to write exceeds the maximum size that can be modified with a single request.
    at /usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:13139:33
    at exceptionGuard (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:695:9)
    at Repo.callOnCompleteCallback (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:13130:13)
    at /usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:12907:19
    at /usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:12078:17
    at PersistentConnection.onDataMessage_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:12111:17)
    at Connection.onDataMessage_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:11394:14)
    at Connection.onPrimaryMessageReceived_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:11388:18)
    at WebSocketConnection.onMessage (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:11289:27)
    at WebSocketConnection.appendFrame_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:10892:18) 
/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:706
            throw e;
            ^

Error: WRITE_TOO_BIG: Data to write exceeds the maximum size that can be modified with a single request.
    at /usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:13139:33
    at exceptionGuard (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:695:9)
    at Repo.callOnCompleteCallback (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:13130:13)
    at /usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:12907:19
    at /usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:12078:17
    at PersistentConnection.onDataMessage_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:12111:17)
    at Connection.onDataMessage_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:11394:14)
    at Connection.onPrimaryMessageReceived_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:11388:18)
    at WebSocketConnection.onMessage (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:11289:27)
    at WebSocketConnection.appendFrame_ (/usr/local/lib/node_modules/firebase-import/node_modules/@firebase/database/dist/index.node.cjs.js:10892:18)
`

Does this work with the new Firestore DB?

Planing to move my no-sql db from mongolab to Firestore, and how to import the data and found this utility and I wonder if this works with firestore.

Only works if file ends in .json.

Since we do a require(), it'll try to parse as js and fail.

Can't import a backup export after fresh install

I installed this from npm (node 6.11.2) and ran firebase-import --database_url https://castle-scott-lowe.firebaseio.com/ --path / --json 2017-09-06T10-50-37Z_castle_data.json to import a recent export, but got the following error:

Reading /Users/scott/Downloads/2017-09-06T10-50-37Z_castle_data.json... (may take a minute)
buffer.js:11
    super(arg1, arg2, arg3);
    ^

RangeError: Invalid typed array length
    at Buffer.Uint8Array (native)
    at FastBuffer (buffer.js:11:5)
    at createUnsafeBuffer (buffer.js:38:12)
    at allocate (buffer.js:181:12)
    at Function.Buffer.allocUnsafe (buffer.js:141:10)
    at tryCreateBuffer (fs.js:475:21)
    at Object.fs.readFileSync (fs.js:520:14)
    at Object.Module._extensions..json (module.js:585:20)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)

Can you give an example Json file

Hi It would be good to show an example json file .

Provide option to use .push instead of .set

It would be nice to use .push so that when data is saved it creates a uniqueidentifier.

TOO_MANY_TRIGGERS: This request would cause too many functions to be triggered.

I import 370Mb json

TypeError: Cannot read property 'length' of undefined

size += JSON.stringify(value).length;
^

TypeError: Cannot read property 'length' of undefined

Update to work with the new SDK

I think this library should be updated in order to work with the new Firebase SDK and auth tools. I'm trying to authenticate with my Firebase project but I can't.

How do you specify credentials?

I tried adding ?auth=<secret> to the url, but received a Failed to connect to Firebase. error.

serviceAccount deprecated warning - status of updates to firebase-import

I run the following command:

firebase-import --database_url https://my_db.firebaseio.com --path / --json data.json --service_account auth.json

and get back a deprecation message about serviceAccount

The 'serviceAccount' property specified in the first argument to initializeApp() is deprecated and will be removed in the next major version. You should instead use the 'firebase-admin' package. See https://firebase.google.com/docs/admin/setup for details on how to get started.

Will firebase-import be updated to reflect this change? (Or should I start migrating my code away from this package?

I am using 2.1.0

Import times out?

I consistently get the following error after trying to upload a JSON file that is ~40MB in size. Any ideas why? It always happens when the upload is 42% complete, and that portion of the JSON is successfully uploaded to Firebase.

FIREBASE INTERNAL ERROR: Server Error: ClientId[-----]:ErrorId[5]: Error on incoming message
Importing [==================== ] 42% (1610/3865)
/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:44
function ac(a){try{a()}catch(b){setTimeout(function(){throw b;},Math.floor(0))
^
AssertionError: false == true
at onComplete (/usr/local/lib/node_modules/firebase-import/bin/firebase-import.js:222:7)
at /usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:128:47
at ac (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:44:20)
at X (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:128:22)
at /usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:121:291
at /usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:85:276
at md.h.bc (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:86:104)
at ad.bc (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:77:364)
at Q.Od (/usr/local/lib/node_modules/firebase-import/node_modules/firebase/lib/firebase-node.js:75:280)
at Ec (/usr/local/lib/node_modules/firebase-i

Please update Readme: firebase-streaming-import is no longer useable

README.md in this repository points to firebase-streaming-import for JSON files larger than 250MB.

After extensive testing with that package, and from reading the issues and comments, and also noticing the last commit was more than 4 years ago, I would request that the link from this package to that one would be removed - due to that package being not supported and in-effect "broken".

(according to the comments in a similar issue, it seems streaming support is ready in firebase-import but needs to be pushed to npm, although with 2.1.0 I'm not able to load 1gb JSON)

"First argument was an invalid path" error

Error: Firebase.child failed: First argument was an invalid path: "". Paths must be non-empty strings and can't contain ".", "#", "$", "[", or "]"

I am calling like so: firebase-import --database_url [redacted url] --path / --json "[redacted path]"

Anyone know what I'm doing wrong? I am on windows 10 using the google cloud sdk shell.