kafkas / firewalk Goto Github PK
View Code? Open in Web Editor NEWA light, fast, and memory-efficient collection traversal library for Firestore and Node.js
Home Page: https://kafkas.github.io/firewalk
License: MIT License
A light, fast, and memory-efficient collection traversal library for Firestore and Node.js
Home Page: https://kafkas.github.io/firewalk
License: MIT License
Might want to add support for exiting traversal early based on some condition.
withConfig()
will return a new Traverser
instance similar to how firestore.Query.where()
and firestore.Query.limit()
work
node_modules/@firecode/admin/lib/api/interfaces/Migrator.d.ts:166:39 - error TS1005: ',' expected.
Especially important for createFastTraverser()
I think that something exposing an async iterator would be cleaner to use.
let count = 0;
let queue = [];
for await (const snapshot of createTraverser(...)) {
count++;
queue.push(sendEmail(snapshot.data());
if (queue.length >= 50) {
await promise.all(queue);
queue = [];
}
}
await promise.all(queue);
console.log(`Processed ${count} users.`);
The items can simply implement the async iterator methods, and fetch in batches internally.
Let's say that we have a number K
of docs to traverse. We should have at most N
docs held in memory at the same time. To get the best performance, should we :
batchSize
possible (=500), and set the maxConcurrentBatchCount
to N / batchSize
maxConcurrentBatchCount
logically possible (which I don't know), and set the batchSize
to N / maxConcurrentBatchCount
Thanks for helping!
The relevant discussion can be found in #23 .
I'm pretty sure that we don't do this but just want to double-check after someone on HN said this might happen. Here is the relevant discussion on HN.
Will eat more memory but traverse much faster.
Will be based on BulkWriter.
Use distinct collections for each migrator method and run those migrations concurrently.
Hello,
Is it possible to use "orderBy" to traverse a collection in a certain order ?
Suppose I have a collection "Data" with these kinds of docs :
{
id : ....,
name : ....,
}
I would like to traverse this collection in a way that the docs are ordered by id
(or by name
) between the different batches. What I mean by that is that the docs of batch1 should all have a "lower" id/name than the docs in batch 2, and so forth. The docs don't have to be ordered inside the same batch.
Is that possible? In what order is the collection traversed by default?
Thanks.
Work is being done on #20.
Currently, this library only supports firebase-admin 9.x
and 10.x
. We should add support for 11.x
.
There have been some typing changes in 11.x
so this is a bit more involved than just updating the dependencies to 11.x
as there are some compilation issues, mainly around the introduction of making firestore.UpdateData
generic.
Traversing Auth users is useful but our traversers are designed to traverse collections only.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.