kafkas / firewalk Goto Github PK

Might want to add support for exiting traversal early based on some condition.

withConfig() will return a new Traverser instance similar to how firestore.Query.where() and firestore.Query.limit() work

Especially important for createFastTraverser()

I think that something exposing an async iterator would be cleaner to use.

  let count = 0;
  let queue = [];
  for await (const snapshot of createTraverser(...)) {
    count++;
    queue.push(sendEmail(snapshot.data());
    if (queue.length >= 50) {
      await promise.all(queue);
      queue = [];
    }
  }
  await promise.all(queue);
  console.log(`Processed ${count} users.`);

The items can simply implement the async iterator methods, and fetch in batches internally.

Let's say that we have a number K of docs to traverse. We should have at most N docs held in memory at the same time. To get the best performance, should we :

• Choose the maximum batchSize possible (=500), and set the maxConcurrentBatchCount to N / batchSize
• Choose the maximum maxConcurrentBatchCount logically possible (which I don't know), and set the batchSize to N / maxConcurrentBatchCount
• Have preferred values for either the batchSize or the maxConcurrentBatchCount depending on N. In this case, what would these values be ?

Thanks for helping!

The relevant discussion can be found in #23 .

I'm pretty sure that we don't do this but just want to double-check after someone on HN said this might happen. Here is the relevant discussion on HN.

Will eat more memory but traverse much faster.

Will be based on BulkWriter.

Use distinct collections for each migrator method and run those migrations concurrently.

Hello,
Is it possible to use "orderBy" to traverse a collection in a certain order ?

Suppose I have a collection "Data" with these kinds of docs :

{
  id : ....,
  name : ....,
}

I would like to traverse this collection in a way that the docs are ordered by id (or by name) between the different batches. What I mean by that is that the docs of batch1 should all have a "lower" id/name than the docs in batch 2, and so forth. The docs don't have to be ordered inside the same batch.

Is that possible? In what order is the collection traversed by default?

Thanks.

Work is being done on #20.

Currently, this library only supports firebase-admin 9.x and 10.x. We should add support for 11.x.

There have been some typing changes in 11.x so this is a bit more involved than just updating the dependencies to 11.x as there are some compilation issues, mainly around the introduction of making firestore.UpdateData generic.

Traversing Auth users is useful but our traversers are designed to traverse collections only.