Significant memory spike in plugin with each() about crystal HOT 3 CLOSED

Update: This appears to not be related to applyTransform or loadOne at all. There's a simpler repro here with just each: https://github.com/scottravio/grafast-memory-spike-repro

from crystal.

This is due to the each() creating a subroutine (since it's not used as the return result of a plan resolver, but instead as input to another step), and subroutines are currently not hoisted (see #2041); as such we're having to run the each transform with a batch size of 1000 (because you're fetching 1000 users, and this is a field in the User type), and for each of those 1,000 user we're then mapping over the 1000 languages ultimately resulting in 1000*1000 = 1,000,000 values.

Since your step here is not dependent on $user, theoretically we should be able to hoist it (see #2041) but we currently do not. Instead, try and ensure that the steps that you use can all be hoisted (see the plan diagram to see which bucket the steps exist in); for example:

    plans: {
      User: {
        totalLanguages() {
          const $allLangugages = withPgClient(
            executor,
            constant(null),
            async (pgClient) => {
              const { rows } = await pgClient.query({
                text: `select language from languages`,
              });
              return rows;
            },
          );
          $allLangugages.hasSideEffects = false;
          return lambda(
            $allLangugages,
            (allLangugages) => allLangugages.length,
          );
        },
      },
    },

However, it would be better to directly address your need, which in this case seems to be counting the number of languages on a per-user basis, so perhaps something like:

    plans: {
      User: {
        totalLanguages($user) {
          const $agg = user_config.find({ user_id: $user.get("id") }).clone("aggregate");
          return $agg.single().select(sql`count(*)`, TYPES.bigint);
        },
      },
    },

Note that .clone is currently marked as an internal API, but that is resolved in #2039.

from crystal.

Closing this issue in favour of the related issues #2039, #2040 and #2041.

from crystal.