Slow performance while having large array of facts. about json-rules-engine HOT 9 OPEN

@jyothis-qb Any solution for speeding up Performance ?

from json-rules-engine.

@mjaniko @jyothis-qb I would need to see the actual rules to get a sense for exactly the cause. However I can say that if you're using JSON path expressions to turn an array of 80,000 objects into an array of 80,000 values then it's going to be slow.

Generally you may be better off trying to flatten/ normalize the data so there are no path expressions to evaluate.

from json-rules-engine.

I'm also facing the same issue but mine is a simple object, but array of object is huge. but the object itself doesn't have a nested path.

was there any resolution on this?

from json-rules-engine.

@JeffrinCh Again I would need to see a specific example to understand exactly but if you're using the path attribute you're going to experience some drop in performance as the results of the facts after applying the path transformation are not cached. @CacheControl there may be an opportunity to do some caching of the job path I can look into.
@JeffrinCh for now one option would be to create dynamic facts instead of using paths. These would benefit from caching and remove the need to parse JSON path expressions.

from json-rules-engine.

@jyothis-qb @JeffrinCh I did some digging and here's my suggestions:

This shows a comparison of runtime across 10,000 executions of calling Almanac.factValue on a fresh Almanac instance so no caching is enabled.
The big take-away is that if you involve the path parameter it will cause a slowdown, that will add up.

In order to speed up your access you could create dynamic facts:

engine.addFact(
  new Fact('factCol1', async (_, almanac) => {
      const f = await almanac.factValue('fact');
      return f.col1;
  });
)

If you're doing lots of path access you could simplify this by creating single facts and using parameters

engine.addFact(
   new Fact('factCol', async ({ col }, almanac => {
     const f = await almanac.factValue('fact');
     return f[`col${col}`];
  })
);

// access your fact with
{
   "fact": "factCol",
   "params": { col: 1 }
   ...
}

This provides a slightly improved performance over using the path value but not quite the same performance benefit of having a specific dynamic fact.

from json-rules-engine.

json-path requires a decent amount of overhead; it's a relatively complex spec. I'm not surprised that using the path feature on a large number of items is causing significant impact.

Its my belief that the underlying json-path library we use (jsonpath-plus) is well optimized for performance, however if there is a performance improvement to be made, it will reside in that library.

I agree with the workaround above of using dynamic facts in place of json-path.

from json-rules-engine.

@CacheControl I also did some digging / profiling of jsonpath-plus and it does seem to be very very optimized. It already caches the results of compiling a path into a function so repeated uses of the same path will not cause re-compilation. The performance is so optimized that even without the caching the behavior is only slightly abnormal.

Suggesting this could probably be closed with Dynamic Facts being the solution.

from json-rules-engine.

@CacheControl I also did some digging / profiling of jsonpath-plus and it does seem to be very very optimized. It already caches the results of compiling a path into a function so repeated uses of the same path will not cause re-compilation. The performance is so optimized that even without the caching the behavior is only slightly abnormal.

Suggesting this could probably be closed with Dynamic Facts being the solution.

Will try these and check

from json-rules-engine.