qai222 / llm_organic_synthesis Goto Github PK

What is this feature about?

1. openai model
2. open-source model
   3. dash app

Is your feature request related to a problem? Please describe.

hard to see the differences between truth and inference

Describe the solution you'd like

something like https://www.jsondiff.com/

What is this feature about?

4. evaluation

Is your feature request related to a problem? Please describe.

Use tree edit distance to describe differences between ground truth and inferred JSON. Preferably field-resolved: ex. 5 edits for all compound.identifiers fields.

Describe the solution you'd like

A function takes into two ORD JSON or messages and produces an edit distance.

Describe alternatives you've considered

1. A naive way is to use deepdiff and count the leaves in outputs. But it gets complicated when edits are branches.
2. proto_compare from proto-matcher. But it only produces text descriptions.
3. Use graph edit distance like GMatch4py which requires conversion from JSON to networkx.
4. Some tree edit distance packages
   1. Apted, it only takes in bracket notations. Here is a parser that may be useful. Some useful tools by the same authors can be found here.
   2. zss, seems buggy tho.