Text to SQL(ask) pipeline performance benchmark about wrenai HOT 4 CLOSED

The following points are the summary of the related discussion with @cyyeh

• All the Experiments and Performance Benchmarks will be recorded on this page
• Didn’t decide to use some (e.g. tool, chart) to show the performance benchmark yet.

from wrenai.

After discussion, we'll use latency, cost and accuracy compared with naive RAG pipeline, optimized RAG pipeline and optimized RAG pipeline with semantics description

from wrenai.

This is not the focus as of now. I'll close it until we need this in the future.

from wrenai.

This is the conclusion of Text-to-SQL(ask) pipeline performance benchmark

• objectives
  • gpt 3.5 turbo + wren engine + with semantics descriptions
  • gpt 3.5 turbo + wren engine
  • gpt 3.5 turbo
• data
  • using part of Spider dataset
  • like data.world’s four dimensions
• metrics
  • accuracy
  • cost
  • latency
  • explainability?(optional)
    • maybe using llm as judge

from wrenai.