Comments (1)
The following is the summary of the related discussion with @cyyeh
- The pipeline evaluation process will follow the "ask pipeline eval" methodology.
- The initial dataset for pipeline evaluation will be the "baseball" data from the Spider dataset.
- For creating the basic dataset needed for evaluation, we will use either GPT-3.5-turbo or GPT-4.
- (Optional) Consider employing a Large Language Model (LLM) to rank responses and assign scores.
- The evaluation must ensure that all queries from the evaluation steps can be executed on the core server.
- Due to the requirement in point 5, it's also necessary to generate the CTE from the step query and verify that the results match those provided by the Spider dataset.
from wrenai.
Related Issues (20)
- how I switch different datasource?add multiple data sources? HOT 3
- wren-ai-service is not starting on windows laptop HOT 1
- Macos Ollama : failed to create asking task + timeout HOT 1
- BigQuery metadata ingestion HOT 2
- all tables have foreign keys, but not a single relation gets shown on the model page HOT 3
- Unknown dialect 'mssql'. Did you mean mysql? HOT 1
- Unable to deploy model with embedding model text-embedding-ada-002 HOT 1
- Mssql database connection HOT 3
- Ibis server error HOT 4
- multilingual input and output HOT 2
- Developer documentation HOT 1
- Change LLM during Docker running HOT 1
- Download in md, json "http://localhost:3000/modeling" HOT 3
- Ibis server error '42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server] HOT 1
- Installation of Wrenai fail HOT 2
- Running on Local
- I am getting error in deploying my database model as I am getting error "Failed to deploy. Please check the log for more details." HOT 8
- Need to have a way to save all Model's description entered by the user. This can be later re-applied to the models when working with other databases using same structure.
- Availability to update MDL file to design semantic model instead of using UI modeling page
- wren-ai-service version mismatch between pyproject.toml and docker image
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wrenai.