Python server for ingesting & parsing documents and other data magic
poetry install
poetry shell
poetry run uvicorn databee:app --reload --port 8001
poetry run pytest
GPUtopia data magic daemon
Home Page: https://databee.gputopia.ai
License: GNU Affero General Public License v3.0
We want to add L402 support to any API endpoint so we can start moving beyond the "account and balance" model to just paying for services you need with bitcoin, no questions asked
We will run a databee and anyone else can run their own databee, specifying what Lightning address they should receive payments at
Probably worth adapting https://github.com/Kodylow/matador
Given a text document (PDF, Markdown, etc.) of any length, we need to extract a list of relevant topics, perhaps 5-20 or more.
This is to help us generate question/answer keypairs programmatically from a document which can be used for finetuning a model.
Once we have a list of topics, we can easily query the entire document for each topic for enough context to generate relevant question/answer keypairs via an LLM.
The question is, what is the best way to generate the list of topics?
Conceivably this is what topic modeling is for, but our current implementation using Gensim and LDA gives bad results. We can explore third-party services like this but unsure topic modeling is even the right approach, and prefer to minimize dependence on third parties.
Maybe we should generate topics another way like using semantic search via a service like Vectara or an LLM API like Claude that works with long documents.
Suggestions welcome!
Bounty: 1M sats to whomever describes an algorithm we end up using for this, or 2M sats if it's fully implemented with a PR we can easily merge.
As discussed, let's get function(s) added to this databee repo enabling RAG over documents (PDF, Markdown, text).
You can use your discretion about what parts of Arguflow should be added in what order, whatever maximally supports the goals explained below.
Code can go within an arguflow.py
file in the databee folder.
Additional background:
Our forthcoming agents UI enables users to create one or more agents, upload docs to their "brain", ask them questions about the docs, give them tasks based on the docs, and finetune models based on the docs.
Our implementation is based partly on the Agent Protocol, but adding support for multiple agents per server.
Here are the (tentative) data models and relationships:
RAG will help to enable:
Thorough integration of Arguflow to enable RAG in support of the above workflow will earn a bounty of 25M sats. (Less for an incomplete integration that we need to do further work before including.)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.