llm_tools's People
Forkers
mhannanillm_tools's Issues
Memory efficiency
Huggingface reads the whole model into main memory and from there to GPU memory. So for a 27GB model we need 27GB of RAM during startup and afterwards we do not need it.
I remember there is a way to stream the model into the GPU memory instead of above behaviour. We should implement this to safe a large portion of RAM.
Warn about long input text
When the lenght of the input text exceeds a models maximum input length the API should throw an exception.
change the way parameters are handled
- drop the current implementation that uses one "args" argument
- add individual arguments temperate, top_k, ...
max_new_tokens parameter crashes the api
curl --location 'http://serv-9205.kl.dfki.de:5000/api/generate?max_new_tokens=10' \
--header 'content-type: application/json' \
--data '{
"doc": "Title: Immune_system\n\nBackground: Pattern recognition receptors are proteins used by nearly all organisms to identify molecules associated with pathogens. Antimicrobial peptides called defensins are an evolutionarily conserved component of the innate immune response found in all animals and plants, and represent the main form of invertebrate systemic immunity. The complement system and phagocytic cells are also used by most forms of invertebrate life. Ribonucleases and the RNA interference pathway are conserved across all eukaryotes, and are thought to play a role in the immune response to viruses.\n\nQuestion: What pathway that plays a role in immune response to viruses is present in all eukaryotes?\n\nAnswer:"}'
Use text-generation-webui
Replate the UI with https://github.com/oobabooga/text-generation-webui
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.