Comments (2)
The type of data required for fine-tuning depends on the task the model is intended to perform.
Best practices for fine-tuning include:
- Using a similar data distribution as the original data
- Using enough data to ensure the model can generalize well
- Using multiple epochs to allow the model to learn from the new data
- Using a small learning rate to prevent overfitting
- Monitoring the model's performance on a validation set
- Using a pre-trained model for better performance
- Regularizing the model to prevent overfitting
from openchatkit.
If you're trying to reproduce the GPT-NeoXT-Chat-Base-20B
model, you can download the dataset by running python data/OIG/prepare.py
from the root of the repository.
We plan to add more documentation about fine tuning on your own data. Please see #10.
from openchatkit.
Related Issues (20)
- We couldn't connect to 'https://huggingface.co' HOT 1
- Add CodeAlpaca-20k dataset to improve coding skills.
- -
- When use one Gpu do model training, met one issue. HOT 3
- Environment Issues On Mac HOT 1
- Example script for continued pre-training? HOT 2
- How to disable AWS_ACCESS_KEY_ID when fine tuning? HOT 2
- LOST in the MIDDLE
- how many card days to Fine-tuning Llama-2-7B-32K-beta
- An error occurred while fine-tuning the model. HOT 3
- Cannot setup environment HOT 1
- Training on BookSum HOT 1
- how to train Fine-tuning Llama-2-7B-32K-beta?
- How to start the combined server/ send commands over HTTP?
- API is not working when inferenced with streamlit
- NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.
- ModuleNotFoundError: No module named 'flash_attn' HOT 1
- What is minimum resource requirement to fine-tuning Llama-2-7B-32K-beta model.
- H
- Newbie question. ยฟCan I use this to build a chat assistant? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openchatkit.