Comments (9)
Thanks @merveenoyan for the awesome feedback.
Isn't fine-tuning technically a subsection of transfer learning? Maybe consider framing it differently.
Well, yes that's right. However, there are nuances. And I have seen lot of folks get confused and debate about this topic. Here and Here. So, in that subsection I want to give a clear idea on how each of them works. Let me know if that sounds reasonable and also feel free to suggest some better sub section titles :)
Also we remove the classifier head and add our own head and unfreeze last layers that are responsible of feature extraction and retrain, is this what you mean here?
Yes, I wanted to write that in short 😅, may be, need to find some better titles for subsections.
I feel like unless we're working on tabular data it's almost always beneficial given it reduces amount of data we need to use and increases performance.
Yes, so, should we go and include this subsection or like remove it. You can also think of as section that includes the best practices.
Maybe one thing you can add is knowledge distillation.
Yes, that would be awesome, but should that be included under fine-tuning and transfer learning, or should I create a different issue for that? Because by definition they are dis-similar?
Thanks and let me know.
from computer-vision-course.
Uhm, it would be better, if we take this discussion on public discord cv-community course channel.
Sure. But how is this supposed to work without having your handles? Feel free to answer me on Discord via @matthiasdroth.
from computer-vision-course.
Hello 👋
I think this is very nice.
Transfer Learning vs Fine-tuning.
Isn't fine-tuning technically a subsection of transfer learning? Maybe consider framing it differently.
Removing the last layers and adding a MLP + additionally re-train a small percentage of the layers
Also we remove the classifier head and add our own head and unfreeze last layers that are responsible of feature extraction and retrain, is this what you mean here?
When to use Fine-tuning and Transfer Learning
I feel like unless we're working on tabular data it's almost always beneficial given it reduces amount of data we need to use and increases performance.
Maybe one thing you can add is knowledge distillation. I've recently experimented around it to add it to transformers docs (will be released today) and observed a good amount of improvement, especially now that we have foundational models that are very good yet hard to fine-tune, they'd be useful for KD.
Overall, good compilation!
from computer-vision-course.
Hey, thanks for the outline 🤗
Here are my thoughts:
- A small introduction to Knowledge Transfer. Why training vision models from scratch are not always the solution.
- Transfer Learning vs Fine-tuning.
These are very fundamentals parts and it is good to cover them. But as we will already have a Transfer Learning/Fine-Tuning section in the CNN chapter, before this one, make sure to check with the team that is working on it (they don't have submitted an outline yet) to avoid too much overlap. I'd say focus on the differences when it comes to Vision Transformers and you can also give a lot of room to the transformers (library) specific methods.
Transfer Learning in depth with 🤗 transformers / torch code.
- Removing the last layer and adding a MLP
- Removing the last layers and adding a MLP + additionally re-train a small percentage of the layers
Fine Tuning in depth with 🤗 transformers / torch code
- General Fine-tuning (i.e. take a small transformer and full fine-tune it)
- Fine-tuning using PEFT (example: LoRA). This should include a small intro on Parameter Efficient Techniques.
I really like that you are using transformers and PEFT here, especially the PEFT methods are something that I hardly see covered in courses yet.
For the MLP part I would says the same as above - try to avoid overlap, best focus on transformer specific attributes, maybe point out some differences with CNNs, that would make a good connection to the former chapter.
In general it would be great to have some task specific fine-tuning examples, which you can connect with your overall outline. Give an example for simple classification at some point, at another use a segmentation task, etc.
I think it is important to not become too theoretical here, but give the participants something they can run themselves and get great results. Afterall that is the magic of fine-tuning - get good results with relatively little effort.
But I'm sure you've got this and it will be an awesome chapter 👍 🙂
from computer-vision-course.
Hey, so quite a while, and addressing all the feedback, we have come up with this structure. We started to work accordingly. Let us know this looks right or not. We will also add the PRs that are open/merged in our and the main repo.
Contents
- Transfer Learning and Fine-tuning definitions with some diagrams.
- Discuss what Transfer Learning is
- Discuss what Fine-tuning is
- How data and amount of data samples is co-related with the above
- Transfer Learning (in depth) with different tasks
- Classification (@shreydan, forked-PR#3, #67)
- Pick a dataset (pick a real world dataset)
- Take a model
- do transfer learning
- Object detection
- Explain Object detection in a very high level (for transformers)
- Pick a dataset
- Take an existing model
- do Transfer learning for that
- Segmentation (assigned to @hanouticelina)
- Explain Object detection in a very high level (for transformers)
- Pick a dataset
- Take an existing model
- do Transfer learning for that
- Classification (@shreydan, forked-PR#3, #67)
- Fine Tuning is depth with different tasks
- Classification (Assigning @Anindyadeep, use the same dataset)
- Pick a dataset
- Take a model
- do transfer learning
- Object detection
- Explain Object detection in a very high level (for transformers)
- Pick a dataset
- Take an existing model
- do Transfer learning for that
- Segmentation
- Explain Object detection in a very high level (for transformers)
- Pick a dataset
- Take an existing model
- do Transfer learning for that
- Classification (Assigning @Anindyadeep, use the same dataset)
- Fine-tuning with LoRA
- Explain what LoRA is in semi-depth
- Pick Dataset for classification, object detection and Segmentation
- FIne-tune using LoRA with all the three datasets
- Compare results with other methods
- Knowledge Distillation
- TBD
- Compare results of Transfer Learning, Fine-tuning, LoRA and Knowledge Distillation
- Recap and Summary
- References
Contribution Guidelines
We are going to work on Shreyas's repository. And here is how we are going to work.
- We will have a main branch
- Make a branch for each section or subsection of things
.
└── course/
└── chapters/
├── chapter3/
│ ├── Transfer Learning and Fine-tuning.mdx
│ └── TransferLearningInDepth.ipynb
├── chapter-n/
│ └── ...mdx
└── _toctree.yml
- This is how we should work/collab
- Create issues for each subsections
- Create a PR for that
- Assign others to review that
- Once done let the cv people review that and merge or whatever needs to be done.
Please assign yourself. So, by the end of next week, I would appreciate some response. So, that we can provide this updated chapter contents to the admin teams, and they get informed on the specifics.
CC: @shreydan @sezan92 @alanahmet
CC: @merveenoyan @johko
from computer-vision-course.
Hi there, I am "contributor 4" on Chapter 3 items
- Common Vision Transformers (Swin, etc.) and
- Transfer Learning / Fine Tuning.
I would like to contribute by
- taking one task of the set (classification, object detection, segmentation) and train a vision transformer on it,
- do almost the same again but with a bigger model and training it using PEFT LoRA (plus showing how to save the adapter locally and how to load it again), and
- add a demo on prompt tuning.
Is everybody OK with that?
@shreydan @sezan92 @alanahmet
CC: @merveenoyan @johko
from computer-vision-course.
@matthiasdroth hello can you please reach out to us on discord. All of the discussions are going on there regarding contributions, mention us in the official discord channel: #cv-community-project. Thank you.
from computer-vision-course.
What are your handles on Discord? It seems only @sezan92 is the same as on GitHub.
from computer-vision-course.
What are your handles on Discord? It seems only @sezan92 is the same as on GitHub.
Uhm, it would be better, if we take this discussion on public discord cv-community course channel.
from computer-vision-course.
Related Issues (20)
- How to include "What you'll learn" section for this course? HOT 6
- Notebook in wrong subdirectory
- Numbering for chapters and notebooks discrepancy HOT 3
- Latex and <Tip> blocks seems to not render correctly HOT 2
- CV task should be located before the CNN
- "Unit 12 - Ethics and Biases in Audio and Computer Vision" HOT 1
- Typos in Unit 1 - Vision HOT 2
- Unfinished File point_cloud.mdx HOT 3
- image not loaded in chapter 2 introduction section.
- Image Classification task is missing in Unit 6 HOT 5
- Missing video link in CNN Unit HOT 2
- z-axis rotation matrix not right in Basics of Linear Algebra for 3D Data HOT 1
- add notebook examples
- Definition of image HOT 3
- Multi-label Image Classification Colab Notebook Error HOT 2
- Unit 4 Transfer Learning Application Notebook Links HOT 5
- Unit 2 - CNN | Transfer Learning/Fine-Tuning Draft Outline HOT 7
- Discord Link is broken in readme HOT 2
- Image Reconstruction/Enhancement with Real-time Image Processing HOT 2
- Issue with rendering the course HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from computer-vision-course.