jamesqfreeman / lora-vit Goto Github PK
View Code? Open in Web Editor NEWLow rank adaptation for Vision Transformer
License: GNU General Public License v3.0
Low rank adaptation for Vision Transformer
License: GNU General Public License v3.0
Thank you for opensource code!
I have a question about lora_vit.head. It seems when save lora fc weight(
Line 100 in 45be249
Line 497 in 45be249
bias=True
in melo.lora_vit.head (examples.ipynb)Excuse me, I know lora a little, I use your method and I find it powerful in training. In my method, I save model as ".pth" after training and I use ".pth" to evaluating model but I find it worse than full params finetuning. So should I also "save_lora_parameters" after training and evaluate it by loading "save_lora_parameters" and ".pth"? Besides, does the parameters of "r" and "alpha" have an effect?
Hi, I am able to successfully do forward passes through my ViT with no issues, but trying to train LoRA instantly crashes with
CUDA out of memory. Tried to allocate 1.50 GiB (GPU 0; 15.78 GiB total capacity; 13.19 GiB already allocated; 1.41 GiB free; 13.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Is it expected that training LoRA adapter weights would use significantly more resources than independently making predictions with the model and training 900,000 parameters? Is it because running a model in with torch.no_grad() saves more memory than shutting off requires_grad on each module?
Thank you for any advice
Thank you for your work and making it available!
I am not into the technical details of LoRA, but I know it accepts the rank and alpha as inputs. I see from the example you are providing that you specify r as input. but I don't see alpha is passed anywhere. so my question is what its current value? is it the same as rank? and how can I customize it for different values?
Hello @JamesQFreeman,
Thanks for your great code for easy re-implementation of LoRA in ViT. It's very useful for us to adapt to our own task.
However, I have observed that only the query and value weights in the attention block are being tuned. I'm curious about the rationale behind this. In the original LoRA paper, the authors recommend tuning all four weights in the attention block (query, key, value, and projection linear layer) for better performance.
Is there a trade-off between computational efficiency and final performance that led to this decision? What's more, have you seen an improvement in results by including the key and projection linear layer weights for tuning in your own task?
Thank you, and I look forward to hearing from you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.