jankrepl / mildlyoverfitted Goto Github PK

View Code? Open in Web Editor NEW

337.0 337.0 125.0 807 KB

Paper implementations from scratch and machine learning tutorials

License: MIT License

Python 92.46% Jupyter Notebook 5.31% Shell 1.92% Dockerfile 0.15% Jinja 0.16%

deep-learning pytorch tutorial youtube

mildlyoverfitted's People

Contributors

Stargazers

Watchers

Forkers

ollihansen90 cv324lib kapitsa2811 zhangherman cherrychou98 lixiaopeng20210428 dongzhenya vishwajitt123 tejshahresearch itzamirali windsmilevalley razirezaei spencerraw wzj207 baranwa2 sundawei dome272 nbaghel777 sergiosanchoasensio ao1neko ruthvik92 metavai tanyu1102 yk135915 emmanuelngenzirabona amine179 garymihalik1 dong-0726 navinelahi nerone19 sleepyeye anatolicvs hungnphan historyw conan9191 anh-ta-2020 f2010126 kasmello cv-ip abid-110 kocchop pedromoya car-coder-907 lbfares amdarwish haoruilee cswu-challenge goinging98 marvnmtz 111111m luckywood sno3mahn cuong3004 prakharraj1302 finesssss mattdias96 radeenxalnw dolmalin zeke19066 ronensc deep-jd eddieay ulsoe05 wany1d1001 tejavaththarun jonychoi andreaderetti liminm glclea001 ggg1024 vasanthengineer4949 mohd-sahil chunde abdulladesmal daksh-s97 sindroc ntaylorox huuhieu0903 creative-python bafatah albertmundu wanburana khalilcse kishorekannan123 richa10 glemhel mactracyhuang enverrrr sixaxis9 ahmedsoror itzkumaran jasoncchsiao distoramos tuanshu bodiman kuta-ndze dalekseevspb vincehass murasso levi-ackman

mildlyoverfitted's Issues

Unable to verify Vision Transformer result using loaded weight from timm

I tried to run this file https://github.com/jankrepl/mildlyoverfitted/blob/master/github_adventures/vision_transformer/verify.py to reproduce the result, the last assertion is not passed. The result from the official model and the custom model is not the same, eventhough the weights are exactly the same, im not sure why. Also, it used to work flawlessly, I suspect it is something to do with torch 2.0 (or maybe timm), but im not sure.

here is the quick runnable notebook to verify the issue
https://colab.research.google.com/drive/1sRyBTvqAMkTgmSWlBFpv79mj6Ir4aLOT?usp=sharing

Mismatch of number of parameters

Hey, the number of parameters in your custom model are not matching with nay of the model here?

https://pytorch.org/vision/stable/models/vision_transformer.html

Issue in verify.py file execution

Thank you for the codes and video explanation. I got the following issue while running the verify.py file. I could run custom.py without any error and I have installed python 3.9, PyTorch version 1.8.
I did all the verification related to the availability of the vit_base_patch16_384 patch in the vision_transformer.py file. This is the error I got.
Could you please help me to solve this issue?

Why is the attn_mask a upper triangle matrix?

mildlyoverfitted/github_adventures/gpt/model.py

Lines 73 to 78 in 94c7983

    
           self.register_buffer( 
        
               "mask", 
        
               (1 - torch.tril(torch.ones(n_positions, n_positions))).to( 
        
                   dtype=torch.bool 
        
               ), 
        
           )

Make sure each project has a README

Containing minimal instructions on how to launch relevant scripts and reproduce the results. The videos should contain everything, however, why not to make it even easier:)

TypeError: 'Parameter' object is not callable

Hi,

Thanks for the video on MLP mixer using pytorch and flax. I am facing an error while running the code. I am sharing the code that I have written and the error I am getting:

class conv1dDepthWiseShared(nn.Module):
  def __init__(self, hidden_dim, kernel_size, k):
    super().__init__()
    #same as input channels of our tensor
    self.hidden_dim = hidden_dim
    # K is any number representing the output features
    self.weight_shared = nn.Parameter(torch.rand(k, 1, kernel_size,))
    self.bias_shared = nn.Parameter(torch.rand(k))

  def forward(self, x):
    weight = self.weight_shared(self.hidden_dim, 1, 1)
    bias = self.bias_shared.repeat(self.hidden_dim)
    res = torch.nn.functional.conv1d(x, weight=weight, bias=bias, groups=self.hidden_dim)
    return res

n_samples, hidden_dim, n_patches = 2, 16, 25
k = 7
x = torch.rand(n_samples, hidden_dim, n_patches)
module_conv = conv1dDepthWiseShared(hidden_dim, n_patches, k)
module_linear = nn.Linear(n_patches, k)
module_conv.weight_shared.data[:, 0, :] = module_linear.weight.data
module_conv.bias_shared.data[:] = module_linear.bias.data
out_conv = module_conv(x).reshape(n_samples, hidden_dim, k)

TypeError                                 Traceback (most recent call last)

<ipython-input-50-310a54adf4b2> in <module>()
----> 1 out_conv = module_conv(x).reshape(n_samples, hidden_dim, k)

1 frames

<ipython-input-40-a54ec45a7e5a> in forward(self, x)
      9 
     10   def forward(self, x):
---> 11     weight = self.weight_shared(self.hidden_dim, 1, 1)
     12     bias = self.bias_shared.repeat(self.hidden_dim)
     13     res = torch.nn.functional.conv1d(x, weight=weight, bias=bias, groups=self.hidden_dim)

TypeError: 'Parameter' object is not callable

Add requirements.txt to each project

Make life easier for those who start from fresh virtual environments.

In the meantime, git grep import should not be too overwhelming.

Cannot Load the pre-trained Weights

Hi, thanks for the nice codes and videos.
If we create an instance of your custom ViT and then try to load the pre-trained weights it's not working and gives errors.

Eval video in "Growing Neural Cellular Automata" code

I find The code for the "Growing Neural Cellular Automata" video very nice and compact.

However, one small problem: the eval video, that is theoretically added to Tensorboard with the writer function, does not seem to appear in Tensorboard after the training is complete (the way it does in the video).

For my personal use, I managed to retrieve it from Tensorboard with the EventAccumulator (so it is indeed produced in the back somewhere), but it would be nice for it to appear directly in the board.
Maybe it's because of something wrong from my end...

How do you load weights for your implementation of ViT?

Hi Jan, thanks for the great work.

I found the answer so I am closing this. However, in case anyone has the same question and read this, it is from the following line:
p_c.data[:] = p_o.data

Missing eval property for the eval video in "Growing Neural Cellular Automata" code

Hi there, I just found a similar issue as the #11

As you can see in the screen shot, based on the training on your code, i cannot control the step with the slider. It seems like there is no attribute "eval" has been written down.

I take the same traning steps and paramater setting as your youtube video. And I use the command line to invoke the tensorboard as below:

tensorboard --logdir=

I wonder if there is anything I missed. Thank you for your attention!

Training guide

Please provide traing guide too. Any other sources about this are also welcome!

Information about DINO in PyTorch

Good morning Mr Jan Krepl,

Cordial greeting,

After running DINO in PyTorch (https://www.youtube.com/ watch? V = psmMEWKk4Uk & list = PLDW9q3PQf5VI0X5ozC3dDgbld8S4lrBuL & index = 15) in google colab because I don't have a GPU, and I'm decreasing the amount of images for training and try, but I get the following error:

The first error found was in the file "train.py" I had to modify

Parameters: vit_name, dim = "vit_deit_small_patch16_224", 384 by # Parameters: vit_name, dim = "vit_small_patch16_224", 384, it wouldn't let me run the file.

Then I ran "python train.py" and I got this error: AttributeError: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'. I was reading in this link a possible solution (https://stackoverflow.com/questions/60730544/tensorboard-colab-tensorflow-api-v1-io-gfile-has-no-attribute-get-filesystem), where I removed tensorflow and apparently, it was working for me but I got this message again:

{'batch_size': 32, 'device': 'cpu', 'logging_freq': 1000, 'momentum_teacher': 0.9995, 'n_crops': 4,' n_epochs': 200, 'out_dim': 1024, 'tensorboard_dir': ' logs', 'clip_grad': 2.0, 'norm_last_layer': True, 'batch_size_eval': 64, 'teacher_temp': 0.04, 'student_temp': 0.1, 'pretrained': False, 'weight_decay': 0.4}
/usr/local/lib/python3.7/dist-packages/torchvision/transforms/transforms.py:853: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int."
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness / freeze if necessary.
cpuset_checked))
0% 0/8 [00:00 <?,? It / s] ^ C

I would appreciate a lot if you could help me, I have not been able to run the model. Thanks a lot.

Interesting error for OpenAI function calling example

I got quite intrigued after watching your YouTube video on OpenAI function calling and got your example.py code to play with. I wanted to see what it would do with an interesting ticker like HMNY. So I asked the following question and it barfed on start and end date. I thought I had unintentionally modified the code when I was perusing but a git diff and a git st both showed nothing had been modified:

# python example.py "If I bought 100 shares of HMNY stock on 2004-01-01, how much did I pay for them and how much are they worth today?"                                                    
                                                                                                                                                                                            
2024-05-11 02:52:22,516 Calling get_price with symbol='HMNY' and date='2004-01-01'                                                                                                          
2024-05-11 02:52:25,216 Calling get_price with symbol='HMNY' and date='2024-05-11'                                                                                                          
2024-05-11 02:52:29,333                                                                                                                                                                     
1 Failed download:                                                                                                                                                                          
2024-05-11 02:52:29,333 ['HMNY']: Exception('%ticker%: Invalid input - start date cannot be after end date. startDate = 1715400000, endDate = 1715395945')                                  
Traceback (most recent call last):                                                                                                                                                          
  File "/root/source/mildlyoverfitted/mini_tutorials/openai_function_calling/example.py", line 105, in <module>
    output = str(get_price(**kwargs))                                                         
  File "/root/source/mildlyoverfitted/mini_tutorials/openai_function_calling/example.py", line 24, in get_price
    return history["Close"].iloc[0].item()                                                    
  File "/opt/openai-functions/lib/python3.10/site-packages/pandas/core/indexing.py", line 1191, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)                                                                                                                                    
  File "/opt/openai-functions/lib/python3.10/site-packages/pandas/core/indexing.py", line 1752, in _getitem_axis
    self._validate_integer(key, axis)                                                         
  File "/opt/openai-functions/lib/python3.10/site-packages/pandas/core/indexing.py", line 1685, in _validate_integer
    raise IndexError("single positional indexer is out-of-bounds")                                                                                                                          
IndexError: single positional indexer is out-of-bounds

So, I asked a different question and this time it didn't crash:

# python example.py "If I bought 100 shares of HMNY stock on 2004-01-01, how much did I pay for them and how much were they worth on 2018-07-31?"
2024-05-11 04:13:56,098 Calling get_price with symbol='HMNY' and date='2004-01-01'
2024-05-11 04:14:03,959 Calling get_price with symbol='HMNY' and date='2018-07-31'
2024-05-11 04:14:05,468 Calling calculate with a=2275, b=100 and op='mul'
2024-05-11 04:14:06,388 Calling calculate with a=0.4959999918937683, b=100 and op='mul'
********************************************************************************
['user', 'system', 'assistant', 'function', 'assistant', 'function', 'assistant', 'function', 'assistant', 'function', 'assistant']
********************************************************************************
You paid $2,275 for 100 shares of HMNY stock on January 1, 2004. 

On July 31, 2018, each share of HMNY stock was worth $0.496. 

Therefore, your 100 shares of HMNY stock were worth $49.60 on July 31, 2018.

This rather puzzled me so later on, I asked the same question that failed before but this time no crash:

# python example.py "If I bought 100 shares of HMNY stock on 2004-01-01, how much did I pay for them and how much are they worth today?"
2024-05-11 04:14:59,328 Calling get_price with symbol='HMNY' and date='2004-01-01'
2024-05-11 04:15:01,679 Calling get_price with symbol='HMNY' and date='2024-05-11'
2024-05-11 04:15:03,448 Calling calculate with a=2275, b=100 and op='mul'
2024-05-11 04:15:04,363 Calling calculate with a=0.00019999999494757503, b=100 and op='mul'
********************************************************************************
['user', 'system', 'assistant', 'function', 'assistant', 'function', 'assistant', 'function', 'assistant', 'function', 'assistant']
********************************************************************************
If you bought 100 shares of HMNY stock on 2004-01-01, you would have paid $2,275 for them. 

As of today (2024-05-11), each share of HMNY stock is worth $0.0002. Therefore, your 100 shares would be worth $0.02.

It looks like OpenAI can be unstable at times...

	self.register_buffer(
	"mask",
	(1 - torch.tril(torch.ones(n_positions, n_positions))).to(
	dtype=torch.bool
	),
	)