For example, from model deion on HuggingFaceH4/starchat-beta, the size of this i

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

Name and size from same model can cause different result about gpu_poor HOT 2 CLOSED

rahulschand commented on July 28, 2024

Name and size from same model can cause different result

from gpu_poor.

Comments (2)

RahulSChand commented on July 28, 2024

Hi @LaniakeaS this happens because to calculate memory requirement, one needs hidden_size, intermediate_size, num_layers, vocab_size of the model (these are usually present in config.json & the app reads it from there)

But when you just input the size, the app has no idea about above values & has to "infer" them based on some assumptions. This part of the code, approximates the values of hidden_size etc. using just the final size (in billion) of the model

gpu_poor/src/App.js

Lines 671 to 705 in a702c37

    
           function getApprox(modelSize) { 
        
               let vocabR = null, 
        
                   headsR = null, 
        
                   numLayersR = null; 
        
               if (modelSize < 5) { 
        
                   vocabR = 32000; 
        
                   headsR = 32; 
        
                   numLayersR = 24; 
        
                   return [vocabR, headsR, numLayersR]; 
        
               } 
        
               if (modelSize < 10) { 
        
                   vocabR = 32000; 
        
                   headsR = 32; 
        
                   numLayersR = 32; 
        
                   return [vocabR, headsR, numLayersR]; 
        
               } 
        
               if (modelSize < 24) { 
        
                   vocabR = 32000; 
        
                   headsR = 40; 
        
                   numLayersR = 40; 
        
                   return [vocabR, headsR, numLayersR]; 
        
               } 
        
               if (modelSize < 55) { 
        
                   vocabR = 32000; 
        
                   headsR = 64; 
        
                   numLayersR = 48; 
        
                   return [vocabR, headsR, numLayersR]; 
        
               } 
        
               vocabR = 32000; 
        
               headsR = 64; 
        
               numLayersR = 80; 
        
               return [vocabR, headsR, numLayersR]; 
        
           }

This causes a difference, because given just the final size of the model it's impossible to say what EXACT hidden_size etc. they had

from gpu_poor.

LaniakeaS commented on July 28, 2024

thank you for your answer!

from gpu_poor.

Recommend Projects

Name and size from same model can cause different result about gpu_poor HOT 2 CLOSED

Comments (2)

Related Issues (9)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	function getApprox(modelSize) {
	let vocabR = null,
	headsR = null,
	numLayersR = null;
	if (modelSize < 5) {
	vocabR = 32000;
	headsR = 32;
	numLayersR = 24;
	return [vocabR, headsR, numLayersR];
	}
	if (modelSize < 10) {
	vocabR = 32000;
	headsR = 32;
	numLayersR = 32;
	return [vocabR, headsR, numLayersR];
	}
	if (modelSize < 24) {
	vocabR = 32000;
	headsR = 40;
	numLayersR = 40;
	return [vocabR, headsR, numLayersR];
	}

	if (modelSize < 55) {
	vocabR = 32000;
	headsR = 64;
	numLayersR = 48;
	return [vocabR, headsR, numLayersR];
	}

	vocabR = 32000;
	headsR = 64;
	numLayersR = 80;
	return [vocabR, headsR, numLayersR];
	}