Hello, Thank you for your very interesting work! I was planning on experimenting w

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Sparsity constraint in channel exchanging,about yikaiw/cen

yikaiw commented on August 18, 2024 1

Hi, for semantic segmentation, the sparsity constraint is implemented at:
(1) different (disjoint) sub-parts,

Lines 396 to 400 in 158e313

    
           if param.requires_grad and name.endswith('weight') and 'bn2' in name: 
        
               if len(slim_params) % 2 == 0: 
        
                   slim_params.append(param[:len(param) // 2]) 
        
               else: 
        
                   slim_params.append(param[len(param) // 2:])

(2) adding the loss of the sparsity constraint,

CEN/semantic_segmentation/main.py

Lines 282 to 283 in 158e313

    
           L1_norm = sum([L1_penalty(m).cuda() for m in slim_params]) 
        
           loss += lamda * L1_norm  # this is actually counted for len(outputs) times

For image-to-image translation, the sparsity constraint is implemented at:
(1) different (disjoint) sub-parts,

CEN/image2image_translation/main.py

Lines 112 to 119 in 158e313

    
           slim_params, insnorm_params = [], [] 
        
           for name, param in G.named_parameters(): 
        
               if param.requires_grad and name.endswith('weight') and 'insnorm_conv' in name: 
        
                   insnorm_params.append(param) 
        
                   if len(slim_params) % 2 == 0: 
        
                       slim_params.append(param[:len(param) // 2]) 
        
                   else: 
        
                       slim_params.append(param[len(param) // 2:])

(2) adding the loss of the sparsity constraint,

CEN/image2image_translation/main.py

Lines 208 to 213 in 158e313

    
           # L1 loss 
        
           l1_loss = params.gama * sum([L1_loss(gen_image, y_) for gen_image in gen_images]) 
        
           if params.lamda > 0: 
        
               slim_loss = params.lamda * sum([slim_penalty(m).cuda() for m in slim_params]) 
        
           else: 
        
               slim_loss = 0

from cen.

hljeong commented on August 18, 2024

Thank you for your reply! However I am still confused as to where the sparsity constraint in terms of channel exchanging is implemented, as the sections of code you referenced seem to be applying the sparsity constraint to the loss calculation.

I am mainly confused about

CEN/semantic_segmentation/models/modules.py

Lines 12 to 15 in 158e313

    
           x1[:, bn1 >= bn_threshold] = x[0][:, bn1 >= bn_threshold] 
        
           x1[:, bn1 < bn_threshold] = x[1][:, bn1 < bn_threshold] 
        
           x2[:, bn2 >= bn_threshold] = x[1][:, bn2 >= bn_threshold] 
        
           x2[:, bn2 < bn_threshold] = x[0][:, bn2 < bn_threshold]

which seems to exchange channels within all of x[0] and x[1], instead of disjoint sub-parts of them.

from cen.

yikaiw commented on August 18, 2024

Hi, take semantic segmentation as an example:
We apply the sparsity constraints on disjoint sub-parts of BN scaling factors in,

CEN/semantic_segmentation/main.py

Lines 396 to 400 in 158e313

    
           if param.requires_grad and name.endswith('weight') and 'bn2' in name: 
        
               if len(slim_params) % 2 == 0: 
        
                   slim_params.append(param[:len(param) // 2]) 
        
               else: 
        
                   slim_params.append(param[len(param) // 2:])

In the case of two modalities, we divide channels into two disjoint sub-parts, which is implemented by adding param[:len(param) // 2 and param[len(param) // 2:] to slim_params. Followed up by the sparsity loss on slim_params, which means only the sub-parts in slim_params are constrained by L1.

We find if a channel is out of the sparsity constraints (L1), its BN scaling factor can be hardly lower than the small threshold during training. Therefore we check the criteria for channel exchanging directly on the whole channels,

CEN/semantic_segmentation/models/modules.py

Lines 12 to 15 in 158e313

    
           x1[:, bn1 >= bn_threshold] = x[0][:, bn1 >= bn_threshold] 
        
           x1[:, bn1 < bn_threshold] = x[1][:, bn1 < bn_threshold] 
        
           x2[:, bn2 >= bn_threshold] = x[1][:, bn2 >= bn_threshold] 
        
           x2[:, bn2 < bn_threshold] = x[0][:, bn2 < bn_threshold]

Since constraining half (disjoint sub-parts) of the channels is already implemented in main.py, checking the exchanging criteria on the whole channels is almost equivalent to disjoint sub-parts.

from cen.

hljeong commented on August 18, 2024

That makes sense. Thank you for your detailed explanation!

from cen.

rginjapan commented on August 18, 2024

@yikaiw 你好（1）我不是特别理解用L1norm来惩罚 scale factor 在loss function 的意义，这一项在loss function里不就是让 scale factor 越来越小么简单的来说。能不能稍微解释一下呢，谢谢🙏
（2）这里的certain portion 就是disjoint 的那部分的意思是么？

from cen.

Sparsity constraint in channel exchanging about cen HOT 5 CLOSED

Comments (5)

Related Issues (17)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	if param.requires_grad and name.endswith('weight') and 'bn2' in name:
	if len(slim_params) % 2 == 0:
	slim_params.append(param[:len(param) // 2])
	else:
	slim_params.append(param[len(param) // 2:])

	L1_norm = sum([L1_penalty(m).cuda() for m in slim_params])
	loss += lamda * L1_norm # this is actually counted for len(outputs) times

	slim_params, insnorm_params = [], []
	for name, param in G.named_parameters():
	if param.requires_grad and name.endswith('weight') and 'insnorm_conv' in name:
	insnorm_params.append(param)
	if len(slim_params) % 2 == 0:
	slim_params.append(param[:len(param) // 2])
	else:
	slim_params.append(param[len(param) // 2:])

	# L1 loss
	l1_loss = params.gama * sum([L1_loss(gen_image, y_) for gen_image in gen_images])
	if params.lamda > 0:
	slim_loss = params.lamda * sum([slim_penalty(m).cuda() for m in slim_params])
	else:
	slim_loss = 0

	x1[:, bn1 >= bn_threshold] = x[0][:, bn1 >= bn_threshold]
	x1[:, bn1 < bn_threshold] = x[1][:, bn1 < bn_threshold]
	x2[:, bn2 >= bn_threshold] = x[1][:, bn2 >= bn_threshold]
	x2[:, bn2 < bn_threshold] = x[0][:, bn2 < bn_threshold]