rigvedrs / yolo-v8-cam Goto Github PK

"Hello, I am trying to explain my model, which is built on yolov8 based on your model, through CAM. The issue is that while the detection of classes on the image is accurate, the heatmap is generated on parts of the image that are completely unrelated to the class. How can I resolve this issue?"

img = cv2.imread(f'/home/fatemeh/Documents/training datasets/adsf/train/images/{img_name}')
img = cv2.resize(img, (640, 640))

rgb_img = img.copy()
img = np.float32(img) / 255
target_layers =[model.model.model[-4]]
cam = EigenCAM(model, target_layers,task='seg')
grayscale_cam = cam(rgb_img)[0, :, :]
cam_image = show_cam_on_image(img, grayscale_cam, use_rgb=True)
plt.imshow(cam_image)
plt.show()
im = cv2.cvtColor(rgb_img, cv2.COLOR_RGB2BGR)
Image.fromarray(np.hstack((im, cam_image)))

File "/home/fatemeh/develop/YOLO_/YOLO-V8-CAM/yolo_cam/base_cam.py", line 87, in forward
targets = [ClassifierOutputTarget(category) for category in target_categories]
UnboundLocalError: local variable 'target_categories' referenced before assignment

Hello :)

Thank you for this awesome tool!

I've ran a "m" model which I have trained on a custom dataset, and I've waited over 3 minutes each time I ran it, without receiving any output. I assume this is some weird behavior that isn't supposed to happen. I do have gpu, and I did try to send my model to gpu but it didn't help somehow. Any tips would be appreciated.

Best,
Ermin

I am getting similar error as #8 , I dont know what is going wrong. I have rerun it multiple times. Also, I dont understand why we have to run it multiple times. For example in below it highlights everything but players and ball. I have tried using different colormaps in cv2 but the result is same.

when I run this line: grayscale_cam = cam(rgb_img)
it appears this error "Kernel Restarting The kernel for pytorch_yolov8/code/YOLO-V8-CAM-main/YOLO v8n EigenCAM.ipynb appears to have died. It will restart automatically. " Could someone help me to solve this problem?? thx!

I'm working on a project where I trained a YOLOv8 model to find E.coli bacteria droplets on leaves. After using Grad-CAM on the YOLO model, which was 100% accurate, I noticed something unexpected. The areas without droplets are marked in red, and the areas with droplets are in blue, which means they're considered less important.

Does this mean the model is actually working well and just the color scheme is flipped, or did I make a mistake somewhere?

Here are the Grad-CAM results and the original image displayed side by side for comparison:

I tried to implement your exact GRAD-CAM on the "puppies" image with your attached "yolov8n.pt".

But some how I seem to get different heatmap. Here is my implementation script

`if name == 'main':
from ultralytics import YOLO
from yolo_cam.eigen_cam import EigenCAM
from yolo_cam.utils.image import show_cam_on_image, scale_cam_image
import warnings
warnings.filterwarnings('ignore')
warnings.simplefilter('ignore')
import torch
import cv2
import numpy as np
import matplotlib.pyplot as plt
import requests
import torchvision.transforms as transforms
from PIL import Image
import io

test = r"D:\GRAD-CAM\images\puppies.jpg"
img = cv2.imread(test)
#img = cv2.resize(img, (640, 640))
rgb_img = img.copy()
img = np.float32(img) / 255

model = YOLO(r"D:\GRAD-CAM\models\yolov8n.pt")  # pretrained YOLOv8n model
#results = model(source = test ,device = 0, line_width = 3, save = True)  # return a list of Results objects
target_layers =[model.model.model[-2]]


cam = EigenCAM(model, target_layers,task='od')
grayscale_cam = cam(rgb_img)[0, :, :]
cam_image = show_cam_on_image(img, grayscale_cam, use_rgb=True)
plt.imshow(cam_image)
plt.show()`

this code generates this image

When I uncomment
results = model(source = test ,device = 0, line_width = 3, save = True) # return a list of Results objects

the generated image is

in both cases "yolov8n.pt" detects the image as follows

so my question: what went wrong here?

Hello, thank you for the great work and implementation!

Edit: I am using a classification model.

I am not really experienced myself, but I noticed that there was a lot of unclear information about which layer to use for EigenCAM in Yolov8. As I understood the paper on EigenCAM, the method is meant to be applied on the very last convolutional layer in the network. This would not be -2, or -4, so many pointed out, but model.model.model[-1].conv I think.

Please correct me if I am mistaken. I might very clearly be mistaken, as Yolo as a model-architecture is not straight forward.

Story: I was really frustrated about finding the right layer, as -2, -3,-4 all performed different in each image, and also for each model-size, so I looked into the yolo architecture and found out, that yolo actually has a conv-network in the classify-layer.

Thank you all and I am very interested in some different views on this.

If I initialise the model multiple times and run the cam function, it seems to be giving an inaccurate result in the output. Could be some issue with the activations of the previous runs not being deleted properly. Will have to look into it.

I am trying to work with your YOLOv8 CAM; however, I am getting an AssertionError in BaseTensor class in the results.py line 32.

Initially, the problem originates from this line:
grayscale_cam = cam(rgb_img)[0, :, :]

It does predict the class correctly:
224x224 golden_retriever 0.41, Labrador_retriever 0.30, Great_Pyrenees 0.09, kuvasz 0.04, cocker_spaniel 0.03, 2.4ms Speed: 1.0ms preprocess, 2.4ms inference, 0.0ms postprocess per image at shape (1, 3, 224, 224)

that comes from here:
ultralytics\yolo\engine\results.py", line 32, in __init__ assert isinstance(data, (torch.Tensor, np.ndarray))

It seems that the model does predict the classes correctly and the data in this class is a tensor/numpy array the first 3 times, but then it becomes a single digit, so it no longer is a tensor or a numpy array. This is outside of your code, but I was wondering if you have encountered anything like this before.

I also tried to convert the image to tensor as it was introduced here:

transform = transforms.ToTensor()
tensor = transform(img).unsqueeze(0)
grayscale_cam = cam(tensor)[0, :, :]

but no success.

Thank you very much for your work, but I am now in great need of gradcam support for yolov8 instance segmention, when will this be possible please!

Hello,

I'm conducting research on Explainable AI using YOLO v8 with a custom dataset and encountered some uncertainties in the documentation. My queries are:

1. What should be specified in the "model" parameter?
2. What is the appropriate input for the "target_layers" attribute?

A detailed example or a notebook demonstrating end-to-end implementation on a sample dataset would be greatly beneficial, especially for beginners like me😬. This could really jazz up the repo!

Best,
Snehit Vaddi

yolo_cam/base_cam.py
def get_target_width_height

code:
width, height = np.shape(input_tensor)[0],np.shape(input_tensor)[1]
problem: np.shape returns the height first, then the width

fix:
height, width = np.shape(input_tensor)[0],np.shape(input_tensor)[1]