rvorias / ind_knn_ad Goto Github PK

Hi,
this repo is great and very helpful !!

I have one question to ask.

I could run the program fine without installing "faiss".
So I am just wondering what is the purpose of using "faiss" in the program.
will the inference speed increase if using "faiss"?

Thanks

The size of my picture is 1280*1024,I use the command streamlit run streamlit_app.py . The result is very good. But part of my picture is missing. The displayed result is not a complete picture. Can the cropping of the picture be changed? I tried to modify the code, but the result was not good.Can the detection speed be improved?Can I just load the model without training every time?

class SPADE(KNNExtractor):
    def __init__(
            self,
            k: int = 5,
            backbone_name: str = "resnet50",
    ):
        super().__init__(
            backbone_name=backbone_name,
            out_indices=(1, 2, 3),
            pool=True,
        )
        self.k = k
        self.image_size_x = 1280
        self.image_size_y = 1024
        self.z_lib = []
        self.feature_maps = []
        self.threshold_z = None
        self.threshold_fmaps = None
        self.blur = GaussianBlur(4)

    def predict(self, sample):
        feature_maps, z = self(sample)

        distances = torch.linalg.norm(self.z_lib - z, dim=1)
        values, indices = torch.topk(distances.squeeze(), self.k, largest=False)

        z_score = values.mean()

        # Build the feature gallery out of the k nearest neighbours.
        # The authors migh have concatenated all features maps first, then check the minimum norm per pixel.
        # Here, we check for the minimum norm first, then concatenate (sum) in the final layer.
        scaled_s_map = torch.zeros(1, 1, self.image_size_y, self.image_size_x)
        for idx, fmap in enumerate(feature_maps):
            nearest_fmaps = torch.index_select(self.feature_maps[idx], 0, indices)
            # min() because kappa=1 in the paper
            s_map, _ = torch.min(torch.linalg.norm(nearest_fmaps - fmap, dim=1), 0, keepdims=True)
            scaled_s_map += torch.nn.functional.interpolate(
                s_map.unsqueeze(0), size=(self.image_size_y, self.image_size_x), mode='bilinear'
            )

        scaled_s_map = self.blur(scaled_s_map)

        return z_score, scaled_s_map

Hey there 👋 Just wanted to let you know that your app on Streamlit Cloud deployed from this repo has gone over its resource limits. Access to the app is temporarily limited. Visit the app to see more details and possible solutions.

Hi, great project! I have 8000 images, and I found that the memory increased a lot during training. My computer has 60G RAM but it is still not enough.

model = SPADE(k=42)  # , backbone_name="hypernet")
train_dataset = StreamingDataset()
app_custom_train_images = "D:\\train\\good"
# train images
for root, dirs, files in os.walk(app_custom_train_images):
    for file in files:
        train_dataset.add_pil_image(Image.open(os.path.join(root, file)))
model.fit(train_dataset)
PATH = "test.pth"
torch.save(model.state_dict(), PATH)

pil_img = plt.open("20210115_raw.png")
img_pil_1 = np.array(pil_img) 
tensor_pil = torch.from_numpy(np.transpose(img_pil_1, (2, 0, 1)))
img = tensor_pil.type(torch.float32)  
img = img.unsqueeze(0) 
pil_to_tensor = img 
img_lvl_anom_score, pxl_lvl_anom_score = model.predict(pil_to_tensor)

The test picture does not seem to need to be transformed. Does it support pictures in other formats?

(env1) D:\Code\ind_knn_ad-master\indad>python run.py padim
DATASETS_PATH in 'D:\dataset\mvtec_anomaly_detection/'
Running padim on bottle dataset.
Training ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 209/209 [00:33<00:00, 6.21it/s]
Testing ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 83/83 [00:15<00:00, 5.20it/s]
Test results bottle - image_rocauc: 1.00, pixel_rocauc: 0.96
Running padim on cable dataset.
Training ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 224/224 [00:28<00:00, 7.85it/s]
Testing ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 150/150 [00:31<00:00, 4.72it/s]
Test results cable - image_rocauc: 0.91, pixel_rocauc: 0.96
Running padim on capsule dataset.
Training ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 219/219 [00:29<00:00, 7.52it/s]
Testing ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 132/132 [00:26<00:00, 4.91it/s]
Test results capsule - image_rocauc: 0.86, pixel_rocauc: 0.98
Running padim on carpet dataset.
Training ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 280/280 [00:41<00:00, 6.71it/s]
Traceback (most recent call last):
File "D:\Code\ind_knn_ad-master\indad\run.py", line 82, in
cli_interface()
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 1137, in call
return self.main(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 1062, in main
rv = self.invoke(ctx)
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "D:\Code\ind_knn_ad-master\indad\run.py", line 77, in cli_interface
total_results = run_model(method, dataset)
File "D:\Code\ind_knn_ad-master\indad\run.py", line 44, in run_model
model.fit(train_ds)
File "D:\Code\ind_knn_ad-master\indad\model.py", line 171, in fit
self.patch_lib = torch.cat(self.patch_lib, 0)
RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:79] data. DefaultCPUAllocator: not enough memory: you tried to allocate 6294077440 bytes.

Where do you get the PaDiM detection data in AUROC in the README table?

From the official paper, there are no provide detailed categories DET AUROC score but only provides DET AUROC total mean score with WR50_2 backbone (95.3) at most. How did you get an overall 97.5 in PaDiM paper performance in Anomaly Detection in README table?

I assume that you copy the wrong data into the README table from the paper, which you copy Localization AUROC into Image-level table and Localization AUPRO into Pixel level table.

Thanks

Hi there,

I am using the pipeline based on the padim method and backbone_name="wide_resnet50_2". However, I see excessive raise in my memory when I check it with htop command in another terminal. I don't have same experience with spade method.

As picture shows, it consumes up to 121G in the memory. I even tried to reduce the 'd_reduced' parameter, but it is still around 100G. May I ask about your opinion how can I make it better? did U also faced same issue?

Best

Now I want to use patchcore to train,but my dateset don't include the ground_truth image.What should I do to let it work?

Hi there,
I was wondering if inside model.py is there a loss or cost value where it would change during training or fitting?

• Or are we even retrain the backbone to the new datasets? is there even learning of backbone in the pipeline where you specify number of epochs?
  I am particularly talking about PADIM. Thanks a lot for your support.
  Best

-How can i make a prediction for an image?
-How can i train the model with my own data?
Thanks.

Hello and thank for your contribution. Really good job

Trying your code i am wondering if this attribute n_reweight is the neighbourhood size described in the Paper.

It seems that code in cli_interface() would train on a pre-trained model and does not interact to filesystem?

Could you please give an example show how to do that or add more parameters to cli_interface()?

in the indad/model.py line 78

the mask must be convert to int type, otherwise, it will get ValueError: continuous format is not supported

Here is the solution: I changed the mask type from float to int

pixel_labels.extend(mask.flatten().numpy().astype(int))

The initial function is like below

	def evaluate(self, test_ds: VisionDataset) -> Tuple[float, float]:
		"""Calls predict step for each test sample."""
		image_preds = []
		image_labels = []
		pixel_preds = []
		pixel_labels = []

		for sample, mask, label in tqdm(test_ds, **TQDM_PARAMS):
			z_score, fmap = self.predict(sample.unsqueeze(0))
			
			image_preds.append(z_score.numpy())
			image_labels.append(label)
			
			pixel_preds.extend(fmap.flatten().numpy())
			pixel_labels.extend(mask.flatten().numpy())
			
		image_preds = np.stack(image_preds)

		image_rocauc = roc_auc_score(image_labels, image_preds)

		pixel_rocauc = roc_auc_score(pixel_labels, pixel_preds)

		return image_rocauc, pixel_rocauc

Hey there 👋 Just wanted to let you know that your app on Streamlit Cloud deployed from this repo has gone over its resource limits. Access to the app is temporarily limited. Visit the app to see more details and possible solutions.

To load model weights, I create an instance of the same model first, and then load the parameters using load_state_dict() method.

model = PatchCore()
train_dataset = StreamingDataset()
test_dataset = StreamingDataset()
train_paths = "D:\\OK"
test_paths = "D:\\NG"
for root, dirs, paths in os.walk(test_paths):
    for path in paths:
        f = Image.open(os.path.join(root, path))
        test_dataset.add_pil_image(f)
# train images
for root, dirs, files in os.walk(train_paths ):
    for file in files:
        train_dataset.add_pil_image(Image.open(os.path.join(root, file)))
# model.to(device)
# model.fit(DataLoader(train_dataset))
PATH = "test.pth"
# torch.save(model.state_dict(), PATH)
print("One image testing ...")
model.load_state_dict(torch.load(PATH))
model.eval()
for test_idx in range(test_dataset.size):
    sample, *_ = test_dataset[test_idx]
    img_lvl_anom_score, pxl_lvl_anom_score = model.predict(sample.unsqueeze(0))

C:\ProgramData\Anaconda3\envs\knn\python.exe D:/ind_knn_ad-master/test.py
One image testing ...
C:\ProgramData\Anaconda3\envs\knn\lib\site-packages\torch\nn\functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  ..\c10/core/TensorImpl.h:1156.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
  File "D:/ind_knn_ad-master/test.py", line 146, in <module>
    img_lvl_anom_score, pxl_lvl_anom_score = model.predict(sample.unsqueeze(0))
  File "D:\ind_knn_ad-master\indad\models.py", line 278, in predict
    resized_maps = [self.resize(self.average(fmap)) for fmap in feature_maps]
  File "D:\ind_knn_ad-master\indad\models.py", line 278, in <listcomp>
    resized_maps = [self.resize(self.average(fmap)) for fmap in feature_maps]
TypeError: 'NoneType' object is not callable

How can I model save, model load and model test for one image?

PATH = '/home/agteks/Desktop/ind_knn_ad/model.pt'
model = PatchCore(f_coreset= .01, backbone_name= "efficientnet_b0", coreset_eps= .95)

model.load_state_dict(torch.load('/home/agteks/Desktop/ind_knn_ad/model.pt'))
model.eval()
model = model.to('cuda')

#sample, *_ = test_dataset[st.session_state.test_idx]
pil_img = Image.open("/home/agteks/Desktop/ind_knn_ad/datasets/fabric_4/test/broken_large/Cropped_2_276_46_768.png")
pil_img = pil_img.resize((224, 224))
pil_to_tensor = transforms.ToTensor()(pil_img).unsqueeze_(0)
img_lvl_anom_score, pxl_lvl_anom_score = model.predict(pil_to_tensor)
score_range = pxl_lvl_anom_score.min(), pxl_lvl_anom_score.max()
show_pred(pil_to_tensor, img_lvl_anom_score, pxl_lvl_anom_score)

-- This code not working. Please help me :(

Hello.

I tested your Patchcore implementation and i need to advice you of a mistake of what the table of results shows
I tested your implementation for 1% of coreset with different B values to get the mean as final results

You can see, that the performance is better than what you described in your table.

Moreover, if we use avgpooling (3, 1, 1) instead avgpooling(3, 2, 0) with Interpolation Lanzcos in the training, results are still better. In fact, paper says that increasing the stride decreases the performance

Here is the results of using this new avgpooling

Hey the app is currently not responding

by the way do you have a tutorial how can i make the app run on a windows maschine? im new to maschine learning and want to test anomaly detection on pictures of industrial parts.

Thanks for your effort!
I saw the average inference time is very short in the paper.
My time to detect a single picture is 0.9686539s.feature_maps, z = self(sample) It takes a lot of time.I try to select a smaller backbone and reduce self.image_size. It has no obvious improvement.
Any comments??? Thanks!

  def predict(self, sample):
      e1 = cv2.getTickCount()
      feature_maps, z = self(sample)
      e2 = cv2.getTickCount()
      t = (e2 - e1) / cv2.getTickFrequency()
      print(t) # 0.6265114s
      distances = torch.linalg.norm(self.z_lib - z, dim=1)
      values, indices = torch.topk(distances.squeeze(), self.k, largest=False)

      z_score = values.mean()
      e4 = cv2.getTickCount()
      t = (e4 - e2) / cv2.getTickFrequency()
      print(t) # 0.0011656s
      # Build the feature gallery out of the k nearest neighbours.
      # The authors migh have concatenated all features maps first, then check the minimum norm per pixel.
      # Here, we check for the minimum norm first, then concatenate (sum) in the final layer.
      scaled_s_map = torch.zeros(1, 1, self.image_size, self.image_size)
      for idx, fmap in enumerate(feature_maps):
          nearest_fmaps = torch.index_select(self.feature_maps[idx], 0, indices)
          # min() because kappa=1 in the paper
          s_map, _ = torch.min(torch.linalg.norm(nearest_fmaps - fmap, dim=1), 0, keepdims=True)
          scaled_s_map += torch.nn.functional.interpolate(
              s_map.unsqueeze(0), size=(self.image_size, self.image_size), mode='bilinear'
          )
      e5 = cv2.getTickCount()
      t = (e5 - e4) / cv2.getTickFrequency()
      print(t) # 0.2722181s
      scaled_s_map = self.blur(scaled_s_map)
      e6 = cv2.getTickCount()
      t = (e6 - e5) / cv2.getTickFrequency()
      print(t) # 0.0311344s
      return z_score, scaled_s_map

        self.blur_kernel = ImageFilter.GaussianBlur(radius=radius)

Hi

possible to detect the type of defect? with anomaly detection

Thank you!

I tried models work on gpu but torch is not working. Program gives a simle GPU errors.

Like (Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor).

I put my veriables(input and model) to cuda on gpu with ".to(device)" or ".cuda()". It's still not working. But if all veriables and model were staying "CPU" code is working. How can I solve this problem? Feature Extractor model is on "GPU" too. But error is same. Could I code rewiev for my problem or this error can be lib error. But I have used Torch1.13 - cuda11.7.

Thanks to your efforts,
I've noticed in the code of patchcore,
doing k nearest neighbor search,

can the current code run with your results?

Hi there, thanks for sharing your codes publicly.
I am trying to follow the custom dataset workflow. My question is about defining my own threshold based on required prec/recall, as you mentioned in here. I am quite newbie and would like to use your repo for my study thesis. My I ask for quick clue where in the code and how can I set a new threshold?

After training with spade method, I test the model on an anomaly test image and got the img anom score: tensor(8.3064), while I obtained even similar value with a good test image.

btw, as you mentioned I defined a custom dataset folder, but do we need to also define a new dataloader? as the already utilized the one in the run.py is for MVTecDataset.
train_ds, test_ds = MVTecDataset(cls).get_dataloaders()
My dataset folder structure is like:
-custom_dataset
--train
---good
--test
---good
---bad
--ground_truth
---bad

I really appreciate your help and effort.

Hi!
In your custom dataset guideline, you put all good samples into one folder. In my custom dataset, I have three class of objects. I was wondering does it make sense if I train the padim for individual classes and do inference for each class with their seperate models? or it is also ok if I put all classes into one bag of good samples and have one model for all?
I really appreciate your suggestions based on your experience.

Best

Test results zipper - image_rocauc: 0.94, pixel_rocauc: 0.98
Traceback (most recent call last):
File "D:\Code\ind_knn_ad-master\indad\run.py", line 82, in
cli_interface()
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 1137, in call
return self.main(*args, **kwargs)
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 1062, in main
rv = self.invoke(ctx)
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "D:\ProgramData\Anaconda3\envs\env1\lib\site-packages\click\core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "D:\Code\ind_knn_ad-master\indad\run.py", line 79, in cli_interface
write_results(total_results, method)
File "D:\Code\ind_knn_ad-master\indad\utils.py", line 65, in write_results
with open(f"./results/{name}.yml", "w") as outfile:
FileNotFoundError: [Errno 2] No such file or directory: './results/spade_26_07_2021_22_47_37.yml'

Hi! The more image samples are, the greater the 28(Fmap_H)*28(Fmap_W)*num_samples,
so it grows because it is projecting. Is there any way to reduce it?

def fit(self, train_dl):
        for sample, _ in tqdm(train_dl, **get_tqdm_params()):
	        feature_maps = self(sample)
        
	        if self.resize is None:
		        largest_fmap_size = feature_maps[0].shape[-2:]
		        self.resize = torch.nn.AdaptiveAvgPool2d(largest_fmap_size)
	        resized_maps = [self.resize(self.average(fmap)) for fmap in feature_maps]
	        patch = torch.cat(resized_maps, 1)
	        patch = patch.reshape(patch.shape[1], -1).T
        
	        self.patch_lib.append(patch)
        
        self.patch_lib = torch.cat(self.patch_lib, 0)
        
        if self.f_coreset < 1:
	        self.coreset_idx = get_coreset_idx_randomp(
		        self.patch_lib,
		        n=int(self.f_coreset * self.patch_lib.shape[0]),
		        eps=self.coreset_eps,
	        )
	        self.patch_lib = self.patch_lib[self.coreset_idx]

Hi, thanks for your efforts and fast responses.

At the question below, the final goal is only to get the anomaly score and not interested into the heatmap visualization.

I was checking the model.py and I noticed that the created mask by us as the ground truth is only required during evaluation (and only for pixel_rocauc measurement) and not utilized for training (model.fit), am I right?

for sample, mask, label in tqdm(test_dl, **get_tqdm_params()):
z_score, fmap = self.predict(sample)

		  image_preds.append(z_score.numpy())
		  image_labels.append(label)
		  
		  pixel_preds.extend(fmap.flatten().numpy())
		  pixel_labels.extend(mask.flatten().numpy())
		  
	  image_preds = np.stack(image_preds)

	  image_rocauc = roc_auc_score(image_labels, image_preds)
	  pixel_rocauc = roc_auc_score(pixel_labels, pixel_preds)
	  return image_rocauc, pixel_rocauc

So, if I completely eliminate the ground_truth mask dataloading from the all pipelines and I only do image_rocauc= model.evaluate(test_ds) instead of image_rocauc, pixel_rocauc = model.evaluate(test_ds), can I still get only the anomaly score from image_rocauc without any mask labeling (have a complete unsupervised pipeline)?

Best

Hello,

Could I please get an updated version of this project? Many features are not working correctly and there are compatibility issues that I am unable to resolve. As a beginner, I have been trying to fix these problems for the past two months without success. I am currently an intern and this is my first year, so I really need help with this.

Thank you very much for your assistance.

Best regards,

I'm trying to convert pytorch to torch script.I want to know whether the detection speed of the converted model has changed. Neither method in the document can be used.
https://pytorch.org/tutorials/advanced/cpp_export.html

model = SPADE(k=42)  # , backbone_name="hypernet")
train_dataset = StreamingDataset()
test_dataset = StreamingDataset()
app_custom_train_images =  "D:\\cwge\\Dataset\\bottle\\train\\good"
# train images
for root, dirs, files in os.walk(app_custom_train_images):
    for file in files:
        train_dataset.add_pil_image(Image.open(os.path.join(root, file)))
model.fit(train_dataset)
PATH = "test.pth"
torch.save(model.state_dict(), PATH)
# model.load_state_dict(torch.load(PATH))
#example = torch.rand(1, 3, 224, 224).cuda()
#traced_script_module = torch.jit.trace(model, example)
script_net = torch.jit.script(model)
script_net.save('model_script.pt')
#traced_script_module.save("traced_resnet_model.pt")

My C + + code reported an error when loading the model.

#include <iostream>
#include "torch/script.h"
#include <vector>
int main()
{torch::jit::script::Module module = torch::jit::load("D:\\model_script.pt");}

Unhandled exception at 0x00007FFB76BA4B89 in ConsoleApplication1.exe: Microsoft C++ exception: c10::NotImplementedError at memory location 0x00000095C3EFDD50.