Hi,

The results of "Fine-Grained Entity Segmentation" look very exciting! When do you plan to release the data and code for this paper?

Thanks!

FileNotFoundError: [Errno 2] No such file or directory: 'D:/Code/Project/Carvana_dataset/testimage\Pic_2023_03_18_091334_5_seg.png'

Thanks for your work。Can the Cropformer-Swin model be converted to tensorRT? How can I improve the speed of predict? I am currently testing 50 images using NVIDIA GeForce RTX 3090, which takes 15 minutes.

Hi,

When do you plan to release the data and code of the paper "Fine-Grained Entity Segmentation"?

Thanks!

@qqlu 哈喽，您好,CropFormer训练代码有实现吗？是用的mask2former的代码吗？

I observe that no global_only mode has the parameter coord, but there is no parameter coord in global_only mode


Is there something wrong with me or should global_only use cascadePSP's model?

Hello, thanks for your contribution. When I used make_entity_mask.py to genere my own annotation, I found a problem in generating entity mask. When a small object is contained by a large object, if the small object is annotated first, the annotation of the small object will be covered by the annotation of the large object later. For example, in the picture below, the object in the red frame is covered

By the way, my task is open world instance segmentation, I think this problem will not be met in the panoptic segmentation because the different annotation format. My method is to annotate objects according to their area from large to small, so that small objects will not be covered.

Great work! I am using the mask2former entity seg model and after extracting the masks and their confidences using prediction.score, it seems that all of the values are 0.7. Any idea why this might be happening? Thank you!

I am trying to training with CUDA11.1 on 2080Ti x4.

The training crashed after ~600 iteration with

[08/28 15:43:34] fvcore.common.checkpoint WARNING: The checkpoint state_dict contains keys that are not used by the model:
  [35mfc1000.{bias, weight}�[0m
[08/28 15:43:34] detectron2.engine.train_loop INFO: Starting training from iteration 0
[08/28 15:56:53] detectron2.engine.train_loop ERROR: Exception during training:
Traceback (most recent call last):
  File "/data/detectron2/detectron2/engine/train_loop.py", line 149, in train
    self.run_step()
  File "/data/detectron2/detectron2/engine/defaults.py", line 495, in run_step
    self._trainer.run_step()
  File "/data/detectron2/detectron2/engine/train_loop.py", line 273, in run_step
    loss_dict = self.model(data)
  File "/opt/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/data/detectron2/projects/EntitySeg/entityseg/arch.py", line 103, in forward
    num_instances = int(torch.max(instance_map)+1)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[08/28 15:56:53] detectron2.engine.hooks INFO: Overall training speed: 653 iterations in 0:13:04 (1.2021 s / it)
[08/28 15:56:53] detectron2.engine.hooks INFO: Total training time: 0:13:05 (0:00:00 on hooks)

Hi, I have tried to realize your method(High Quality Segmentation for Ultra High-resolution Images), it works well. But do you have met the problem that the wrong mask segmentation happens when the RGB-image area is similar to the real label area color. For example , I want to segment the sky area, but the white wall also to be thinked as sky.

Hi，I am interested in your this paper, but I have some question. Can you explain the pseudo-label dataset generation stage? How to calculated the score of any prediction and (constant)score threshold δ is meaning box confidence threshold?

Hi,

Thanks for releasing such a great dataset! I am trying to replicate the result using the released weights on Huggingface, but found that some config files are missing, such as this one. Could you upload these files?

Thanks

I am running EntitySegRLE and there seems to be some files missing.

I am getting a ModuleNotFoundError: No module named 'entityseg.arch_rle_aug' error.

Additionally, in the config files, there seems to be only a few config files compared to the available models. How do I get the relevant files? Thank you!!

It seems that entity loads gt_bitmasks from .npz files built from PNG files, instead of using gt_masks from the RLEs in the instances_*.json.
Is there any particular reason for doing so?
If it has to be like this, it would be favorable to provide scripts to convert coco RLE jsons into PNG files which could then be used in make_entity_mask.py

作者您好：
很高兴看到您的UniGS论文被cvpr 2024接收，我对您这篇工作比较好奇和感兴趣，特别生成和分割统一表示的部分以及之前您做过的高质量分割，请问啥时候可以开源下代码~

Could you explain the pretrained models form this page?

Were all of these models pretrained using the COCO dataset and the numbers are evaluation numbers on the EntitySeg-{semantic, instance, panoptic, entity} datasets? Or are they just starting with COCO pretrained weights, and then fine-tuned on the EntitySeg dataset and the evaluation numbers are from some other dataset?

Hello!
This work of AIMS is intersting.
When do you plan to release the code of the paper "AIMS: All-Inclusive Multi-Level Segmentation", please?

When i print the ori_scores and scores in demo_result_and_vis.py, the scores is is not sorted from high to low. The reason is mask rescoring. But score in scores which higher than args.confidence_threshold is discarded in visualization.

Thanks for your work. I met a problem when training coco panoptic model by using multi GPUs.

My command is python3 projects/EntitySeg/train_net.py --config-file projects/EntitySeg/configs/entity_swin_t_1x.yaml --num-gpus 4 OUTPUT_DIR entity_model . Below is my error:

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

Any help is greatly appreciated. Thank you!

hello, thanks for your great work High Quality Segmentation.
I want to ask the following 2 questions

1. I don't see any difference in P position information between images. Even in an image, the information of the variable rel_cell is the same in all locations. So why can CRM generate detailed segmentation masks?
2. What is the meaning of calculating 3 features in P because it seems to me that it is hard-fixed and concatenated into each feature of the image?
   below is the map that I printed out for the variables

你好，请问模型公开了吗？点击进去，没有可以下载的模型。

Hello guys,
I've read "High Quality Segmentation for Ultra High-resolution Images" recently. Great work!
I am very excited about this paper.
When do you plan to release code for this paper?

Thanks in advance.

Hi, I am getting the following error when running this coarse partition network with train2017 dataset

ERROR [08/05 15:18:06 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
File "/home/hndx/detectron2-main/detectron2/engine/train_loop.py", line 149, in train
self.run_step()
File "/home/hndx/detectron2-main/detectron2/engine/defaults.py", line 494, in run_step
self._trainer.run_step()
File "/home/hndx/detectron2-main/detectron2/engine/train_loop.py", line 268, in run_step
data = next(self._data_loader_iter)
File "/home/hndx/detectron2-main/detectron2/data/common.py", line 234, in iter
for d in self.dataset:
File "/home/hndx/anaconda3/envs/llz0/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/hndx/anaconda3/envs/llz0/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
return self._process_data(data)
File "/home/hndx/anaconda3/envs/llz0/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
data.reraise()
File "/home/hndx/anaconda3/envs/llz0/lib/python3.7/site-packages/torch/_utils.py", line 434, in reraise
raise exception
AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/hndx/anaconda3/envs/llz0/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/hndx/anaconda3/envs/llz0/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch
data.append(next(self.dataset_iter))
File "/home/hndx/detectron2-main/detectron2/data/common.py", line 201, in iter
yield self.dataset[idx]
File "/home/hndx/detectron2-main/detectron2/data/common.py", line 90, in getitem
data = self._map_func(self._dataset[cur_idx])
File "/home/hndx/detectron2-main/detectron2/utils/serialize.py", line 26, in call
return self._obj(*args, **kwargs)
File "/home/hndx/detectron2-main/detectron2/projects/EntitySeg/entityseg/data/dataset_mapper.py", line 197, in call
instances.instanceid = instance_id_list
File "/home/hndx/detectron2-main/detectron2/structures/instances.py", line 66, in setattr
self.set(name, val)
File "/home/hndx/detectron2-main/detectron2/structures/instances.py", line 84, in set
), "Adding a field of length {} to a Instances of length {}".format(data_len, len(self)) ##lizhi long

Where can I find the code of the CA-SSL?

results of pretrained model mit_b5_1x.pth on COCO val set are not good.Here is my command:

python projects/EntitySeg/train_net.py --config-file projects/EntitySeg/configs/entity_mit_b5_1x.yaml --num-gpus 1 --eval-only MODEL.WEIGHTS data/models/mit_b5_1x.pth MODEL.CONDINST.MASK_BRANCH.USE_MASK_RESCORE "True"

and the results:

[08/11 16:11:11 d2.engine.defaults]: Evaluation results for coco_2017_val_entity in csv format: [08/11 16:11:11 d2.evaluation.testing]: copypaste: Task: bbox [08/11 16:11:11 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl [08/11 16:11:11 d2.evaluation.testing]: copypaste: 0.0010,0.0019,0.0008,0.0004,0.0014,0.0013 [08/11 16:11:11 d2.evaluation.testing]: copypaste: Task: segm [08/11 16:11:11 d2.evaluation.testing]: copypaste: AP,AP50,AP75,APs,APm,APl [08/11 16:11:11 d2.evaluation.testing]: copypaste: 0.0006,0.0015,0.0005,0.0002,0.0008,0.0009

It seems current entity_evaluation.py can only calculate entity AP for coco datasets only. Yet in your paper, there are results for other datasets like ade20k and cityscapes. Could you please provide code for the evaluation of those datasets? Thank you very much.

Hello, you‘ve mentioned at the beginning of your abstract that the traditional approach have the problem of balancing computational cost and computational time. So I want to know how much graphics card memory and computing time your method needs when conducting 2K or 6K images？And what kind of graphics card we need to run this code? Thank you.

Hi, why do I get this error when I run the coco instance_2017 dataset with the source program?
This error is as follows

Traceback (most recent call last):
File "/home/hndx/detectron2-main/detectron2/projects/EntitySeg/train_net.py", line 81, in
args=(args,),
File "/home/hndx/detectron2-main/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "/home/hndx/detectron2-main/detectron2/projects/EntitySeg/train_net.py", line 67, in main
trainer = Trainer(cfg)
File "/home/hndx/detectron2-main/detectron2/engine/defaults.py", line 378, in init
data_loader = self.build_train_loader(cfg)
File "/home/hndx/detectron2-main/detectron2/projects/EntitySeg/train_net.py", line 33, in build_train_loader
return build_detection_train_loader(cfg, mapper)
File "/home/hndx/detectron2-main/detectron2/config/config.py", line 207, in wrapped
explicit_args = _get_args_from_config(from_config, *args, **kwargs)
File "/home/hndx/detectron2-main/detectron2/config/config.py", line 245, in _get_args_from_config
ret = from_config_func(*args, **kwargs)
File "/home/hndx/detectron2-main/detectron2/data/build.py", line 350, in _train_loader_from_config
proposal_files=cfg.DATASETS.PROPOSAL_FILES_TRAIN if cfg.MODEL.LOAD_PROPOSALS else None,
File "/home/hndx/detectron2-main/detectron2/data/build.py", line 278, in get_detection_dataset_dicts
assert len(dataset_dicts), "No valid data found in {}.".format(",".join(names))
AssertionError: No valid data found in coco_2017_train_entity.

Hi there, thanks for your work! It looks very interesting and i would like to test it out, however I keep running into the same problem when testing the visualization script and wondering if you could help me out.

Steps I did:

1. Build detectron2 from source using the same commit version and pytorch version in your README
2. clone this repo into projects/EntitySeg
3. Download the pretrained R50.pth model
4. Install the panopticapi library
5. Run python3 projects/EntitySeg/demo_result_and_vis.py --config-file projects/EntitySeg/configs/entity_swin_lw7_1x.yaml --input data/input/*.jpg --output data/output MODEL.WEIGHTS data/pretrained_model/R_50.pth MODEL.CONDINST.MASK_BRANCH.USE_MASK_RESCORE "True"

Below is my error:

Any help is greatly appreciated. Thank you!

Thanks for your great work "Open-World Entity Segmentation". Could you provide the evaluation code on ADE-20K dataset, when using models trained on COCO?

Hello! Have you tried to refine the mask which is more than one class at same time, such as 3 classes or 4 classes? And how about the result? Thank you very much.

where can i find code of the paper?

When I run "python3 entity_to_json.py", the following problems occur：
Traceback (most recent call last):
File "entity_to_json.py", line 23, in
catid_map = {category['id']: [cid, category["isthing"], category["name"], category["supercategory"]] for cid, category in enumerate(categories_list)}
File "entity_to_json.py", line 23, in
catid_map = {category['id']: [cid, category["isthing"], category["name"], category["supercategory"]] for cid, category in enumerate(categories_list)}
KeyError: 'supercategory'

I find that the problem occurs in categories_list, and there is no "supercategory" key in categories_list(from detectron2.data.datasets.builtin_meta import COCO_CATEGORIES) .

When I train with 2 GPUs, I have the same problem in facebookresearch/detectron2#4191.
Thus, after reading your README.MD, I want to know how to set the "find_unused_parameters=True" in distributed training of my own detectron2.
I have tried to set the "find_unused_parameters=True" in the detectron2/engine/defaults.py

Thus, I want to ask you whether I change it correctly.
I am looking forward to your response.

'torch.cuda.empty_cache()' seems don`t work after process one chunk image, I get an RuntimeError: CUDA out of memory.

First of all, many thanks to the authors for open sourcing such a great project. I was a little confused about the image format conversion.
Why use 'BGR' to input image in demo_crop, set cfg.INPUT.FORMAT = 'RGB', and then flip image to 'RGB' before the image is fed into the network? Doesn't this cause confusion, or is there something special going on.

1. In detectron2/projects/CropFormer/demo_cropformer/demo_from_dirs.py
   img = read_image(path, format="BGR")
2. at this time cfg.INPUT.FORMAT = 'RGB'
3. and in detectron2/projects/CropFormer/demo_cropformer/predictor.py
   if self.input_format == "RGB":
   # whether the model expects BGR inputs or RGB
   original_image = original_image[:, :, ::-1]
   While the end result is correct, the process is confusing.
   If there is any special purpose, I really look forward to your explanation for me. Thank you again for such a good work.
   

Hi，thank you for your contributions to entity segmentation task. When I train the model followed your steps, I encountered the following problems: FloatingPointError: Loss became infinite or NaN at iteration=20! . I am looking forward to you reply.

Thanks for your great work. I have a question that how the panoptic results of PanoFCN or DETR transfered to the entity segmentation results?

Why do you use an imperfect library detectron2?

I run test.py but i met this error, miss "_seg.png" file

python test.py --dir ./data/DUTS-TE --model ./model_10000 --output ./output --clear
/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/torchvision/io/image.py:11: UserWarning: Failed to load image Python extension: /home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/torchvision/image.so: undefined symbol: _ZNK2at10TensorBase21__dispatch_contiguousEN3c1012MemoryFormatE
warn(f"Failed to load image Python extension: {e}")

before_Parser_time: 1659253874.6776164
Hyperparameters: {'dir': './data/DUTS-TE', 'model': './model_10000', 'output': './output', 'global_only': False, 'L': 900, 'stride': 450, 'clear': True, 'ade': False}
ASPP_4level
12 images found

before_for_time: 1659253881.0989463 ; before_for_time - before_Parser_time: 6.421329975128174
Traceback (most recent call last):
File "test.py", line 106, in
for im, seg, gt, name, crm_data in progressbar.progressbar(val_loader):
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/progressbar/shortcuts.py", line 10, in progressbar
for result in progressbar(iterator):
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/progressbar/bar.py", line 547, in next
value = next(self._iterable)
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/hoang/Desktop/luanvan/implicitmodel/4cham/Entity/High-Quality-Segmention/dataset/offline_dataset_crm_pad32.py", line 138, in getitem
im, seg, gt = self.load_tuple(self.im_list[idx])
File "/home/hoang/Desktop/luanvan/implicitmodel/4cham/Entity/High-Quality-Segmention/dataset/offline_dataset_crm_pad32.py", line 110, in load_tuple
seg = Image.open(im[:-7]+'_seg.png').convert('L')
File "/home/hoang/anaconda3/envs/tranSOD/lib/python3.6/site-packages/PIL/Image.py", line 2975, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: './data/DUTS-TE/sun_abzsyxfgntlvd_seg.png'

Hi! I wonder how many and what kinds of GPUs did you use in UniGS? How long does the training process take? Thanks!