donahowe / autostudio Goto Github PK
View Code? Open in Web Editor NEWAutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation
Home Page: https://arxiv.org/abs/2406.01388
AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation
Home Page: https://arxiv.org/abs/2406.01388
good job!
在上报了Issues#3之后,修改如下:
再次运行验证脚本后,终端窗口输出如下:
D:\AITest\AutoStudio\model\pipeline_stable_diffusion.py:41: FutureWarning: Importing DiffusionPipeline
or ImagePipelineOutput
from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
from diffusers.pipeline_utils import DiffusionPipeline
Using box scale: (512, 512)
D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\models\GroundingDINO\ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight']
safety_checker=None
. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at huggingface/diffusers#254 .matplotlib.pyplot.figure
) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam figure.max_open_warning
). Consider using matplotlib.pyplot.close()
.全是 error ,但不知道是哪里错了。。。
请项目大佬帮忙分析指点一下,这么多人尝试,有成功跑出来的吗???
生成图像报错如下!!
│ │
│ /xxx/workpace/AIGC/AutoStudio/DETECT_SAM/efficient_sam.py:31 in inference_with_box │
│ │
│ 28 │ bbox_labels = torch.reshape(torch.tensor([2, 3]), [1, 1, 2]) │
│ 29 │ img_tensor = ToTensor()(image) │
│ 30 │ │
│ ❱ 31 │ predicted_logits, predicted_iou = model( │
│ 32 │ │ img_tensor[None, ...].to(device), │
│ 33 │ │ bbox.to(device), │
│ 34 │ │ bbox_labels.to(device), │
│ │
│ /xxx/anaconda/envs/th23/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 │
│ in _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: shape '[0, 3, 1, 2]' is invalid for input of size 1572864
Hi,
Thank you for your interest in AutoStudio. I have fixed the bugs mentioned in the issue and released the SDXL version. The main issue was related to the environment file. Please reconfigure according to the new requirements.txt. Additionally, I have made some minor bug fixes in run.py, autostudio.py, and diffusionpipeline/diffusionpipelinexl.py.
cond = cond.reshape(cond.shape[0], height, width, cond.shape[2])
RuntimeError: shape '[3, 64, 64, 640]' is invalid for input of size 23592960
12288无法分解成64*64。可是使用1.0的却正常。
由于是SD这方面的新手,在模型配置方面有点疑惑,想请教一下本地Pretrained模型的配置问题
StableDiffusion
模型按照推荐下载了dreamlike-anime-1.0IP-Adaptor
下载efficient_sam_s_gpu.jit
下载groundingdino_swint_ogc.pth
bert-base-uncased
在run.py
脚本中:
sd_path
配置了dreamlike-anime-1.0的文件夹路径vae_path
我也使用了dreamlike-anime-1.0/vae文件夹路径unet
我也使用了dreamlike-anime-1.0/unet文件夹路径ip_ckpt
和image_encoder_path
用了1.5plus
的版本RTX 4090/24GB CUDA 11.8
不知道这样的配置会不会有什么问题,因为输出效果很差,完全无法复现repo和论文中提到的效果
python 3.10
from groundingdino import _C
导入失败
warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
发生异常: NameError
name '_C' is not defined
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/models/GroundingDINO/ms_deform_attn.py", line 53, in forward
output = _C.ms_deform_attn_forward(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/models/GroundingDINO/ms_deform_attn.py", line 338, in forward
output = MultiScaleDeformableAttnFunction.apply(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/models/GroundingDINO/transformer.py", line 793, in forward
src2 = self.self_attn(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/models/GroundingDINO/transformer.py", line 584, in forward
output = checkpoint.checkpoint(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/models/GroundingDINO/transformer.py", line 266, in forward
memory, memory_text = self.encoder(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/models/GroundingDINO/groundingdino.py", line 334, in forward
hs, reference, hs_enc, ref_enc, init_box_proposal = self.transformer(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/util/inference.py", line 79, in predict
outputs = model(image[None], captions=[caption])
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/Grounding-DINO/groundingdino/util/inference.py", line 218, in predict_with_classes
boxes, logits, phrases = predict(
File "/home/admin123/repository/lmm/AutoStudio/DETECT_SAM/detectSam.py", line 100, in process_image
detections = detect_model.predict_with_classes(
File "/home/admin123/repository/lmm/AutoStudio/model/autostudio.py", line 341, in generate
seg_img, detection = process_image(detect_model=dino_model, same_model=same_model, input_image=i[1][0], categories=character_prompt_full, device=self.device)
File "/home/admin123/repository/lmm/AutoStudio/run.py", line 231, in
output = autostudio.generate(
NameError: name '_C' is not defined
发现:
_C.cpython-38-x86_64-linux-gnu.so
_C.cpython-39-x86_64-linux-gnu.so
没有_C.cpython-310-x86_64-linux-gnu.so
这个问题怎么解决?是不是不支持python310?能不能提供3.10的文件
On line 199 in run.py, " output = theatergen2.generate(", the object "theatergen2" doesn't get imported , and it has nowhere to be found in this repo.
Great work, are there any plans to release the entire model?
I've been stuck on that vae config.json error all day. Tried to download a bunch of them from Huggingface such as https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/tree/main to no avail even after modified the path on run.py script. Would be nice if any of you can share how you get it working by sharing a run.bat or something.
TheaterGen论文是基于sd1-5,但是本文Table2的TheaterGen有sdxl版本的结果,
请问这个结果是怎么得到的
当我运行 run.py 时,缺少以下三个 Hugging Face 模型:
/data2/chengjunhao/THEATERGEN/pretrained_models/vae_ft_mse
/data2/chengjunhao/THEATERGEN/pretrained_models/diffusion_1.5_comic
/diffusion_1.5/unet
您能否提供这些模型或其权重的正确 Hugging Face 链接?此外,如果您能提供一个用于此设置的 Docker 容器,将不胜感激。
谢谢!
Getting this problem on an i9-9900k, RTX 2080 + RTX 3060 Ti system. Any suggestions to fix?
NumPy 2.0.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "/home/parsa/HDD/machine learning/AutoStudio/run.py", line 19, in <module>
import torch
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/torch/__init__.py", line 1382, in <module>
from .functional import * # noqa: F403
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/torch/functional.py", line 7, in <module>
import torch.nn.functional as F
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/torch/nn/__init__.py", line 1, in <module>
from .modules import * # noqa: F403
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/torch/nn/modules/__init__.py", line 35, in <module>
from .transformer import TransformerEncoder, TransformerDecoder, \
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/torch/nn/modules/transformer.py", line 20, in <module>
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/torch/nn/modules/transformer.py:20: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:84.)
device: torch.device = torch.device(torch._C._get_default_device()), # torch.device('cpu'),
/home/parsa/HDD/machine learning/AutoStudio/model/pipeline_stable_diffusion.py:41: FutureWarning: Importing `DiffusionPipeline` or `ImagePipelineOutput` from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
from diffusers.pipeline_utils import DiffusionPipeline
/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/gradio_client/documentation.py:106: UserWarning: Could not get documentation group for <class 'gradio.mix.Parallel'>: No known documentation group for module 'gradio.mix'
warnings.warn(f"Could not get documentation group for {cls}: {exc}")
/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/gradio_client/documentation.py:106: UserWarning: Could not get documentation group for <class 'gradio.mix.Series'>: No known documentation group for module 'gradio.mix'
warnings.warn(f"Could not get documentation group for {cls}: {exc}")
Using box scale: (512, 512)
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "/home/parsa/HDD/machine learning/AutoStudio/run.py", line 31, in <module>
from model.autostudio import AUTOSTUDIO, AUTOSTUDIOPlus, AUTOSTUDIOXL, AUTOSTUDIOXLPlus
File "/home/parsa/HDD/machine learning/AutoStudio/model/autostudio.py", line 16, in <module>
from detectSam import process_image
File "/home/parsa/HDD/machine learning/AutoStudio/DETECT_SAM/detectSam.py", line 14, in <module>
import cv2
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/cv2/__init__.py", line 181, in <module>
bootstrap()
File "/home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/cv2/__init__.py", line 153, in bootstrap
native_module = importlib.import_module("cv2")
File "/home/parsa/.pyenv/versions/3.10.6/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
AttributeError: _ARRAY_API not found
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/parsa/HDD/machine learning/AutoStudio/run.py:31 in <module> │
│ │
│ 28 │
│ 29 from model.unet_2d_condition import UNet2DConditionModel │
│ 30 from model.utils import show_boxes, show_image, get_global_prompt │
│ ❱ 31 from model.autostudio import AUTOSTUDIO, AUTOSTUDIOPlus, AUTOSTUDIOXL, AUTOSTUDIOXLPlus │
│ 32 │
│ 33 from detectSam import EFFICIENT_SAM_MODEL, GROUNDING_DINO_MODEL │
│ 34 │
│ │
│ /home/parsa/HDD/machine learning/AutoStudio/model/autostudio.py:16 in <module> │
│ │
│ 13 │
│ 14 from PIL import Image │
│ 15 from typing import List │
│ ❱ 16 from detectSam import process_image │
│ 17 from diffusers.pipelines.controlnet import MultiControlNetModel │
│ 18 from safetensors import safe_open │
│ 19 from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection │
│ │
│ /home/parsa/HDD/machine learning/AutoStudio/DETECT_SAM/detectSam.py:14 in <module> │
│ │
│ 11 sys.path.append(f"{dpath}/YOLO-World/") │
│ 12 sys.path.append(f"{dpath}/Grounding-DINO/") │
│ 13 │
│ ❱ 14 import cv2 │
│ 15 import time │
│ 16 import contextlib │
│ 17 import numpy as np │
│ │
│ /home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/cv2/__init__.py:181 in │
│ <module> │
│ │
│ 178 │ if DEBUG: print('OpenCV loader: DONE') │
│ 179 │
│ 180 │
│ ❱ 181 bootstrap() │
│ 182 │
│ │
│ /home/parsa/HDD/machine learning/AutoStudio/lib/python3.10/site-packages/cv2/__init__.py:153 in │
│ bootstrap │
│ │
│ 150 │ │
│ 151 │ py_module = sys.modules.pop("cv2") │
│ 152 │ │
│ ❱ 153 │ native_module = importlib.import_module("cv2") │
│ 154 │ │
│ 155 │ sys.modules["cv2"] = py_module │
│ 156 │ setattr(py_module, "_native", native_module) │
│ │
│ /home/parsa/.pyenv/versions/3.10.6/lib/python3.10/importlib/__init__.py:126 in import_module │
│ │
│ 123 │ │ │ if character != '.': │
│ 124 │ │ │ │ break │
│ 125 │ │ │ level += 1 │
│ ❱ 126 │ return _bootstrap._gcd_import(name[level:], package, level) │
│ 127 │
│ 128 │
│ 129 _RELOADING = {} │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: numpy.core.multiarray failed to import
如题。
DETECT_SAM/efficient_sam.py报错,TorchScript相关的错误
报错日志:
Traceback (most recent call last):
File "AutoStudio/run.py", line 231, in <module>
output = autostudio.generate(
File "AutoStudio/model/autostudio.py", line 891, in generate
seg_img, detection = process_image(detect_model=dino_model, same_model=same_model, input_image=i[1][0], categories=character_prompt_full, device='cuda:1') #if CUDA out of spa
File "AutoStudio/DETECT_SAM/detectSam.py", line 108, in process_image
detections.mask = inference_with_boxes(
File "AutoStudio/DETECT_SAM/efficient_sam.py", line 62, in inference_with_boxes
mask = inference_with_box(image, box, model, device)
File "AutoStudio/DETECT_SAM/efficient_sam.py", line 31, in inference_with_box
predicted_logits, predicted_iou = model(
File "/data/condaenvs/autostudio/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: shape '[0, 3, 1, 2]' is invalid for input of size 1572864
Hi, I would like to ask that what is the average character-character similarity (aCCS) metric mentioned in your paper, I cannot find its information on the internet. I'm looking forward to your reply, thanks!
大佬,微信群过期了呢,麻烦更新一下群呢
good job!
looking forward to the SDXL code!
code release?
可否提供以下安装步骤的说明 ,谢谢
Prepare /DETECT_SAMefficient_sam_s_gpu.jit and /DETECT_SAM/Grounding-DINO/groundingdino_swint_ogc.pth for groundingdino and efficientSAM
图片是走文生图的模式生成的,能否提前预设一张或多张图片? @donahowe 谢谢
theatergen2在Theatergen的基础上做了什么修改呢
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /root/code/AutoStudio/run.py:199 in │
│ │
│ 196 │ │ │ ind_offset = repeat_ind * LARGE_CONSTANT2 + args.seed_offset │
│ 197 │ │ │ vis_location = [dialogue, turn] │
│ 198 │ │ │ │
│ ❱ 199 │ │ │ output = autostudio.generate( │
│ 200 │ │ │ │ │ │ │ │ │ │ │ GROUNDING_DINO_MODEL, │
│ 201 │ │ │ │ │ │ │ │ │ │ │ EFFICIENT_SAM_MODEL, │
│ 202 │ │ │ │ │ │ │ │ │ │ │ character_database, │
│ │
│ /root/code/AutoStudio/model/autostudio.py:367 in generate │
│ │
│ 364 │ │ │ │ │ │ continue │
│ 365 │ │ │ │
│ 366 │ │ │ # prepare latent_guidance │
│ ❱ 367 │ │ │ latent_guidance_mask, latent_guidance_image = prepare_mid_image(guide_masks, │
│ 368 │ │ │ latent_guidance_mask = latent_guidance_mask.resize((int(width/8), int(height │
│ 369 │ │ │ latent_guidance_mask = np.array(latent_guidance_mask) │
│ 370 │ │ │ latent_guidance_mask = torch.from_numpy(latent_guidance_mask).to(self.device │
│ │
│ /root/code/AutoStudio/model/utils.py:21 in prepare_mid_image │
│ │
│ 18 print(f"Using box scale: {box_scale}") │
│ 19 │
│ 20 def prepare_mid_image(mask_tensor_list_512, single_obj_img_list, bboxes, height, width, │
│ ❱ 21 │ mask_tensor_512 = mask_tensor_list_512[0] │
│ 22 │ #m,n = mask_tensor_512.size() │
│ 23 │ m,n = width, height │
│ 24 │ new_mask_tensor = np.zeros((n, m)).astype(np.uint8) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range
The above error has been taking place every time and wouldn't get fixed by alternating inputs.
Good project, I want to know what is the different between with StoryDiffusion.
env: windows 11 x64, Python3.10.11 + torch 2.1.0+ cu11.8, prepared all stable-diffusion-v1-5/sd-vae-ft-mse/IP-Adapter checkpoints.
error message in the terminal windows:
D:\AITest\AutoStudio\model\pipeline_stable_diffusion.py:41: FutureWarning: Importing DiffusionPipeline
or ImagePipelineOutput
from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
from diffusers.pipeline_utils import DiffusionPipeline
Using box scale: (512, 512)
D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\models\GroundingDINO\ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\AITest\AutoStudio\run_me.py:31 in │
│ │
│ 28 │
│ 29 from model.unet_2d_condition import UNet2DConditionModel │
│ 30 from model.utils import show_boxes, show_image, get_global_prompt │
│ ❱ 31 from model.autostudio import AUTOSTUDIO, AUTOSTUDIOPlus, AUTOSTUDIOXL, AUTOSTUDIOXLPlus │
│ 32 │
│ 33 from detectSam import EFFICIENT_SAM_MODEL, GROUNDING_DINO_MODEL │
│ 34 │
│ │
│ D:\AITest\AutoStudio\model\autostudio.py:16 in │
│ │
│ 13 │
│ 14 from PIL import Image │
│ 15 from typing import List │
│ ❱ 16 from detectSam import process_image │
│ 17 from diffusers.pipelines.controlnet import MultiControlNetModel │
│ 18 from safetensors import safe_open │
│ 19 from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection │
│ │
│ D:\AITest\AutoStudio/DETECT_SAM\detectSam.py:36 in │
│ │
│ 33 │ RESULTS = "results" │
│ 34 │ │
│ 35 │ DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") │
│ ❱ 36 │ EFFICIENT_SAM_MODEL = load(device=DEVICE) │
│ 37 │ GROUNDING_DINO_MODEL = Model(f"{dpath}/Grounding-DINO/groundingdino/config/Grounding │
│ 38 │ │
│ 39 │ BOUNDING_BOX_ANNOTATOR = sv.BoundingBoxAnnotator() │
│ │
│ D:\AITest\AutoStudio/DETECT_SAM\efficient_sam.py:14 in load │
│ │
│ 11 │
│ 12 def load(device: torch.device) -> torch.jit.ScriptModule: │
│ 13 │ if device.type == "cuda": │
│ ❱ 14 │ │ model = torch.jit.load(GPU_EFFICIENT_SAM_CHECKPOINT) │
│ 15 │ else: │
│ 16 │ │ model = torch.jit.load(CPU_EFFICIENT_SAM_CHECKPOINT) │
│ 17 │ model.eval() │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\torch\jit_serialization.py:152 in load │
│ │
│ 149 │ │
│ 150 │ if isinstance(f, str): │
│ 151 │ │ if not os.path.exists(f): # type: ignore[type-var] │
│ ❱ 152 │ │ │ raise ValueError(f"The provided filename {f} does not exist") # type: ignor │
│ 153 │ │ if os.path.isdir(f): │
│ 154 │ │ │ raise ValueError(f"The provided filename {f} is a directory") # type: ignor │
│ 155 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The provided filename D:\AITest\AutoStudio\DETECT_SAM/pretrain/efficient_sam_s_gpu.jit does not exist
Press any key to continue . . .
I find and download the efficient_sam_s_gpu.jit from "https://huggingface.co/merve/EfficientSAM" to put it above directory, and go ahead.
2.------
D:\AITest\AutoStudio\model\pipeline_stable_diffusion.py:41: FutureWarning: Importing DiffusionPipeline
or ImagePipelineOutput
from diffusers.pipeline_utils is deprecated. Please import from diffusers.pipelines.pipeline_utils instead.
from diffusers.pipeline_utils import DiffusionPipeline
Using box scale: (512, 512)
D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\models\GroundingDINO\ms_deform_attn.py:31: UserWarning: Failed to load custom C++ ops. Running on CPU mode Only!
warnings.warn("Failed to load custom C++ ops. Running on CPU mode Only!")
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\AITest\AutoStudio\python310\lib\site-packages\transformers\configuration_utils.py:629 in │
│ _get_config_dict │
│ │
│ 626 │ │ │ │
│ 627 │ │ │ try: │
│ 628 │ │ │ │ # Load from local folder or from cache or download from model Hub and ca │
│ ❱ 629 │ │ │ │ resolved_config_file = cached_file( │
│ 630 │ │ │ │ │ pretrained_model_name_or_path, │
│ 631 │ │ │ │ │ configuration_file, │
│ 632 │ │ │ │ │ cache_dir=cache_dir, │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\transformers\utils\hub.py:417 in cached_file │
│ │
│ 414 │ user_agent = http_user_agent(user_agent) │
│ 415 │ try: │
│ 416 │ │ # Load from URL or cache if already cached │
│ ❱ 417 │ │ resolved_file = hf_hub_download( │
│ 418 │ │ │ path_or_repo_id, │
│ 419 │ │ │ filename, │
│ 420 │ │ │ subfolder=None if len(subfolder) == 0 else subfolder, │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\huggingface_hub\utils_validators.py:110 in │
│ inner_fn │
│ │
│ 107 │ │ │ kwargs.items(), # Kwargs values │
│ 108 │ │ ): │
│ 109 │ │ │ if arg_name in ["repo_id", "from_id", "to_id"]: │
│ ❱ 110 │ │ │ │ validate_repo_id(arg_value) │
│ 111 │ │ │ │
│ 112 │ │ │ elif arg_name == "token" and arg_value is not None: │
│ 113 │ │ │ │ has_token = True │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\huggingface_hub\utils_validators.py:158 in │
│ validate_repo_id │
│ │
│ 155 │ │ raise HFValidationError(f"Repo id must be a string, not {type(repo_id)}: '{repo │
│ 156 │ │
│ 157 │ if repo_id.count("/") > 1: │
│ ❱ 158 │ │ raise HFValidationError( │
│ 159 │ │ │ "Repo id must be in the form 'repo_name' or 'namespace/repo_name':" │
│ 160 │ │ │ f" '{repo_id}'. Use repo_type
argument if needed." │
│ 161 │ │ ) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name':
'/data2/chengjunhao/THEATERGEN/pretrained_models/dino_bert'. Use repo_type
argument if needed.
During handling of the above exception, another exception occurred:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\AITest\AutoStudio\run_me.py:31 in │
│ │
│ 28 │
│ 29 from model.unet_2d_condition import UNet2DConditionModel │
│ 30 from model.utils import show_boxes, show_image, get_global_prompt │
│ ❱ 31 from model.autostudio import AUTOSTUDIO, AUTOSTUDIOPlus, AUTOSTUDIOXL, AUTOSTUDIOXLPlus │
│ 32 │
│ 33 from detectSam import EFFICIENT_SAM_MODEL, GROUNDING_DINO_MODEL │
│ 34 │
│ │
│ D:\AITest\AutoStudio\model\autostudio.py:16 in │
│ │
│ 13 │
│ 14 from PIL import Image │
│ 15 from typing import List │
│ ❱ 16 from detectSam import process_image │
│ 17 from diffusers.pipelines.controlnet import MultiControlNetModel │
│ 18 from safetensors import safe_open │
│ 19 from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection │
│ │
│ D:\AITest\AutoStudio/DETECT_SAM\detectSam.py:37 in │
│ │
│ 34 │ │
│ 35 │ DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") │
│ 36 │ EFFICIENT_SAM_MODEL = load(device=DEVICE) │
│ ❱ 37 │ GROUNDING_DINO_MODEL = Model(f"{dpath}/Grounding-DINO/groundingdino/config/Grounding │
│ 38 │ │
│ 39 │ BOUNDING_BOX_ANNOTATOR = sv.BoundingBoxAnnotator() │
│ 40 │ MASK_ANNOTATOR = sv.MaskAnnotator() │
│ │
│ D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\util\inference.py:142 in init │
│ │
│ 139 │ │ model_checkpoint_path: str, │
│ 140 │ │ device: str = "cuda" │
│ 141 │ ): │
│ ❱ 142 │ │ self.model = load_model( │
│ 143 │ │ │ model_config_path=model_config_path, │
│ 144 │ │ │ model_checkpoint_path=model_checkpoint_path, │
│ 145 │ │ │ device=device │
│ │
│ D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\util\inference.py:40 in load_model │
│ │
│ 37 def load_model(model_config_path: str, model_checkpoint_path: str, device: str = "cuda") │
│ 38 │ args = SLConfig.fromfile(model_config_path) │
│ 39 │ args.device = device │
│ ❱ 40 │ model = build_model(args) │
│ 41 │ checkpoint = torch.load(model_checkpoint_path, map_location="cpu") │
│ 42 │ model.load_state_dict(clean_state_dict(checkpoint["model"]), strict=False) │
│ 43 │ model.eval() │
│ │
│ D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\models_init_.py:17 in │
│ build_model │
│ │
│ 14 │ │
│ 15 │ assert args.modelname in MODULE_BUILD_FUNCS.module_dict │
│ 16 │ build_func = MODULE_BUILD_FUNCS.get(args.modelname) │
│ ❱ 17 │ model = build_func(args) │
│ 18 │ return model │
│ 19 │
│ │
│ D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\models\GroundingDINO\groundingdino. │
│ py:395 in build_groundingdino │
│ │
│ 392 │ dec_pred_bbox_embed_share = args.dec_pred_bbox_embed_share │
│ 393 │ sub_sentence_present = args.sub_sentence_present │
│ 394 │ │
│ ❱ 395 │ model = GroundingDINO( │
│ 396 │ │ backbone, │
│ 397 │ │ transformer, │
│ 398 │ │ num_queries=args.num_queries, │
│ │
│ D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\models\GroundingDINO\groundingdino. │
│ py:115 in init │
│ │
│ 112 │ │ │
│ 113 │ │ # bert │
│ 114 │ │ self.tokenizer = get_tokenlizer.get_tokenlizer(text_encoder_type) │
│ ❱ 115 │ │ self.bert = get_tokenlizer.get_pretrained_language_model(text_encoder_type) │
│ 116 │ │ self.bert.pooler.dense.weight.requires_grad(False) │
│ 117 │ │ self.bert.pooler.dense.bias.requires_grad_(False) │
│ 118 │ │ self.bert = BertModelWarper(bert_model=self.bert) │
│ │
│ D:\AITest\AutoStudio\DETECT_SAM/Grounding-DINO\groundingdino\util\get_tokenlizer.py:25 in │
│ get_pretrained_language_model │
│ │
│ 22 │
│ 23 def get_pretrained_language_model(text_encoder_type): │
│ 24 │ if text_encoder_type == "bert-base-uncased" or (os.path.isdir(text_encoder_type) and │
│ ❱ 25 │ │ return BertModel.from_pretrained("/data2/chengjunhao/THEATERGEN/pretrained_model │
│ 26 │ if text_encoder_type == "roberta-base": │
│ 27 │ │ return RobertaModel.from_pretrained(text_encoder_type) │
│ 28 │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\transformers\modeling_utils.py:2251 in │
│ from_pretrained │
│ │
│ 2248 │ │ # Load config if we don't provide a configuration │
│ 2249 │ │ if not isinstance(config, PretrainedConfig): │
│ 2250 │ │ │ config_path = config if config is not None else pretrained_model_name_or_pat │
│ ❱ 2251 │ │ │ config, model_kwargs = cls.config_class.from_pretrained( │
│ 2252 │ │ │ │ config_path, │
│ 2253 │ │ │ │ cache_dir=cache_dir, │
│ 2254 │ │ │ │ return_unused_kwargs=True, │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\transformers\configuration_utils.py:547 in │
│ from_pretrained │
│ │
│ 544 │ │ assert config.output_attentions == True │
│ 545 │ │ assert unused_kwargs == {"foo": False} │
│ 546 │ │ ```""" │
│ ❱ 547 │ │ config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwarg │
│ 548 │ │ if "model_type" in config_dict and hasattr(cls, "model_type") and config_dict["m │
│ 549 │ │ │ logger.warning( │
│ 550 │ │ │ │ f"You are using a model of type {config_dict['model_type']} to instantia │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\transformers\configuration_utils.py:574 in │
│ get_config_dict │
│ │
│ 571 │ │ """ │
│ 572 │ │ original_kwargs = copy.deepcopy(kwargs) │
│ 573 │ │ # Get config dict associated with the base config file │
│ ❱ 574 │ │ config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwar │
│ 575 │ │ if "_commit_hash" in config_dict: │
│ 576 │ │ │ original_kwargs["_commit_hash"] = config_dict["_commit_hash"] │
│ 577 │
│ │
│ D:\AITest\AutoStudio\python310\lib\site-packages\transformers\configuration_utils.py:650 in │
│ _get_config_dict │
│ │
│ 647 │ │ │ │ raise │
│ 648 │ │ │ except Exception: │
│ 649 │ │ │ │ # For any other exception, we throw a generic error. │
│ ❱ 650 │ │ │ │ raise EnvironmentError( │
│ 651 │ │ │ │ │ f"Can't load the configuration of '{pretrained_model_name_or_path}'. │
│ 652 │ │ │ │ │ " from 'https://huggingface.co/models', make sure you don't have a l │
│ 653 │ │ │ │ │ f" name. Otherwise, make sure '{pretrained_model_name_or_path}' is t │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OSError: Can't load the configuration of '/data2/chengjunhao/THEATERGEN/pretrained_models/dino_bert'. If you were
trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name.
Otherwise, make sure '/data2/chengjunhao/THEATERGEN/pretrained_models/dino_bert' is the correct path to a directory
containing a config.json file
Press any key to continue . . .
I check the "https://github.com/IDEA-Research/GroundingDINO" and "https://github.com/donahowe/Theatergen", but can't find your pretrained model "dino_bert".
Please help me to check and analayze the error message, thanks!
能提供抱脸脸的试玩也行~要是有1键包就更好了,谢谢
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/run.py:32 in │
│ │
│ 29 │
│ 30 from model.unet_2d_condition import UNet2DConditionModel │
│ 31 from model.utils import show_boxes, show_image, get_global_prompt │
│ ❱ 32 from model.autostudio import AUTOSTUDIO, AUTOSTUDIOPlus, AUTOSTUDIOXL, AUTOSTUDIOXLPlus │
│ 33 │
│ 34 from detectSam import EFFICIENT_SAM_MODEL, GROUNDING_DINO_MODEL │
│ 35 │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/model/autostudio.py:16 in │
│ │
│ 13 │
│ 14 from PIL import Image │
│ 15 from typing import List │
│ ❱ 16 from detectSam import process_image │
│ 17 from diffusers.pipelines.controlnet import MultiControlNetModel │
│ 18 from safetensors import safe_open │
│ 19 from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/DETECT_SAM/detectSam.py:22 in │
│ │
│ 19 import supervision as sv │
│ 20 import torch │
│ 21 │
│ ❱ 22 from groundingdino.util.inference import Model, load_image │
│ 23 │
│ 24 #from mmengine.runner import Runner │
│ 25 #from mmengine.config import Config │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/util/inference.py:20 in │
│ │
│ │
│ 17 sys.path.append(f"{ppath}/") │
│ 18 │
│ 19 import groundingdino.datasets.transforms as T │
│ ❱ 20 from groundingdino.models import build_model │
│ 21 from groundingdino.util.misc import clean_state_dict │
│ 22 from groundingdino.util.slconfig import SLConfig │
│ 23 from groundingdino.util.utils import get_phrases_from_posmap │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/models/init.py:8 in │
│ │
│ │
│ 5 # Licensed under the Apache License, Version 2.0 [see LICENSE for details] │
│ 6 # ------------------------------------------------------------------------ │
│ 7 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved │
│ ❱ 8 from .GroundingDINO import build_groundingdino │
│ 9 │
│ 10 │
│ 11 def build_model(args): │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/models/GroundingDINO/init.p │
│ y:15 in │
│ │
│ 12 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved. │
│ 13 # ------------------------------------------------------------------------ │
│ 14 │
│ ❱ 15 from .groundingdino import build_groundingdino │
│ 16 │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/models/GroundingDINO/groundingd │
│ ino.py:54 in │
│ │
│ 51 │ generate_masks_with_special_tokens, │
│ 52 │ generate_masks_with_special_tokens_and_transfer_map, │
│ 53 ) │
│ ❱ 54 from .transformer import build_transformer │
│ 55 from .utils import MLP, ContrastiveEmbed, sigmoid_focal_loss │
│ 56 │
│ 57 │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/models/GroundingDINO/transforme │
│ r.py:36 in │
│ │
│ 33 from groundingdino.util.misc import inverse_sigmoid │
│ 34 │
│ 35 from .fuse_modules import BiAttentionBlock │
│ ❱ 36 from .ms_deform_attn import MultiScaleDeformableAttention as MSDeformAttn │
│ 37 from .transformer_vanilla import TransformerEncoderLayer │
│ 38 from .utils import ( │
│ 39 │ MLP, │
│ │
│ /shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/models/GroundingDINO/ms_deform_ │
│ attn.py:28 in │
│ │
│ 25 from torch.autograd.function import once_differentiable │
│ 26 from torch.nn.init import constant_, xavier_uniform_ │
│ 27 │
│ ❱ 28 from groundingdino import _C │
│ 29 try: │
│ 30 │ from groundingdino import _C │
│ 31 except: │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ImportError: cannot import name '_C' from 'groundingdino' (/shared/jiayu_ti/git_project/backing_up/AutoStudio/groundingdino/init.py)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.