weixi-feng / layoutgpt Goto Github PK
View Code? Open in Web Editor NEWOfficial repo for LayoutGPT
License: MIT License
Official repo for LayoutGPT
License: MIT License
Can anybody tell me why distinguish between spatial and counting data sets?
I have a couple of questions regarding the usage and retrieval of CSS structures in conjunction with conditions and images. I would appreciate some clarification on the following points:
2D Image Layout Generation
The generated layout will be saved to ./llm_output/counting by default. To generate images based on the layouts, run
`(layoutgpt) ubuntu@host13:~/nj/text-3d/LayoutGPT/gligen$ python gligen_layout_counting.py --file ../llm_output/counting/gpt4.counting. k-similar.k_8.px_64.json --batch_size 5
Images will be saved at ./generation_samples/counting/gpt4.counting.k-similar.k_8.px_64
Traceback (most recent call last):
File "gligen_layout_counting.py", line 427, in
_main(args)
File "gligen_layout_counting.py", line 410, in _main
run(meta_list, args, starting_noise)
File "/home/ubuntu/anaconda3/envs/layoutgpt/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "gligen_layout_counting.py", line 282, in run
model, autoencoder, text_encoder, diffusion, config = load_ckpt(meta["ckpt"])
File "gligen_layout_counting.py", line 74, in load_ckpt
saved_ckpt = torch.load(ckpt_path)
File "/home/ubuntu/anaconda3/envs/layoutgpt/lib/python3.8/site-packages/torch/serialization.py", line 791, in load
with _open_file_like(f, 'rb') as opened_file:
File "/home/ubuntu/anaconda3/envs/layoutgpt/lib/python3.8/site-packages/torch/serialization.py", line 271, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/home/ubuntu/anaconda3/envs/layoutgpt/lib/python3.8/site-packages/torch/serialization.py", line 252, in init
super().init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'gligen_checkpoints/checkpoint_generation_text.pth'
`
No such file or directory: 'gligen_checkpoints/checkpoint_generation_text.pth'
For GLIGEN, download the Box+Text checkpoint and put it under gligen/gligen_checkpoints, so I can download diffusion_pytorch_model.bin under that website.
But I cann't find which step do this job(downloading checkpoint_generation_text.pth), can you help me?
Hi,Thank you for your amazing job. I would like to try this layoutGPT for 3D indoor scene generation. So, do I need to do these steps below?
Hi,
I'm running into this issue in the layout to image step:
Traceback (most recent call last):File "/mnt/LayoutGPT/gligen/gligen layout counting.py", line 427, in main(args)
File "/mnt/LayoutGPT/gligen/gligen layout counting.py", line 410, in _mainrun(meta list, args, starting noise)File "/root/miniconda3/envs/myconda/lib/python3.9/site-packages/torch/utils/ contextlib.py", line 115, in decorate contextreturn func(*args,**kwargs)File "/mnt/LayoutGpT/gligen/gligen layout counting.py", line 282, in run
model, autoencoder, text encoder, diffusion, config = load ckpt(metar"ckpt"])
File "/mnt/LayoutGPT/gligen/gligen layout counting.py", line 85, in load ckpttext encoder.load state dict( saved ckptr"text encoder"File "/root/miniconda3/envs/myconda/lib/python3.9/site-packapes/torch/mn/modules/module,py", line 2041, in load state dictraise RuntimeError('Error(s) in loading state dict for :nlt'.format(RuntimeError: Error(s) in loading state dict for FrozenCLIPEmbedder:
Unexpected kev(s) in state dict:
"transformer.text model.embeddings .position ids"
I'm sure I used the right GLIGEN checkpoint as instructed, but not able to figure out why there were unexpected keys.
Hello, I would like to express my appreciation for your admirable work. I must say that the visualizations in the paper are quite impressive. I am curious about how you created the 3D indoor scenes. Would it be possible for you to share the codes or scripts that you used? I would be grateful. Thank you.
BTW, may I ask about the timeline for open-sourcing?
When I run the scriptrun_layoutgpt_3d.py
, the following error occurs:
This model's maximum context length is 4097 tokens. However, your messages resulted in 4109 tokens. Please reduce the length of the messages.Input too long. Will shrink the prompting examples
Hi Feng, Thank you for releasing the code of your great work!
As for one of the baseline approached mentioned in the paper, i.e., the modified LayoutTransformer, are you planning on also releasing your re-implementation? It would be of great help since the original LayoutTransformer does not receive textual description as input.
Best Regard
I cannot find any information about your supporting set, which is NSR-1K/counting/counting.train.json
.
The only thing I know is it is extracted from the MSCOCO-2017 dataset.
Could you give me more specific information about this json file? (i.e. how you choose some images)
Hi! This is a great repo!
I have finished the data preparation steps and generated the scene by run_layoutgpt_3d.py.
I'm trying to visualize the generated 3D layout by rendering the image scenes:
basedir="/mnt/data/agic/data"
visualization_output_dir="./vis"
output_directory=$basedir"/output_pickle/threed_future_model_bedroom.pkl"
python render_from_file.py ../config/bedrooms_eval_config.yaml $visualization_output_dir $output_directory ../demo/floor_plan_texture_images \
../../llm_output/3D/gpt3.5.bedroom.k-similar.k_8.px_regular.json \
--up_vector 0,1,0 --camera_position 2,2,2 --split test_regular --export_scene
I met the following error:
Traceback (most recent call last):
File "render_from_files.py", line 383, in <module>
main(sys.argv[1:])
File "render_from_files.py", line 318, in main
renderables, trimesh_meshes = get_textured_objects(
File "/mnt/data/agic/LayoutGPT/ATISS/scene_synthesis/utils.py", line 31, in get_textured_objects
raw_mesh = TexturedMesh.from_file(furniture.raw_model_path)
File "/home/chenzheng/anaconda3/envs/atiss/lib/python3.8/site-packages/simple_3dviz/renderables/textured_mesh.py", line 300, in from_file
mtl = read_material_file(mesh.material_file)
File "/home/chenzheng/anaconda3/envs/atiss/lib/python3.8/site-packages/simple_3dviz/io/__init__.py", line 27, in read_material_file
return {
File "/home/chenzheng/anaconda3/envs/atiss/lib/python3.8/site-packages/simple_3dviz/io/material.py", line 25, in __init__
self.read(filename)
File "/home/chenzheng/anaconda3/envs/atiss/lib/python3.8/site-packages/simple_3dviz/io/material.py", line 113, in read
self._Ns = float([
IndexError: list index out of range
I do not understand why this error occurred and don't know how to resolve it. Could you help me?
I found that simple_3dviz/io/material.py tried to read data from
3D-FUTURE-model/c2fecd9b-c61e-423a-a48d-08c63931cd1f/model.mtl
And simple_3dviz assumed that model.mtl contains the specular exponent, namely a line starting with "Ns". But in fact, I do not find that. How should I resolve this gap?
The content of model.mtl:
newmtl solid_001_wire
d 1
Tr 0
Tf 1.000000 1.000000 1.000000
illum 2
Ka 0.000000 0.000000 0.000000
Kd 0.000000 0.000000 0.000000
Ks 0.313725 0.313725 0.313725
Ke 0.000000 0.000000 0.000000
map_Ka ./texture.png
map_Kd ./texture.png
newmtl solid_002_wire
d 1
Tr 0
Tf 1.000000 1.000000 1.000000
illum 2
Ka 0.196078 0.196078 0.196078
Kd 0.196078 0.196078 0.196078
Ks 0.705882 0.705882 0.705882
Ke 0.000000 0.000000 0.000000
map_Ka ./texture.png
map_Kd ./texture.png
newmtl solid_003_wire
d 1
Tr 0
Tf 1.000000 1.000000 1.000000
illum 2
Ka 0.500000 0.500000 0.500000
Kd 0.500000 0.500000 0.500000
Ks 0.117647 0.117647 0.117647
Ke 0.000000 0.000000 0.000000
map_Ka ./texture.png
map_Kd ./texture.png
newmtl solid_004_wire
d 1
Tr 0
Tf 1.000000 1.000000 1.000000
illum 2
Ka 0.500000 0.500000 0.500000
Kd 0.500000 0.500000 0.500000
Ks 0.098039 0.098039 0.098039
Ke 0.000000 0.000000 0.000000
map_Ka ./texture.png
map_Kd ./texture.png
newmtl solid_005_wire
d 1
Tr 0
Tf 1.000000 1.000000 1.000000
illum 2
Ka 0.500000 0.500000 0.500000
Kd 0.500000 0.500000 0.500000
Ks 0.137255 0.137255 0.137255
Ke 0.000000 0.000000 0.000000
map_Ka ./texture.png
map_Kd ./texture.png
Traceback (most recent call last):
File "/data/users/yyy/LayoutGPT/run_layoutgpt_2d.py", line 20, in
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
File "/home/yyy/anaconda3/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1825, in from_pretrained
init_kwargs = json.load(tokenizer_config_handle)
OSError: Can't load tokenizer for 'gpt2'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'gpt2' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.
how to fix it?
I downloaded the backbone through the link and cannot proceed with the download due to permission reasons
The error display is as follows:
wget https://penzhanwu2bbs.blob.core.windows.net/data/GLIPv1_Open/models/swin_large_patch4_window12_384_22k.pth -O swin_large_patch4_window12_384_22k.pth
wget : PublicAccessNotPermittedPublic access is not permitted on this storage account. RequestId:54b491d5-a01e-0070-1f81-cb068d000000 Time:2023-08-10T11:57:32.7421018Z
Hi,
After running evaluation scripts, I got similar accuracy to those in the paper except for numerical reasoning accuracy. I got only around 0.24 accuracy after running the glip evaluation.
I haven't checked the evaluation code's details, but may I have a hint about what might be the reason causing this issue?
Hi, I was visualising the bboxes of the NSR-1K dataset, the boxes seems incorrect compared to the original-coco boxes.
counting = read_json("LayoutGPT/dataset/NSR-1K/counting/counting.train.json")
caps = read_json('COCO/annotations/captions_train2017.json')
image_info = caps['images']
def visualize_data(idx):
sample = counting[idx]
img_id = sample['image_id']
for k in image_info:
if k['id'] == img_id:
img_name = k['file_name']
H = k['height']
W = k['width']
print(img_id)
img = Image.open(f'COCO/train2017/{img_name}').convert('RGB')
draw = ImageDraw.Draw(img)
for lst in sample['object_list']:
text = lst[0]
x,y,w,h = lst[1]
x = (x*W)
y = (y*H)
w = (w*W)
h = (h*H)
print(x,y,w,h)
draw.rectangle([(x,y), (x+w, y+h)], outline=(255,0,0))
print(text)
draw.text((x,y-10), text, (0,0,0))
plt.imshow(img)
plt.show()
The boxes for image_id = 45247 in counting.train.json files in [x,y,w,h] format (assumed) are:
[0.75 ,93.37, 273.27, 183.09] and [280.38, 112.05, 202.96, 147.68]
whereas in the original annotations are:
[0.75, 56.71, 273.27, 274.91] and [280.38, 84.75, 202.96, 221.75]
The x-axes matches completely but the y-axes is wrong in almost all the images. Here's an example
I might be doing this wrong, any help is appreciated!!
Red boxes are taken from NSR-1K dataset, Green are from MSCOCO GT
Hi, thanks for the great work. However, I have problems when reproducing the counting accuracy result.
Any hint for what might have caused the problem is helpful!
Hi, there. Thank you for your wonderful work~ I am wondering why we need to process the 3D layout dataset. Since your method is training-free, we only need to sample the furniture from the 3D-FUTURE dataset and place them using the CSS code generated by your method. 😄
I am asking this for ATISS's buggy code of data processing.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.