cvlab-columbia / pix2gestalt Goto Github PK

View Code? Open in Web Editor NEW

102.0 8.0 7.0 14.65 MB

Code for the paper "pix2gestalt: Amodal Segmentation by Synthesizing Wholes" (CVPR 2024)

Home Page: https://gestalt.cs.columbia.edu/

License: Other

Python 100.00%

3d amodal-instance-segmentation amodal-perception cvpr2024 stable-diffusion zero-shot recognition

pix2gestalt's People

Contributors

Stargazers

Watchers

Forkers

egeozguroglu sorokinvld keyman9848 suweenl marenan thanhpham1987 yaojin17

pix2gestalt's Issues

Want to download the source code

We are doing agricultural research, the crop will be more folded when growing in the field, so we need to use algorithms that can segment the shaded part, I noticed your project, and would like to take the source code and try it out to see if it can be applied to our agricultural research!

Demo or Inference code

May I ask how we can demo or do inference on custom image (meaning our own image)?

Also, I wish to ask what is computational burden for running inference of the model on GPU.

config.json file missing from HF

Hi there!

When will you be releasing the config.json file?

Questions during Run Gradio demo for amodal completion and segmentation

When I run the command python app.py, it gives me the error

Loading model from ./ckpt/epoch=000005.ckpt
Global Step: 3200
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py:101: UserWarning: Failed to download model from HuggingFaceCompatibleDownloader downloader. Trying to download from HuggingFaceCompatibleDownloader downloader.
  warnings.warn(
/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py:107: UserWarning: Failed to download model from HuggingFaceCompatibleDownloader downloader. No fallback downloader available.
  warnings.warn(
Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f18e4694070>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Carve/tracer_b7/resolve/d8a8fd9e7b3fa0d2f1506fe7242966b34381e9c5/tracer_b7.pth (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f18e4694070>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 170, in download_model_base
    r = requests.get(hugging_face_url, stream=True)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 725, in send
    history = [resp for resp in gen]
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 725, in <listcomp>
    history = [resp for resp in gen]
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Carve/tracer_b7/resolve/d8a8fd9e7b3fa0d2f1506fe7242966b34381e9c5/tracer_b7.pth (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f18e4694070>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 98, in download_model
    return self.download_model_base(file_name)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 190, in download_model_base
    raise ConnectionError(
ConnectionError: Exception caught when downloading model! Model name: tracer_b7.pth. Exception: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Carve/tracer_b7/resolve/d8a8fd9e7b3fa0d2f1506fe7242966b34381e9c5/tracer_b7.pth (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f18e4694070>: Failed to establish a new connection: [Errno 101] Network is unreachable')).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connection.py", line 616, in connect
    self.sock = sock = self._new_conn()
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f18f0108c40>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Carve/tracer_b7/resolve/d8a8fd9e7b3fa0d2f1506fe7242966b34381e9c5/tracer_b7.pth (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f18f0108c40>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 170, in download_model_base
    r = requests.get(hugging_face_url, stream=True)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Carve/tracer_b7/resolve/d8a8fd9e7b3fa0d2f1506fe7242966b34381e9c5/tracer_b7.pth (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f18f0108c40>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chenhx/software/pix2gestalt/pix2gestalt/app.py", line 307, in <module>
    fire.Fire(run_demo)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/chenhx/software/pix2gestalt/pix2gestalt/app.py", line 208, in run_demo
    interface = create_carvekit_interface()
  File "/home/chenhx/software/pix2gestalt/pix2gestalt/ldm/util.py", line 48, in create_carvekit_interface
    interface = HiInterface(object_type="object",
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/api/high.py", line 56, in __init__
    self.u2net = TracerUniversalB7(
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/ml/wrap/tracer_b7.py", line 50, in __init__
    model_path = tracer_b7_pretrained()
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/ml/files/models_loc.py", line 53, in tracer_b7_pretrained
    return downloader("tracer_b7.pth")
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 118, in __call__
    return self.download_model(file_name)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 105, in download_model
    return self.fallback_downloader.download_model(file_name)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 111, in download_model
    raise e
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 98, in download_model
    return self.download_model_base(file_name)
  File "/home/chenhx/miniconda3/envs/pix2gestalt/lib/python3.9/site-packages/carvekit/utils/download_models.py", line 190, in download_model_base
    raise ConnectionError(
ConnectionError: Exception caught when downloading model! Model name: tracer_b7.pth. Exception: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /Carve/tracer_b7/resolve/d8a8fd9e7b3fa0d2f1506fe7242966b34381e9c5/tracer_b7.pth (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f18f0108c40>: Failed to establish a new connection: [Errno 101] Network is unreachable')).

When will the code released

Dear author.

Thank you for your awesome work, and congratulations on your work being accepted by CVPR 2024.

I wonder when the entire code will be released, I am very interested in your work and hope to replicate it.

Thank you!

How long does it take to train each epoch?

Hi, I'm very impressed with your solid work, while I feel curious that how much time is needed to train each epoch and how many epochs may be enough for a competent model. Looking forward to your reply, thanks.

Testing script for amodal segmentation on COCOA

Thank you for the interesting work!
I would like to ask if you have the script to generate the result in Table 1 in the paper for amodal segmentation results.

Question of Demo or Inference in the app.py

Hello, I have some doubts.

Question about the RGB / RGBA output format

Thanks for your excellent work and it indeed inspired me a lot! I am wondering is it possible to generate completed images in RGBA format for easier segmentation?

Question of the app.py

Hi, thanks for your great work.
My question is, in your paper you add a text prompt (such as bench, person, etc), however, in your app.py, no where to find the text prompt. The only prompt is the click prompt. Did i misunderstood your paper?

Thank you.

How can I change the port in app.py?

I would like to know how can I change the webui port in app.py, I can not find it.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.