text2cinemagraph / text2cinemagraph Goto Github PK

View Code? Open in Web Editor NEW

349.0 10.0 43.0 77.71 MB

Text2Cinemagraph: Text-Guided Synthesis of Eulerian Cinemagraphs [SIGGRAPH ASIA 2023]

Home Page: https://text2cinemagraph.github.io/website/

License: MIT License

Python 99.83% Dockerfile 0.17%

animation cinemagraph computer-vision diffusion-models pytorch text-to-cinemagraph image-to-video optical-flow

text2cinemagraph's Issues

Online version?

Is there an online version I could try myself?

Max retries exceeded with url: /openai/clip-vit-large-patch14/resolve/main/tf_model.h5

Hello, because I can't load the model on huggingface online, I need to change the place where clip-vit-large-patch14 is loaded online in the code to load it locally; but I can't find the modified location. Can you help point out this model loading? Well, I'm looking forward to your reply.

How to determine ODISE vocab?

Hi, thanks a lot for your great works.

I'm wondering if we want to generate for different scene, do we need to change ODISE vocab? If so, how can we choose a good set of words?

What kind of hardware configuration is required?

I use nvdia 3080 with 10GB video memory. After running, the process kills itself.

Encountered a problem when executing "conda install -c "nvidia/label/cuda-11.6.1" libcusolver-dev"

Firstly, I created a docker using the command docker run --gpus all -itd --ipc=host --name=guyu --rm -v /gdata/cold1/guyu:/userhome pytorch/pytorch:1.13.1-cuda11.6-cudnn8-devel, and then used the command docker exec -it guyu /bin/bash to enter the docker. After that, I configured following the instructions in README.md. But I encountered a problem when executing conda install -c "nvidia/label/cuda-11.6.1" libcusolver-dev. Here is the error report.

(t2c) root@535ca7f76d3b:/userhome/text2cinemagraph# conda install -c "nvidia/label/cuda-11.6.1" libcusolver-dev
Collecting package metadata (current_repodata.json): done                                                                                                                                                                                      
Solving environment: done                                                                                                                                                                                                                      
                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                               
==> WARNING: A newer version of conda exists. <==                                                                                                                                                                                              
  current version: 22.11.1                                                                                                                                                                                                                     
  latest version: 23.5.2                                                                                                                                                                                                                       
                                                                                                                                                                                                                                               
Please update conda by running                                                                                                                                                                                                                 
                                                                                                                                                                                                                                               
    $ conda update -n base -c defaults conda                                                                                                                                                                                                   
                                                                                                                                                                                                                                               
Or to minimize the number of packages updated during conda update use                                                                                                                                                                          
                                                                                                                                                                                                                                               
     conda install conda=23.5.2                                                                                                                                                                                                                
                      


## Package Plan ##

  environment location: /opt/conda/envs/t2c

  added / updated specs:
    - libcusolver-dev


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libcusolver-dev-11.3.3.112 |       h9f1add7_0        55.0 MB  nvidia/label/cuda-11.6.1
    ------------------------------------------------------------
                                           Total:        55.0 MB

The following NEW packages will be INSTALLED:

  libcusolver-dev    nvidia/label/cuda-11.6.1/linux-64::libcusolver-dev-11.3.3.112-h9f1add7_0 


Proceed ([y]/n)? y


Downloading and Extracting Packages
                                                                                                                                                                                                                                               
class: CondaSSLError
message:
Encountered an SSL error. Most likely a certificate verification issue.

Exception: HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Max retries exceeded with url: /6137df1016c93053c05a6684/6215bb7c03b93eb2b1de2209?response-content-disposition=attachment%3B%20filename%3D%22libcusolver-dev-11.3.3.112-h9f1add7_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27libcusolver-dev-11.3.3.112-h9f1add7_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=600&X-Amz-Date=20230724T083633Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHcaCXVzLWVhc3QtMSJIMEYCIQC6yayOjRAUqP5Axp1ZbVZpmFJvAVu6nNAINJbMY8rISwIhAInfNg6ApcYSh9to9DsA5Wqjsc6YPTBwq9j3WUm3r5X0KrMFCBAQABoMNDU1ODY0MDk4Mzc4IgxWYvMUS44diKr6uXgqkAXYlysqO%2FevcawMaRTJtxdDkWNF7X5%2BLDvkWJU9hg0AYlZJzSgUPezN9mRBx8XfthV23QVrAQGLKaU%2Bcx9jVpcSXn6Bs7k5LLcbokSDSilV3%2BsijMpYAPILkqQxoQZu6AfKqrGs2mFdmJHS8rZ8xkBv4%2FjswzA9hAxwpR1eUZAZLk1nWeira%2B4W9%2F1SRR0oghZpr8YMPdWF18vYqxeHQKzp8LHYW5kV8ANt3MaVvkJMb1hUqAmiDzSZnzzKdTNWzfaumgbTOsPkajrbPrL8nlrYUteDWBVXSBLRujay07vDuPIfc3gOL0awXaHj0%2B5tSwvqtr4GvGQpBVBeIs90yPXKvKKPzBF%2Fau5zLfHkPIBCpNebz8Q%2BSPuGP%2FpKlnFNxnbnCpgSwuXGHKtoHvbaeD7lyHVOKJ759sNZkGh8JHWtRtcU5j4jFtnbgmbJp5xMmAKyyZrD3OhyLokCHC8negRz8R%2BUWzmUNCLqp9mp6r0zG%2FX6zZ1vm8Tqk4HGl2FsPj8WA%2BWAAIVFyXX0OOpifZNiZ9M3SCAXb4yO8o3lvQYgDTXwLRAlRno4dLKYMxDYWeA08o%2FqphBNMOI9%2BWuCQI%2FygYYaE38Nh%2BiPdGKwy2ToVZdiD1dookK%2BIalM31%2BHa7y0t0lvPaoTCixgX7KuVnLKo5BsvmOqWMedVJnHAgCttPcTMhJmpwTFSb6%2B3Qf3CEbCV%2BveC5slq5Z3ewoKMNGEHDzMX7Eus6eclL06fjfjQMtJHUoIhemMJKBZxPh2gnkCYFcvoqCJYZW7ghfsRvxyUd3InEI8vj77czLcmCe29ksvTpJgBTUFfZsQtCTvRhNr4JnMVSiwn%2BtTI9T%2BmwCvf13tN7Fo%2FTnk2CWSiBT8bzDlwvilBjqwARHV11O7SoYUIeRBVjriqPm5MJ0Po8QyB%2Fv9hM1qibDm9KZmuA4Dl0Q5jGGAedSFTDQmu8WzpTWDZB153DIzZ90PHCm8La5rwQW8G7Hua7YQV1GjiJHw6qjw5z%2BfrqWluj1Cfz8zNhZLaQz4QsQ08kwLN6WSx95Mkz%2FmQOYUIV7UC5yUNGPsozkP9bOcdiYMvi1A4dzW0jPWumYgilFFg%2FgD9KZliKkUQ35fiE%2Fd9w2%2B&X-Amz-Credential=ASIAWUI46DZFOYYALWVN%2F20230724%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=822a1e199d29eee1933475f1a4a60d5ee3d0104d9fc68548830fd71f3ca5a943 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:997)')))

kwargs:
{}

Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1118, in __call__
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/conda/cli/main.py", line 69, in main_subshell
    exit_code = do_call(args, p)
  File "/opt/conda/lib/python3.10/site-packages/conda/cli/conda_argparse.py", line 91, in do_call
    return getattr(module, func_name)(args, parser)
  File "/opt/conda/lib/python3.10/site-packages/conda/notices/core.py", line 109, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/conda/cli/main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "/opt/conda/lib/python3.10/site-packages/conda/cli/install.py", line 332, in install
    handle_txn(unlink_link_transaction, prefix, args, newenv)
  File "/opt/conda/lib/python3.10/site-packages/conda/cli/install.py", line 357, in handle_txn
    unlink_link_transaction.download_and_extract()
  File "/opt/conda/lib/python3.10/site-packages/conda/core/link.py", line 204, in download_and_extract
    self._pfe.execute()
  File "/opt/conda/lib/python3.10/site-packages/conda/core/package_cache_data.py", line 805, in execute
    raise CondaMultiError(exceptions)
conda.CondaMultiErrorclass: CondaSSLError
message:
Encountered an SSL error. Most likely a certificate verification issue.

Exception: HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Max retries exceeded with url: /6137df1016c93053c05a6684/6215bb7c03b93eb2b1de2209?response-content-disposition=attachment%3B%20filename%3D%22libcusolver-dev-11.3.3.112-h9f1add7_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27libcusolver-dev-11.3.3.112-h9f1add7_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=600&X-Amz-Date=20230724T083633Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEHcaCXVzLWVhc3QtMSJIMEYCIQC6yayOjRAUqP5Axp1ZbVZpmFJvAVu6nNAINJbMY8rISwIhAInfNg6ApcYSh9to9DsA5Wqjsc6YPTBwq9j3WUm3r5X0KrMFCBAQABoMNDU1ODY0MDk4Mzc4IgxWYvMUS44diKr6uXgqkAXYlysqO%2FevcawMaRTJtxdDkWNF7X5%2BLDvkWJU9hg0AYlZJzSgUPezN9mRBx8XfthV23QVrAQGLKaU%2Bcx9jVpcSXn6Bs7k5LLcbokSDSilV3%2BsijMpYAPILkqQxoQZu6AfKqrGs2mFdmJHS8rZ8xkBv4%2FjswzA9hAxwpR1eUZAZLk1nWeira%2B4W9%2F1SRR0oghZpr8YMPdWF18vYqxeHQKzp8LHYW5kV8ANt3MaVvkJMb1hUqAmiDzSZnzzKdTNWzfaumgbTOsPkajrbPrL8nlrYUteDWBVXSBLRujay07vDuPIfc3gOL0awXaHj0%2B5tSwvqtr4GvGQpBVBeIs90yPXKvKKPzBF%2Fau5zLfHkPIBCpNebz8Q%2BSPuGP%2FpKlnFNxnbnCpgSwuXGHKtoHvbaeD7lyHVOKJ759sNZkGh8JHWtRtcU5j4jFtnbgmbJp5xMmAKyyZrD3OhyLokCHC8negRz8R%2BUWzmUNCLqp9mp6r0zG%2FX6zZ1vm8Tqk4HGl2FsPj8WA%2BWAAIVFyXX0OOpifZNiZ9M3SCAXb4yO8o3lvQYgDTXwLRAlRno4dLKYMxDYWeA08o%2FqphBNMOI9%2BWuCQI%2FygYYaE38Nh%2BiPdGKwy2ToVZdiD1dookK%2BIalM31%2BHa7y0t0lvPaoTCixgX7KuVnLKo5BsvmOqWMedVJnHAgCttPcTMhJmpwTFSb6%2B3Qf3CEbCV%2BveC5slq5Z3ewoKMNGEHDzMX7Eus6eclL06fjfjQMtJHUoIhemMJKBZxPh2gnkCYFcvoqCJYZW7ghfsRvxyUd3InEI8vj77czLcmCe29ksvTpJgBTUFfZsQtCTvRhNr4JnMVSiwn%2BtTI9T%2BmwCvf13tN7Fo%2FTnk2CWSiBT8bzDlwvilBjqwARHV11O7SoYUIeRBVjriqPm5MJ0Po8QyB%2Fv9hM1qibDm9KZmuA4Dl0Q5jGGAedSFTDQmu8WzpTWDZB153DIzZ90PHCm8La5rwQW8G7Hua7YQV1GjiJHw6qjw5z%2BfrqWluj1Cfz8zNhZLaQz4QsQ08kwLN6WSx95Mkz%2FmQOYUIV7UC5yUNGPsozkP9bOcdiYMvi1A4dzW0jPWumYgilFFg%2FgD9KZliKkUQ35fiE%2Fd9w2%2B&X-Amz-Credential=ASIAWUI46DZFOYYALWVN%2F20230724%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=822a1e199d29eee1933475f1a4a60d5ee3d0104d9fc68548830fd71f3ca5a943 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:997)')))

kwargs:
{}

: <exception str() failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/conda", line 13, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/conda/cli/main.py", line 112, in main
    return conda_exception_handler(main, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1418, in conda_exception_handler
    return_value = exception_handler(func, *args, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1121, in __call__
    return self.handle_exception(exc_val, exc_tb)
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1150, in handle_exception
    return self.handle_application_exception(exc_val, exc_tb)
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1164, in handle_application_exception
    self._print_conda_exception(exc_val, exc_tb)
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1168, in _print_conda_exception
    print_conda_exception(exc_val, exc_tb)
  File "/opt/conda/lib/python3.10/site-packages/conda/exceptions.py", line 1095, in print_conda_exception
    stderrlog.error("\n%r\n", exc_val)
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 1506, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 1633, in handle
    if (not self.disabled) and self.filter(record):
  File "/opt/conda/lib/python3.10/logging/__init__.py", line 821, in filter
    result = f.filter(record)
  File "/opt/conda/lib/python3.10/site-packages/conda/gateways/logging.py", line 48, in filter
    record.msg = record.msg % new_args
  File "/opt/conda/lib/python3.10/site-packages/conda/__init__.py", line 107, in __repr__
    errs.append(e.__repr__())
  File "/opt/conda/lib/python3.10/site-packages/conda/__init__.py", line 62, in __repr__
    return f"{self.__class__.__name__}: {self}"
  File "/opt/conda/lib/python3.10/site-packages/conda/__init__.py", line 66, in __str__
    return str(self.message % self._kwargs)
ValueError: unsupported format character 'B' (0x42) at index 289

I haved tried following solutions, but they didn't help:

Change the conda channel to tsinghua.
Use the command conda config --set ssl_verify false to close the ssl verification.
Update the conda using either conda update -n base -c defaults conda or conda install conda=23.5.2.
Follow https://stackoverflow.com/questions/33699577/conda-update-fails-with-ssl-error-certificate-verify-failed?rq=1, use the certification by the command conda config --set ssl_verify /etc/ssl/certs/ca-certificates.crt.

Could you help me with the problem?

SD 1.5?

Would this work with SD1.5? This really looks awesome

Can't load the configuration of '../Tune-A-Video/checkpoints/stable-diffusion-v1-4' at stage s5

Hi, I used Google Colab with V100 (24GB RAM) to run inference_t2c.py file.
I followed the same instructions to setup the environment (using conda to create virtual environment) and downloaded necessary pretrained models. Everything was fine until stage s5 (optical flow part). I got the following error messages:

Traceback (most recent call last):
File "/content/text2cinemagraph/inference_t2c.py", line 149, in
main()
File "/content/text2cinemagraph/inference_t2c.py", line 139, in main
predict_flow(exp_config)
File "/content/text2cinemagraph/test_motion.py", line 79, in predict_flow
model = create_model(opt)
File "/content/text2cinemagraph/models/models.py", line 13, in create_model
model.initialize(opt)
File "/content/text2cinemagraph/models/pix2pixHD_model.py", line 45, in initialize
self.text_encoder = CLIPTextModel.from_pretrained(pretrained_model_path, subfolder="text_encoder")
File "/usr/local/envs/t2c/lib/python3.9/site-packages/transformers/modeling_utils.py", line 2079, in from_pretrained
config, model_kwargs = cls.config_class.from_pretrained(
File "/usr/local/envs/t2c/lib/python3.9/site-packages/transformers/models/clip/configuration_clip.py", line 134, in from_pretrained
config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/usr/local/envs/t2c/lib/python3.9/site-packages/transformers/configuration_utils.py", line 565, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/usr/local/envs/t2c/lib/python3.9/site-packages/transformers/configuration_utils.py", line 641, in _get_config_dict
raise EnvironmentError(
OSError: Can't load the configuration of '../Tune-A-Video/checkpoints/stable-diffusion-v1-4'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '../Tune-A-Video/checkpoints/stable-diffusion-v1-4' is the correct path to a directory containing a config.json file.

Is this a PATH problem or anything (model or config) I have to download as well?
Really appreciate your help!

Endless Loop *.gif

Thank you very much for the interesting code!

I tried it and the result is not an Endless Loop Gif, but a 2 second movie.
Will you provide the code to generate the Endless Loop?

git: 'lfs' is not a git command

Use sudo apt-get install git-lfs to install git-lfs, in case sudo is not allowed, use the following commands,

wget https://github.com/git-lfs/git-lfs/releases/download/v3.2.0/git-lfs-linux-amd64-v3.2.0.tar.gz
tar -xzf git-lfs-linux-amd64-v3.2.0.tar.gz
PATH=$PATH:/<absolute-path>/git-lfs-3.2.0/
git lfs install
git lfs version```

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.