Comments (10)
a change in width and height causes the model to slow down for the first inference on the new dimensions.
it can generate the result in 1 second. If I change the width and height, it will generate the next result in ~10 seconds.
We trace a static computation graph and optimize it to reduce the inference time. The static computation graph is also assuming static inputs shape to enable memory allocation ahead of time. So when finding an input with a new shape, it will trigger a graph compilation, which tasks a round 7 seconds.
We provide an offline compile mode to avoid online compilation costs when we already know all kinds of inputs shape. We can compile graphs for these shapes offline, and then load the compiled results online. Loading a graph only takes less than 1 second.
We also provide a shared graph feature to save more memory and compile time.
Here this the test for offline compile / shared graph: https://github.com/Oneflow-Inc/diffusers/blob/oneflow-fork/tests/test_pipelines_oneflow_graph_load.py
You need to update oneflow diffusers and oneflow to the most recent version.
python3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/master/[PLATFORM]
cd diffusers
git checkout oneflow-fork
git pull origin oneflow-fork
How to load the compiled result of graph
Compile and save graph
- Turn on
pipe.enable_save_graph()
; - Call pipe to generate images, which will trigger compilation and cache the compilation results;
- Call
pipe.save_graph(graph_save_path)
to save the graph, and the cached compilation results will be saved at this time; note that the graph_save_path folder needs to already exist;
The graph cached under the graph cache of the previous pipe is stored under graph_save_path;
Load the graph and use
- Execute
pipe.load_graph(graph_save_path, compile_unet=True, compile_vae=True)
, the previously saved cache will be restored
In this way, the previously saved graph is loaded into the graph cache of the pipe, and when the pipe is called for reasoning later, it will hit the cache of the graph, thus avoiding compilation;
Compile and share between graphs with different input shapes but the same parameters
Just turn on pipe.enable_graph_share_mem()
;
After opening, multiple graphs with different input shapes but the same parameters can be shared:
- Compile pass optimization results;
- Constant folded parameters;
This can save memory and compile time;
In addition, sorting the input shape from large to small to trigger graph compilation can make the memory-sharing effect of the activation part better and further reduce the memory.
from onediff.
Changing the width or height causes a recompilation, and a new feature about this is working in process.
from onediff.
Hey that is great.
Can we get a bit of information about the new feature and what optimizations will it bring towards especially stable diffusion ?
from onediff.
from onediff.
@shangguanshiyuan Hi, is this issue fixed in the new update? if so, how to do that?
from onediff.
Thanks for your attention. This feature has not been released during the testing phase, which will significantly reduce compilation time for multi-shape.
from onediff.
@shangguanshiyuan Great, thanks! any approximation on when it will be released to public?
from onediff.
@strint Thanks a lot for your answer! I am trying your instructions and it gives me the following error while trying to save graph for stable diffusion:
AttributeError: 'VaeGraph' object has no attribute 'enable_save_runtime_state_dict'
from onediff.
AttributeError: 'VaeGraph' object has no attribute 'enable_save_runtime_state_dict'
It's because oneflow has not been updated to the latest version.
You can use this to get the oneflow version:
python3 -m oneflow --doctor
To install the latest oneflow, install nightly:
- Nightly
python3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/master/[PLATFORM]
- All available
[PLATFORM]
:Platform CUDA Driver Version Supported GPUs cu117 >= 450.80.02 GTX 10xx, RTX 20xx, A100, RTX 30xx cu102 >= 440.33 GTX 10xx, RTX 20xx cpu N/A N/A
Here is the full update list
Update oneflow:
- Nightly
python3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/master/[PLATFORM]
- All available
[PLATFORM]
:Platform CUDA Driver Version Supported GPUs cu117 >= 450.80.02 GTX 10xx, RTX 20xx, A100, RTX 30xx cu102 >= 440.33 GTX 10xx, RTX 20xx cpu N/A N/A
Update transformers
Delete the local folder which contains the oneflow fork of transformers, directly use the official transformers
python3 -m pip install transformers>=4.26
Update diffusers
cd diffusers
git checkout oneflow-fork
git pull origin oneflow-fork
python3 -m pip install -e .[oneflow]
After updating oneflow/transformers/diffusers, you can run the test:
python3 diffusers/tests/test_pipelines_oneflow_graph_load.py
from onediff.
looks like it has been resolved, feel free to reopen if not.
from onediff.
Related Issues (20)
- Compiled oneflow model Used more GPU Memory HOT 2
- oneflow update HOT 8
- Does not work with PAG HOT 1
- Once used, Model Speedup cannot be deactivated HOT 2
- Update examples to use onediffx HOT 1
- diffusers0.29动态分辨率切换报错 HOT 1
- sdxl模型,使用A800显卡在batch size等于8时出现core dump HOT 3
- [Feature Request] AuraFlow with onediff support HOT 2
- Lack citation info HOT 2
- Does not work with Tiled Diffusion, makes a new error when importing the nodes together HOT 1
- [Doc]undocumented matrix of dependencies when install HOT 5
- SD1.5的UNet2DConditionModel 额外增加输入模块后使用nexfort接口进行编译报错 HOT 10
- [Bug] compile model with nexfort backend on sm75 device HOT 5
- [Bug] undefined symbol: __nvJitLinkAddData_12_5, version libnvJitLink.so.12 HOT 4
- kolors got poor results when text encoder quantize[Bug] HOT 9
- [Bug] ComfyUI中使用 Model Graph Saver 节点不能保存正确的graph HOT 8
- Add acceleration support for FLUX models HOT 8
- [Bug] unet + Controlnet compile fail HOT 17
- [Bug] Nexfort isn't compatible with PyTorch 2.4.0 HOT 9
- [Bug] onediff-webui-extension ejoom with controlnet HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onediff.