Comments (1)
Additional Debugging
When running the test program with PYTHONMALLOC=debug
environment variable, Python will check the status of the GIL when performing memory allocation and deallocation:
PYTHONMALLOC=debug python sdxl_onediff_test.py
The captured stack trace is as follows:
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x000072fcf0ac7859 in __GI_abort () at abort.c:79
#2 0x00000000004a6ef3 in fatal_error_exit (status=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.10.13/Python/pylifecycle.c:2556
#3 fatal_error (fd=<optimized out>, header=header@entry=1, prefix=prefix@entry=0x6b6a90 <__func__.5.lto_priv.0> "_PyMem_DebugFree", msg=msg@entry=0x627e68 "Python memory allocator called without holding the GIL", status=status@entry=-1) at /usr/local/src/conda/python-3.10.13/Python/pylifecycle.c:2667
#4 0x00000000004a727d in _Py_FatalErrorFunc (func=func@entry=0x6b6a90 <__func__.5.lto_priv.0> "_PyMem_DebugFree", msg=msg@entry=0x627e68 "Python memory allocator called without holding the GIL") at /usr/local/src/conda/python-3.10.13/Python/pylifecycle.c:2753
#5 0x0000000000423122 in _PyMem_DebugCheckGIL (func=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.10.13/Objects/obmalloc.c:2646
#6 0x00000000004dd024 in _PyMem_DebugFree (ctx=0x743040 <_PyMem_Debug+96>, ptr=0x72f37c77f4c0) at /usr/local/src/conda/python-3.10.13/Objects/obmalloc.c:2670
#7 0x000072f57c11946c in ?? () from /opt/conda/lib/python3.10/site-packages/oneflow/_oneflow_internal.cpython-310-x86_64-linux-gnu.so
#8 0x000072f45e27ec0d in std::_Sp_counted_ptr_inplace<oneflow::one::LocalTensor, std::allocator<oneflow::one::LocalTensor>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /opt/conda/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-d0782696.so
#9 0x000072f57c097a1a in oneflow::deleter(DLManagedTensor*) () from /opt/conda/lib/python3.10/site-packages/oneflow/_oneflow_internal.cpython-310-x86_64-linux-gnu.so
#10 0x000072fc54ddb28c in c10::deleteInefficientStdFunctionContext(void*) () from /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so
#11 0x000072fcbf9bb7b6 in c10::StorageImpl::~StorageImpl() () from /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so
#12 0x000072fc54e02ca7 in c10::intrusive_ptr<c10::StorageImpl, c10::detail::intrusive_target_default_null_type<c10::StorageImpl> >::reset_() () from /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so
#13 0x000072fc54dfacb3 in c10::TensorImpl::~TensorImpl() () from /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so
#14 0x000072fc54dfae49 in c10::TensorImpl::~TensorImpl() () from /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so
#15 0x000072fcbfc6b738 in THPVariable_clear(THPVariable*) () from /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so
#16 0x000072fcbfc6bac5 in THPVariable_subclass_dealloc(_object*) () from /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so
#17 0x00000000004dd8e4 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.10.13/Objects/object.c:2295
#18 _Py_DECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.10.13/Include/object.h:500
#19 _Py_XDECREF (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.10.13/Include/object.h:567
#20 list_dealloc (op=0x72f37c78b200) at /usr/local/src/conda/python-3.10.13/Objects/listobject.c:355
#21 0x00000000004f1820 in _Py_Dealloc (op=<error reading variable: dwarf2_find_location_expression: Corrupted DWARF expression.>) at /usr/local/src/conda/python-3.10.13/Objects/object.c:2295
#22 _Py_DECREF (op=<optimized out>) at /usr/local/src/conda/python-3.10.13/Include/object.h:500
#23 _Py_XDECREF (op=<optimized out>) at /usr/local/src/conda/python-3.10.13/Include/object.h:567
#24 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=Frame 0x6c043530, for file /opt/conda/lib/python3.10/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py, line 1100, in __call__ (self=<StableDiffusionXLControlNetPipeline(_internal_dict=<FrozenDict(vae=('diffusers', 'AutoencoderKL'), text_encoder=('onediff', 'MixedOneflowDeployableModule'), text_encoder_2=('onediff', 'MixedOneflowDeployableModule'), tokenizer=('transformers', 'CLIPTokenizer'), tokenizer_2=('transformers', 'CLIPTokenizer'), unet=('onediff', 'MixedOneflowDeployableModule'), controlnet=('onediff', 'MixedOneflowDeployableModule'), scheduler=('diffusers', 'EulerDiscreteScheduler'), force_zeros_for_empty_prompt=True, _name_or_path='stabilityai/stable-diffusion-xl-base-1.0', _FrozenDict__frozen=True) at remote 0x72f8b83427b0>, vae=<AutoencoderKL(_internal_dict=<FrozenDict(in_channels=3, out_channels=3, down_block_types=['DownEncoderBlock2D', 'DownEncoderBlock2D', 'DownEncoderBlock2D', 'DownEncoderBlock2D'], up_block_types=['UpDecoderBlock2D', 'UpDecoderBlock2D', 'UpDecoderBloc...(truncated), throwflag=<optimized out>) at /usr/local/src/conda/python-3.10.13/Python/ceval.c:1882
This reveals that the Python memory allocator is being called without holding the GIL.
from onediff.
Related Issues (20)
- [Doc]undocumented matrix of dependencies when install HOT 5
- SD1.5的UNet2DConditionModel 额外增加输入模块后使用nexfort接口进行编译报错 HOT 10
- [Bug] compile model with nexfort backend on sm75 device HOT 5
- [Bug] undefined symbol: __nvJitLinkAddData_12_5, version libnvJitLink.so.12 HOT 4
- kolors got poor results when text encoder quantize[Bug] HOT 9
- [Bug] ComfyUI中使用 Model Graph Saver 节点不能保存正确的graph HOT 8
- Add acceleration support for FLUX models HOT 8
- [Bug] unet + Controlnet compile fail HOT 17
- [Bug] Nexfort isn't compatible with PyTorch 2.4.0 HOT 9
- [Bug] onediff-webui-extension ejoom with controlnet HOT 1
- [Bug] name 'is_disable_oneflow_backend' is not defined HOT 2
- [Bug] IPA 工作流切换到普通文生图工作流再切到 IPA 工作流生图后异常报错导致 ComfyUI 重启
- [Bug] !!! Exception during processing!!!
- [Bug] comfyui_ipa HOT 2
- 如何使用该节点在comfyui内加速kolors模型
- [Bug] The second API call will fail after switching model from SD to SDXL, vice versa.
- A10(24G) 运行720x1280尺寸的animatediff带controlnet的视频生成,总是OOM
- Each pipe run generates different image (run with the same params and seed)
- about Nexfort compile cache HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from onediff.