Comments (9)
WendaDeng
commented on June 11, 2024
2
Hi, @mwydmuch, thanks for replying! Your guess is right, We are using Qemu to run ARM64 docker on x86 machine. We also try to build ViZDoom on ARM64 machine directly, but encounter the same problem. Below is the log output:
[ 36%] Linking C static library libdumb.a
[ 36%] Built target dumb
[ 36%] Generating sc_man_scanner.h
[ 37%] Generating xlat_parser.c, xlat_parser.h
Segmentation fault (core dumped)
src/vizdoom/src/CMakeFiles/vizdoom.dir/build.make:79: recipe for target 'src/vizdoom/src/xlat_parser.c' failed
make[2]: *** [src/vizdoom/src/xlat_parser.c] Error 139
make[2]: *** Deleting file 'src/vizdoom/src/xlat_parser.c'
CMakeFiles/Makefile2:712: recipe for target 'src/vizdoom/src/CMakeFiles/vizdoom.dir/all' failed
make[1]: *** [src/vizdoom/src/CMakeFiles/vizdoom.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 37%] Linking CXX shared library bin/libvizdoom.so
[ 37%] Built target libvizdoom_shared
[ 38%] Linking CXX static library bin/libvizdoom.a
[ 38%] Built target libvizdoom_static
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2
Installation failed, you may be missing some dependencies.
Please check https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md for details
Traceback (most recent call last):
File "/home/smart/ViZDoom/setup.py", line 158, in <module>
setup(
File "/home/smart/miniconda3/envs/game/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
return distutils.core.setup(**attrs)
File "/home/smart/miniconda3/envs/game/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/home/smart/miniconda3/envs/game/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/smart/miniconda3/envs/game/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/smart/miniconda3/envs/game/lib/python3.10/site-packages/setuptools/dist.py", line 1208, in run_command
super().run_command(command)
File "/home/smart/miniconda3/envs/game/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/smart/ViZDoom/setup.py", line 149, in run
subprocess.check_call(['make', '-j', str(cpu_cores)])
File "/home/smart/miniconda3/envs/game/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '191']' returned non-zero exit status 2.
Without information about Qemu, but other messages are the same.
We also try to use pip install vizdoom command directly on ARM64 machine, but got following output log:
[ 36%] Linking C static library libdumb.a
[ 36%] Built target dumb
[ 37%] Generating xlat_parser.c, xlat_parser.h
[ 37%] Generating sc_man_scanner.h
Segmentation fault (core dumped)
src/vizdoom/src/CMakeFiles/vizdoom.dir/build.make:79: recipe for target 'src/vizdoom/src/xlat_parser.c' failed
make[2]: *** [src/vizdoom/src/xlat_parser.c] Error 139
make[2]: *** Deleting file 'src/vizdoom/src/xlat_parser.c'
CMakeFiles/Makefile2:712: recipe for target 'src/vizdoom/src/CMakeFiles/vizdoom.dir/all' failed
make[1]: *** [src/vizdoom/src/CMakeFiles/vizdoom.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 38%] Linking CXX static library bin/libvizdoom.a
[ 38%] Linking CXX shared library bin/libvizdoom.so
[ 38%] Built target libvizdoom_static
[ 38%] Built target libvizdoom_shared
Makefile:90: recipe for target 'all' failed
make: *** [all] Error 2
Installation failed, you may be missing some dependencies.
Please check https://github.com/mwydmuch/ViZDoom/blob/master/doc/Building.md for details
Traceback (most recent call last):
File "/home/smart/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/home/smart/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/smart/.local/lib/python3.6/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 262, in build_wheel
metadata_directory)
File "/tmp/pip-build-env-lnjm524l/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 231, in build_wheel
wheel_directory, config_settings)
File "/tmp/pip-build-env-lnjm524l/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 215, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-lnjm524l/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 268, in run_setup
self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-lnjm524l/overlay/lib/python3.6/site-packages/setuptools/build_meta.py", line 158, in run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 178, in <module>
keywords=['vizdoom', 'doom', 'ai', 'deep learning', 'reinforcement learning', 'research']
File "/tmp/pip-build-env-lnjm524l/overlay/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-lnjm524l/overlay/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "setup.py", line 137, in run
subprocess.check_call(['make', '-j', str(cpu_cores)])
File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '191']' returned non-zero exit status 2.
----------------------------------------
ERROR: Failed building wheel for vizdoom
Failed to build vizdoom
ERROR: Could not build wheels for vizdoom, which is required to install pyproject.toml-based projects
These error messages are the same with build from source code.
We found that the error is related to using the lemon tool to generate xlat_parser code. So, we tried to build the source code using debug mode and see what is wrong with the lemon tool. We encounter some problems in the building process, but we finally build the project suceeseefully and did not encounter the error related to the lemon. That is kind of embarrassing.
Problems we encountered in the building process are following:
1
/root/ViZDoom/src/vizdoom/src/d_main.cpp: In function 'void D_DoomInit()':
/root/ViZDoom/src/vizdoom/src/d_main.cpp:2229:15: error: '_FPU_EXTENDED' was not declared in this scope
2229 | cw = (cw & ~_FPU_EXTENDED) | _FPU_DOUBLE;
| ^~~~~~~~~~~~~
/root/ViZDoom/src/vizdoom/src/d_main.cpp:2229:32: error: '_FPU_DOUBLE' was not declared in this scope
2229 | cw = (cw & ~_FPU_EXTENDED) | _FPU_DOUBLE;
| ^~~~~~~~~~~
We fixed it by commenting out the cmake command related to FPU_CONTROL_DIR in src/vizdoom/src/CMakeLists.txt file.
- find_path( FPU_CONTROL_DIR fpu_control.h )
- if( FPU_CONTROL_DIR )
- include_directories( ${FPU_CONTROL_DIR} )
- add_definitions( -DHAVE_FPU_CONTROL )
- endif( FPU_CONTROL_DIR )
+ # find_path( FPU_CONTROL_DIR fpu_control.h )
+ # if( FPU_CONTROL_DIR )
+ # include_directories( ${FPU_CONTROL_DIR} )
+ # add_definitions( -DHAVE_FPU_CONTROL )
+ # endif( FPU_CONTROL_DIR )
2
/root/ViZDoom/src/vizdoom/src/p_spec.cpp:1282:4: error: narrowing conversion of '-1' from 'int' to 'char' [-Wnarrowing]
We fixed it by changing the type of hexenScrollies[24][2] from static const char to static const signed char in src/vizdoom/src/p_spec.cpp file.
- static const char hexenScrollies[24][2] =
+ static const signed char hexenScrollies[24][2] =
3
[100%] Built target pk3
[100%] Assembling pip package in /root/ViZDoom/build/bin/python3.10/pip_package
Required ViZDoom's resources do not exist. Aborting.
make[2]: *** [src/lib_python/CMakeFiles/python_pip_package.dir/build.make:71: src/lib_python/CMakeFiles/python_pip_package] Error 3
make[1]: *** [CMakeFiles/Makefile2:767: src/lib_python/CMakeFiles/python_pip_package.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
We fixed it by copying vizdoom binary file from vizdoomd binary file under bin directory. And we also added one command cp ${BOTS_PATH} ${SCENARIOS_DEST_DIR} right after the command mkdir -p ${SCENARIOS_DEST_DIR} in scripts/assemble_pip_package.sh file.
mkdir -p ${SCENARIOS_DEST_DIR}
+cp ${BOTS_PATH} ${SCENARIOS_DEST_DIR}
These fixes work and we can import vizdoom successfully in python program. But those fixes are too complex and not elegant. So we hope you can help us to find a better solution, thanks!
from vizdoom.
mwydmuch
commented on June 11, 2024
Hi @WendaDeng! First, please note that we do not officially support ARM64 architecture at the moment (Apple Silicon is the exeption). So there is some probability that ViZDoom may not compile/work on ARM64, since it was not tested by us. But it seems not to be an issue here.
I don't understand why you can just use pip install on your ARM machine. But if I understand correctly, this process of using the X86_64 machine to build the ARM64 version in Docker is using Qemu, which is a machine emulator and virtualizer, and in your log, we can see that it crashes during compilation (qemu: uncaught target signal 11 (Segmentation fault) - core dumped).
This might be entirely Qemu's bug we do not control. It seems that there is plenty of similar issues for other packages/software that people try to build this way. So at the moment, I'm not sure what can be done about that.
from vizdoom.
mwydmuch
commented on June 11, 2024
Thank you @WendaDeng for the detailed explanation and investigation!
-
I'm not sure if this is the best way of fixing this, so I need to investigate. Could you provide your docker image, so I could also try to build it?
-
This was already reported in the old issue #336. Also, I checked other improved ZDoom forks, and they contain this fix. So without a doubt, it should be changed in ViZDoom too.
-
This is because you compiled in DEBUG mode. In this mode, exe name is changed to vizdoomd instead of just vizdoom. So this doesn't really require fixing.
from vizdoom.
WendaDeng
commented on June 11, 2024
Thank you @mwydmuch for providing related information. Yes, we can our provide docker image. But right now our public docker repository encounter a problem, we can not pull the public docker without login docker repository. We have reported this problem to our docker service provider and waiting for their reply. So please wait for a while. We will provide our docker image once this problem is fixed. Thanks!
from vizdoom.
WendaDeng
commented on June 11, 2024
Hi @mwydmuch, I'm glad to tell you that we have fixed the docker repository problem. Our docker image is bj-sm-ai-registry.cn-beijing.cr.aliyuncs.com/smai-public/miniconda:ubuntu20.04-cpu. We build it from official ubuntu:20.04 docker image. And we change apt and pip source cause we are in China. We have backed up the original apt source file to /etc/apt/sources.list.bk so you can reuse it easily. We also install some basic softwares in the image, such as cmake vim git g++ python3 python3-pip python3-distutils python3-dev python-is-python3, etc. We install miniconda(version 22.11.1-1) in the image and the default python version is 3.10.8. To use it in x86 machine, you may need to pull multiarch/qemu-user-static:latest docker image first.
from vizdoom.
WendaDeng
commented on June 11, 2024
Hey @mwydmuch, is the docker image we provided available? Is there any progress?
from vizdoom.
mwydmuch
commented on June 11, 2024
Hi @WendaDeng, Iām sorry, due to the ICML deadline in January and the vacations Iām on right now I haven't yet delve into it enough. But I remember about it, and should definitely work on the last weak of February. We aim to provide full ARM support in the feature so this is a priority for me.
from vizdoom.
mwydmuch
commented on June 11, 2024
Hi @WendaDeng, I want to thank you for all the effort you put into this issue. It took some time, but I fixed all the issues, in the PR referenced above, soon it will be merged and we will do the next official release.
from vizdoom.
WendaDeng
commented on June 11, 2024
Hi @mwydmuch, I want to thank you for your effort and help too. I'm going to try the new feature and tell you if it work for me. Thanks again.
from vizdoom.
Related Issues (20)
- Create documentation in farama.org style
- Update to the newest Gymnasium API
- Create a git pre-commit hook with black, flake8 and isort for gym_wrapper HOT 3
- Create a GitHub workflow with black, flake8 and isort checks HOT 1
- Add docstrings to pybind11 module
- Fix SetuptoolsDeprecationWarning in setup.py
- Cannot pickle vizdoom.vizdoom.GameState in python HOT 1
- Curious why can't I install vizdoom with conda deps?
- from vizdoom import gymnasium_wrapper doesn't work HOT 2
- Reset error in deadly corridor: screen buffer HOT 7
- Occasional start-up crash when using DirectDraw on Windows 11 HOT 4
- ViZDoom's Development Roadmap
- Checklist for Maturity
- Cannot install via pip HOT 6
- Can I use VizDoom in other FPS games? HOT 3
- First Time User Encountering Pip Install Error M1 HOT 6
- Bug for VizdoomEnv HOT 3
- ATTACK command not working properly, The SELECTED_WEAPON_AMMO variable does not change when pressed HOT 9
- ViZDoom for behavioral modelling; stabilizing time-diff between states HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vizdoom.