Comments (25)
To access RBD, you also need to compile the Ceph RBD python bindings (ideally from the Ceph version you have in use). I think these are sadly not packaged for CentOS 7 in a Python3 version. What we use, is:
CEPH_VERSION=13.2.5
pip install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rados
pip install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
from benji.
I setup a fresh CentOS7 VM and ran through the steps and hit the same issue.
Pretty simple config.
configurationVersion: '1'
databaseEngine: sqlite:////tmp/benji.sqlite
defaultStorage: Ceph-Backup
storages:
- name: Ceph-Backup
storageId: 1
module: rbd
configuration:
path: /Ceph-Backups2
ios:
-
name: file
module: file -
name: rbd
module: rbd
I get the same error when initializing the database, unless I change my storage module to file.
[root@ceph-backups ~]# benji database-init
INFO: $ /bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.io.rbd not found or related import error.
from benji.
Fantastic information and I appreciate it! Is there any distro that is better suited for benji?
from benji.
Is there any distro that is better suited for benji?
I'll leave that for others to answer, since we are also using CentOS 7 as it is so well supported by Ceph 😉 .
But I think that with CentOS 8 sometime later this year (hopefully) which comes with Python 3, things will improve also for us.
from benji.
Running into a issue missing the RBD development headers.
[root@ceph-backups ~]# pip3.6 install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/[email protected]#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to v13.2.5) to /tmp/pip-xm500h70-build
Complete output from command python setup.py egg_info:
/tmp/pip-xm500h70-build/src/pybind/rbd/tmp0bqxxywk/rbd_dummy.c:3:24: fatal error: rbd/librbd.h: No such file or directory
#include <rbd/librbd.h>
^
compilation terminated.
Compile Error: RBD development headers not found
from benji.
Running into a issue missing the RBD development headers.
[root@ceph-backups ~]# pip3.6 install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/[email protected]#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to v13.2.5) to /tmp/pip-xm500h70-build
Complete output from command python setup.py egg_info:
/tmp/pip-xm500h70-build/src/pybind/rbd/tmp0bqxxywk/rbd_dummy.c:3:24: fatal error: rbd/librbd.h: No such file or directory
#include <rbd/librbd.h>
^
compilation terminated.Compile Error: RBD development headers not found
Got around this by installing the following.
yum install librados-devel librbd-devel
Now I am seeing this.
[root@ceph-backups ~]# pip3.6 install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/[email protected]#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to v12.2.11) to /tmp/pip-l35mn9rd-build
Complete output from command python setup.py egg_info:
WARNING: Cython is not installed.
ERROR: Cannot find Cythonized file rbd.c
Cython is indeed installed as well.
[root@ceph-backups tmp]# rpm -qa | grep Cython
Cython-0.19-5.el7.x86_64
from benji.
That version of cython is not a python 3.6 version. It seems there is no Cython for Python 3.6 in EPEL.
We are using a python 3.6 virtualenv in which we install cython, the rbd/rados libraries and Benji itself.
from benji.
Great input, got me a bit further. Decided to try out the python virtualenv but getting hung up still after getting the Ceph RBD python bindings in place.
(benji) [root@ceph-backups ~]# pip install git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd
Collecting git+https://github.com/ceph/[email protected]#subdirectory=src/pybind/rbd
Cloning https://github.com/ceph/ceph (to revision v12.2.11) to /tmp/pip-req-build-3o4cqquh
Requirement already satisfied (use --upgrade to upgrade): rbd==2.0.0 from git+https://github.com/ceph/[email protected]#subdirectory=src/pybind/rbd in /usr/local/benji/lib/python3.6/site-packages
Still results in the same error.
(benji) [root@ceph-backups ~]# /usr/local/benji/bin/benji backup --snapshot-name 2019-03-18T14:38:53 --rbd-hints /tmp/vm-100-disk-0.diff rbd://Test/vm-100-disk-0@2019-03-18T14:38:53 vm-100-disk-0
INFO: $ /usr/local/benji/bin/benji backup --snapshot-name 2019-03-18T14:38:53 --rbd-hints /tmp/vm-100-disk-0.diff rbd://Test/vm-100-disk-0@2019-03-18T14:38:53 vm-100-disk-0
ERROR: Module file benji.io.rbd not found or related import error.
No luck finding benji.io.rbd.
(benji) [root@ceph-backups ~]# find / | grep benji.io.rbd
/usr/local/benji/lib/python3.6/site-packages/benji/io/rbd.py
/usr/local/benji/lib/python3.6/site-packages/benji/schemas/v1/benji.io.rbd.yaml
from benji.
Mhm - the path:
/usr/local/benji/lib/python3.6/site-packages/benji/io/rbd.py
shoult contain the class and load fine.
Here's the complete set of things we do (since we do it via puppet, I had to convert things a bit):
- Add IUS repos for Python 3.6 ( https://ius.io/ )
- Install
gcc
andgit
. - Install
librados-devel
andlibrbd-devel
from Ceph repos. - Install
python36u-devel
,python36u-pip
,python36u-libs
,python36u-setuptools
and create a virtualenv with Python 3.6. - Install
aes-keywrap
,pycryptodome
,Cython
and rados / rbd (as described above) via pip3 in the VirtualEnv. - Finally, install Benji via pip3 with features
compression,readcache
.
That being said - do you get any error if, inside the VirtualEnv, you run python and then type:
import benji.io.rbd
and hit "return"?
from benji.
Mhm - the path:
/usr/local/benji/lib/python3.6/site-packages/benji/io/rbd.py
shoult contain the class and load fine.
Here's the complete set of things we do (since we do it via puppet, I had to convert things a bit):
- Add IUS repos for Python 3.6 ( https://ius.io/ )
- Install
gcc
andgit
.- Install
librados-devel
andlibrbd-devel
from Ceph repos.- Install
python36u-devel
,python36u-pip
,python36u-libs
,python36u-setuptools
and create a virtualenv with Python 3.6.- Install
aes-keywrap
,pycryptodome
,Cython
and rados / rbd (as described above) via pip3 in the VirtualEnv.- Finally, install Benji via pip3 with features
compression,readcache
.That being said - do you get any error if, inside the VirtualEnv, you run python and then type:
import benji.io.rbd
and hit "return"?
Here is that output.
(benji) [root@ceph-backups ~]# import benji.io.rbd
import: unable to open X server `' @ error/import.c/ImportImageCommand/369.
One thing I want to mention. In order to get rados/rdb I had to install python-devel because it was looking for python-config. Shouldn't it be using python3.6-config which was installed.
from benji.
@adambmedent, CentOS 7 is fine, but, as @olifre already mentioned, is missing the Ceph Python modules for Python 3. The Docker images are also CentOS 7 based and compile the necessary modules themselves. Some workarounds are required as you've already noticed. Please have a look at the command sequence in https://github.com/elemental-lf/benji/blob/8446afbad81f36432a9c0dabf0be59f1ed42b238/images/benji/Dockerfile, lines 11 to 31.
If you install Docker (or another compatible container runtime) you could also use the pre-compiled Docker image that we provide. With the right options to docker run
put into a script wrapper you could make it behave like a native CLI program. You wouldn't have to deal with the dependencies then, but you'd have to live with the Ceph version used in the image.
from benji.
@adambmedent, CentOS 7 is fine, but, as @olifre already mentioned, is missing the Ceph Python modules for Python 3. The Docker images are also CentOS 7 based and compile the necessary modules themselves. Some workarounds are required as you've already noticed. Please have a look at the command sequence in https://github.com/elemental-lf/benji/blob/8446afbad81f36432a9c0dabf0be59f1ed42b238/images/benji/Dockerfile, lines 11 to 31.
If you install Docker (or another compatible container runtime) you could also use the pre-compiled Docker image that we provide. With the right options todocker run
put into a script wrapper you could make it behave like a native CLI program. You wouldn't have to deal with the dependencies then, but you'd have to live with the Ceph version used in the image.
I think this might have answered my questions.
RUN python3.6 -m venv $VENV_DIR &&
ln -s /usr/bin/python3.6-config $VENV_DIR/bin/python-config &&
ln -s /usr/bin/python3.6m-x86_64-config $VENV_DIR/bin/python3.6m-x86_64-config &&
. $VENV_DIR/bin/activate &&
pip install --upgrade 'pip==10.0.*' &&
sed -i "s|self.run_command(['clone', '-q', url, dest])|self.run_command(['clone', '--depth=1', '--branch', rev if rev else 'master', '-q', url, dest])|" $(python -c 'import site; print(site.getsitepackages()[0])')/pip/_internal/vcs/git.py
So I shouldn't install python-devel to get python-config. I should create a symbolic link. Thats gotta be my issue. I am going to start fresh right now and see what I end up with.
I am not overly familiar with docker so I am hoping to just run our standard C7 load because we are C7 shop.
from benji.
The symbolic links got me further and I am no longer getting the same error. I now have benji.io.rbd.
(benji) [root@linux151 benji]# find / | grep benji.io.rbd
/usr/lib/python3.6/site-packages/benji/io/benji.io.rbd
(benji) [root@linux151 bin]# benji database-init
INFO: $ /usr/local/benji/bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.
from benji.
@adambmedent I just noticed: You can skip the sed
hack, it should work without. This hack is just so that the Ceph repository gets cloned with --depth=1
because it's quite large and we're only interested in the most recent revision. This reduces build time.
from benji.
Sorry, I overlooked this one:
(benji) [root@linux151 bin]# benji database-init
INFO: $ /usr/local/benji/bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.
Could you please run /usr/local/benji/bin/benji --log-level DEBUG database-init
and report the output back here? And while you're at it could you try to import Ceph's rbd
manually? Something like python3.6 -c 'import rbd'
and see what is says... No output would mean that it could be imported correctly.
I suspect that the problem is that you either need to (re)create the virtual environment with --system-site-packages
or you need to install Ceph's rados
and rbd
modules into the virtual environment instead of globally. I would suggest the latter.
from benji.
Ok I started with a brand new CentOS7 VM and did the following steps to a T.
- yum install -y git gcc
- Install ceph repo package
- yum install -y deltarpm tzdata epel-release https://centos7.iuscommunity.org/ius-release.rpm
- yum install -y python36u-devel python36u-pip python36u-libs python36u-setuptools librados-devel librbd-devel
- python3.6 -m venv /usr/local/benji
- VENV_DIR=/usr/local/benji
- ln -s /usr/bin/python3.6-config $VENV_DIR/bin/python-config
- ln -s /usr/bin/python3.6m-x86_64-config $VENV_DIR/bin/python3.6m-x86_64-config
- . $VENV_DIR/bin/activate
- pip install --upgrade 'pip==10.0.*'
- pip install cython
- CEPH_VERSION=12.2.11
- pip install "git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rados"
- pip install "git+https://github.com/ceph/ceph@v$CEPH_VERSION#subdirectory=src/pybind/rbd"
- pip install benji[compression,readcache,s3,b2]
There were no errors at all and it looked to be successful, but I am still hitting the same exact issue.
(benji) [root@linux151 ~]# benji database-init
INFO: $ /usr/local/benji/bin/benji database-init
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.
Here is the database-init debug output.
(benji) [root@linux151 ~]# /usr/local/benji/bin/benji --log-level DEBUG database-init
INFO: $ /usr/local/benji/bin/benji --log-level DEBUG database-init
DEBUG: commands.database_init(**{})
DEBUG: Using block hash BLAKE2b with kwargs {'digest_bits': 256}.
DEBUG: Resolved schema for benji.io.base-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}.
DEBUG: Resolved schema for benji.io.file-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}.
DEBUG: Resolved schema for benji.io.base-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}}}}.
DEBUG: Resolved schema for benji.io.rbd-v1: {'configuration': {'type': 'dict', 'nullable': True, 'required': True, 'schema': {'simultaneousWrites': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'simultaneousReads': {'type': 'integer', 'empty': False, 'min': 1, 'default': 3}, 'cephConfigFile': {'type': 'string', 'empty': False, 'default': '/etc/ceph/ceph.conf'}, 'clientIdentifier': {'type': 'string', 'empty': False, 'default': 'admin'}, 'newImageFeatures': {'type': 'list', 'empty': False, 'default': ['RBD_FEATURE_LAYERING'], 'schema': {'type': 'string', 'regex': '^RBD_FEATURE_.*'}}}, 'empty': False}}.
DEBUG: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.
Traceback (most recent call last):
File "/usr/local/benji/lib64/python3.6/site-packages/benji/factory.py", line 50, in _import_modules
module = importlib.import_module('{}.{}.{}'.format(package, cls._MODULE, module))
File "/usr/lib64/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'benji.storage.rbd'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/benji/lib64/python3.6/site-packages/benji/scripts/benji.py", line 338, in main
func(**func_args)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/commands.py", line 492, in database_init
benji_obj = Benji(self.config, init_database=True)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/benji.py", line 50, in init
StorageFactory.initialize(self.config)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/factory.py", line 74, in initialize
cls._import_modules(config, storages)
File "/usr/local/benji/lib64/python3.6/site-packages/benji/factory.py", line 53, in _import_modules
package, cls._MODULE, module))
benji.exception.ConfigurationError: Module file benji.storage.rbd not found or related import error.
ERROR: An exception of type ConfigurationError occurred: Module file benji.storage.rbd not found or related import error.
(benji) [root@linux151 ~]# python3.6 -c 'import rbd'
(benji) [root@linux151 ~]#
I was in the python virtual enviroment when I installed the rados and rbd modules.
from benji.
My ceph cluster is 12.2.11, so I figured I would stick with this version on my dedicated benji VM. I am starting to wonder if that is my issue as well? Maybe I need to move to the newer version for the benji VM?
from benji.
My ceph cluster is 12.2.11, so I figured I would stick with this version on my dedicated benji VM. I am starting to wonder if that is my issue as well? Maybe I need to move to the newer version for the benji VM?
Hmm, I've think I've also tested with Luminous last year and Benji only uses very basic calls to the RBD API which have existed for years.
from benji.
My ceph cluster is 12.2.11, so I figured I would stick with this version on my dedicated benji VM. I am starting to wonder if that is my issue as well? Maybe I need to move to the newer version for the benji VM?
Hmm, I've think I've also tested with Luminous last year and Benji only uses very basic calls to the RBD API which have existed for years.
You write:I was in the python virtual enviroment when I installed the rados and rbd modules.
But you also have written this:
Not quite sure what you mean here?
from benji.
Sorry, I've deleted this as I was on the wrong track, but I pressed the comment button by accident, please ignore it.
But I just found this in your output:
ModuleNotFoundError: No module named 'benji.storage.rbd'
And this matches the fact the you can load Ceph's rbd
module fine. So why is Python not able to find benji.storage.rbd
?
from benji.
Sorry, I've deleted this as I was on the wrong track, but I pressed the comment button by accident, please ignore it.
But I just found this in your output:ModuleNotFoundError: No module named 'benji.storage.rbd'
And this matches the fact the you can load Ceph's
rbd
module fine. So why is Python not able to findbenji.storage.rbd
?
It doesn't seem to exist any were. Do you happen to know what provides this module?
(benji) [root@linux151 ~]# find / | grep benji.storage.rbd
(benji) [root@linux151 ~]#
from benji.
First of all your grep
is wrong as the dots translate to directories in Python. But I found the issue. I didn't see the forest for the trees. You're actually trying to load a module that doesn't exist. In your configuration:
name: Ceph-Backup
storageId: 1
module: rbd # Change this to file
configuration:
path: /Ceph-Backups2
Storage modules are file
, s3
or b2
. IO modules are file
and rbd
.
from benji.
First of all your grep is wrong as the dots translate to directories in Python.
Actually I was wrong, the dots are okay as they match any character.
from benji.
Yep, I am in action now.
When I was hitting the issue of missing "benji.io.rbd" I started messing around with that conf file and totally forgot. Once I changed it back, benji is now working. I appreciate the assistance and I will let you guys know how things go!
from benji.
That's great to hear! Please report back any issues you encounter. And if you have any feature ideas, I'd be interested to hear them, too.
from benji.
Related Issues (20)
- benji database-init should be called first time running the container HOT 1
- Restored data does not always match original (Ceph bug) HOT 21
- Problems with snaptrims being pended - Work around HOT 3
- Memory error on benji rm or benji enforce HOT 2
- benji-backup-pvc is no longer working since upgrade HOT 3
- IO scheme k3s is undefined HOT 2
- openstack cinder-backup integration HOT 2
- MySQL compatability problem HOT 1
- benji with celery ? HOT 1
- Publish to a Helm repository in order to support Terraform deployments HOT 2
- Metrics scaling HOT 5
- Coordination with Velero HOT 3
- How to use with rook-ceph ? HOT 2
- Python 3.10 suport HOT 1
- NBD server losing connection to PostgreSQL HOT 4
- nbdserver does not start with python 3.10
- Unable to install benji to Python 3.11 venv in Proxmox 8 (Debian 12 (bookworm)) HOT 4
- NBD export fail "nbd0: truncated" to mount with block size other than 512
- Bug: high ram usage during `benji rm`
- Some pool names are not supported
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benji.