Comments (6)
Hi everyone,
When two or more clients request rbd-target-api at the same time, the configuration file gateway.conf will be modified randomly. When used at scale, performance is slow, will the community support concurrency?
Sorry for late.
What do you mean modified randomly ? BTW, have you seen any issue ? Currently when changing the gateway.conf
object it will acquire the exclusive lock first from Rados. And only the auth gateway node could change the corresponding disk config.
from ceph-iscsi.
Yes, this exclusive lock cannot lock concurrent requests.
We ran into a problem:
-
When concurrency with different rbd-target-api
Both rbd-target-apis are executed to step 3. At this time, rbd-target-api-1 gets the lock, writes gateway.conf to rados successfully, and releases the lock. rbd-target-api-2 also modifies the configuration file in memory at this time, gets the lock, and writes gateway.conf will overwrite the gateway.conf of rbd-target-api-1. -
When the same rbd-target-api is used concurrently
There is also this problem
I think the cause of the problem:
- gateway.conf is to read and write the entire rados object, and the granularity is too large.
- rbd-target-api is not a distributed service.
from ceph-iscsi.
Currently the sequence is:
1, exclusive lock
2, read gateway.conf
object to tmp_config
3, update the tmp_config in memory
4, store tmp_config to gateway.conf
5, exclusive unlock
So for the step 3 in your picture, the exclusive lock should have already been acquired.
And also for each sections, such as for:
o- iscsi-targets ............................................................. [Targets: 1]
o- iqn.2003-01.com.redhat.iscsi-gw:ceph-gw1 ................... [Auth: CHAP, Gateways: 2]
| o- disks ................................................................... [Disks: 1]
| | o- rbd/disk_1 .............................................. [Owner: rh7-gw2, Lun: 0]
| o- gateways ..................................................... [Up: 2/2, Portals: 2]
| | o- rh7-gw1 .................................................... [192.168.122.69 (UP)]
| | o- rh7-gw2 .................................................... [192.168.122.14 (UP)]
o- host-groups ........................................................... [Groups : 0]
o- hosts ................................................ [Auth: ACL_ENABLED, Hosts: 1]
We can see that it's auth is Gateways: 2
, and the ceph-iscsi will only allow the auth gateway 2 to update this secions in gateway.conf
, so there shouldn't have any conflict of it, or it's buggy in the corresponding code.
from ceph-iscsi.
Yes, we can test it. Similar configuration file errors often occur when concurrently accessing different nodes rbd-target-api.
Jun 1 08:00:44 node51 rbd-target-api[2744]: KeyError: u'rbd/disk226'
The following are two scripts that simulate concurrent operation of rbd-target-api services on different nodes and add luns to the same target. (It is very easy to reproduce the problem)
access rbd-target-api-1
for i in `seq 1 100`;
do
curl --insecure --user admin:admin -d mode=create -d create_image=true -d pool=rbd -d size=1T -X PUT http://192.168.122.52:5000/api/disk/rbd/disk$i ;
curl --insecure --user admin:admin -d disk=rbd/disk$i -X PUT http://192.168.122.52:5000/api/targetlun/iqn.2003-01.com.redhat.iscsi-gw:ceph-gw1;
curl --insecure --user admin:admin -d disk=rbd/disk$i -X PUT http://192.168.122.52:5000/api/clientlun/iqn.2003-01.com.redhat.iscsi-gw:ceph-gw1/iqn.2022-05.com.xstor.client0005;
done
access rbd-target-api-2
for i in `seq 101 200`;
do
curl --insecure --user admin:admin -d mode=create -d create_image=true -d pool=rbd -d size=1T -X PUT http://192.168.122.53:5000/api/disk/rbd/disk$i ;
curl --insecure --user admin:admin -d disk=rbd/disk$i -X PUT http://192.168.122.53:5000/api/targetlun/iqn.2003-01.com.redhat.iscsi-gw:ceph-gw1;
curl --insecure --user admin:admin -d disk=rbd/disk$i -X PUT http://192.168.122.53:5000/api/clientlun/iqn.2003-01.com.redhat.iscsi-gw:ceph-gw1/iqn.2022-05.com.xstor.client0005;
done
from ceph-iscsi.
Cool, so it's a bug IMO.
Recently I am busy with cephfs project, since you can reproduce it and if you'd like please raise on PR to fix it.
from ceph-iscsi.
I think this problem is very difficult to fix, it involves design issues and changes will be huge.
from ceph-iscsi.
Related Issues (20)
- rbd-target-api / Unable to decode password for iqn.1998-01.com ,,, HOT 19
- Help! The rbd-target-api service of iSCSI gw cannot be started HOT 2
- ceph-iscsi / tcmu-runner bad pefromance with vmware esxi HOT 1
- I can't get return from method _add_dev_to_lio_user_rbd in ARM environment, it seems to be hang HOT 1
- Unable to remove / disassociate a disk from initiator HOT 1
- rbd-target-api will not start on Ubuntu Impish and later
- Failed to deploy iscsi service HOT 1
- Is ceph iscsi still being maintained?Or has it been abandoned? HOT 5
- gwcli error: REST API failure, code: 500
- About using cephadm to deploy iSCSI Gateway
- Cannot remove a non-existent gatway by confirm=true HOT 1
- rbd-target-api fails to start : error : Failed to run ceph osd blocklist ls HOT 1
- rbd-target-api stuck, tcmu-runner crash / iSCSI Login negotiation failed
- HELP:Failed : disk create/update failed on ceph-2. Unhandled exception: __init__() got an unexpected keyword argument 'control'
- HELP: Failed : disk create/update failed on ceph-2. LUN allocation failure
- rbd-target-api: Disk limit 256 HOT 4
- rtslib_fb: `db_root: cannot be changed because it's in use`
- PSA: Purging ceph-iscsi on Debian rendered system borked
- rbd-target-api bug in centos9 with default python 3.9.18 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ceph-iscsi.