tendrl / gluster-integration Goto Github PK
View Code? Open in Web Editor NEWExtracts all data from a Gluster cluster for consumption by Tendrl
License: GNU Lesser General Public License v2.1
Extracts all data from a Gluster cluster for consumption by Tendrl
License: GNU Lesser General Public License v2.1
For unit test i have imported a module called Eventer (from gluster_bridge.manager.eventer import Eventer). When i try to run a test file it says import error like (ImportError: No module named tendrl.gluster_bridge.common.db.event). I dont see any module called Common in gluster_bridge.
There are references to many ceph related terminologies in gluster bridge in files like:
manager/eventer.py
manager/user_request.py
manager/manager.py
manager/request_factory.py
These have to be removed
This is required while processing a job from the job queue. If the current node's id falls within the list of node_ids passed as parameters, the job would be picked and processed by that node. Only one node would pick and process the job from the list of node_ids passed in parameter for job.
As part of re-factoring, the unit test coverage is decreased. More unit tests should be written for better coverage.
Currently the output of gluster get-state
command is read from a file. This file is in ini
format and we just read that into json. Each section in the file becomes one dictionary with entries as keys in dictionary. It would be better parseable/processable if we create a proper hierarchical dictionary as below out of this data and then update the detail into central store
{
'Global': {
'MYUUID': '3780fd01-8047-402c-81df-e361a1935bb8',
'op-version': '30900'
},
'Peers': [
.....
],
'Volumes': [
{
'name': 'vol1',
'id': 'ff0bceae-9901-46b0-9464-7fa4836d2535',
............
'Bricks': [
'Brick1': {
'hostname': '<FQDN>',
'path': '<FQDN>:<mounth path>',
..........
},
..........
],
'Options': {
'option1': 'val1',
'option2': 'val2',
.........
},
...............
},
........
],
'Services': [
{
'name': 'glustershd',
'online_status': "Offline",
},
...............
],
'Misc': {
'Base Port': 49152,
'Last allocated port': 49153
}
]
}
This would make looping through the details and updation in central store more predictable and errors can be handled more effectively (as in case a KeyError
for one volume doesnt cause other volume details updation failures and the error could be logged for that specific volume only)
Repository names need to be updated to follow lowercase, hyphenated convention.
Add preferred tags to the flow definition file, which can be used to generate jobs that are targeted towards specific type of nodes. Also change the sample job format to include this.
Currently to import a cluster, user expected to provide all the nodes in the cluster as input. This will be inconvenient if the size of the cluster is large. Instead the usecase flow is expected to be modified as:
select one of the node in cluster
use the selected node to get the details of the cluster
submit the request to all the nodes to import the cluster
This issue is to create a flow to get the cluster details
Package tendrl-gluster-integration-0.0.1-1.el7.centos.noarch.rpm
from https://copr-be.cloud.fedoraproject.org/results/tendrl/tendrl/epel-7-x86_64/ copr doesn't mark configuration files as configuration.
This breaks Fedora packaging policy and will have impact on user changes in
affected configuration files during package upgrades.
# rpm -ql tendrl-gluster-integration | grep ^/etc
/etc/tendrl/gluster_integration
/etc/tendrl/gluster_integration_logging.yaml
# rpm -qc tendrl-gluster-integration | grep ^/etc
#
While expected result would be:
# rpm -ql tendrl-gluster-integration | grep ^/etc
/etc/tendrl/gluster_integration
/etc/tendrl/gluster_integration_logging.yaml
# rpm -qc tendrl-gluster-integration | grep ^/etc
/etc/tendrl/gluster_integration_logging.yaml
#
for more information see: Tendrl/commons#72
I've done package check from https://copr.fedorainfracloud.org/coprs/tendrl/tendrl/build/480512/.
All errors should be fixed and it will be great to fix also 3 warnings.
$ rpmlint tendrl-gluster-integration-0.0.1-1.el7.centos.noarch.rpm
tendrl-gluster-integration.noarch: W: incoherent-version-in-changelog 0.0.1-1 ['0.0.1-1.el7.centos', '0.0.1-1.centos']
tendrl-gluster-integration.noarch: E: script-without-shebang /usr/share/tendrl/gluster_integration/tendrl.conf.sample
tendrl-gluster-integration.noarch: W: non-conffile-in-etc /etc/tendrl/gluster_integration_logging.yaml
tendrl-gluster-integration.noarch: W: no-manual-page-for-binary tendrl-gluster-integration
tendrl-gluster-integration.noarch: E: unknown-key RSA#e879b811 (MD5
1 packages and 0 specfiles checked; 2 errors, 3 warnings.
Instead of writing configuration details in file, store everything in etcd
The attribute name is repeated in the sds_atoms_gluster. One of them has to be replaced with description.
The common module provides all the base classes for flows, atoms, manager, job queue handler, persister and contains all the utilities which are generic in nature like exceptions, command executor, service manager and package installer. The gluster_integration modules should extend/use these common module changes.
Some of the test cases are failing in gluster integration, we have to fix those issues.
When launched tendrl-gluster-integration command it continuously prints:
ERROR - Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tendrl/gluster_integration/manager/rpc.py", line 161, in _run
self._server.run()
File "/usr/lib/python2.7/site-packages/tendrl/gluster_integration/manager/rpc.py", line 83, in run
self._acceptor()
File "/usr/lib/python2.7/site-packages/tendrl/gluster_integration/manager/rpc.py", line 70, in _acceptor
if "etcd_client" in raw_job['parameters']:
KeyError: 'parameters'
While running tendrl-gluster-integration as service, its not able to read data written at /tmp/glusterd-state by glusterd service.
This happens because each service gets its own variation of /tmp and glusterd and tendrl-gluster-integration are two different services here in this case.
It could be set as /var/run/glusterd-state as its a runtime file.
For CI and integration testing it is important to install all Tendrl components from packages.
tendrl-gluster-integration-0.0.1-1.el7.centos.noarch.rpm
have a dependency on tendrl-common
. tendrl-common
was recently renamed to tendrl-commons
so the package should reflect it in a dependency list.
# rpm -qp https://copr-be.cloud.fedoraproject.org/results/tendrl/tendrl/epel-7-x86_64/00498961-tendrl-gluster-integration/tendrl-gluster-integration-0.0.1-1.el7.centos.noarch.rpm --requires
/bin/sh
.
.
systemd
tendrl-common
rpmlib(PayloadIsXz) <= 5.2-1
#
# rpm -qp https://copr-be.cloud.fedoraproject.org/results/tendrl/tendrl/epel-7-x86_64/00498961-tendrl-gluster-integration/tendrl-gluster-integration-0.0.1-1.el7.centos.noarch.rpm --requires
/bin/sh
.
.
systemd
tendrl-commons
rpmlib(PayloadIsXz) <= 5.2-1
#
The current output from gluster get-state command also contains volume options details.
We should update the code to add volume options details to central store.
Package version should comes from tendrl/gluster_integration/init.py and this should automatically updated from the github if we run make gitversion.
In etc/tendrl/tendrl.conf.sample is attribute called log_path in Common section. It should be probably called log_cfg_path.
According to installation documentation [1] some configuration files from gluster-integration etc directory should be copied to specific directories but etc directory structure is different from what is specified in documentation.
According to documentation logging.yaml.timedrotation.sample should be located at etc/samples/logging.yaml.timedrotation.sample but it is located at etc/tendrl/gluster-integration/logging.yaml.timedrotation.sample.
According to documentation there should be etc/tendrl/tendrl.conf.sample but there is no such file.
Documentation and etc directory should be united.
The check_commit_msg.py complains that the commis message should be of the form :
'tendrl-bug-id:<tendrl_repo>/issue_id'
It should probably say :
tendrl-bug-id:Tendrl/<tendrl_repo_name>#<issue_id>
Current name of systemd service file is tendrl-glusterd.service
, while the name of package is tendrl-gluster-integration
and the name of binary which implements the service is /usr/bin/tendrl-gluster-integration
.
For this reason, the systemd service file should be renamed to tendrl-gluster-integration.service
, so that this file name matches component name.
When one configure gluster_bridge
following the Install docs and run tendrl-gluster-bridge
, the following exception appears in the logs:
[manager.py:48 - _run() ] [Errno 2] No such file or directory
[manager.py:48 - _run() ] [Errno 2] No such file or directory
[rpc.py:96 - _run() ] EtcdThread run...
[client.py:521 - read() ] Issuing read for key /api_job_queue with args {}
[connectionpool.py:383 - _make_request() ] "GET /v2/keys/api_job_queue HTTP/1.1" 404 79
[rpc.py:99 - _run() ] Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tendrl_gluster_bridge-0.1-py2.7.egg/tendrl/gluster_bridge/manager/rpc.py", line 97, in _run
self._server.run()
File "/usr/lib/python2.7/site-packages/tendrl_gluster_bridge-0.1-py2.7.egg/tendrl/gluster_bridge/manager/rpc.py", line 50, in run
self._acceptor()
File "/usr/lib/python2.7/site-packages/tendrl_gluster_bridge-0.1-py2.7.egg/tendrl/gluster_bridge/manager/rpc.py", line 28, in _acceptor
jobs = self.client.read("/api_job_queue")
File "/usr/lib/python2.7/site-packages/etcd/client.py", line 536, in read
timeout=timeout)
File "/usr/lib/python2.7/site-packages/etcd/client.py", line 848, in wrapper
return self._handle_server_response(response)
File "/usr/lib/python2.7/site-packages/etcd/client.py", line 928, in _handle_server_response
etcd.EtcdError.handle(r)
File "/usr/lib/python2.7/site-packages/etcd/__init__.py", line 304, in handle
raise exc(msg, payload)
EtcdKeyNotFound: Key not found : /api_job_queue
[manager.py:48 - _run() ] [Errno 2] No such file or directory
[rpc.py:96 - _run() ] EtcdThread run...
[client.py:521 - read() ] Issuing read for key /api_job_queue with args {}
[connectionpool.py:383 - _make_request() ] "GET /v2/keys/api_job_queue HTTP/1.1" 404 79
[rpc.py:99 - _run() ] Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tendrl_gluster_bridge-0.1-py2.7.egg/tendrl/gluster_bridge/manager/rpc.py", line 97, in _run
self._server.run()
File "/usr/lib/python2.7/site-packages/tendrl_gluster_bridge-0.1-py2.7.egg/tendrl/gluster_bridge/manager/rpc.py", line 50, in run
self._acceptor()
File "/usr/lib/python2.7/site-packages/tendrl_gluster_bridge-0.1-py2.7.egg/tendrl/gluster_bridge/manager/rpc.py", line 28, in _acceptor
jobs = self.client.read("/api_job_queue")
File "/usr/lib/python2.7/site-packages/etcd/client.py", line 536, in read
timeout=timeout)
File "/usr/lib/python2.7/site-packages/etcd/client.py", line 848, in wrapper
return self._handle_server_response(response)
File "/usr/lib/python2.7/site-packages/etcd/client.py", line 928, in _handle_server_response
etcd.EtcdError.handle(r)
File "/usr/lib/python2.7/site-packages/etcd/__init__.py", line 304, in handle
raise exc(msg, payload)
EtcdKeyNotFound: Key not found : /api_job_queue
[manager.py:48 - _run() ] [Errno 2] No such file or directory
[manager.py:195 - shutdown() ] Signal handler: stopping
Instead of the exception, gluster_bridge
should validate the data it's getting from etcd and log the issue properly without raising the exception.
Version specified in setup.py
file is 0.0.1
, while the latest git tag (and releases page provided for these tags by github) is v1.1
.
$ grep version setup.py
version="0.0.1",
$ git tag
v1.0
v1.1
https://packaging.python.org/distributing/#choosing-a-versioning-scheme
https://packaging.python.org/single_source_version/#single-sourcing-the-version
Segregate all the flows specific to object, under that object instead of having it in global name space. Note that such flows should be independent other objects
Specification: Tendrl/specifications#34
All the actions that are specific to an object should be defined as flows under that particular object. Currently such flows are defined at global level. For example, the start volume, stop volume, delete volume flows should be under volume object.
Specification PR: Tendrl/specifications#10
Specification issue: Tendrl/specifications#34
Service failed to start due to environment variable issue which is used in one of the tendrl configuration file. Service configuration file can not expand the path of a environment variables like $HOME or $PWD.
This issue suggest to remove requirements.txt
file and move items which this file contains into install_requires
setuptools keyword in setup.py
file.
Based on explanation of difference between install_requires
keyword and requirements.txt
file from upstream Python Packaging User Guide, I understand that:
install_requires
requirements.txt
files are used to define the requirements for a complete python environmentSince this project is a single Tendrl component, which is expected to be installed with other Tendrl components together, using requirement file seems to be inappropriate. Listing dependencies of single reusable python library/tool/daemon, which is our case here, is not mentioned among 4 common uses of Requirements files as listed in description of the feature in packaging guide. Moreover, it creates additional issues:
requirements.txt
file makes us to install the project in non-production way, which means that we will find problems with listing requirements later than we should, see eg. issue https://github.com/Tendrl/performance_monitoring/issues/17 - this is very blatant issue, but was not noticed by any developer, unit tests or CI - since all of those are using installation method which doesn't resemble what an admin using this project would dorequirements.txt
file while specifying install_requires
in setup.py
requires to maintain function to read requirements.txt
file in setup.py
, which is unnecessary maintenance burdenrequirements.txt
is higher and includes features which should not be done in production setupUse new logging framework for job status updates in flows and atoms
For testing gluster_bridge, tox required bridge_common as site-package, we have to give bridge_common in test_requirement as pip installable.
refs : https://gist.github.com/GowthamShanmugam/bf66174d7d6062fab936706c713d3d0c
gluster-integration service should read the cluster-id from the conf file instead of expecting a command line argument.
Unfortunately, it looks like the gluster_bridge is missing config file that wasn't added. we have to fix this issue.
The pre and post runs are dummy placeholders at the moment for the volume operations
Implement the actual logic and test cases for the same as part of this fix.
There are some failures while updating the volume options. Correct the same
Volume status is not updating properly at etcd server when start and stop actions performed.
Copy the logging config to /etc/tendrl/gluster-integration
Also update the 'log_cfg_path' in etc/tendrl/gluster-integration/gluster-integration.conf.yaml.sample to reflect the same
We need to collectively start improving the unit test code coverage for this project.
Current coverage: https://coveralls.io/github/Tendrl/gluster_bridge
Unit tests framework: http://doc.pytest.org/en/latest/
How to write unit tests: http://docs.openstack.org/developer/horizon/topics/testing.html
Contribute patches here: https://github.com/Tendrl/gluster_bridge/tree/master/gluster_bridge/tests
Lets work on this issue together to get code coverage up to 90%, please share doubts/inputs on this issue itself
We need to fix pep8 errors from latest builds
How to detect pep8 errors:
Current build: https://travis-ci.org/Tendrl/gluster_bridge
The new job structure is as per https://github.com/Tendrl/bridge_common/blob/master/etc/samples/tendrl_api_job.sample.json.
Based on this rpc.py which reads and processes the jobs from etcd job queue should be modified to figure out the actual flow name and invoke the same.
Currently the volume is marked as deleted=True as an stop gap arrangement. Modify the same to delete the etcd entry post volume deletion flow.
There should be document which describes:
[was originally at https://tendrl.atlassian.net/browse/TEN-36]
2.From rpc.py, a while loop is running without any check in _acceptor method. I cant test that function.
and control never come out from that function.
3.From ini2json.py, there a method called dget it is not called anywhere and object for class StrictConfigParser is created and initialized and destroyed inside method ini_to_dict so calling dget from other method is useless.
It's not directly possible to ship tendrl-gluster-integration
component with configuration file(s) which would contain all default values known in advance (eg. before installation).
Eg. we know that we need to set this piece of configuration into /etc/tendrl/tendrl.conf
(see installation.rst):
[gluster_integration]
log_cfg_path = /etc/tendrl/gluster_integration_logging.yaml
But it's not possible to ship this in a default configuration file in binary package, because tendrl.conf
configuration file is owned by tendrl-node-agent
component.
Based on my understanding, tendrl-gluster-integration requires a tendrl-node-agent, but this is not directly mentioned in installation.rst file. Is my understanding correct (so that we need to add this into the docs)?
Salt package is not a requirement for gluster bridge. So it has to be removed from the requirements file.
Moving on, all git commits should contain mandatory mention on newline of
"tendrl-bug-id: <tendrl_repo>/" and optional mention of "tendrl-spec: <tendrl_spec_name>" in the commit msg.
eg:
"""
Packaging python-etcd
Added Makefile and spec file to create python-etcd package
Now one can use "make rpm" to pull the latest source from github
and build the source rpm
and using the Makefile one can also build rpm using "make rpm"
which will use mock to build the rpm using srpm.
tendrl-bug-id: gluster_integration/18
tendrl-spec: refactor_gluster_integration_get_state_dump
"""
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.