mtth / azkaban Goto Github PK
View Code? Open in Web Editor NEWLightweight Azkaban client
Home Page: http://azkabancli.readthedocs.org
License: MIT License
Lightweight Azkaban client
Home Page: http://azkabancli.readthedocs.org
License: MIT License
Traceback (most recent call last):
File "/usr/local/bin/azkaban", line 9, in <module>
load_entry_point('azkaban==0.9.5', 'console_scripts', 'azkaban')()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/local/lib/python2.7/dist-packages/azkaban/__main__.py", line 98, in <module>
from azkaban.project import Project
File "/usr/local/lib/python2.7/dist-packages/azkaban/project.py", line 8, in <module>
from .util import AzkabanError, Adapter, flatten, temppath, write_properties
File "/usr/local/lib/python2.7/dist-packages/azkaban/util.py", line 13, in <module>
from requests.packages.urllib3 import disable_warnings
ImportError: No module named packages.urllib3
checked from https://azkaban.readthedocs.io/en/latest/ajaxApi.html#, I found that there were several api that were not included in this python cli, such as fetchProjectLogs, fetchprojectflows, fetchflowgraph. I thinks they're also important. So why we don't have them and is there any plan to add them?
When running a flow, azkaban-cli only support 'concurrent' and 'skip' values for the concurrentOption request parameters. However Azkaban server 3.0 supports 'skip', 'pipeline' and 'queue'.
It could be really great to supports all this values.
Hi, as shown in the following full dependency graph of azkaban, azkaban requires urllib3 * , while the installed version of requests(2.22.0) requires urllib3 <1.26,>=1.21.1.
According to Pip's “first found wins” installation strategy, urllib3 1.25.7 is the actually installed version.
Although the first found package version urllib3 1.25.7 just satisfies the later dependency constraint (urllib3 <1.26,>=1.21.1), it will lead to a build failure once developers release a newer version of urllib3.
azkaban - 0.9.13
| +- docopt(install version:0.6.2 version range:*)
| +- requests(install version:2.22.0 version range:>=2.4.0)
| | +- certifi(install version:2019.9.11 version range:>=2017.4.17) m
| | +- chardet(install version:3.0.4 version range:<3.1.0,>=3.0.2)
| | +- idna(install version:2.8 version range:>=2.5,<2.9)
| | +- urllib3(install version:1.25.6 version range:<1.26,>=1.21.1)
| +- six(install version:1.13.0 version range:>=1.6.1)
| +- urllib3(install version:1.25.6 version range:*)
Thanks for your attention.
Best,
Neolith
Running azkaban info -p jobs.py2
still works when the actual file is named jobs.py
. Should change the way the script is dynamically loaded: move from __import__
to imp.load_source
.
Give the option to create complex flows that have multiple jobs that depend to each other. For example it would be really useful if we were able to generate job files like this:
# foo.job
type=command
command=echo foo
# bar.job
type=command
dependencies=foo
command=echo bar
Is there any chance we can create conditional flow on Job? https://azkaban.readthedocs.io/en/latest/conditionalFlow.html
To enable integration tests I created ~/.azkabanrc
with
[azkaban]
test.alias = local
[alias.local]
url = http://localhost:8081
Then I ran
nosetests
There was an error:
..............................................Azkaban password for JuhoAutio@http://localhost:8081:
Azkaban password for JuhoAutio@http://localhost:8081: Azkaban password for JuhoAutio@http://localhost:8081: E
======================================================================
ERROR: test_remote.TestCreateDelete.test_create_delete_project
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/JuhoAutio/.pyenv/versions/3.7.0/lib/python3.7/site-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/Users/JuhoAutio/ideaprojects/AzkabanCli/azkaban/test/test_remote.py", line 97, in test_create_delete_project
ok_(not self.project_exists(self.project))
File "/Users/JuhoAutio/ideaprojects/AzkabanCli/azkaban/test/test_remote.py", line 90, in project_exists
self.session.upload_project(project, path)
File "/Users/JuhoAutio/ideaprojects/AzkabanCli/azkaban/azkaban/remote.py", line 651, in upload_project
self._refresh() # ensure that the ID is valid
File "/Users/JuhoAutio/ideaprojects/AzkabanCli/azkaban/azkaban/remote.py", line 765, in _refresh
if not 'Incorrect Login.' in err.message:
AttributeError: 'AzkabanError' object has no attribute 'message'
Hmm not sure if I can reproduce this test run consistently.. but any way it shows the error AttributeError: 'AzkabanError' object has no attribute 'message'
which is real. I believe since python 2.6 or something Exception
doesn't have message
any more.
Apart from that, how can I successfully run the integration tests?
I set the url with credentials:
[alias.local]
url = azkaban:azkaban@http://localhost:8081
After this I nosetests
were run through, but there were quite many errors. For example:
AssertionError: 'Execution queued successfully wi' != 'Execution submitted successfully'
Azkaban requires all job names to be unique within a project. The current behavior when there's a naming collision is for Azkaban to throw a 500 error that doesn't provide useful details to the deploying user about what went wrong. This leads to a lot of confusion for developers who are encountering the issue for the first time.
Instead of the current behavior, I propose a pre-deploy check for uniqueness among job names in a project. This provides the benefit of being able to provide a more detailed error message that will allow developers to resolve their naming collision more quickly.
azkaban (create | delete)
create
and delete
should return a user prompt to complete action by filling in project name and description.
azkaban-cli requires ConfigParser package but this package is built into the system on python 3. So ideally, we would want to try import ConfigParser and then if don't work, then contigparser
Hi,
It seems that the the lightweight azkaban client does not support the pipelineLevel query parameters (see https://github.com/azkaban/azkaban/blob/71b5dd9a83935c2f253b0d944e47909b918b06bc/azkaban-common/src/main/java/azkaban/server/HttpRequestUtils.java#L94). Enabling this will allow azkaban-cli to fully support all concurrent modes.
After I upload the job, here is what I have in the Azkaban log:
> Time User Type Message
> 2014-07-07 16:29 35s jdoe Uploaded Uploaded project files zip file.zip
What I would like to have
> Time User Type Message
> 2014-07-07 16:29 35s jdoe Uploaded Uploaded project files my-job-0.0.4-SNAPSHOT-azkaban.zip
On
Line 44 in f69e98d
requests
library. This results in a bunch of InsecureRequestWarning
s from urllib3
in addition to it just being plain old unsafe.
Can we remove the passing of that flag and let users configure urllib themselves if they want to disable SSL verification like that?
When the Azkaban server is overloaded, jobs can hang in preparing state which delays the creation of the log file, which causes azkabanpig
to error out.
Probably would be best to add it to the RC file
I'm getting HTTPError: 411 Client Error: Length Required for url
Revamp .azkabanrc
:
[azkaban]
default_alias = ...
[azkabanpig]
default_project = pig_you
default_type = pig
[aliases]
foo = you@http://that
bar = http://this
[session_ids]
foo = abc
We are currently running Azkaban 3.1.0 in our production environment and all aspects of this code works great on 3.1.0.
However we are in the process of getting 3.59.0 working in our lab, in preparation of upgrading to a more current version. Almost everything works, however we have one issue that we have discovered.
Issue
azkaban build -cp project.py -a lab-azkaban
Azkaban password for username@http://azkaban.example.com:
Traceback (most recent call last):
File "/path/to/virtualenvs/venv_name/lib/python2.7/site-packages/azkaban/util.py", line 310, in wrapper
return func(*args, **kwargs)
File "/path/to/virtualenvs/venv_name/lib/python2.7/site-packages/azkaban/__main__.py", line 502, in main
['ZIP', '--url', '--alias', '--replace', '--create', '--option']
File "/path/to/virtualenvs/venv_name/lib/python2.7/site-packages/azkaban/__main__.py", line 460, in build_project
res = _upload_zip(session, project.name, _zip, _create, archive_name)
File "/path/to/virtualenvs/venv_namelib/python2.7/site-packages/azkaban/__main__.py", line 320, in _upload_zip
callback=_callback,
File "/path/to/virtualenvs/venv_name/lib/python2.7/site-packages/azkaban/remote.py", line 673, in upload_project
data=form,
File "/path/to/virtualenvs/venv_name/lib/python2.7/site-packages/azkaban/remote.py", line 902, in _request
raise err
HTTPError: 410 Client Error: Gone for url: http://azkaban.example.com/manager
This error only manifests when we have the -c
argument being passed and the project does not already exist. If the project already exists, then the build and upload works without issue.
Are there any recommendations regarding where this issue might be arising? I was hoping you might have some insight.
Thank you.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.