zerodayyy / spotcli Goto Github PK
View Code? Open in Web Editor NEWCLI for Spot Elastigroups management
Home Page: https://github.com/zerodayyy/spotcli/releases/latest
License: MIT License
spotcli's People
Contributors
Stargazers
spotcli's Issues
Add "list" command
spotcli list [aliases|scenarios] — pretty-print available aliases/scenarios.
Tests
- Fix failing tests
- Write missing tests
- Add code coverage report
- Add test stage to CI
- Configure repo to only allow merging PRs with green tests
- Add shields to README
S3 provider
To make sure configs aren't edited by unauthorized users, a provider with access control is needed. S3 is a perfect choice for this use case.
TODO:
- Research S3 client libs, use
boto3as a fallback - Implement S3Provider class
- Add tests for S3Provider
After running suspend/unsuspend, there is an exception
Describe the bug
After running suspend/unsuspend on any scenario.
The groups got suspended or unsuspended, but the following exception is thrown:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ spotcli/cli.py:375 in │
│ │
│ 372 │
│ 373 │
│ 374 if name == "main": │
│ ❱ 375 │ main() │
│ │
│ /Users/lior.franko/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py:829 in │
│ call │
│ │
│ 826 │ │
│ 827 │ def call(self, *args, **kwargs): │
│ 828 │ │ """Alias for :meth:main.""" │
│ ❱ 829 │ │ return self.main(*args, **kwargs) │
│ 830 │
│ 831 │
│ 832 class Command(BaseCommand): │
│ │
│ /Users/lior.franko/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py:782 in main │
│ │
│ 779 │ │ try: │
│ 780 │ │ │ try: │
│ 781 │ │ │ │ with self.make_context(prog_name, args, **extra) as ctx: │
│ ❱ 782 │ │ │ │ │ rv = self.invoke(ctx) │
│ 783 │ │ │ │ │ if not standalone_mode: │
│ 784 │ │ │ │ │ │ return rv │
│ 785 │ │ │ │ │ # it's not safe to ctx.exit(rv) here! │
│ │
│ /Users/lior.franko/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py:1259 in │
│ invoke │
│ │
│ 1256 │ │ │ │ Command.invoke(self, ctx) │
│ 1257 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │
│ 1258 │ │ │ │ with sub_ctx: │
│ ❱ 1259 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ 1260 │ │ │
│ 1261 │ │ # In chain mode we create the contexts step by step, but after the │
│ 1262 │ │ # base command has been invoked. Because at that point we do not │
│ │
│ /Users/lior.franko/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py:1066 in │
│ invoke │
│ │
│ 1063 │ │ """ │
│ 1064 │ │ _maybe_show_deprecated_notice(self) │
│ 1065 │ │ if self.callback is not None: │
│ ❱ 1066 │ │ │ return ctx.invoke(self.callback, **ctx.params) │
│ 1067 │
│ 1068 │
│ 1069 class MultiCommand(Command): │
│ │
│ /Users/lior.franko/.pyenv/versions/3.7.7/lib/python3.7/site-packages/click/core.py:610 in invoke │
│ │
│ 607 │ │ args = args[2:] │
│ 608 │ │ with augment_usage_errors(self): │
│ 609 │ │ │ with ctx: │
│ ❱ 610 │ │ │ │ return callback(*args, **kwargs) │
│ 611 │ │
│ 612 │ def forward(*args, **kwargs): # noqa: B902 │
│ 613 │ │ """Similar to :meth:invoke but fills in default keyword │
│ │
│ spotcli/cli.py:137 in run │
│ │
│ 134 │ │ console.print("\n") │
│ 135 │ auto_approve or click.confirm("Continue?", abort=True) │
│ 136 │ t_start = time.time() │
│ ❱ 137 │ s.run() │
│ 138 │ t_end = time.time() │
│ 139 │ duration = t_end - t_start │
│ 140 │ console.print( │
│ │
│ /Users/lior.franko/Repos/spotcli/spotcli/configuration/tasks.py:282 in run │
│ │
│ 279 │ def run(self): │
│ 280 │ │ results = [] │
│ 281 │ │ for task in self.tasks: │
│ ❱ 282 │ │ │ results.extend(task.run()) │
│ 283 │ │ return results │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'NoneType' object is not iterable
To Reproduce
Steps to reproduce the behavior:
- Run command 'spotcli run unsuspend-is'
- See error
Expected behavior
No error
Console output
The above exception
Configurations
If applicable, add all existing configs.
System (please complete the following information):
- OS: [e.g. macOS 11.0.0, Ubuntu 18.04]
- Python version [e.g. 3.8.5]
- Python installation source [e.g. system, apt, homebrew, pyenv]
Additional info
Add any other info about the problem here.
Threading errors on Python 3.9
Describe the bug
SpotCLI is not working on Python 3.9 due to a bug in threading library.
To reproduce
Steps to reproduce the behavior:
- Use Python 3.9 interpreter
- Run any action
Expected behavior
The action runs successfully
Console output
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/zerodayyy/.pyenv/versions/3.9.0/bin/spotcli:8 in <module> │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py:829 in __call__ │
│ │
│ 826 │ │
│ 827 │ def __call__(self, *args, **kwargs): │
│ 828 │ │ """Alias for :meth:`main`.""" │
│ ❱ 829 │ │ return self.main(*args, **kwargs) │
│ 830 │
│ 831 │
│ 832 class Command(BaseCommand): │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py:782 in main │
│ │
│ 779 │ │ try: │
│ 780 │ │ │ try: │
│ 781 │ │ │ │ with self.make_context(prog_name, args, **extra) as ctx: │
│ ❱ 782 │ │ │ │ │ rv = self.invoke(ctx) │
│ 783 │ │ │ │ │ if not standalone_mode: │
│ 784 │ │ │ │ │ │ return rv │
│ 785 │ │ │ │ │ # it's not safe to `ctx.exit(rv)` here! │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py:1259 in invoke │
│ │
│ 1256 │ │ │ │ Command.invoke(self, ctx) │
│ 1257 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │
│ 1258 │ │ │ │ with sub_ctx: │
│ ❱ 1259 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ 1260 │ │ │
│ 1261 │ │ # In chain mode we create the contexts step by step, but after the │
│ 1262 │ │ # base command has been invoked. Because at that point we do not │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py:1066 in invoke │
│ │
│ 1063 │ │ """ │
│ 1064 │ │ _maybe_show_deprecated_notice(self) │
│ 1065 │ │ if self.callback is not None: │
│ ❱ 1066 │ │ │ return ctx.invoke(self.callback, **ctx.params) │
│ 1067 │
│ 1068 │
│ 1069 class MultiCommand(Command): │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py:610 in invoke │
│ │
│ 607 │ │ args = args[2:] │
│ 608 │ │ with augment_usage_errors(self): │
│ 609 │ │ │ with ctx: │
│ ❱ 610 │ │ │ │ return callback(*args, **kwargs) │
│ 611 │ │
│ 612 │ def forward(*args, **kwargs): # noqa: B902 │
│ 613 │ │ """Similar to :meth:`invoke` but fills in default keyword │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/spotcli/cli.py:157 in scale │
│ │
│ 154 │ KIND is the scaling direction: up or down. │
│ 155 │ GROUP is elastigroup name, alias or regex. │
│ 156 │ """ │
│ ❱ 157 │ action(action=f"{kind}scale", target=group, amount=amount, auto_approve=auto_approve │
│ 158 │
│ 159 │
│ 160 main.add_command(version) │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/spotcli/cli.py:185 in action │
│ │
│ 182 │ console.print("\n") │
│ 183 │ auto_approve or rich.prompt.Confirm.ask("Continue?") or sys.exit(1) │
│ 184 │ t_start = time.time() │
│ ❱ 185 │ task.run() │
│ 186 │ t_end = time.time() │
│ 187 │ duration = t_end - t_start │
│ 188 │ console.print(f"\n[bold green]Task run complete! Ran 1 task in {duration:.2f} second │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/spotcli/tasks.py:134 in run │
│ │
│ 131 │ │ loom = pexecute.thread.ThreadLoom(max_runner_cap=PARALLEL_THREADS) │
│ 132 │ │ for target in self.targets: │
│ 133 │ │ │ loom.add_function(work, [], dict(target=target, amount=self.amount, console= │
│ ❱ 134 │ │ return loom.execute() │
│ 135 │
│ 136 │
│ 137 @Task.register("downscale") │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pexecute/base_loom.py:125 in │
│ execute │
│ │
│ 122 │ │ │ runner = self.runners.pop(0) │
│ 123 │ │ │ runner.start() │
│ 124 │ │ │ self.started.append(runner) │
│ ❱ 125 │ │ │ while self.get_active_runner_count() >= self.max_runner_cap: │
│ 126 │ │ │ │ time.sleep(self.POOL_FILL_PAUSE) │
│ 127 │ │ │
│ 128 │ │ while self.get_active_runner_count(): │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pexecute/base_loom.py:141 in │
│ get_active_runner_count │
│ │
│ 138 │ │ │
│ 139 │ │ count = 0 │
│ 140 │ │ for runner in self.started: │
│ ❱ 141 │ │ │ if runner.is_running() or not runner.is_tracker_updated(): │
│ 142 │ │ │ │ count += 1 │
│ 143 │ │ │
│ 144 │ │ if count == 0: │
│ │
│ /Users/zerodayyy/.pyenv/versions/3.9.0/lib/python3.9/site-packages/pexecute/thread/runner.py:21 │
│ in is_running │
│ │
│ 18 │ def is_running(self): │
│ 19 │ │ """ Returns True if runner is active else False """ │
│ 20 │ │ │
│ ❱ 21 │ │ return self.runner and self.runner.isAlive() │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'Thread' object has no attribute 'isAlive'
System (please complete the following information):
- macOS 11.0
- Python 3.9.0 (pyenv)
Telemetry
Since SpotCLI is a distributed app, it's important to track all kinds of exceptions in it, as well as the adoption rate of new versions. For this, Sentry can be utilized.
This should be an opt-out feature, which one can disable in the app config: no data should be sent once user opts out of telemetry.
Document all modes and config options
Put essential stuff in README, full documentation — in GitHub Wiki or on readthedocs.org.
Detect new versions
To avoid app distributions from becoming outdated, a new version detection mechanism is needed.
Implementation idea: compare app version to latest tag name on GitHub. Use packaging.version.parse to compare versions. Display update command.
Add command to list instances in an Elastigroup
Is your feature request related to a problem? Please describe.
This will add more visibility when doing operations using this tool
Describe the solution you'd like
A command like spotcli list instances example-group that returns the list of group's instances with some metadata like IPs, healthchecks etc.
Add child class register decorator to providers submodule
Can be implemented same as in tasks submodule
Stateful mode
Sometimes you need to revert exactly to the state before the scenario/action run. Stateful mode should solve this problem.
- Implement state gathering
- Implement state storage (JSON + TinyDB)
- Implement state diffs
- Detect state changes besides saved states
- Implement resource locks
- Scenario parameter & CLI flag for stateful mode
- Implement revert action
- Display & log revert command in the end when in stateful mode
"List" command filtering
Is your feature request related to a problem? Please describe.
At some point the output of spotcli list x will become way too large to be readable.
Describe the solution you'd like
A simple regex or substring filter, and a CLI option to enable it, like spotcli list scenarios -f ow
Describe alternatives you've considered
Using grep on the table is not convenient
Additional context
From implementation standpoint, the existing filter in spotcli.elastigroup module can be reused if separated into its own module.
Audit logging
We need a possibility to log all actions somewhere. Good first candidates for "somewhere" are Slack and ELK.
TODO:
- Implement Slack provider
- Implement Logstash providers
- Add
logparameter to Scenario and CLI - Add SlackProvider tests
- Add LogstashProvider tests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.