ploxiln / fab-classic Goto Github PK

View Code? Open in Web Editor NEW

64.0 64.0 11.0 21.23 MB

Simple, Pythonic remote execution - Fork of Fabric 1.x

Home Page: https://ploxiln.github.io/fab-classic/

License: BSD 2-Clause "Simplified" License

Python 100.00%

fab-classic's People

Contributors

Stargazers

Watchers

Forkers

jealouscloud devopscraftsman stealthii bkbonde jaysoffian pathcl timsnyder-siv arpitjain799 babywyrm nephila andrei-kondakov

fab-classic's Issues

[feature request] parse bool args automatically

Like this:
https://stackoverflow.com/a/19536667
https://stackoverflow.com/a/21138900

Can automatically convert bool argument in fabric

[Bug] rsync_project exclude flag broken in Python3+

Python3 added the __iter__ attribute to strings so the following check doesn't work:

fab-classic/fabric/contrib/project.py

Lines 107 to 108 in 8cff965

    
           if not hasattr(exclude, '__iter__'): 
        
               exclude = (exclude,)

This results in the exclude rsync flag splitting up a string like HELLO into H E L L O.

Looking at the newer Fabric's rsync they do the following:

https://github.com/fabric/patchwork/blob/d653591cc8130765978a91d02c4cd108e76cca06/patchwork/transfers.py#L82-L83

With six.string_types switching depending on whether they are using Py3 or 2:

https://github.com/pyinvoke/invoke/blob/45dc9d03639dac5b6d1445831bf270e686ef88b4/invoke/vendor/six.py#L40-L49

An UnicodeDecodeError error occurred when I used the 'capture=True' of local function

fab --version
                                                        
fab-classic 1.18.0
paramiko-ng 2.8.2
Python 3.7.0

The python code
outstr = local('mvn clean package',capture=True)

Actually the execute result did not have 'UTF-8' encoding.

output

[localhost] local: mvn clean package
Traceback (most recent call last):
  File "d:\Code\fabfile.py", line 130, in <module>
    outstr =local('mvn clean package',capture=True)
  File "D:\Programs\Python37\lib\site-packages\fabric\operations.py", line 1212, in local
    out = _AttributeString(stdout.decode('utf-8').strip() if stdout else "")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 1480: invalid start byte

Can you add an encoding parameter in the local function?

thks

Why use paramiko-ng?

Hi,

Some people might already depend on paramiko, so having both paramiko and paramiko-ng will result in a dependency hell.

What are the reasons to use paramiko-ng instead of paramiko==2.7.1?

put command fails when bashrc contains a cd

Nice to see fab-classic ! I didn't know it exists until now.

I have a question that I hope someone can help with. I noticed that my put commands fail if my .bashrc contains a cd command ... in my case, I'm sourcing another file from my .bashrc and at the end of this file I do a cd /my/folder. This works totally fine, but then if I try to run a put("something", "/remote/folder/xyz"), it always fails with

[[email protected]] put: /my/local/file -> /my/remote/file

Fatal error: sudo() received nonzero return code 1 while executing!

Requested: mv "3bfff637b1dc4ccabd86becf6542a3d6" "/my/remote/file"
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "mv \"3bfff637b1dc4ccabd86becf6542a3d6\" \"/my/remote/file\""

I know I can probably remove the cd from my .bashrc, but is there another way around it?

I even try to only source the file if it's not a tty, e.g. tty -s && source .my_rc_file, but it is still sourced with fabric.

Something is opening file descriptors and not closing them

Easy to reproduce. Just run a fabric task across 1024+ hosts on. The per process file descriptor will trigger and the first place fab-classic breaks is on this select:

!!! Parallel execution exception under host 'test.host:
Process test.host:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/repo/python-pp-fabric/fabric/tasks.py", line 217, in _parallel_wrap
    queue.put({'name': name, 'result': task.run(*args, **kwargs)})
  File "/home/ubuntu/repo/python-pp-fabric/fabric/tasks.py", line 168, in run
    return self.wrapped(*args, **kwargs)
  File "/home/ubuntu/repo/python-pp-fabtools/pp_fabtools/system_cmds.py", line 20, in runcmd_as_root
    cmd = sudo(cmd)
  File "/home/ubuntu/repo/python-pp-fabric/fabric/network.py", line 700, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/repo/python-pp-fabric/fabric/operations.py", line 1130, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/home/ubuntu/repo/python-pp-fabric/fabric/operations.py", line 923, in _run_command
    timeout=timeout, capture_buffer_size=capture_buffer_size)
  File "/home/ubuntu/repo/python-pp-fabric/fabric/operations.py", line 807, in _execute
    worker.raise_if_needed()
  File "/home/ubuntu/repo/python-pp-fabric/fabric/thread_handling.py", line 26, in raise_if_needed
    six.reraise(e[0], e[1], e[2])
  File "/home/ubuntu/.virtualenvs/python-pp-fabric/lib/python3.6/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/ubuntu/repo/python-pp-fabric/fabric/thread_handling.py", line 13, in wrapper
    callable(*args, **kwargs)
  File "/home/ubuntu/repo/python-pp-fabric/fabric/io.py", line 269, in input_loop
    r, w, x = select([f], [], [], 0.0)
ValueError: filedescriptor out of range in select()

As far as where this is happening, the best read I have on it seems like sys.stdout and/or other sys based output streams are being duplicated through the threadable call to io.py and output/input_looper functions.

I can say that this doesn't happen at all in the original fabric package. It also doesn't seem to happen under fab-classic using python2.7 - I've tested 3.6.9 and 3.7.5 and they both have this problem.

Doing something like this at the bottom of the tasks.execeute():

os.close(sys.stdin.fileno())
os.close(sys.stdout.fileno())
os.close(sys.stderr.fileno())

While silly, this stops the leak. It breaks all the other output of the program. There is a noted difference between fd.close() and os.close(fd).

The best I can tell is it seems like fd.close() leaves the C level file descriptor open in python 3.x but I'm not sure why yet.

network.py hiding error, prompts for password

https://github.com/ploxiln/fab-classic/blob/ca674ab/fabric/network.py#L568

How about having a print(e) above that line?

ssh agent forwarding

I'm trying to use ssh agent forwarding using -A but for some strange reason it's not working for me... using ssh with agent-forwarding works however ... any idea on how to debug / fix this?

@parallel doesn't work with Python 3.8

Hi,

First of all, thanks for the fork and for the work done!

We're migrating from Fabric3==1.14.post1 to fab-classic.

Both packages have the same problem: parallel doesn't work with Python 3.8.

You can import it as:
a. from fabric.api import parallel or
b. from fabric.decorators import parallel

Here is an example:

@parallel
@roles('nginx')
def test():
    run('echo test!')

Result:

(venv) MacBook-Pro:my-website mmalysh$ fab test
[[email protected]] Executing task 'test'
Traceback (most recent call last):
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/main.py", line 759, in main
    execute(
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/tasks.py", line 412, in execute
    ran_jobs = jobs.run()
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/job_queue.py", line 138, in run
    _advance_the_queue()
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/job_queue.py", line 123, in _advance_the_queue
    job.start()
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_execute.<locals>.inner'

No problems with Python 3.7.7.

Public key authentication fails with Ubuntu 22.04 LTS servers

Steps to replicate:

Create a virtual server running Ubuntu 22.04 LTS and configure with an appropriate keypair (I have replicated this on instances running on both DigitalOcean and AWS)

Create a minimal fabfile:

from fabric.api import env, run, task

env.host_string = "X.X.X.X"  # replace with IP address of your instance
env.user = "myuser"  # replace with the username on the instance

@task
def hello():
    run("echo hello")

Execute fab hello

Expected result:

The instance executes the hello command on the remote server and then disconnects

Actual result:

Authentication fails and the user is prompted to enter a password

➜ ~ fab hello
[X.X.X.X] run: echo hello
Connect error: Authentication failed.
[X.X.X.X] Login password for 'myuser':

If I follow these same steps, but instead try it on a Ubuntu 20.04 LTS server it works correctly. I haven't tried this with any non-LTS releases.

Versions:

Local machine:

➜ ~ fab --version
fab-classic 1.19.2
paramiko-ng 2.8.10
Python 3.11.4

I get the same result with other versions of Python too (e.g. 3.8.x).

Remote machine:

[email protected]:~# ssh -V
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022

Fabric forcing password when not required

I recently upgraded Ubuntu and fab-classic versions, and now old code that used to work is failing because Fabric is now demanding a password when none is required.

chris@localhost:~$ ssh chris@localhost
Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-91-generic x86_64)
chris@localhost:~$ exit
Connection to localhost closed.
chris@localhost:~$ . .env/bin/activate
(.env) chris@localhost:~$ pip freeze | grep fab-classic
fab-classic==1.19.2
(.env) chris@localhost:~$ python
(.env) chris@localhost:~$ python
Python 3.12.1 (main, Dec 10 2023, 15:07:36) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fabric.api import run, env
/home/chris/git/burlap/.env/lib/python3.12/site-packages/paramiko/transport.py:178: CryptographyDeprecationWarning: Blowfish has been deprecated
  'class': algorithms.Blowfish,
>>> env.host_string = 'localhost'; result = run('uname -s')
[localhost] run: uname -s
Connect error: Authentication failed.
[localhost] Login password for 'chris':

Why is Fabric asking for a password when my localhost doesn't require one for that user?

add type annotations

@ploxiln would you be interested in a PR that adds type annotations to fab-classic? I could slog through doing it if you have interest in accepting it.

mesg: ttyname failed: Inappropriate ioctl for device

Sorry for the delay reporting it. We ended up reverting back to fabric 1.14.1 to resolve this, and I didn't get a chance to test again.

Using fab-classic 1.18.0 with python 2.x I'm getting this error message when I launch fab commands over a remote SSH, e.g.

$ ssh [email protected] "fab run_command -H [email protected]"

[[email protected]] ... some output
[[email protected]] out: mesg: ttyname failed: Inappropriate ioctl for device

This doesn't happen with fabric 1.14.1

In most cases, the error is harmless, but some commands seem to not execute correctly when this happens (still not 100% sure what fails)

running the fab command locally rather than via ssh works ok.

Error when updating to fab-classic 1.19

Hello friends,

after upgrading to fab-classic 1.19 I get the following error while trying to deploy:

Traceback (most recent call last):
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\main.py", line 769, in main
    execute(
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\tasks.py", line 377, in execute
    results[host] = _execute(
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\tasks.py", line 268, in _execute
    return task.run(*args, **kwargs)
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\tasks.py", line 168, in run
    return self.wrapped(*args, **kwargs)
  File "C:\progr\py3\foodrec\foodrec\fabfile.py", line 59, in full_deploy
    commit()
  File "C:\progr\py3\foodrec\foodrec\fabfile.py", line 21, in commit
    local("git add .")
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\operations.py", line 1249, in local
    with settings(hide('stdout', 'stderr') if capture else None):
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\context_managers.py", line 241, in settings
    return nested(*managers)
  File "c:\progr\py3\foodrec\venv\lib\site-packages\fabric\context_managers.py", line 40, in __init__
    self.enter_context(manager)
  File "C:\Users\serafeim\AppData\Local\Programs\Python\Python38-32\lib\contextlib.py", line 424, in enter_context
    _exit = _cm_type.__exit__
AttributeError: type object 'NoneType' has no attribute '__exit__'

My fabfile is the following:

from __future__ import with_statement
from fabric.api import env, cd, run, local, settings
import os


def black():
    "Run black"
    print("Running black...")
    local("black .")
    print("Black ok!")


def flake8():
    "Run flake8 checks"
    print("Check with flake8")
    # local("flake8 .")
    print("flake8 ok!")


def commit():
    local("git add .")
    with settings(warn_only=True):
        local("git commit")
    with settings(warn_only=True):
        local("git push origin master")
    print("Commit ok")


def pull():
    with cd(env.directory):
        run("git fetch origin")
        run("git merge origin/master")
    print("fetch / merge ok")


def work():
    "Do work on server (copy settings, migrate and run collect static)"
    with cd(env.directory):
        requirements_txt = "requirements/" + env.env + ".txt"
        if os.stat(requirements_txt).st_size > 0:
            virtualenv("pip install -r {0}".format(requirements_txt))
        virtualenv("python manage.py migrate")
        virtualenv("python manage.py update_permissions")
        virtualenv("python manage.py collectstatic --noinput")
        if env.env == "prod":
            virtualenv("python manage.py compres")


def touch_wsgi():
    print("Restarting uwsgi")
    if env.env == "prod":
        run(r"cat /home/serafeim/aismanager/gunicorn.pid | xargs kill -HUP")


def full_deploy():
    "Reformat - check - commit - pull - do work - and restart uwsgi"
    black()
    flake8()
    commit()
    pull()
    work()
    touch_wsgi()


def virtualenv(command):
    run(env.activate + "&&" + command)


def uat():
    "UAT settings"
    env.env = "uat"
    env.user = "serafeim"
    env.hosts = ["test.gr"]
    env.directory = "/home/serafeim/foodrec/foodrec"
    env.activate = "source /home/serafeim/foodrec/venv/bin/activate"


def prod():
    "PROD settings"
    env.env = "prod"
    env.user = "serafeim"
    env.hosts = [""]
    env.directory = "/home/serafeim/foodrec/foodrec"
    env.activate = "source /home/serafeim/foodrec/venv/bin/activate"

It seems that there's something fishy while trying to run local when encapsulating in the settings context manager...

I am using Python 3.8.1 and the command I run is fab uat full_deploy. Notice that this works flawlessly with fab-classic 1.18.1.

TIA

pexpect does not work fabric-classic

Migrating from fabric3 to fabric 1.18.0 breaks import of pexpect as fabric api put method requires mode value in octal and it is putting pexpect module with put with integer mode values.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/fabric/main.py", line 685, in main
    docstring, callables, default = load_fabfile(fabfile)
  File "/usr/local/lib/python3.8/dist-packages/fabric/main.py", line 177, in load_fabfile
    imported = importer(os.path.splitext(fabfile)[0])
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/gscorrea/fabfilelab.py", line 23, in <module>
    from ilogue.fexpect import expect, expecting, erun, esudo
  File "/usr/local/lib/python3.8/dist-packages/ilogue/fexpect/__init__.py", line 1, in <module>
    from ilogue.fexpect.api import expect, expecting, run, sudo, local
  File "/usr/local/lib/python3.8/dist-packages/ilogue/fexpect/api.py", line 2, in <module>
    from ilogue.fexpect.internals import wrapExpectations, wrapExpectationsLocal, ExpectationContext
  File "/usr/local/lib/python3.8/dist-packages/ilogue/fexpect/internals.py", line 23
    fabric.api.put(pexpect_module,'/tmp/', mode=0777)
                                                   ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

fab --show=stdout does not return execution output

WIth fabric 1.14 and fabric3 I only need to add --show=stdout to get the stdout output to the host system and now with fabric 1.18.0 it does not show the output anymore and I had to add status (--show,stdout,status) to get the execution output.

Load flakiness retry logic mismatches paramiko-ng

fab-classic/fabric/network.py

Lines 496 to 506 in eb89992

    
           # If we get SSHExceptionError and the exception message indicates 
        
           # SSH protocol banner read failures, assume it's caused by the 
        
           # server load and try again. 
        
           # 
        
           # If we are using a gateway, we will get a ChannelException if 
        
           # connection to the downstream host fails. We should retry. 
        
           if (e.__class__ is ssh.SSHException and msg == 'Error reading SSH protocol banner') \ 
        
              or e.__class__ is ssh.ChannelException: 
        
               if _tried_enough(tries): 
        
                   raise NetworkError(msg, e) 
        
               continue

specifically the msg == 'Error reading SSH protocol banner' seems to be too strictly checking the message.

I had a long-running @parallel fabric thing crash with this stacktrace:

Exception: Error reading SSH protocol banner
Traceback (most recent call last):
  File ".conda-env/lib/python3.9/site-packages/paramiko/transport.py", line 2049, in _check_banner
    buf = self.packetizer.readline(timeout)
  File ".conda-env/lib/python3.9/site-packages/paramiko/packet.py", line 360, in readline
    buf += self._read_timeout(timeout)
  File "conda-env/lib/python3.9/site-packages/paramiko/packet.py", line 575, in _read_timeout
    raise EOFError()
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".conda-env/lib/python3.9/site-packages/paramiko/transport.py", line 1904, in run
    self._check_banner()
  File ".conda-env/lib/python3.9/site-packages/paramiko/transport.py", line 2053, in _check_banner
    raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
Exception: Error reading SSH protocol banner[Errno 104] Connection reset by peer
Traceback (most recent call last):
  File ".conda-env/lib/python3.9/site-packages/paramiko/transport.py", line 2049, in _check_banner
    buf = self.packetizer.readline(timeout)
  File ".conda-env/lib/python3.9/site-packages/paramiko/packet.py", line 360, in readline
    buf += self._read_timeout(timeout)
  File ".conda-env/lib/python3.9/site-packages/paramiko/packet.py", line 573, in _read_timeout
    x = self.__socket.recv(128)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/centos/src/project_data/federation_pit-1175/firesim/.conda-env/lib/python3.9/site-packages/paramiko/transport.py", line 1904, in run
    self._check_banner()
  File "/home/centos/src/project_data/federation_pit-1175/firesim/.conda-env/lib/python3.9/site-packages/paramiko/transport.py", line 2053, in _check_banner
    raise SSHException(
paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer
Fatal error: Needed to prompt for a connection or sudo password (host: 10.2.0.5), but input would be ambiguous in parallel mode
Aborting.

I have env.connection_attempts = 10 and I only see three nested exceptions. I'm also using key-based auth. The last one:

paramiko.ssh_exception.SSHException: Error reading SSH protocol banner[Errno 104] Connection reset by peer

I'm wondering if the SSHException message is ending up with more stuff in it and we changed the referenced code to be 'Error reading SSH protocol banner' in msg it would correctly retry in this case. @ploxiln would you consider this a 'bug fix' or am I reaching too far?

	# If we get SSHExceptionError and the exception message indicates
	# SSH protocol banner read failures, assume it's caused by the
	# server load and try again.
	#
	# If we are using a gateway, we will get a ChannelException if
	# connection to the downstream host fails. We should retry.
	if (e.__class__ is ssh.SSHException and msg == 'Error reading SSH protocol banner') \
	or e.__class__ is ssh.ChannelException:
	if _tried_enough(tries):
	raise NetworkError(msg, e)
	continue