pyinfra-dev / pyinfra Goto Github PK

Directories get made, but files are not copied?

Ie inventory.group('master'). Returns list of host objects.

Don't re-calculate over and over.

1. There is already a number of similar tools, far more advanced than this one, isn't it easier to just write a Salt formula or Puppet recipe?
2. Docker + Mesos & Kubernetes solves the problem even better, why not to choose this way?

Rather than just --key-password BLAH.

When deploying to lots of hosts, using lots of files, pyinfra often hits the file open limit. This is because a new file object is opened per-file, per-host (so uploading the same file in parallel work). Use Python's resource module to guestimate good values for config.PARALLEL.

During an operation, possible open files/host:

1. Main one for commands
2. Second SFTP connection
3. Local file for SFTP
4. Two per nested local Popen commands (callbacks)
5. Nested fact commands (callbacks) uses main SSH connection

Max files / host should be 3 - which is SSH + Popen. SFTP uses main SSH connection and file handle is closed immediately after - so will never clash with Popen/etc.

Track runlevels enabled/disabled such that init modules enabled=True/False kwargs can become stateful (currently always generate commands).

Eg support lists: pyinfra -i inventories/dev.py --run apt.packages [htop,iftop],latest=true

Eg 00 07 * * * ENV=production PYTHONPATH=/opt/myapp /opt/envs/myapp/bin/python -m myapp will always echo to end of file, even if line is present.

Likely issues matching with the *'s.

For use in CLI mode: from pyinfra import inventory in inventory data and deploy scripts.

Ie enable passing Something={{ host.data.env }} as an argument, such that multiple "result" strings get bunched as one operation (because the op_hash will be the same). Use jinja2 w/host arg to convert string args into final format before pushing to state.

I have been running simple pyinfra deploy scripts like this one
for doing my daily apt-get update/upgrade combo:

apt.py

from pyinfra.modules import apt
from pyinfra.modules import server

apt.update()
apt.upgrade()
server.shell(commands="apt-get -y autoremove")

initially I always ran it as the root user, like so

pyinfra -i somehost --user root -v apt.py

ie. using simply somehost's hostname, and taking advantage of the
fact, that I have an ssh-agent running, ie. I am not asked for a password or
key phrase (whether it's a good thing to allow ssh root access, is another question,
but it certainly makes my adminstrator's life easier).

Then I got bored of always having to type --user root, so instead I introduced
simple inventories (one for each host I use so far):

somehost.py

HOSTS = [
    ('somehost', {'ssh_user': "root"})
]

that way I can just call:

pyinfra -i somehost.py apt.py -v

I guess it makes sense to have ssh_user set as "root", because most of
the operations I do on a host require root permissions.

However not always: a similar emacs/cask update I typically do as user rx
(rather then root):

cd ~/.emacs.d && EMACS="emacs" ~/cfg/e/cask/bin/cask outdated

Now what I would like is: just call

pyinfra -i somehost.py -v cask.py

given that I have prepared a similar cask deploy script:

cask.py

from pyinfra.modules import server

server.shell(chdir="~/.emacs.d"
            , commands='EMACS="emacs" ~/cfg/e/cask/bin/cask update'
            , user="rx")

Ie. I want to just use the somehost.py inventory, and have my cask.py deploy script
figure out, that a certain task requires user "rx" only.
Note that this is my wishlist thinking: the server.shell command doesn't
currently have a user=... parameter

I do not want to have to think for every little task:

• oh, apt operations require root permissions, thus call them with an inventory
  that has 'ssh_user': "root"
• cask operations I do as user "rx", thus call them with another inventory
  that has 'ssh_user': "rx"

Rather I want the individual deploy scripts/operations therein be able
to take a user param.

currently what I am resorting to in the case of cask above:

def caskUpdate():
    server.shell(commands='su - rx -c "cd ~/.emacs.d && EMACS=emacs ~/cfg/e/cask/bin/cask update"')

caskUpdate()

which is not really the style I want to write my operations.

Maybe this is possible with pyfinfra alreay, and I just haven't discovered it yet?

Also on my wishlist: maybe create a pyinfra mailing list, where I could ask these
kind of user questions ?

Not that in (haskell) propellor, which I am also using, this
is possible: the userScriptPropterty
there takes among other params a User param, cf eg.

http://hackage.haskell.org/package/propellor-3.0.5/docs/Propellor-Property-Cmd.html#v:userScriptProperty

ie. in propellor I just have

caskUpdate :: Property UnixLike
caskUpdate = propertyList "cask update" $ props
             & userScriptProperty (User "rx")
             [
              "(cd ~/.emacs.d && EMACS=\"emacs\" ~/cfg/e/cask/bin/cask update)"
             ] `assume` MadeChange

Thanks for creating pyinfra in the first place, and keep up your good work.

Ie during deploy build error: highlight the target host that failed

Currently will override each other, and one without data may override one with data.

Extensions supply operations & facts. Need spec/example & bootstrapping support in pyinfra.

Parsing is still iffy, needs better support - JSON?

• Package manage w/ npm.packages w/ optional dir arg (ie non-global install)

/ in replacement lines breaks the expression.

So restarted=something.changed doesn't generate multiple ops where the value changes.

HookError should be hook.Error.

python.execute still suffers the global name issue in #26, except that there's no hackfix because the function is essentially copied out of the namespace.

Current fix is just to do imports inside the function...

Same as sudo/sudo_user/su_user, etc - add get_pty to paramiko commands. Default to False.

Needed for certain command line tools that require a terminal because they're annoying like that.

Currently ignoring users SSH keys or ssh-agent added keys.

Annoyingly both yum and rpm will install multiple versions of the same package, so we need to handle that.

apt-get upgrade <package> != upgrade

pyinfra.api.deploy would be similar to the operation module, but @deploy wrapped functions define groups of operations, a "deploy".

Need spec/example/bootstrapping support in pyinfra.

Using the following states does not result in the home directory being created. The use is created and /home/deploy is written in /etc/passwd, but the directory is not created.

    server.user(
        'deploy',
        group='deploy',
        ensure_home=True,
    )

Using pyinfra-0.1.dev26 (pip installed) on Ubuntu 16.04 x86_64 in a Linux container.

Just started using this project and liking it so far! Had a quick question: any recommend way of dealing with needing a sudo password for a user? Is just disabling it the way to go?

As an aside it would also be awesome if the deploy bailed out as soon as a command requires some input (like prompting for a sudo password) since at the moment it just hangs.

Investigate bunching commands up together - would be esp. effective in fact gathering. Problem is facts are bound with sudo/sudo_user as well, so need to handle that (ie can't just join all the commands together and expect it to work as currently).

Also, this might be considered over-optimisation, and as such has low priority.

iPython does this - means executing pyinfra in PowerShell will work as expected.

Look at distlib.

The following code, when executed with exec, raises a NameError:

hello = 'world'

def test():
    return hello

thing = test()

The "fix" is to exec the file twice, using the locals from the first as globals in the second: https://github.com/Fizzadar/pyinfra/blob/f12674e6ce5389f0dab19ea57a663ef8405661ab/pyinfra/api/util.py#L23

files.sync, files.put and possibly files.template can't handle binary data.

modules.common.packaging or something. Needs to support diffing packages against fact AND versions - so each op will need to pull any versions passed in (currently no version comparison, just installed/not).

Common bits from following to be merged:

• https://github.com/Fizzadar/pyinfra/blob/develop/pyinfra/modules/apt.py
• https://github.com/Fizzadar/pyinfra/blob/develop/pyinfra/modules/gem.py
• https://github.com/Fizzadar/pyinfra/blob/develop/pyinfra/modules/pip.py
• https://github.com/Fizzadar/pyinfra/blob/develop/pyinfra/modules/pkg.py
• https://github.com/Fizzadar/pyinfra/blob/develop/pyinfra/modules/yum.py

Ie:

files.sync(
    'some/dir',
    '/remote/dir',
    exclude='*.pyc'
)

Currently no version comparison, just package names(!)

Sometimes we upload a blank/empty file. Something to do with IO objects.

EDIT: because uploading from one IO object in parallel.