olipo186 / git-auto-deploy Goto Github PK

I think we should add systemd service alongside sys-v init file

As I described in the future README, script should fetch basic configuration from Docker ENV variables, even if no config file is specified.

Could you help me to add this feature to the script? I guess we need to add to get_config() something like this:

• not raise exceptions on lines 403 and 410 if there is env variable AUTODEPLOY_URL
• add entry to beginning of self._config['repositories'] list from all env variables before cycle on 418 begins

Am I on the right way?

I would like to have one server, one GitAutoDeploy script listening on one port that deploys to multiple sites (folders). The standard way of setting up a web server with multiple sites is to separate them by users. Each user has their own key, repo and folder respectively.

To allow GitDeploy to do this, the only ways are to either. A) Set up multiple GitAutoDeploy scripts on different ports (8001, 8002, 8003), under each user, which, my guess will increase the CPU usage quite a bit? Or, B) Set up one GitDeploy user and allow access to the other users web folder, which, then the owner of the files is the GitDeploy user because he's the one doing the pulling. I would assume this is also a security risk. (Allowing one user write to multiple users documents.)

Is my thinking all wrong or is this really just designed for one website/repo/server?

Hi.

I often get the following error:

fatal: Unable to create '/project/.git/index.lock': File exists.

If no other git process is currently running, this probably means a
git process crashed in this repository earlier. Make sure no other git
process is running and remove the file manually to continue.
Pull result: 128

And I don't see any reason of that. Even if previous time I get successful pull and deploy then next time it can fail again.

My solution was check exit code and to try several times if it fails. And added 5 sec pause between tries. Like:

if self.pull(path, branch) == 0 or self.pull(path, branch) == 0 or \
   self.pull(path, branch) == 0 or self.pull(path, branch) == 0:
    self.deploy(path)
else:
    print "Error ..."

Two times are often enough. Four times are always enough.

1. Could you suppose a reason of .git/indexlock failure?
2. Do you know a better solution?
3. If I fix by this way in my branch is it worth to merge it to your repository?

Regards, Sergei.

Changing the port in config.json does not affect the deployed server. Also the default "config.json" is not
read, you need to pass in the path on the cli.

See issue #88

When adding an SSH URL from GitLab to the config file it seems to switch the URL and uses the HTTP URL instead which requires a Username/Password.

Example:
[email protected]:username/repo.git

when i'm trying to run the script I get this error

File "GitAutoDeploy.py", line 65
finally:
^
SyntaxError: invalid syntax

any thought? I don't know anything about python, looked the script but can't figure it out.
Could be a version problem?

thanks a lot

https://github.com/olipo186/Git-Auto-Deploy/blob/master/docs/Configuration.md#repositories under path

you mean when a pull is executed, not a "clone" right?

allow configuration of setting uid/gid per repo

When you try enable enable this systemd unit you got:

The unit files have no [Install] section. They are not meant to be enabled
using systemctl.
Possible reasons for having this kind of units are:
1) A unit may be statically enabled by being symlinked from another unit's
   .wants/ or .requires/ directory.
2) A unit's purpose may be to act as a helper for some other unit which has
   a requirement dependency on it.
3) A unit may be started when needed via activation (socket, path, timer,
   D-Bus, udev, scripted systemctl call, ...).

There were several changes to run as a user other than root, these would have been much easier if the following variables and command line args had used each other.

DAEMON_UID=root
DAEMON_GID=root
RUNDIR=/var/run/$NAME
PIDFILE=/var/run/$NAME.pid
PWD=/opt/Git-Auto-Deploy/
OPTIONS="--daemon-mode --quiet --pid-file /var/run/git-auto-deploy.pid --log-file /var/log/git-auto-deploy.log"
USER=root

For instance the PIDFILE is not used in OPTIONS, and running as a user is a pain when forced into /var/run. Also the logfile is forced to /var/log, which is not accessible when running as a user.

The changes I made seem to run nicely and clean up the above vars. However they may have side-effects on other systems or configs so no PR.

I have many different sites with corresponding repositories, the config looks like it might support this usage, but the docs don't mention it.

Seems GitLab integration is not working correctly. Getting following error from webhook:

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 33991)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
    method()
  File "GitAutoDeploy.py", line 117, in do_POST
    repo_urls = self.get_repo_urls_from_request()
  File "GitAutoDeploy.py", line 169, in get_repo_urls_from_request
    elif user_agent.lower().find('bitbucket') != -1:
AttributeError: 'NoneType' object has no attribute 'lower'
----------------------------------------

Seems the header detection is not functioning correctly as changing the method to following works fine:

    def get_repo_urls_from_request(self):
        """Parses the incoming request and extracts all possible URLs to the repository in question. Since repos can
        have both ssh://, git:// and https:// URIs, and we don't know which of them is specified in the config, we need
        to collect and compare them all."""
        import json
        #content_type = self.headers.getheader('content-type')
        length = int(self.headers.getheader('content-length'))
        body = self.rfile.read(length)
        data = json.loads(body)
        repo_urls = []
        if not 'repository' in data:
            print "ERROR - Unable to recognize data format"
            return repo_urls
        for k in ['url', 'git_http_url', 'git_ssh_url', 'push_data']:
            if k in data['repository']:
                repo_urls.append(data['repository'][k])
        return repo_urls

Hi

I try to install all the dependencies in the hook by adding bower install into the deploy.sh whose absolute path (/home/son/preprod/deploy.sh) is put into the "deploy" path in the config file. However, I got this error message when the hook run

/bin/sh: 1: /home/son/preprod/deploy.sh: Permission denied

Does anyone have an idea why ?
I have tried doing a cd before running the script but still got the same error message.

Thanks,
Son.

Hi

I got it up and running towards my GitLab repos, but every time I do a push that triggers the git pull from Git-Auto-Deploy, it asks for username and password. I cannot find anything in the documentation that explains how to get around this. Maybe I'm missing out on something here, but please explain how to do this. Shall I add something in the conf.json file?
Thanks.

It is found when running without the --log-file option. Here is the stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/git-auto-deploy/gitautodeploy/__main__.py", line 5, in <module>
    main()
  File "gitautodeploy/gitautodeploy.py", line 600, in main
    app.run()
  File "gitautodeploy/gitautodeploy.py", line 507, in run
    log_file_path = os.path.expanduser(self._config['logfilepath'])
  File "/usr/lib/python2.7/posixpath.py", line 254, in expanduser
    if not path.startswith('~'):
AttributeError: 'NoneType' object has no attribute 'startswith'

It looks like default options have been commented out. This doesn't sound like the root cause. It is just uncovering the issue.
v0.2.1...v0.2.2#diff-57696f53643a33ca14f7fd021378ba75R418

I've set the url in the config as git url and using --sshkey-gen argument to start the daemon, and it shows me this error. It is working fine with HTTPS url.

Traceback (most recent call last):
File "GitAutoDeploy.py", line 661, in
app.run()
File "GitAutoDeploy.py", line 585, in run
self.ssh_key_scan()
File "GitAutoDeploy.py", line 480, in ssh_key_scan
port = repository['port']
KeyError: 'port'

I tried to start it by Docker, but it exited on error code 0 without any messages or error codes.

> docker-compose up
Starting kmtravel_deployer_1
Attaching to kmtravel_deployer_1
kmtravel_deployer_1 exited with code 0

docker-compose.yml seem like:

deployer:
  image: qnch/github-gitlab-auto-deploy
  environment:
    GAD_REPO_URL: [email protected]:my/repo.git
    GAD_REPO_DEPLOY: pwd

I know it propably will need SSH keys and maybe some other settings, but it should tell what it needs or print some error messages.

This product looks good, but it likely doesn't work well with Gitlab.

I didn't use docker for installation, everything is configured from the Makefile, gitautodeploy is launched using /etc/init.d/gitautodeploy start

I get this message on start:

Directory /home/deploy/www/test not found
Cloning into '/home/deploy/www/test'...
remote: Counting objects: 12, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 12 (delta 2), reused 0 (delta 0)
Receiving objects: 100% (12/12), 261.06 KiB | 0 bytes/s, done.
Resolving deltas: 100% (2/2), done.
Checking connectivity... done.
Repository [email protected]:fronteed/deploy.git successfully cloned
Starting Git Auto Deploy in daemon mode

As a result, repo is successfully cloned into /home/deploy/www/test/!

Here's the code from /opt/Git-Auto-Deploy/GitAutoDeploy.conf.json:

{
        "pidfilepath": "/var/run/gitautodeploy/gitautodeploy.pid",
        "host": "localhost",
        "port": 8001,
        "global_deploy": [
                "echo Deploy started!",
                "echo Deploy completed!"
        ],
        "repositories":
        [{
                "url": "[email protected]:fronteed/test.git",
                "branch": "master",
                "path": "/home/deploy/www/test"
        }]
}

I've a droplet at DO (digitalocean.com), so my host is the static IP, which looks like http://m.y.i.p:8001 (your-host:8001). I'm trying to paste this address into the URL field in "web hooks" section of my project at Gitlab.com. I get an error "Hook execution failed. Ensure hook URL is correct and service is up". When I push updates into repo, I don't see any changes at /home/deploy/www/test 😞

Summary:

• public keys are likely ok, since repo is nicely cloned at the beginning.
• web hook at 8001 port doesn't work for Gitlab, while it's reachable and open.

I'm using Ubuntu 14.04, Nginx and Git v2.6.3.

ps -ef | grep python:

root    30370    1  0 16:22 ?     00:00:00 python /opt/Git-Auto-Deploy/GitAutoDeploy.py --daemon-mode

lsof -i :8001

COMMAND   PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
python  30370 root    3u  IPv4 17339996      0t0  TCP *:8001 (LISTEN)

Am I doing something wrong?

deploy script for the master branch runs when the webhook for develop gets called.

It would be nice to have a project logo. @olipo186 do you have any preferences?

add python setup.py file to make this project to a python package - pip install git-auto-deploy its also will support auto installation to /usr/bin or /otp

I've set up the server and started it as a daemon when I do a pull I see this output on the daemon:

ec2-52-28-30-194.eu-central-1.compute.amazonaws.com - - [17/Aug/2015 09:40:49] "POST / HTTP/1.1" 200 -

but no pull il happening.
the repo is always the same on the server.

any thought?

thanks

We're planning on creating a Debian package, but in order to do so we need to know the licence that this project has. @olipo186 You're the creator (iirc) of this project, so it's up to you to decide it.

We need to consider to use argparse

https://github.com/olipo186/Git-Auto-Deploy/pull/33/files#r37361447

Use pep8 code docstrings convention. This will allow automatic code documentation using Sphinx doc

Thank you for the scripts its really helpful

All is working fine in normal mode like if i start process with
python gitautodeploy/

On post push request from github automatically deploying as configuration but, If I start the process daemon mode using this command its not working. As I am using this script in Digital ocean server i need to run the process in daemon mode.
python gitautodeploy/ --daemon-mode

Error are given below
---------------------------------------- 2016-04-08 17:59:34,275 [INFO ] 2016-04-08 17:59:34,275 [INFO ] Exception happened during processing of request from 2016-04-08 17:59:34,275 [INFO ] 2016-04-08 17:59:34,276 [INFO ] ('192.30.252.41', 42233) 2016-04-08 17:59:34,276 [INFO ] 2016-04-08 17:59:34,276 [ERROR] Traceback (most recent call last): 2016-04-08 17:59:34,276 [ERROR] File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock 2016-04-08 17:59:34,277 [ERROR] self.process_request(request, client_address) 2016-04-08 17:59:34,277 [ERROR] File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request 2016-04-08 17:59:34,277 [ERROR] self.finish_request(request, client_address) 2016-04-08 17:59:34,277 [ERROR] File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request 2016-04-08 17:59:34,278 [ERROR] self.RequestHandlerClass(request, client_address, self) 2016-04-08 17:59:34,279 [ERROR] File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__ 2016-04-08 17:59:34,280 [ERROR] self.handle() 2016-04-08 17:59:34,280 [ERROR] File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle 2016-04-08 17:59:34,280 [ERROR] self.handle_one_request() 2016-04-08 17:59:34,280 [ERROR] File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request 2016-04-08 17:59:34,280 [ERROR] method() 2016-04-08 17:59:34,280 [ERROR] File "gitautodeploy/httpserver.py", line 28, in do_POST 2016-04-08 17:59:34,281 [ERROR] File "gitautodeploy/httpserver.py", line 72, in figure_out_service_from_request 2016-04-08 17:59:34,281 [ERROR] ImportError: No module named parsers 2016-04-08 17:59:34,281 [INFO ] ----------------------------------------

Please help me out to figure and solve this problem.

Thank you in advance

How do I use my deployment key so the server doesn't cause git to ask for a sign-in?

Are your supporting Gitlab CI webhooks? I had your script working great with my Gitlab instance but I was thinking that maybe it is a better idea for me listen to Gitlab CI webhooks and auto deploy only if the build_status is success.

Thanks for you awesome script!

I really like the PPA for ubuntu packages.
Could you please add packages for Ubuntu Xenial 16.04.

Thanks!

Hi, just found your nice fork of this code, after creating my own :) duh!

I have some of my projects on Bitbucket, so .. I got this:
https://github.com/mxcoder/GitAutoDeploy

I think it would be great if we can collaborate on adding bitbucket's hooks to your version.
Please let me know if you are interested on it, and if I can help you anyhow, I'm not a python dev, but well, just let me know.

In case you're interested, the "bad" parts of bitbucket is that it uses application/json for the pullrequest hook, and form-urlencoded for the push hook. So, had to deal with that.

Regards!

Seems when running the following crontab it cannot find the config file in Ubuntu 14.04
@reboot /usr/bin/python /path/to/GitAutoDeploy.py --daemon-mode --quiet

The Error:

Could not load ./GitAutoDeploy.conf.json file

Traceback (most recent call last):
  File "/path/to/GitAutoDeploy.py", line 665, in <module>
    app.run()
  File "/path/to/GitAutoDeploy.py", line 602, in run
    self.get_config()
  File "/path/to/GitAutoDeploy.py", line 410, in get_config
    raise e
IOError: [Errno 2] No such file or directory: './GitAutoDeploy.conf.json'

There are certain cases when the deploy script will end with an error or will output something.
If that is the case, and sending via email is enabled, send the output and the exit code via email.

• Waiting for https://gitlab.com/gitlab-org/gitlab-ce/issues/590

I'll add a Makefile to the project so users would only need to do make install to get everything up and running.

The Makefile should cover the following things:

• Install files in the correct paths
• Detect init system (initV, upstart, systemd for now)
• Install the right init script and enable it.
• Update the README

We're attempting to set up GAD to work with our gitlab server. We enabled the webhook in gitlab for push events and have properly connected it to GAD. When testing the webhook or manually pushing to our dev branch, we get an error in the GAD log:

2016-04-12 21:02:14,722 [ERROR]  File "/usr/lib/python2.7/dist-packages/gitautodeploy/httpserver.py", line 50, in do_POST
2016-04-12 21:02:14,722 [ERROR]  File "/usr/lib/python2.7/dist-packages/gitautodeploy/httpserver.py", line 50, in do_POST
2016-04-12 21:02:14,722 [ERROR]  logger.warning('Unable to find any of the repository URLs in the config: %s' % ', '.join(repo_urls))
2016-04-12 21:02:14,722 [ERROR]  logger.warning('Unable to find any of the repository URLs in the config: %s' % ', '.join(repo_urls))
2016-04-12 21:02:14,722 [ERROR]  NameError: global name 'repo_urls' is not defined
2016-04-12 21:02:14,722 [ERROR]  NameError: global name 'repo_urls' is not defined

We haven't done anything unusual to our gitlab server that would cause it to submit malformed requests, I don't think.

This project appears as a fork of logsol/GitHub-Auto-Deploy, but the projects have diverted so much that they are not even similar anymore. Ask a GitHub admin (from contact area) to unlink this project from the other one.

Today README.md looks like documentation for command line utility. But it is Docker image and it should be used differentely.

So, is is excess to have in README:

• dependencies of Python utility
• configuration by copying files
• how to start daemon (it should start automatically in this image)

But in README for Docker image should be:

• example how to run it (docker run ...)
• example how to configure it by docker run parameters
• recommanded way how to link it with application container

There are two issues with the init script:

1. It sends log output to console, to whoever started/restarted it first/last
2. It doesn't properly detach the process, so when you disconnect from the machine the process dies

Hi,
I'm trying to execute a deployment command after the git pull, but I'm not getting to do it. I put a command in the "deploy" option, in repository configuration, but nothing occurs, just the pull. I am doing anything wrong or could it be a bug?

Thanks!

I'd like to run a script that corrects permissions after deploying. Is it possible to call that through this tool?

It would be useful if this produced a log file with everything it does (git pull results, deploy command statuses, etc)

When using the Debian/Sys-v init script I get the following error:

Starting Git Auto Deploy in daemon mode
Traceback (most recent call last):                                                                                                                                                                                                       
  File "/opt/Gitlab_Auto_Deploy/GitAutoDeploy.py", line 616, in <module>
    app.run()
  File "/opt/Gitlab_Auto_Deploy/GitAutoDeploy.py", line 559, in run
    GitAutoDeploy.create_daemon()
  File "/opt/Gitlab_Auto_Deploy/GitAutoDeploy.py", line 504, in create_daemon
    os.chdir('/home/tvk')
OSError: [Errno 2] No such file or directory: '/home/tvk'

If I set up a repository in the configuration with a branch as in:

    {
      "url": "http://gitlab/playground/deploy.git",
      "branch": "master",
      "deploy": "echo deploying"
    }

And then I do a push of commits on the dev branch, a deploy is still triggered.

when running the script it stops with this error:
IOError: [Errno 2] No such file or directory: u'/var/run/gitautodeploy/gitautodeploy.pid'

I'm not running the script as sudo, don't have that access on the server.

Sorry I don't know anything of this stuff.

thanks a lot