In parallel or serialized one-by-one (per hook id)?

We are going to build a lightweight CI process on top of webhook. The flow would be like this:

1. start webhook which gets webhooks of github on git push (hook id: "build-github-version")
2. user commits and pushes a file to github
3. github calls webhook
4. webhook (hook id: "build-github-version") does some time consuming build stuff (git checkout, docker build, ...)

Step 4) will take some time and eventually someone will commit in between while there is still a running hook command "build-github-version". We want to serialize all incoming requests per hook id to prevent concurrent executions of the hook command.

Is this a functionality webhook currently supports / would you consider it as a new feature? We can workaround the problem by using lock files etc., but it seems to me like a common issue when dealing with webhooks which execute more complex command.

I have a build server in the office that is not publically addressable.
To make it receive bitbucket web hooks I can use ngrok.
Does this sound like a good idea for your project ?

Hey there, I've used this on a centos server and it went smoothly, but I'm trying to run this on debian and while installing, I get the following error:

root@vagrant:~# go get github.com/adnanh/webhook
# golang.org/x/sys/unix
go/src/golang.org/x/sys/unix/asm.s:8 6a: No such file or directory: textflag.h

go version
go version go1.3.3 linux/amd64

uname -a
Linux vagrant 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux

Environment

root@vagrant:~# go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

I have the following in a dot rc:

export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

Hi,
I have following settings:

[
  {
    "id": "redeploy-webhook",
    "execute-command": "/home/joseluis/test/hooks/redeploy.sh",
    "command-working-directory": "/home/joseluis/test",
    "response-message": "I got the payload!"
  }
]

I request the hook:

$ curl http://localhost:9000/hooks/redeploy-webhook
I got the payload!

On the terminal I got this error:

$ webhook -hooks hooks.json -verbose 
[webhook] 2015/12/06 16:58:02 version 2.3.6 starting
[webhook] 2015/12/06 16:58:02 setting up os signal watcher
[webhook] 2015/12/06 16:58:02 attempting to load hooks from hooks.json
[webhook] 2015/12/06 16:58:02 loaded 1 hook(s) from file
[webhook] 2015/12/06 16:58:02   > redeploy-webhook
[webhook] 2015/12/06 16:58:02 os signal watcher ready
[webhook] 2015/12/06 16:58:02 starting insecure (http) webhook on :9000
[webhook] 2015/12/06 16:58:07 Started GET /hooks/redeploy-webhook
[webhook] 2015/12/06 16:58:07 redeploy-webhook got matched (1 time(s))
[webhook] 2015/12/06 16:58:07 redeploy-webhook hook triggered successfully
[webhook] 2015/12/06 16:58:07 Completed 200 OK in 265.558µs
[webhook] 2015/12/06 16:58:07 executing /home/joseluis/test/hooks/redeploy.sh 
(/home/joseluis/test/hooks/redeploy.sh) with arguments 
["/home/joseluis/test/hooks/redeploy.sh"] and environment [] 
using /home/joseluis/test as cwd
[webhook] 2015/12/06 16:58:07 command output: 
[webhook] 2015/12/06 16:58:07 error occurred: 
fork/exec /home/joseluis/test/hooks/redeploy.sh: exec format error
[webhook] 2015/12/06 16:58:07 finished handling redeploy-webhook

My simple echo "Hello" command in redeploy.sh doesn't execute.
What am I doing wrong?

Description of problem:
While working on the tests for #76, I ran into what appears to be another bug. If you have two hooks from bitbucket, then it breaks the program.

The code crashes here: https://github.com/adnanh/webhook/blob/master/webhook.go#L172-L179
But it is still unclear to me what breaks it.

How reproducible:
Always on develop & master branch

Steps to reproduce:

1. Modify the testing template with 2 hooks (identical or different) for bitbucket
2. Rerun tests with go test / goconvey
3. Test fails - observe the logs

ex of addition to hooks template:

  {
    "id": "bitbucket-called-twice",
    "execute-command": "{{ .Hookecho }}",
    "command-working-directory": "/",
    "include-command-output-in-response": true,
    "response-message": "success",
    "parse-parameters-as-json": [
      {
        "source": "payload",
        "name": "payload"
      }
    ],
    "trigger-rule": {
      "and": [
        {
          "match": {
            "type": "value",
            "parameter": {
              "source": "payload",
              "name": "payload.canon_url"
            },
            "value": "https://bitbucket.org"
          }
        },
        {
          "match": {
            "type": "value",
            "parameter": {
              "source": "payload",
              "name": "payload.repository.absolute_url"
            },
            "value": "/webhook/testing/"
          }
        },
        {
          "match": {
            "type": "value",
            "parameter": {
              "source": "payload",
              "name": "payload.commits.0.branch"
            },
            "value": "master"
          }
        }
      ]
    }
  },
  {
    "id": "bitbucket-called-twice",
    "execute-command": "{{ .Hookecho }}",
    "command-working-directory": "/",
    "include-command-output-in-response": true,
    "response-message": "success",
    "parse-parameters-as-json": [
      {
        "source": "payload",
        "name": "payload"
      }
    ],
    "trigger-rule": {
      "and": [
        {
          "match": {
            "type": "value",
            "parameter": {
              "source": "payload",
              "name": "payload.canon_url"
            },
            "value": "https://bitbucket.org"
          }
        },
        {
          "match": {
            "type": "value",
            "parameter": {
              "source": "payload",
              "name": "payload.repository.absolute_url"
            },
            "value": "/webhook/testing/"
          }
        },
        {
          "match": {
            "type": "value",
            "parameter": {
              "source": "payload",
              "name": "payload.commits.0.branch"
            },
            "value": "master"
          }
        }
      ]
    }
  },

Addition to hookHandlerTests:

    {
        "bitbucket",
        "bitbucket-called-twice",
        nil,
        `payload={"canon_url": "https://bitbucket.org","commits": [{"author": "marcus","branch": "master","files": [{"file": "somefile.py","type": "modified"}],"message": "Added some more things to somefile.py\n","node": "620ade18607a","parents": ["702c70160afc"],"raw_author": "Marcus Bertrand <[email protected]>","raw_node": "620ade18607ac42d872b568bb92acaa9a28620e9","revision": null,"size": -1,"timestamp": "2012-05-30 05:58:56","utctimestamp": "2014-11-07 15:19:02+00:00"}],"repository": {"absolute_url": "/webhook/testing/","fork": false,"is_private": true,"name": "Project X","owner": "marcus","scm": "git","slug": "project-x","website": "https://atlassian.com/"},"user": "marcus"}`,
        true,
        http.StatusOK,
        `{"message":"success"}`,
    },

Actual result:
The other hook handlers have been commented (for readability), but the template still contain their definitions.

When rerun with previous modifications:

> go test -verbose
[webhook] 2016/06/09 17:41:45 version 2.3.8 starting
[webhook] 2016/06/09 17:41:45 setting up os signal watcher
[webhook] 2016/06/09 17:41:45 attempting to load hooks from /tmp/webhook-config-559176871/hooks.json
[webhook] 2016/06/09 17:41:45 os signal watcher ready
[webhook] 2016/06/09 17:41:45 loaded 6 hook(s) from file
[webhook] 2016/06/09 17:41:45   > github
[webhook] 2016/06/09 17:41:45   > bitbucket-called-twice
[webhook] 2016/06/09 17:41:45   > bitbucket-called-twice
[webhook] 2016/06/09 17:41:45   > bitbucket
[webhook] 2016/06/09 17:41:45   > gitlab
[webhook] 2016/06/09 17:41:45   > uncalled-hook
[webhook] 2016/06/09 17:41:45 starting insecure (http) webhook on 127.0.0.1:39343
[webhook] 2016/06/09 17:41:45 Started GET /
[webhook] 2016/06/09 17:41:45 Completed 404 Not Found in 110.68µs
[webhook] 2016/06/09 17:41:45 Started POST /hooks/bitbucket-called-twice
[webhook] 2016/06/09 17:41:45 bitbucket-called-twice got matched (2 time(s))
[webhook] 2016/06/09 17:41:45       Hook: &{ID:bitbucket-called-twice ExecuteCommand:/tmp/hookecho-test-411381864/hookecho CommandWorkingDirectory:/ ResponseMessage:success ResponseHeaders:[] CaptureCommandOutput:true PassEnvironmentToCommand:[] PassArgumentsToCommand:[] JSONStringParameters:[{Source:payload Name:payload}] TriggerRule:0xc8200c0ee0}
[webhook] 2016/06/09 17:41:45 bitbucket-called-twice hook triggered successfully
[webhook] 2016/06/09 17:41:45 executing /tmp/hookecho-test-411381864/hookecho (/tmp/hookecho-test-411381864/hookecho) with arguments ["/tmp/hookecho-test-411381864/hookecho"] and environment [] using / as cwd
[webhook] 2016/06/09 17:41:45 command output: 
[webhook] 2016/06/09 17:41:45 finished handling bitbucket-called-twice
[webhook] 2016/06/09 17:41:45       Hook: &{ID:bitbucket-called-twice ExecuteCommand:/tmp/hookecho-test-411381864/hookecho CommandWorkingDirectory:/ ResponseMessage:success ResponseHeaders:[] CaptureCommandOutput:true PassEnvironmentToCommand:[] PassArgumentsToCommand:[] JSONStringParameters:[{Source:payload Name:payload}] TriggerRule:0xc8200c1020}
[webhook] 2016/06/09 17:41:45 error parsing JSON: invalid character 'm' looking for beginning of value
[webhook] 2016/06/09 17:41:45 http: multiple response.WriteHeader calls
[webhook] 2016/06/09 17:41:45 bitbucket-called-twice got matched (2 time(s)), but executed only 1 times. Please review previous log messages
[webhook] 2016/06/09 17:41:45 Completed 400 Bad Request in 3.249099ms
--- FAIL: TestWebhook (2.24s)
    webhook_test.go:77: failed "bitbucket" (id: bitbucket-called-twice):
        expected status: 200, response: {"message":"success"}
        got status: 200, response: {"message":"success"}error parsing JSON: invalid character 'm' looking for beginning of value
FAIL
exit status 1
FAIL    github.com/xlr-8/webhook    2.243s

The line:
[webhook] 2016/06/09 17:41:45 error parsing JSON: invalid character 'm' looking for beginning of value seems very suspicious to me, as both hooks are identical, I did not expect a different behaviour.

Expected results:
Have not the test failing but continuing (because more output is expected), but not have the program crashing due to JSON issue.

Additional info:

If the performance becomes an issue, we should consider switching to echo.

Hi, I just updated my local checkout to the latest version, but cannot build webhook anymore. I have previously built version 2.3.5 with the same steps.

$ git rev-parse master
8de1a519269b2171f925d085e744ddc7167effa8
$ go version
go version go1.3.3 linux/arm
$ go build
# _/.../git/webhook
./webhook.go:183: h.ParseJSONParameters(&headers, &query, &payload) used as value
./webhook.go:197: assignment count mismatch: 2 = 1
./webhook.go:238: assignment count mismatch: 2 = 1
./webhook.go:244: h.ExtractCommandArgumentsForEnv undefined (type *hook.Hook has no field or method ExtractCommandArgumentsForEnv) 

Hello,

I've packaged webhook for Debian:

https://packages.debian.org/sid/webhook

It'd be nice to mention this way of getting webhook ("apt install webhook") in the README.

If you prefer not to, please feel free to close the issue.

PS: FYI, I had to apply a couple of patches to the upstream code, see:

https://anonscm.debian.org/cgit/pkg-go/packages/webhook.git/tree/debian/patches

(mainly package paths and interfaces that have changed). Feel free to merge
whatever makes sense to you.

Take care and thansk

this can be useful for slack /slash command integration for the return message property

Here is the request I made to the server. I've used httpie during the tests.

$ http -v --form GET http://localhost:9000/hooks/date foo=bar
GET /hooks/date HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Content-Length: 7
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:9000
User-Agent: HTTPie/0.8.0

foo=bar

As you can see the Content-Type header contains the value application/x-www-form-urlencoded; charset=utf-8. In the code, you don't allow charset to be present in the header resulting in an empty payload.

When the executed command exits with a non-0 exit code, the http status code should not be 200.
This would allow for simper webhook monitoring.

Proposal

As projects and software industry evolve towards API, providing the possibility to trigger API call instead of running commands could be great. To do so relying on oauth mechanism, supported by github/bitbucket/slack, seems the most logical/straight forward.

Implementation

Oauth token

Because of the way outgoing hooks are implemented it seems easier to add a parameter to the URL such as: access_token cf. RFC 2.3, that will be then forwarded as Authorization: Bearer $TOKEN cf. RFC 2.1

Now I'm aware of the security threat that it creates to have the token in the URL. However I believe everyone should be aware of such condition (the webhook documentation could be adapted, to emphasise it), but given the current possibilities provided by github/bitbucket/slack, I don't see much more solution around it.
The use of SSL and also secrets is obviously advised.

Hook modifications

Adding API call, implies to have:

• Token (option):
  would be implemented in the hookHandler method if present
• URI to call:
  would be in the new struct of Hook on the same logic than ExecuteCommand
• Headers:
  would be in the new Argument struct on the same logic than PassArgumentsToCommand
• Body:
  same logic than Headers.

Token parameter in URI

As specified in the previous section.

New Hook structure

Modification of the hooks structure - addition of APICall and PassArgumentsToAPI:

type Hook struct {
    ID                       string          `json:"id,omitempty"`
    ExecuteCommand           string          `json:"execute-command,omitempty"`
    CommandWorkingDirectory  string          `json:"command-working-directory,omitempty"`
    APICall                  string          `json:"execute-api-call,omitempty"`
    ResponseMessage          string          `json:"response-message,omitempty"`
    ResponseHeaders          ResponseHeaders `json:"response-headers,omitempty"`
    CaptureCommandOutput     bool            `json:"include-command-output-in-response,omitempty"`
    PassEnvironmentToCommand []Argument      `json:"pass-environment-to-command,omitempty"`
    PassArgumentsToCommand   []Argument      `json:"pass-arguments-to-command,omitempty"`
    PassArgumentsToAPI       []Argument      `json:"pass-arguments-to-api,omitempty"`
    JSONStringParameters     []Argument      `json:"parse-parameters-as-json,omitempty"`
    TriggerRule              *Rules          `json:"trigger-rule,omitempty"`
}

New argument/source type

Introduce two new source types: apiheader & apibody.
Body would be like the default SourceString but would have to match apibody.
Headers will be key/value parameters that should eventually be concatenated to avoid breaking the definition of the Get method:

func (ha *Argument) Get(headers, query, payload *map[string]interface{}) (string, bool) {

That would give something like this as source:

const (
    SourceHeader        string = "header"
    SourceQuery         string = "url"
    SourcePayload       string = "payload"
    SourceString        string = "string"
    SourceAPIHeader     string = "apiheader"
    SourceAPIBody       string = "apibody"
    SourceEntirePayload string = "entire-payload"
    SourceEntireQuery   string = "entire-query"
    SourceEntireHeaders string = "entire-headers"
)

The Argument struct would also have to be adapted to read body/headers. So two new variables are added to reflect the new implementation Name string & Key string:

type Argument struct {
    Source  string `json:"source,omitempty"`
    Name    string `json:"name,omitempty"`
    Key     string `json:"key,omitempty"`
    Value   string `json:"value,omitempty"`
    EnvName string `json:"envname,omitempty"`
}

The rest would follow the same logic: extraction of arguments, re-using them in the hookHandler & hookHandle methods.

Testing

The testing of the outgoing call, cannot of course be tested fully. So the idea would be to mock webhooks from github/bitbucket or slack, create the API call and then call a mini http-server mocking answers based on: missing body, missing headers, 4xx/5xx return code, etc.

Backward compatibility

There should not be any problem of backward compatibility as it is an addition of feature, thus both should remain possible at any time.

Proposal

Deprecate the MatchRule.Regex field in favor of using MatchRule.Value.

{
  "match":
  {
    "type": "regex",
    "value": ".*",
    "parameter":
    {
      "source": "payload",
      "name": "ref"
    }
  }
}

I can't think of a good reason to keep Regex around as a stand-alone string field when the Value field can serve the same purpose.

Backward Compatibility

Keep MatchRule.Regex around for a time and simply copy it's value into MatchRule.Value.

Hello,

1/ Do you plan on returning an error code other than 200 when the command fail?
2/ Do you plan on adding an option to return the output of the command on the HTTP response

if no, would you accept code for any of these 2 features?

best

It looks like the changes introduced in revision ecbcf11 (issue #106) caused building to fail:

$ go test
# github.com/wjessop/webhook
./webhook.go:201: cannot range over errors (type error)
./webhook.go:261: cannot assign error to errors (type []error) in multiple assignment
./webhook.go:269: cannot assign error to errors (type []error) in multiple assignment
FAIL	github.com/wjessop/webhook [build failed]

It also looks like the tests don't pass, they still assume ParseJSONParameters returns a single error:

err := h.ParseJSONParameters(tt.headers, tt.query, tt.payload)

Current behavior in verbose mode:

λ webhook.exe -verbose 
[webhook] 2015/07/22 22:01:25 version 2.3.4 starting                                                            
[webhook] 2015/07/22 22:01:25 attempting to load hooks from hooks.json                                          
[webhook] 2015/07/22 22:01:25 couldn't load hooks from file! open hooks.json: The system cannot find the file specified.                                                                                                            
[webhook] 2015/07/22 22:01:25 starting insecure (http) webhook on :9000

Without the -verbose flag the program silently fails to find the hooks.json file (expectedly), but doesn't tell the user that. I think it would be helpful to fail immediately if no hooks file is found (and assuming -hotreload is false). What are your thoughts?

All it does is add ambiguity and makes our codebase dirty.

The usefulness of the feature is questionable and can be achieved by having a single hook with a script/program that will decide what needs to be done.

Hi, this looks really promising, thanks!

Is there a way to have a single id filter based on payload? I may have misread the docs
but it seems like each endpoint can have multiple filters but I can't expose multiple commands
via a single id (so it's not possible to put several commands at one web hook URL).

I'm hoping to use this with marathon, so can specify a single URL for all web hooks

(specced here for reference: https://mesosphere.github.io/marathon/docs/event-bus.html )

There are about a dozen different payloads sent to that URL and I'd like to ignore some,
and run different commands based on the payload fields.

I have a suggestion to use a generic parser configs, will choose what is more to taste.
URL: https://github.com/vstakhov/libucl#basic-structure
Support syntax: JSON, Compacted JSON, nginx style, YAML

• Go modules

ucl: parsed input in 0.3002 seconds
ucl: emitted config in 0.1174 seconds
ucl: emitted json in 0.1174 seconds
ucl: emitted compact json in 0.0991 seconds
ucl: emitted yaml in 0.1354 seconds

Proposal

Collapse the Parameter fields into the MatchRule struct:

type MatchRule struct {
    Regex     string   `json:"regex,omitempty"`
    Secret    string   `json:"secret,omitempty"`
    Source    string   `json:"source,omitempty"` // Parameter.Source
    Subject   string   `json:"name,omitempty"`   // Parameter.Name
    Type      string   `json:"type,omitempty"`
    Value     string   `json:"value,omitempty"`
}

Which would give us a simplified match definition:

"match":
{
  "type": "value",
  "source": "payload",
  "subject": "ref",
  "value": "refs/heads/master"
}

Backward Compatibility

It should be possible to leave the MatchRule.Parameter field in place for backward compatibility. We can just copy Parameter.Source and Parameter.Name into their MatchRule counterparts. We would declare Parameter as deprecated in the documentation and remove it at some later date.

The Argument Type

We would keep the Argument type for use with pass-arguments-to-command, etc.

It would be handy to be able to pass static string to the command that will be executed as a parameter, along with dynamic parameters...

Parameter should be specified using the following notation

{
    "source": "string",
    "name": "argument"
}

Hi,

just recently I noticed something, where I am not 100% sure if it has been the same all along (currently using 2.3.9).

I have for example to following hook:

[
  {
    "id": "test",
    "execute-command": "echo",
    "trigger-rule":
    {
      "match":
      {
        "type": "payload-hash-sha1",
        "secret": "yoursecret",
        "parameter":
        {
          "source": "url",
          "name": "token"
        }
      }
    }
  }
]

When I now call http://my-ip:9000/hooks/test (for example through my browser) I get the following answer displayed:
Hook rules were not satisfied.
and simultaneously webhook will log the following:

[webhook] 2016/06/27 15:07:32 test got matched (1 time(s))
[webhook] 2016/06/27 15:07:32 test got matched (1 time(s)), but didn't get triggered because the trigger rules were not satisfied
[webhook] 2016/06/27 15:07:32 Completed 200 OK in 119us

If I now change my url to http://my-ip:9000/hooks/test?token=test
I will now get the following response:

error evaluating hook: invalid payload signature 39b5b946e0d0144df4ea088c3e195c9118b86dd9

And this is basically telling me: "hey, you used the wrong secret! try this one instead!" (since the payload signature in the response is acutally the sha1 of my defined secret.

Is this a configuration mistake? or a bug?

Is there or will there be any kind of authentication?

Currently this possible use case is not appearing neither in the docs nor in examples:

    "trigger-rule":
    {
      "match":
      {
        "type": "value",
        "value": "123",
        "parameter":
        {
          "source": "url",
          "name": "token"
        }
      }
    }

I mean "source": "url" here. GET params are referred as query, which is what I have attempted to name them in the first place: { "source": "query", "name": "token" }. It took a while to figure out that url needs to be used instead of query (had to read the source for that).

Would be good all accepted values for source were listed somewhere in README.md or at least used in the example.

Thanks a lot for making this tool available @adnanh!

Hi, this tool is just what I'm looking for but am having difficulty getting it working, I've got a simple hooks.json:

[
  {
        "id": "test-hook",
        "execute-command": "/root/scripts/delme.sh",
        "trigger-rule":
        {
                "match":
                {
                        "type": "value",
                        "value": "development",
                        "parameter":
                        {
                                "source": "payload",
                                "name": "branch"
                        }
                }
        }
  }
]

when I POST the following:

{
"branch": "development",
"something": "else"
}

I get the following:

test-hook got matched (1 time(s)), but didn't get triggered because the trigger rules were not satisfied

Not sure what I'm doing wrong :(

Hello,

Could you add webhook builds to your github releases?
You can use tools like gox to create builds for multiple platforms without too much hassle (in theory).

I was going to submit changes (something like this) to enable Travis CI testing until I noticed that there's tons of Docker stuff in the existing .travis.yml file. So, I wanted to open a discussion about the scope of this repository.

There's no connection between Docker and webhook, and I don't think the files contributed by @almir belong in this repo. While they may be useful to some, they are out of place in this repo IMHO. I'd suggest @almir create his own webhook-docker repo for the Docker config. His Dockerfile could be updated to git-pull the repo before building. The webhook README or from the wiki pages could link to his repo to make it easier for people to find him work.

Additionally, I've had the misfortune of maintaining RPM specs in the past in a similar situation, and I don't care to go down that road again. I mean no offense to @timhughes, who did all the heavy lifting, but that's the distro package maintainer's job, and the spec file doesn't belong in the webhook repo. There's too may distros and too many variations; that noise belongs somewhere else.

I feel the same way about the init script contributed by @timhughes. Are we going to need to add and maintain a systemd config? nosh? upstart? Run away!!

If nothing else, I'd like to see these contributions moved to a contrib folder, which historically means "we're not maintaining these files, but you can try them if you want..good luck."

I'm only a contributor like everyone else (well, except @adnanh). I'm empathetic to @timhughes and @almir since I know the work that goes into creating and submitting their changes to this project. I don't mean to discourage people from contributing. I'm just worried that the repo is going to turn into a hodgepodge of hacks to get webhook running in everyone's favorite environment when those hacks really belong somewhere else.

It would be useful if multiple JSON files containing hooks could be loaded instead of one huge file.
A huge file is difficult to maintain and difficult to read.

This functionality can be implemented either by modifying the -hooks switch to accept array instead of string, or by specifying -hooks parameter multiple times (preferred way), for example:

webhook -verbose -hooks 1sthooksfile.json -hooks 2ndhooksfile.json -hooks 3rdhooksfile.json -hotreload

Cool work, thanks a lot!

Description of problem:
I'm currently testing webhook, and I found out, that you can't have multiple hooks triggered with one name. Theoretically you can, but in practice you can't, I'm still learning go, so please correct me, if there is something I did not get.

In your webhook.go code we can see this loop: https://github.com/adnanh/webhook/blob/master/webhook.go#L190-L242
Snippets:

if err != nil {
        msg := fmt.Sprintf("error parsing JSON: %s", err)
        log.Printf(msg)
        w.WriteHeader(http.StatusBadRequest)
        fmt.Fprintf(w, msg)
        return
}

if err != nil {
        msg := fmt.Sprintf("error evaluating hook: %s", err)
        log.Printf(msg)
        w.WriteHeader(http.StatusInternalServerError)
        fmt.Fprintf(w, msg)
        return
}

if ok {
        log.Printf("%s hook triggered successfully\n", h.ID)

        for _, responseHeader := range h.ResponseHeaders {
                w.Header().Set(responseHeader.Name, responseHeader.Value)
        }   

        if h.CaptureCommandOutput {
                response := handleHook(h, &headers, &query, &payload, &body)
                fmt.Fprintf(w, response)
        } else {
                go handleHook(h, &headers, &query, &payload, &body)
                fmt.Fprintf(w, h.ResponseMessage)
        }   

        return
}

Which has many return that break the for loop (whether it succeeded or not), I would suggest to have a less restrictive behaviour with continue and log an error message at the end if really nothing executed.

How reproducible:
Always

Steps to reproduce:

1. Create hooks.json with different hooks and 2 with the same name, but different command/arg
2. Rerun webhook if not hotreload
3. Do action to trigger the hook
4. Observe the logs

ex of hooks.json file used (I also varied the orders of hooks, just in case):

> cat hooks.json
[
  {
    "id": "webhook2",
    "execute-command": "/tmp/script2.sh",
    "pass-arguments-to-command":
    [
      {
        "source": "payload",
        "name": "head_commit.id"
      },
      {
        "source": "payload",
        "name": "head_commit.author.name"
      },
      {
        "source": "payload",
        "name": "head_commit.author.email"
      }
    ]
  },
  {
    "id": "webhook",
    "execute-command": "/tmp/script3.sh",
    "pass-arguments-to-command":
    [
      {
        "source": "payload",
        "name": "head_commit.id"
      },
      {
        "source": "payload",
        "name": "head_commit.author.name"
      },
      {
        "source": "payload",
        "name": "head_commit.author.email"
      }
    ]
  },
  {
    "id": "webhook",
    "execute-command": "/tmp/script1.sh",
    "pass-arguments-to-command":
    [
      {
        "source": "payload",
        "name": "head_commit.id"
      },
      {
        "source": "payload",
        "name": "head_commit.author.name"
      },
      {
        "source": "payload",
        "name": "head_commit.author.email"
      }
    ]
  }
]

Testing scripts look like this (just to know what is called with which arguments):

> tail -n +1 /tmp/script*
==> /tmp/script1.sh <==
#!/bin/bash

echo $0
echo $@

==> /tmp/script2.sh <==
#!/bin/bash

echo $0
echo $@

==> /tmp/script3.sh <==
#!/bin/bash

echo $0
echo $@

Expected results:
To have all matching hooks executed, because I don't want the execution to be stopped. Whether it ran fine or not (if typo in one, if service unavailable, if you just want multiple tasks run, etc).

Additional info:
Attached is a first draft of a patch suggested.
fix-return.patch.txt

It'd be great to be able to specify environment variables that will get set that aren't prefixed by HOOK_. The situation I just hit was needing to set HOME on a command. The underlying (puppet) libraries that were complaining about HOME not being set would have been a PITA to map HOOK_HOME to HOME in (probably requiring us to run our own forks).

https://github.com/urfave/negroni/ has had some commits lately both introducing a new Logger interface and reverting the changes.

It looks like in the end, backwards incompatible changes were introduced into github.com/urfave/negroni/ and this project needs to be updated to support future versions.

This should be reproducible in any environment that does not already have an older version of negroni, for example docker:

$ docker run golang go get github.com/adnanh/webhook
# github.com/adnanh/webhook
src/github.com/adnanh/webhook/webhook.go:114: l.Logger undefined (type *negroni.Logger has no field or method Logger)
src/github.com/adnanh/webhook/webhook.go:117: l.Logger undefined (type *negroni.Logger has no field or method Logger)

After commit 2947e5e, Webhook will abort the command execution if missing arguments were reference from within pass-arguments-to-command. The same can be said of the new environment arguments as well.

When passing arguments to a command via environment variables, Webhook isn't keeping the existing OS environment.

• Diff GitHub-GitLab Hook [1] [2]
  • gitlabhq-7.11 Web hooks

Hi, I've found https://hub.docker.com/r/adnanh/webhook/ and wanted to use it, to build my own container which would respond to web hooks. However I saw that in beginning of November after at first adding a description about the image, it got removed again few days later.

May I ask why the docker documentation got removed again?

Error "parsing JSON: couldn't retrieve argument for {Source:payload Name:payload}" work with BitBucket.
before everything was good
whats wrong?

Thanks

Specifically it would be useful to specify: Access-Control-Allow-Origin in the response so that the hook could be called from js served from a different location or port.

First off, thanks for publishing this project. It looks promising.

I ran into an issue trying to receive a POST hook from Bitbucket. Bitbucket sends a x-www-form-urlencoded POST with the JSON content in the payload form variable -- they don't send the JSON payload in the body. I was able to modify hookHandler() to decode the JSON, but I had to hardcode the payload map index:

} else if contentType == "application/x-www-form-urlencoded" {
    fd, err := url.ParseQuery(string(body))
    if err != nil {
        log.Printf("error parsing form payload %+v\n", err)
    } else {
        fv := helpers.ValuesToMap(fd)
        decoder := json.NewDecoder(strings.NewReader(fv["payload"].(string)))
        decoder.UseNumber()
        err := decoder.Decode(&payload)
        if err != nil {
            log.Printf("error parsing JSON payload %+v\n", err)
        }
    }
}

With that change, I'm able to match simple paths (canon_url and repository.absolute_url), but I'm unable to match commits.0.branch. I've yet to look into why.

I'm testing webhook with curl and sending Bitbucket's sample payload. Here's a simplified version of what I'm doing:

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" \
  -d 'payload={"canon_url": "https://bitbucket.org"}' \
  http://localhost:9000/hooks/example

I wanted to see if you can duplicate the form decoding problem. I also wanted to see what you think about adding another option to the hook definition format. I'm thinking something like form-variable would work. I would set that to payload, and webhook would know to access that form variable to find the JSON payload.

Hi,

first time user, maybe it's a conf issue, maybe a hook issue.
I tried both the docker and compile from source.

deploy/webhook -hooks ./deploy/deployhook.json -verbose

[webhook] 2016/06/09 09:27:33 version 2.3.8 starting
[webhook] 2016/06/09 09:27:33 setting up os signal watcher
[webhook] 2016/06/09 09:27:33 attempting to load hooks from ./deploy/deployhook.json
[webhook] 2016/06/09 09:27:33 os signal watcher ready
[webhook] 2016/06/09 09:27:33 loaded 1 hook(s) from file
[webhook] 2016/06/09 09:27:33 > deploy-webhook
[webhook] 2016/06/09 09:27:33 starting insecure (http) webhook on :9000

after each hook is triggered, I get:

[webhook] 2016/06/09 09:27:51 Started POST /
[webhook] 2016/06/09 09:27:51 Completed 404 Not Found in 141.541µs

My config is the following:

[
{
"id": "deploy-webhook",
"execute-command": "/home/test/deploy/deploy.sh",
"command-working-directory": "/home/test/www/"
}
]

Any idea what the error could be?

If you configure:

"trigger-rule": {
      "match": {
        "type": "payload-hash-sha1",
        "secret": "<secret>",
        "parameter": {
          "source": "header",
          "name": "X-Hub-Signature"
        }
      }
    }

And you make a curl request to the server, that contains an invalid X-Hub-Signature (the header must be present, but invalid)

curl -H 'X-Hub-Signature: dummy' http://webhook-app1.foo/echo-webhook

Then the server will respond:

error evaluating hook: invalid payload signature <signature>

Then repeating the curl with the returned signature will succeed:

curl -H 'X-Hub-Signature: <signature>' http://webhook-app1.foo/echo-webhook

Although useful for debugging, it also defeats the signature verification based security.

I'm using webhook in a docker container. The hook has to store some data from bitbucket into etcd. Is there any way I can access environment variables passed into the docker container (notably ETCDCTL_ENDPOINT in my case) from within my hook script? Right now I'm using a bit of a kludge by having a run.sh file save the relevant variables into a file, then launching webhook, and then accessing the contents of the file when the hook is invoked. I'd really like to define the hook - passing in environment variables along the lines of

"pass-parameter-to-command":
[
{
"source": "environment",
"name": "ETCDCTL_ENDPOINT"
}

or using pass-environment-to-command in some way?

Some webhooks allow you to return data in the response, e.g. hipchat webhooks

If include-command-output-in-response is set to true the output is wrapped inside a json, which prevents responding with a valid json as expected by the caller of the webhook.

An enhancement would be to allow raw-command-output-in-response which would simply pass all stdout content to the http response body, without wrapping it.

Hello, first thank you for writing this webhook, its fantastic and your documentation is top notch. Really helped me process events!

my issue is this, Im using this Go webhook as a client to execute a shell script based on a JIRA ticket webhook POST

ive setup Go-webhook with a hook.json

[
  {
    "id": "jira-new",
    "execute-command": "/root/golang/jira_out.sh",
    "command-working-directory": "/root/golang",
    "pass-arguments-to-command":
    [
        {
          "source": "payload",
          "name": "key"
        }
    ]
  }
]

have a shell script that outputs the JIRA issue ID # into a flat text file
cat /root/golang/jira_out.sh

#!/bin/sh
echo $1 > file.out

at this point I just want to test that my client is receiving a JSON from JIRA and spitting out the JIRA key # into a file, very basic stuff,

my JIRA json looks like this,

{  
   "expand":"renderedFields,names,schema,transitions,operations,editmeta,changelog,versionedRepresentations",
   "id":"10213",
   "self":"https://green.vagrant.local/rest/api/latest/issue/10213",
   "key":"POC-35",
   "fields":{  
      "issuetype":{  
         "self":"https://green.vagrant.local/rest/api/2/issuetype/10002",
         "id":"10002",
         "description":"A task that needs to be done.",
         "iconUrl":"https://green.vagrant.local/secure/viewavatar?size=xsmall&avatarId=10318&avatarType=issuetype",
         "name":"Task",
         "subtask":false,
         "avatarId":10318

I need to get the key (key: POC_35)

When I run the webhook client on a custom port, its recieving the JIRA POST but failing to pass the Key parameter to the shell script,

[root@green ~/golang]# [webhook] 2016/09/29 12:10:09 version 2.4.0 starting
[webhook] 2016/09/29 12:10:09 setting up os signal watcher
[webhook] 2016/09/29 12:10:09 attempting to load hooks from hooks.json
[webhook] 2016/09/29 12:10:09 found 1 hook(s) in file
[webhook] 2016/09/29 12:10:09 loaded: jira-new
[webhook] 2016/09/29 12:10:09 os signal watcher ready
[webhook] 2016/09/29 12:10:09 serving hooks on http://127.0.0.1:7788/hooks/{id}
[webhook] 2016/09/29 12:10:09 listen tcp 127.0.0.1:7788: bind: address already in use
[webhook] 2016/09/29 12:10:16 Started POST /hooks/jira-new
[webhook] 2016/09/29 12:10:16 jira-new got matched
[webhook] 2016/09/29 12:10:16 jira-new hook triggered successfully
[webhook] 2016/09/29 12:10:16 Completed 200 OK in 1.961989ms
[webhook] 2016/09/29 12:10:16 error extracting command arguments for environment: couldn't retrieve argument for {Source: Name: EnvName:}
[webhook] 2016/09/29 12:10:16 executing /root/golang/jira_out.sh (/root/golang/jira_out.sh) with arguments ["/root/golang/jira_out.sh"] and environment [] using /root/golang as cwd
[webhook] 2016/09/29 12:10:16 command output:
[webhook] 2016/09/29 12:10:16 finished handling jira-new

I cant figure out how to get this parameter from the JSON header and pass it to shell script, Any ideas? Thanks!

I think the helpers package adds unnecessary segmentation into the codebase. Each function in helpers is only used by a single package:

• ValuesToMap is used by main
• CheckPayloadSignature and ExtractParameter are used by hook

I'd recommend moving those functions and dropping the helpers package. What do you think?

Currently the include-command-output-in-response config option does this:

... boolean whether webhook should wait for the command to finish and return the raw output as a response to the hook initiator. If the command fails to execute or encounters any errors while executing the response will result in 500 Internal Server Error HTTP status code, otherwise the 200 OK status code will be returned.

It would be great to have two separate config options for this, one to return the output in the response and one to just wait until the command is executed. I.E:

• include-command-output-in-response - Does what it says on the tin and sticks with the current functionality
• wait-for-command-completion - If the command fails to execute or encounters any errors while executing the response will result in 500 Internal Server Error HTTP status code, otherwise the 200 OK status code will be returned. does not return the command output in the response

If I set pass-arguments-to-command to include

{
  "source": "entire-headers"
}

or

{
  "source": "header",
  "name": "X-GitHub-Event"
}

The resulting argument is always empty. Can headers only be referenced in match, or is the intention that they should also work in arguments?

The hook package has some logging side effects that should probably be factored out. I envision the hook package being completely self-contained such that someone could write their own frontend or incorporate it into a larger application.

The log calls should be refactored into error returns. Let the caller decide how to handle the errors.