What is the reason to have an optional Load Balancer? In which cases API Gateway is not enough?

We are following the suggested approach, but we are getting an 403 error which we are not able to solve. This is the basic configuration logged by axios when the lambda function it is triggered from the webhook

config: {
    url: 'https://hostname/app/stash/rest/api/latest/projects/GRANTSPP/repos/sda-docref-poc/archive?at=refs/heads/GRANTSPP-6363-FT&format=zip',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, */*',
      Authorization: 'Bearer XXXXXXX',
      'Accept-Encoding': 'gzip,deflate',
      'User-Agent': 'axios/0.20.0'
    }

If we send a similar request via SOAP UI, which resemble to someting like

GET https://hostname/app/stash/rest/api/latest/projects/GRANTSPP/repos/sda-docref-poc/archive?at=refs%2Fheads%2FGRANTSPP-6363-FT&format=zip HTTP/1.1
Accept-Encoding: gzip,deflate
Authorization: Bearer XXXXX
Host: webgate.ec.europa.eu
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

Everything works as expected.

As the lambda function gets a 403 there's connectivity with our bitbucket serve. Any idea about possible reason about refusal of the axios request?

Hi Alex,

Thanks for putting this together. I have encountered the following error.

From bitbucket webhook, I get the 500 error code.

I checked the log in cloudwatch, here is the error message:

Can you help point out where I should look? Thanks.

Yang

Hi there,

I have followed the article but unfortunately, Lambda is not able to download zipped files using Bitbucket server API. From the error message, it looks like an authentication issue but I cannot really figure out what is it. Please find the logs below:

INFO error Error: Request failed with status code 401
at createError (/var/task/node_modules/axios/lib/core/createError.js:16:15)
at settle (/var/task/node_modules/axios/lib/core/settle.js:17:12)
at RedirectableRequest.handleResponse (/var/task/node_modules/axios/lib/adapters/http.js:231:9)
at RedirectableRequest.emit (events.js:400:28)
at RedirectableRequest.emit (domain.js:470:12)
at RedirectableRequest._processResponse (/var/task/node_modules/follow-redirects/index.js:399:10)
at ClientRequest.RedirectableRequest._onNativeResponse (/var/task/node_modules/follow-redirects/index.js:57:10)
at Object.onceWrapper (events.js:520:26)
at ClientRequest.emit (events.js:400:28)
at ClientRequest.emit (domain.js:470:12) {
config: {
url: '/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
method: 'get',
headers: {
Accept: 'application/json, text/plain, /',
Authorization: 'Bearer WEBHOOKTOKEN',
'User-Agent': 'axios/0.21.1'
},
baseURL: 'https://SERVER_URL:443',
transformRequest: [ [Function: transformRequest] ],
transformResponse: [ [Function: transformResponse] ],
timeout: 0,
adapter: [Function: httpAdapter],
responseType: 'stream',
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: -1,
maxBodyLength: -1,
validateStatus: [Function: validateStatus],
data: undefined
},
request: <ref *1> ClientRequest {
_events: [Object: null prototype] {
abort: [Function (anonymous)],
aborted: [Function (anonymous)],
connect: [Function (anonymous)],
error: [Function (anonymous)],
socket: [Function (anonymous)],
timeout: [Function (anonymous)],
prefinish: [Function: requestOnPrefinish]
},
_eventsCount: 7,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: false,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
socket: TLSSocket {
_tlsOptions: [Object],
_secureEstablished: true,
_securePending: false,
_newSessionPending: false,
_controlReleased: true,
secureConnecting: false,
_SNICallback: null,
servername: 'bitbucket.dpti.sa.gov.au',
alpnProtocol: false,
authorized: true,
authorizationError: null,
encrypted: true,
_events: [Object: null prototype],
_eventsCount: 9,
connecting: false,
_hadError: false,
_parent: null,
_host: 'bitbucket.dpti.sa.gov.au',
_readableState: [ReadableState],
_maxListeners: undefined,
_writableState: [WritableState],
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: undefined,
_server: null,
ssl: [TLSWrap],
_requestCert: true,
_rejectUnauthorized: true,
parser: null,
_httpMessage: [Circular *1],
[Symbol(res)]: [TLSWrap],
[Symbol(verified)]: true,
[Symbol(pendingSession)]: null,
[Symbol(async_id_symbol)]: 5,
[Symbol(kHandle)]: [TLSWrap],
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0,
[Symbol(connect-options)]: [Object],
[Symbol(RequestTimeout)]: undefined
},
_header: 'GET /rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip HTTP/1.1\r\n' +
'Accept: application/json, text/plain, /\r\n' +
'Authorization: Bearer TOKEN\r\n' +
'User-Agent: axios/0.21.1\r\n' +
'Host: bitbucket.dpti.sa.gov.au\r\n' +
'Connection: close\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: [Function: noopPendingOutput],
agent: Agent {
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
defaultPort: 443,
protocol: 'https:',
options: [Object],
requests: {},
sockets: [Object],
freeSockets: {},
keepAliveMsecs: 1000,
keepAlive: false,
maxSockets: Infinity,
maxFreeSockets: 256,
scheduling: 'lifo',
maxTotalSockets: Infinity,
totalSocketCount: 1,
maxCachedSessions: 100,
_sessionCache: [Object],
[Symbol(kCapture)]: false
},
socketPath: undefined,
method: 'GET',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
path: '/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
_ended: false,
res: IncomingMessage {
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 1,
_maxListeners: undefined,
socket: [TLSSocket],
httpVersionMajor: 1,
httpVersionMinor: 1,
httpVersion: '1.1',
complete: true,
headers: [Object],
rawHeaders: [Array],
trailers: {},
rawTrailers: [],
aborted: false,
upgrade: false,
url: '',
method: null,
statusCode: 401,
statusMessage: '',
client: [TLSSocket],
_consuming: false,
_dumped: false,
req: [Circular *1],
responseUrl: 'https://SERVER_URL:443/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
redirects: [],
[Symbol(kCapture)]: false,
[Symbol(RequestTimeout)]: undefined
},
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: 'bitbucket.dpti.sa.gov.au',
protocol: 'https:',
_redirectable: Writable {
_writableState: [WritableState],
_events: [Object: null prototype],
_eventsCount: 2,
_maxListeners: undefined,
_options: [Object],
_ended: true,
_ending: true,
_redirectCount: 0,
_redirects: [],
_requestBodyLength: 0,
_requestBodyBuffers: [],
_onNativeResponse: [Function (anonymous)],
_currentRequest: [Circular *1],
_currentUrl: 'https://SERVER_URL:443/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
[Symbol(kCapture)]: false
},
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype] {
accept: [Array],
authorization: [Array],
'user-agent': [Array],
host: [Array]
}
},
response: {
status: 401,
statusText: '',
headers: {
server: 'nginx/1.20.0',
date: 'Tue, 31 Aug 2021 05:54:11 GMT',
'content-type': 'application/json;charset=UTF-8',
'transfer-encoding': 'chunked',
connection: 'close',
'x-arequestid': '@JBU598x924x1700989x0',
'x-asen': 'SEN-10832278',
'cache-control': 'no-cache, no-transform',
vary: 'X-AUSERNAME,Accept-Encoding',
'www-authenticate': 'OAuth realm="https%3A%2F%2Fbitbucket.dpti.sa.gov.au"',
'x-content-type-options': 'nosniff'
},
config: {
url: '/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
method: 'get',
headers: [Object],
baseURL: 'https://bitbucket.dpti.sa.gov.au:443',
transformRequest: [Array],
transformResponse: [Array],
timeout: 0,
adapter: [Function: httpAdapter],
responseType: 'stream',
xsrfCookieName: 'XSRF-TOKEN',
xsrfHeaderName: 'X-XSRF-TOKEN',
maxContentLength: -1,
maxBodyLength: -1,
validateStatus: [Function: validateStatus],
data: undefined
},
request: <ref *1> ClientRequest {
_events: [Object: null prototype],
_eventsCount: 7,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
destroyed: false,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: false,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: 0,
_hasBody: true,
_trailer: '',
finished: true,
_headerSent: true,
socket: [TLSSocket],
_header: 'GET /rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip HTTP/1.1\r\n' +
'Accept: application/json, text/plain, /\r\n' +
'Authorization: Bearer TOKEN\r\n' +
'User-Agent: axios/0.21.1\r\n' +
'Host: bitbucket.dpti.sa.gov.au\r\n' +
'Connection: close\r\n' +
'\r\n',
_keepAliveTimeout: 0,
_onPendingData: [Function: noopPendingOutput],
agent: [Agent],
socketPath: undefined,
method: 'GET',
maxHeaderSize: undefined,
insecureHTTPParser: undefined,
path: '/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
_ended: false,
res: [IncomingMessage],
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: 'bitbucket.dpti.sa.gov.au',
protocol: 'https:',
_redirectable: [Writable],
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype]
},
data: IncomingMessage {
_readableState: [ReadableState],
_events: [Object: null prototype],
_eventsCount: 1,
_maxListeners: undefined,
socket: [TLSSocket],
httpVersionMajor: 1,
httpVersionMinor: 1,
httpVersion: '1.1',
complete: true,
headers: [Object],
rawHeaders: [Array],
trailers: {},
rawTrailers: [],
aborted: false,
upgrade: false,
url: '',
method: null,
statusCode: 401,
statusMessage: '',
client: [TLSSocket],
_consuming: false,
_dumped: false,
req: [ClientRequest],
responseUrl: 'https://bitbucket.dpti.sa.gov.au:443/rest/api/latest/projects/SP/repos/codepipeline-poc/archive?at=refs/heads/master&format=zip',
redirects: [],
[Symbol(kCapture)]: false,
[Symbol(RequestTimeout)]: undefined
}
},
isAxiosError: true,
toJSON: [Function: toJSON]
}

It would be useful if the URL of the API GW would also be added to the Output, if used. Right now it seems only the ELB case is in the template.

As per the blog it says

Fill out the URL field with the API Gateway or Load Balancer URL. To obtain this URL, choose the Outputs tab of the AWS CloudFormation stack.

(currently putting this in myself, so may just send a PR)

Hey Alex,

It's really a wonderful solution which has almost taken care of our problem to include Bitbucket Server to CodePipeline.

Everything was working as expected. But suddenly we got this below mentioned error while connecting to API Gateway from Bitbucket.

ERROR : "The server didn't respond you may retry when it comes back up". Please refer to the screenshot for better explanation.

Not able to figure out where it is failing as the event is not coming to the API gateway itself. Is it a Bitbucket Server Firewall error or there is a problem with the API Gateway endpoint which was working absolutely fine till last week.

Please Note that we are using Enterprise Bamboo Server and our AWS VPC has connection to on-prem bitbucket server with Transit Gateway and my Lambda function is running on a specific VPC to be able to download the code from Bitbucket.

I have aws codepipeline created with source -> set to bitbucket repo.
my webhook gets triggered on code change and aws codepipeline runs.

recently , when a code change was pushed on bitbucket repo.

webhook didn't trigger an aws codepipeline.

How I solved it.
-> I clicked on the release change ( to start pipeline )
After that it automatically triggers pipeline on code change.

Problems:
-> Why did pipeline not get triggered on code change?
-> Does this happen frequently and How to solve it?

Whenever I push a change to my Bitbucket repository, the webhook fails with this error:
java.lang.RuntimeException: java.net.SocketTimeoutException: 20,000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]

As documentation suggests:

Subnets where the Application Load Balancer run. Leave it blank if you are using the API Gateway endpoint type.

We leave empty for API Gateway, then we get below error while creating stack:

Parameter validation failed: parameter value for parameter name LBSubnets does not exist. Rollback requested by user.

Hey Alex,

Thank you so much for putting out this type of solution for Bitbucket Server integration with CodePipeline. You're awesome for this!!!!!!

I followed your instructions in the documentation and used an API Gateway instead of a Load Balancer. And found a few errors on my end, so I could be doing something and if so please let me know.

In this create-stack command:

aws cloudformation create-stack --stack-name CodePipeline-Bitbucket-Integration --template-body file:///package.yaml --parameters file:///infra/parameters.json --capabilities CAPABILITY_NAMED_IAM

It would not work unless I removed a "/" from --parameters and --template argument's file path.

Removing them like below will allow the stack to be created.

aws cloudformation create-stack --stack-name CodePipeline-Bitbucket-Integration --template-body file://package.yaml --parameters file://infra/parameters.json --capabilities CAPABILITY_NAMED_IAM

After I made the above change and filled in the parameters.json file, CloudFormation successfully created the stack. From the AWs UI console I deployed the API Gateway: CodePipeline-Bitbucket-Integration and copied the endpoint URL and pasted it into my Bitbucket Server's repository webhook.

In my Bitbucket repository (which is running in the same VPC and same subnet that the Lambda has connection too) When testing the connection I got a status 200. But After I triggered a build by adding a comment to a commit, I got this response below.

Webhook event details

REQUEST

HEADER
X-Hub-Signature: sha256=7bb0ec6ef324a2a0637f740efdb6d1267e1ee44947c0f78bc48fef7fb300c31e
X-Request-Id: 27f2dfb5-912e-4246-af9b-f0e52d1eb281
Content-Type: application/json; charset=utf-8
X-Event-Key: repo:comment:added

BODY:
{"eventKey":"repo:comment:added","date":"2019-11-27T23:22:07+0000","actor":{"name":"xxxx","emailAddress":"[email protected]","id":1,"displayName":"Lenny Castaneda","active":true,"slug":"xxxxxx","type":"NORMAL","links":{"self":[{"href":"http://ec2-xxxxxxxxx.us-west-2.compute.amazonaws.com/users/xxxxxxxx"}]}},"comment":{"properties":{"repositoryId":13},"id":2,"version":0,"text":"Test comment #2","author":{"name":"xxxxx","emailAddress":"[email protected]","id":1,"displayName":"Lenny Castaneda","active":true,"slug":"xxxxxx","type":"NORMAL","links":{"self":[{"href":"http://xxxxxxxxxxx.us-west-2.compute.amazonaws.com/users/xxxxxxx"}]}},"createdDate":1574896927696,"updatedDate":1574896927696,"comments":[],"tasks":[],"severity":"NORMAL","state":"OPEN"},"repository":{"slug":"ami","id":13,"name":"AMI","scmId":"git","state":"AVAILABLE","statusMessage":"Available","forkable":true,"project":{"key":"xxxxx","id":1,"name":"xxxxxxx","description":"Devops Testing","public":false,"type":"NORMAL","links":{"self":[{"href":"http://xxxxxxxxx.compute.amazonaws.com/projects/xxxxxx"}]}},"public":false,"links":{"clone":[{"href":"ssh://[email protected]:7999/xxxx/ami.git","name":"ssh"},{"href":"http://xxxxxxxxxx.us-west-2.compute.amazonaws.com/scm/xxx/ami.git","name":"http"}],"self":[{"href":"http://xxxxxxxxxxxx.us-west-2.compute.amazonaws.com/projects/xxxxx/repos/ami/browse"}]}},"commit":"6899921754f8ec3f5d05b68d77c6abe75783d41c"}

RESPONSE:

HEADERS
Access-Control-Allow-Origin: *
x-amzn-RequestId: d52f551a-36ca-4069-9380-178061b43e7c
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept
Content-Length: 54
Access-Control-Allow-Methods: POST, GET
X-Amzn-Trace-Id: Root=1-5ddf051f-413343e0556a3471887773fe;Sampled=0
Date: Wed, 27 Nov 2019 23:22:08 GMT
x-amz-apigw-id: D128-GLNPHcF1hQ=
Content-Type: application/json
Via: 1.1 localhost (Apache-HttpClient/4.5.5 (cache))
Connection: keep-alive

BODY
{"statusCode":500,"fault":"Some weird thing happened"}

Any help or guidance on how you got your internal Bitbucket Server to successfully send its webhook POST to the API Gateway I would be extra extra Thankful for.

Looking forward to hear from you. Have a Happy Thanksgiving.

Hi there

I'd managed to launch the cloudformation stack when I tested the first time in my organisation's sandbox account but now I'm getting the following error when trying to launch it in another account:

Resource handler returned message: "The provided execution role does not have permissions to call CreateNetworkInterface on EC2

I've had a look at solutions online and the role that is created definitely has the permissions to call CreateNetworkInterface. Has anyone else come across the same error?

Having successfully installed the stack, I configured the webhook in Bitbucket server.

The URL is something along
https://123456789.execute-api.ap-southeast-2.amazonaws.com/prod/

I set the secret/password to what I set in the configuration file (hard to get wrong as BB displays it in cleartext...).

Clicking Test Connection I get a 403 and it seems to come from the API Gateway as I see no Cloudwatch logs that indicate the Lambda triggered.

Looking at what Bitbucket sent and received I unfortunately only get:

Request details
Event type:Test connection event
URL endpoint:https://123456789.execute-api.ap-southeast-2.amazonaws.com/prod/
Headers

X-Event-Key: diagnostics:ping
X-Request-Id: 9cfdecbc-d891-4185-b2d3-a167c57b4dbb
Content-Type: application/json

Body

{"test": true}

And

Response details
HTTP status:403
Headers

x-amz-apigw-id: HtfLdEb1SwMFplw=
Server: Server
Connection: keep-alive
x-amzn-RequestId: e87fd1b8-4015-4edf-abb1-05ddad3cdf7b
x-amzn-ErrorType: ForbiddenException
Content-Length: 23
Date: Tue, 11 Feb 2020 02:32:41 GMT
Via: 1.1 localhost (Apache-HttpClient/4.5.5 (cache))
Content-Type: application/json

Body

{"message":"Forbidden"}

I have no other API GW logs as it's not configured and I don't have the permissions to do so. Making it somewhat hard to determine why the correct mutual secret would cause an issue. Then again, that should be a 401. As far as I can see, the template "comes with" all required IAM permissions, so it seems strange to get a 403.

Testing directly at the API GW level shows the the Lambda is being triggered, though lack of a complete sample event in the repo for testing in the AWS API Gateway console. Using merely the Atlassian sample payloads is incomplete and will fail at the normalisation stage with the Lambda returning a 500. Of course this also bypasses authentication entirely, so isn't really useful.

whenever i pushed the code to my repo in Bitbucket, event is getting triggers.
But in Lambda logs getting the error "proxy: undefined" while downloading the file thru downloadFile() function.

We are deploying the stack in eu-central-1 region, and getting below error:

Error occurred while GetObject. S3 Error Code: PermanentRedirect. S3 Error Message: The bucket is in this region: us-east-1. Please use this region to retry the request (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 4deba866-40a6-40b8-9a10-612291cf533e)

I am getting an error during the creation of the LamdaFunction:

Your access has been denied by S3, please make sure your request credentials have permission to GetObject for lexlabs3bucketmgmtapps/f4a779b4953357ca1517a6a0e76994f4. S3 Error Code: AccessDenied. S3 Error Message: Access Denied (Service: AWSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException; Request ID: 25909cf8-1365-4872-a6ad-b4e6f21d2001)

The account I am using has full admin access. Not really sure what to check.

Thanks,
Tony

The CF template uses a lambda which runs on Node Js 14.x runtime which is already deprecated. Can you guys please update this to Node 20.x .

As of now I have tried running it with Node 16.x and it worked after adding the axios and crypto, which are dependent packages of the lambda.
code provided via blog is in V2 SDK
Can you please refactor the code for V3 SDK and migrate to NodeJs 20.x

Is the license:

MIT License as per LICENSE file -

or

Apache 2.0 as per package.json -