Code Monkey home page Code Monkey logo

sn-cicd's Introduction

CICD Server for Service-Now (V4)

This is the core CICD Server in version 4.
For an implementation example, please have a look at https://github.com/bmoers/sn-cicd-example/tree/release/4.

Table of contents

About

The aim of this project is to CICD-enable ServiceNow. Where CICD is used heavily in professional application development these days, there is no such thing in ServiceNow out-of-the-box. As the number of projects we implemented in ServiceNow grow, as more we struggled with doing thing right. It is just too dangerous to oversee an unwanted record in an update set or not having a clue if a change would do any harm on the platform.
The difference between 'classic' software development and the way it works in ServiceNow is how changes are captured. In ServiceNow that's the job of an 'update set' or a 'scoped application' (see it as db row-dump in XML) - the others just use GIT for that.

So how difficult must it be to extract all code inside an 'update set' into a GIT repo and send it to a CICD pipeline?

Actually not very....

With this CICD-Server we mimic a local developer working in its IDE and committing code to a GIT repo and pushing it to origin. Every update set is represented as a branch on which then also the build process runs (build on branch, merge to master). If the build on branch is successful, the update set is deployed to the integration ServiceNow environment (e.g. test or integrated dev). Have a look at the Process Flow for detailed information.

As the code is now in a GIT repo, standard tools like Bamboo or Jenkins etc can be used to trigger a longer pipeline with further stages to bring changes to production.

Whats New

Developers (End Users)

  • Deploy update set from git repo
    • extract the code from the XML saved to git instead from the source environment
  • Run CICD on scoped apps
    • automatically create an update set containing the complete scoped app and send it to the pipeline
  • All changes are now exported to GIT
    • Fields containing JavaScript are still created as .js files
    • All other files / fields are exported as JSON
    • No 'empty pull request' anymore
  • The Update-Set, on which the CICD process runs, is also exported
  • Branches are automatically deleted on pull request merge
  • SSL support for portal
  • New Jobs Dashboard to display progress
  • More detailed 'build run' log

Platform

  • Pull request proxy
    • route PR information from public git repo to CICD server
  • REST API endpoints to integrate with standard build tools
    • trigger ATF test or deployment
  • Use of scripted REST API to interact with ServiceNow
  • Message-Queue driven Master/Worker architecture
    • Easy to scale up by adding additional workers (on local- or remote server)
  • ATF runs as worker job on server (and not anymore on build process/build tool)
  • Support for external build tool
    • Use CICD Server to extract code from ServiceNow and run a pipeline on e.g. Bamboo
    • Build results are automatically sent from build process (remote) to CICD server
  • Gulp tasks now configurable
    • Allow to modify and extend build stages
  • Better automated conflict detection on GIT merge
    • Can be extended to inform about 'last commit wins' issues
  • Null value support for empty fields
    • ServiceNow sometimes treats null as empty or empty as null, to avoid displaying unrelated changes in GIT empty is treated as null
  • Project DB (filesystem meta information) now in server and not in project
  • Option to extend or overwrite CICD server modules
  • Credentials only stored on CICD server (as env. variable) no Oauth token used anymore

Features

Export, Build, Test and Deploy an Service-Now Update-Set.

On request (received from Service-Now) do:

  • create a nodejs project
  • init a git repo, link to remote repo, switch to master branch
  • export all (related) files from Service-Now production instance
  • switch to update-set branch
  • export all files (sys_update_xml) from Service-Now development instance
  • run npm install
  • run gulp
    • document the files with JSDoc
    • check the quality with ESLint
    • run the test cases (ATF tests) IN Service-Now via mocha wrapper
  • if all gulp tasks are successful
    • raise a pull request in remote git repo
    • wait for code review and pull request completion
    • on complete
      • set update-set complete
      • deploy update-set to target

Pull Request

pul request example

Slack Example

slack message example

UI Example

web ui example

Process Flow

Invoke the CICD pipeline

Steps Dev Prod (master) Code Comment
Run CICD
Setup GIT repo on GIT host if required lib\modules\run.js Filter on commit message 'no-cicd' to avoid the build to trigger.
Clone GIT
Extract files from prod into master branch (refresh in case changes made on prod without using this pipeline) lib\modules\export-files-from-master.js
Push master to GIT Filter on commit message 'no-cicd' to avoid the build to trigger.
Create branch for update set if not already exists lib\modules\export-update-set.js
Refresh update set branch with changes made on master. To avoid merge collision later in the process.
Export update set XML
Export update set files
Configure Lint / JsDoc / ATF
Push branch to GIT This will cause CICD pipeline to start if build on branch is enabled.
If CICD_EMBEDDED_BUILD is 'true'
Build the branch locally lib\modules\build-project.js

Build project

Steps Dev (Source) Test (Target) Code Comment
Gulp Init lib\project-templates\gulpfile.js Get build information from server (/build/config)
Gulp Lint, Doc Lint and jsDoc results are zipped and sent to the CICD server (/build/task)
Gulp Test Run ATF suites and tests lib\project-templates\atf-wrapper.js Start hidden browser as test runner on CICD server.Send zipped mocha results to CICD server (/build/task)
Gulp complete lib\project-templates\gulpfile.js Send build complete info to server (/build/complete).
Build complete lib\modules\build-complete.js
If CICD_GIT_PR_ENABLED is 'true'.
Raise pull request Raise PR in GIT system
Else If deploy target information in place and CD_CD_DEPLOY_ON_BUILD_PASS is 'true'
Complete update set Update set can now be moved
Deploy to target lib\modules\deploy-update-set.js Deployment can be done either via update set pull from source or pull from GIT (CICD_CD_DEPLOY_FROM_GIT)
Else notify via Slack build has completed

Pull request resolved

Steps Dev (Source) Test (Target) Code Comment
Pull request completed lib\cicd.js PR information coming from GIT system (/pull_request)
If pull request resolved
If CICD_GIT_DELETE_BRANCH_ON_MERGE remove local branch Delete branch locally and remote
Complete update set Update set can now be moved
If CICD_CD_DEPLOY_ON_PR_RESOLVE is 'true'
Deploy to target lib\modules\deploy-update-set.js Deployment can be done either via update set pull from source or pull from GIT (CICD_CD_DEPLOY_FROM_GIT)

Trigger Build

Checkout code from GIT
Run

  • npm install
  • gulp

This will test the application on the default host (source). If configured it will also raise a pull request against master branch.

Trigger Tests

Checkout code from GIT
Run

  • npm install
  • gulp test --commit-id <commit-id> --on-host <test-host.service-now.com>

--commit-id: the commit ID of the build --on-host: the host on which the ATF test will run. If CICD_ATF_RUN_ON_PRODUCTION is 'false' it will not allow to run on the master environment.

Trigger Deliver

Checkout code from GIT
Run

  • npm install
  • gulp deploy --commit-id <commit-id> --git --deliver-to <target-host.service-now.com> --deliver-from <source-host. service-now.com>

--commit-id: the commit ID of the build

--git: if exists, update set will be taken and deployed from GIT

--deliver-to: the environment to deliver to

-- deliver-from: the environment from which to deliver. If GIT is enabled, this environment will act as a proxy to connect to GIT.

Trigger Deploy

Checkout code from GIT
Run

  • npm install
  • gulp deploy --commit-id <commit-id> --git --deploy-to <target-host.service-now.com> --deploy-from <source-host. service-now.com>

--commit-id: the commit ID of the build

--git: if exists, update set will be taken and deployed from GIT

--deploy-to: the environment to deploy to

--deploy-from: the environment from which to deploy. If GIT is enabled, this environment will act as a proxy to connect to GIT.

Trigger Deploy/Deliver via REST call

Method POST
URL /deploy/us
Header "x-access-token": CICD_DEPLOY_ACCESS_TOKEN, "accept": "application/json"
Body {"commitId": <the commit ID of the build>, "from": <source-host>, "to": <target-host>, "deploy": <true = deploy| false = deliver> [false], "git": <true = via git | false = via source> [false] }

Rest client must support long polling and follow the redirects in the response header.

System Properties

Git clone might run long on large GIT repos and deployments time out. To avoid consider following settings:

  • glide.http.outbound.max_timeout.enabled=false
  • glide.rest.outbound.ecc_response.timeout=300

More information about REST calls via MID server can be found here KB0694711

Contribute

Please fork, please contribute.

Project dependencies

The project is designed to use extensions. This core project contains all 'shared' features. Customization which are dedicated to your Service-Now environment or CICD pipeline shall be added to the 'extending' project (like https://github.com/bmoers/sn-cicd-example-v3)

sn-cicd's People

Contributors

bmoers avatar dependabot[bot] avatar g-matharu avatar gitlabbin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sn-cicd's Issues

ATF Cases Not executing

I have test cases setup for my project; the builds go through however test cases are not being executed. ATF is enabled on the instance and the ATF user is created with the correct roles; also ATF is launching as part of the build but not the test cases; I am getting the below:

Execute ATF: Wrapper
execute Mocha Tests...

Execute ATF: Execution Issue
√ >>>>> Passing for now, but no test cases specified!

1 passing (8ms)
[17:29:28] Finished 'test' after 8.23 s

Task Ended 'test'

ZIP: test adding folder C:\Users\user1\AppData\Local\Temp\2\ac6be813-8f0a-4faf-b6b5-a556f205f1f3\test
[17:29:28] Starting 'build'...

Starting Task 'build'

Build Done: Thanks for submitting
[17:29:28] Finished 'build' after 230 ms

Task Ended 'build'

No config found for task 'build'
ZIP: test 359935 total bytes
ZIP: test archiver has been finalized and the output file descriptor has closed.
Upload Results test Upload successful! Server responded with: File uploaded!

What am I missing?

Error while building the application

hi,
getting the below error in node when i click on build this application ui action in servicenow. can you help?

[01:48:35.102] [LOG] [19196] [/worker#lwLR4eV-RSALbmdVAAAA] Starting module: projectSetup
[01:48:35.105] [LOG] [19196] [/worker#lwLR4eV-RSALbmdVAAAA] STARTING PROMISE JOB 'projectSetup'
[01:48:35.112] [ERROR] [19196] [/worker#lwLR4eV-RSALbmdVAAAA] PROMISE JOB ERROR: Job 'projectSetup' { type: 'exe',
name: 'projectSetup',
options:
{ updateSet: '261c169d1b9c48106933fc88cc4bcb2b',
application:
{ id: '268e9d5fdbb32300fcf417803996195e',
name: 'CICD Test Application' },
requestor:
{ userName: 'admin',
fullName: 'System Administrator',
email: '[email protected]' },
atf: { updateSetOnly: false },
git: { repository: 'x_11413_cicd_test' },
source: { name: 'https://dev1234.service-now.com/' },
master: { name: null },
target: { name: null } },
host: 'LAPTOP-JS3P7BMM',
id: '4ef3da58-0405-4c17-8d50-87f0474dac90',
emitter: { _events: {}, _eventsCount: 2 },
created: 1570168115100,
completed: null,
workerId: '/worker#lwLR4eV-RSALbmdVAAAA',
result: undefined,
error: undefined } Error: Configuration Error: source server host name not defined in config.host.name
at Promise.try (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\lib\modules\project-setup.js:198:23)
at tryCatcher (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\util.js:16:23)
at Function.Promise.attempt.Promise.try (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\method.js:39:29)
at CICD.configure (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\lib\modules\project-setup.js:191:27)
at CICD.module.exports (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\lib\modules\project-setup.js:497:22)
at Promise.try.then (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\lib\eb\run-module.js:75:34)
at tryCatcher (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\util.js:16:23)
at Promise._settlePromiseFromHandler (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\promise.js:517:31)
at Promise._settlePromise (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\promise.js:574:18)
at Promise._settlePromiseCtx (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\promise.js:611:10)
at _drainQueueStep (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\async.js:142:12)
at _drainQueue (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\async.js:131:9)
at Async._drainQueues (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\async.js:147:5)
at Immediate.Async.drainQueues [as _onImmediate] (C:\Users\ganes\Desktop\sn-cicd-master\sn-cicd-master\node_modules\bluebird\js\release\async.js:17:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)

Orlando Deployment - Deployment phase fails at preview with 'Could not find a record in sys_app_category for column category referenced in this update'

Running into a deployment issue with 2 Orlando instances. The pipeline breaks at update set preview step on the instance. The error logged is as follow:
'Could not find a record in sys_app_category for column category referenced in this update'

Looked it up in the sys_app_category table and did not find a "category" column there on either sides (source or target); also checked the same tables on 2 previews versions (New York & Madrid) and tables are similar like the Orlando one.

I looked up the pull request in Git for the scoped app and looked at the exported XML and noticed the tag with a default value "customer" in there in many places; I was wondering if the process was complaining about it; even though the same process runs fine in New York & Madrid. Wondering what's breaking the process in Orlando.

Logs below:

[22:38:54.978] [LOG] [30032] Request: https://dev88759.service-now.com/api/devops/cicd/deploy_step?__step=previewUpdateSet&__status=304
[22:38:55.291] [LOG] [30032] Job in progress. Wait for 1000 ms ...
[22:38:56.294] [LOG] [30032] Request: https://dev88759.service-now.com/api/devops/cicd/deploy_step?__step=previewUpdateSet&__status=304
[22:38:56.651] [LOG] [30032] Request: https://dev88759.service-now.com/api/devops/cicd/deploy_step?__step=commitUpdateSet&__status=303
[22:38:56.990] [ERROR] [30032] [/worker#hOdmxdADXHiF0GxlAAAC] deploy-update-set.deployUpdateSet : Commit requires manual interaction! - Update conflicts must be resolved manually. { name: 'Preview Review Problem',
message: '- Update conflicts must be resolved manually. ',
updateSet:
'https://dev88759.service-now.com/sys_remote_update_set.do?sys_id=941a68681b8b5010271ced3abc4bcb89',
issues: true,
previewProblems:
[ { type: 'error',
name:
'Could not find a record in sys_app_category for column category referenced in this update',
link:
'https://dev88759.service-now.com/sys_update_preview_problem.do?sys_id=ba1aa8681b8b5010271ced3abc4bcb3e' } ],
dataLossWarnings: [],
payload:
{ version: null,
sourceSysId: 'd9f4edf71b321010271ced3abc4bcb72',
targetEnvironment: 'https://dev88759.service-now.com/',
updateSetSysId: '54dca2c77a76f3d36b4d6dde84a86503',
limitSet: [ '4VQVmZl3hVXr2A6B' ],
sourceEnvironment: 'https://dev91367.service-now.com',
deploy: true,
collisionDetect: false,
conflicts: { resolutions: {}, defaults: [Object] },
step: 'commitUpdateSet',
progressId: null,
state: 'previewed',
remoteUpdateSetSysId: '941a68681b8b5010271ced3abc4bcb89',
missingRecords: {} } }
[22:38:58.482] [WARN] [30032] Email notification is disabled. Following message was not sent: "DevOps Integration" <>, DEPLOYMENT CONFLICTS - Update Set is causing conflicts: Deployment of 'my_demo_app - 1.0.0 › #1' requires manual interaction

Deployment Conflicts in Update Set 'my_demo_app - 1.0.0 › #1'


The 'preview' step during the update set deployment on target https://dev88759.service-now.com detected conflicts and requires them to be resolved manually.


Please open the Target Update Set and resolve all conflicts to proceed with this deployment.




















Source: https://dev91367.service-now.com
Update Set: my_demo_app - 1.0.0
Target: https://dev88759.service-now.com
Update Set: https://dev88759.service-now.com/nav_to.do?uri=sys_remote_update_set.do%3Fsys_id%3Dnull%26sysparm_view%3Dcicd_problems


Update conflict:

TypeDescriptionLink
ERRORCould not find a record in sys_app_category for column category referenced in this updateopen


[22:38:58.506] [LOG] [30032] Change Run State to 'failed (manual interaction required)'
[22:38:58.561] [LOG] [25396] /worker#hOdmxdADXHiF0GxlAAAC is done (/worker/done) with job: acce5236-3d01-49c9-8c3e-42cfea859706 Type: exe

CI User wasn't picked up when deploy update set

Issue:
config snippet from .env file :

# user to load update-set form 'source' and 'master'
CICD_CI_USER_NAME=svc_cicd
CICD_CI_USER_PASSWORD=*****

# user to deploy update-set to 'target'
CICD_CD_USER_NAME=admin
CICD_CD_USER_PASSWORD=****

while running deployment jobs, it always using 'admin' to connect to the source instance, it will cause authentication failure

{ error:
   { message: 'User Not Authenticated',
     detail: 'Required to provide Auth information' },
  status: 'failure' }

Deployment to target breaks with - Remote update-set not found

The pipeline was working fine between both my instances -now it is breaking with Remote update-set not found exception at the deployment stage; It's failing for existing applications and also for applications created from scratch,

What is the reason behind this exception? How can we fix it?

workerId: '/worker#zpr5LetrJJeqMv9yAAAB',
result: 'BACKGROUND-JOB-IN-PROGRESS',
error:
{ StatusCodeError: 400 - {"error":{"detail":"","message":"Remote update-set not found with "remote_sys_id" 3f2434266a5a6800d7ae05b98e08b293"},"status":"failure"}
at new StatusCodeError (C:\Users\user1\Documents\sncicd-server\node_modules\request-promise-core\lib\errors.js:32:15)
at Request.plumbing.callback (C:\Users\user1\Documents\sncicd-server\node_modules\request-promise-core\lib\plumbing.js:104:33)
at Request.RP$callback [as _callback] (C:\Users\user1\Documents\sncicd-server\node_modules\request-promise-core\lib\plumbing.js:46:31)
at Request.self.callback (C:\Users\user1\Documents\sncicd-server\node_modules\request\request.js:185:22)
at Request.emit (events.js:198:13)
at Request. (C:\Users\user1\Documents\sncicd-server\node_modules\request\request.js:1154:10)
at Request.emit (events.js:198:13)
at Gunzip. (C:\Users\user1\Documents\sncicd-server\node_modules\request\request.js:1076:12)
at Object.onceWrapper (events.js:286:20)
at Gunzip.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1145:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
name: 'StatusCodeError',
statusCode: 400,
message:
'400 - {"error":{"detail":"","message":"Remote update-set not found with \"remote_sys_id\" 3f2434266a5a6800d7ae05b98e08b293"},"status":"failure"}',
error: { error: [Object], status: 'failure' },
options:
{ json: true,
baseUrl: null,
gzip: true,
encoding: 'utf8',
headers: [Object],
jar: false,
rawResponse: true,
autoPagination: true,
retry: 1,
delay: 100,
auth: [Object],
followRedirect: false,
method: 'POST',

Pull Request Slack Message

The Slack PR message currently must come from the Git system, the CICD server is not capable to send that information directly (however it is exposing an API to the Git system to send slack messages)

Extend the CICD server to send PR Slack messages directly as discussed here
#30 (comment)

Issue When Building App - Install Node App fails

Install Node App steps fails as follow - Please advise.

[12:11:44.983] [LOG] [10336] Using SYS_* Field White-List 'sys_class_name, sys_created_by, sys_created_on, sys_id, sys_mod_count, sys_name, sys_package, sys_policy, sys_scope, sys_update_name, sys_updated_by, sys_updated_on'
[12:11:44.983] [LOG] [10336] Project Datastore: use remote
[12:11:44.985] [LOG] [10336] [/worker#ZbGaE-8YwQO-wT-JAAAA] build-project : install node application
[12:11:45.015] [LOG] [10336] install node app in C:\Users\user1\AppData\Local\Temp\3\git-root\459c6b10-47eb-4605-be75-f78bffe92478 [ 'ci', '--silent', '--no-audit', '--no-optional' ]
[12:11:55.729] [LOG] [10336] npm install process exited with code: 1
[12:11:55.730] [ERROR] [10336] [/worker#ZbGaE-8YwQO-wT-JAAAA] build-project : install node application failed:
Error
at C:\Users\user1\Documents\sncicd-server\node_modules\sn-cicd\lib\modules\build-project.js:41:76
at tryCatcher (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\util.js:16:23)
at Promise._settlePromiseFromHandler (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:547:31)
at Promise._settlePromise (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:604:18)
at Promise._settlePromise0 (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:649:10)
at Promise._settlePromises (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:725:18)
at _drainQueueStep (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:93:12)
at _drainQueue (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:86:9)
at Async._drainQueues (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:15:14)
at processImmediate (internal/timers.js:456:21) Error
at C:\Users\user1\Documents\sncicd-server\node_modules\sn-cicd\lib\modules\build-project.js:41:76
at tryCatcher (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\util.js:16:23)
at Promise._settlePromiseFromHandler (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:547:31)
at Promise._settlePromise (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:604:18)
at Promise._settlePromise0 (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:649:10)
at Promise._settlePromises (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\promise.js:725:18)
at _drainQueueStep (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:93:12)
at _drainQueue (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:86:9)
at Async._drainQueues (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (C:\Users\user1\Documents\sncicd-server\node_modules\bluebird\js\release\async.js:15:14)
at processImmediate (internal/timers.js:456:21)
[12:11:55.737] [LOG] [10336] [/worker#ZbGaE-8YwQO-wT-JAAAA] build-project : cleaning up project
[12:11:55.740] [LOG] [10336] deleting C:\Users\user1\AppData\Local\Temp\3\git-root\459c6b10-47eb-4605-be75-f78bffe92478\node_modules
[12:11:57.512] [ERROR] [10336] [/worker#ZbGaE-8YwQO-wT-JAAAA] PROMISE JOB ERROR: Job 'buildProject' {
type: 'exe',
name: 'buildProject',
options: 'xUauor5G8AKEmCLY',
host: 'ncias-d2371-c',
id: '0895740f-60fe-41ee-990b-09621667cfb5',
emitter: { _events: {}, _eventsCount: 2 },
created: 1586880704932,
completed: null,
workerId: '/worker#ZbGaE-8YwQO-wT-JAAAA',
result: undefined,
error: undefined

WorkerNode job worker always in `running` for ever, even job finished

4 cores cpu, having 8 worker nodes.
Steps to produce the issue:

  1. trigger the build job 8 times, all going well. but when all job done, the JobWorker still show running in the web UI
  2. once the 9th build job triggered, no worker node available. the job will pending for ever

Only way is to restart the SN-CICD server

Suport custom domains

The CICD process does not support "custom urls" - environment must be running under the "service-now.com" domain.

Replace hardcoded ".service-now.com" with ENV var as discussed here
#30 (comment)

"Global" Update Set Name instead of App Name

The update sets deployed for Globally scoped applications is currently named as Global #. This results in update sets with duplicate names.
The deployed update set should be named as #

exportFilesFromMaster : socket hang up

Header is too large if pagination is enabled.

An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.: org.apache.coyote.http11.HeadersTooLargeException: An attempt was made to write more data to the response headers than there was room available in the buffer. Increase maxHttpHeaderSize on the connector or write less data into the response headers.: org.apache.coyote.http11.Http11OutputBuffer.checkLengthBeforeWrite(Http11OutputBuffer.java:544)
org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:468)
org.apache.coyote.http11.Http11OutputBuffer.write(Http11OutputBuffer.java:454)
org.apache.coyote.http11.Http11OutputBuffer.sendHeader(Http11OutputBuffer.java:414)
org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Processor.java:1388)
org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:373)
org.apache.coyote.Response.action(Response.java:208)
org.apache.coyote.Response.sendHeaders(Response.java:418)
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:304)
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:285)
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118)
com.glide.size_aware.SizeAwareServletOutputStream.flush(SizeAwareServletOutputStream.java:68)
java.util.zip.DeflaterOutputStream.flush(DeflaterOutputStream.java:282)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141)
java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
com.glide.rest.serializer.impl.ResponseSerializerImpl.closeOutputStream(ResponseSerializerImpl.java:166)
com.glide.rest.processors.RESTAPIProcessor.endRequestProcessing(RESTAPIProcessor.java:383)
com.glide.rest.processors.RESTAPIProcessor.endRequestProcessing(RESTAPIProcessor.java:372)
com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:301)
com.glide.processors.AProcessor.runProcessor(AProcessor.java:553)
com.glide.processors.AProcessor.processTransaction(AProcessor.java:240)
com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:177)
com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:166)
com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31)
com.glide.sys.Transaction.run(Transaction.java:2192)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

email notification in case slack is not available in the environment

Currently e-mail notifications works for failures (builds, deployments, preview, etc). Would it be possible to add e-mail notifications for the success criteria when build completes, pull request is open, pull request is merged, deployment in progress, and deployment complete?

The reason I am asking for this is because collaboration tools such as Slack is not available or allowed in the environment I currently work with and we would rather use e-mail notifications for now.

Please advise.

Error During Project Setup

An Error is being encountered related to certificate validation at project setup - Error logs shown below:

[18:27:36.164] [ERROR] [2664] [/worker#Fh0RJQ1ejjKBQFxzAAAB] PROMISE JOB ERROR: Job 'projectSetup' { type: 'exe',
name: 'projectSetup',
options:
{ build: { requestor: [Object] },
atf: { updateSetOnly: false },
updateSet: '1db3fd101b41281494edddb6bc4bcb6a',
application:
{ id: 'bfb3a72c1b94641094edddb6bc4bcb2a',
name: 'MyCICDTestApp',
organization: 'company' },
git:
{ repository: 'x_g_nci_mycicdtest',
remoteUrl: null,
enabled: true,
pullRequestEnabled: true },
host: { name: 'https://service-sandbox.xxx.xxx/' } },
host: 'ncias-d2371-c',
id: 'bb5861f3-4db6-4fe8-b3b8-fd0dfb8df731',
emitter: { _events: {}, _eventsCount: 2 },
created: 1607470055957,
completed: null,
workerId: '/worker#Fh0RJQ1ejjKBQFxzAAAB',
result: undefined,
error: undefined } { Error: unable to verify the first certificate
at TLSSocket.onConnectSecure (_tls_wrap.js:1058:34)
at TLSSocket.emit (events.js:198:13)
at TLSSocket._finishInit (_tls_wrap.js:636:8) code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE' }
[18:27:36.217] [LOG] [5464] /worker#Fh0RJQ1ejjKBQFxzAAAB is done (/worker/done) with job: bb5861f3-4db6-4fe8-b3b8-fd0dfb8df731 Type: exe
[18:27:36.217] [ERROR] [5464] Job failed on worker { id: 'bb5861f3-4db6-4fe8-b3b8-fd0dfb8df731',
type: 'exe',
host: 'ncias-d2371-c',
platform: 'win32',
background: undefined } %j { name: 'RequestError',
message: 'Error: unable to verify the first certificate',
cause:
{ stack:
'Error: unable to verify the first certificate\n at TLSSocket.onConnectSecure (_tls_wrap.js:1058:34)\n at TLSSocket.emit (events.js:198:13)\n at TLSSocket._finishInit (_tls_wrap.js:636:8)',
message: 'unable to verify the first certificate',

Error when click in "Build this Application"

I'm having an error in the Node app, when click in "Build this application button".
Config:

  • CentOS 7
  • Virtual Box VM
  • 2GB RAM
  • 30GB HD
  • MID server running in the same server

Error:
image

Error text:
Unhandled rejection TypeError: Cannot read property 'match' of null
at CICD.module.exports (/home/vjbs/int-cicd/lib/modules/project-setup.js:471:36)
at Promise.try.then (/home/vjbs/int-cicd/lib/eb/run-module.js:75:34)
at tryCatcher (/home/vjbs/int-cicd/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/home/vjbs/int-cicd/node_modules/bluebird/js/release/promise.js:517:31)
at Promise._settlePromise (/home/vjbs/int-cicd/node_modules/bluebird/js/release/promise.js:574:18)
at Promise._settlePromiseCtx (/home/vjbs/int-cicd/node_modules/bluebird/js/release/promise.js:611:10)
at _drainQueueStep (/home/vjbs/int-cicd/node_modules/bluebird/js/release/async.js:142:12)
at _drainQueue (/home/vjbs/int-cicd/node_modules/bluebird/js/release/async.js:131:9)
at Async._drainQueues (/home/vjbs/int-cicd/node_modules/bluebird/js/release/async.js:147:5)
at Immediate.Async.drainQueues [as _onImmediate] (/home/vjbs/int-cicd/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)

Test connection in CICD Integration properties

In CICD Integration > Properties for the "The CICD Server Hostname" Property, it would save time to test to see if the connection is good before having to finish the set up and run a test CICD pipeline to see if the connection is good or not. This could also be used in MID server to be used.

How sn-cicd handle data migration

When we migrate updateset from dev instance to test instance,
some updateset requires additional data or configuration changes.

  • Configure system properties or other instance-specific settings
  • group or role

How will SN-CICD handle it? or any good practice for that case?

Build UI action not being displayed in New York release ?

I'm trying to implement sn-cicd in New York taking the example but I'm currently stuck in the place where I need to build the test application which i uploaded.
The UI action thats supposed to run the build isnt displayed in New York release.
Any heads up ?

Update set commit fails when deploying from git.

When using git as the source for update set, update set is being loaded in target environment, but commit fails because of scope errors. This happens both for scoped and global applications.
Cannot commit Update Set 'Default' because: Update scope id '03852e7f4f613300a3cc121f9310c728' is different than update set scope id 'x_317910_latest28'. Resolve the problem before committing.
Same update set works if i disable deployment from git.
Not sure it is related but i see below error in Logs:-
GetKeys failed (com.glide.processors.soap.SOAPProcessingException: Web service not found: api/devops/cicd/source/sys_scope com.glide.processors.soap.SOAPProcessingException: Web service not found: api/devops/cicd/source/sys_scope

Incorrect baselineCommitId for merged-deployment after a fix-deployment

Scenario

Update Set Name Merge Sequence Commit ID Deployed to Target
UpdateSet1 1 commit-1 ✔️
UpdateSet2 2 commit-2
FIX: UpdateSet3 3 commit-3
UpdateSet4 4 commit-4
UpdateSet5 5 commit-5

Deploy commit-3 to Target; As expected, commit-2 is not deployed since commit-3 is a FIX deployment

Update Set Name Merge Sequence Commit ID Deployed to Target
UpdateSet1 1 commit-1 ✔️
UpdateSet2 2 commit-2
FIX: UpdateSet3 3 commit-3 ✔️
UpdateSet4 4 commit-4
UpdateSet5 5 commit-5

Now deploy commit-5 to Target,
Finding: commit-2 is still not deployed to Target
Cause: baselineCommitId should ideally be commit-1 and not commit-3

Update Set Name Merge Sequence Commit ID Deployed to Target
UpdateSet1 1 commit-1 ✔️
UpdateSet2 2 commit-2
FIX: UpdateSet3 3 commit-3 ✔️
UpdateSet4 4 commit-4 ✔️
UpdateSet5 5 commit-5 ✔️

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.