tpayne / devops-coord-framework Goto Github PK
View Code? Open in Web Editor NEWOpen-source DevOps framework example for on-boarding 1:N projects Jenkins (or other CI tool) & CLI use
License: MIT License
Open-source DevOps framework example for on-boarding 1:N projects Jenkins (or other CI tool) & CLI use
License: MIT License
Framework unit testing needs runtime properties read from a properties file, not be hardcoded
The framework needs simple Docker services added...
Add option to send email as text or html as using HTML by default will cause email issues.
The relevant framework routines need to be JavaDoc'ed for Javadoc run
Add the ability for default properties file to be read to store config information in
Various branches of the code have been created with pull requests that need processing
Currently, the sendMail() routine is coded to work with servers that use secure protocols - TLS etc. A new version needs to be done that can work with very simple SMTP servers.
This also requires a very simple broadcast SMTP server to be setup
Create a pipelineTemplate() call singleton for wrapping default containers...
Need containers created - only example...
vars/pipelineTemplate.groovy...
def call(Closure body) {
podTemplate(
label: 'default',
containers: [
containerTemplate(name: 'centos', image: "centos", resourceRequestMemory: '1Gi', resourceLimitMemory: '1Gi', resourceRequestCpu: '500m'),
containerTemplate(name: 'tools', image: "tools", alwaysPullImage: true, ttyEnabled: true, command: 'cat', resourceRequestMemory: '512Mi', resourceLimitMemory: '512Mi')
])
{
body()
}
}
The framework needs SCM support added for simple GIT functions...
The service classes need to be extended to add one for uploading artefacts to repositories
Add an option to the constructors for ComponentManifest to support XML rather than JSON.
XML is easier to read, but not so efficient as JSON.
XMLSlurper and MarkUpBuilder rather than JsonBuilder & JsonSlurper.
Long running services - like containers - need DSL plugin implemented for them to overcome 5min timeout issue.
Send mail needs the ability to be able to read file contents and send them via email - not attachments, just reading text from a file
Class for manipulating the value stream manifest file needs to be created
The documentation needs to be updated to cover available service APIs
Framework needs a number of Jenkins samples created to show how to use it
The following use-case...
@Library('devops-framework')
import org.devops.*
@GrabResolver(name='devops-core', root='file:///Volumes/WorkDisk/GROOVY/devops-framework/devops-framework-core/target/lib/')
@Grab('org.devops.framework.core:devops-framework-core:0.0.1-SNAPSHOT')
// Import the framework classes...
import org.devops.framework.core.*
def config = [
property1: 'value1',
property2: 'value2',
property3: 'value3'
]
def label = "${JOB_NAME}".toLowerCase().replaceAll("[^A-Za-z0-9]", "-").replaceAll("(-)+", "-")
// This needs full specification, else you get a namespace conflict with Jenkins...
def intP = new org.devops.Integration(this, config)
node {
File fetchDir = new File("/Volumes/WorkDisk/tmp/BuildJobs/")
File targetRepo = new File("/Volumes/WorkDisk/tmp/Repos/")
File targetRepoFile = new File(targetRepo.getAbsolutePath()+File.separator+"repo.json")
String slackURI = "https://hooks.slack.com/services/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"
//
// Setup the Integration process...
//
// Where am I building...
println "Using temporary directory "+fetchDir.getAbsolutePath()
// Register a getComponentList callback...
intP.getComponentList(body:{
ComponentManifest manifest = new ComponentManifest(targetRepoFile)
println ">Get the component list...<"
manifest.getComponentList().each {
k, v -> println "Component list is \"${v.componentName}-${v.componentVersion}-${v.componentLocation}\""
}
manifest = null
})
// Register a prepareForDeploy callback...
intP.prepareForDeploy(body:{
println ">Prepare Work Area - clean my files up...<"
if (fetchDir.exists()) {
Utilities.deleteDirs(fetchDir)
}
fetchDir.mkdirs()
})
// Register a get deploy assets callback...
intP.getDeployAssets(body:{
ComponentManifest manifest = new ComponentManifest(targetRepoFile)
println ">Get deployment assets...<"
manifest.getComponentList().each {
k, v ->
StringBuffer outputStr = new StringBuffer()
// Parse the names of source and target...
String location = v.componentLocation
String fileName = location.substring(0,location.length()-v.componentVersion.length()-1)
fileName = fileName.substring(targetRepo.getAbsolutePath().length(),fileName.length())
File srcFile = new File(location)
File tFile = new File(fetchDir.getAbsolutePath()+File.separator+fileName)
// Perform copy...
boolean retStat = Repository.pullAssetFromRepo(ConfigPropertiesConstants.FILE,
srcFile,tFile,null,null,outputStr)
if (!retStat) {
println "Repo pull failed "+outputStr.toString()
Notifications.messageSlackChannel(slackURI,"${label}: Repo pull failed "+outputStr.toString())
error("Repo pull failed "+outputStr.toString())
}
srcFile = null
outputStr = null
}
manifest = null
})
String packageName = "tomCat-"+System.currentTimeMillis()
// Register a pre-deploy callback...
intP.preDeploy(body:{
ComponentManifest manifest = new ComponentManifest(targetRepoFile)
println ">Do pre-deploy...<"
File dockerFile = new File(fetchDir.getAbsolutePath()+File.separator+"Dockerfile")
String txt = new String()
txt = "FROM tomcat"
manifest.getComponentList().each {
k, v ->
// Parse the names of source and target...
String location = v.componentLocation
String fileName = location.substring(0,location.length()-v.componentVersion.length()-1)
fileName = fileName.substring(targetRepo.getAbsolutePath().length(),fileName.length())
File tFile = new File(fetchDir.getAbsolutePath()+File.separator+fileName)
//
// Copy files...
txt+= "\nCOPY "+tFile.getName()+" /usr/local/tomcat/webapps/"
}
txt+= "\nEXPOSE 8080"
txt+= "\nCMD [\"catalina.sh\", \"run\"]"
Utilities.writeFile(dockerFile,txt)
// Bake an image...
StringBuffer outputStr = new StringBuffer()
//boolean retStat = Container.buildContainer(ConfigPropertiesConstants.DOCKER,packageName.toLowerCase(),
// fetchDir,null,outputStr)
def retStat = devOpsFrameworkBuildContainerStep buildDirectory: fetchDir.getAbsolutePath(),
containerFile: dockerFile.getAbsolutePath(),
containerName: packageName.toLowerCase()
if (retStat) {
println "Bake worked"
Notifications.messageSlackChannel(slackURI,"${label}: Bake worked")
} else {
println "Bake failed "+outputStr
Notifications.messageSlackChannel(slackURI,"${label}: Bake failed "+outputStr.toString())
error("Bake failed "+outputStr.toString())
}
manifest = null
dockerFile.delete()
})
// Register a run-deploy callback...
intP.runDeploy(body:{
println ">Do deploy...<"
// Run an image...
StringBuffer outputStr = new StringBuffer()
boolean retStat = Container.runContainer(ConfigPropertiesConstants.DOCKER,
packageName.toLowerCase(),
"ls /usr/local/tomcat/webapps/jpetstore.war",
outputStr)
if (retStat && outputStr.toString().contains("/usr/local/tomcat/webapps/jpetstore.war")) {
println "Deploy worked"
Notifications.messageSlackChannel(slackURI,"${label}: Deploy worked")
} else {
println "Deploy failed "+outputStr
Notifications.messageSlackChannel(slackURI,"${label}: Deploy failed")
error("Deploy failed "+outputStr.toString())
}
})
// Register a post-deploy callback...
intP.postDeploy(body:{
println ">Post-deploy - clean my files up...<"
if (fetchDir.exists()) {
Utilities.deleteDirs(fetchDir)
}
fetchDir.mkdirs()
})
// Run the integration pipeline...
intP.runPipeline()
}
Will generate a serialisation error...
Running container build task...
Running on master
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
an exception which occurred:
in field org.devops.framework.core.ComponentManifest.manifest
in object org.devops.framework.core.ComponentManifest@cbf1c3b
in field com.cloudbees.groovy.cps.impl.BlockScopeEnv.locals
in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@169e3527
in field com.cloudbees.groovy.cps.impl.CallEnv.caller
in object com.cloudbees.groovy.cps.impl.FunctionCallEnv@7614196e
in field com.cloudbees.groovy.cps.Continuable.e
in object com.cloudbees.groovy.cps.Continuable@5b1b50a5
in field org.jenkinsci.plugins.workflow.cps.CpsThread.program
in object org.jenkinsci.plugins.workflow.cps.CpsThread@8e7e4ad
in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.threads
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7a5c51e2
in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7a5c51e2
Caused: java.io.NotSerializableException: org.devops.framework.core.ComponentManifest$ManifestFile
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:926)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.HashMap.internalWriteEntries(HashMap.java:1793)
at java.util.HashMap.writeObject(HashMap.java:1363)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callWriteObject(JDKSpecific.java:156)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:191)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1028)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
at java.util.concurrent.ConcurrentSkipListMap.writeObject(ConcurrentSkipListMap.java:1437)
at sun.reflect.GeneratedMethodAccessor511.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callWriteObject(JDKSpecific.java:156)
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:191)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1028)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1082)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1040)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:920)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.lambda$writeObject$0(RiverWriter.java:144)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:237)
at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:143)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:552)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:529)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgramIfPossible(CpsThreadGroup.java:516)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:440)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
This is caused by using a DSL callback + the ComponentManifest object. Something about the ManifestFile
definition causes an issue
The outputStr functionality in pull/push routines is not working.
Also needs test cases to support
Currently, the messageSlackChannel() method uses CURL OS CLI. It needs to be replaced with some form of Java or Groovy package to remove OS dependencies
The readme needs a few improvements to better describe the DevOps framework process flow
Jenkins still has the usual problems problems when trying to run OS jobs via standard Java or Groovy OS process runners.
Need to implement a Jenkins friendly runner using the Jenkins plugin-java API.
This needs the POM to also include Jenkins dependencies.
Also need to be able to detect and switch runner mode on the fly when in OS running and Jenkins running.
Framework routines that are written using Java style code need the appropriate final types added to stop parameters being changed
The dependencies in the pom.xml
files need to be updated as they are over a year old.
The plugins still work and build ok, but they are running on old versions open to security threats
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These problems occurred while renovating this repository. View logs.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
Warning
Renovate failed to look up the following dependencies: Failed to look up maven package org.devops.framework:devops-framework
, Failed to look up maven package org.devops.framework.core:devops-framework-core
.
Files affected: devops-framework-core/pom.xml
, devops-framework-pipeline/pom.xml
, devops-framework-plugin/pom.xml
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
.gitpod.Dockerfile
devops-framework-pipeline/src/main/resources/Dockerfile.linux-base
devops-framework-pipeline/src/main/resources/Dockerfile.linux-dev
docker/Dockerfile
maven 3.8-jdk-11
.github/workflows/codeanalysis.yaml
actions/checkout v3
actions/setup-java v3
actions/cache v3
actions/cache v3
.github/workflows/codeql-analysis.yml
actions/checkout v3
github/codeql-action v1
github/codeql-action v1
github/codeql-action v1
.github/workflows/ghcr-io-docker-build.yaml
actions/checkout v3
docker/setup-buildx-action b167a89702b8b5314c104ab9d211b3dcf774f2b1
docker/login-action f75d088332b07a08afadf6ac53c74509b9453f12
docker/metadata-action 314ddf6d4a82333bf1bc7630399df41bf68eba09
docker/build-push-action f2a1d5e99d037542a71f64918e516c093c6f3fc4
.github/workflows/maven.yml
actions/checkout v3
actions/setup-java v3
actions/checkout v3
devops-framework-core/pom.xml
org.devops.framework:devops-framework 0.0.1-SNAPSHOT
org.codehaus.groovy:groovy-all 2.5.19
org.apache.ant:ant 1.10.12
javax.mail:javax.mail-api 1.6.2
com.sun.mail:javax.mail 1.6.2
junit:junit 4.13.2
com.cloudbees:groovy-cps 1.34
org.codehaus.gmavenplus:gmavenplus-plugin 1.13.1
org.apache.maven.plugins:maven-compiler-plugin 3.10.1
org.codehaus.groovy:groovy-eclipse-compiler 2.9.3-01
org.codehaus.groovy:groovy-eclipse-batch 2.5.14-02
org.apache.maven.plugins:maven-eclipse-plugin 2.10
org.jacoco:jacoco-maven-plugin 0.8.8
org.apache.maven.plugins:maven-dependency-plugin 3.5.0
org.apache.maven.plugins:maven-surefire-plugin 3.0.0-M7
devops-framework-pipeline/pom.xml
org.devops.framework:devops-framework 0.0.1-SNAPSHOT
org.codehaus.groovy:groovy-all 2.5.19
org.devops.framework.core:devops-framework-core 0.0.1-SNAPSHOT
junit:junit 4.13.2
com.cloudbees:groovy-cps 1.34
org.codehaus.gmavenplus:gmavenplus-plugin 1.13.1
org.apache.maven.plugins:maven-compiler-plugin 3.10.1
org.codehaus.groovy:groovy-eclipse-compiler 2.9.3-01
org.codehaus.groovy:groovy-eclipse-batch 2.5.14-02
org.apache.maven.plugins:maven-eclipse-plugin 2.10
org.jacoco:jacoco-maven-plugin 0.8.8
org.apache.maven.plugins:maven-assembly-plugin 3.4.2
org.apache.maven.plugins:maven-surefire-plugin 3.0.0-M7
devops-framework-plugin/pom.xml
org.devops.framework:devops-framework 0.0.1-SNAPSHOT
org.codehaus.groovy:groovy-all 2.5.19
org.devops.framework.core:devops-framework-core 0.0.1-SNAPSHOT
org.apache.ant:ant 1.10.12
junit:junit 4.13.2
com.cloudbees:groovy-cps 1.34
org.jenkins-ci.plugins.workflow:workflow-step-api 2.24
org.jenkins-ci.plugins.workflow:workflow-api 2.47
org.jenkins-ci.plugins.workflow:workflow-job 2.42
org.jenkins-ci.plugins.workflow:workflow-cps 2803.v1a_f77ffcc773
org.jenkins-ci.plugins.workflow:workflow-basic-steps 2.24
org.jenkins-ci.plugins:structs 1.24
org.codehaus.gmavenplus:gmavenplus-plugin 1.13.1
org.apache.maven.plugins:maven-compiler-plugin 3.10.1
org.codehaus.groovy:groovy-eclipse-compiler 2.9.3-01
org.codehaus.groovy:groovy-eclipse-batch 2.5.14-02
org.apache.maven.plugins:maven-eclipse-plugin 2.10
org.jacoco:jacoco-maven-plugin 0.8.8
org.apache.maven.plugins:maven-assembly-plugin 3.4.2
org.apache.maven.plugins:maven-surefire-plugin 3.0.0-M7
pom.xml
org.jenkins-ci.plugins:plugin 4.56
com.sonatype.clm:clm-maven-plugin 2.42.0-01
org.owasp:dependency-check-maven 6.5.3
The environment used for the unit-tests with Git actions does not work due to components not being installed - like ansible.
These comments either need to be installed into the build environment or the tests skipped.
Interface methods need to be added to the SCM and Container classes to put in tool specific layers on top of the generic interface
Get groovy doc plugin to work with the pom.xml
Add service support for Nexus OSS repo
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.