Code Monkey home page Code Monkey logo

lockable-resources-plugin's Introduction

Jenkins Lockable Resources Plugin

Jenkins Plugin GitHub release Jenkins Plugin Installs Build Status GitHub license Maintenance Crowdin Join the chat at https://gitter.im/jenkinsci/lockable-resources

This plugin allows defining lockable resources (such as printers, phones, computers, etc.) that can be used by builds. If a build requires a resource which is already locked, it will wait for the resource to be free.


Support

“Open source” does not mean “includes free support”

You can support the contributor and buy him a coffee. coffee Every second invested in an open-source project is a second you can't invest in your own family / friends / hobby. That`s the reason, why supporting the contributors is so important.

Thx very much for supporting us.


Usage

Adding lockable resources

  1. In Manage Jenkins > Configure System go to Lockable Resources Manager
  2. Select Add Lockable Resource

Each lockable resource has the following properties:

  • Name - A mandatory name (not containing spaces!) for this particular resource, i.e. DK_Printer_ColorA3_2342
  • Description - Optional verbose description of this particular resource, i.e. Printers in the Danish Office
  • Labels - Optional space-delimited list of Labels (A label can not containing spaces) used to identify a pool of resources. i.e. DK_Printers_Office Country:DK device:printer, DK_Printer_Production, DK_Printer_Engineering
  • Reserved by - Optional reserved / locked cause. If non-empty, the resource will be unavailable for jobs. i.e. All printers are currently not available due to maintenance. This option is still possible, but we recommend to use the page <jenkinsRootUrl>/lockable-resources/

A resource is always the one thing that is locked (or free or reserved). It exists once and has an unique name (if we take the hardware example, this may be office_printer_14). Every resource can have multiple labels (the printer could be labeled dot-matrix-printer, in-office-printer, a4-printer, etc.). All resources with the same label form a "pool", so if you try to lock an a4-printer, one of the resources with the label a4-printer will be locked when it is available. If all resources with the label a4-printer are in use, your job waits until one is available. This is similar to nodes and node labels.

Using a resource in a freestyle job

When configuring the job, select This build requires lockable resources. Please see the help item for each field for details.

Using a resource in a pipeline job

When the lock step is used in a Pipeline, if the resource to be locked isn't already defined in the Jenkins global configuration, an ephemeral resource is used: These resources only exist as long as any running build is referencing them.

Examples:

Acquire lock

Example for scripted pipeline:

echo 'Starting'
lock('my-resource-name') {
  echo 'Do something here that requires unique access to the resource'
  // any other build will wait until the one locking the resource leaves this block
}
echo 'Finish'

Example for declarative pipeline:

pipeline {
  agent any

  stages {
    stage("Build") {
      steps {
        lock(label: 'printer', quantity: 1, resource : null) {
          echo 'printer locked'
        }
      }
    }
  }
}

Take first position in queue

lock(resource: 'staging-server', inversePrecedence: true) {
    node {
        servers.deploy 'staging'
    }
    input message: "Does ${jettyUrl}staging/ look good?"
}

It is not allowed to mixed inversePrecedence and priority.

start time job resource inversePrecedence
00:01 j1 resource1 false
00:02 j2 resource1 false
00:03 j3 resource1 true
00:04 j4 resource1 false
00:05 j5 resource1 true
00:06 j6 resource1 false

Resulting lock order: j1 -> j5 -> j3 -> j2 -> j4 -> j6

lock (queue) priority

lock(resource: 'staging-server', priority: 10) {
    node {
        servers.deploy 'staging'
    }
    input message: "Does ${jettyUrl}staging/ look good?"
}
start time job resource priority
00:01 j1 resource1 0
00:02 j2 resource1 <default == 0>
00:03 j3 resource1 -1
00:04 j4 resource1 10
00:05 j5 resource1 -2
00:06 j6 resource1 100

Resulting lock order: j1 -> j6 -> j4 -> j2 -> j3 -> j5

Resolve a variable configured with the resource name and properties

lock(label: 'some_resource', variable: 'LOCKED_RESOURCE') {
  echo env.LOCKED_RESOURCE
  echo env.LOCKED_RESOURCE0_PROP_ABC
}

When multiple locks are acquired, each will be assigned to a numbered variable:

lock(label: 'some_resource', variable: 'LOCKED_RESOURCE', quantity: 2) {
  // comma separated names of all acquired locks
  echo env.LOCKED_RESOURCE

  // first lock
  echo env.LOCKED_RESOURCE0
  echo env.LOCKED_RESOURCE0_PROP_ABC

  // second lock
  echo env.LOCKED_RESOURCE1
  echo env.LOCKED_RESOURCE1_PROP_ABC
}

Skip executing the block if there is a queue

lock(resource: 'some_resource', skipIfLocked: true) {
  echo 'Do something now or never!'
}

Detailed documentation can be found as part of the Pipeline Steps documentation.

Jenkins label parser allows sophisticated filtering

The plugin uses the Jenkins-internal label parser for filtering lockable resources. A full list of supported operators and syntax examples can be found in the official documentation.

lock(label: 'labelA && labelB', variable : 'someVar') {
    echo 'labelA && labelB acquired by: ' + env.someVar;
}

lock(label: 'labelA || labelB', variable : 'someVar') {
    echo 'labelA || labelB acquired by: ' + env.someVar;
}

lock(label: 'labelA || labelB || labelC', variable : 'someVar', quantity : 100) {
    echo 'labelA || labelB || labelC acquired by: ' + env.someVar;
}

Multiple resource lock

lock(label: 'label1', extra: [[resource: 'resource1']]) {
	echo 'Do something now or never!'
}
echo 'Finish'
lock(
  variable: 'var',
  extra: [
    [resource: 'resource4'],
    [resource: 'resource2'],
    [label: 'label1', quantity: 2]
  ]
) {
  def lockedResources = env.var.split(',').sort()
  echo "Resources locked: ${lockedResources}"
}
echo 'Finish'

More examples are here.


Node mirroring

Lockable resources plugin allow to mirror nodes (agents) into lockable resources. This eliminate effort by re-creating resources on every node change.

That means when you create new node, it will be also created new lockable-resource with the same name. When the node has been deleted, lockable-resource will be deleted too.

Following properties are mirrored:

  • name.
  • labels. Please note, that labels still contains node-name self.
  • description.

To allow this behavior start jenkins with option -Dorg.jenkins.plugins.lockableresources.ENABLE_NODE_MIRROR=true or run this groovy code.

System.setProperty("org.jenkins.plugins.lockableresources.ENABLE_NODE_MIRROR", "true");

Note: When the node has been deleted, during the lockable-resource is locked / reserved / queued, then the lockable-resource will be NOT deleted.


Improve performance

To be safe thread over all jobs and resources, need to be all operations synchronized. This might lead to slow down your jobs. The jenkins self, has low CPU usage, but your jobs are very slow. Why?

The most time are spend to write the lockable-resources states into local file system. This is important to get the correct state after Jenkins reboots.

To eliminate this saving time has been added a property DISABLE_SAVE.

  • The best way is to use it with JCaC plugin. So you are sure, you have still correct resources on Jenkins start.
  • When you set pipeline durability level to PERFORMANCE_OPTIMIZED, it makes also sense to set this property to true.

Note: Keep in mind, that you will lost all your manual changes!

Note: This option is experimental. It has been tested in many scenarios, but no body know.

To allow this behavior start jenkins with option -Dorg.jenkins.plugins.lockableresources.DISABLE_SAVE=true or run this groovy code.

System.setProperty("org.jenkins.plugins.lockableresources.DISABLE_SAVE", "true");

Detailed lock cause

Tle plugin step lock() will inform you in the build log detailed block cause. The size of cause depends on count of ordered resources and size of current queue. To eliminate big unreadable logs we limited the size. To see all cause change the properties as follow:

System.setProperty("org.jenkins.plugins.lockableresources.PRINT_BLOCKED_RESOURCE", "-1");
System.setProperty("org.jenkins.plugins.lockableresources.PRINT_QUEUE_INFO", "-1");

PRINT_BLOCKED_RESOURCE means how many of ordered resources are printed. Per default 2. PRINT_QUEUE_INFO how many queue items are printed. Per default 2.

0 means disabled -1 means all / unlimited.

Configuration as Code

This plugin can be configured via Configuration-as-Code.

Example configuration

unclassified:
  lockableResourcesManager:
    declaredResources:
      - name: "S7_1200_1"
        description: "S7 PLC model 1200"
        labels: "plc:S7 model:1200"
        reservedBy: "Reserved due maintenance window"
      - name: "S7_1200_2"
        labels: "plc:S7 model:1200"
      - name: "Resource-with-properties"
        properties:
          - name: "Property-A"
            value: "Value"

Properties description, labels and reservedBy are optional.


lockable-resources overview

The page <jenkinsRootUrl>/lockable-resources/ provides an overview over all lockable-resources.

Resources

Provides an status overview over all resources and actions to change resource status.

Name Permission Description
Reserve RESERVE Reserves an available resource for currently logged user indefinitely (until that person, or some explicit scripted action, decides to release the resource).
Unreserve RESERVE Un-reserves a resource that may be reserved by some person already. The user can unreserve only own resource. Administrator can unreserve any resource.
Unlock UNLOCK Unlocks a resource that may be or not be locked by some job (or reserved by some user) already.
Steal lock STEAL Reserves a resource that may be or not be locked by some job (or reserved by some user) already. Giving it away to currently logged user indefinitely (until that person, or some explicit scripted action, later decides to release the resource).
Reassign STEAL Reserves a resource that may be or not be reserved by some person already. Giving it away to currently logged user indefinitely (until that person, or some explicit scripted action, decides to release the resource).
Reset UNLOCK Reset a resource that may be reserved, locked or queued.
Note RESERVE Add or edit resource note.

Labels

Provides an overview over all lockable-resources labels.

Note: Please keep in mind, that lockable-resource-label is not the same as node-label!

Queue

Provides an overview over currently queued requests. A request is queued by the pipeline step lock(). When the requested resource(s) is currently in use (not free), then any new request for this resource will be added into the queue.

A resource may be requested by:

  • name, such as in lock('ABC') { ... }
  • label, such as in lock(label : 'COFFEE_MACHINE')

Note: Please keep in mind that groovy expressions are currently supported only in free-style jobs. Free-style jobs do not update this queue and therefore can not be shown in this view.

Note: An empty value in the column 'Requested at' means that this build has been started in an older plugin version - 1117.v157231b_03882 and early. In this case we cannot recognize the timestamp.


Upgrading from 1102.vde5663d777cf

Due an issue is not possible anymore to read resource-labels from the config file org.jenkins.plugins.lockableresources.LockableResourcesManager.xml, which is generated in the release 1102.vde5663d777cf

This issue does not effect instances configured by Configuration-as-Code plugin.

A possible solution is to remove the <string> tags from your org.jenkins.plugins.lockableresources.LockableResourcesManager.xml config file manually, before you upgrade to new version (Keep in mind that a backup is still good idea).

Example:

change this one

<labels>
  <string>tests-integration-installation</string>
</labels>

to

<labels>
  tests-integration-installation
</labels>

Changelog


Report an Issue

Please report issues and enhancements through the Jenkins issue tracker in GitHub


Contributing

Contributions are welcome, please refer to the separate CONTRIBUTING document for details on how to proceed! Join Gitter channel to discuss your ideas with the community.


License

All source code is licensed under the MIT license. See LICENSE

lockable-resources-plugin's People

Contributors

abayer avatar adelcast avatar aeon512 avatar akiasi avatar amuniz avatar basil avatar bbara avatar bozaro avatar cronik avatar dcendents avatar dependabot[bot] avatar github-actions[bot] avatar gounthar avatar jimklimov avatar jmmeessen avatar kddubb avatar markewaite avatar mpokornyetm avatar niranjanib avatar offa avatar oleg-nenashev avatar primoz-p avatar r3d-dragon avatar rjarry avatar sheepdreamofandroids avatar sterys avatar tobix avatar tomasbjerre avatar wadeck avatar yrodiere avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

lockable-resources-plugin's Issues

No such property: resource for class: groovy.lang.Binding

For the last few months I've been using a build of #50 as I need to know the name of the resource that I am locking. I recently noticed that PR was closed in favor of #49, so I again went back to using the official build of the lockable resources plugin.

However, now I am getting the following error when a build tries to acquire a lock after having to wait due to all available resources being locked:

16:37:29 Trying to acquire lock on [Label: auto, Quantity: 1]
16:37:29 Found 0 available resource(s). Waiting for correct amount: 1.
16:37:29 [Label: auto, Quantity: 1] is locked, waiting...
16:39:06 Lock acquired on [Label: auto, Quantity: 1]
[Pipeline] {
[Pipeline] }
16:39:06 Lock released on resource [Label: auto, Quantity: 1]
[Pipeline] // lock
[Pipeline] echo
16:39:06 groovy.lang.MissingPropertyException: No such property: resource for class: groovy.lang.Binding

As long as there is a resource available when the lock is acquired, things work fine. It's just when we need to wait for a lock that we get this error.

It's also worth noting that #50 didn't suffer from this problem

lock resource queue is not working.

i have 13 resources with label atResource, so the first 13 resources are locked and automation runs on all 13 of them but after that the wait doesnt happens it directly return null and automation is triggered on null resources, this was working fine in older release until we migrated recently to latest one.
//called 20 times
lock(label: atResource, quantity: 1) {
//each time lock a resource and run automation on it.
runAutomation(env.resource0)
}

Note:after migration to latest one the step is changed as below as env.resource0 no more works
lock(label: atResource, quantity: 1,variable:"resource0") {
//each time lock a resource and run automation on it.
runAutomation(env.resource0)
}

Feature Request: Show all jobs waiting for resource

Hi,
I have a problem. We have several hundred build jobs and e.g. sometimes they share the same resources, e.g. db or integration servers, deployment targets and so on. In the lockable resource list, I have listed all active locks, which job currently uses the lock and the possibility to lock or unlock the resource.

Is there any possibility to show the jobs waiting for the lockable resource or may you add a possibility to view this?

That would be great.

Yours

Christian

Using QueueTaskDispatcher to block jobs using a locked resource before deploying to node

Hi,

I don't know if I'm writing this in the correct place, but I have been wondering about how to solve the following problem:

Given:
A declarative Jenkinsfile 'A' that references label 'B':
Pipeline { agent { label: 'B'} ... }

One of the stages in the Pipeline takes Lock 'L'.

When:

  • There exists N jobs using Jenkinsfile 'A'
  • K nodes with labels 'B'

Then:
If any job of the set N takes the lock 'L', I want the rest N-1 to be blocked with CauseOfBlockage set to BecauseLabelIsBusy.

So only 1 node should be running and the rest K-1 should be free for other jobs of another set to execute on.

Note: This is needed to lock resources to different hardware setups, in reality the lock string is a parameter passed by the job. Some jobs don't need any resource and thus will pass BUILD_TAG as a unique lock string.
This way we should be able to have higher utilization of the nodes and avoid having dedicated nodes for specially constrained jobs.

I have been looking around and I found two plugins that if combined in someway might give a good solution:

  • Lockable plugin (to manage the locking mechanism)
  • Build-blocker-plugin (to hook to QueueTaskDispatcher extension point)

Am I looking in the right direction? or can this be totally done by the Lockable plugin?

Thank you in advance for your feedback!

In version 2.0 can't execute Windows batch or Ant script

I have a problem to use version 2.0 on Windows Server 2008. After locking resources with name dev-build-sql I can't execute Windows batch or call Ant script. I'm getting java error:

[lockable-resources] acquired lock on [dev-build-sql]
Building in workspace D:\jenkins\jobs\Test-lock\workspace
[workspace] $ cmd /c call C:\Users\CSIDEV\AppData\Local\Temp\jenkins583201663619856164.bat
The parameter is incorrect
FATAL: command execution failed
java.io.IOException: CreateProcess error=87, The parameter is incorrect
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
Caused: java.io.IOException: Cannot run program "cmd" (in directory "D:\jenkins\jobs\Test-lock\workspace"): CreateProcess error=87, The parameter is incorrect
at java.lang.ProcessBuilder.start(Unknown Source)
at hudson.Proc$LocalProc.(Proc.java:245)
at hudson.Proc$LocalProc.(Proc.java:214)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:850)
at hudson.Launcher$ProcStarter.start(Launcher.java:384)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:730)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
at hudson.model.Run.execute(Run.java:1735)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:415)
Build step 'Execute Windows batch command' marked build as failure
[lockable-resources] released lock on [dev-build-sql]
Finished: FAILURE

In case no lock is set batch executed successfully. I suspect that setting environment variable during resource lock operation is causing the error. Either the name of the environment variable is not compatible with windows environment, or the value is too long.

Changing resource name to DEV_BUILD_SQL didn't help. 

Forced to rollback to the plugin version 1.10.

Lock blocks queue while node is free

Hi,

Example for incorrect behavior

Lockable_job_1 is running with LOCK_A on node 120.

Starting Lockable_job_2 with LOCK_A on node 110, but getting locked by LOCK_A, so it remains in the waiting queue, which is the correct behavior.

Starting Lockable_job_3 with LOCK_B on node 110, but it gets locked by Lockable_job_B in the queue, while node 110 is free.

I think the plugin should not block the entire queue.

Remove locks from the Lockable Resources Manager

Hi,

We are using some Multibranch piplines and we are creating branch-specific locks. We now have hundreds of unused locks in our Lockable Resources Manager.

Is there a way to delete them or to clean this list at the end of a build? or to unpersist them in this list?

Lockable resource with null name causes NPE

let's have two lockable resources defined "lock1" and a resource with null name, following pipeline will cause NPE

pipeline {

    agent none
    options { skipDefaultCheckout() }

    stages {

        stage('Distro') {
            agent {
                node {
                    label 'node1'
                }
            }
            steps {
                lock("lock1") {
                    echo "hello"
                }
            }
        }
    }
    
    post {

        always {
            node("node1") {
                echo "hello post"
            }
        }
    }
}
java.lang.NullPointerException
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.freeResources(LockableResourcesManager.java:316)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:355)
	at org.jenkins.plugins.lockableresources.LockStepExecution$Callback.finished(LockStepExecution.java:84)
	at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
	at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:166)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Lock won't clear

Jenkins ver. 2.60.3

I have a lock that won't clear. When clearing a lock from the console:

Stack trace

java.lang.NullPointerException
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.getNextQueuedContext(LockableResourcesManager.java:440)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlockNames(LockableResourcesManager.java:353)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:342)
	at org.jenkins.plugins.lockableresources.LockableResourcesManager.unlock(LockableResourcesManager.java:330)
	at org.jenkins.plugins.lockableresources.actions.LockableResourcesRootAction.doUnlock(LockableResourcesRootAction.java:104)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:343)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:184)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:117)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:129)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
Caused: javax.servlet.ServletException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:765)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
	at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:86)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

The build that holds the lock died on a Jenkins bug without clearing.

Have tried updating the plugin etc without success. Is there a command-line way to remove the lock?

Critical section in pipeline is entered without lock

It seems that sometimes the critical section in pipeline is entered without lock. Please see the job configs below, which are "a bit" simplified from our real ones. The problem happens in the following scenario: JOB1 is running a build and has locked the resource:"lab1" via "CI_LAB"-label. I trigger JOB2, the build for JOB2 correctly starts waiting for the lock on "lab1". When JOB1 ends, JOB2 enters the critical section, and the lock tells that it got the lock for "lab1" - but when I look at the "Lockable Resources" UI in Jenkins, the resource has no lock, and other jobs can lock the resource even though JOB2 is still in the critical section!

lockable resources plugin: 1.11.2
jenkins: 1.644
pipeline:2.4

JOB1 does:

LAB_LABEL="CI_LAB"
SEQUENCER_LABEL="LONG_RUN_SEQUENCER"
lock ( label: "$SEQUENCER_LABEL") {
    lock (label: "$LAB_LABEL", Quantity:1) {
        // job1 critical section
    }
}

JOB2 does:

LAB_RESOURCE=lab1
lock ( "$LAB_RESOURCE" ) {
   // job2 critical section
}

We have the following lockable resources(simplified listing for the problem scenario):

Name:LONG_RUN_SEQUENCER, Labels:LONG_RUN_SEQUENCER //only one resource with this label
Name:lab1, Labels:LAB1 CI_LAB
Name:lab2, Labels:LAB2 CI_LAB

Ideas:

  • could the nested lock in JOB1 cause this somehow?
  • or could it be caused by JOB1 having locked the resource using label, while JOB2 locks the same resource using the resource name?
  • or maybe just some other concurrency bug?

Groovy Expression for pipeline

I would like to use this plugin in some more advanced scenarios.
For example

  • I would like to take all resources with label but with specific description
  • I would like to exclude some resources
  • I would like that every retry to lock different resource
    retry(3) {
    lock(label: "LABEL", variable: "VARIABLE", quantity: 1) {
    def v = VARIABLE // never the same
    }
    }

I would be able to handle every case with groovy expression for lock block.

No changelog for version 2.0

Jenkins says

Warning: the new version of this plugin claims to use a different settings format than the installed version. Jobs using this plugin may need to be reconfigured, and/or you may not be able to cleanly revert to the prior version without manually restoring old settings. Consult the plugin release notes for details.

but I can't find any changelog for the new version. Can you explain what was changed?

How to remove all resources with label

I'm using pipeline workflow to create some resources with label and after the build is done I want to delete/remove all resources with this label.

To create it i'm using this lines for example:

LockableResourcesManager.get().createResourceWithLabel("my_resource1", "my_label")
LockableResourcesManager.get().createResourceWithLabel("my_resource2", "my_label")

and now i need to delete all resources with "my_label" using pipeline (groovy)
is there a way to do it? (even loop each of "my_resourcesX" and delete it)
can you add deleteResourcesWithLabel("my_lable")
or lockableResource.delete()

History

Please make possible to get the history of usage of a lockable resource

Feature request: ability to add/remove lockable resources through API

It would be really useful to able to add/remove lockable resources via an API (either REST or through the groovy console). This would enable the locks to represent real world non-static things - i.e. as you scale your infrastructure, you can scale your locks to reflect this.

Can not lock on Labels in Free-flow jobs

Originally documented here - probably with better formatting than here

Now added to Jenkins JIRA JENKINS-44388

The current 2.0 version does apparently not work with labels, when used in a free-flow job!

I set up a small test scenario to play with, and it seems like waiting for resources works, but not waiting for Labels.

My tests below:
I used a resource named TEST, with two labels TEST_1 and TEST_2.
Looking at the available resources, it's reported that I have the "TEST" resource, and one instance of the two labels - as expected.
In a free-flow job, I enable This build requires lockable resources, and the tested the following four setups:

Trial 1
Resources = TEST
Number of resources to request = 0
Expectation: The resource TEST and all labels consumed during the test.
On execution:
The test project does not wait for anything.
None of the resources nor labels are reported used.
Note: This might be correct behaviour - provided 0 is used literal, unlike documented in the plugin: "empty value or 0 means all "

Trial 2
Resources = "TEST"
Number of resources to request = 1
Expectation: All labels on the TEST resource will be consumed.
On execution:
The test project does wait for the TEST resource and does consume all Labels.

Trial 3
Lable = "TEST_1"
Number of resources to request = 0
Expectation: The TEST_1 label will be consumed, leaving "TEST_2" available.
On execution:
The test project does not wait for anything.
None of the resources nor labels are reported used.
Note: This might be correct behaviour - provided 0 is used literal, unlike documented in the plugin: "empty value or 0 means all "

Trial 4
Lable = "TEST_1"
Number of resources to request = 1
Expectation: The TEST_1 label will be consumed, leaving "TEST_2" available.
On execution:
The test project does not wait for anything.
None of the resources nor labels are reported used.
Trial 4 is the error I'm really concerned about, as it seems to indicate that only resources can be taken, and not labels.

Question:Rest API to change label of a resource.

Hi,

where can i see the rest api documentations, is there a way i can change the label of a resource.?

want to achieve below using rest api.
selected.get(0).setLabels(selected.get(0).getLabels() + "_Fault");

regards,
Vikash

Jobs waiting to lock resource could be searching for newly created resources

FEATURE REQUEST

It would be nice feature if we could on the fly add resources, while jobs are already waiting for resources with specified label to lock.
So, if a new resource added, (after the jobs started to run), matches the requirements, the resource should be acquired normally.

CURRENT BEHAVIOR

Resources added after job started polling for resources to lock, are not visible to the job.

Support for locking for multiple resource groups

It seems that one Jenkins build can only get a resource lock on one resource group. I'd like to get a lock on multiple resource groups.

RASPBERRIES (label): RP_1, RP_2, RP_3
TABLES (label): GALAXY_1, GALAXY_2, GALAXY_3

My build has a place where I can specify that I'd like a lockable resoirce from RASPBERRIES but I cannot get a lock on one RP_X and one GALAXY_X

Acquiring of empty resource if timeout is expired

Following the hard-coded timeout is for how long the resource might be queued(reserved) before it's acquired.
Class LockableResource.java:
private static final int QUEUE_TIMEOUT = 5;
If the timeout is expired a resource will be unqueued from the queue and an empty resource will be acquired by plugin:
[lockable-resources] acquired lock on []
The possible workaround might be to increase the timeout to some significant value.
However, it does not fix the issue. Plugin should check if acquired resource is empty and somehow react, for example, ask to acquire a resource one more time.

Mixing direct/inverse precedence locks doesn't yield the expected result

We have a jenkins job that uses locks to avoid running a specific stage in parallel even if the job has multiple instances running at the same time.

We normally lock the resource with inversePrecedence: false to have a FIFO-like behaviour. In some cases, however, we want one specific run to 'jump' over the queue, and that is possible with a build-specific parameter. However, the build that acquires the lock with inversePrecedence: true is still executed after all the other builds are done.

The LIFO-like behaviour is only respected when all the locks are acquired with inversePrecedence: true.

It would be great if the plugin could handle this mixed behaviour.

Since 2.0, resource locks are broken in multi configuration projects

See https://issues.jenkins-ci.org/browse/JENKINS-43574

For multi configurations projects, lockable resources no longer behave as they should.

In my setup, I have a single lockable resource, which is locked by name, and a single multi configuration job with 2 different configurations executing on different slave nodes, locking that resource.
The parent job itself is executed on a 3rd slave node, by random scheduling choice.

When the parent project starts, all child jobs are enqueued and the first child job starts, while the other job is enqueued. The web interface shows, that the second job would still be waiting on the resource, and the resource itself shows up as "queued" state (not locked!).

After a short delay, the second job also starts, in parallel with the other child job requiring exactly the same resource. The resource now shows as "free", despite both jobs still running.

Both child jobs have the following line in their log:

[lockable-resources] acquired lock on []

As you can see, no resource has actually been locked.

Downgrading to 1.11.2 resolves the issue for me, so the breaking change must have been introduced with any of the changes in 2.0.

Nothing else in an of the jenkins logs, hinting at what could possibly have gone wrong.

Plugin doesn't lock the quantity requested

When I specify that I require a quantity of 'n' resources in a lock, the plugin waits for at least that quantity to become available, but then acquires all locks, not just the quantity required.

We have a pipeline with a fast compile-and-test step followed by longer running end-to-end tests. Every compile-and-test step triggers the end-to-end tests. I want to run 2 concurrent end-to-end tests and I want to supersede any old ones still queued, because we can’t get through them fast enough.

I have created two lockable resources test-queue-1 and test-queue-2, each with the label test-queue. I set up a lock in the end-to-end tests like this:

lock(label: 'test-queue', inversePrecedence: true, quantity: 1) {

    milestone()

    // run end-to-end tests...
}

I had hoped that this would acquire one of the two lockable resources with label test-queue, leaving the other for a second concurrent build. What actually happens is that both resources get locked, so subsequent builds queue until both resources are released. At that point, we run the latest queued build and kill off earlier builds (the superseding logic works). I just can't set up 2 concurrent builds.

I think it should lock the quantity requested.

Unclear about declarative pipeline

I am trying to rewrite my pipeline script for maintainablity but am unclear how to lock the resource for the entire script, is it possible to do something like:

pipeline {
    agent any 
    resource: ('lockable resource')

    stages {
        stage('Build') { 
            steps { 
                sh 'make' 
            }
        }
        stage('Test'){
            steps {
                sh 'make check'
                junit 'reports/**/*.xml' 
            }
        }
        stage('Deploy') {
            steps {
                sh 'make publish'
            }
        }
    }
}

checksum mismatch when trying to install plugin

Trying to install version 2.0 plugin from Jenkins 2.64, and getting this error

java.io.IOException: Downloaded file /var/lib/jenkins/plugins/lockable-resources.jpi.tmp does not match expected SHA-1, expected 'tuLn+UmElP/9wLnBhRme0QZAues=', actual 'f9lfA56w1mbVlpFYsOkHsb3HRDs='
	at hudson.model.UpdateCenter.verifyChecksums(UpdateCenter.java:1810)
	at hudson.model.UpdateCenter.access$1100(UpdateCenter.java:149)
	at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1963)
	at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1194)
	at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1680)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1874)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1651)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
	at java.lang.Thread.run(Thread.java:745)

Flaked tests on LockStepTest

Sometime test failures occurs on LockStepTest and for avoiding the build to fail flaked tests have been allowed through Maven Surefire Plugin.

Instead of allowing flaked tests it could be a better strategy to find the reason why the test fail and fix the root problem.

Any idea about these test failures?

https://jenkins.ci.cloudbees.com/job/plugins/job/lockable-resources-plugin/101/console
https://jenkins.ci.cloudbees.com/job/plugins/job/lockable-resources-plugin/102/console

Request: Delete resources after they haven been used

Hy there,
I want to use this plugin to coordinate some of my tests in my pipeline.
There are MANY tests I need to coordinate.
Every test requires a lock ("Path-" + actualPath) { // do test}
and therefore creates a lockable resource.
This is good for the moment, but after the test has been done this results in a list of over 50 (many more tests are going to be migrated to pipeline) more or less useless lockable resources that crowd the lockable resource list and make it unusable.
This makes finding a consistent lockable resource like 'testing on testing3' or 'prod_lock' very hard.

Is it possible to make resources temporary with lock (nameOfResource:String, diposeAfterUse:boolean) {}

Do you know another way to automatically dispose a useless lockable resource?

Thank you :)

how to declare total available quantity in pipeline?

I see a lot of examples on how to use the lock resource to request quantities to lock,

lock(resource: '2GB-RAM', quantity: 2) {
...
}

but I havent found an example that declares how many total quantity for a particular resource there are.

Can anyone provide an example of that?

Injecting Environment Variables based on resource selected

Would it be possible to have environment variables associated with each resource? I have 3 identical embedded boards I'm testing with as my resources. Each board has unique:

  • Nework Discovery IDs
  • Static IP Addresses
  • JTAG-POD Addresses

Once one of the boards is selected, I'd like the Job to be able to pick up these values from the resource and use them when running shell or batch scripts.

Many thanks.

P.S. Excellent plug-in!

Non-blocking or timeouted lock step

It would be nice to have a tryLock; body; releaseLock execution flow in addition to an existing lock { body }.
This will allow to implement any custom resource selection logic, i.e.

  1. Try lock the best (fastest, cheapest, your personal favorite, etc) available environment.
  2. Oh, it is occupied at the moment, let's fallback to another one
body = { business logic here }
// say, I have two resources: "cheap" and "expensive", both share the label "test env"
lockObj = tryLock label:'cheap env'
if (! lockObj) // okay-okay, let's just wait for any of envs available
    lock label:'test env', body: body //no timeout here

Until label locking is not implemented, this can help work it around:

// I have 10 similar resources: env1, env2, env3...env10
i=0
waitUntil {
    i++
    if (i>10) {
        i=0 //enumarate again
        sleep 10 //make it a little less fanatical
    }
    tryLock "env$i" // should return something coerced to true on success
}

The lock obtaining func should return some lock object and user is obliged to save its value for further releasing. Good-old try-finally to ensure that resource is released:

try{
    lockObj = tryLock 'env'
} finally {
    lockObj.release() //this will help user not to forget to save a variable and will not allow to release others' resources.
}

Alternatively we could extend existing lock step with timeout filed which limits the phase of lock obtaining. The step itself shoud return boolean whether it manage to acquire a lock or not:

body = { business logic here }
// say, I have two resources: "cheap" and "expensive", both share the label "test env"
boolean lockObtained = lock label:'cheap env', body: body, timeout: 1 /*min*/
if (! lockObj) // okay-okay, let's just wait for any of envs available
    lock label:'test env', body: body //no timeout here

Feature request : weights or other preference to matched resources

Currently AFAIK this plugin returns a random (or first?) one of the available resources matched by the requested label(s) or groovy script.

This feature request is to provide an ability to attach "preference weights" to returned values, so some resources are more likely to be locked for a particular request (e.g. customized groovy expression in a Jenkins job).

Use-case: we are testing builds of our product on a farm of devices with several tests - relatively fast basic testing, and then those versions which passed the basic tests are subjected to longer integration testing. Tests do take quite a while, so sometimes we have several built versions of the software ready to be tested - so several boxes at once can be utilized by a test (one for each new version).

While any device can run the basic test, just a subset of them can be used for integration tests because only they have the special connections (further HW appendages) needed for those test suites to succeed. Sometimes boxes are also grabbed and locked by developers to experiment their new features or support investigations on hardware, so they fall out from the CI-testing pool for that time.

Currently all these boxes are labeled (for this plugin) to use for basic tests, and that subset with appendages is also labeled for the specific tests. Effectively any of the unequal boxes can get occupied by a basic test, so if all special boxes are running the basic test, the special tests queue up and wait.

With this requested feature in place, the boxes without appendages would have a greater preference in the basic test job - so a queue of new builds would first use these boxes for the basic tests, and would only spill over to use available special boxes if there are many enough builds to be tested - more than the currently available pool of simpler devices. This way the special boxes are mostly available to pick up incoming integration testing jobs right away, but still can help with basic testing in bigger bursts of development.

Bonus points: if both a basic and an integration testing job are waiting in queue, and a "special" device becomes available, an integration job should have a way to be awarded with this device first. Maybe some magic with an even higher preference weight?..

Queued resources with variable names lose reference to variable

With this pipeline code, and two available resources in 'resourcePool', stages that are queued lose the reference to the variable name.

def steps = [:]

steps['stage1'] = {
    lock(label: 'resourcePool', quantity: 1, variable: "STAGE1_VM"){
        echo "locked resource: ${env.STAGE1_VM}"
        echo sh(returnStdout: true, script: 'env')
        sleep 10
    }
}

steps['stage2'] = {
    lock(label: 'resourcePool', quantity: 1, variable: "STAGE2_VM"){
        echo "locked resource: ${env.STAGE2_VM}"
        echo sh(returnStdout: true, script: 'env')
        sleep 15
    }
}
steps['stage3'] = {
    lock(label: 'resourcePool', quantity: 1, variable: "STAGE3_VM"){
        echo "locked resource: ${env.STAGE3_VM}"
        //echo sh(returnStdout: true, script: 'env')
        sleep 5
    }
}
steps['stage4'] = {
    lock(label: 'resourcePool', quantity: 1, variable: "STAGE4_VM"){
        echo "locked resource: ${env.STAGE4_VM}"
        //echo sh(returnStdout: true, script: 'env')
        sleep 20
    }
}
steps['stage5'] = {
    lock(label: 'resourcePool', quantity: 1){
        echo "locking non variable resource"
        //echo sh(returnStdout: true, script: 'env')
        sleep 2
    }
}

node('master'){
    parallel steps
}

Lock when required

It would be an great feature to be able to specify in the build task list when to lock / unlock resources.
Say for example I have a build that takes 30 minutes to execute, but the locked resource in only in used during 3 minutes of that build (especially at the end of the build).

It would allow for a configuration such as this:

  • Task A (27 mins)
  • Lock Resource: X
  • Task B (3 mins)

This will allow multiple long builds to do a lot of their grunt work up until such a time where they need to await the unlock of a resource.

What are valid scopes for `lock` in a declarative pipeline job?

The documentation seems to imply that lock must be used within a step, but could you confirm?

I have a situation where I need to lock a resource while performing a step. I need to always cleanup that resource, so that my step works correctly.

step('foo') {
    lock('foo-resource') {
        bat '... acquire the resource...'
        bat '... use the resource...'
        bat '... cleanup the resource...'
    }
}

I need to move the cleanup step to a post / always, so the step reports the correct exit code, but now it's outside of the lock.

step('foo') {
    lock('foo-resource') {
        bat '... acquire the resource...'
        bat '... use the resource...'
    }
}
post {
    always {
        bat '... cleanup the resource...'
    }
}

I've got other jobs waiting for this resource and, based on the timing, they're trying to use the resource that this job is now cleaning up.


In the freestyle/UI-based job configuration, it looks like a lock is scoped to the entire build, not one step.

image

Groovy Expression in Free-Flow jobs is either broken or not documented enough to be useable.

Originally documented here - probably with better formatting than here

Now added to Jenkins JIRA as [JENKINS-44389|https://issues.jenkins-ci.org/browse/JENKINS-44389]

I was initially attempting to get the Lockable Resources Plugin to lock on a specific resource while also evaluation the value of a system environment variable.
Evaluation the system environment variable is working perfectly, however, I can't seem to get a lock on the correct resource.
This question focused on the specific locking problem!
I have created a three resources called A_TEST, B_TEST and C_TEST. None of them have any Labels. They are all visible from my Jenkins_URL/lockable-resources/ where they can be taken and released without problems.
In my Jenkins Job configuration, I have selected the This build requires lockable resources option, which allows me to specify a Resource, Label or Groovy Expression (and Additional classpath). It additionally allows me to specify Reserved resources variable name and Number of resources to request.
According to the documentation, the Groovy script must return a boolean value, so I determined to try it out:

Test 1
The first test I did was to verify the basic functionality, by setting the following:
Resource = B_TEST
Groovy Expression = Not checked
Number of resources to request = 1
This results in the job executing with a lock on the B_TEST resource. The console output reads:
[lockable-resources] acquired lock on [B_TEST]

Test 2
In this test I set the following:
Resource = B_TEST
Groovy Expression = Checked
Groovy Script = return false
Number of resources to request = 1
When attempting to execute the job, this wrongly waits forever with the text: (pending--Waiting for the resourced [B_TEST])

Test 3
In this test I set the following:
Resource = B_TEST
Groovy Expression = Checked
Groovy Script = return true
Number of resources to request = 1
This results in the wrong resource A_TEST to be locked. The console output reads:
[lockable-resources] acquired lock on [A_TEST]

Test 4
After rereading the help for each option in the plugin, I discovered that the plugin apparently only allows me to specify either a Resource, Label or Groovy Expression So in this test I set the following:
Groovy Expression = Checked
Groovy Script = return false
Reserved resources variable name = MyResourceName
This results in the job executing wrongly without a lock on any resource. The console output reads:
[lockable-resources] acquired lock on []

Test 5
So in this test I set the following:
Groovy Expression = Checked
Groovy Script = return true
Reserved resources variable name = MyResourceName
This results in the job wrongly locking on all resource. The console output reads:
[lockable-resources] acquired lock on [A_TEST, B_TEST, C_TEST]

Test 6
According to the documentation of the plugin, in Jenkins-Pipelines this plugin can be used as this:
echo 'Starting'
lock('my-resource-name') {
echo 'Do something here that requires unique access to the resource'
// any other build will wait until the one locking the resource leaves this block
}
echo 'Finish'
so I started experimenting with the Groovy Script containing variations of the lock('B_TEST') call, but this mostly lead to more confusion and errors while the job attempted to start, such as this:
No signature of method: Script1.lock() is applicable for argument types: (java.util.LinkedHashMap) values: [[resource:B_TEST]] Possible solutions: each(groovy.lang.Closure), wait(), run(), run(), any(), find())
But I guess this all makes good sense, as the lock(){ } call seems suited to take and release the lock only during its scope.

The Question
The big question now is, how does all of this actually work? My guess is that somewhere there's a groovy command to specify the Resources/Labels you want to reserve, and the return value (true/false) determines to actually take the lock or not.

Workflow support

Hi,

Anyone currently working on updating this plugin to be compatible with Workflow?

Feature request : deferred manual locking of resource by user

Users can click a button in Lockable Resources page on Jenkins to grab a resource. They can also "Unreserve+Reserve" a resource to take it away from a job that currently uses it - which can have adverse effects depending on that job's structure (e.g. calling sub-jobs using the same limited resource).

This feature proposes that there should be a way for users to request a resource (any by label, or specific by name) to be assigned to them after this resource is freed by its current consumer, as well as to cancel such pending requests.

The use-case is for example to schedule maintenance on hardware nodes of the test farm or validator postmortem investigation after failed tests (requesting a deferred lock of a specific device), or for developer use/experiments (deferred lock on any device with a label match).

Bonus points for notifications (IM, Mail) to the users when they are finally granted exclusive access to that resource - so they can begin working with it.

Add documentation example on how to pass build arguments into the Groovy script

Context: the Lockable Resources plugin includes an option to specify the needed resource by one of: exact resource name, matching a label, or evaluating a Groovy script for each configured resource that would return true if this resource is okay for this job. Then one of the "okay" resources (assuming "Number of resources to request" setting is 1) is reserved while the build runs.

My use-case, now solved by code in PR #72, is that we have a test farm, where usually any of the systems can be used for a test (so label-matching is in place), except when developers want to run the job against some particular system - and protect it from being used by any other job during this time. The build job (or actually a Multiphase job calling several others) has build arguments, including specification of the environment to use for the test (a "CONTROLLER" in examples below).

For the test, I used a build of the plugin with PR above integrated (made by mvn package), and in the job I selected "Meta Data / This build requires lockable resources", further selected "Groovy Expression" and entered the script below. The "Resources" and "Label" fields were left empty, and the "Number of resources to request" was set to 1 (disregard the warning that Given amount 1 is greater than amount of resources: 0.), and "Reserved resources variable name" is LOCKED_CONTROLLER.

Note the println lines below end up in jenkins.log, and can be pretty noisy, so comment them away when your job definition works well :)

// We need an available testbed resource marked with label "rc-validation-farm"
// If the user specified a particular testbed name in CONTROLLER var, require that one instead (regardless of labels)

/*
println "Inspecting the resource to lock for requested CONTROLLER='" +
    CONTROLLER + "' (looking at resourceName='" + resourceName +
    "' resourceDescription='" + resourceDescription + "' resourceLabels='"
    + resourceLabels + "')"
*/

if ( CONTROLLER.startsWith("LOCK_LABEL:") ) {
    def LOCK_LABEL = (CONTROLLER =~ /^LOCK_LABEL:(.*?)$/ )[0][1];
/*
    println "Looking for LOCK_LABEL='" + LOCK_LABEL + "' among '" +
        resourceLabels + "' for '" + resourceName + "' (" + resourceDescription + ")"
*/
    if (resourceLabels.contains( LOCK_LABEL )) {
//        println "ACCEPTED '" + resourceName + "'"
        return true;
    }
} else {
//    println "Looking for 'rc:" + CONTROLLER + "' in the name '" + resourceName + "'"
    if (resourceName == ("rc:"+CONTROLLER) ) {
//        println "ACCEPTED '" + resourceName + "'"
        return true;
    }
}

// println "Resource '" + resourceName + "' is not suitable for this job"
return false; // Tested resource is not appropriate for this build

The numerous corresponding lockable resource definitions in Manage Jenkins define a name (like rc:controller1) and labels (like rc-validation-farm rc-model-3), and sometimes comments about nuances of the controller model.

The job build parameter CONTROLLER is a predefined Global Choice Parameter, which includes the names like controller1 that our devs can pick, and a default value of LOCK_LABEL:rc-validation-farm.

The script above was made generic enough to ease copy-pasting, so it reacts to the build arguments starting with LOCK_LABEL: and picks the rest of the string as the label to look for in resources; otherwise it would look for the requested resource name (with the prefix rc:).

As you could guess, there is a second side to the medal: the default CONTROLLER build argument is not usable as a definitive host name, so that build argument has to be replaced with the actually picked value :)

For this we use the EnvInject plugin, so in the job I further selected "Prepare an environment for the run" and entered an "Evaluated Groovy script" with:

def map = [:]

/*
println "Requested build arg CONTROLLER=='" + CONTROLLER +
    "'; the locked resource == '" + LOCKED_CONTROLLER + "'";
*/

if ("LOCK_LABEL:rc-validation-farm".equals(CONTROLLER)) {
    def CTLNAME = ( (LOCKED_CONTROLLER =~ /^rc:(.*?)$/)[0][1] )
    println "Extracted CONTROLLER:='" + CTLNAME + "'"
    map << [ CONTROLLER : CTLNAME ]
}

return map

(the script actually evaluates optional overrides for a few other variables as well, hence this structure and not a simpler one). Note that println here ends up in the build job's log.

Finally note, that for sub-jobs called from this one, you should also specify "Predefined parameters" including CONTROLLER=${CONTROLLER} so the one mapped above is in place for those jobs that would access it, and not the default token. The "Build on same node" and "Current build parameters" are useful for such cases as well :)

Node dependend resources

I need to create resources that are node dependend. That means a job must pick a target node where the requested resource is available. Does anybody else has similar needs?
I consider extending this plugin but I worry that the effort would possibly exceed my coding skills.

Pipeline stage not shown in queue while waiting for resource

I recently switched from

lock('LOCK') {
    stage('Integration test') {
        # Doing stuff
    }
}

to

stage('Integration test') {
    options {
        lock('LOCK')
    }
    # Doing stuff
}

The reason was to avoid having an executor slot occupied while waiting for the resource.
I expected the build to instead be shown in the build queue as non Pipeline builds are while waiting for a resource. But this is not the case, instead the build is invisible until the resource is acquired.

Job DSL plugin issue

When using the lockable-resources plugin with the Job DSL plugin, the generated snippet within config.xml appears to be incorrect:

        <org.jenkins.plugins.lockableresources.RequiredResourcesProperty>
            <resourceNames>lock-resource</resourceNames>
        </org.jenkins.plugins.lockableresources.RequiredResourcesProperty>

As expected, running the job(s) fails to lock the resource.

If I edit the generated job manually and configure the lockable resource, its replaced with this:

      <org.jenkins.plugins.lockableresources.RequiredResourcesProperty plugin="[email protected]">
        <resourceNames>lock-resource</resourceNames>
      </org.jenkins.plugins.lockableresources.RequiredResourcesProperty>

This works as expected, so I'm not sure if the issue lies with JobDSL or lockable-resources plugin.

Jenkins: 1.638
Java: 1.8
Lockable Resources: 1.7
Job DSL: 1.40

Nodes are shown as locked although they are not

Jenkins nodes are shown as locked although they are not currently used. Is there a way to unlock them and could be possible to check if there are really some jobs currently running on those node before show them as locked?

09:54:38 Trying to acquire lock on [deploy-node-01]
09:54:38 Found 0 available resource(s). Waiting for correct amount: 1.
09:54:38 [deploy-node-01] is locked, waiting...

Update
Ok, I found a way to unlock it but still, there should be a check to ensure if a resource is really locked or not.

org.jenkins.plugins.lockableresources.LockableResourcesManager.get().unlockNames(['deploy-node-01'], null, true);

Pipeline: Resource label as a variable

I want to pass the label of the locked resource as a variable to the scripts executed inside the step, but I couldn't find a clean explanation about.

Is this even possible?

stage("Test") {
  steps {
    lock(resource: 'test_hosts', quantity: 1) {
      sh 'ping <resource label>'
    }
  }
}

If not possible it would be a great feature.

Feature Request: Create and remove locks within a jenkinsfile

It would be ideal to create an arbitrary lock and use it for a period of time, and optionally delete it. Having a self-creating and self-deleting lock would give the ability to lock resources which definitions change frequently but is already tracked in code.

Example usage could look like:

def app_name = 'my-awesome-app'
// ...snip...
lock(resource: "${app_name}:staging", ephemeral: true) {
   // ... lock created and acquired... deploy this app to the staging environment
}
// by this point the lock is released and destroyed

where the lock name my-awesome-app:staging would be created and reserved by the pipeline, and once the lock is out of scope it is deleted. Maybe even allow someone the ability to control create and delete actions: lock(resource: ..., create: true, delete: false) { ... }

I realize this is very similar to #103 but didn't want to hijack that conversation for something that has clearly different features (but also a lot of overlap).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.