Comments (10)
Hi @Gayanvir ,
Yeah, I think it's a major issue. Need to check how to fix it for the parallel execution. Will keep you updated.
Thank you
from mpl.
Hi @Gayanvir ,
I just prepared the fix - it still need to be polished, but looks like working well. The one thing I need to check - is the complicated tests execution.
Could you please check the fix with your pipeline logic?
Thank you
from mpl.
@Gayanvir thank you for testing - I will test it little bit more and merge with master asap.
from mpl.
Sooo, unfortunately fix will not be simple - MPL need to split the execution context... With configs I think it's simple - CFG will be a full copy of the parent config - here is no issue. But the shared context like activeModules, postStepsErrors, modulePostSteps - will be affected.
I think we can use step id from the pipeline to navigate the right branch. It will help with choosing the right context and will allow to execute any number of parallel threads per pipeline.
from mpl.
I am new to MPL and Jenkins. please correct me if I am wrong. What I could notice is activeModules cannot have duplicate modules. Here https://github.com/griddynamics/mpl/blob/master/vars/MPLModule.groovy#L36 it says "Loop protection: There is no way to run currently active module again". So I guess activeModules need to have unique values. This might not well suited for parallel stages. But if we have this restriction, I think we can simply remove a module from activeModule by just removing by its path. Just like
activeModules -= path
instead of
activeModules.pop()
from mpl.
No, this activeModules also a part of different mechanisms inside MPL to determine the poststeps and so on. I was able to run multiple Test modules from the parallel stages, but I think it's just a luck - and for sure if we will go forth with the complex parallel builds - it will cause a multiple issues.
Infinity loop protection - is here to make sure the module will not call itself again (but it can call the parent one).
from mpl.
Checking the ways to solve the issue - it seems that the parallel builds in pipeline are still not good for daily use, so be careful: env is shared in the pipeline.
Anyway I think there is another way to pass the build identifier, hopefully I will find it soon.
from mpl.
Seems I'm moving forward with the next solution:
- Adding StepStartNode / StepEndNode guards to the running of the module (additionally I need to identify them somehow)
- Using
CpsThread.current().head.get().getEnclosingBlocks()
determining the current branch of the modules - Following the branch to make sure we using the right MPLManager context.
First step is partially done, now the Pipeline Steps
of the build looks much better - now each module in the separated block:
from mpl.
Just improved view of the Pipeline Steps
:
- Hid the rows with checking existence of
.jenkins
project modules - usedHelper.pathExists()
instead of the stepfileExists()
. - Added
LabelAction
andArgumentsAction
to store the MPL data in the blocks.
from mpl.
Hi @sparshev ,
Sorry for the late reply and I highly appreciate your efforts on this. Our pipeline is not much complex but has parallel builds. I tried your fixes and it is working well. Good job !!!
Thanks
from mpl.
Related Issues (20)
- Overriding Modules in MPL HOT 4
- [Q] Passing variables between stages HOT 5
- Passing pipeline CFG as env variable in build environment HOT 3
- [Question] How to pass my own configs to pipeline HOT 2
- Real examples of file names for modules HOT 3
- Unable to properly execute jenkins test on PR from forked repo HOT 1
- Change the circleci links
- Jenkins is going to remove built-in junit HOT 1
- Demo video links in wiki broken? HOT 7
- Problems when trying out MPL HOT 9
- Can we have a top-level getter for cron similar to agent_label? HOT 5
- mpl-nested example tests depend on mpl 1.0.0-SNAPSHOT HOT 4
- Allow access to the pipeline configs from MPL instance
- Generate reference documentation for modules and steps HOT 2
- Defaults are lost in case you use them with closure. HOT 4
- Any way to introduce / reorder stages? HOT 2
- CircleCI build failed due to jenkins repo moved to https
- The pipeline can't find the embedded modules anymore
- java.lang.ArrayIndexOutOfBoundsException HOT 4
- After Upgrade: Unable to find not active module to execute: -X> modules/Build/Build.groovy HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mpl.