adoptium / tkg Goto Github PK

TestKitGen (TKG)

License: Apache License 2.0

Makefile 14.53% Perl 13.55% Shell 2.66% Java 50.15% Python 19.11%

tkg's Introduction

TKG

TestKitGen (TKG) is a lightweight test harness for bringing together a diverse set of tests or commands under some common behaviour. All testing that runs at the AdoptOpenJDK project is run using TKG to yoke together many types of testing (that run on a variety of test frameworks).

TKG standardizes:

test target generation (generates make targets based on contents of playlist.xml files)
test output, defaulting to Test Anything Protocol (TAP) as the simplest, lowest common denominator
the way tests are tagged, grouped, included and excluded (categorizing by elements in the playlist file, by group, impl, version, etc)

tkg's People

Contributors

Stargazers

Watchers

Forkers

renfeiw jennychenyj bitpang98 sophia-guo llxia keithc-ca theresa-m longyuzhang andrew-m-leonard piyush286 zheddie tajila psoujany knn-k smlambert ymanton willsparker pshipton vermash vsebe chengjin01 oscarqq nikolamilijevic1 sxa jennli joeyleeeeeee97 liqunl adamfarley icohedron mesbah-alam brngylni singh264 say-droid427 2001asjad vanichitkara eapenzhan babsingh hangshao0 an3sha hzongaro jasonfengj9 amandanguyeen xius666 harshikaninawe psaunderualberta nadeensami jeromeju dsouzai cjjdespres yeeskywalker jdekonin zdtsw gdams isabella232 bhavanisn j-braley momanmalik alexrogalskiy adem1414 ultramancoder panxuefeng-loongson tellison luhenry arkaprabhachakraborty vedant-z flow2ml chandrashritii joe-braley johnoliver cas-atlantic jasonkatonica karianna edith-mercy sameera-simha adirala-krishna surekha-nalam sobirada varshm16 prodroy89 adambrousseau zzambers annaibm ashithavijay radekcap pushkarnk sophiaxu0424

tkg's Issues

Verify Rerun links produced by TKG contain correct information

We do not put all parameters in the Rerun link that TKG produces (by design). For the parameters we do add, we should verify that they are correct.

Recently grabbed TKG output from a Corretto test job (https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk11_corretto_sanity.openjdk_x86-64_linux/321/console) that has many test targets failing.

14:52:35  To rebuild the failed test in a jenkins job, copy the following link and fill out the <Jenkins URL> and <FAILED test target>:
14:52:35  <Jenkins URL>/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=<FAILED test target>
14:52:35  
14:52:35  For example, to rebuild the failed tests in <Jenkins URL>=https://ci.adoptopenjdk.net/job/Grinder, use the following links:
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk_lang_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk_math_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk_util_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk_svc_sanity_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk11_tier1_cipher_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk11_tier1_buffer_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk11_tier1_iso8859_1
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=jdk11_tier1_pack200_1
14:52:35  rebuild the failed tests in one link:
14:52:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=hotspot&JDK_VENDOR=adoptopenjdk&BUILD_LIST=openjdk&PLATFORM=x86-64_linux_xl&TARGET=testList%20TESTLIST=jdk_lang_1,jdk_math_1,jdk_util_1,jdk_svc_sanity_1,jdk11_tier1_cipher_1,jdk11_tier1_buffer_1,jdk11_tier1_iso8859_1,jdk11_tier1_pack200_1

In this case, we should check that JDK_VENDOR is correct and we may also wish to explicitly include a few other variables (even though they have default values when not set) in order to be more transparent on the starting values of the Grinder job that gets set up.

Create a generic name for asm jar

We use asm.jar name contains the specific version.
https://github.com/AdoptOpenJDK/TKG/blob/70d471246847943ac59a8840998c6e625f181056/scripts/getDependencies.pl#L66

The problem is that every time we upgrade the jar, we need to update all the files that use it. (See https://github.com/eclipse/openj9/pull/10057/files)
To avoid this, we should have a generic name (similar to testing, commander, etc). We can just call it asm.jar

Support specifying minor version in playlist

The <version> tag is currently only at the granularity of major version (and can support special branches like Valhalla/Panama). We do not currently have a mechanism to say 'only 11 and not less than 11.0.9'. We need to investigate on how to support minor version properly in TKG.

Related issue: adoptium/aqa-tests#1963 (comment)

core files for failures on z/OS

We did some work to collect core files on z/OS. I'm thinking it only works for one core file.

Looking at the artifact from the last zos nightly build, it only contains one core for the cmdLineTester_SCCommandLineOptionTests_1 test, but should contain 7 cores.
job/Test_openjdk11_j9_sanity.functional_s390x_zos_Nightly/234/

@renfeiw said:
Here is the pr to capture core from test: https://github.com/AdoptOpenJDK/TKG/pull/6 Maybe the script moveDmp.pl doesn't capture all cores, fyi @Lan-Xia.

Not able to set `--iterations` when running the tests locally

TestKitGen has a iterations argument --iterations on line 39 in makeGen.mk[1] Is there a way to set that for local tests?
I have tried exporting TEST_ITERATIONS before make compile but didn’t have any luck. From the looks it the variable being substituted on line 39 in makeGen.mk[1] --iterations=$(TEST_ITERATIONS) doesn’t look like an environment variable substitution. I am also not too familiar with make files so I could be wrong

If it is not supported I could try adding it as being able to run a grinder locally would be handy for intermittent bugs

[1] https://github.com/AdoptOpenJDK/TKG/blob/master/makeGen.mk#L39

test.getDependency does not include jtreg jar

In #189, we added jtreg jar in test.getDependency Jenkins job. However, test.getDependency only archives jtreg-4.2.0-tip.tar.gz.sha256sum.txt, but not jtreg_4_2_0_tip.tar.gz

https://ci.eclipse.org/openj9/job/test.getDependency/12758/
https://ci.adoptopenjdk.net/job/test.getDependency/1063/

TKG: Unable to detect zLinux31 architecture.

While executing SVT tests on zLinux31, TKG failed to detect system.os.arch as s390.

Error:
System.getProperty('os.arch')=s390
17:40:14
17:40:14 System.getProperty('java.fullversion')=8.0.7.0 - pxz3180sr7-20210609_01(SR7)
17:40:14 JRE 1.8.0 Linux s390-31-Bit 20210608_6725 (JIT enabled, AOT enabled)
17:40:14 OpenJ9 - 46b10e7
17:40:14 OMR - e4c94e9
17:40:14 IBM - c24a144
17:40:14
17:40:14 Cannot determine System.getProperty('os.arch')=s390
17:40:14
17:40:14 makefile:66: recipe for target 'envDetect' failed

Link:- hyc-runtimes-jenkins.swg-devops.com/job/Grinder/16159

Validate playlist against XML Schema

We would like to add a validation step to check for any syntax issue in playlist.xml

Test material compile fails but running tests is attempted

I noticed that even when the test material fails to compile and this portion of the build fails, running the tests are still attempted.

https://ci.eclipse.org/openj9/job/Grinder/1256/

13:03:32      [javac]   symbol:   variable MAC
13:03:32      [javac]   location: class XlpOptionsTestRunner
13:03:32      [javac] /Users/****/workspace/Grinder/openjdk-tests/functional/VM_Test/src/j9vm/test/xlp/XlpOptionsTestRunner.java:271: error: cannot find symbol
13:03:32      [javac] 			xlpOptionsList.add(new XlpOption("-Xlp:objectheap:pagesize=4M,pagesize=2M", 2 * ONE_MB, XlpUtil.XLP_PAGE_TYPE_NOT_USED, osName == MAC));
13:03:32      [javac] 			                                                                                                                                  ^
13:03:32      [javac]   symbol:   variable MAC
13:03:32      [javac]   location: class XlpOptionsTestRunner
13:03:32      [javac] Note: Some input files use or override a deprecated API.
13:03:32      [javac] Note: Recompile with -Xlint:deprecation for details.
13:03:32      [javac] Note: Some input files use unchecked or unsafe operations.
13:03:32      [javac] Note: Recompile with -Xlint:unchecked for details.
13:03:32      [javac] 5 errors
13:03:32      [javac] 41 warnings
13:03:32  
13:03:32  BUILD FAILED
13:03:32  /Users/****/workspace/Grinder/openjdk-tests/TKG/scripts/build_test.xml:60: The following error occurred while executing this line:
13:03:32  /Users/****/workspace/Grinder/openjdk-tests/functional/VM_Test/build.xml:214: The following error occurred while executing this line:
13:03:32  /Users/****/workspace/Grinder/openjdk-tests/functional/VM_Test/build.xml:110: Compile failed; see the compiler error output for details.
13:03:32  
13:03:32  Total time: 12 seconds
13:03:32  1
13:03:32  make[1]: *** [compile] Error 1
13:03:32  make: *** [compile] Error 2
13:03:32  + '[' '!' -z 0 ']'
13:03:32  + '[' darwin15.6.0 = cygwin ']'
13:03:32  + echo PROCESSCATCH: Checking for any leftover java processes on the machine
13:03:32  PROCESSCATCH: Checking for any leftover java processes on the machine
13:03:32  + ps -fu ****
13:03:32  + grep java
13:03:32  + egrep -v 'remoting.jar|agent.jar|grep'
13:03:32    502 47421 47420   0  9:10PM ??         3:52.80 java -jar slave.jar
13:03:32  + echo PROCESSCATCH: There are rogue processes still on the machine as per the list above
13:03:32  PROCESSCATCH: There are rogue processes still on the machine as per the list above
[Pipeline] }
13:03:34  $ ssh-agent -k
13:03:34  unset SSH_AUTH_SOCK;
13:03:34  unset SSH_AGENT_PID;
13:03:34  echo Agent pid 85305 killed;
13:03:38  [ssh-agent] Stopped.
[Pipeline] // sshagent
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] timestamps
[Pipeline] {
[Pipeline] echo
13:03:42  Running tests...
[Pipeline] echo
13:03:42  ITERATION: 1/1
[Pipeline] sh
13:03:46  ++ uname
13:03:46  + '[' Darwin = AIX ']'
13:03:46  ++ uname
13:03:46  + '[' Darwin = SunOS ']'
13:03:46  + MAKE=make
13:03:46  + make -f ./openjdk-tests/TKG/testEnv.mk testEnvTeardown
13:03:46  make: Nothing to be done for `testEnvTeardown'.
[Pipeline] sh
13:03:54  + ./openjdk-tests/maketest.sh ./openjdk-tests _testList TESTLIST=xlpTests_1,xlpCodeCacheTests_1
13:03:54  JAVA_HOME is set to /Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image
13:03:54  /Users/jenkins/Xcode7/Xcode.app/Contents/Developer/usr/bin/make -f makeGen.mk AUTO_DETECT=true MODE=tests TESTTARGET=testList TESTLIST=xlpTests_1,xlpCodeCacheTests_1
13:03:54  perl scripts/configure.pl
13:03:54  /Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java -cp "./bin/TestKitGen.jar:./lib/json-simple.jar" org.testKitGen.MainRunner --mode=tests --spec=osx_x86-64_cmprssptrs --jdkVersion=8 --impl=openj9 --buildList=functional/VM_Test --iterations=1 --testFlag= --testTarget=testList --testList=xlpTests_1,xlpCodeCacheTests_1 --numOfMachines=1 --testTime= --TRSSURL= 
13:03:54  Modes data parsed from resources/modes.xml and resources/ottawa.csv.
13:03:54  
13:03:54  Starting to generate test make files.
13:03:54  
13:03:54  Parsing /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../functional/VM_Test/playlist.xml
13:03:54  
13:03:54  Generated /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../functional/VM_Test/autoGen.mk
13:03:54  
13:03:54  Generated /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../functional/autoGen.mk
13:03:54  
13:03:54  Generated /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../autoGen.mk
13:03:54  
13:03:54  Generated /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/utils.mk
13:03:54  
13:03:54  Make files are generated successfully.
13:03:54  
13:03:54  /Users/jenkins/Xcode7/Xcode.app/Contents/Developer/usr/bin/make -f runtest.mk _testList
13:03:54  
13:03:54  Running make 3.81
13:03:54  set TEST_ROOT to /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/..
13:03:54  set JDK_VERSION to 8
13:03:54  set JDK_IMPL to openj9
13:03:54  set JVM_VERSION to openjdk8-openj9
13:03:54  set JCL_VERSION to latest
13:03:54  set JAVA_HOME to /Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image
13:03:54  set SPEC to osx_x86-64_cmprssptrs
13:03:54  Running testList ...
13:03:54  There are 2 test targets in testList
13:03:54  "/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -version
13:03:54  openjdk version "1.8.0_282-internal"
13:03:54  OpenJDK Runtime Environment (build 1.8.0_282-internal-jenkins_2020_12_03_22_39-b00)
13:03:54  Eclipse OpenJ9 VM (build master-8401f565b48, JRE 1.8.0 Mac OS X amd64-64-Bit Compressed References 20201203_580 (JIT enabled, AOT enabled)
13:03:54  OpenJ9   - 8401f565b48
13:03:54  OMR      - f7a2761067b
13:03:54  JCL      - 0655969557f based on jdk8u282-b04)
13:03:54  
13:03:54  ===============================================
13:03:54  Running test xlpTests_1 ...
13:03:54  ===============================================
13:03:54  xlpTests_1 Start Time: Fri Dec  4 13:03:53 2020 Epoch Time (ms): 1607105033205
13:03:54  "/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
13:03:54  
13:03:54  Attempting to destroy all caches in cacheDir /Users/jenkins/javasharedresources/
13:03:54  
13:03:54  JVMSHRC806I Compressed references persistent shared cache "sharedcc_jenkins" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
13:03:54  JVMSHRC005I No shared class caches available
13:03:54  cache cleanup done
13:03:54  variation: Mode610
13:03:54  JVM_OPTIONS:  -Xcompressedrefs -Xjit -Xgcpolicy:gencon 
13:03:54  { itercnt=1; \
13:03:54  	mkdir -p "/Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16071050334664/xlpTests_1"; \
13:03:54  	cd "/Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16071050334664/xlpTests_1"; \
13:03:54  	"/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java" -Dplatform=osx_x86-64_cr -cp "/Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar" \
13:03:54  	j9vm.runner.Menu -test="j9vm.test.xlp" -exe=""/Users/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java"  -Xcompressedrefs -Xjit -Xgcpolicy:gencon  -Xdump" \
13:03:54  	-jar="/Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../../jvmtest/functional/VM_Test/VM_Test.jar" \
13:03:54  	-xids=all,osx_x86-64_cr,; \
13:03:54  	if [ $? -eq 0 ] ; then echo ""; echo "xlpTests_1""_PASSED"; echo ""; cd /Users/jenkins/workspace/Grinder/openjdk-tests/TKG/..;  else echo ""; echo "xlpTests_1""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/Users/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16071050334664/TestTargetResult";
13:03:54  Error: Could not find or load main class j9vm.runner.Menu
13:03:55  
13:03:55  xlpTests_1_FAILED

SHA mismatch for asm-8.0.1.jar

Cloned TKG and ran a make compile, getting the following error.

--------------------------------------------
Starting download third party dependent jars
--------------------------------------------
downloading dependent third party jars to /home/peter/test/openjdk-tests/TKG/../TKG/lib
downloading https://repo1.maven.org/maven2/org/ow2/asm/asm-all/6.0_BETA/asm-all-6.0_BETA.jar
--> file downloaded to /home/peter/test/openjdk-tests/TKG/../TKG/lib/asm-all.jar
downloading https://repository.ow2.org/nexus/content/repositories/releases/org/ow2/asm/asm/8.0.1/asm-8.0.1.jar
--> file downloaded to /home/peter/test/openjdk-tests/TKG/../TKG/lib/asm-8.0.1.jar
Expected sha is: 3f5199523fb95304b44563f5d56d9f5a07270669,
Actual sha is  : a236c777042003fc6fe13c53825e06494d7185fb.
ERROR: sha checksum error.
Please delete /home/peter/test/openjdk-tests/TKG/../TKG/lib/asm-8.0.1.jar and rerun the program!makefile:40: recipe for target 'getdependency' failed
make: *** [getdependency] Error 255

auto test exclusion enhancements

Thanks to @renfeiw , we enabled auto test exclusion in openj9 repo (PR eclipse-openj9/openj9#11785).

A couple of enhancements that we would like to have:

update year in the copyright
add test excluded label in the issue upon merging the PR in openj9/adopt repo

KEEP_REPORTDIR does not work as expected

When KEEP_REPORTDIR is set to false, we should not capture any passed test artifacts. However, this does not work as expected. Please see the build link below:
https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.functional_s390x_linux_aot_Personal_testList_0/4/

It looks like all test artifacts are captured.

I think this is due to test pipeline uses boolean https://github.com/AdoptOpenJDK/openjdk-tests/blob/e55242930ee3bd56ceb2cbdcb898df130fbfc9c9/buildenv/jenkins/JenkinsfileBase#L78
https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/buildenv/jenkins/JenkinsfileBase#L264

and TKG uses string https://github.com/AdoptOpenJDK/TKG/blob/b7d19e42ab2c131d71ba037304b6efd8e07d5ad5/settings.mk#L195

Move functional tests related 3rd party libs into functional level

getDependencies.pl contains functional tests related 3rd party libs. They should not be the pre-requisite for all tests. They need to be moved into the functional level.

EPIC: Testing environment checks

TKG should do general testing env checks before and/or after running the test. Below are something we could take a look to see if we should implement:

diskspace (fail if we do not have enough diskspace)
memory (fail if we do not have enough memory)
core (fail if there is core generated after a test run regardless of the test result)
list existing running processes before and after the test (fail if there is any dangling process?), this includes docker images leftover from aborted runs, those are not expected to be present and should get pruned
a flag to enable/disable this test environment health checks

Remove TEST_RESULTSTORE and related logic

👾 Description of the issue
We are using artifactory to store test outputs. In the past we used a different artifact storage system which we can now remove.
The logic for TEST_RESULTSTORE can be removed, L135-L151 of the openj9Settings.mk file can be removed.

📋 Step by Step

To solve this issue and contribute a fix you should check the following step-by-step list. A more detailed documentation of the workflow can be found here. Note: that you do not need to add yourself to a Contributors.md file as outlined in the detailed instructions.

🤔❓ Questions

If you have any questions just ask us directly in this issue by adding a comment. You can join our community chat at Slack. Next to this you can find a general manual about open source contributions here.

Remove extra step to generate jvmtest folder

We are generating jvmtest folder and copy all the compiled material to the folder duration compilation. It was used for internal builds and is not necessary now, so we can remove this extra step.

z/OS OpenJ9 ADD_JVM_LIB_DIR_TO_LIBPATH needs jdk/lib on LIBPATH

job/Test_openjdk11_j9_sanity.functional_s390x_zos_Nightly
jsigjnitest_0
shrtest_zos_0
thrstatetest_0
Fatal error: JVM interface version mismatch: expecting 4, got 6.

The problem is the jdk/lib directory isn't on the LIBPATH. z/OS uses an additional library libwrappers.so. When jdk/lib isn't on the LIBPATH the tests would fail like

EDC5205S DLL module not found. (errno2=0xC40B0025)
iconv_init: libwrappers.so not found

however there are JDK8 directories on the pre-existing LIBPATH, such as /home/jenkins/openj9_resources/J8.0_64/lib/s390x. The test finds the JDK8 library instead of JDK11 (due to the missing lib dir) and this results in the version mismatch.

Update rerun link based on Jenkins server

At the end of the run, TKG provides rerun link (below).

16:40:35  To rebuild the failed test in a jenkins job, copy the following link and fill out the <Jenkins URL> and <FAILED test target>:
16:40:35  <Jenkins URL>/parambuild/?JDK_VERSION=11&JDK_IMPL=openj9&BUILD_LIST=openjdk&PLATFORM=ppc64_aix&TARGET=<FAILED test target>
16:40:35  
16:40:35  For example, to rebuild the failed tests in <Jenkins URL>=https://ci.adoptopenjdk.net/job/Grinder, use the following links:
16:40:35  https://ci.adoptopenjdk.net/job/Grinder/parambuild/?JDK_VERSION=11&JDK_IMPL=openj9&BUILD_LIST=openjdk&PLATFORM=ppc64_aix&TARGET=jdk_io_0

We need to update rerun link based on Jenkins server. We can use HUDSON_URL for Jenkins server. The default value should still be https://ci.adoptopenjdk.net/.

Add support for JDK_VENDOR in playlists

There may be several use cases where we want to differentiate by JDK_VENDOR (which is over and beyond differentiating by JDK_IMPL).

We have currently blended the meaning of JDK_IMPL with vendor (hotspot, sap, corretto, dragonwell are all hotspot VM impl based, but rightly we should categorize them as JDK_VENDOR (adopt, sap, amazon, alibaba) or whatever value is returned by a query of java.vendor. (related: adoptium/temurin-build#465)

As there are tests that may be vendor specific (many build/packaging tests for example).

Enable mixed for non-openj9 JVM

Currently, we only check if a SDK is mixed or not for openj9/ibm.
https://github.com/adoptium/TKG/blob/master/src/org/openj9/envInfo/JavaInfo.java#L72
https://github.com/adoptium/TKG/blob/master/src/org/openj9/envInfo/JavaInfo.java#L92

As a result, non-openj9 JVM only runs tests that match with PLATFORM (i.e., nooptions and nocompressedrefs mode tests). We should enable mixed for non-openj9 JVM. Once this is enabled, non-openj9 SDK will run both compressedrefs and nocompressedrefs mode tests in one test build.

Ideally, we should remove the mixed check in TKG, but not all platforms are switched to mixed at the moment.

FYI @smlambert @renfeiw

Enhance platform requirements mechanism for TKG

We are using <platformRequirements> to specify the spec to test against. It provides some degree of freedom to define os, arch, bit information, but it's limited (only support and relationship) and sometimes difficult to read and write.

A better way is to utilize regular expression to define the platform. Another step to improve the mechanism to use platform instead of spec to define the information. Since they are more clear and intuitive because it matches the build names.

Update runAqaArgParse.py to use new aqa-tests url

Currently, run aqa action failed due to set openjdk_testrepo to the old AdoptOpenJDK/openjdk-tests:master

#195 (comment)
https://github.com/adoptium/TKG/runs/2841898695?check_suite_focus=true

/usr/bin/git clone --depth 1 -b master https://github.com/AdoptOpenJDK/openjdk-tests.git
Cloning into 'openjdk-tests'...
Error: ENOENT: no such file or directory, chdir '/home/runner/work/TKG/TKG' -> 'aqa-tests'

We should update runAqaArgParse.py to use new aqa-tests URL:

TKG/scripts/testBot/runAqaArgParse.py

Line 63 in 1476a6f

    
           parser.add_argument('--openjdk_testrepo', default=['AdoptOpenJDK/openjdk-tests:master'], nargs='+')

Set defined test modes for Solaris

At the moment Solaris is running three different modes when running tests e.g.:

01:30:59  PASSED test targets:
01:30:59  	jdk_lang_0
01:30:59  	jdk_lang_1
01:30:59  	jdk_lang_2
01:30:59  	jdk_jdi_0
01:30:59  	jdk_jdi_1
01:30:59  	jdk_jdi_2

We should not be running all three modes.
Related to the issue regarding getting Solaris working: adoptium/aqa-tests#799 (comment)

Add support in TKG to specify a path for parallelList.mk

We need to implement a mechanism in TKG to be able to specify a path at which to generate parallelList.mk. This is needed by jck_target_folder_check script, as explained by Lan here: adoptium/aqa-tests#2064 (comment)

FYI @renfeiw @llxia

extended.perf Tests Executed in Incorrect Order

Summary

In one extended.perf run, each failed test target would be followed by the re-execution of earlier test targets.

Note that while stack overflows have been seen elsewhere, I've only seen this problem once.

Details

In a full extended.perf run, I expect each test target to be executed once apiece, in order. Here, though, we'd execute a test target, sometimes it would fail, and failed targets would be declared to be a different (earlier) test target, followed by the targets that were meant to be run after the different (incorrect) test target.

It's as if our position in the list of test targets is being repeatedly reset to the position of the earlier test target.

See this issue for the full story.

This issue is raised specifically to cover the problem where the ordering of the test targets could be changed during execution. Raising this as a TKG issue as that's where the playlist file is passed in, so that's where I assume the list (and ordering) of test targets is meant to be stored.

Link
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.perf_ppc64_aix/15/

Grinder Link
Attempting to reproduce. Link.
Result: Saw one StackOverflow, but we need more than one to reproduce this.

Bigger repro: https://ci.adoptopenjdk.net/job/Grinder/114/

Repro Link
https://ci.adoptopenjdk.net/job/Grinder/parambuild/?SDK_RESOURCE=upstream&TARGET=extended.perf&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=0&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&TEST_PARALLELIZATION_PARAMS=&PLATFORM=ppc64_aix&KEEP_REPORTDIR=false&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&TEST_OPTIONS_PARAMS=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&NON_AQA_TEST_REPOS_HELP_TEXT=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&TEST_REPO_PARAMS=&TEST_SELECTION_PARAMS=&TEST_PARALLELIZATION_PARAMS_HELP_TEXT=&JDK_SELECTION_PARAMS=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=11&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fibmruntimes%2Fopenj9-openjdk-jdk11&PLATFORM_AND_MACHINE_HELP_TEXT=&RELEASE_TAG=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&UPSTREAM_JOB_NAME=build-scripts%2Fjobs%2Fjdk11u%2Fjdk11u-aix-ppc64-openj9&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=openj9&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&POST_RUN_PARAMS_HELP_TEXT=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_SELECTION_PARAMS_HELP_TEXT=&JDK_IMPL=openj9&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&TEST_SELECTION_PARAMS_HELP_TEXT=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=perf&UPSTREAM_JOB_NUMBER=964&TEST_REPO_PARAMS_HELP_TEXT=&POST_RUN_PARAMS=&TIME_LIMIT=10&TEST_OPTIONS_PARAMS_HELP_TEXT=&JVM_OPTIONS=&PARALLEL=None

Consider moving env vars for zos compiler to settings.mk

Opening this issue to discuss whether we move the env vars out of Jenkins files into TKG, to support command line running of zOS native testing

related: adoptium/aqa-tests#1478

Add support for excluding tests based on JDK_OPTIONS

Some tests are not compatible to be run with particular JDK options. For example, a test that uses -XX:+ClassRelationshipVerifier is not compatible with options -Xfuture or -Xverify:all and will result in an JVM error if either of these options are used. Instead, it would be ideal for the test to be excluded if incompatible options are specified.

Not all artifacts saved during javac crash

For the following particular job:
https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108/console

With the following "Rerun in Grinder" link:
https://ci.eclipse.org/openj9/job/Grinder/parambuild/?TARGET=-f+parallelList.mk+testList_0&SDK_RESOURCE=upstream&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&RUNTIME_NAME=testList_0&PLATFORM=x86-64_linux&KEEP_REPORTDIR=false&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&EXTRA_OPTIONS=&BUILD_IDENTIFIER=eclipse%2Fopenj9%2311393&CUSTOMIZED_SDK_URL=https%3A%2F%2F140-211-168-230-openstack.osuosl.org%2Fartifactory%2Fci-eclipse-openj9%2FBuild_JDK8_x86-64_linux_Personal%2F982%2FOpenJ9-JDK8-x86-64_linux-20210104-160146.tar.gz+https%3A%2F%2F140-211-168-230-openstack.osuosl.org%2Fartifactory%2Fci-eclipse-openj9%2FBuild_JDK8_x86-64_linux_Personal%2F982%2Ftest-images.tar.gz&ADOPTOPENJDK_BRANCH=master&ARTIFACTORY_SERVER=ci-eclipse-openj9&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=8&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fibmruntimes%2Fopenj9-openjdk-jdk8.git&OPENJ9_BRANCH=master&OPENJ9_SHA=origin%2Fpr%2F11393%2Fmerge&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&UPSTREAM_JOB_NAME=&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=openj9&LABEL_ADDITION=&ARTIFACTORY_REPO=ci-eclipse-openj9&ARTIFACTORY_ROOT_DIR=Test&UPSTREAM_TEST_JOB_NUMBER=728&DOCKERIMAGE_TAG=&JDK_IMPL=openj9&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&TKG_SHA=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=ab89294b-5ba1-48e9-8c85-107daca5a2e9&NUM_MACHINES=&OPENJDK_SHA=&BUILD_LIST=functional&UPSTREAM_JOB_NUMBER=&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None

It appears only the system core dump is saved in the artifacts. However from stdout we can see the javacore, jitdump, and Snap trace are all generated.

18:07:27  compile:
18:07:27       [echo] Ant version is Apache Ant(TM) version 1.10.5 compiled on July 10 2018
18:07:27       [echo] ============COMPILER SETTINGS============
18:07:27       [echo] ===fork:                         yes
18:07:27       [echo] ===executable:                   /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdkbinary/j2sdk-image/bin/javac
18:07:27       [echo] ===debug:                        on
18:07:27       [echo] ===destdir:                      /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/TKG/../../jvmtest/functional/IllegalAccessError_for_protected_method
18:07:27      [javac] Compiling 2 source files to /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/bin
18:07:29      [javac] Unhandled exception
18:07:29      [javac] Type=Segmentation error vmState=0x00000000
18:07:29      [javac] J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
18:07:29      [javac] Handler1=00007F92D5DC1DA0 Handler2=00007F92D56A2020 InaccessibleAddress=00000000FFB80418
18:07:29      [javac] RDI=0000000000048D00 RSI=0000000000048D00 RAX=0000000000048D00 RBX=00000000FFB80400
18:07:29      [javac] RCX=00000000FFB80400 RDX=00000000020E0001 R8=0000000000000000 R9=00000007FFB42360
18:07:29      [javac] R10=0000000000051300 R11=00007F92D73B8150 R12=00000000000E3558 R13=0000000000000001
18:07:29      [javac] R14=00007F927C6680D3 R15=00007F92D6D02910
18:07:29      [javac] RIP=00007F92D5D4D763 GS=0000 FS=0000 RSP=00007F92D6D02590
18:07:29      [javac] EFlags=0000000000010246 CS=0033 RBP=0000000000000000 ERR=0000000000000004
18:07:29      [javac] TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=00000000FFB80418
18:07:29      [javac] xmm0 0000000000000005 (f: 5.000000, d: 2.470328e-323)
18:07:29      [javac] xmm1 cc31990665dbbe9e (f: 1708899968.000000, d: -1.104629e+59)
18:07:29      [javac] xmm2 00007f92d6d02940 (f: 3603966208.000000, d: 6.930192e-310)
18:07:29      [javac] xmm3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
18:07:29      [javac] xmm4 43e0000000000000 (f: 0.000000, d: 9.223372e+18)
18:07:29      [javac] xmm5 000000003d9bcb21 (f: 1033620288.000000, d: 5.106763e-315)
18:07:29      [javac] xmm6 000000004928d030 (f: 1227411456.000000, d: 6.064219e-315)
18:07:29      [javac] xmm7 00000000be871679 (f: 3196524032.000000, d: 1.579293e-314)
18:07:29      [javac] xmm8 410440d286af43d9 (f: 2259633152.000000, d: 1.659143e+05)
18:07:29      [javac] xmm9 43e0000000000000 (f: 0.000000, d: 9.223372e+18)
18:07:29      [javac] xmm10 3f847ae147ae147b (f: 1202590848.000000, d: 1.000000e-02)
18:07:29      [javac] xmm11 3fefae147ae147ae (f: 2061584256.000000, d: 9.900000e-01)
18:07:29      [javac] xmm12 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
18:07:29      [javac] xmm13 4107ffc000000000 (f: 0.000000, d: 1.966000e+05)
18:07:29      [javac] xmm14 0000000048473240 (f: 1212625408.000000, d: 5.991166e-315)
18:07:29      [javac] xmm15 00000000469f5b68 (f: 1184848768.000000, d: 5.853931e-315)
18:07:29      [javac] Module=/home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdkbinary/j2sdk-image/jre/lib/amd64/compressedrefs/libj9vm29.so
18:07:29      [javac] Module_base_address=00007F92D5D28000
18:07:29      [javac] Target=2_90_20210104_982 (Linux 4.4.0-169-generic)
18:07:29      [javac] CPU=amd64 (4 logical CPUs) (0x5e2f07000 RAM)
18:07:29      [javac] ----------- Stack Backtrace -----------
18:07:29      [javac] (0x00007F92D5D4D763 [libj9vm29.so+0x25763])
18:07:29      [javac] (0x00007F92D5D3AA50 [libj9vm29.so+0x12a50])
18:07:29      [javac] (0x00007F92D5DFE7A2 [libj9vm29.so+0xd67a2])
18:07:29      [javac] ---------------------------------------
18:07:29      [javac] JVMDUMP039I Processing dump event "gpf", detail "" at 2021/01/04 19:07:28 - please wait.
18:07:29      [javac] JVMDUMP032I JVM requested System dump using '/home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/core.20210104.190728.11521.0001.dmp' in response to an event
18:07:34      [javac] JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.11546.
18:07:34      [javac] 
18:07:34      [javac] JVMDUMP010I System dump written to /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/core.20210104.190728.11521.0001.dmp
18:07:34      [javac] JVMDUMP032I JVM requested Java dump using '/home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/javacore.20210104.190728.11521.0002.txt' in response to an event
18:07:34      [javac] JVMDUMP010I Java dump written to /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/javacore.20210104.190728.11521.0002.txt
18:07:34      [javac] JVMDUMP032I JVM requested Snap dump using '/home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/Snap.20210104.190728.11521.0003.trc' in response to an event
18:07:34      [javac] JVMDUMP010I Snap dump written to /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/Snap.20210104.190728.11521.0003.trc
18:07:34      [javac] JVMDUMP032I JVM requested JIT dump using '/home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/jitdump.20210104.190728.11521.0004.dmp' in response to an event
18:07:34      [javac] JVMDUMP010I JIT dump written to /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/jitdump.20210104.190728.11521.0004.dmp
18:07:34      [javac] JVMDUMP013I Processed dump event "gpf", detail "".
18:07:34  
18:07:34  BUILD FAILED
18:07:34  /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/TKG/scripts/build_test.xml:60: The following error occurred while executing this line:
18:07:34  /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/build.xml:52: The following error occurred while executing this line:
18:07:34  /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/build.xml:81: The following error occurred while executing this line:
18:07:34  /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/functional/IllegalAccessError_for_protected_method/build.xml:54: Compile failed; see the compiler error output for details.
18:07:34  
18:07:34  Total time: 18 seconds
18:07:34  1
18:07:34  20210104-19:07:33 - Successfully moved core file core.20210104.190728.11521.0001.dmp to /home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/TKG/../TKG/test_output_compilation
18:07:34  compile.mk:32: recipe for target 'compile' failed
18:07:34  make[1]: *** [compile] Error 1
18:07:34  make[1]: Leaving directory '/home/****/workspace/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/openjdk-tests/TKG'
18:07:34  makefile:59: recipe for target 'compile' failed
18:07:34  make: *** [compile] Error 2
[Pipeline] }
18:07:35  $ ssh-agent -k
18:07:35  unset SSH_AUTH_SOCK;
18:07:35  unset SSH_AGENT_PID;
18:07:35  echo Agent pid 11173 killed;
18:07:36  [ssh-agent] Stopped.
[Pipeline] // sshagent
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Post)
[Pipeline] timestamps
[Pipeline] {
[Pipeline] step
18:07:37  TAP Reports Processing: START
18:07:37  Looking for TAP results report in workspace using pattern: **/*.tap
18:07:46  Did not find any matching files. Setting build result to FAILURE.
[Pipeline] archiveArtifacts
18:07:46  Archiving artifacts
[Pipeline] junit
18:07:49  Recording test results
[Pipeline] getArtifactoryServer
[Pipeline] newBuildInfo
[Pipeline] artifactoryUpload
18:09:00  [consumer_0] Deploying artifact: https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108/SHA.txt
[Pipeline] publishBuildInfo
18:09:47  Deploying build info to: https://140-211-168-230-openstack.osuosl.org/artifactory/api/build
18:09:47  Deploying build descriptor to: https://140-211-168-230-openstack.osuosl.org/artifactory/api/build
18:09:48  Build successfully deployed. Browse it in Artifactory under https://140-211-168-230-openstack.osuosl.org/artifactory/webapp/builds/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108
18:09:48  Sending request for build retention, deleting build artifacts, max number of builds to store: 20.
[Pipeline] echo
18:09:48  Test output artifactory URL:'https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108/'
[Pipeline] sh
18:09:51  + tar -cf - ./openjdk-tests/TKG/test_output_compilation
18:09:51  + pigz -9
[Pipeline] getArtifactoryServer
[Pipeline] newBuildInfo
[Pipeline] artifactoryUpload
18:09:59  [consumer_0] Deploying artifact: https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108/functional_test_output.tar.gz
[Pipeline] publishBuildInfo
18:10:54  Deploying build info to: https://140-211-168-230-openstack.osuosl.org/artifactory/api/build
18:10:55  Deploying build descriptor to: https://140-211-168-230-openstack.osuosl.org/artifactory/api/build
18:10:55  Build successfully deployed. Browse it in Artifactory under https://140-211-168-230-openstack.osuosl.org/artifactory/webapp/builds/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108
18:10:55  Sending request for build retention, deleting build artifacts, max number of builds to store: 20.
[Pipeline] echo
18:10:55  Test output artifactory URL:'https://140-211-168-230-openstack.osuosl.org/artifactory/ci-eclipse-openj9/Test/Test_openjdk8_j9_sanity.functional_x86-64_linux_Personal_testList_0/108/'
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // stage
[Pipeline] echo
terminateJavaProcesses iteration: 1
[Pipeline] echo
PROCESSCATCH: Checking for any leftover java processes on the machine
[Pipeline] sh
+ ps -fu jenkins
+ grep java
+ egrep -v remoting.jar|agent.jar|grep
jenkins  10104 10103  0 15:30 ?        00:01:56 java -jar slave.jar
[Pipeline] echo
PROCESSCATCH: There are rogue processes still on the machine as per the list above
[Pipeline] echo
PROCESSCATCH: Removed processes - here is what is remaining:
[Pipeline] sh
+ ps -fu jenkins
+ egrep+  -v remoting.jar|agent.jar|grep
grep java
jenkins  10104 10103  0 15:30 ?        00:01:56 java -jar slave.jar
[Pipeline] cleanWs
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[WS-CLEANUP] done
[Pipeline] sh
+ find /tmp -name *core* -print -exec rm -f {} ;
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

We should fix this so all artifacts are saved.

Update dependency jtreg version

Similar issue to adoptium/aqa-tests#2642

Enhancement to JDK_VERSION support

We currently can tag test material as applicable to a particular JDK_VERSION, meaning major version numbers (8, 11, 11+, etc), but there are use cases where being able to specify update version within a major version (see discussion in adoptium/aqa-tests#1963).

Test of Grinder_Simple does not work

Please see the following results.
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder_Simple/2124/console
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder_Simple/2123/console

I can see the message "autoGen.mk:16: C:/Users/jenkins/workspace/Grinder_Simple/openjdk-tests/\TKG\dependencies.mk: No such file or directory".

Proposal: TKG PR builds

Currently, there are no TKG PR builds. All PRs need to be tested using Grinder by contributors/committers. Had a brief discussion with @renfeiw, we can leverage run-aqa GitHub action and setup workflow for TKG PR builds first. This can be the foundation for the workflow for openj9 PR builds, as TKG PR builds are much simpler (i.e., do not need to build the SDK). The major workflow should be similar to adoptium/aqa-tests#1977, but most values in the workflow need to be parameterized.

Details:

users trigger the PR builds (in Github Actions) with comments (similar to auto PR for test exclusion). Triggering syntax should be similar to openj9 PR builds: action <instruction> <test(s)> <platform(s)> <jdkversion(s)> <impl(s)>.

Our plan is to extend this and use it in openj9 PR builds in the future, so we do not want to create another syntax to complicate the user story. Thanks Shelley for the suggestion.

For example:

action nightly sanity.functional xlinuxlargeheap jdk11
action releases testA,testB xlinux,osx jdk8,jdk11 hotspot,openj9

<instruction> - actions that the users want to do.

compile: compile only // currently available in openj9
test: compile and run test(s) // currently available in openj9
nightly: only run test(s) using SDK from nightly
releases: only run test(s) using SDK from releases
<CUSTOMIZED_SDK_URL>:<token>: only run test(s) using SDK from <CUSTOMIZED_SDK_URL>

For TKG PR builds, we only need nightly/releases (case 3 and case 4). We just listed all the cases here.

<impl(s)> can be omitted and the default value is openj9

script is needed to translate trigger comment to parameters that can be passed used for run-aqa

a map is needed to map the platform to os (i.e., xlinux => ubuntu-16.04)
set version/impl/target/build_list, etc (maybe we should use the same names as the ones exist in TKG?)
default build_list can be functional,system for now.
Possible enhancement: set build_list value according to <test(s)> value. For example, if <test(s)> is sanity.openjdk, then the build_list to openjdk.

run-aqa needs to be able to accept TKG_OWNER_BRANCH for TKG PR builds

Open this issue for discussion. FYI @smlambert @pshipton

SPEC for hotspot IMPL JDK setting by AUTO_DETECT as without _cmprssptrs

Running with github action or running in developer's local environment without export SPEC EnvDetector will generate autoGenEnv.mk to setting DETECTED_SPEC. And DETECTED_SPEC will be set as SPEC as nodefine SPEC.
https://github.com/AdoptOpenJDK/TKG/blob/master/envSettings.mk#L25

For non-openj9 JDK, DETECTED_SPEC will be set as without _cmprssptrs.
https://github.com/AdoptOpenJDK/TKG/blob/master/src/org/openj9/envInfo/JavaInfo.java#L75

This is inconsistent with jenkins story, which set SPEC per platform. All non large heap 64 platforms is setting as _cmprssptrs.
https://github.com/AdoptOpenJDK/openjdk-tests/blob/master/buildenv/jenkins/openjdk_tests

Also this inconsistence is hidden by the check https://github.com/AdoptOpenJDK/TKG/blob/master/envSettings.mk#L33

capture core for TKG compilation

In the current code, we capture core files for test material compilation, but we do not capture core files for TKG compilation. We should add it for TKG as well.

Remove unnecessary macros

Due to historical reasons, we have D, SQ, and Q macros defined in makefile in TKG and used in test repos.

For example:
https://github.com/AdoptOpenJDK/TKG/blob/master/compile.mk#L17
https://github.com/AdoptOpenJDK/TKG/blob/master/makeGen.mk#L26
https://github.com/AdoptOpenJDK/TKG/blob/master/makeGen.mk#L28

It will be more clear and simple if these values can be explicitly written in makefile and test files. For example,

we do not need to modify the value when copying from the playlist for running system_custom
macro is not defined issue will not happen

I think we should verify and remove the macros from makefiles and test repos if they are unnecessary.

Update testKitGen to generate makefiles on the fly

We used to generate all makefiles for all targets before executing a specific make target (e.g., sanity). This approach requires an extra step (i.e., make -f run_configure.mk) before testing. This was designed for some legacy reasons. Since the legacy issues are no longer existing and the generation process only takes seconds, we decide to generate makefiles on the fly when we executing the target. This means we only generate the specific make target that is executing, so no extra step is needed beforehand. This update will make tkg simpler, faster, and take less disk space. This is also an important step to achieve parallel testing in the future.

Test jobs produce FAILURE when no tests are run as Playlist does not contain the given target

Some sanity.external sub test jobs do not contain any targets matching the "external" target, eg:netty and spring:
https://ci.adoptopenjdk.net/view/Failing%20Test%20Jobs/job/Test_openjdk11_hs_sanity.external_x86-64_linux_netty/
https://ci.adoptopenjdk.net/view/Failing%20Test%20Jobs/job/Test_openjdk11_hs_sanity.external_s390x_linux_spring/1/console

22:25:12  TAP Reports Processing: START
22:25:12  Looking for TAP results report in workspace using pattern: **/*.tap
22:25:12  Did not find any matching files. Setting build result to FAILURE.

update openj9jtregtimeouthandler.jar URL to point to new Openj9 Jenkins job

The new Openj9 Jenkins is https://openj9-jenkins.osuosl.org/, so we need to update the links for openj9jtregtimeouthandler.jar

TKG/scripts/getDependencies.pl

Line 121 in 6ddecd4

    
           url => 'https://ci.eclipse.org/openj9/view/Infrastructure/job/Build_JDK_Timeout_Handler/lastSuccessfulBuild/artifact/openj9jtregtimeouthandler.jar',

TKG/scripts/getDependencies.pl

Line 123 in 6ddecd4

    
           shaurl => 'https://ci.eclipse.org/openj9/view/Infrastructure/job/Build_JDK_Timeout_Handler/lastSuccessfulBuild/artifact/openj9jtregtimeouthandler.jar.sha256sum.txt',

getDependentLibs should fail if the download step fails

Currently, the test script proceeds even if the download step fails in getDependentLibs.

     [exec] --------------------------------------------
     [exec] Starting download third party dependent jars
     [exec] --------------------------------------------
     [exec] downloading dependent third party jars to /home/runner/work/aqa-tests/aqa-tests/openjdk-tests/TKG/../TKG/lib
     [exec] downloading https://ci.eclipse.org/openj9/view/Infrastructure/job/Build_JDK_Timeout_Handler/lastSuccessfulBuild/artifact/openj9jtregtimeouthandler.jar.sha256sum.txt
     [exec] --> file downloaded to /home/runner/work/aqa-tests/aqa-tests/openjdk-tests/TKG/../TKG/lib/openj9jtregtimeouthandler.jar.sha256sum.txt
     [exec] downloading https://ci.eclipse.org/openj9/view/Infrastructure/job/Build_JDK_Timeout_Handler/lastSuccessfulBuild/artifact/openj9jtregtimeouthandler.jar
     [exec] --> file downloaded to /home/runner/work/aqa-tests/aqa-tests/openjdk-tests/TKG/../TKG/lib/openj9jtregtimeouthandler.jar
     [exec] downloading https://ci.eclipse.org/openj9/view/Infrastructure/job/Build_JDK_Timeout_Handler/lastSuccessfulBuild/artifact/openj9jtregtimeouthandler.jar.sha256sum.txt
     [exec] --> file downloaded to /home/runner/work/aqa-tests/aqa-tests/openjdk-tests/TKG/../TKG/lib/openj9jtregtimeouthandler.jar.sha256sum.txt
     [exec] Expected sha is: <!doctype,
     [exec] Actual sha is  : d46d58c000b511cd0fde4eb331fb09810511709ab0107974bb6148c583a3c2ff.
     [exec] ERROR: sha checksum error.
     [exec] Please delete /home/runner/work/aqa-tests/aqa-tests/openjdk-tests/TKG/../TKG/lib/openj9jtregtimeouthandler.jar and rerun the program!
     [exec] Result: 255
getJtreg:
    [mkdir] Created dir: /home/runner/work/aqa-tests/aqa-tests/jvmtest/openjdk
     [exec] gzip: jtreg_4_2_0_tip.tar.gz: No such file or directory

getDependentLibs should be updated.

TKG/scripts/getDependencies.xml

Line 26 in 0717152

buildPlatformMap.properties is not suitable for hotspot impl

A couple of side-effects of passing in that properties file to hotspot impl are mentioned in:

We need a different approach for hotspot (either a different map, or no map depending on what is best approach to fix)

Update openjdk-test and TSC docs once this repo is populated

Centralize test doc it is currently laid out across the various test locations, the 'best' overall view currently resides in the openjdk-tests/README.md file, but we should have a ToC and a description of why test material lives across locations, and how to find info in an efficient way.

run AQA Workflow didn't set up jdksource correctly if sdk_resource=build-jdk

run AQA Workflow didn't set up jdksource correctly if sdk_resource=build-jdk. Related with #175

See #175 (comment)

The quick fix could be:

Add ID for step https://github.com/adoptium/TKG/blob/master/.github/workflows/runAqa.yml#L115
Update the step AQA https://github.com/adoptium/TKG/blob/master/.github/workflows/runAqa.yml#L146 with if condition if: matrix.sdk_resource != 'build-jdk'
Add a second AQA step with if: matrix.sdk_resource == 'build-jdk' and add `env:
TEST_JDK_HOME: ${{ steps.buildJDKID.outputs.BuildOpenJ9JDK }}

Auto-detect does not work for arm 32 builds

See https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/1267, think auto-detect does not work as the value of java.vm.name is unexpected.

01:47:02 AUTO_DETECT is set to true
01:47:02 /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image/bin/java -cp ./bin/TestKitGen.jar org.openj9.envInfo.EnvDetector JavaInfo
01:47:02 System.getProperty('os.name')=Linux
01:47:02
01:47:02 System.getProperty('os.arch')=arm
01:47:02
01:47:02 System.getProperty('java.fullversion')=null
01:47:02
01:47:02 Cannot determine System.getProperty('os.arch')=arm
01:47:02
01:47:02 System.getProperty('java.vm.name')=OpenJDK Zero VM
01:47:02
01:47:02 run_configure.mk:29: recipe for target 'testconfig' failed
01:47:02 make: *** [testconfig] Error 1

Add live monitor and count tests scripts

Related to adoptium/aqa-tests#1744

Get Relevant Machine Info for Setting CPU Affinities

Problem Description

Currently, we are just getting the total number of CPUs, as shown in the snippet below:

https://github.com/AdoptOpenJDK/TKG/blob/84b082615e8cf443f3ce07e2eea9a334dbd8f52e/src/org/openj9/envInfo/MachineInfo.java#L32

In order to set affinities, we need the following info:

Number of Logical Cores
Number of Physical Cores
Number of Numa Nodes
Threads Per Core

Besides number of physical cores, we can directly get everything from lscpu. We have multiple ways of calculating number of physical cores. One of them can be: CPU(s)/ Thread(s) per core.

For example, looking at the CPU info for a Power machine which has SMT-8:

Number of Logical Cores = 160
Number of Physical Cores = 160/8 = 20
Number of Numa Nodes = 2
Threads Per Core = 8

lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                160
On-line CPU(s) list:   0-159
Thread(s) per core:    8
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Model:                 1.0 (pvr 004c 0100)
Model name:            POWER8NVL (raw), altivec supported
CPU max MHz:           4023.0000
CPU min MHz:           2061.0000
L1d cache:             64K
L1i cache:             32K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0-79
NUMA node1 CPU(s):     80-159

Other Relevant Snippet

https://github.com/AdoptOpenJDK/TKG/blob/84b082615e8cf443f3ce07e2eea9a334dbd8f52e/src/org/openj9/envInfo/EnvDetector.java#L75-L84

Proposed Changes

Add the support to fetch that info in https://github.com/AdoptOpenJDK/TKG/blob/master/src/org/openj9/envInfo/

Related Issues

Ability to Set CPU Affinities for Benchmarking (adoptium/aqa-tests#1587)

Remove ant-contrib dependency

We should investigate to see if it is possible to remove ant-contrib dependency. This will reduce our Prerequisites and simplify the playbook.

MachineInfo executes multiple times

MachineInfo is a special test and it is labeled with all three levels. However, when we try to run extended.external with BUILD_LIST=external/quarkus, it executed three times.

Grinder/8029

Dependencies can no loger be downloaded

While doing a fresh install I started to get the following messages:

downloading dependent third party jars to /home/mpirvu/FullVM/openj9-openjdk-jdk8/openj9/test/TKG/../TKG/lib
downloading http://central.maven.org/maven2/org/ow2/asm/asm-all/6.0_BETA/asm-all-6.0_BETA.jar
ERROR: downloading http://central.maven.org/maven2/org/ow2/asm/asm-all/6.0_BETA/asm-all-6.0_BETA.jar failed, return code: 2048
makefile:68: recipe for target 'getdependency' failed
make: *** [getdependency] Error 8

Trying to do the download manually I get:

501 HTTPS Required. 
Use https://repo1.maven.org/maven2/
More information at https://links.sonatype.com/central/501-https-required

Searching on the internet found that:
"As of 1/15 Maven central repo no longer accepts unencrypted http traffic"

Dynamic compilation - auto detect BUILD_LIST based on TARGET

It would be nice if we can auto-detect BUILD_LIST value based on TARGET. It means that BUILD_LIST will be an optional value. Similar to JDK_VERSION. The goal is to further reduce test build running time in Grinder and comment triggered PR builds and to make our test framework more user friendly.

Part 1: clean up cmdlinetest tests that need multiple folders in BUILD_LIST
For example, cmdlinetests need 3 folders:

BUILD_LIST=functional/cmdline_options_tester,functional/cmdline_options_testresources,functional/cmdLineTests/jvmtitests

Part 2: clean up dependencies in rest of functional tests
Part 3: add feature in TKG to generate BUILD_LIST based on BUILD_LIST and target
Part 4: clean up dependencies in other repos

Remove platform specific code in getDependencies script

We are using curl on zos and wget on the other platforms. These two pieces of code can be merged so we use curl across all platforms.