galasa-dev / projectmanagement Goto Github PK

What happens to tests if there is a blip in the connection to the CPS and RAS? Does it recover automatically, do we need to redrive the connection?

1. "Best practice" on what you should and should not do in a Galasa test
2. How to find out about available managers
3. how to structure a test
   ...

Using the template, create single page document per Manager for the reference information. The scripts are to raise a PR with the changes for review.

Create templates and programs to generate the Manager reference information direct from the source code and update the doc site.

Will need automatically submit pull requests when run.

Items to complete are:-

1. Support Swarm
2. Authenticate to the docker server/swarm
3. Handle authentication to the Docker Registries
4. Check for orphaned containers on the swarm and clean up
5. Clean up after test run
6. Handle extract of the container log
7. Issue commands to the container
8. Retrieve files from the container
9. put files in the container
10. Include an IVT
11. JUnit should reach atleast 40% coverage, final target is 80%. Mock the HTTP Manager.
12. Include the reference documentation

Create default Grafana dashboard to monitor the Galasa Ecosystem

For those that don't want to use etcd3 or couchdb, create extensions for having the CPS, RAS, DSS, CREDs in one or MariaDB databases.

Can be the basis of a Db2 version

If relevant, create a UCD plugin to run tests against your deployed application

Run through the tutorial.

Update all the repos README with information about where to raise issues, add feature requests etc.

Create the CICS TS Manager bundle. This bundle will contain the main interaction with CICS, but will not include provisioning or site specific code.

At the moment will contain only the CECI component

Either using helm, or a custom "controller" deploy and maintain the Galasa Ecosystem in Kubernetes.

Will need to setup the CPS, RAS, PVCs, the defaults in the CPS dependent on hostnames. Setup the Resource Management, Engine Controler and run IVTs to ensure it all hangs together.

if possible run from a simple
kubectl -n xxxxx run -it galasadev/ecosystem-kubernetes

command, should be interactive and setup deployments, statefulsets, pvcs, configmaps etc.

Create a RAS, intended to run in parallel with the CouchDB RAS, so that tests are automatically archived in zip files for long term storage, for audit purposes.

Possible targets are FTPS, SFTP, HTTP, S3, Google Filestore, whatever the IBM equiv is, etc

Increase the project wide code coverage to 30%

Using a provided zOS 3270 terminal that is at the initial CECI screen, provide the following functionality:-

1. Get/Put containers
2. Link program with commarea or containers

Create a list of available managers,
a list of proposed managers and possible deliverables
a list pointing to the manager reference topics

Create the integration testing for the Docker Manager, above JUnit and IVT.

Will need to provision a Linux server with Docker installed on it, (via Openstack)

When connecting to a TLS telnet port and the tls=true cps property has not been set for the image, the client will fail with:-

Exception in thread "main" dev.galasa.zos3270.spi.NetworkException: Unable to connect to Telnet server
at dev.galasa.zos3270.internal.comms.Network.connectClient(Network.java:102)
at dev.galasa.zos3270.spi.Terminal.connect(Terminal.java:43)
at t.TestTerminalTso.main(TestTerminalTso.java:9)
Caused by: dev.galasa.zos3270.spi.NetworkException: Expected fffd28 but received fffd2e
at dev.galasa.zos3270.internal.comms.Network.expect(Network.java:251)
at dev.galasa.zos3270.internal.comms.Network.negotiate(Network.java:156)
at dev.galasa.zos3270.internal.comms.Network.connectClient(Network.java:93)
... 2 more

as fffd2e means IAC DO START_TLS, we should be able to continue from this point using TLS and a SSLSocket.

Will mean the inputstream/outputstream fields need to be reworked in dev.galasa.zos3270.internal.comms.Network.createSocket() and move the IAC DO TN3270E check earlier

Documentation for the new CECI Component (CICS TS):-

New Features
and any other docs above the Manager reference material

Documentation above the reference material

Give the managers and tests the ability to validate events within a test lifecycle so that errors can be thrown at interesting points and avoid duplicating validation code in tests.

Example.

Each time a z/OS Batch Job is read back, a validation event should be issued so that, possible, a customer manager can check the batch job for common errors and throw an exception if foundm, which will stop the test from running.

Each Manager will need to publish what validation event points are available and what is in the parameters.

Create a sample markdown file of the http Manager.

Convert to a template that can be used by the documentation tool.

Having started SIMBank running I connected to it using a PECOM session and connected though to the simulated CICS screen.

At this stage I got distracted onto something else and on return saw a CICS logo, cleared the screen, typed "cemt inq task" and pressed enter.

Apart from a brief "x-System" at the bottom, nothing seemed to happen and after a bit of thinking I remembered this was a SimBank session and I should have entered BANK so I:

Pressed clear - nothing happened
Pressed F3 - nothing happened
Deleted all the characters, type bank and pressed enter - screen changed to have "bank inq task" and nothing else happened.

During a recreate of this, no messages were issued to my SimBank console.

I was finally able to continue by overtyping the "inq task" with spaces and pressing enter.

Obviously this is a simulation so does not have to be perfect and there are many things it could do:,

a) Invalid input could result in the data being removed from the screen and issue a message to the screen that xxx was invalid.
b) Recognise and process clear and f3 keys being pressed (perhaps an error for any other key).
c) Mimic CICS behaviour more closely and take the first word of input or its first four characters if longer. Then my unintentional "bank inq task" would be seen as "bank" with some ignored input.(Perhaps with a) above also incorporated).

Create a CECI Manager to the CICS TS Manager.

The CECI Manager will need to provide an annotated field the tester can use. CECISession?. The tester will need to obtain a zOS3270 Terminal, signon onto the target CICS region and start the CECI transaction. Can then pass the terminal instance to a method of the CECI component to perform actions. For the initial release, the following functionality will be available:-

1. Put/get containers
2. Link program with commarea or container.
   Will need to cope with large data structures.

Create the functionality in SimPlatform and SimBank to simulate the zOS/MF Jobs restful service. Possible job would be creating testing accounts in the SimBank data lake.

Write a tutorial to extend the SimBank testing to include Batch Jobs.

The SimBank application is to receive Jobs from SimPlatform. If the JCL doesn't match EXACT what it is expecting, return unprocessed (simplatform will pass to future applications for processing).

If the JCL, eg PGM DDs DD * etc matches, load the Derby database with the accounts provided.

If relevant, create a maven plugin goal to be able to run tests against your built application

Test the Galasa Ecosystem at extreme scale, running 100s of tests in parallel, GBs of result data etc

Create a RAS, intended to run in parallel with the CouchDB RAS, so that tests are automatically archived in a PDF file for long term storage, for audit purposes.

Possible targets are FTPS, SFTP, HTTP, S3, Google Filestore, whatever the IBM equiv is, etc

Finish writing the Docker Manager to be able to create and cleanup docker containers (not volumes or networks) on a Docker swarm.

Store the credentials in Eclipse Secure Storage. Will need to use the http listener to get to them, and therefore convert to https, so need to pass the self-signed public certificate in the overrides, via the certificate management. This would be for local runs only.

Start running integration tests in the Shared Environments in the pipeline

If there is a space in the directory name, the New Example Project fails for Galasa:-

eclipse.buildId=4.7.3.M20180330-0640
java.version=1.8.0_221
java.vendor=Oracle Corporation
BootLoader constants: OS=macosx, ARCH=x86_64, WS=cocoa, NL=en_GB
Framework arguments: -product org.eclipse.epp.package.java.product -product org.eclipse.epp.package.java.product -keyring /Users/mikebyls/.eclipse_keyring
Command-line arguments: -os macosx -ws cocoa -arch x86_64 -product org.eclipse.epp.package.java.product -data file:/Users/mikebyls/ws/galive4/ -product org.eclipse.epp.package.java.product -keyring /Users/mikebyls/.eclipse_keyring

dev.galasa.smbank.ui
Error
Mon Dec 02 14:34:12 GMT 2019
Error

java.net.URISyntaxException: Illegal character in path at index 26: file:/Applications/Eclipse 2.app/Contents/Eclipse/plugins/dev.galasa.simbank.ui_0.3.0/lib/galasa-simbanktests-parent-examples.zip
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parseHierarchical(URI.java:3105)
at java.net.URI$Parser.parse(URI.java:3053)
at java.net.URI.(URI.java:588)
at java.net.URL.toURI(URL.java:953)
at dev.galasa.eclipse.ui.wizards.simbank.ExampleSimbankOperation.run(ExampleSimbankOperation.java:75)
at org.eclipse.ui.actions.WorkspaceModifyDelegatingOperation.execute(WorkspaceModifyDelegatingOperation.java:68)
at org.eclipse.ui.actions.WorkspaceModifyOperation.lambda$0(WorkspaceModifyOperation.java:107)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2240)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2267)
at org.eclipse.ui.actions.WorkspaceModifyOperation.run(WorkspaceModifyOperation.java:128)
at org.eclipse.jface.operation.ModalContext.runInCurrentThread(ModalContext.java:437)
at org.eclipse.jface.operation.ModalContext.run(ModalContext.java:353)
at org.eclipse.jface.wizard.WizardDialog.run(WizardDialog.java:980)
at dev.galasa.eclipse.ui.wizards.simbank.ExampleSimbankWizard.performFinish(ExampleSimbankWizard.java:40)
at org.eclipse.jface.wizard.WizardDialog.finishPressed(WizardDialog.java:778)
at org.eclipse.jface.wizard.WizardDialog.buttonPressed(WizardDialog.java:417)
at org.eclipse.jface.dialogs.Dialog.lambda$0(Dialog.java:619)
at org.eclipse.swt.events.SelectionListener$1.widgetSelected(SelectionListener.java:81)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:249)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:86)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4258)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1502)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1525)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1510)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1314)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4082)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3699)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:818)
at org.eclipse.jface.window.Window.open(Window.java:794)
at org.eclipse.ui.actions.NewExampleAction.run(NewExampleAction.java:121)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:473)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:565)
at org.eclipse.jface.action.ActionContributionItem.lambda$4(ActionContributionItem.java:397)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:86)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4258)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1502)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1525)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1510)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1314)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4082)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3699)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1150)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1039)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:153)
at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:680)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:594)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:151)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:653)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:590)
at org.eclipse.equinox.launcher.Main.run(Main.java:1499)

Add annotations to the test and thereby the test catalog for Engine capabilities so tests are run on the correct engine.

eg, Selenium, IPv6

Explore and create the ability to backup the CPS and RAS. Also restore the data. Test in DR mode

Extend SimPlatform to support:-

1. Submit
2. status
3. extract
4. purge

Should simulate the queues, input, active, output, held.

Submit should hand the JCL to SimBank for validation and through a generic JCL Error if not happy. ie, for a submit, the simulation should receive the JCL without error and return, in the background pass to SimBank. If SimBank not happy, place on the "held" queue with a generic JCL error. otherwise simulate the job complete.

Either using compose, or a custom "controller" deploy and maintain the Galasa Ecosystem in native Docker or a Docker Swarm.

Will need to setup the CPS, RAS, volumes, networks, the defaults in the CPS dependent on hostnames. Setup the Resource Management, Engine Controler and run IVTs to ensure it all hangs together.

if possible run from a simple

docker run -it galasa/ecosystem-docker

command, should be interactive and setup restartable containers.

Create a Galasa Ecosystem as a shared environment and run as many IVTs against it, via the Jenkins plugin.

Handle certificates in the framework so that we don't need to install self-signed/testing certificates onto the operating systems.

Public Certificates should go into the CERTS
Private keys should go into the CREDs.

As it is key/value pairs, need to base64 them.

Need UI to make it easier to extract and add to the CERTS/CREDs.

The Shared Environments framework will allow Galasa to create environments to be tested against across multiple test runs. Usually used for complicated integrated environments or long lasting.

The initial use case is to create a Galasa Ecosystem in Docker and Kubernetes to test Galasa against and run all the Integrated tests, including eclipse testing. So will need to spin up multiple Linux and Windows servers and tie them all together. And clean up afterwards.

Future enhancements, not included in this epic, would be create environments before tests need them, esp for environment that take a long time to create, eg zPDT. Create environment to develop tests on, so you dont have to wait for them to spin up everytime you make a tiny mod.

using the enhancements in SimBank to receive a batcj job with new accounts to create in the data lake, create a tutorial to demonstrate this.

Takeaways would be:-

1. How to submit, wait and check batch jobs
2. Possible scenario for solving test data and data lakes.
3. Deep integration with z/OS

The static part of the run log messages are too long, eg :-

27/11/2019 12:25:24.841 DEBUG dev.galasa.framework.TestRunManagers. - The following Managers are active:-

which means you have to look a long way right to get to the interesting stuff.

Change the log4j properties so that for non-trace settings the log line should look like:-

27/11/2019 12:25:24 DEBUG d.g.f.TestRunManagers - The following Managers are active:-

Until the CICS TS provision is in place, any integration testing will need to be performed with a static CICS TS Region somewhere.

The beta FV testing to exercise CECI against the real CICS region.

Support the new Shared Environments run structure, what ever that may be

Allow for credentials to be secured in Vault for local and automated test runs

Will povide a lifecycle for checking if a defect is open to change the overall result. Also update a issue/workitem for test result.

Will not actually do the update but pass over to an extension for github or rtc

At the moment have to manually encrypt and store credentials. Add Eclipse tools to do it. Key is stored in CPS. May look at storing that elsewhere.

For use with the existing test run step, similar to the JAT one.

ie read the test catalog and extract the tests required