integrity-tf / integrity Goto Github PK
View Code? Open in Web Editor NEWIntegrity Test Framework - a sophisticated integration test automation framework for Java
Home Page: http://www.integrity-tf.org
License: Eclipse Public License 1.0
Integrity Test Framework - a sophisticated integration test automation framework for Java
Home Page: http://www.integrity-tf.org
License: Eclipse Public License 1.0
One should be able to request a java.sql.Date instead of a java.util.Date and get dates converted into that object type. However, at the same time the conversion to java.util.Date should still be used if just a java.util.Date is requested.
Loaded test resource 'C:\ew\integrity_test\SuiteOrganization\Integrity\suite.integrity': 0 errors.
Loaded test resource 'C:\ew\integrity_test\SuiteOrganization\Integrity\fixture.integrity': 0 errors.
Resolving the test model...done!
Test execution has begun...
Now entering suite 2: suiteorganization.rootSuite
Now entering suite 3: suiteorganization.otherSuite
Defined variable result with initial value: 8
Now executing call 1: Multiply 12.5 with 8...
SUCCESS!
Now running test 1: Divide 100.0 by 8...
SUCCESS!
Now leaving suite 3: suiteorganization.otherSuite
Now leaving suite 3: suiteorganization.rootSuite
Finished executing 3 suites with 1 tests and 1 calls in 79 msecs!
1 tests finished sucessfully, accompanied by 0 failures and 0 exceptions.
Even in the tutorial example, suite numbering seems to be offset, and the "leaving" messages are wrong as well. Total suite count is wrong, too.
When using primitive types for a parameter in a fixture signature the call to it may fail, if the user does not give any parameter for it.
The runner tries to call the fixture with an Object[1] = { null }
array, which leads to a stacktrace like:
java.lang.IllegalArgumentException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at de.gebit.integrity.fixtures.FixtureWrapper.execute(FixtureWrapper.java:262)
...[snip]...
Please:
@FixtureParameter(optional = true)
, which is not necessary to reproduce this, but is an indication that something is wrong here.Even though the exact same test script is run, different forks may have different colors in the Test Runner view. It would be nicer if they had the same color.
I learned from #50 that there is some kind of operation possible on variables of the kind. That leaves me more puzzled than it was enlightening to me as I could not find documentation for this here: http://www.integrity-tf.org/docs/tutorials/tutorial2.
That leaves me with the following questions:
one.toLowercase
actually "one".toLowercase()
or "one.toLowercase"
?Possibly more things one should know about operators, but I never dared to ask?
My very first impression: Feels like pandora's boxed which just opened...
If a fixture which has a description with conditional parts in it is called inside a suite, which itself offers some parameters and forwards an optional one of those to the fixture call, the conditional text for the "parameter is present" situation is used, even though the parameter is actually not present in the call to the suite.
call createBO class: "Country"
+twolettercode: "US"
+threelettercode: "USA"
+digitcode: "840"
+name: "United States of America"
+phonecode: "049" -> usa
call createBO class: "Province"
+code: "AL"
+name: "Alabama"
+country: usa
Assuming "createBO" is a fixture which creates an instance of a simple bean class, the code above results in an undesired autoconversion of the object inside the variable "usa" to a map. In this situation, this conversion is counterproductive and should not be done.
The XSLT transform after running all tests can take some time when dealing with big test suites. During that time, the test control view shows "running tests" as info text. There should be a special text for this phase, to let the user know that the client has finished test execution and is now post-processing result data.
packagedef testpack with
suitedef suite1 with
suiteend
suitedef suite2 requires suite1 with
suiteend
suitedef suite3 requires suite2 with
suiteend
packageend
should result in execution order
suite1
suite2
suite3
but suite1 is completly ignored
In case of a fixture like this...
@FixtureMethod
public Object echo(@FixtureParameter(name="value") Object aValue) {
System.out.println("echo got value: " + aValue);
return aValue;
}
...the Integrity-internal ...Impl classes wrapping script-provided values are delivered with no conversion. Thus a test fixture based on the method above fails since for example IntegerImpl is not meant to be compared to another IntegerImpl. Generally it should probably be best to apply default conversions to values going in a fixture method like the above one (specifying just Object as target type), because the fixture author shouldn't have to deal with Integritys' internal types.
A fixture that is defined to have two parameters "one" and two" is usually called this way:
call testFixture one: 1 two: 2
but this is also allowed:
call testFixture one: 1 two: 2 one: 2
the fixture will be called with param "one" = 2.
I defined a call simpleParamFixture that receives 2 objects. One object is defined as java.lang.Number. Afterwards I call the that fixture with a numerical String value that exceeds the maximum integer e.g. "2147483648". Integrity will try to convert the String into a Number using the StringToInteger conversion which obviously fails. Integrty than returns with an error.
However Integrity should use the StringToLong conversion in this case which would not fail.
suitedef number_test with
variable bla
call hello.echo value: "2147483648" -> bla
call hello.echo value: bla
call simpleParamFixture obj: {
param1: bla
param2: bla
}
suiteend
At least the ISO time format (possibly others as well) should be able to deal with millisecond precision. Currently the grammar doesn't even allow to enter them.
Possibly, for conversion in a java.sql.Timestamp, even nanoseconds might be useful.
Something like this creates an exception during conversion:
call arbitraryCall +testparam1: 1, 2, 3, 4
Please provide some kind of hook for failed tests.
Background is that we are using Integrity with Selenium Fixtures and we would love to have the chance to include additional information for failed tests to the output.
What we want to be able to do is to include a plain page dump ๐ and a screenshot ๐ท to each failed test, so we can either see or reproduce a failed test more easily.
(The plain page dump allows us to rerun a given selector on the plain html output)
I could think of more platforms (e.g. Swing) where screenshots would be helpful or other trigger (e.g. screenshots after each command, for debugging purposes) - but for a first, a trigger for failed tests would be the most helpful one.
What could be a kind of problem is that one most likely needs some kind of context from the fixture (the Selenium instance in our case) which collects the actual data and pipes this to the test output, where it should be included somehow.
This does seem to depend on a certain test structure, as it does not happen in any case, but is reproducible if you've got the problem once.
That handling should be improved.
The documentation from Tutorial3 mentions that:
These dependencies are executed before the actual suite that depends on those is run, but only if the dependent suites haven't already been run in the current "execution stack trace". This sounds complicated, but is actually a very simple system.
Dependency management in general is actual far from beeing very simple.
The current implementation doesn't explain what the current execution stack trace is, but from what I've learned it seems to be the current test execution, making the statement an the dependency is run if it is not run in this test execution before.
It would be analogous to an @BeforeClass
in JUnit on the top test suite.
However the concludedby
keyword is tied to a different scope than the duplication mechanism, which prevents the dependency from beeing triggered more than once in it's scope.
Imagine a suite login
which is concluded by it's equivalent logout
. Given the following scenario:
Suite Root
+- Suite A (requires login)
+- Suite B (requires login)
will result in the following execution:
Root
login
as a dependency of Suite A
A
logout
because login is a dependency of A
and is concluded by logout
- this is the place where the concluded by triggersB
(note that Suite login
is not triggered as it is already run in the current execution stack trace)The duplication detection of the requires
keyword is test execution scope where as the concludedby
keyword scope is Suite scope.
I would assume those two to be a pair (like @Before
and @After
) and would at least expect them to have an equivalent scope.
However this seems to be not the case, the duplication detection of requires
is somewhat equivalent of an @BeforeClass
and the concludedby
keyword is equivalent to @After
, which is an odd pair.
If Unix-style line endings are used on Windows, or Windows-style endings on Unix, the formatter cannot correctly format line indentations currently. Ideally this should work even with the wrong line endings, since the Eclipse editor generally supports any line ending style on any system by auto-detecting the used line endings and adapting file display.
java.lang.IllegalArgumentException: Argument cannot be null
at org.eclipse.swt.SWT.error(SWT.java:4263)
at org.eclipse.swt.SWT.error(SWT.java:4197)
at org.eclipse.swt.SWT.error(SWT.java:4168)
at org.eclipse.swt.widgets.Widget.error(Widget.java:468)
at org.eclipse.swt.widgets.Text.setText(Text.java:1967)
at de.gebit.integrity.eclipse.views.IntegrityTestRunnerView.updateDetailPanel(IntegrityTestRunnerView.java:1673)
at de.gebit.integrity.eclipse.views.IntegrityTestRunnerView.access$26(IntegrityTestRunnerView.java:1542)
at de.gebit.integrity.eclipse.views.IntegrityTestRunnerView$15.selectionChanged(IntegrityTestRunnerView.java:1104)
at org.eclipse.jface.viewers.Viewer$2.run(Viewer.java:164)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.ui.internal.JFaceUtil$1.run(JFaceUtil.java:49)
at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:175)
at org.eclipse.jface.viewers.Viewer.fireSelectionChanged(Viewer.java:162)
at org.eclipse.jface.viewers.StructuredViewer.updateSelection(StructuredViewer.java:2188)
at org.eclipse.jface.viewers.StructuredViewer.handleSelect(StructuredViewer.java:1211)
at org.eclipse.jface.viewers.StructuredViewer$4.widgetSelected(StructuredViewer.java:1241)
at org.eclipse.jface.util.OpenStrategy.fireSelectionEvent(OpenStrategy.java:239)
at org.eclipse.jface.util.OpenStrategy.access$4(OpenStrategy.java:233)
at org.eclipse.jface.util.OpenStrategy$1.handleEvent(OpenStrategy.java:403)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4165)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3754)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2701)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2665)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2499)
at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:679)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:668)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:123)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
at org.eclipse.equinox.launcher.Main.main(Main.java:1386)
Dots are used to split fully or partly qualified identifiers in package name (parts) and identifier name. Thus it is illegal to declare a variable which has a dot in its identifier. However, the validator currently doesn't seem to see such a construct as an error (even though it's not usable correctly later).
When primitive types are used as types in fixture methods, IllegalArgumentExceptions can be thrown if there is no value provided in the test script for that parameter.
Sensible defaults (0 in case of numbers) should be delivered instead.
It would be nice when the Integrity Plugin would fully support the outline feature of eclipse. This is especially very useful for big test-suites.
Currently the plugin supports only outlines for packagedef, testdef and suitedef. Outlines of tests are not supported (like test and tabletest), see this screenshot:
...even though it should, since it conforms to the ISO time format
Is is quite exhausting to compare expected maps with result maps in order to find out which values does not match. This is especially true if many attributes are involved.
DefaultResultComparator.performEqualityCheckForMaps() knows which value differs. It should pass this information to the reporting layer and the reporting layer should visualize this information either by color, detail info or both.
Example, two suite calls:
suite aSuite optionalParam: 123
suite aSuite
The latter call will receive the parameter "optionalParam" set to "123", even though that's clearly not intended.
Currently the resulting html page can expand and collapse individual entries. ๐
However what get's annoying is that if one wants to copy data from an entry the entry collapses on the first click.
What would be better is when there would be an explicit collapse button which collapses the entry instead of "anywhere on the panel" as it is now.
Title is pretty self-explanatory; feature is commonly known with Java files.
If one uses a suite which is not part of any file passed to the console runner, the following exception is shown.
Exception in thread "main" org.jdom.IllegalDataException: The data "null" is not legal for a JDOM attribute: A null is not a legal XML value.
Because it can't get the name of the suite which is referenced (in this case the suite which was specified as required could not be resolved).
This is not a very helpful error message, please provide a better one, which shows the error cause and the information which is available (which should be at least the name of the referenced suite as given in the suite which is referencing it).
Integrity cannot map between a variable that is defined using an operation and the target type of String (and maybe others).
The parameter param1 will get result the value: (de.gebit.integrity.dsl.impl.StandardOperationImpl@1b5c3de (operators: [+]))
instead of 13 as expected
variable bla initially 12
variable blubb initially (bla + 1)
call simpleParamFixture obj: {
param1: blubb
}
where simpleParameterFixture is a Fixture that gets an Object composed of a String parameter named "param1".
rejected
This name override attribute doesn't really seem to work correctly, and it has never really been used anywhere and is generally a bit too much abstraction. It shall be removed.
If a construct like this is created...
variable aVariable
call echoMap map: {
key: (aVariable + 2)
}
...a NullPointerException is thrown during dry run phase. This does not interfere with test execution, but is of course irritating.
Eclipse plugin does not support:
operations (12 + 1)
or custom operations ["Helo" tLowercase]
There are conversions converting various kinds of dates to SQLDate, but none for SQLTime.
Something like this does not work currently:
constant transactionConst {
subject: "Lottery Win"
value: 1000.00
}, {
subject : "Cash Withdrawal"
value : -220.00
}
This is due to the fact that constants and variables can only be pre-allocated with single values, not multiple values, as supported in pretty much all other places where fixed values are used.
It would be beneficial to extend the constant/variable feature to support multiple values as well.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.