When a test calls a method on another object, and it registers an error that doesn't have the play-mode test's id set as its error source, the TestRunner doesn't collect the error and thus doesn't log that the play-mode test failed. The same happens when any error's are thrown without having the id of the current play mode test given as the error's source.

E.g, the following behaves correctly, and the play-mode test fails:

extends PlayModeTest

func _ready():
	throw_error("Example error.")
	queue_free()

but this code passes, when it should fail just the same:

extends PlayModeTest

func _ready():
	Testing.throw_error(self, "Example error.")
	queue_free()

Note: This issue belongs to the experimental branch! It specifically refers to this commit.

Try to find a way to run tests in play mode, not in editor. This will:

• Eliminate the need for user's to worry about @tool annotations in their autoloads.
• Make it easy to create tests that can instance scenes and test those scenes while they are processing.
• Make it possible for unexpected failed assertions pauses execution of tests (there should be a "Pause on Error" setting).
• get_stack will work, since the built-in debugger will be active!

See #34 for rough overview of implementation.

In the following code, Case 1 works as expected, but Case 2 fails on its error_happens call:

func test_():
	# Case 1
	error_happens(
		func():
			throw_error("error 1")
			,
		"Case 1")
	
	# Case 2
	error_happens(
		func():
			throw_error("error 1")
			throw_error("error 2")
			,
		"Case 2")

Create diagrams to provide an overview of the project structure and design.

• Sequence diagram for running tests
• Class diagram of core classes in the project
• Use case diagrams for various types of testing

When done, mention the Project UML Diagrams file as a good place to get more documentation on the plugin.

Remove the src param in the methods of testing.gd and user_test.gd. Use the built in get_stack() method in Godot 4 instead of requiring users to provide a source for errors manually!

The constructor and src property in simple_error.gd will also need to be changed accordingly.

Currently, if the user doesn't annotate their singletons with the @tool annotation, strange errors will happens if they try to use the public methods on those autoloads.

Users should be warned in the README to always add the annotation to their autoloads to ensure they work when being called by the testing framework.

The TestUtils singleton should be loaded first in the main plugin script on _enter_tree, since the Testing singleton might depend on it for initialization in the future.

In some cases, e.g. when testing user made singletons (autoloads), their code is unreachable and gives does not exist exceptions when trying to access the singelton's members.

Add a gen_msg property to the SimpleError class that may contain additional information provided by the Testing methods. This property serves as a way for the Testing methods to generate an additional error message alongside the error message written by the user.

For example, in the testing method error_happens, the gen_msg might contain the outputs of all errors that happened while running the code callable.

The gen_msg should be displayed in the dock as a child of the error's root TreeItem.

Note: This is mainly being added as a way to make creation of the expected testing method possible. This method is outlined in issue #24.

The src param in the istrue(..) and error_happens(..) methods of the UnitTest class is not neccesary, since the source of the message can automatically specified by the method bodies when calling the actual implementations of these methods on the Testing singelton.

Add inline docs to the method in testing.gd and explain the expect method in the project wiki/README

Would be good if you could add a link to the plugin in the readme and maybe a screenshot.

Create a new testing method to go along with the existing assertions in the plugin, nl. istrue and error_happens.

This method will automatically generate a gen_msg containing a comparison between the expected and the found parameters.

Sample Usage

extends UnitTest

# Unit tests for the Transaction class.

func test_add():
	var a = Person.new("Person A")
	var b = Person.new("Person B")
	var t1 = Transaction.new(10, a, b)
	var t2 = Transaction.new(7, b, a)
	var expected = Transaction.new(3, a, b)
	var found = t1.add(t2)
	
	expect(expected, found, "Adding two transactions should result in a single transaction which can replace their effect.")

If the expect method should fail, the dock will display the user provided error message, along with the gen_error message. The gen_error in this case might look something like this:
expected: Transaction(3, a, b) found Transaction(3, b, a)

Make the err_msg param in the assertion methods of the plugin optional. This is for when users want to do many assertions consequtively inside a test without writing an error message for each.

Assertions like these without error messages should not be titled by their message in the dock, but by the assertion method used, and the line it was used at inside the unit test.

Introduction

The current sample code found in res://sample tests/ does not provide a good overview of how the plugin is meant to be used. The existing contents should be deleted, and new sub-directories named basic, intermediate, and advanced should be added. These directories should each contain a clear and easy-to-understand scenario in which the SimpleTesting plugin can be used, demonstrating the proper testing approach for the given scenario.

The code to be tested in each case should be saved in separate sub-directories called "scenario", e.g., res://sample tests/basic/scenario/, and the unit tests for that scenario can be saved in sub-directories called "tests", e.g., res://sample tests/basic/tests/.

Example scenario for sample tests/basic:

The sample tests/basic directory might contain a scenario with a user-created score system ../scenario/score.gd. The sample might then include a unit test ../tests/score_tests.gd that tests the functionality of the score system with a few straightforward examples of how to use the UnitTest.istrue() and UnitTest.error_happens() methods.

Further Requirements

The 'intermediate' and 'advanced' directories could contain more complex scenarios, such as verifying proper user data storage on the local system. Additional scenarios could involve instantiating scenes and performing operations on them, like loading the game world with the player character and testing their stun-immunity status effect or health regeneration rate. [Note: If testing scenes by running them is currently not possible, a new issue should be added to address this limitation].

Conclusion

These improvements will offer users a more comprehensive understanding of the plugin's capabilities and how to utilize it effectively in various situations. This will also serve as a way for beginners in testing to gain a better grasp of testing fundamentals.

Display all the would-be error messages that istrue and error_happens methods would have generated if they failed under their respective method nodes in the dock tree. Display them in green to show that they've passed.

After implementing issue #6 - "Improve legibility of errors in the dock", the display of these green error messages should have the same format, simply green instead of red.

Note: This should be an optional setting!

Purpose

This will improve the testing dock by having it show all the assertions that did pass, to give more context when errors don't pass.

This way the repo can just be cloned into any project's res://addons directory for installation.

This would also make it easier to use this project as a git submodule in other git projects, which would be great for users and people trying to work on the plugin's development by testing it in various other projects.

Project will also have to be renamed to desired plugin folder name!

Add the option to select items in the dock and copy them as text. Also add quick "Copy All" button, or leave as copy dock hierarchy root.

This will make it easier to paste testing results into LLMs like ChatGPT.

Some classes use a custom method name for generating a string representation of the class's data. Change any of those methods found to override the built-in _to_string method in Godot instead.

Just look through all the scripts to see if you can find any of these cases. If you've covered all scripts in the project, this issue may be closed.

Making testing for any element and its children optional, will enable people to skip testing for certain parts of their projects which might take a long time to test.

Create a checkbox or visibility icon for each Tree Item and use it to enable or disable that item and its children (recursively) for testing. Also remember the user's settings by saving it to the user preferences file.

Do not display the entire error output in one item of the dock. Only display the error message and then add the rest of the error details as children to the message item in the tree.

Note: If possible, truncate error source string from the left, not the right. So the end of the source string is always visible.

Figure out why the code results in two unsuccessful tests

func test_add_with_non_statSheet():
	var s1 = StatSheet.new({Stat.CPOWER: 10})
	
	# Adding with a non-StatSheet object
	var non_statSheet = "This is a string, not a StatSheet"
	var error_prone_code = func():
		var s3 = s1.add(non_statSheet)
	error_happens(error_prone_code, "Adding with a non-StatSheet should raise an error")

	# Adding with a null object
	var null_object = null
	error_prone_code = func():
		var s3 = s1.add(null_object)
	error_happens(error_prone_code, "Adding with a null object should raise an error")

but the following is successful

func test_add_with_non_statSheet():
	var s1 = StatSheet.new({Stat.CPOWER: 10})
	
	# Adding with a non-StatSheet object
	var non_statSheet = "This is a string, not a StatSheet"
	var error_prone_code = func():
		var s3 = s1.add(non_statSheet)
	TestUtils.error_happens(error_prone_code, null, "Adding with a non-StatSheet should raise an error")

	# Adding with a null object
	var null_object = null
	error_prone_code = func():
		var s3 = s1.add(null_object)
	error_happens(error_prone_code, "Adding with a null object should raise an error")

Source: https://github.com/accmltr/Veiled/commit/0b6d4a7c24315f10cc5f33a86e9de454128a8569

I noticed that the Auto run and Auto refresh toggles in the dock become re-enabled when reloading the project after turning them off.

They should load with the same states they were in when the project was last closed.