cuberite / cuberitepluginchecker Goto Github PK

The scenario file should be able to manipulate files and folders:

• Create file with (text) contents
• Copy file
• Rename file
• Delete file
• Create folder (recursive)
• Rename folder
• Delete folder (recursive)

Command fuzzing could be made much more intelligently:

1. Execute a command with (unique) random parameters
2. If the handler just outputs a message to the player, try to see if it contains the command name, if so, try to parse usage (number of parameters) from it, otherwise just retry a few times with a different count of parameters
3. If the command handler accesses special APIs, take note which parameter is used for the API call and use a proper value for the next retry.
   • cRoot:GetWorld() -> the param is a world name
   • cWorld:GetPlayer(), cWorld:DoWithPlayer() -> the param is a player name
   • tonumber -> a number is expected for this param
4. Retry a few times with params adjusted based on the heuristics in step 3.

The checker will need more information on how to test the plugin as much as possible. While the special API implementations can add callback requests (such as when registering a command, we can add a request to call the handler with various params), the plugins will need more specific testing, such as specific commands in specific sequences. Or perhaps generating in-game events in a specific order (player connected, player executing a command, player interacting with the blocks, ...)

When adding redirection or using FS operations in the scenario file, the paths default to relative-to-Checker. However, it would be better to use relative-to-ScenarioFile paths, not to depend on specific Checker location.

Inspired by the difference between TravisCI and CircleCI, each clone the project out to a different folder structure.

It seems that using cPlayer:GetWorld() crashes the checker, because it attempts to convert it to a cBroadcastInterface somehow:

Attempting to use an unknown Global value "cBroadcastInterface", nil will be returned.
stack traceback:
	Simulator.lua:1168: in function <Simulator.lua:1152>
	Simulator.lua:501: in function 'createApiEndpoint'
	Simulator.lua:554: in function <Simulator.lua:548>
	/home/ubuntu/Core/spawn.lua:6: in function 'SendPlayerToWorldSpawn'
	/home/ubuntu/Core/spawn.lua:57: in function 'Function'
	Simulator.lua:1392: in function 'processCallbackRequest'
	Simulator.lua:966: in function 'executePlayerCommand'
	Scenario.lua:179: in function 'fuzzSingleCommand'
	...
lua: Simulator.lua:501: attempt to index field '?' (a nil value)
stack traceback:
	Simulator.lua:501: in function 'createApiEndpoint'
	Simulator.lua:554: in function <Simulator.lua:548>
	/home/ubuntu/Core/spawn.lua:6: in function 'SendPlayerToWorldSpawn'
	/home/ubuntu/Core/spawn.lua:57: in function 'Function'
	Simulator.lua:1392: in function 'processCallbackRequest'
	Simulator.lua:966: in function 'executePlayerCommand'
	Scenario.lua:179: in function 'fuzzSingleCommand'
	...

Plugin: cuberite/Core#197 (comment)
CI log: https://circleci.com/gh/cuberite/Core/57

It should be pretty straight-forward to provide the AutoAPI and ManualAPI files from the official buildserver - just archive the stuff together when building Cuberite.

These are key to properly testing the plugins' callback-storing.

Using scenario files' initializePlugin action it would be possible to load multiple plugins and test them together (allowing testing the inter-plugin calls).

However, it should still be possible to use the Checker in the classic single-plugin way, using the -p parameter.

I had to use a hardcoded path in cuberite/Core#194 to make the ci tests pass.
@madmaxoft noted that might be a bug in the plugin checker see comment.

I'd like someone to look into this.

Cuberite can currently generate a list of API functions that are automatically exported by ToLua++, via its src/Bindings/GenerateBindings.lua script. The ApiLoader should load the API from that format, as well as have a way to specify more API description files to read (for the manual bindings).

More player actions should be available in the Scenario file:

• movement
• placing block
• breaking block
• using a block (entity)
• damaging an entity
• chat message

The table name in Utils.lua is util, I think that should be utils.

Edit: Found one more problem the function deleteFolderContents is recursive and the call should be then utils.deleteFolderContents(fullName) instead of deleteFolderContents(fullName).

When an API function takes a callback, the call to that callback should be postponed until the current scenario step is finished, and only then should it be executed. This helps find the following bugs:

local function commandHandler(a_Player, a_Split)
  a_Player:GetWorld():QueueTask(
    function()
      a_Player:SendMessage("BUG!")
      -- a_Player may be invalid here
    end
  )
end

If a plugin stores a parameter value from a callback for later, it should be possible to provide better diagnostics about where exactly the parameter was stored. If the callback is a 100+ line function, it can be quite difficult for a human to analyze it.

Using the debug library it should be possible to examine the function's locals and upvalues, seeing which one contains the saved instance. For example if the callback creates a new closure with the value captured into the closure, it should be possible to see that - there'll be a local value (the closure) of type function whose upvalue would be equal to the saved object.

It may be necessary to re-execute the handler in order to be able to see its internals via the debug library, possibly even installing a debug hook so that code can be executed while the function is still on the stack. Perhaps this could be made into a new detection mode that installs a hook for each callback's return and checks all its locals / upvalues.

Luacheck should be added as another test in the CI.

We need to document how this thing is to be used, what the various command line parameters do, what checking methods are supported and how to integrate this for checking plugins in a CI.

A static function cannot be assigned a special implementation, the param matcher will not recognize it.
Even worse for functions that are optionally-static, such as cPluginManager:BindCommand()

Take an existing Cuberite plugin and add a CI test for it using the Checker.

Document the CI usage in the main docs file - provide the CI config as an example.

Scenarios should allow testing (and fuzzing?) the webadmin handlers.

If possible, the webadmin tests should also detect cWorld methods usage and log a warning for these (world thread vs webadmin thread deadlock)

Constructors and operators need to be created when creating the class meta-table, because their usage doesn't go through the __index meta-method.

Console commands should get registered and should have Scenario-file support similar to in-game commands - executing a console command and fuzzing the console command handlers.