lpghatguy / lemur Goto Github PK

Now that we have a high performance timer built into Lemur, we can take a whack at implementing a (rough) implementation of Roblox's microprofiler.

Thankfully, the interface that Lua has is super simple and should be trivial to replicate:

debug.profilebegin(name: string)
debug.profileend()

Notably, I want to use this in Roact to have as much instrumentation as possible into performance.

For reference, here's a microprofiler flame graph from a project I'm working on that uses Roact with added instrumentation, which isn't part of Roact right now:

Right now, you can use Instance.new("TestService") (or any other illegal class) when you usually can't on Roblox. You shouldn't have to use game:GetService for all of these as that'll end up making tests more difficult.

Instance.new should work like it does on Roblox for the project's sake. At the same time, there should be an internal function like Instance.newForced or something along those lines that'll bypass those checks entirely.

For Instances, ClassName is not considered a property in the internal properties table. Although the only scenario this has mattered in is #5 (which doesn't matter because ClassName never changes anyway), this could become an issue in the future. Best to implement readonly properties and include ClassName now.

We need for this a feature of Roact!

This line in the README:

local root = habitat:loadFromFs("src/roblox", ReplicatedStorage)

Doesn't actually work. You get the error:

passedOptions must be type `table`, got type `Instance`

This isn't a PR to fix it because I'm not sure if it's supposed to be able to take instances, or if it's just a typo.

Roblox types override __eq. This is sort of fine, but accessing properties on instances creates Lua versions of those values every time, which means they aren't rawequal and also don't hash the same! This is especially surprising for things like signals on instances that don't ever change.

This was a Lemur difference that caused a real bug in Roact. Basically:

local signalMap = {}
local part = Instance.new("Part")

signalMap[part.Changed] = "foo"

print(signalMap[part.Changed]) -- "foo" in Lemur, nil in Roblox

Basically everything related to callbacks in the code uses a variable to track function calls. They litter the signal spec code.

https://github.com/LPGhatguy/lemur/blob/705238b707d48ee8ce926814aa0cf22393caa32e/lib/Signal_spec.lua

Busted provides something called "spies" for this exact usage, where it tracks if a given function is called. This would help make the code more readable.

I assume this line is to replicate the behavior that disconnects all events when :Destroy() is called, but this (at least from reading the code, I haven't tested it) also breaks GetPropertyChangedSignal connections.

local part = Instance.new("Part")

part:GetPropertyChangedSignal("Name"):connect(function()
	print(part.Name)
end)

part.Changed:connect(print)

part.Name = "A"
part:Destroy()
part.Name = "B"

This code will print A and B. In lemur (again, presumably from reading the code) it'd only print A.

Lemur should support the same sort of file structures that Rojo does.

In Roact, I wrote code to do this in the benchmark and spec scripts, which seems non-ideal. Essentially every project I've started brings in that function or something equivalent to it.

Rojo is (currently) the standard tool for syncing code into Roblox from the filesystem, so lining up with its conventions makes sense. Rojo also happens to line up with how stock Lua loads folders as modules.

I would like to test out my data store module with lemur to make sure I don't seriously break something with a new update. I think lemur should have fake DataStores that perhaps just delay for a while on usage (#1), have emulated throttles, and sometimes just error like data stores actually do.

We should implement the newest HttpService method, RequestAsync. Implementing GetAsync and PostAsync aren't as important, but they should be okay to bring along.

The thing that will be annoying is error codes -- Roblox's API relies on string matching with cURL's error numbers. I think RequestAsync is a little better (there's an enum!) but implementing GetAsync and PostAsync might be problematic enough to not bother with.

I want to use this to enable end-to-end testing for Rojo.

This is the first part of #1.

Matching the convention Rojo set here makes sense. Lemur should find .server.lua and .client.lua files and import them as Script and LocalScript instances, respectively.

I know this is placeholder while there isn't a thread scheduler, but why this behavior? Why not an error?

There should be some way to temporarily churn through a Roblox-style event loop to let things like wait, spawn, and RunService work.

It's important that the system doesn't permanently transfer work so that the wrapping code can still regain control over the process, like in CI test scripts.

https://developer.roblox.com/api-reference/datatype/NumberRange

Fun fact, did you know Roblox doesn't either?

> print(Color3.new("Doge"))
0, 0, 0
> print(Color3.new("Doge", "is", "Funny"))
0, 0, 0

There's now a base class for all *Value objects, this would be valuable for testing small things that depend on IsA.

It doesn't look like any of the ValueBases do any type checking, they just have different default values. They should be moved to InstanceProperty.typed.

I'm not sure if this is out of scope for lemur, but I want to run extensive unit testing on my RemoteEvents to make sure they're bulletproof.

Currently, neither of these objects are supported in lemur. For this to be added, I'm sure that fake players would also need to be implemented. Roblox itself runs tests like this (as shown by TestService.NumberOfPlayers).

I was setting up lemur for another one of my projects, but I made a typo in the file name so it pointed to a file/directory that didn't exist. Lemur silently returned nil, and I had to read the code of Lemur to figure out it doesn't check for you.

Right now, tick maps to os.clock, which only has second resolution.

We should let tick still fall back to os.clock if LuaSocket is missing, but we also need to have some way to figure out if the high performance timer is enabled.

Currently instance types do not extend off of any other class.

Ideally, this would not only help with methods like :IsA but also would make creating new instances easier.

Simple typo

This is a lofty one, so I'm making a tracking issue for it.

Roblox/roact#52 requires support for GetFullName in order to run tests using lemur.

Not sure how exact the property should be (e.g. should it create a Character/Humanoid?), but this is something I'd like to see. Right now I have to shim GetPlayerFromCharacter.

GenerateGUID doesn't actually generate a GUID--it gives a constant one. This is fine, but it's deceptive and looks random. I don't see a problem in changing it to something to where developers can easily tell its a shim (like 000000-...)

We already have BindableEvent!

But does in Roblox, which means this code:

return function(part)
	if part.Parent == nil then return end

	local parent = part

	repeat
		parent = parent.Parent

		if parent:FindFirstChild("Humanoid") then
			return parent
		end
	until parent:IsA("Model")
end

needs to be written as

return function(part)
	if part.Parent == nil then return end

	local parent = part

	repeat
		parent = parent.Parent

		if parent:FindFirstChild("Humanoid") then
			return parent
		end
	until parent:IsA("Model") or parent:IsA("Workspace")
end

It can be useful to test APIs that aren't accessible at all levels, or to make sure that your code isn't accessing any APIs that can't be used by regular scripts.

Lemur should probably have a notion of script security levels. This is pretty tricky with regards to modules!

Right now, you can write code that accesses things like io.open and your Lemur tests will still pass, while obviously failing in Roblox itself.

I imagine there are cases where it's useful to access those APIs intentionally, but I think there should be a Lemur API to disable them. There's already precedent for this (sort of accidentally) in the way that require shadows Lua's stock require function.

There should probably be a way to intentionally inject new global APIs into the environment without forking Lemur.

Lemur doesn't work in anything higher than Lua 5.1 because of its use of newproxy and setfenv. This is completely fine, and makes testing more accurate to Roblox, however the errors you get if you're accidentally running on a more recent Lua aren't helpful if you don't know about the removal of these functions.

Lemur should detect if you're on an updated Lua build in init.lua, and give a better error message if so.

We should detect if newproxy is available and tell the user that Lemur needs Lua 5.1, not 5.2 or newer.

It would be nice if Lemur had instances for RemoteEvent and Remote function. Additionally, it would be nice if when it is building its dependency tree it could automatically create them when *.model.json (a Rojo thing) files are encountered of those types.

Work around is simple (just add them to the dependencies to be injected by the unit test (kind of like a module script mock), but it would be great if it just worked.

MockDataStoreService uses it.

I am doing CI testing via lemur+testEZ for this project: https://github.com/buildthomas/MockDataStoreService. In one of my files, I need to swap between a fake datastore implementation or the real DataStoreService depending on the environment:

[...]
local shouldUseMock = false
if game.GameId == 0 then -- Local place file
	shouldUseMock = true
[...]

Unfortunately, it doesn't seem like lemur's Game class has any of the properties of DataModel, so I can't perform the test I want to perform on this file right now.

My suggestion is to add some properties to DataModel with sensible defaults (as if it's running in an offline Studio file):

(Properties GearGenreSetting and Genre excluded here because I don't think anyone will use these / they seem candidates for deprecation)

I might be using Lemur, and TestEZ incorrectly, but coming from a C# and Java background where I'm used to unit testing frameworks like NUnit, JUnit, Mockito, and Moq where IoC principles are normally used when developing the units under test, and dependencies are controlled by the instantiator of the class (not the class itself) which makes them easy to mock.

I made a small change to Lemur, that will use the native lua "require" instead of the Lemur habitat implementation, if the parameter is a string. This allows me to load mock implementations from the filesystem, rather than the Lemur habitat.

E.g.
Here is a simple "class" that I want to test. Most code has been left out for simplicity, but this module has several dependencies that I don't want to test (e.g., DataStore2)

local PlayerEntity = {}

local dependencies = require(script.Dependencies).Get()

-- Constructor
function PlayerEntity.new(player)
	local p = { 
		Player = player,
	}
	setmetatable(p, PlayerEntity)
	p.coinStore = dependencies.Datastore("coins", player)
	p.poleStore = dependencies.Datastore("poles", player)
	p.backpack = playerBackpack.new(player)
	p.animationController = dependencies.PlayerAnimationController.new(player)
	return p
end

PlayerEntity.__index = PlayerEntity
return PlayerEntity

Instead of explicitly defining dependencies, I use a script called Dependencies.lua that is a child of the PlayerEntity script.

local Dependencies = { }

local injected = nil

function Dependencies.Get()
    return injected or {
        DataStore = require (game.ServerScriptService.lib.datastore2),
        PlayerBackpack = require (script.Parent.Parent.PlayerBackpack),
        PlayerAnimationController = require(script.Parent.Parent.PlayerAnimationController)
    }
end
 
function Dependencies.Inject(newDependencies)
     injected = newDependencies
end
 
return Dependencies

This allows the unit test to override the dependencies with ad hoc tables, or scripts from a mock repository.

return function()
    local uutDependencies = require(script.Parent.Dependencies)

    local DataStoreMock = require("DataStore2Mock")
    local PlayersInGame = {}

    setmetatable(PlayersInGame, {})

    uutDependencies.Inject({
        DataStore = DataStoreMock,
        PlayerBackpack = require("PlayerBackpackMock"),
        PlayerAnimationController = require("PlayerAnimationControllerMock")
    })

    local uut = require(script.Parent)

    describe("GetUserId", function()
        it("Should return the player's UserID", function()
            local player = uut.new({ UserId = "Unit Tester 0", Name = "bob"})
            expect(player:GetUserId()).to.equal("Unit Tester 0")
        end)
    end)
end

This works because of a small change I made to Leumur.

environment.require = function(path)
	if type(path) ~= "string" then
		return habitat:require(path)
	else
		return require(path)
	end
end

This will allow the main test script to add mock repository locations to the lua require path. It seems like this would be a good feature for everybody to use (especially those of us used to basing the majority of our testing on IoC principles and mainstream testing tools), assuming that it isn't a huge departure from the vision of Lemur.

It should be possible to codegen a test harness that will tell us how close to basic compliance we have in a few different places.

It would save a lot of time if Lemur could read rojo.json files and initialize your tree for you from that. It would save a lot of duplicated effort!

This one modifies a built-in like math, which means we'll have to start shallow-copying it for each script.

Enums right now are numbers, they should be a special type, known as EnumItem, which has a property that allows you to retrieve the underlying numeric value.

This is important for improving the typechecking of our values!

To implement HttpService:JSONDecode and HTTPService:JSONEncode, we need to either bundle a Lua JSON library into Lemur's source, or depend on a new rock like dkjson for parsing.

http://robloxdev.com/api-reference/function/Instance/GetDescendants

I was going to implement this, but I don't know of a good PRNG that doesn't need bitmath, and I'm not entirely comfortable making a bit library a dependency.

Right now, if you call loadFromFs on a directory, it will load all of the directory's children into the instance that you give it.

Instead, I'd like to extend loadFromFs to work on files, and return the object that you loaded instead. That means that loadFromFs("lib") would give you an instance named lib, instead of loading all of the pieces from lib and putting them into an existing Folder.

This is a breaking change!

To get MockDataStoreService working with Lemur without monkey patching, RunService:IsServer() (at the time of writing, an unimplemented API) needs to return true. However, I'm not comfortable making RunService:IsServer() always return true on Lemur. I imagine there are many other cases like this.

Habitats should be easily customizable for arbitrary options for situations like this. In this case, there should be an easy way to tell Lemur "I want RunService:IsServer() to return true" without having to monkey patch.

This project would probably be separated from Lemur but closely related to it.

I want a layer that supports Lua 5.1 through 5.4 that provides the minimum amount of functionality needed to write a library that's compatible with Roblox and not, mostly notably the module system.

There are a couple tricks that could be used to implement this:

• Small instance emulation layer, basically to implement script and friends
• Static analysis of source to replace require calls that reference instances with their string equivalents and/or pack the entire project into a single file like Browserify or Webpack from the web/JS ecosystem.