Async functions from different modules treated as normal coroutines about mlua HOT 4 OPEN

I tried implementing this (change poll_pending to use a set value instead of an address) locally this morning, but that didn't fully solve the issue.
With the change, the poll_pending value from an async function defined in a different library is recognized, but the function never completes.

I added many print statements throughout the relevant mlua functions and Delay (from the tokio tutorial) to figure out what's wrong and it seems that the waker does not get set correctly in this scenario.

So when the Future impls for async functions defined in a different library call wake, they're doing so on the default noop waker.

For async functions defined in the same library, the waker is set and used correctly.

The WakerGuard does set the waker, but at some point in the process, the waker is lost.
It seems like it's somewhere between calling resume_inner on the Thread and calling get_poll in the lua chunk created by create_async_callback.

I'm not very familiar with mlua internals/the lua c api, but it seems like maybe coroutines have their own lua sub-state and ExtraData.
So I'm thinking that the waker isn't getting copied into the sub-state, but I can't figure out why that'd be the case for async functions defined in another library, but not ones defined in the same library.

from mlua.

After digging through the mlua docs and code, I believe I've tracked the issue down to the fact that the address of ASYNC_POLL_PENDING is used for the Lua::poll_pending value rather than a constant value.
This means every binary/library has a different value for poll_pending.

Yeah, this is by design. It's exactly how it should be.
Modules and the main app does not share internals, everything they have on Rust side lives in their own state, isolated and unshared.
This also applies not only to futures but to registered userdata objects and so on.

Is there any specific reason (such as possible security concerns) that the address of ASYNC_POLL_PENDING is used for poll_pending rather than a constant value?

Rust does not provide stable ABI, the only API we can rely on is Lua C API which is stable and consitent.
Modules and the main app can be compiled by different compiler version, they can use different libraries (such as tokio). Even if libraries are exactly the same there is no guarantee that structs layouts are not reordered by llvm optimizer just because it decided to apply some optimization for this specific code.

For async functions defined in the same library, the waker is set and used correctly.

The WakerGuard does set the waker, but at some point in the process, the waker is lost.
It seems like it's somewhere between calling resume_inner on the Thread and calling get_poll in the lua chunk created by create_async_callback.

For the same reason as I explained above. WakerGuard is an internal entity and cannot be shared between modules/app.

I'm not very familiar with mlua internals/the lua c api, but it seems like maybe coroutines have their own lua sub-state and ExtraData.
So I'm thinking that the waker isn't getting copied into the sub-state, but I can't figure out why that'd be the case for async functions defined in another library, but not ones defined in the same library.

Each module and the main app have their own independent state, the ExtraData object. Waker object for polling futures lives inside that non-shared state and this is why it's visible inside own library but not externally.

from mlua.

The best way to integrate async functionality between loadable modules and the main app is to integrate independent event loops together using stable primitives.

The idea is to poll coroutines normally through Lua API but recognize their ASYNC_POLL_PENDING state and pass/receive oneshot channels to share futures state. When remote future is ready you should receive a message through channel and only then continue polling coroutine.
What use as a channel is up to you, it can be pipes or tokio channels that can operate in mixed sync/async module (sync on foreign side and async on native side).

The same principle is used to integrate async rust and async c++, async Lua apps (such as openresty OR neovim) and async rust modules.

from mlua.

Integrating futures through FFI is not easy, you can take a look to https://docs.rs/async-ffi/latest/async_ffi/ for some other solutions outside of Lua world (just to understand scope of the problem!)

from mlua.