feat: WASM support about flutter_soloud HOT 27 OPEN

@filiph

1. Yes, we don't need to write a new backend. Miniaudio acts as the audio backend for SoLoud C++ on all platforms.
2. I spent some time compiling an example provided by SoLoud and it works with miniaudio! I uploaded it here (click to start).
3. Yes, I already compiled what Dart FFI uses for binding C++. In the web dir there is a compile script that compiles sources stored in the src dir, bindings.cpp included which is the cpp source that contains all the FFI functions needed by flutter_soloud plugin.
4. Yes it is possible (I didn't try). But this option is still on the beta channel. It is supposed to compile the Flutter code to wasm and we just need to use the wasm lib from Flutter app in any way you compiled it.

I am not familiar with this kind of stuff, but my thoughts to proceed are:

1. compile src with emscripten which generates a .wasm and a .js
2. use a conditional import like @wolfenrain pointed out. So, when using the plugin on the web, we should import something like bindings_player_web.dart instead of bindings_player_ffi.dart.
3. in bindings_player_web.dart we should then use JS interop to call the .js functions generated with 1)

Problems:

• the plugin will have to provide an already compiled wasm and js. Not really a problem but they should be linked to the plugin.
• write bindings_player_web.dart code.
• since the plugin uses an isolate to manage some functionalities, I don't know if this is feasible because AFAIK, JS doesn't support threads(?), but for sure there are other ways(?).

Don't take the things I have written as certainties, because as I have already written, they are not things I am familiar with.

from flutter_soloud.

I'm afraid I don't know. I haven't yet touched WASM, and I'm afraid it'll be some time before I can make time. I'm trying to get someone from Flutter to chip in, but in the meantime I found these (that I intend to read for inspiration):

• flutter/flutter#114242
• flutter/flutter#114243
• https://github.com/sachaarbonel/flutter_web_wasm
• dart-lang/sdk#37355
• https://pub.dev/packages/wasm_interop

from flutter_soloud.

Word from the Flutter team:

in the current state of world, using dart:js_interop to interop Flutter is a viable route. In the future, there maybe new tooling to allow Dart FFI works directly on web, but it’s probably not happening anytime soon.

from flutter_soloud.

I've made some attempts to compile on Linux. In the web directory there is the compile script.
This succesfully generates the .wasm and .js in the web/build folder. I stopped here.

IMHO the next step should be to use dart:js_interop to interop Flutter with the created JS.

from flutter_soloud.

Hi,

I tried to import the generated wasm into a simple html file.

This is the error I got,

Uncaught (in promise) TypeError: WebAssembly.instantiate(): Imports argument must be present and must be an object

what I noticed is that, it changes based on the object created for importObject

const importObject = {
  imports: { imported_func: (arg) => console.log(arg) },
};
            
     WebAssembly.instantiateStreaming(
      fetch("libflutter_soloud_plugin.wasm"),
      importObject,
    ).then(
          (obj) => console.log(obj),
);

My understanding is that we want to skip the js part and use the wasm directly within the app, anyone correct me If I got it totally wrong.

I havent tried to use/import the js part without the wasm.

cant fix, the stupid formatting :(

from flutter_soloud.

My understanding is that we want to skip the js part and use the wasm directly within the app, anyone correct me If I got it totally wrong.

I think that's the best approach at this stage, yes. The whole Flutter-on-web story seems to be (slowly but surely) approaching a time when apps and games are just compiled to WASM. So if it's possible for one WASM binary to use another WASM binary as a "DLL", then that's what we want.

from flutter_soloud.

If I understand correctly, it is or it will be possible to use just the compiled WASM and use that with all the platforms? Making it possible to get rid of the FFI part? This would be wow!

from flutter_soloud.

My understanding is that we want to skip the js part and use the wasm directly within the app, anyone correct me If I got it totally wrong.

I think that's the best approach at this stage, yes. The whole Flutter-on-web story seems to be (slowly but surely) approaching a time when apps and games are just compiled to WASM. So if it's possible for one WASM binary to use another WASM binary as a "DLL", then that's what we want.

have you got, projects/repo where to "steal" the build phase. Currently it build, but I'm not sure if the errors I am having are because of me or the wasm created...

also, I'm missing something else, how the flutter/dart will interact with it ?

https://flutterweb-wasm.web.app/ in the console is

how ? :D

@filiph

from flutter_soloud.

Word from the Flutter team:

in the current state of world, using dart:js_interop to interop Flutter is a viable route. In the future, there maybe new tooling to allow Dart FFI works directly on web, but it’s probably not happening anytime soon.

Does this mean we should go for a federated plugin only for web? If so, I've tried this approach and taken some steps, but I haven't reached a good point. I could write these few steps in detail.

from flutter_soloud.

TBH, I don't know what it means. I don't think it necessarily means a need for a federated plugin?

from flutter_soloud.

TBH, I don't know what it means. I don't think it necessarily means a need for a federated plugin?

You can have both internal binding implementation (IO and web) live in the same package, as long as the correct implementation is getting imported using this:

import 'package:flutter_soloud/io_implementation.dart' 
    if (dart.library.html) 'package:flutter_soloud/web_implementation.dart';

from flutter_soloud.

A few points that I need clarified:

1. Do we need to wire Flutter SoLoud to WebAudio (e.g. by programming a backend), or is this already done by the miniaudio backend that the package uses?
2. Is there a precedent of SoLoud (C++) with miniaudio backend running on the web as a WASM binary?
3. Is it possible to compile the flutter_soloud Flutter package and its Dart FFI SoLoud (C++) dependency into one WASM binary?
4. Is it possible to compile a Flutter app as a WASM binary (apparently yes?)?
5. Is it possible to somehow "link" the Flutter app WASM binary with the flutter_soloud Flutter WASM binary? OR, is every package (even an FFI one) just automatically compiled to one WASM blob?

/cc @johnpryan @alnitak

from flutter_soloud.

By the way, there is this issue opened, they also talk about audio in the comments

dart-lang/sdk#46690

In the meantime I tried to get the js and wasm to load, where the js maps the functions inside the wasm.

The command I tried where copied from this thread jarikomppa/soloud#356 (comment)

unfortunately that didnt work

Uncaught (in promise) TypeError: WebAssembly.instantiate(): Import #0 module="env": module is not an object or function
SoLoud_welcome.js:4226 Stack overflow detected.  You can try increasing -sSTACK_SIZE (currently set to 65536)
handleException @ SoLoud_welcome.js:4226
0029d602:0xb05 Uncaught (in promise) RuntimeError: memory access out of bounds
    at 0029d602:0xb05
    at SoLoud_welcome.js:691:12
    at callMain (SoLoud_welcome.js:4608:15)
    at doRun (SoLoud_welcome.js:4658:23)
    at run (SoLoud_welcome.js:4673:5)
    at runCaller (SoLoud_welcome.js:4585:19)
    at removeRunDependency (SoLoud_welcome.js:629:7)
    at receiveInstance (SoLoud_welcome.js:821:5)
    at receiveInstantiationResult (SoLoud_welcome.js:839:5)

so point 3 that @alnitak suggested I'm not sure will work, cause the js and wasm are still linked to each other and needed to be imported in order to work ( happy to be wrong )

multi thread in the js world is webworker, see this link for a short explanation (https://medium.com/techtrument/multithreading-javascript-46156179cf9a)

Like, this example (https://blog.logrocket.com/getting-started-webassembly-flutter-web/) calls wasm from Flutter but is very basic the audio is huge in comparison

from flutter_soloud.

@superciccio I got the same error when trying to compile a standalone example. I had to add -sSTACK_SIZE=1048576 -sALLOW_MEMORY_GROWTH (maybe just the latter is needed).

The command I tried where copied from this thread ...

Did you use that because the web/compile script isn't working for you? What are the very few steps you took to start trying? I mean, is it worth creating a new project just for the web to try all this?

from flutter_soloud.

from flutter_soloud.

Maybe I could help with this.
Save the following code to main.cpp in an empty dir. I have added some comments inside. Hope it help somehow.

#include <stdio.h>

// Bypass CORS:
// For Chrome: edit shortcut or with cmd: C:\Chrome.exe --disable-web-security
// For Firefox: Open Firefox and type about:config into the URL bar.
// Search for: security.fileuri.strict_origin_policy set to false

/// Run Chrome as below and open generated main.html
/// chromium --disable-web-security --user-data-dir=~/chromeTemp --disable-site-isolation-trials

/// Build with:
/// emcc -o main.html ./main.cpp -O2 -s EXPORT_ALL=1 -sASSERTIONS -sSTACK_SIZE=1048576 -sALLOW_MEMORY_GROWTH

int main(int argc, char *argv[])
{
    printf("START\n");

    int n = 0;
    while (n < 3)
    {
        printf("loop %d\n", n);
        n++;
    }

    printf("END\n");
    return 0;
}

from flutter_soloud.

from flutter_soloud.

I think we miss each other I was talking an hello world with soloud 😅

Oh, I see ;)

Have you ever go it loaded in your browser ( wasm + is ) only ?

Yes, the example I linked before is compiled mostly in the same way as my previous "Hello World", it uses just SoLoud lib. But if you mean our flutter_soloud C++ sources, I didn't.

from flutter_soloud.

from flutter_soloud.

I created a new plugin with only web platform to see what problems we could hit.
Calling simple C functions (bound in the JS) from Dart seems to work, but things have gone horribly with some more complex C functions.

Some more info are written in the README and I have opened an issue where I explain better the problems.

from flutter_soloud.

Hi @filiph, can you check out the notes in dart-lang/sdk#46690 (comment), and then perhaps respond over there with feedback?

from flutter_soloud.

• since the plugin uses an isolate to manage some functionalities, I don't know if this is feasible because AFAIK, JS doesn't support threads(?), but for sure there are other ways(?).

Hi, I not expert in this tema but thread is posible by using Web Workers. Also supports shared workers, which allows you to synchronize across multiple tabs.

This approach by using Web Workers is implemented right now in the drift package. In his docs show how handle about.

from flutter_soloud.

+1

from flutter_soloud.

Umm not quite @yeikel16 , the answer to multithreading for audio in the browser is not Web Workers, its Audio Worklets which are specifically:

The AudioWorklet interface of the Web Audio API is used to supply custom audio processing scripts that execute in a separate thread to provide very low latency audio processing.

Audio Worklets support wasm, there are lots of examples to be found.

This is actually the path I was going on to be able to use Dart to write audio processing functions but I don't think Darts WASM support currently allows for compiling standalone functions into WASM.

from flutter_soloud.

Hi @maks, thanks for your feedback.

I have explored Audio Worklets despite my complete ignorance of JS.
Managing our own audio callback with a custom Audio Worklet, implies not using SoLoud/miniaudio C lib and therefore a separate audio engine written in JS (if I am not wrong).
But I saw that since some months, miniaudio supports Audio Worklets. When compiling to WASM, emscriten generates 2 more files: [libname].aw.js and [libname].ww.js
I suppose they are the audio buffer and the audio callback. Still, I think they have to be managed with a custom JS code.

Currently I am working on a separate testing project to add web support to flutter_soloud. I reached some results, but I am stuck mainly on having a common way for all platforms to communicate between a thread and the main Dart Isolate. More info in this issue

from flutter_soloud.

@alnitak yes sorry its taken me a while to respond here.

I had also previously looked into using miniaudio from Dart too but from the point of view of writing DSP code (ie code thats generating samples inside the usual callback function of miniaudio and other libraries like SDL, etc) but of course ran into the issue of not being able to do so due to lack of sync callbacks. My understanding of audio worklets is that they basically work the same way as other operating system audio APIs and libraries that wrap them such as miniaudio, SDL, etc, in that you provide a callback function implementation which gets called by the library/framework that needs to fill in a buffer of audio samples before it returns.

But getting back to the point at hand, I would suggest that perhaps using Soloud is not a good match for the case of browsers, because the web audio api provides much of the functionality that Soloud does. Instead it may be better to instead try to provide Solouds api built on top of web audio, though of course thats much more work than just doing a ffigen over solouds api.

Alternatively it may more sense to have a Dart package that provides a light abstraction over flutter_soloud and then it can make use of the existing new web package to call into the browser webaudio apis to implement that abstraction on the web platform.

from flutter_soloud.

@maks agree with you: on the web the Audio Worklet is mandatory if you want to implement a DSP.

Maybe I am wrong, but by compiling SoLoud/miniaudio with emscripten to WASM and JS, your web app will use Web Audio. If this is true the SoLoud/miniaudio audio samples callback is used on the web (and maybe can be replaced by your custom DSP code instead of writing a custom audio worklet in JS).

Also if SoLoud/miniaudio uses Web Audio, then the compiled lib will have all the SoLoud functionalities it provides.

In short, the testing project works on the web. But I am having some problems like sending events from C to Dart (ie when a sound ends). I have seen the new NativeCallable but it's not compatible with web since it uses dart:ffi and my wish is/was to use one code-base for all platforms.

Alternatively it may more sense to have a Dart package that provides a light abstraction over flutter_soloud and then it can make use of the existing new web package to call into the browser webaudio apis to implement that abstraction on the web platform.

Agree with this, IMHO should be the right direction now.

from flutter_soloud.