context: atsushieno/aap-core#34

We seek for possible UI integration for AAP, and for aap-juce it used to be planned to build wasm-based UI using juce_emscripten. It was then abandoned because Android Chrome did not support Atomics (and SharedArrayBuffer). They are now on Android Chrome (needs some feature flags enabled) so it is technically possible again.

I have updated my own fork of juce_emscripten at https://github.com/atsushieno/JUCE/tree/juce_emscripten_607 so that it may be usable. There is no CMake integration yet though.

It is only an initial draft but the basic out-process UI integration strategy is outline at https://gist.github.com/atsushieno/eb8155cbde052ded330ff9667b51e937 . To make it happen on aap-juce, there will have to be some injection wherever AudioProcessorEditor accesses AudioProcessor. It is likely that we have to patch plugin apps code to have intermediate AudioProcessor that taps into call to AudioProcessor API uses by the editor, if that is possible and sufficient. Needs some experiment.

Currently there are couple of plugins that builds multiple plugin artifacts (and that I am aware of):

• ChowPhaser (Mono/Stereo)
• ADLplug-AE (ADL/OPN)
• DDSP-VST (Effect/Synth)

So far aap-juce build system only brings in one plugin, but that is not ideal. We should be able to build and package multiple plugins within an apk.

Currently not successfully imported. See atsushieno/aap-juce-os251.

(copied from atsushieno/aap-core#38)

There is most likely some intensive operation that prevents realtime processing. Needs some investigation.

One possible improvement is that other plugin API wrappers are based on AudioPlayHead (e.g. VST3Wrapper and LV2Wrapper) while our current implementation is quite simple and minimum.

both aap-juce-plugin-host and aap-juce-helio seems to freeze when it tries to instantiate a plugin. It was not like that a few months ago, so it must be fairly recent regression.

The biggest issue on diagnosing issues in JUCE based plugins is that it is impossible to debug with Android Studio for some reason. Some reason actually is, debugging plugin requires apps kept alive while debugging the service part but it is somehow not possible.

(1) Usually when I debug LV2-based plugins, I launch standalone app activity, then go back to home screen (or app list) to launch a host to connect to the plugin service. The service is still alive and I can simply add android.os.Debug.waitForDebugger() in MainActivity with no problem.
(2) But the Main Activity (JuceActivity) from JUCE-based plugin apps, if I go back to home screen or app list, it terminates the application and debugger stops. Even after I modify app AndroidManifest.xml and make the Activity launchMode='standard'.
(3) If I create another launch configuration that does not launch main-launcher Activity and add android.os.Debug.waitForDebugger() at JuceApp.onCreate(), the debugger still often does not break and I have no idea why.

It may depend on call to Java.initialiseJUCE (this) in JuceApp but we need some solid debugging experience.

taking over #14

JUCE AudioPluginHost is still weird. OB-Xd always crashes. Any other synths generate sound, but their key is weird. Also it frequently crashes.

It is not limited to JUCE-based synths. mda-lv2 synths are also suffered.

former issue: #1

Now that we have migrated to juce6, it is looking possible to switch to CMake-based build solution, which is likely better than dealing with Projucer.

There is actually a couple of minor concerns:

• CMake requirements: it requires CMake 3.15, which means we need additional download steps on CI servers as well as local.properties hack to add CMake directory settings.
• JUCE's CMake support is not like, Projucer can generate a natural CMakeLists.txt outcome. It has to be manually written.

On a side note, the Android Studio issue on taking infinite time on indexing is actually irrelevant to Projucer oldness. It will happen even with modern CMake-based solution.

Since the enhancement is not really big, we'd leave it as an enhancement issue.

Currently my local build shows it's using 12GB space. GitHub Actions offers only 500MB for Free and 1GB for Pro. Even AudioPluginHost costs 1GB-ish and Andes seems to cost 1.5GB, they have to be significantly reduced, or we dump GitHub Actions again.

I'm logging some failed attempts to port more JUCE plugins here (I have the same topic on aap-lv2 issues).

Currently if the target plugin product name contains a whitespace (e.g. "Surge XT", or "DDSP Effect") metadata generator fails to handle it. It is because our hack is ugly and resort to some builder shell script (generate-metadata.sh). It has to be cross-platform across Unix-y platforms and that brings in headache.

It should not happen if we simply migrate to full CMake world.

It feels like there is some blank sampling bytes within the audio processing block.

When we had that kind of issue last time, it was due to missing channel (like, only L was processed while something expected both L and R).

In this code, it tries to let JavaVM attach to current thread.

https://github.com/atsushieno/aap-juce/blob/main/modules/juceaap_audio_plugin_client/juceaap_AAPWrappers.h#L41

But this constructor is actually called by the plugin service binder, which means the caller can be an audio thread.

Although if we simply remove this #if ANDROID part, the exception below happens:

java.lang.RuntimeException: Can't create handler inside thread Thread[Binder:5724_4,5,main] that has not called Looper.prepare()
        at void android.os.Handler.<init>(android.os.Handler$Callback, boolean) (Handler.java:227)
        at void android.os.Handler.<init>() (Handler.java:129)

The code sometimes causes run-time crash for unknown reason.

We will have to determine what to do here.

Currently we kinda have to run AAP metadata generator because otherwise it is annoying to write everything manually. But nowadays AAP supports dynamic parameter lists at runtime, so it would make more sense to avoid static parameter list. They do not really contribute to plugin listup anyways.

It will practically result in eliminating need for the metadata generator, which has been PITA.

Currently our aap-juce presets scanning involves loading of entire preset data to just get its name. Preset scanning does not happen often, but it takes too long time on some plugins. The newest example now is aap-juce-byod - it involves heavy-weight ML operations where each of the preset takes seconds to load. It is not really desirable.

While JUCE AudioPluginHost generates some audio outputs for Andes, it only generates noises on OBXd and Magical8bitPlug2. And no synth generates valid sound on aaphostsample.

Since MDA pianos generate valid outputs on both hosts it is specific to JUCE based instruments. Weird is that Andes still generates audible sound on aaphostsample...

It is confirmed on juce6 branch.

Currently, aap-juce plugin ports launch JuceActivity, which is the main entrypoint of the app. But they are in general useless because they are designed to run as a plugin UI (we are seeing Standalone build of the app). It is not what we intend to show.

What we should do instead is to show the "plugin settings and playground UI" aka. PluginListActivity in androidaudioplugin-ui-compose. Then it can present "Show UI" command to switch to it (or show overlay Window if possible).

Bonus if it can control the plugin instance in use (but needs connection management).

One of the reasons why we created this repo is to ease unifying multiple projects, and makes it easy to enable Oboe in JUCE for Android builds.

It took like an entire night to finish indexing at "/home" (which actually is ~/, not /home) which is stupid. The problem is that unless AS finishes that "indexing" step, it does not load device list and no further exec/debug operation cannot be performed.

AAP itself has no problem. Projucer-generated default GUI App has no problem either. Something between them and this is causing the issue.

We run aap-metadata-generator to generate aap_metadata.xml from AudioProcessor implementation class. To achieve that, we first build Linux or Darwin version of the plugin, then link it to the extractor console tool.

That takes long time before starting the actual Android project build, while it is only to generate the metadata. And the metadata is actually unchanged unless code changes happen. It is good to run the metadata generator whenever code update occurs, but it actually rarely happens.

Thus, it is probably better to split out metadata generation task from the ordinary builds.

While creating aap-juce-audible-planets port, I noticed that there would be common build task to copy some local contents as assets to extract under app-specific file at the initial run time. AudiblePlanets requires its presets repo contents copied into certain local data directory.

aap-lv2-sfizz does first-run setup for sfz too. We should provide some common build targets to achieve that.

It has some difficulty. It's project .jucer files are scattered around, and neither plugin nor standalone even builds without tricky tweaks. JUCE modules are embedded, without Projucer, and we're not sure if they contain any changes to the original. And now I have an Android app project that builds, which does not show anything (due to missing juce_CreateApplication()).

It is super annoying, but when we launch aap-juce-simple-host and aap-juce-plugin-host, it tries to access ALL midi devices including MidiDeviceServices and actually opens some MIDI devices. It does not make sense. It tries to open my fluidsynth-midi-service-j which is MIDI OUTPUT device. It happens due to the infamous JUCE on Android bug that it mis-recognizes MIDI output devices as MIDI input devices.

That feature should be completely scrapped within JUCE itself. Therefore we will need another patch to fix it.

The same goes for JUCE plugins because we use Standalone apps, but fortunately there is #if JUCE_DONT_AUTO_OPEN_MIDI_DEVICES_ON_MOBILE conditional check in the JUCE sources, so we could individually apply that. We already disabled that auto open garbage in projuce-app.sh so projucer-based apps won't open it. We still have to apply this individually in CMake based projects though.

Currently OB-Xd is saved as OB_Xd on Linux and it's causing build failures on Mac. It used to be the other way i.e. only Mac build passed.

2020-11-15T08:21:20.0293600Z xcodebuild: error: 'OB_Xd.xcodeproj' does not exist.

See full build log at https://github.com/atsushieno/aap-juce/runs/1401869604

While generating aap_metadata.xml for aap-juce-simple-reverb, I noticed that aap-metadata-generator fails to retrieve the default values from its parameters. They are mostly 0.5f but the generated xml does not contain any pp:default attribute.

This in general applies to the following components, this time:

• aap-juce to the latest
• AAP to 0.6.20
• JUCE to 6.1.2
• AGP to 7.2.0-alpha04 (along with gradle 7.3-rc-1)

Progress:

• aap-juce-plugin-host
• aap-juce-helio
• aap-juce-obxd
• aap-juce-dexed
• aap-juce-odin2
• aap-juce-hera
• aap-juce-ports
• aap-juce-frequalizer -> still not usable due to atsushieno/aap-juce-witte-eq#4 (common JUCE effect plugin issue)
• aap-juce-witte-eq
• aap-juce-chow-phaser -> still not usable (ditto, but the problem surface looks different)
• aap-juce-augene -> skip it, too old

SARAH and Magical8bitPlug2 generate no sound, while andes and mda EPiano does generate valid output. Tested on AudioPluginHost and aaphostsample.

As part of UI integration scenario, wasm builds are part of the big picture in AAP to support in-process editor UI. atsushieno/aap-core#34

We need those builds automated within this repo. A big issue here is however, Emscripten builds are not fully generatable via Projucer yet. We either commit Builds/Emscripten, or enhance Projucer to add support for juce_emscripten (which I kind of avoid, because it's going to make merging upstream harder).

This is what Magical8bitPlug2 shows when it is being loaded and used by aaphostsample:

I have no idea why, but those juceaap modules have flipped module names. I thought I fixed them once, but apparently not.

To avoid confusion, we would rename those modules to the following:

• juceaap_audio_processors => aap_audio_plugin_client
• juceaap_audio_plugin_client => aap_audio_processors

This will involve a lot of changes everywhere in aap-juce-xxx repos.

From #10 (comment):

Now Dexed crashes here:

9935-9958/com.digitalsuburban.dexed I/JUCE: JUCE Assertion failure in juce_AudioSampleBuffer.h:275

(snip)

It is most likely related to atsushieno/aap-core#44 i.e. inconsistency between list of "defined" ports vs. the actual JUCE channels. While supporting dynamic ports with static known ports is doable, it must be properly distinguished at implementation.

There is a JUCE on Android bug that crashes those juce::DocumentWindow variations on either AudioPluginHost or aap-juce-simple-host:

• GenericAudioProcessorEditor
• preset selector

They are problematic but likely irrelevant to AAP codebase.

The new OPNplug port results in aap_metadata.xml with a bogus content:

       <port name="Volume model" direction="input" default="nan" minimum="0.0"
            maximum="0.0" content="other"/>

nan is not a valid value here.

This triggered an AAP core problem too: atsushieno/aap-core#52

List of plugins that are known to crash at startup. They might be due to the same reason.

• dexed
• SARAH
• Magical8bitPlug2

We used to plan the GUI support as Web UI (#22). As of AAP 0.8.0, we have in-process plugin UI support, so we do not necessarily have to take that approach anymore. If we could reuse AudioProcessorEditor within the native UI context, that would be probably easier.

Having said that, JUCE on Android does not provide a way to show plugin UI. It only supports Standalone apps, and that means it expects its own JuceActivity. We need much "less integrated" JUCE UI, ideally running just as an android.view.View.

It would be nice if we can just make only a few lines of changes to the existing CMakeLists.txt.

For example, clap-juce-extensions makes it to build CLAP version (from its README):

add_subdirectory(libs/clap-juce-extensions EXCLUDE_FROM_ALL) 
clap_juce_extensions_plugin(TARGET my-target
        CLAP_ID "com.my-cool-plugs.my-target"
        CLAP_FEATURES instrument "virtual analog" gritty basses leads pads)

We could probably do the same.

(inherited from atsushieno/aap-core#11)

tracktion_engine builds on Android, and it is most likely that it can handle audio plugin related bits in JUCE. Since we have juceaap_audio_processor, technically tracktion_engine should be already usable for AAP.

augene project can be used as a reference music player. It is very simple auduo plugin based music player based on tracktion_engine.

Problem: even if we get this working, there is no "instrumental" sample yet.

apart from #6 I wanted to have a working DAW, which is like an ultimate goal towards 1.0 release.

Helio is based on JUCE, and it supports Android, so I ended up to add AAP support on it and created a repo as aap-juce-helio. At this state, helio can list up AAPs, and it seems able to instantiate them. Though it is still not "working", especially in that:

• the plugin listing is weird. It is rather a helio issue (text too big, without wrapping)
• no audio output port is recognized, so it is simply impossible to connect to audio out and emit any sound.

I would visit this issue on occasion.

After the successful aap-lv2 repository split, this comes next.

rationale

We should make aap-juce module widely usable, not just by forking and/or adding more projects to this repo. This repo is getting fat and now it takes 3 hours to build. Each project takes 30~ minutes.

JUCE porting builds are almost identical in each project, but they are time consuming, and there are still minor glitches (such as OB-Xd renaming project files for Mac-only due to Xcode naming mismatch).

Also, there are CMake based projects nowadays, which is not compatible with jucer-based solution (such as Tracktion Engine). I'm in the other process making CMake support working in standard Android Studio project which incorporates CMake, not by hacky and immature project structure generated by Projucer.

Migration steps

The first step is to split out samples from this repo. It is being done at aap-juce-ports repo, which so far contains whatever this repo should still provide (modules, tools, etc.) as hack, and existing sample project support files (Makefile and patches).

Once it goes green, we remove references to those samples from this repo. Maybe even AudioPluginHost: it is essential, but it takes 30-ish minutes to build it, which is extraneous for plugin projects that will submodule this repo.

Then I can look to CMake migration and possible build time optimization. Maybe after further repository split (to ease CI build caching).

It seems aap-juce-simple-reverb does not seem to receive parameter changes on aaphostsample (and probably AudioPluginHost, cannot verify it as it's too unstable and always crashes before I connect some instrument and SimpleReverb, play some notes, change parameters and play some notes again).

The current state of aap-juce support is a big hack and not practically ready for external developers. We have to restructure the entire repo to ease porting.

• JUCE-AAP modules: copy AAP header files directly under the module sources so that any subsequent huge checkout becomes unnecessary.
• distribute androidaudioplugin.aar and prebuilt native dependencies via Maven so that at least they can be simply added as a Maven dependencies, along with those header files above. (It is rather an issue in android-audio-plugin-framework.)

Emscripten support is another issue that needs a lot more effort, so it can be skipped for now.

Things have been improved from "build manually copied sources with modification" to "copy from submodule, apply tiny patch, alter .jucer and build" now. As a next step, if the .jucer files can be automatically converted from original sources, that would be great.

Most of the changes are already described on README.md, and it wouldn't be too hard to create some sort of converter script that manipulates the .jucer XML files.

This is what's going on with aap-juce-odin2 from 4888e7f. It should not have audio input ports.