Publish documentation of command line android build of SDK and Samples about diligentengine HOT 51 OPEN

I am wondering why Android is such a headache. Other 5 platforms were very straightforward to set up.

from diligentengine.

Android build system is very poorly set up at the moment, but unfortunately we now have no bandwidth to improve it and any help would be greatly appreciated.

from diligentengine.

Command-line build will also enable running Android build on CI servers as right now it is the only platform that is skipped by CI

from diligentengine.

Ok, so yeah, that's something I'd like to make sure we can automate. I feel like Android Studio makes some things easier (recommending fixes for missing components), but when it fails, it can be puzzling. In most cases one can get better information from the command line build with gradle. I'll see what I find.

from diligentengine.

Will look into this over the weekend.

from diligentengine.

First problem I ran into was related to Error:Could not find com.android.support.constraint:constraint-layout:1.0.1 required by AntTweakBar. I did a search on this in the mvn repositories (it is a support library), and it is not in mvnrepository, oddly, so is probably packaged by the Android Studio version, but not determined what version. Lots of people have hit this problem on other projects. The best thing to do is change all versions of this support lib from 1.0.1 to 1.1.0, as that is in the mvnrepository.

from diligentengine.

I'll propose a fix for the above, on to the next issue: Task :Common:generateJsonModelDebug fails with a cmake error (probably NDK related). Need to investigate.

from diligentengine.

Error:Could not find com.android.support.constraint:constraint-layout:1.0.1

Where is it specified? Can't find it in Gradle files or in the project generally. BTW, do you have the latest code? I recently made few changes/fixes on Android

from diligentengine.

Oh, let me pull the latest. All of these constraint things are in the samples:

./AntTweakBar/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.1.0'
./Atmosphere/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial01_HelloTriangle/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial02_Cube/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial03_Texturing/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial04_Instancing/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial05_TextureArray/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial06_Multithreading/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial07_GeometryShader/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial08_Tessellation/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial09_Quads/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial10_DataStreaming/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'
./Tutorial11_ResourceUpdates/build.gradle:    implementation 'com.android.support.constraint:constraint-layout:1.0.1'

from diligentengine.

Yeah, I double checked the history and these were indeed removed recently
DiligentGraphics/DiligentSamples@847e1c3

from diligentengine.

Ok, that was a good update, fixed that last problem. Hit a new one:

Execution failed for task ':AntTweakBar:processDebugResources'.
> Android resource linking failed
/.gradle/caches/transforms-2/files-2.1/b229f0a2317aecc488ca7e6d10403f98/res/values-v28/values-v28.xml:9:5-12:13: AAPT: error: resource android:attr/dialogCornerRadius not found.

/dev/repos/DiligentEngine/DiligentSamples/Android/build/AntTweakBar/intermediates/incremental/mergeDebugResources/merged.dir/values-v28/values-v28.xml:11: AAPT: error: resource android:attr/dialogCornerRadius not found.

/.gradle/caches/transforms-2/files-2.1/afd4068bb0a5baef6d45864dae65a31e/res/values/values.xml:89:5-125:25: AAPT: error: resource android:attr/fontVariationSettings not found.

/.gradle/caches/transforms-2/files-2.1/afd4068bb0a5baef6d45864dae65a31e/res/values/values.xml:89:5-125:25: AAPT: error: resource android:attr/ttcIndex not found.

Probably something I'm missing in my support libraries or SDK version. Looking into that.

from diligentengine.

I'll revisit this later in the week (SDK updating right now), and I have to jump onto something else.

from diligentengine.

Thanks for looking into this.
TBH, I have no idea what these errors mean. Looks like none of these attributes are ever specified by any of build files...
Also, sometimes incremental build does not handle the file changes properly, so it may be useful to delete all intermediate files.

from diligentengine.

As an aside, since I am only building with gradlew + a standalone Android SDK, I have to go through this trial and error. The Android Studio will usually suggest/recommend fixes that will often work. However, I have to sort out the meaning of the errors. There are a lot of moving parts (platform/os, SDK tool version, support libs, ndk tool version, cmake version, and probably a few other things I forgot). I'll keep digging. Those errors are probably some library version or tools version that I am missing.

from diligentengine.

You are right. Android studio indeed suggests installing a number of components. Maybe if you install it and open the project, you will know what components are missing.

from diligentengine.

So as far as this error:

Execution failed for task ':AntTweakBar:processDebugResources'.
> Android resource linking failed
/.gradle/caches/transforms-2/files-2.1/b229f0a2317aecc488ca7e6d10403f98/res/values-v28/values-v28.xml:9:5-12:13: AAPT: error: resource android:attr/dialogCornerRadius not found.

/dev/repos/DiligentEngine/DiligentSamples/Android/build/AntTweakBar/intermediates/incremental/mergeDebugResources/merged.dir/values-v28/values-v28.xml:11: AAPT: error: resource android:attr/dialogCornerRadius not found.

/.gradle/caches/transforms-2/files-2.1/afd4068bb0a5baef6d45864dae65a31e/res/values/values.xml:89:5-125:25: AAPT: error: resource android:attr/fontVariationSettings not found.

/.gradle/caches/transforms-2/files-2.1/afd4068bb0a5baef6d45864dae65a31e/res/values/values.xml:89:5-125:25: AAPT: error: resource android:attr/ttcIndex not found.

it appears that this can be fixed by adding:

compileSdkVersion 28

to the build.gradle in the offending project AntTweakBar.

from diligentengine.

I'm getting another error after fixing the last one in the cmake section of building AntTweakBar

    gnustl_static is no longer supported.  Please switch to either c++_shared
    or c++_static.  See
    https://developer.android.com/ndk/guides/cpp-support.html for more
    information.

from diligentengine.

So this above issue is a problem with the changes to the NDK. @DiligentGraphics unfortunately for maintainers, SDK tools, OS versions, and support tools provided by the Android SDK toolchains don't stand still, and stuff that might build perfectly in your configuration might not on another system due to the variety of differences that might exist. It is sometims necessary to save off versions of tools in case things aren't being archived by Google (I think for the most part they do archive things, but they've moved away from the standalone SDK tools to Android Studio, which pushes the burden of maintaining Studio also. The ideal strategy that I use is to lock down configuration and make sure that I document what I'm using, such that anyone else picking up my project will be able to use the same versions and be guaranteed success in the build. Use something different and the guarantees are completely thrown off. I think that's where this is at, unless you are telling me that you've got other people building on other systems for you to validate and that maybe it's my configuration that's crazy.

In any case, I'll keep plugging away here until I nail down a configuration that works. My team is sufficiently motivated to get the samples working and making builds reproducible.

from diligentengine.

So the above C++ / cmake issue mentioned is down in the guts of the NDK. Probably with NDK 19 or sometime earlier, they no longer support gnustl_static. To fix this requires a change to:

Common/NativeApp/Android/build.gradle

and change the line for the cmake and just don't specify the STL

externalNativeBuild {
            cmake {
                arguments '-DANDROID_PLATFORM=android-21',
                        '-DANDROID_TOOLCHAIN=clang', '-DENABLE_TESTS=TRUE'
            }
        }

That fixes the other bug I hit.

from diligentengine.

In the midst of building [19/65] Linking CXX static library DiligentTools/TextureLoader/libTextureLoader.a

I get about 20 errors, starting with:

DiligentEngine/DiligentSamples/External/TwBarLib/src/TwGraphImpl.cpp:135:32: error: reference to type 'const Diligent::TextureData' could not bind to an rvalue of type 'Diligent::TextureData *'
      _Dev->CreateTexture( desc, &TexData, _Tex );
                                 ^~~~~~~~

from diligentengine.

I'm going to try on an older setup with an older sdk / ndk. These problems maybe are due to the change away from the gnustl-static.

from diligentengine.

Do you get this error after removing -DANDROID_STL=c++_static? The error seems to be a compiler error, which shouldn't be related to c++ runtime library controlled by DANDROID_STL.

from diligentengine.

In Android Studio, the build succeeds with the following build settings:

cmake {
    arguments '-DANDROID_TOOLCHAIN=clang', '-DENABLE_TESTS=TRUE'
}

I can't understand why it causes so many errors in a standalone toolchain.

from diligentengine.

BTW, are you able to build the teapot sample from android ndk samples (which is where I copied the settings from)?

from diligentengine.

So not sure as far as why the standalone toolchain is having an issue, but as I said, I'm on Mac OS X, exactly same errors with Android Studio as with building from the command line. I'm using the same toolchain, just that Android Studio manages the tools that get installed. My belief is these Android build issues are 95% of the time configuration issues, not necessarily a bug in the settings per se or code issues. In other words, something in my environment, tools versions, or other binary that might be in my path that doesn't match the settings or api version.

from diligentengine.

In terms of the gnustl static issue, described in this error message below, that's just an NDK version issue. If you are still on an older NDK that won't affect your build. So the first question is, what NDK version do you have installed? I'll separately confirm that the teapot builds.

gnustl_static is no longer supported. Please switch to either c++_shared
or c++_static. See
https://developer.android.com/ndk/guides/cpp-support.html for more
information.

from diligentengine.

gnustl_static is no longer supported. Please switch to either c++_shared
or c++_static. See
https://developer.android.com/ndk/guides/cpp-support.html for more
information.

I do use c++_static, but you said it generates an error. I'm little confused now.

from diligentengine.

@DiligentGraphics so if you don't mind, can you state which version the Android NDK you are using?

A second point, that I forgot was that this project has submodules, and to keep everything working together, I have to remember to do the git submodule update step.

I'm setting up a windows system with a pure command line set up (Google has un-hidden the Android Command Line SDK tools). I'll keep the mac with the Android studio.

About to retest

from diligentengine.

So after resetting my workspace back to a clean state and doing a submodule update --recursive, I get an error I've seen before on other projects.

Ran a ./gradlew after setting the gradlew to be executable.

* What went wrong:
A problem occurred configuring project ':AntTweakBar'.
> No toolchains found in the NDK toolchains folder for ABI with prefix: mips64el-linux-android

So on a non-linux platform, we aren't going to have mips64el-linux-android cross compiler under the toolchains. That affects windows and mac. The solutions are numerous, but I'll refer to the Google Filament issue as somewhat authoratative:

google/filament#15

The second to last solution worked for me.

Continuing to build.

from diligentengine.

I currently have NDK 19.1

I can try to update it to 20.0
I can successfully build everything on Mac and Linux with Android Studio 3.4.1. It used to work on Windows too, but it was unusably slow and I haven't tried it for a while.

from diligentengine.

running: gradlew build

hits the same problem seen in comment: #42 (comment)

Before I repeat the entire sequence of errors that I found previously, what tag or branch should I be building from, if not tag v2.4.b?

from diligentengine.

The head right now is in a good shape and everything should work fine. v2.4.b is quite old.
So, just want to make sure, you can't even build the head in Android Studio?

from diligentengine.

Let me switch to do that. I was going off the instructions on the project https://github.com/DiligentGraphics/DiligentEngine#cloning-the-repository

I'll switch to HEAD if that's working. Yes, it looks like that tag is from way back in Jan or Feb.

from diligentengine.

Switching to head. Numerous problems gone away.

from diligentengine.

The above picture of the Android "config" might be enough to help support documentation of the Android build. I'd suggest embedding that into the Android section of the docs.

from diligentengine.

Numerous problems gone away.

So did you manage to build it or some issues still remain?

The above picture of the Android "config" might be enough to help support documentation of the Android build. I'd suggest embedding that into the Android section of the docs.

I can add Android NDK and build tools version to the section.

from diligentengine.

It's building now from HEAD, not from the documented tag. I'd suggest that the Android section should recommend a particular tag that works for android, or tell people to gamble on the HEAD. And when I say it's building now, I think it will build overnight. It's been cranking for an hour, so I think it will finish, but it's only at 8%. Blame gradle and an artificially slow mac.

from diligentengine.

Also confirmed I can build teapot. So this was all completely because I followed the instructions (the problems found). In any case, I'll document the gradlew commands that people can use, and this should allow you to cut over to a CI server type environment for builds and maybe testing.

from diligentengine.

As far as making this build as an included library or something in an external project, I think that's a different effort. If my team is going to use this, I have to figure that part out too. I want to be able to distribute the code we are working on without having to also include the SDK in our repository. I'll try to figure that out on a different issue once I see this running on a phone or whatever.

from diligentengine.

Darn ... I spoke to soon, build fails during linting:

> Task :AntTweakBar:lint
Calling mockable JAR artifact transform to create file: /Users/dkords/.gradle/caches/transforms-2/files-2.1/df2e2769b2482aa61510ddb3163c3052/android.jar with input /Users/dkords/Library/Android/sdk/platforms/android-28/android.jar
Ran lint on variant release: 3 issues found
Ran lint on variant debug: 3 issues found
Wrote HTML report to file:///DiligentEngine/DiligentSamples/Android/build/AntTweakBar/reports/lint-results.html
Wrote XML report to file:///DiligentEngine/DiligentSamples/Android/build/AntTweakBar/reports/lint-results.xml

> Task :Atmosphere:lint
Ran lint on variant debug: 3 issues found
Ran lint on variant release: 3 issues found
Wrote HTML report to file:///DiligentEngine/DiligentSamples/Android/build/Atmosphere/reports/lint-results.html
Wrote XML report to file:///DiligentEngine/DiligentSamples/Android/build/Atmosphere/reports/lint-results.xml

> Task :Common:lint
Ran lint on variant release: 6 issues found
Ran lint on variant debug: 6 issues found
Wrote HTML report to file:///DiligentEngine/Common/NativeApp/Android/build/reports/lint-results.html
Wrote XML report to file:///DiligentEngine/Common/NativeApp/Android/build/reports/lint-results.xml

> Task :Common:lint FAILED

FAILURE: Build failed with an except

from diligentengine.

I think I can force it to skip linting. I'm assuming those are things that don't matter. It claims to have found an error:

Some methods, such as View#onDetachedFromWindow, require that you also call the super implementation as part of your method.
To suppress this error, use the issue id "MissingSuperCall" as explained in the Suppressing Warnings and Errors section.

../../src/main/java/com/diligentengine/android/common/DiligentApplicationBase.java:27: Overriding method should call super.onCreate
 24 import android.widget.Toast;
 25 
 26 public class DiligentApplicationBase extends Application {
 27     public void onCreate(){                                                                         
 28         Log.w("native-activity", "onCreate");
 29 
 30         final PackageManager pm = getApplicationContext().getPackageManager();

from diligentengine.

I'll let this run again, not sure but I may have to suppress the lint warnings within each occurrence.

from diligentengine.

Will check in on this tomorrow.

from diligentengine.

Probably the Android Studio has some built in settings to ignore these warnings or errors.

from diligentengine.

I updated all tools to the most recent versions and everything builds fine (though there are some clang warnings).

I don't have any lint errors. Did you configure lint separately?

from diligentengine.

No, I don't explicitly ask for a lint , however, the default ./gradlew build

will end up running a lint target. I am able to disable it with:

./gradlew build -x lint

I think the effort is different here. I'm building from the command line with gradle. The configurations will be different as Android studio has defaults that it may enforce, as well as user defaults per project. When you finally hook up your project to a nightly CI build, you will probably have times where the CI will fail but you will be looking at Android Studio and you will think "works for me".

So with the lint ignored, the build completes. I will see about the commands to install on device, and just verify these final steps of actually getting it running on a real device.

from diligentengine.

I've been dealing with it since 2007. The tools are not bad, and I can't complain that they are free. However, sometime back Google decided Gradle was better, because ... ? I guess because that's what Jetbrains/Android Studio was built on as a build tool, and this caused years of difficulty, probably 8 years to get the tools to the point where it just worked pretty well, but native NDK integration was still a challenge. Android had traditionally used these Android.mk files which is sort of a specialized GNU Makefile. It was goofy but it worked. Google had a few years where it seemed like they didn't know how to deal with the native integrations via gradle. But then they moved to the cmake plugin for gradle. That's been a mixed bag, but I think mostly good long term. As for the weird errors that are on the java side, that's mostly compatibility stuff and api issues. Google releases a lot of new code every year. It's a mess to keep up with.

I recommend maintaining as low an api compatibility support level as possible, as it will ensure the code runs on the most devices. Avoid the temptation to lock on to new Android features. Android users typically aren't able to upgrade to the latest Android without buying a new device, so many will be stuck on an old OS for 2-3 years, or the duration of owning a particular device.

from diligentengine.

Yeah, I originally used NDK build, and it wasn't always very easy. Besides, the project was much smaller and only supported couple of platforms. The fact that they made CMake their native build system simplified the problem dramatically. At least with the native part I have very few problems and when I do they are most of the time very clear and easy to fix. It is the Gradle part that I dont fully understand and that causes all wierd issues.

I'm targeting API level 21 I guess which should be pretty ok

from diligentengine.

Yeah, I guess the point is with Android, unlike with iOS, Google doesn't have a good scheme to push people forward onto the latest OS. That's really up to carriers and or 1st tier phone manufacturers to determine if they want the newer OSs. And in large part, people stay on older OSs for a long time. Great that you guys are sticking with API 21 actually.

from diligentengine.

do you still need help? if so i think i could help a lot with getting this to build on android :)

from diligentengine.

We now have the command-line build set up. The build runs on every commit as part of CI
https://github.com/DiligentGraphics/DiligentEngine/actions/workflows/android.yml

from diligentengine.