Monday, 22 October 2018

How do I parallelize externalNativeBuild for different ABIs?

Background

I'm currently working on an Android project which contains a fair amount of native code. The native code is built for multiple ABIs. For various reasons, the native code has up until now been built using a custom Gradle task that invokes ndk-build.cmd.

Now I'm in the process of changing this to use externalNativeBuild for the NDK integration, and CMake lists instead of the old Android.mk files. It seems like the better supported/documented way of doing things.


Problem

One nice thing about the old ndk-build method was that it would build for multiple ABIs in parallel - e.g. an ARM version and an x86 version of the library could be built in parallel.
This is especially helpful in my case, because I am required to use a 3rd party tool during the linking phase which 1) takes a very long time (minutes) to finish, and 2) is mostly single-threaded. Hence, building the library for multiple ABIs in parallel helped shorten my build times a lot.

When building with CMake / Ninja I cannot replicate this behavior. Ninja is supposed to parallelize builds by default, and it may well be doing that for a given ABI (i.e. compiling multiple source files in parallel for the same ABI). But from what I can tell, it never starts building the library for another ABI until it has finished building for the current ABI.


Question

When using CMake / Ninja through externalNativeBuild, is there any way I can tell the build system that I want it to build my native code for multiple ABIs in parallel?


Example

A minimal example to demonstrate this is as simple as the "New project" template in Android Studio, i.e. something like:

CMakeLists.txt:

cmake_minimum_required(VERSION 3.4.1)

add_library(native-lib SHARED
            src/main/cpp/native-lib.cpp )

app/build.gradle:

apply plugin: 'com.android.application'

android {
    compileSdkVersion 26
    defaultConfig {
        applicationId "com.example.cmakemultiabis"
        minSdkVersion 21
        targetSdkVersion 26
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        externalNativeBuild {
            cmake {
                cppFlags ""
            }
        }
        ndk {
            abiFilters 'armeabi-v7a', 'x86'
        }
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
    externalNativeBuild {
        cmake {
            path "CMakeLists.txt"
        }
    }
}

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'com.android.support:appcompat-v7:27.1.1'
    implementation 'com.android.support.constraint:constraint-layout:1.1.3'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
}

And then building with e.g. gradlew assembleRelease will give you:

> Task :app:externalNativeBuildRelease
Build native-lib x86
[1/2] Building CXX object CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o
[2/2] Linking CXX shared library ..\..\..\..\build\intermediates\cmake\release\obj\x86\libnative-lib.so
Build native-lib armeabi-v7a
[1/2] Building CXX object CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o
[2/2] Linking CXX shared library ..\..\..\..\build\intermediates\cmake\release\obj\armeabi-v7a\libnative-lib.so

It may not be obvious from that output that the two libraries are built sequentially, but it does become obvious if you have something that takes a noticeable amount of time to link.

I have tried this both with the CMake / Ninja versions that you can download with the SDK Manager, as well as some newer versions (CMake 3.12.3, Ninja 1.8.2), and saw the same behavior in both cases.



from How do I parallelize externalNativeBuild for different ABIs?

No comments:

Post a Comment