ikerhurtado.com
You're in
Iker Hurtado's pro blog
Developer | Entrepreneur | Investor
Software engineer (entrepreneur and investor at times). These days doing performant frontend and graphics on the web platform at Barcelona Supercomputing Center

Native activity, app glue lib, lifecycle and threads with NDK

22 Jan 2015   |   iker hurtado  
Share on Twitter Share on Google+ Share on Facebook
I think this post is a good foundation text to approach the creation of a first purely native application (using app glue lib), that is, a Android app without Java code. I also refer to other resources for more complete information.

Native activity

In Android, it's possible to write a completely native activity with the helper class NativeActivity provided by Android SDK; so we can write a completely native application.

This convenience class handles the communication between the Android framework and our native code. All we need to do is declare our application like native in the AndroidManifest.xml file (as I showed in my previous post Creating a new Android project with NDK support).

The NativeActivity also notifies the native code of any activity lifecycle callbacks (onCreate(), onPause(), onResume(), etc). Our native application will implement this callbacks.

Native activities do not change the fact that Android applications still run in their own virtual machine, sandboxed from other applications. Because of this, you can still access Android framework APIs through the JNI. There are, however, native interfaces to access resources such as sensors, input events, and assets that can be accessed directly from the native side.

The Android NDK provides with two ways to implement your native activity:

1. Create your own native activity

The NDK provides with the native_activity.h header that defines a native version of the NativeActivity class. It contains the callback interface and data structures in order to create your native activity. This solution has a single thread approach, so we must worry about returning quickly from our native callback methods. Otherwise, the application can become unresponsive to user actions because the GUI thread is busy executing the callback function. The way to solve this issue is to use multiple threads.

Read the comments in native_activity.h file for more information.

2. Use the helper app glue lib

The android_native_app_glue.h file defines a static helper library built on top of the native_activity.h interface. It spawns another thread to handle things such as callbacks or input events. This prevents any callbacks from blocking the main thread and adds some flexibility in how to implement the callbacks (changing the programming model to make things easier).

Read the comments in android_native_app_glue.h file for more information.

Understanding and using the native app glue lib

As said, Android NDK provides a static library named android_native_app_glue to help execute callback functions and handle user inputs in a separate thread.

app glue lib implementation

I put some high level notes about the implementation.

The following diagram indicates how the main and background thread work together to create the multi-threaded native activity:

At startup the main thread creates a unidirectional pipe for communication between the main UI thread and the background thread.

When an activity lifecycle or UI event occurs, the main thread simply writes a command to the write end of the pipe. Because the signal is left before actual processing of the events, the main thread can return from the callback function quickly without worrying about the possible long processing.

In the background thread the the true loop implemented in the android_main function will poll for events. Once an event is detected, the function calls the event handler, which reads the exact command from the read end of the pipe and handles it.

The android_native_app_glue library implements all the main thread stuff and part of the background thread stuff for us. We only need to supply the polling loop and the event handler.

The library attaches two event queues for the event loop to be created by us in the background thread, including the activity lifecycle event queue and the input event queue.

There are very detailed info about the native app glue implementation at Android Native Development Kit Cookbook and at Android NDK Beginner’s Guide books.

app glue lib use

In addition to the configuration process described in my previous post, the following steps should be followed to use the android_native_app_glue library:

1. Implement a function named android_main. This function should implement an event loop which will poll for events continuously. This method will run in the background thread created by the library.

2. Two event queues are attached to the background thread by default, including the activity lifecycle event queue and the input event queue. We have to poll the events using the looper created by the library. We can identify the event origin by checking the returned identifier (either LOOPER_ID_MAIN or LOOPER_ID_INPUT).

3. When an event or more are detected the looper finishes and the data pointer will point to an android_poll_source data structure. Then we call the process function of this structure. The process is a function pointer which points to android_app->onAppCmd for activity lifecycle events, and android_app->onInputEvent for input events. We can provide our own processing functions and direct the corresponding function pointers to these.

Code example
void android_main(android_app* pApplication) {

 android_poll_source* lSource;
 android_app* application= pApplication;
 application->onAppCmd= activity_event_callback;
 application->onInputEvent = callback_input;

 app_dummy();

 while (true) {
  // Loop the events accumulated (it can be several)
  while ((lResult = ALooper_pollAll(running, NULL, &lEvents, (void**)&lSource)) >= 0) 
  { 
    if (lSource != NULL) {// Lifecycle or input event
     // Launch event app processing via application->onAppCmd 
     // or application->onInputEvent
     lSource->process(application, lSource); 
    }

    // Check if we are exiting.
    if (application->destroyRequested) {
      return;
    }
  }

 // DO TASKS

 }
}

The android_main() method runs the native event loop, which is itself composed of two nested while loops. The outer one is an infinite loop, terminated only when application destruction is requested. Destruction request flag can be found in android_app "application context" provided as an argument to the android_main() method by the native glue.

Inside the main loop is an inner loop which processes all pending events with a call to ALooper_pollAll(). This method is part of the ALooper API which is a general-purpose event loop manager provided by Android. When timeout is -1, ALooper_pollAll() remains blocked while waiting for events. When at least one is received, ALooper_pollAll() returns and code flow continues. The android_poll_source structure describing the event is filled and used for further processing.

Queues are processed by native app glue internal methods process_cmd() and process_input() for the command and input queue, respectively. However both are triggered by our own code when we write lSource->process() in the android_main() function. Then, internally, process_cmd() and process_input() calls itself our own callback.

These is a good example online at android.app.NativeActivity official documentation.

More about polling calls

Poll for events can be done by either one of the following two functions:

int ALooper_pollOnce(int timeoutMillis, int* outFd, int*outEvents, void** outData);

int ALooper_pollAll(int timeoutMillis, int* outFd, int* outEvents,void** outData);

timeoutMillis specifies the timeout for polling. If it is set to zero, then the functions return immediately; if it is set to negative, they will wait indefinitely until an event occurs. The functions return the identifier (greater than zero) when an event occurs from any input queues attached to the looper. In this case, outFd , outEvents, and outData will be set to the file descriptor, poll events, and data associated with the event. Otherwise, they will be set to NULL.

It remains pending me to know if the ALooper_pollAll function can return after more that one event dispatch. The documentation and the outEvents param seem to indicate this, but in practice it always returns after one event dispatch.

If my assumption is true, the lSource->process function should process all these events in one step. Do not?

The android_app structure

The native event loop receives an android_app structure that contains some contextual information such as:

void* userData : This is a pointer to any data we want. It's esud to give some contextual information to the activity event callback.

void (*pnAppCmd)(...), int32_t (*onInputEvent)(...) : These are callbacks triggered respectively when an activity and an input event occur.

ANativeActivity* activity : This describes the Java native activity (its class as a JNI object, its data directories, and so on) and gives the necessary information to retrieve a JNI context.

AConfiguration* config : This contains information about current hardware and system state, such as the current language and country, the current screen orientation, density, size, and so on.

void* savedState size_t savedStateSize : This is used to save a buffer of data when an activity (and thus its native thread) is destroyed and restored later.

AInputQueue* inputQueue : This handles input events (used internally by the native glue).

ALooper* looper : This allows attaching and detaching event listeners (used internally by the native glue). The listeners poll and wait for events represented as data on a Unix file descriptor.

ANativeWindow* window, ARect contentRect : This represents the "drawable" area, in which graphics can be drawn. The ANativeWindow API declared in native_window.h allows retrieving window width, height and pixel format and changing these settings.

int activityState : This describes the current activity state, that is, APP_CMD_START, APP_CMD_RESUME, APP_CMD_PAUSE, and so on.

int destroyRequested : This is a flag. When equals to 1 indicates that application is about to be destroyed and native thread must be terminated immediately. This flag has to be checked in the event loop.

Commands (lifecycle events) from Main (UI) thread

As seen, when something happens in NativeActivity callback handlers are immediately triggered on the native side (main or UI thread). These handlers notify the event by calling internal method android_app_write_cmd() with some of this events:

onStart, onResume, onPause, onStop:
Changes the application state by setting android_app.activityState with the appropriate APP_CMD_* value.

onSaveInstance:
Sets the application state to APP_CMD_SAVE_STATE and waits for the native application to save its state (custom saving has to be implemented in its own command callback).

onDestroy:
Notifies the background thread that destruction is pending, and then frees memory when background thread acknowledges (and does what it needs to frees resources). Structure android_app is not usable anymore and application itself terminates.

onConfigurationChanged, onWindowFocusedChanged, onLowMemory:
Notifies the background thread of the event (APP_CMD_GAINED_FOCUS, APP_CMD_LOST_FOCUS, and so on).

onNativeWindowCreated,onNativeWindowDestroyed:
Calls function android_app_set_window() which provides and requests the background thread to change its display window.

onInputQueueCreated, onInputQueueDestoyed:
Uses a specific method android_app_set_input() to register an input queue. Input queue comes from NativeActivity and is usually provided after native thread loop has started.

Input events

First, note that android_native_app_glue attaches the input event queue for us by default.

When an input event occurs, it will be handled by process_input (the function pointer source->process in the while true loop we called points to process_input if the event is an input event). Inside, AInputQueue_getEvent is firstly called to retrieve the event. Then, AInputQueue_preDispatchEvent is called to send the key for pre-dispatching. This could possibly result in it being consumed by the current Input Method Editor (IME) before the app. Followed by this is the android_app->onInputEvent, which is a function pointer-pointing to an event handler provided by us. If no event handler is provided by us, it's set to NULL . After that, AInputQueue_finishEvent is called to indicate that event handling is over.

How to handle input events

We first called AInputEvent_getType to get the input event type. The android/input.h header file defines two input event types:

AINPUT_EVENT_TYPE_KEY: the input event is a key event. We can call AKeyEvent_getAction, AKeyEvent_getFlags, and AKeyEvent_getKeyCode to get the action, flags, and key code of a key event.

An important problem with NativeActivity is that there is no easy way to display a virtual keyboard. We have to resort to JNI facilities to show it.

AINPUT_EVENT_TYPE_MOTION: the input event is a motion event. We can call AMotionEvent_getAction and AMotionEvent_getX (the same with Y) to get the action and the x position of a motion event. Note that the AMotionEvent_getX function requires the second input argument as the pointer index. The pointer index is obtained by using the following code:

pointer_index = (action & AMOTION_EVENT_ACTION_POINTER_INDEX_MASK) 
                 >> AMOTION_EVENT_ACTION_POINTER_INDEX_SHIFT;

There are a lot more input event functions which can be found at andoid/input.h

Note that callback_input() (and processInputEvent()) return an integer value (semantically a boolean). This value indicates that an input event has been processed by the application and does not need to be processed further by the system. For example, return 1 when the back button is pressed to stop event processing and prevent activity from getting terminated.

The touch API is rather rich. Some calls:

AMotionEvent_getAction(): To detect whether a finger is entering in contact with the screen, leaving it, or moving over the surface. The result is an integer value composed of the event type (on byte 1, for example, AMOTION_EVENT_ACTION_DOWN) and a pointer index (on byte 2, to know which finger the event refers to).

AMotionEvent_getX(), AMotionEvent_getY():To retrieve touch coordinates on screen, expressed in pixels as a float (sub-pixel values are possible).

AMotionEvent_getDownTime(), AMotionEvent_getEventTime():To retrieve how much time finger has been sliding over the screen and when the event has been generated in nanoseconds.

AMotionEvent_getPressure(), AMotionEvent_getSize(): To detect how careful users are with their device. Values usually range between 0.0 and 1.0 (but may exceed it). Size and pressure are generally closely related. Behavior can vary greatly and be noisy depending on hardware.

AMotionEvent_getHistorySize(), AMotionEvent_getHistoricalX(), AMotionEvent_getHistoricalY(): Touch events of type AMOTION_EVENT_ACTION_MOVE can be grouped together for efficiency purpose. These methods give access to these "historical points" that occurred between previous and current events.

It's worthy to note that touch screen event coordinates are absolute and the origin is the upper-left corner (unlike to OpenGL -on the lower-left corner).

If you look more deeply at AMotionEvent API, you will notice that some events have a second parameter pointer_index, which ranges between 0 and the number of active pointers. This is due to multi-touch support. Two or more fingers on a screen are translated in Android by two or more pointers. To manipulate them, look at:

AMotionEvent_getPointerCount():To know how many fingers touch the screen

AMotionEvent_getPointerId(): To get a pointer unique identifier from a pointer index. This is the only way to track a particular pointer (that is, finger) over time, as its index may change when fingers touch or leave the screen.

More on Java and native code lifecycles

Native libraries have a different lifecycle than usual Android activities. When an usual activity is destroyed and recreated for any reason (ie, screen rotation), every data is lost in the activity. However, native library and its global data are likely to remain in memory; it persists between executions and that has impact on the need of memory management.

Therefore, it's important to release memory when an application is destroyed and be careful with create and destroy events. For example, onDestroy() event can sometimes being executed after an activity instance is recreated. The possible solutions are explained in:

Threading

There are two possible types of threads in an application: the usual Java thread (called the managed thread too) created by the VM and the native thread (called non-VM thread too) that can be created from native code.

This native thread can be attached (and detached) to a VM by calling a JNI function. Once attached, the native thread works just like a Java thread, running inside a native method. It's a interesting possibility.

More interesting information in section Pros and Cons of using Java Threads for Native Code (192 page) of the book Pro Android C++ with the NDK.
Also, very interesting info in section Synchronizing POSIX Threads (202 page) of the aforementioned book.
W.Boeke  |  08:57 - 7 Jul
Hi, Good story, clear and informative! About the Java and native code lifecycles: After state->destroyRequested you can do a return or an exit(0). In the latter case the native threads will be killed and all resources released. After a return the resources are kept (as you described), they have to be be released in native code. In an application where all graphics are done in native code, using OpenGL, it appears that only a brute exit(0) works. Else, when the app is restarted, nothing is visible on the screen. Rendering is performed as usual, but it's not visible. I did not want an exit(0) because global data will get lost, but I could'nt find a solution.
Iker Hurtado  |  09:16 - 7 Jul
W.Boeke, thanks for your contribution.
Artem  |  07:16 - 14 Aug
About threading: even if you attach native thread to VM you probaly won't be able to find java classes from your app, only system ones.
Peter  |  09:12 - 22 Sep
Very good post!
thp  |  20:38 - 2 Jan
Wow this stuff is complex. So with the app glu think we obtain a classic single threaded "processEvents (poll event loop) -> update -> render (eglSwapBuffer)" lifecycle ? I didn't well understand the threads, not very sure about it, so the input handling is blocking code unlike in java ? it's in the same thread ? so why we describe it inside callbacks function ? weird
Ankit Singh  |  08:35 - 9 Oct
Very helpful. Thanks a lot.
Viktor Sehr  |  08:09 - 30 Oct
Very good post, thank you.

POST A COMMENT: