Background, usage of AAC

Hi-Q uses AAC audio encoding for M4A format. The AAC is the codec for encoding audio, and the M4A is the file extension for MPEG-4 container format.

In Android, there is a built-in encoder for AAC via the class MediaCodec. To start an AAC encoder, we call

MediaCodec.createEncoderByType("audio/mp4a-latm");

Then we need to call configure on the resulting object, providing it with a MediaFormat. We set up MediaFormat as follows:

format = MediaFormat.createAudioFormat("audio/mp4a-latm", 44100, 1);
format.setInteger(KEY_AAC_PROFILE, AACObjectHE);
format.setInteger(KEY_BIT_RATE, 64000);

Then after it is set, we sent the output of the codec to a MediaMuxer, which will format an MPEG-4 container with the AAC-encoded stream as the sole track.

Here is how we get the audio data encoded by the codec, simplified:

1. We call codec.dequeueInputBuffer() to get an integer, that we pass to codec.getInputBuffer() to get a ByteBuffer.
2. Put the raw audio data to the buffer.
3. Call codec.queueInputBuffer(), passing the buffer with additional information such as the current time, size and offset in bytes, etc.
4. Call codec.dequeueOutputBuffer() to get an integer, that we pass to codec.getOutputBuffer() to get a ByteBuffer.
5. Read bytes from the buffer, and pass that to the muxer.

This works most of the time. Hi-Q users are able to enjoy a better audio quality and compression in m4a compared to mp3.

The random crash

However, since the first time we release this feature, there has been seemingly random crash as follows:

Fatal Exception: java.lang.IllegalStateException
       at android.media.MediaCodec.native_dequeueInputBuffer(MediaCodec.java)
       at android.media.MediaCodec.dequeueInputBuffer(MediaCodec.java:2635)

Fatal Exception: java.lang.IllegalStateException: Failed to stop the muxer
       at android.media.MediaMuxer.nativeStop(MediaMuxer.java)
       at android.media.MediaMuxer.stop(MediaMuxer.java:454)

Fatal Exception: java.lang.IllegalStateException
       at android.media.MediaCodec.native_dequeueOutputBuffer(MediaCodec.java)
       at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2698)

The rate of this happening is ~800 users per month, out of more than 200K monthly users that we have.

Unfortunately we can’t find the reason of this happening, we tried multiple ways of reproducing and we never find any reliable way of triggering this error. We suspect that this was a faulty implementation of the codec, but this happened not only on a specific device model but arbitrarily.

Experiment to reliably reproduce crash

Only until one of our customer, Alex, told me that he often get a crash when he sets the Gain to the max while using the M4A format. This might explain the situation!

I tried to do the same myself, setting the gain to the max and recording using the M4A format. I tried it several times. Almost always Hi-Q crashed after several seconds, with similar stack trace as above. This was enlightening!

So we tried to reproduce the crash in a systematic manner. Instead of using audio data taken from real live, we generate the audio data in code:

for (int i = 0, len = block.len; i < len; i++) {
    final short[] data = block.data;
    if (i % 100 == 0) up = !up;
    if (up) {
        data[i] = (short) (32500 + i);
    } else {
        data[i] = (short) (-32500 - i);
    }
}

A block contains 4400 samples. The code above generates a sharp and loud audio signal, having amplitudes 32500~32599 out of max value of 32767. On the third block, the encoding process always crashes.

Hypothesis 1: The loud signals causes the codec to output a lot of data, so we need to repeatedly drain the output by calling dequeueOutputBuffer multiple times until BUFFER_FLAG_END_OF_STREAM is given. I was optimistic when I found that this was not currently done in the code.

This did not work. The same error still happened.

Hypothesis 2: We implemented waiting for the input buffer by calling dequeueInputBuffer with a wrong timeout.

To prevent deadlock, dequeueInputBuffer has a timeout parameter. Probably it was too short or too long. Unfortunately, there is no guideline or clear standard on the number we should put.

Hypothesis 3: We should have used the recommended asynchronous API instead of enqueueing and dequeueing the buffers ourselves.

Since Build.VERSION_CODES.LOLLIPOP, the preferred method is to process data asynchronously by setting a callback before calling configure.

In the asynchronous, we set a callback object to the codec that will receive input buffers to be filled in via onInputBufferAvailable, output buffers from where bytes are extracted via onOutputBufferAvailable, and errors via onError. No more calling dequeue methods, and no more timeout values to be specified!

Sadly, the same error still happens, via onError. The error has code=1101, diagnostic info “android.media.MediaCodec.error_1101”, and isRecoverable=false.

In conclusion, we feel that it is quite likely that the error was not caused by dequeueing and enqueueing the input and output buffers at the wrong order.

Changes made

Hypothesis 4: The codec is configured in a way that does not allow encoding of audio signals outside of a certain limit.

We tried to see more carefully what parameters we passed to the codec, and we realized that we had this:

format.setInteger(KEY_AAC_PROFILE, AACObjectHE);

Then I thought: Why this has to be AACObjectHE? Could I change it to something else and maybe everything would work well? In online examples of encoding audio signals in AAC, people would use AACObjectLC (const value 2) or AACObjectHE (const value 5). I tried to change it to AACObjectLE and… voila! No more crashes.

This change is published starting in Hi-Q 2.8.0. I hope everything goes well and we have much more reliable M4A recording. We apologize for the earlier crashes. 🙇‍♂️