xasworks / esp-libopus Goto Github PK

I just discovered this, and in the interest of telling the folks who might be interested to know, I am posting it here.

As you know, esp-idf is the official sdk / build system / libraries for esp32. There is also esp-adf, which is specifically geared towards audio processing on the esp32. It bundles esp-idf (a slightly older version), and a bunch of extra components - mainly audio_pipeline_handle_t and all a load of audio elements you can use. Notably, there is an opus encoder element (components/esp-adf-libs/esp_codec/include/codec/opus_decoder.h) and opus decoder element (components/esp-adf-libs/esp_codec/include/codec/opus_encoder.h)

esp-idf and esp-adf libraries (they call them components) mostly consist of source code. You build those components on your machine when you build your project.

But esp-adf-libs is a distribution in binary form. People are already asking for the code (espressif/esp-adf-libs#12)

If you look at the header files for opus_decoder.h and opus_encoder.h in esp-adf-libs, the functionality is only exposed behind the audio pipeline abstraction (audio_element_handle_t).

How is the actual opus encoding/decoding being done? You might think, from reading the espressif documentation, that the "codec chip" is doing this in hardware.

Nope, they've built libopus and bundled it into esp-adf-libs.

todd@euclid ~/esp-adf
#
# some symbols defined in libopus are present in libesp_codec.a
#
1 (v2.4) % readelf -as components/esp-adf-libs/esp_codec/lib/esp32/libesp_codec.a | grep opus_decoder_create
    59: 00000000   149 FUNC    GLOBAL DEFAULT   35 opus_decoder_create

# 
# are these symbols declared in the libesp_codec header files? they are not
# 
0 (v2.4) % ag opus_decoder_create components/esp-adf-libs/esp_codec/include

I can confirm that you can just #include header files from https://github.com/xiph/opus in your esp-adf project, and it will build, link, and run. (Ok, I haven't tried the encoder, but the decoder works just fine).

Notably, this lets you decode opus and play it directly with the i2s driver, without using the audio_pipeline abstraction at all.

I wonder whether real-time encoding (delay 0-300ms) on the most recent ESP32 CPUs is acceptable assuming 48kHz with Opus complexity 4 or higher.

Has anyone tried that?

Is it possible?