During office hours @PranavPeshwe suggested we document resource usage. Example metrics worth tracking we discussed:

• Memory overhead of minimal app, with Golioth, per Golioth feature
• Memory, especially peak
• Binary size

We briefly discussed instrumenting these and other metrics as part of the CI process. In the past I've worked with:

• Bloaty
• Valgrind
• gprof
• LLVM sanitizers

Zephyr also has some tools.

Of course Memfault has a series on the topic.

I'm having an issue configuring a custom APN when using Golioth, however the same method works fine when using the vanilla Nordic examples

Background;
My NBIoT capable sim card requires a custom APN to be set for a data connection to be opened. If you do not set a custom APN the sim registers on and receives the default operators APN, which seems to work but blocks any data at all from being sent.
I believe the suggested way to add a custom APN with Zephyr for the latest NRF toolchain is with the following added in your prj.conf file:

#APN setup 
CONFIG_PDN=y
CONFIG_PDN_SYS_INIT=y
CONFIG_PDN_DEFAULTS_OVERRIDE=y
CONFIG_PDN_DEFAULT_APN="flickswitch"

# I also set NBIoT mode 
CONFIG_LTE_NETWORK_MODE_NBIOT=y

If I add the above to the Nordic samples such as at_client or asset_tracker, which compile and connect just fine without issues.

However when adding the same to a Golioth sample prj.conf, such as hello or dfu samples, the APN is not set correctly. One of the early log messages shows:
[00:00:12.480,377] <err> pdn: Failed to configure CID 0, err, -8

Full output of my modified hello sample as follows:

uart:~$ *** Booting Zephyr OS build v2.6.99-ncs1-1  ***
[00:00:00.215,728] <inf> golioth_system: Initializing
uart:~$ +CEREG: 2,"5209","099D410D",9
+CSCON: 1
+CEREG: 1,"5209","099D410D",9,,,"11100000","00000110"
[00:00:13.809,906] <err> pdn: Failed to configure CID 0, err, -8
[00:00:13.818,756] <err> pdn: Failed to configure default CID, err -8
[00:00:13.828,277] <dbg> golioth_hello.main: Start Hello sample
[00:00:13.llo.main: Hi Discord :D
[00:00:13.844,848] <wrn> at_notif: Already initialized. Nothing to do
uart:~$ The Golioth Hello AT host sample started
[00:00:13.858,093] <dbg> golioth_hello.main: Calling Golioth Client Start
[00:00:13.867,492] <dbg> golioth_heo.main: Starting while loop
[00:00:13.876,098] <inf> golioth_hello: Sending hello! 0
uart:~$ 
        [00:00:13.893,157] <dbg> golioth_hello.main: Sleeping 5s
[00:00:13.901[00:00:18.901,123] <inf> golioth_hello: Sending hello! 1
[00:00:18.909,576] <wrn> golioth_hello: Failed
[00:00:18.918,182] <dbg> golioth_hello.main: Sleeping 5s
[00:00:23.926,147] <inf> golioth_hello: Sending hello! 2
[00:00:23.934,600] <wrn> golioth_hello: Failed[00:00:23.943,206] <dbg> golioth_hello.main: Sleeping 5s
[00:00:28.951,171] <inf> golioth_hello: Sending hello! 3
[00:00:28.959,625] <wrn> golioth_hello: Faileeeping 5s
[00:00:33.976,196] <inf> golioth_hello: Sending hello! 4
[00:00:33.984,649] <wrn> golioth_hello: Failed to send hello!
[00:00:33.993,255] <dbg> golioth_hello.main: Sleeping 5s
[00:00:36.953,826] <inf> golioth_system: Client connected!
[00:00:39.001,220] <inf> golioth_hello: Sending hello!

I modified the sample to allow the use of the LTE Link Monitor Utility, this shows how the device is registering with the default operator network APN (lte.vodacom)

I then modified the sample to manually set the APN using the AT+CGDCONT command (first turning the modem off with AT+CFUN=0, then back on after) using the following code:

        if(at_cmd_write("AT+CFUN=0", NULL, 0, NULL) != 0)
        {
            LOG_INF("modem issue!\n");
        }else{
            LOG_INF("Modem off \n");
        }

        k_sleep(K_SECONDS(1));


        err = at_cmd_write("AT+CGDCONT=0,\"IP\",\"flickswitch\"", NULL, 0, NULL);
        if (err) {
                printk("Could not define PDP context +CGDCONT, error: %d", err);
                return err;
        }else{
                LOG_INF("APN Set\n");
        }

        k_sleep(K_SECONDS(1));


        if(at_cmd_write("AT+CFUN=1", NULL, 0, NULL) != 0)
        {
           LOG_INF("modem issue!\n");
        }else{
            LOG_INF("Modem on \n");
        }

This has the desired effect of setting the APN correctly and my device is assigned the correct IP address allocated to this SIM card

However, Golioth was still unable to connect! No logs are sent to the cloud:

ramb0t@ubuntu:~/zephyr-nrf/modules/lib/golioth$ goliothctl logs --interval=10m
no logs found

Note also the somewhat sketchy UART output, not sure why that corruption is happening.

This may well be a problem with my toolchain setup, I'm not experienced enough to know. Once again however if I compile the Nordic asset tracker example with just the PDN config settings in prj.conf, then the device communicates with the Nordic cloud as it should.

My (sanitised) .config output from build/zephyr/.config attached:
config.txt

Right now the docs point to hello. However, that example has several overlays that can be confusing to a new user. Also, the sample code could use a bit of clean up/simplication.

Can we create a new quickstart sample, with a single overlay and the simplest logging example possible? We also might switch to using kconfig gui for the sample, so let's make sure that works too!

For example

https://github.com/golioth/zephyr-sdk/blob/ff39bf1c51e8185964c7de8b253ca4d657393ca8/samples/hello/src/main.c#L43

This could be set in an initializer function or a setter of some sort.

My one suggestion would be to provide an example or sample to retrieve basic network information from the modem like APN, IP address etc (AT+CGDCONT) at a user level, so it is not too difficult if you are not familiar with all zephyr configs

Originally posted by @ramb0t in #83 (comment)

The Golioth SDK fully manages connections to the Golioth Server, dealing with initialization, timeouts, retries, etc. While that is still the ideal design for the happy path, a developer might want to exert a little more control over the network/modem. One specific use case is a low power application. Ex. the device is collecting sensor data and only periodically connects to send batched information. In this use case, the developer has the best understanding of their power budget and would like to manage the connectivity like they manage other peripherals in the pursuit of their use case.

/via @ubieda during 6/30 office hours

During OH @mniestroj described how we would like to move to an asynchronous, event driven model to make exposing the ability to manage the connection easier to implement but for now there could be a workaround to manually setup and manage things.

Zephyr includes a settings subsystem that provides an abstract way to store persistent per-device configuration and runtime state. Settings is very flexible but some examples might include adjusting a motor speed, a sensor sampling rate or network configuration.

The enhancement is develop a Device Service to manage Settings and the corresponding UI. As an example, a version of this could be built on LightDB with some device-side logic to integrate OBSERVEd updates with Settings. Also, the UI could be built on top of the current REST API for LightDB. Effectively, this would turn that example into a fully managed Device Service.

/via @ubieda during 6/30 office hours

zephyrproject-rtos/zephyr#37155

Once merged, we should change our configuration to take advantage of this better UX.

Per #31 (comment), "we need to start using Zephyr from tip of master (right now we still rely on stable 2.6.0)."

@keenanjohnson asked in Discord for an example showing how to setup/deploy a device in the field via the web browser. Some of that discussion (Keenan - please add any more details!):

Given that we're open source and sort of limited in developer resources, something that we can develop once for all platforms ( like a UI in the web) would be the most ideal that allows our users to connect their devices to their home wifi!
So when you get your device you got to sensor.local or something similar to connect it to your home wifi etc. That works well for our full linux devices today and not sure if it's technically feasible for an esp32
...
The device broadcasts an AP, the user connects to that, enters the credentials and then the device switches to that AP.
...
I've seen other ESP devices do something similar, like this one: https://www.bhencke.com/pixelblazegettingstarted, so it seems technically feasible.

Basically create a sample that serves a web page for entering network credentials, similar to our mobile demo (which does more than wi-fi credentials.)

A non-Zephyr library I've used is called esp32-wifi-manager We could do something similar with civetweb as the webserver, dns-sd for .local discovery & settings to store credentials.

https://piolabs.com/blog/news/platformio-oss-july-2021-updates.html

I believe we decided to remove these, right?

The LightDB LED example updates a counter needlessly. This is left over from a previous sample which was more general

Describe the feature

lightDB delete was added to the api with #155. This should be backported to 3095973 so that it is available when building for nRF91 targets.

Issue was discovered when building an app with golioth_lightdb_delete() on ESP32, then testing it on nRF9160 and build fails:

/home/mike/golioth-compile/AgTech-Soil/fw/src/main.c: In function 'delete_control_endpoint':
/home/mike/golioth-compile/AgTech-Soil/fw/src/main.c:231:8: warning: implicit declaration of function 'golioth_lightdb_delete'; did you mean 'golioth_lightdb_set'? [-Wimplicit-function-declaration]
  231 |  ret = golioth_lightdb_delete(client,
      |        ^~~~~~~~~~~~~~~~~~~~~~
      |        golioth_lightdb_set

Current workaround is:

/* FIXME: Golioth SDK for nRF91 is behind and doesn't yet include this command */
#ifndef CONFIG_SOC_NRF9160_SICA
ret = golioth_lightdb_delete(client, GOLIOTH_LIGHTDB_PATH("cmd_led"));
#endif

Existing feature reference

golioth_lightdb_delete()

During office hours, the community suggested it would be helpful to have samples that help new users as they are getting started with the platform but might run into issues while troubleshooting.

The general idea would be able to test that the different parts of the stack are working. It becomes hard to determine if you're not seeing LightDB Stream data if the cellular modem isn't reliably connecting to a tower. To troubleshoot that specific example, the app could log data locally, publish the the RSSI of the tower, etc. Since it is not intended for production, the sample should err on sending more data and enabling most services (aka a "kitchen sink demo.") It was also suggested that we should have separate samples for Wi-Fi and Cellular since you want different information for each communication medium.

I'd love to use this issue to gather suggestions and requirements!

The paths we use for the services can be located centrally

Link provided in https://github.com/golioth/zephyr-sdk/tree/main/samples/hello/README.rst (https://docs.zephyrproject.org/latest/guides/networking/qemu_setup.html#networking-with-qemu) is no longer valid. We should provide a link to stable documentation release, not the "latest" one. Potentially all links to Zephyr documentation pages are not affected with the same.

Re-name wifi.c to something more indicative of it's use to support our samples.

I can build most of the samples for ESP32
but when building lightdb_stream with
~/zephyrproject/modules/lib/golioth$ west build -b esp32 samples/lightdb_stream/ -p

the build fails with:

/home/ramb0t/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.bfd: zephyr/zephyr_prebuilt.elf section bss' will not fit in region dram0_0_seg'
/home/ramb0t/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.bfd: DRAM segment data does not fit.
/home/ramb0t/.espressif/tools/xtensa-esp32-elf/esp-2020r3-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld.bfd: region dram0_0_seg' overflowed by 1008 bytes
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
FATAL ERROR: command exited with status 1: /usr/bin/cmake --build /home/ramb0t/zephyrproject/modules/lib/golioth/build

Possibly related to esp32-c3 support recently added, as discussed https://discord.com/channels/819247807662915605/896524295298248754/912466408762404864

Create an issue template for issues that track collections of other issues

Its confusing to subscribe to the 'observed' resource. The other samples query 'counter' which is intuitive as a resource. We should consider changing 'observed' to 'counter'

https://github.com/golioth/zephyr-sdk/blob/409ff7b4e6d4d8649e665cbe4e165ad2284ba464/samples/lightdb/observe/src/main.c#L71

zephyrproject-rtos/zephyr#36981

We should identify what is not needed from a typical Golioth application. We may add more groups over time. Note that the user can always manually include a group through flags or their own west.yaml.

Hardware Target

all

Golioth Dsdk Commit Hash

3095973

What Host OS are you using?

Other (Mention Below)

What happened?

Using empty PSK/PSK_ID results in errors appearing from Zephyr logging subsystem, but there is no clear way what is the source of the problem. We should at least point users into PSK/PSK_ID being invalid, as that is often the case.

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

• #45
• #107
• #108
• #112
• #114
• #129

Hardware Target

nRF9160

Golioth Dsdk Commit Hash

main

What Host OS are you using?

Mac

What happened?

Currently TLS_PSK_ID is set at compile time. In some cases, someone may want to use an onboard unique I identifier (like an IMEI) as the device ID. Setting this during Golioth init would be handy for this purpose (since that IMEI is not retrievable until runtime)

Happy to provide more context but hope that helps.

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

It would be handy to have a way to query the state of the Golioth cloud connection, connected/disconnected etc. This would helo with usability for the end-user. Then for example we could have status LEDs on a device to show network connectivity and cloud connectivity status and help with debugging troublesome devices by showing potential network issues at a glance.

https://discord.com/channels/819247807662915605/819251195940044800/938741598559305769

Hardware Target

ESP32 DevKit1

Golioth Dsdk Commit Hash

440b700

What Host OS are you using?

Ubuntu

What happened?

Trying to get the Golioth DFU sample working on ESP32 with the MCUBoot support.

OTA update never finishes and seems to restart at about 566KB downloaded.

To setup, follow this guide: https://github.com/golioth/zephyr-sdk/blob/main/samples/dfu/README.rst#espressif-mcuboot-target
And this https://github.com/mcu-tools/mcuboot/blob/399720d1cabd26c4356445d351f263b31e942961/docs/readme-espressif.md

Command history:

cd bootloader/mcuboot/
 git submodule update --init --recursive --checkout boot/espressif/hal/esp-idf
 git submodule update --init --recursive ext/mbedtls
 cd boot/espressif/hal/esp-idf
 ./install.sh 
 cd ../..
 cmake -DCMAKE_TOOLCHAIN_FILE=tools/toolchain-esp32.cmake -DMCUBOOT_TARGET=esp32 -B build -GNinja
 cmake --build build/
 esptool.py --chip esp32 elf2image --flash_mode dio --flash_freq 40m -o build/mcuboot_esp32.bin build/mcuboot_esp32.elf
 esptool.py -p /dev/ttyUSB0 -b 115200 --before default_reset --after hard_reset --chip esp32 write_flash --flash_mode dio --flash_size detect --flash_freq 40m 0x1000 build/mcuboot_esp32.bin

mcuboot bootloader then successfully loaded on ESP32. On to DFU sample

Setup the wifi credentials and psk in prj.conf as only change to sample. Compiled the example as follows:

west build -b esp32 -p
 west sign -t imgtool -- --key ~/zephyrproject/bootloader/mcuboot/root-rsa-2048.pem
 west flash --bin-file build/zephyr/zephyr.signed.bin --hex-file build/zephyr/zephyr.signed.hex

After flashing successfully, signed new version and created artefact and release:

west sign -t imgtool --no-hex -B new.bin -- --key ~/zephyrproject/bootloader/mcuboot/root-rsa-2048.pem --version 0.0.1
goliothctl dfu artifact create new.bin --version 0.0.1 
goliothctl dfu release create --release-tags 0.0.1 --components [email protected] --rollout true

At this point ESP32 detected updated and starts OTA process:

[esp32] [INF] *** Booting MCUboot build v1.7.0-421-g399720d ***
[esp32] [INF] Primary image: magic=bad, swap_type=0x4, copy_done=0x2, image_ok=0x2
[esp32] [INF] Scratch: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[esp32] [INF] Boot source: none
[esp32] [INF] Swap type: none
[esp32] [INF] br_image_off = 0x10000
[esp32] [INF] ih_hdr_size = 0x20
[esp32] [INF] DRAM segment: start=0x1a010, size=0x23d4, vaddr=0x3ffb0000
[esp32] [INF] IRAM segment: start=0x1c3e4, size=0x1946c, vaddr=0x40080000
0x40080000: _WindowOverflow4 at /home/ramb0t/zephyrproject/zephyr/arch/xtensa/core/window_vectors.S:54

[esp32] [INF] start=0x400917b8
0x400917b8: __start at /home/ramb0t/zephyrproject/zephyr/soc/xtensa/esp32/loader.c:86



uart:~$ *** Booting Zephyr OS build v2.7.99-2737-g4521eb7d2c5a  ***
[00:00:02.350,000] <inf> golioth_system: Initializing
[00:00:02.350,000] <inf> net_config: Initializing network
[00:00:02.350,000] <inf> net_config: Waiting interface 1 (0x3ffb2368) to be up...
[00:00:02.351,000] <inf> esp32_wifi: WIFI_EVENT_STA_START
[00:00:02.351,000] <inf> net_config: Interface 1 (0x3ffb2368) coming up
[00:00:02.351,000] <inf> net_config: Running dhcpv4 client...
[00:00:03.048,000] <inf> esp32_wifi: WIFI_EVENT_STA_DISCONNECTED
[00:00:03.106,000] <inf> esp32_wifi: WIFI_EVENT_STA_CONNECTED
[00:00:09.375,000] <inf> net_config: IPv4 address: 192.168.88.169
[00:00:09.375,000] <inf> net_config: Lease time: 600 seconds
[00:00:09.375,000] <inf> net_config: Subnet: 255.255.255.0
[00:00:09.375,000] <inf> net_config: Router: 192.168.88.1
[00:00:09.375,000] <dbg> golioth_dfu.main: Start DFU sample
[00:00:09.375,000] <err> golioth_dfu: Failed to confirm image: -5
[00:00:09.375,000] <inf> golioth_system: Starting connect
[00:00:11.700,000] <dbg> golioth_dfu: Desired
                                      a3 01 1a 61 fc 19 9a 02  78 40 31 37 37 39 64 61 |...a.... x@1779da


...

[00:04:13.264,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 580608
[00:04:13.673,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 581632
[00:04:14.799,000] <inf> mcuboot_util: Swap type: none
[00:04:14.799,000] <inf> golioth_dfu: swap type: none
[00:04:15.209,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 1024
[00:04:15.516,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 2048
[00:04:15.788,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 3072
[00:04:16.200,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 4096

seems to crash at a slightly different offset each time but close to 566KB, not sure if thats significant

[00:12:12.914,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 528384
[00:12:13.928,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 0
[00:12:13.928,000] <inf> mcuboot_util: Swap type: none
[00:12:13.928,000] <inf> golioth_dfu: swap type: none
[00:12:14.336,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 1024

also ran
uart:~$ mcuboot

lots of bad in the output

[00:00:26.355,000] <inf> mcuboot_util: Swap type: none
swap type: none
confirmed: 0

primary area (1):
  version: 0.0.3+0
  image size: 638976
  magic: bad
  swap type: revert
  copy done: bad
  image ok: bad

secondary area (2):
  version: 0.0.3+0
  image size: 638976
  magic: bad
  swap type: unknown
  copy done: bad
  image ok: bad

Link to discussion: https://discord.com/channels/819247807662915605/896524295298248754/938862700958744618

Relevant log output

[esp32] [INF] *** Booting MCUboot build v1.7.0-421-g399720d ***
[esp32] [INF] Primary image: magic=bad, swap_type=0x4, copy_done=0x2, image_ok=0x2
[esp32] [INF] Scratch: magic=unset, swap_type=0x1, copy_done=0x3, image_ok=0x3
[esp32] [INF] Boot source: none
[esp32] [INF] Swap type: none
[esp32] [INF] br_image_off = 0x10000
[esp32] [INF] ih_hdr_size = 0x20
[esp32] [INF] DRAM segment: start=0x1a010, size=0x23d4, vaddr=0x3ffb0000
[esp32] [INF] IRAM segment: start=0x1c3e4, size=0x1946c, vaddr=0x40080000
0x40080000: _WindowOverflow4 at /home/ramb0t/zephyrproject/zephyr/arch/xtensa/core/window_vectors.S:54

[esp32] [INF] start=0x400917b8
0x400917b8: __start at /home/ramb0t/zephyrproject/zephyr/soc/xtensa/esp32/loader.c:86



uart:~$ *** Booting Zephyr OS build v2.7.99-2737-g4521eb7d2c5a  ***
[00:00:02.350,000] <inf> golioth_system: Initializing
[00:00:02.350,000] <inf> net_config: Initializing network
[00:00:02.350,000] <inf> net_config: Waiting interface 1 (0x3ffb2368) to be up...
[00:00:02.351,000] <inf> esp32_wifi: WIFI_EVENT_STA_START
[00:00:02.351,000] <inf> net_config: Interface 1 (0x3ffb2368) coming up
[00:00:02.351,000] <inf> net_config: Running dhcpv4 client...
[00:00:03.048,000] <inf> esp32_wifi: WIFI_EVENT_STA_DISCONNECTED
[00:00:03.106,000] <inf> esp32_wifi: WIFI_EVENT_STA_CONNECTED
[00:00:09.375,000] <inf> net_config: IPv4 address: 192.168.88.169
[00:00:09.375,000] <inf> net_config: Lease time: 600 seconds
[00:00:09.375,000] <inf> net_config: Subnet: 255.255.255.0
[00:00:09.375,000] <inf> net_config: Router: 192.168.88.1
[00:00:09.375,000] <dbg> golioth_dfu.main: Start DFU sample
[00:00:09.375,000] <err> golioth_dfu: Failed to confirm image: -5
[00:00:09.375,000] <inf> golioth_system: Starting connect
[00:00:11.700,000] <dbg> golioth_dfu: Desired
                                      a3 01 1a 61 fc 19 9a 02  78 40 31 37 37 39 64 61 |...a.... x@1779da

...

[00:04:13.264,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 580608
[00:04:13.673,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 581632
[00:04:14.799,000] <inf> mcuboot_util: Swap type: none
[00:04:14.799,000] <inf> golioth_dfu: swap type: none
[00:04:15.209,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 1024
[00:04:15.516,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 2048
[00:04:15.788,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 3072
[00:04:16.200,000] <dbg> golioth_dfu.data_received: Received 1024 bytes at offset 4096

Code of Conduct

• I agree to follow this project's Code of Conduct

Describe the feature

https://github.com/zephyrproject-rtos/sdk-ng/releases/tag/v0.14.0

The official toolchain provided by Zephyr was recently migrating to a new packaging and distribution system. This should have no affect on Linux (but we should verify) but introduces official packages for macOS & Windows. This is a significant improvement on the user experience for users on macOS & Windows.

Note: the Espressif toolchain is still separate, but there is a draft PR that will add it to the new distribution. This will be benefit all three platforms.

Existing feature reference

No response

Extra layer of file structure can cause confusion

Zephyr Commit hash

d51a67b7c098c8bb5479a7b87f05dc40fb76e065

Golioth SDK Commit Hash

cc65be0

Hardware Target

all platforms

What host OS are you using?

This affects every OS

Current Behavior

Known Issue

This is a known issue, we're filing this as a bug to make the fix more visible

Anyone who installed the Golioth SDK prior to May 2022 and tries to update it will see a FileNotFoundError for the west.yml when updating their repo and running west update.

Expected Behavior

How to fix

Overview

Edit the config file in the <project root>/.west directory to point to west-zephyr.yml (or west-ncs.yml if you are using the NCS version of our SDK)

Steps to fix:

1. Edit zephyrproject/.west/config
2. Change file = west.yml to file = west-zephyr.yml
3. Update west

   cd ~/zephyrproject/modules/lib/golioth
   git pull
   west update
   

Steps To Reproduce

Start with a Golioth Zephyr SDK installed prior to May 2022

1. cd ~/zephyrproject/modules/lib/golioth
2. git pull
3. west update

Logs, console output, or any kind of debug information

$ cd ~/zephyrproject/modules/lib/golioth
$ git pull
$ west update
Traceback (most recent call last):
  File "/home/mike/zephyrproject/.venv/bin/west", line 8, in <module>
    sys.exit(main())
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/app/main.py", line 781, in main
    app.run(argv or sys.argv[1:])
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/app/main.py", line 106, in run
    self.run_command(argv)
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/app/main.py", line 336, in run_command
    cmd.run(args, unknown, self.topdir, manifest=self.manifest)
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/commands.py", line 140, in run
    self.do_run(args, unknown)
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/app/project.py", line 802, in do_run
    self.update_all()
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/app/project.py", line 856, in update_all
    self.manifest = Manifest.from_file(
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/manifest.py", line 1153, in from_file
    return Manifest(**kwargs)
  File "/home/mike/zephyrproject/.venv/lib/python3.8/site-packages/west/manifest.py", line 1283, in __init__
    source_data = source_file.read_text()
  File "/usr/lib/python3.8/pathlib.py", line 1236, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/mike/zephyrproject/modules/lib/golioth/west.yml'

Need to implement an issue template for bugs

We missed a bug with TLS PSK length because we were not testing NRF91 + NCS at buildtime.

@mniestroj to look into what it will take to automate this.

• #118

Currently, only one sample shows how to use CBOR, despite CBOR support from 3 Device Services. This came up today when someone asked about implementing CBOR with LightDB Stream. We should decide if we want to add JSON AND CBOR examples, add CBOR as an alternative in each sample (if/else, )reference a snippet from the samples or something else. I don't have a recommendation at this time but multiple samples feel the least maintainable.

This also relates to #141.

Create an application_support directory to contain helper code like wifi.c

Build files are attempting to be tracked. Lets get these ignored.

Currently, when sending timeseries data using LightDB Stream, the timestamp is set when the message is received by the cloud. However, there are use cases where a device might have a time service and want to set the timestamp from the moment the event was created. Ex. mostly disconnected devices that stores events in a queue.

This request is to add an API to set the timestamp. We would leave the source of time up to the developer for now, but in the future we may provide a Time Device Service.

/via @ubieda during 6/30 office hours

As reported by @ubieda

I just tried a fresh build for the nrf9160 of the hello sample and it's giving me an error (log with info attached). Are you able to build the latest version using this guide? https://docs.golioth.io/hardware/nrf91/quickstart/set-up-zephyr

golioth_err_log_211201.txt

ChrisGammell:
@mniestroj is looking into this, but a short term fix is to add the following to the /net/golioth/system_client.c

ChrisGammell:
//Add #define for NCS (which doesn't configure this variable for some reason)
#ifndef CONFIG_MBEDTLS_PSK_MAX_LEN
#define CONFIG_MBEDTLS_PSK_MAX_LEN 64
#endif

CBOR is an ideal compact binary encoding format that we recommend where bits and bytes matter, like Cellular. Our Device Services already support the format. The SDK kinda supports it today but requires a fair bit of boilerplate. The ideal scenario is that the developer set a value to use CBOR and starts sending data. It would also be nice to set this globally via KConfig but still make it possible to set this at runtime.

I consider this indirectly related to #100.

Aproving.

Follow up discussion:

1. Can we create a single source of truth for our 'flagship' boards
2. Can we run CI on those boards based on their device tree?

Originally posted by @0Grit in #118 (review)

Comment our usage of the SYS_INIT macro as this could be highly confusing to users unfamiliar with the complexity of this macro

https://github.com/golioth/zephyr-sdk/blob/ff39bf1c51e8185964c7de8b253ca4d657393ca8/net/golioth/system_client.c#L185-L186

Possibly move the macro into a more intuitive location as well?

The currently supported version of NCS is 1.7.1.

We should update our SDK to support the latest current version of NCS, 1.8.0.

Reported in #157

So far we have only build tests in CI. Those work well, but they do not check if something was broken in runtime. We need to develop automated tests, which will run whenever we refactor code, add more features or want to test with newer Zephyr (or some library) release.

In the first step it makes most sense to use simulated device (e.g. QEMU, native_posix) to execute SDK samples / tests. This will cover 80-90% of the codebase already. Next step would be to use reference hardware boards officially supported by Golioth. This however requires more effort, but is the only way to increase code coverage of the samples we provide.

Originally reported by @mniestroj

Describe the feature

The DFU sample works on ESP32, but is not mentioned in the README:
https://github.com/golioth/zephyr-sdk/blob/main/samples/dfu/README.rst

Instructions should be added, or just a link to the blog post:
https://blog.golioth.io/firmware-update-for-esp32-devices/

Existing feature reference

dfu sample

Current implementation of DFU by observing single CoAP /update resource is very simple, but also very restricted in design. This Issue is a collection of nice to have features, which we want to take into account when designing a more powerful and flexible DFU mechanism. This list includes ideas for both cloud part as well SDK part.

• be able to store multiple firmware versions "in golioth" instead of always providing local file, which might be useful to have "phased rollout" with group of devices
• delta firmware upgrade - support DFU by sending diff of two firmware images, in order to reduce network bandwidth (tested bsdiff some time ago and simple changes in 200k firmware could result in 3k of diff binary, while significant code changes were still around ~50k, i.e. 25% original size); might be useful for some applications
• allow more control over DFU process, so that:
  • downloaded firmware on device is not automatically selected for upgrade on all devices; instead in some situations it might be desirable to transfer firmware to group of devices and only then mark that firmware as bootable on all devices at once,
  • newly booted firmware is confirmed by cloud after it successfully connects back to golioth or adding some status on progress of DFU,
• supporting multiple image DFU (e.g. to upgrade nrf52+esp combo),
• support XIP images (e.g. mcuboot's BOOT_DIRECT_XIP):
  • In order to prevent application swap, we can prepare 2 variants of the firmware. First one will support running from the first application slot, where second will support running from the second application slot. Those two variants would need to be uploaded to cloud (golioth). During firmware download process, we should somehow figure out which firmware variant should we download and flash (depending on which application slot are we running from now).
  • What will we gain by supporting this option:
    • faster "reboot from older to newer firmware" - this would address quite long time (up to ~1min for ~500kB firmware) that is needed for swapping two application images,
    • faster revert to previous firmware (in case of any errors),
    • smaller bootloader size, just like the BOOT_DIRECT_XIP option description says; this means more space for applications that uses golioth SDK,
    • smaller flash wear (it shouldn't be a problem in most cases, but it still improves it a bit).
  • Downsides:
    • requires building of two firmware variants,
    • golioth server and/or golioth SDK needs to be aware of those two firmware variants and choose the correct one (bit more complex).

Our goal is to enable developers to trust our IoT clients and services so they can focus on the application specific detail of their work.

Their applications depend on the client not the other way around.
Therefore I propose we pull out the samples directory into either it's own repository or multiple repositories. (1 per example)

The samples can then act as a working template for the developer to start from and eventually fork.

There is no lightdb delete API yet. Add that and provide sample that utilizes it (so we can test that it actually works).

Originally reported by: @alvarowolfx

• Implementation
• Doxygen
• Sample

So far we were not notified by DTLS layer about any session negotiation problems. There is a PR which seems to address that: zephyrproject-rtos/zephyr#37794

Reported by @mniestroj

Hardware Target

nrf9160dk_nrf9160_ns

Golioth Dsdk Commit Hash

440b700

What Host OS are you using?

Ubuntu

What happened?

I am evaluating using Golioth for my company for one of our projects that is on NCS 1.8.0

I tried both the samples provided in the Golioth SDK and making my own test sample:

void main(void)
{
    int counter = 0;
    int err;

    LOG_DBG("Start Logging sample");

    err = lte_lc_init_and_connect_async(lte_handler);
    if (err)
    {
        printk("lte_lc_init_and_connect_async, error: %d\n", err);
        return;
    }

    k_sem_take(&lte_connected, K_FOREVER);

    client->on_message = golioth_on_message;
    golioth_system_client_start();

    while (true)
    {
        LOG_INF("INFO: the counter is %d", counter);

        counter++;

        k_sleep(K_SECONDS(5));
    }
}

On NCS 1.7.1 this works as intended -> The logs reach your servers
On NCS 1.8.0 this does not work, a Failed to connect: -95 is reported.

Is this a known issue? Are there any configuration changes required to make this work?

Regards,
Tjaž

Relevant log output

On NCS 1.7.1

uart:~$ *** Booting Zephyr OS build v2.6.99-ncs1-1  ***
[00:00:00.209,167] <inf> golioth_system: Initializing
[00:00:00.209,472] <dbg> main.main: Start Logging sample
uart:~$ Connected to: home network
[00:00:02.420,623] <inf> main: INFO: the counter is 0
[00:00:02.420,745] <inf> golioth_system: Starting connect
[00:00:02.512,390] <inf> golioth_system: Client connected!
[00:00:07.420,715] <inf> main: INFO: the counter is 1
[00:00:12.420,806] <inf> main: INFO: the counter is 2
[00:00:17.420,928] <inf> main: INFO: the counter is 3
[00:00:22.421,020] <inf> main: INFO: the counter is 4
...

--------------------------------------------------------
On NCS 1.8.0

uart:~$ *** Booting Zephyr OS build v2.7.0-ncs1  ***
[00:00:00.208,129] <inf> golioth_system: Initializing
[00:00:00.208,526] <dbg> main.main: Start Logging sample
uart:~$ Connected to: home network
[00:00:04.649,475] <inf> main: INFO: the counter is 0
[00:00:04.649,597] <inf> golioth_system: Starting connect
[00:00:05.043,823] <err> golioth_system: Failed to connect: -95
[00:00:05.043,823] <wrn> golioth_system: Failed to connect: -95
[00:00:09.649,627] <inf> main: INFO: the counter is 1
[00:00:10.043,945] <inf> golioth_system: Starting connect
[00:00:13.044,189] <err> golioth_system: Failed to connect: -95
[00:00:13.044,219] <wrn> golioth_system: Failed to connect: -95
[00:00:14.649,719] <inf> main: INFO: the counter is 2
[00:00:18.044,281] <inf> golioth_system: Starting connect
[00:00:19.649,810] <inf> main: INFO: the counter is 3
...

Code of Conduct

• I agree to follow this project's Code of Conduct

bounded by mutex in get sample:
https://github.com/golioth/zephyr-sdk/blob/ff39bf1c51e8185964c7de8b253ca4d657393ca8/samples/lightdb/get/src/main.c#L59

not bounded in observe sample
https://github.com/golioth/zephyr-sdk/blob/ff39bf1c51e8185964c7de8b253ca4d657393ca8/samples/lightdb/observe/src/main.c#L91