monome / norns-image Goto Github PK

Seems like in some cases (haven't fully pinned this down yet, seems to be at least straight after booting) I'm not getting an IP address in the "stats" menu even though I'm connected to wifi.
Once I go into system > wifi and back to the root menu with the stats the IP is shown.
I guess we're missing a refresh of the IP address on that page.

it appears as if the ~we/norns-image directory included in the 220306 release image is cloned from a personal fork and not https://github.com/monome/norns-image.git

additionally there are local (uncommitted) changes to setup.sh which may or may not be included in this repo.

randomly there can be a stop-job that makes power-off take a long time (a full 1:30)

it's sometimes more frequent when battery powered, but not always.

to replicate:

• when battery powered, sudo shutdown now (via wifi or norns menu)
• then connect usb-serial to laptop
• you'll see the count down

we need to identify why and which job takes so long at shutdown and make sure we have a quick shutdown. it's very confusing to have the device not shut down when you don't have a terminal hooked up.

instructions (including photos) for using usb-disk mode and reading/writing the complete disk image.

this will not be a common procedure, but just in case.

received a report that the battery went from 100 to zero within an hour, but the unit ran another hour.

We'll need to figure out how to automatically build the image, especially useful during development so everyone who has a device is actually using the same image, but obviously also nice once real releases start happening.

Current notes for how to build the image live here https://github.com/tehn/norns-image/blob/master/image.txt

Followup from #66

We'll have to determine how we want to update the firmware related stuff (that's all the firmware related files in /boot, but also the tools like vcgencmd and the libs to make these tools work. There are packages in raspbian for them:

dpkg -L raspberrypi-bootloader
/.
/boot
/boot/LICENCE.broadcom
/boot/bootcode.bin
/boot/fixup.dat
/boot/fixup_cd.dat
/boot/fixup_db.dat
/boot/fixup_x.dat
/boot/start.elf
/boot/start_cd.elf
/boot/start_db.elf
/boot/start_x.elf
/usr
/usr/share
/usr/share/doc
/usr/share/doc/raspberrypi-bootloader
/usr/share/doc/raspberrypi-bootloader/changelog.Debian.gz
/usr/share/doc/raspberrypi-bootloader/copyright
we@norns:~$ dpkg -L libraspberrypi-bin
/.
/opt
/opt/vc
/opt/vc/bin
/opt/vc/bin/containers_check_frame_int
/opt/vc/bin/containers_datagram_receiver
/opt/vc/bin/containers_datagram_sender
/opt/vc/bin/containers_dump_pktfile
/opt/vc/bin/containers_rtp_decoder
/opt/vc/bin/containers_stream_client
/opt/vc/bin/containers_stream_server
/opt/vc/bin/containers_test
/opt/vc/bin/containers_test_bits
/opt/vc/bin/containers_test_uri
/opt/vc/bin/containers_uri_pipe
/opt/vc/bin/dtmerge
/opt/vc/bin/dtoverlay
/opt/vc/bin/dtoverlay-post
/opt/vc/bin/dtoverlay-pre
/opt/vc/bin/edidparser
/opt/vc/bin/mmal_vc_diag
/opt/vc/bin/raspistill
/opt/vc/bin/raspivid
/opt/vc/bin/raspividyuv
/opt/vc/bin/raspiyuv
/opt/vc/bin/tvservice
/opt/vc/bin/vcdbg
/opt/vc/bin/vcgencmd
/opt/vc/bin/vchiq_test
/opt/vc/bin/vcmailbox
/opt/vc/bin/vcsmem
/opt/vc/sbin
/opt/vc/sbin/vcfiled
/usr
/usr/bin
/usr/share
/usr/share/doc
/usr/share/doc/libraspberrypi-bin
/usr/share/doc/libraspberrypi-bin/LICENCE
/usr/share/doc/libraspberrypi-bin/changelog.Debian.gz
/usr/share/doc/libraspberrypi-bin/copyright
/opt/vc/bin/dtparam
/usr/bin/dtmerge
/usr/bin/dtoverlay
/usr/bin/dtoverlay-post
/usr/bin/dtoverlay-pre
/usr/bin/dtparam
/usr/bin/edidparser
/usr/bin/raspistill
/usr/bin/raspivid
/usr/bin/raspividyuv
/usr/bin/raspiyuv
/usr/bin/tvservice
/usr/bin/vcdbg
/usr/bin/vcgencmd
/usr/bin/vchiq_test

So we can update them via apt-get but there's no way to to guarantee idempotency for the result since we can't fix the version of these packages. In other words at a different point in time you'll get a different result, not a good thing for guaranteeing working results.

On the other hand I don't really want to be managing/installing these files manually or via scripts, so maybe our best bet is to mirror the .deb packages for the current version of them (1.20190215-1 for all of them at the moment) and install them that way? And then periodically update them in tandem with the kernel.

you can query current consumption with

cat /sys/class/power_supply/bq27441-0/current_now

this shows charge (positive values) and discharge (negative). i've added monitoring of this value inside the norns status window for debugging. the number is weird, has 3 extra zeros suggesting micro-amps, so ignore the three 0's and you get mA.

so the issue at hand: reducing power consumption everywhere possible when appropriate.

one method is soft-disconnecting the wifi dongle. i'm not sure we can tell the hub to electrically disconnect it, but it seems it should be able to be put in usb-suspend mode. initial googling gave me this:

sudo sh -c "echo 0 > /sys/bus/usb/devices/1-1.4/authorized"

where my wifi is plugged into 1-4 based on lsusb -t (furthest from the usb power)

it surely drops current consumption, but it's still taking 50-60ma more than when it's fully pulled out.

i'll track down other possible power-savers but i think this is going to be the major one. when wifi is on the dongle pulls a little over 200ma is is major. i'm thinking wifi should be OFF by default on powerup.

This has been bugging me for a while so I figured I should create an issue for it.

I'm wondering how we're going to keep the necessary processes running. And what should happen in case any of them crashes or hangs but doesn't crash?
Do we have any idea about how we want to handle this?

This is mainly triggered by the fact that supercollider (or more specifically sclang) doesn't really seem to be made for running in the background, see supercollider/supercollider#2655.

As far as I know we have the following components that need to be running, if I missed anything please let me know :)

• jack
• scsynth
• sclang/crone
• matron
• maiden

We also have ws-wrapper, which would wrap sclang/crone and matron. I haven't looked into it's internals but does this properly handle/pass signals? Does it reap zombie (sub)processes? And what happens when ws-wrapper itself dies?

Since these are all effectively daemons for our purposes normally they would be executed as a daemon by some process supervisor that can guarantee that when it dies it will be restarted. We currently don't have one yet on the buildroot image so we'll need to pick one if we choose to go this way.
But that does mean the components need to be able to run as a daemon which currently isn't really the case.

Going through the Raspberry Pi build instructions in #52 and as of Mon Jul 23 02:48:01 UTC 2018 it looks like catfact.net is offline. Not sure which Debian packages are included from that repo...

I'm using a WM8731 DAC and would like to make use of the built-in headphone amp in that chip. Norns hardware seems to have a separate chip handling the headphone output (and uses some i2c code to change those values).

I can't seem to find where norns is sending alsa/jack volume change commands for the main outputs. I figured this might be a good place to look for implementing headphone controls with the WM8731.

Any suggestions?

need build guide for off-the-shelf pi 3

systemd units should have explicit (Requires) dependencies as illustrated below:

• norns-init
  • norns-jack (also require sound.target)
    • norns-crone
  • norns-matron (also require sockets.target)
  • norns-maiden

norns.target should explicitly require all of the above.

• where should the norns software live?
• should /home be its own partition?

when loading the driver at boot via dtoverlay the OLED is much less stable than manually loading it with modprobe.

redrawing of frames sometimes aborts mid-draw, and sometimes pixels are lost/skipped which shifts the redraw by an offset.

need to test further if this is an issue with the DC manual wiring to pin 5-- some sort of electrical unreliability.

manual driver load command is sudo modprobe fbtft_device custom name=fb_ssd1322 width=128 height=64 speed=16000000 gpios=dc:7,reset:6

Not really an issue, but I'd like to make sure we have a solid understanding of our requirements for norns's OS before we decide how to build it.

From the discussions so far I've distilled the following end-user targeted requirements:

• Should boot quickly. More specifically, should show signs of life to the user max 6s after the device is started by holding down button 1 for 3s
• The user should be able to apply updates without opening the device
• Applying updates should be safe/not brick the device
• Updates should not wipe the user's work
• Minimum of 2GB storage available to the user
• The configuration of the kernel should enable all norns hardware to work out of the box
• Some additional drivers should be enabled as kernel modules to allow them to be used with norns out of the box. For example MIDI USB devices. Which ones t.b.d.
• Kernel/console output should be available on mini-usb on the back
• Mobile/battery powered usage should be possible, currently we have no minimum amount of hours battery life defined

And the following development related requirements:

• Be able to have a quick develop, build, test feedback cycle.
• Be able to develop using the developers desired development tools

i nuked ipc_wrapper from the norns repo. so the start scripts are broken (sorry)

they should use websocket wrapper right? i haven't changed it myself cause i'm not sure of the appropriate URLs to use.

I did flash the 200424 image. I could boot it successfully in the monome shield (200323)
However:

• the display was flipped (upside down)
• the encoder knobs have no effect
• the buttons have no effect
• "awake" does not play on startup

Is this a common bug or only on my shield? ("200106" did work fine)

From the build readme (build-dev-image.md), the kernel download url here comes back with "403 Forbidden"

wget https://monome.nyc3.digitaloceanspaces.com/kernel-4.9.59-rt52-0.0.4.tar.gz

so we don't have to run a script manually

GPIO Pins are listed here:
https://github.com/monome/norns-image/blob/master/readme-hardware.md

This is confusing as I think what's listed is the actual pin # rather than the GPIO#

Also - Pin 39 (GPIO39 as listed) is GND

Are these the correct assignments? Where is the dt_overlay for these?

Here's a reference image showing pin number as well as GPIO#

Both norns.local hostname and matron's _osc._udp service are unpublished in an unknown timeframe after starting wifi.

lots of tiny issues to navigate

https://community.blokas.io/t/supercollider-headless-autostart-and-or-single-click-sh/246/23

https://github.com/notro/fbtft/wiki/Bootsplash

Every now and then I get the following errors in the kernel's log (dmesg or journalctl -t kernel):

May 24 05:18:35 norns kernel: bq27xxx-battery 1-0055: error reading temperature
May 24 05:21:55 norns kernel: bq27xxx-battery 1-0055: error reading current
May 24 05:21:55 norns kernel: power_supply bq27441-0: driver failed to report `current_now' property: -121

mainly optimize systemd and remove anything that can be postponed, but here are some interesting data points: https://www.raspberrypi.org/forums/viewtopic.php?f=72&t=84734

for example, defer wifi connection until after main norns app bootup

/usr/share/locale takes up 133MB over 190 locales, most of which will never be used. The image could be stripped off them to save space, which comes at a premium on the old Compute Modules.

• wifi off
• wifi scan (returns list of detected networks, one network per line)
• wifi select (two args: SSID, password)
• wifi on (activates last used network)
• wifi hotspot (makes a passworded network to connect to from a laptop)

wpa_supplicant should handle basically all of this, and then some hotspot reference here: https://frillip.com/using-your-raspberry-pi-3-as-a-wifi-access-point-with-hostapd/

use case scenario: hotspot mode can be activated via the screen interface. you then connect a laptop via wifi, set up connection to main wifi router, then have the norns connect to the router so it can talk to the internet when needed.

When running the 190303 beta image on a CM3+ I get the following error message when reading the CPU temperature

we@norns:~$ vcgencmd measure_temp
VCHI initialization failed
we@norns:~$ echo $?
255

This is probably caused by the firmware not matching with the vcgencmd binary.
Need to figure out how to update both the firmware and the other stuff from https://github.com/raspberrypi/firmware without updating the kernel.

oled module setup at boot, so we don't have to manually run modprobe script

https://github.com/alidaf/raspberryPi/blob/master/displayPi/ssd1322-spi/fbtft/ssd1322-spi-overlay.dts
https://github.com/alidaf/raspberryPi/wiki/SSD1322(SPI)--fbtft-framebuffer-driver

On OG aluminium Norns with 4GB of storage, the new image takes up quite a bit of space. There's less than a 1GB left for user data after a clean install so it would be nice if we could somehow make logging more efficient.

I'm looking at daemon.log and find it pretty big after ~12 hours of runtime. It's 66MB with a number of repeated lines (sorted here):

 258721 norns jackd[384]: JackAudioDriver::ProcessGraphAsyncMaster: Process error
 246914 norns jackd[384]: JackEngine::XRun: client = softcut was not finished, state = Running
  76709 norns jackd[453]: JackAudioDriver::ProcessGraphAsyncMaster: Process error
  73652 norns jackd[453]: JackEngine::XRun: client = softcut was not finished, state = Running
   6375 norns jackd[384]: JackEngine::XRun: client = softcut was not finished, state = Triggered
   5636 norns jackd[384]: JackEngine::XRun: client softcut finished after current callback
   2804 norns ws-wrapper[2550]: station  0  queing next file  1  of  1
   2804 norns ws-wrapper[2550]: station 0 playing file /dev/shm/weather.wav
   1937 norns ws-wrapper[2550]: MP3.start: command to execute is:
   1675 norns jackd[453]: JackEngine::XRun: client = softcut was not finished, state = Triggered

As you can see, there's 250 thousand xruns reported. This log survives a restart.

in the process of trying to squeeze out more performance for MLR, i belatedly realized that the audio glitches we experience due to CPU load are not typically symptomatic of jack/ALSA buffer underruns.

• these are kind of drawn-out micro-stutters that start being apparent when scsynth usage approaches ~50% of one CPU in top
• top also shows that the CPU load distribution is pretty reasonable, and using tasksel to forcefully redistribute processes has no effect.
• no xruns are reported by jack in the output of scsynth.

so, i dug a little more and gathered a few more data points:

• glitchiness seemed unaffected by ALSA period size (default 128, tried up to 1024.)

• ran the system using the CM3's default kernel7.img instead of the realtime kernel. this has no support for monome-snd so used a USB audio interface. no glitches! our heaviest use case is reading+writing with all four voices in MLR, plus reverb and compression; scsynth uses ~60% CPU with no audible ill effects.

• tried setting elevator=deadline in /boot/commandline.txt, this seems to have no effect with the default kernel and makes the artifacts in the RT kernel noticeably worse.

• USB soundcard on the RT kernel is just as bad, which seems to rule out a monome-snd driver problem.

so i think we should try without the RT patches. we can't do everything we want using <50% of one core for scsynth.

AFAICT, this PR (which was merged to norns:dev) requires that scsynth be running instead of supernova, but norns-image:dev is still launching supernova (https://github.com/monome/norns-image/blob/dev/config/norns-supernova.service)

Noticed cron is still started (by cron.service), since we don't have any need for it I think we can/should disable it.
@artfwo what do you think?

In some cases USB soundcards get ALSA card ID 0, which is the one we use to run JACK. This means sound is going to the wrong soundcard, or in some cases (MIDI devices without audio) JACK is unable to start.

I've been unable to reproduce this on the buildroot image, it might also not happen on the norns hardware/image itself but only when running on a regular RPi with raspbian, still gathering info.

Output on norns running the buildroot image:

# cat /proc/asound/cards
 0 [sndrpimonome   ]: snd_rpi_monome - snd_rpi_monome
                      snd_rpi_monome
 1 [eX             ]: USB-Audio - ESI MIDIMATE eX
                      Ploytec GmbH ESI MIDIMATE eX at usb-3f980000.usb-1.1, full speed
# 

not sure why yet.

I guess the simplest way would be to use hw:sndrpimonome instead of hw:0 in the JACK service https://github.com/monome/norns-image/blob/master/config/norns-jack.service#L11

Alternatively we could look into making sure monome-snd always gets ID 0. The advantage to this would be that there would be no need to change any config in case the norns stack is used with a different setup/board which uses a different soundcard. The OS would just need to be configured correctly to ensure the desired soundcard is at ID 0.

Currently ws-wrapper logs every message received through OSC which might become a large amount of data if this is automated modulation. Case in point, after making a MIDI to Norns script parameter automation M4L device, I set it up automated via Ableton Live as part of longterm testing of a slightly overclocked Norns.

After around 24 hours of runtime, I can see daemon.log being 350MB big (!!!) with

• 2.5 million occurrences of norns ws-wrapper[512]: setting parameter /release to ...
• 1.25 million occurrences of norns ws-wrapper[512]: setting parameter /pw to ...
• 284 thousand occurrences of norns ws-wrapper[512]: setting parameter /cutoff to ...
• 181 thousand occurrences of norns ws-wrapper[512]: setting parameter /delay_rate to ...
• and so on, and so on

You get my point.

I understand that:

• people are unlikely to run Norns for 24+ hours at a time; and
• logrotate is supposed to compress old logs

However, I think that:

• 24 sessions of 1 hour will end up with the same amount of log data; and
• writing this information to disk shortens the lifespan of the eMMC which is not replaceable without replacing the entire compute module which is not easily obtainable these days

Therefore, I think ws-wrapper shouldn't log this activity unless instructed via sending it /debug 1 or some Unix signal.

This issue is about the problems with the included wifi nub, purely on an OS level. The integration with matron/the menu is out of scope. Same for hotspot related issues.

We've been having quiet a lot of reports from users and experiences ourselves with the wifi being unreliable. We should figure out why that is and try to find a solution to make it work reliably.

Symptoms seem a bit random, for example the network connection works fine (ping google.com no drops), reboot, works fine, reboot again, works fine, reboot another time, it doesn't work anymore.

As the first part of #9 I'd like to start with automatically building the kernel image.
To be sure I'm building the right thing I'd like to check the requirements I've gathered so far, mainly from image.txt and looking through the issues:

• Kernel sources from https://github.com/raspberrypi/linux.git
• In image.txt raspberrypi/linux@807c5d9 is used as a specific commit to checkout. I guess we want to track the stable tags provided by the raspberry kernel repo? For example https://github.com/raspberrypi/linux/releases/tag/raspberrypi-kernel_1.20171029-1
• Apply rt patches from https://www.kernel.org/pub/linux/kernel/projects/rt/4.9/older/
• Apply https://raw.githubusercontent.com/fedberry/kernel/master/usb-dwc_otg-fix-system-lockup-when-interrupts-are-threaded.patch to make sure USB OTG FIQ still works.
  Does anyone know if this can be/will be upstreamed to the rpi3 kernel?
• Add the monome-snd driver
• Add/enable fbtft driver
• Start config with bcm2709_defconfig
• Apply custom kernel config. What exactly? image.txt lists the following:
  • Enable RT
  • Enable "BQ27xxx battery driver (+I2C)"
  • Enable monome-snd
  • Disable DEBUG_INFO

Also there was mention of using https://github.com/raspberrypi/linux/blob/5433720f3e8d6d06a59b30cce44210d814f34b0b/drivers/input/misc/rotary_encoder.c in the norns repo, is this still something we want?

Ideally I'd like to build a kernel that works on the RPi3 as well as when using QEMU, but I'll have to look into if that still requires any patches or changes to the config.