Check that OPKG update works
Check that OPKG install works (doesn't fail) for some stable package

There are two parts to this task:

The physical setup needs to ensure that, even with the usb kmod failing, the ethernet connection lets boardfarm download an image to fix itself.

Apart from that the software needs to be able to recover itself from a fail state and flash a stable image, so that any PR that breaks boardfarm won't break all other PRs.

Acceptance Criteria:

• If NAND is empty, still able to program the NAND
• Use bootloader to flash, firmware0 by default and gets image from a accessible IP location (not from USB)
• Flash the image generated by Jenkins PR job
• Steps from Flashing from UBoot Prompt

Because the packages don't exist yet for our arch. We should switch to some "core" packages for these tests or make a full release for lede

Testlink: CK-106
Basic Ping test to a dumb Clicker at a known address

InterfacesShow is passing now while it should be failing due to eth0 showing up as unknown, the test needs to properly differentiate between the interfaces to figure out how many of them should show up with a proper UP/DOWN status.

We should be checking whether we can access Uboot variables after flashing, a simple test to check if fw_printenv prints boot_partition value and returns a zero exit code would be enough.

Confirm that both IPv4 and IPv6 addresses are allocated to each network interface

Failure for OpkgUpdate test:

boardfarm.OpkgUpdate() (from (empty))
Failing for the past 1 build (Since Unstable#282 )
Took 30 sec.
Stacktrace

Traceback (most recent call last):
  File "/home/jenkins/workspace/CreatorDev_openwrt_ci40-LYIXBKXU7P6M7VKINYW64SFJHCZJFZ43LACISJI2LBBWG2Q4GTUA/boardfarm/tests/linux_boot.py", line 72, in testWrapper
    self.runTest()
  File "/home/jenkins/workspace/CreatorDev_openwrt_ci40-LYIXBKXU7P6M7VKINYW64SFJHCZJFZ43LACISJI2LBBWG2Q4GTUA/boardfarm/tests/opkg.py", line 41, in runTest
    board.expect('Updated list of available packages')
  File "/home/jenkins/workspace/CreatorDev_openwrt_ci40-LYIXBKXU7P6M7VKINYW64SFJHCZJFZ43LACISJI2LBBWG2Q4GTUA/boardfarm/devices/openwrt_router.py", line 455, in expect
    return super(OpenWrtRouter, self).expect(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1418, in expect
    timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1433, in expect_list
    timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1535, in expect_loop
    raise TIMEOUT(str(err) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded.
<devices.openwrt_router.OpenWrtRouter object at 0x7f1028cd70d0>
version: 3.1
command: /bin/bash
args: ['/bin/bash', '-c', u'miniterm.py -b 115200 /dev/ttyUSB0']
searcher: <pexpect.searcher_re object at 0x7f1028d5f310>
buffer (last 100 chars): 'ck failed.\r\r\nSignature check failed.\r\r\nSignature check failed.\r\r\nroot@OpenWrt:/# \r\r\nroot@OpenWrt:/# '
before (last 100 chars): 'ck failed.\r\r\nSignature check failed.\r\r\nSignature check failed.\r\r\nroot@OpenWrt:/# \r\r\nroot@OpenWrt:/# '
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 23615
child_fd: 3
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: <devices.base.o_helper instance at 0x7f1028dab488>
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1

Failure for OpkgInstall test:

boardfarm.OpkgInstall() (from (empty))
Failing for the past 1 build (Since Unstable#282 )
Took 30 sec.
Stacktrace

Traceback (most recent call last):
  File "/home/jenkins/workspace/CreatorDev_openwrt_ci40-LYIXBKXU7P6M7VKINYW64SFJHCZJFZ43LACISJI2LBBWG2Q4GTUA/boardfarm/tests/linux_boot.py", line 72, in testWrapper
    self.runTest()
  File "/home/jenkins/workspace/CreatorDev_openwrt_ci40-LYIXBKXU7P6M7VKINYW64SFJHCZJFZ43LACISJI2LBBWG2Q4GTUA/boardfarm/tests/opkg.py", line 57, in runTest
    board.expect("Configuring {}".format(pkg))
  File "/home/jenkins/workspace/CreatorDev_openwrt_ci40-LYIXBKXU7P6M7VKINYW64SFJHCZJFZ43LACISJI2LBBWG2Q4GTUA/boardfarm/devices/openwrt_router.py", line 455, in expect
    return super(OpenWrtRouter, self).expect(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1418, in expect
    timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1433, in expect_list
    timeout, searchwindowsize)
  File "/usr/lib/python2.7/dist-packages/pexpect/__init__.py", line 1535, in expect_loop
    raise TIMEOUT(str(err) + '\n' + str(self))
pexpect.TIMEOUT: Timeout exceeded.
<devices.openwrt_router.OpenWrtRouter object at 0x7f1028cd70d0>
version: 3.1
command: /bin/bash
args: ['/bin/bash', '-c', u'miniterm.py -b 115200 /dev/ttyUSB0']
searcher: <pexpect.searcher_re object at 0x7f1028b99950>
buffer (last 100 chars): 'ed errors:\r\r\n * opkg_install_cmd: Cannot install package nano.\r\r\nroot@OpenWrt:/# \r\r\nroot@OpenWrt:/# '
before (last 100 chars): 'ed errors:\r\r\n * opkg_install_cmd: Cannot install package nano.\r\r\nroot@OpenWrt:/# \r\r\nroot@OpenWrt:/# '
after: <class 'pexpect.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 23615
child_fd: 3
closed: False
timeout: 30
delimiter: <class 'pexpect.EOF'>
logfile: None
logfile_read: <devices.base.o_helper instance at 0x7f1028dab488>
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1

Testlink: CK-132/133

Sysupgrade to and from the last major release version.

Testlink: CK-145

Testlink: CK-136
Onboard device
Check for WLAN IP
Ping AP

Running boardfarm gives this warning in the logs

WARNING: Unknown board model 'ci40'. Please check spelling, or write an appropriate class to handle that kind of board.

Given a CI40 with BT/BLE enabled

• scan for BT/BLE devices
• when a wireless device connects, then connection and ping over BT/BLE is successful

Requires an additional BT/BLE device (RPi or other device) to be in the boardfarm setup

Probably due to the fact that LEDE code is currently configuring eth0 to be a bridged lan where as openwrt configures it as a wan

We need to test upgrades, there are lots of scenarios to test but atm anything is better than nothing.

Acceptance criteria:

• test that a build can upgrade the other nand partition to itself
• confirms a reboot does switch parititions

As a boardfarm developer, I want to check that my boardfarm change has not broken anything before it is merged
As a developer, I want to manually run a hosted creator image through boardfarm, so that I know the creator image extensions haven't broken the core platform.

Intended to check that boardfarm works prior to a PR getting merged to the boardfarm repo.

• Uploads the latest major release (hard code this to 1.0.1 for now)
• Runs boardfarm tests on PR change
• Optional build param allows to add a link to hosted image to be flashed

• Check that OPKG remove works
• Check that OPKG removes (doesn't fail) some stable package with no dependencies
• Check that OPKG removes (doesn't fail) some stable package which has dependencies installed
• Check that OPKG removed (doesn't fail) some kmod package
• Check removing a package which is not even installed, should be handled
• Test must be put through a fail case to ensure that it will report failure correctly

Given a CI40 in softAP mode, when a wireless device connects, then connection and ping over wireless is successful.
Requires an additional WLAN RPi to be in the boardfarm setup

Acceptance

• Runs on weekly builds only (not CI)
• Runs the following for 12 hours
  stress --cpu 20 --io 20 --vm 4 --vm-bytes 16M & iperf3 -c <test-server-ip> -b0 -t0 -i 10 &
• Flags a failure if the board crashes /coredumps/becomes unresponsive

• Check the current version of U-Boot from prompt (can use "strings /dev/mtd0 | grep "U-Boot"" command to find out the version)
• Flash a different version of bootloader using "mtd write" command
• Verify that the new version is flashed correctly using "mtd verify" command

The logs for iperf tests show these error messages /bin/ash: mpstat: not found yes the test still passes?

Currently once programming is finished we reboot the device and wait 180 seconds.

Instead we could just wait for a few seconds (to avoid accidentally interrupting boot loader) and then sit in a loop waiting for the console to appear. This shouldn't take more than 40 seconds. If it does timeout exception should be raised.

Testlink: CK-149

The test logs can be seen in the overall build log but I would expect the test report UI to be able to show full logs of each test. Currently it can only shows a stack trace of failed tests.