proj_template/ make prog doesn't select which python3 it needs about cfu-playground HOT 24 CLOSED

Hi Marc: Thanks for the bug reports. There's a lot to unpack here.

We haven't paid much attention to the build environment, yet. It's probably time.

To perhaps get you over a hump:

• Any modern Python (3.7+) should work fine.
• I use a python virtualenv for cfu-playground. I enter it before running scripts/setup, and also in each new shell.
• Install requests via pip (pypi link)

With regards to a more general solution, @tcal-x maybe we should consider litex-buildenv again?

from cfu-playground.

Probably make-env (https://github.com/SymbiFlow/make-env) is a better solution than litex-buildenv now.

from cfu-playground.

Thanks @alanvgreen I'll give it another shot. The main issue is that the published steps don't work as is, and I think assume that you're doing extra steps with virtualenv, that aren't listed.
I do have python 3.9 on my debian system, but clearly there are missing steps on having the correct python with the correct modules installed. It seems to depend on a bunch of modules (understandably) without installing them (or the install failed silently for me?)
I'll have another look, or if you know of obvious steps that are missing in the doc, please let me know :)

from cfu-playground.

@mithro will that work for non-Symbiflow based toolchains?

@marcmerlin yes, something like litex-buildenv, make-env or maybe a Docker container are likely the best solution here. Even Python virtualenvs aren't really hermetic.

from cfu-playground.

I agree that a standardized build environment is the right way to go here. If we do want a short-term solution, we could consider adding the requests package into third_party/python like we have with migen, litex, litex-boards etc.

from cfu-playground.

Basic instructions will help I think. The first link I found doesn't even have working info:

sauron:~/fpga/CFU-Playground$ python3 -m venv env
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt-get install python3-venv

You may need to use sudo with that command.  After installing the python3-venv
package, recreate your virtual environment.

Failing command: ['/home/merlin/fpga/CFU-Playground/env/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']
sauron:~/fpga/CFU-Playground$ sudo apt-get install python3-venv
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-venv is already the newest version (3.9.2-3).
sauron:~/fpga/CFU-Playground$ dpkg -S ensurepip
libpython2.7-stdlib:amd64: /usr/lib/python2.7/ensurepip/__init__.py
libpython2.7-stdlib:amd64: /usr/lib/python2.7/ensurepip
libpython2.7-stdlib:amd64: /usr/lib/python2.7/ensurepip/__main__.py
python3.9-venv: /usr/lib/python3.9/ensurepip/__init__.py
python3.9-venv: /usr/lib/python3.9/ensurepip/_uninstall.py
python3.9-venv: /usr/lib/python3.9/ensurepip
libpython2.7-stdlib:amd64: /usr/lib/python2.7/ensurepip/_uninstall.py
python3.9-venv: /usr/lib/python3.9/ensurepip/__main__.py
sauron:~/fpga/CFU-Playground$ python --version
Python 3.9.2

looks like

sauron:~/fpga/CFU-Playground$ virtualenv env
sauron:~/fpga/CFU-Playground$ type python3; source env/bin/activate; type python3
python3 is hashed (/home/merlin/fpga/fomu-toolchain-linux/bin/python3)
python3 is /home/merlin/fpga/CFU-Playground/env/bin/python3
(env) sauron:~/fpga/CFU-Playground$ scripts/setup

is better even though that doc tells you not to use that for python3.

but still, things don't work after that

(env) sauron:~/fpga/CFU-Playground$ cd proj/proj_template
(env) sauron:~/fpga/CFU-Playground/proj/proj_template$ make prog
/home/merlin/fpga/CFU-Playground/scripts/pyrun /home/merlin/fpga/CFU-Playground/proj/proj_template/cfu_gen.py
Traceback (most recent call last):
  File "/home/merlin/fpga/CFU-Playground/proj/proj_template/cfu_gen.py", line 38, in <module>
    main()
  File "/home/merlin/fpga/CFU-Playground/proj/proj_template/cfu_gen.py", line 31, in main
    new_verilog = verilog.convert(cfu, name='Cfu', ports=cfu.ports)
  File "/home/merlin/fpga/CFU-Playground/third_party/python/nmigen/nmigen/back/verilog.py", line 61, in convert
    return _convert_rtlil_text(rtlil_text, strip_internal_attrs=strip_internal_attrs)
  File "/home/merlin/fpga/CFU-Playground/third_party/python/nmigen/nmigen/back/verilog.py", line 10, in _convert_rtlil_text
    yosys = find_yosys(lambda ver: ver >= (0, 9))
  File "/home/merlin/fpga/CFU-Playground/third_party/python/nmigen/nmigen/_toolchain/yosys.py", line 228, in find_yosys
    raise YosysError("Could not find an acceptable Yosys binary. The `nmigen-yosys` PyPI "
nmigen._toolchain.yosys.YosysError: Could not find an acceptable Yosys binary. The `nmigen-yosys` PyPI package, if available for this platform, can be used as fallback
make: *** [../proj.mk:187: generate_cfu] Error 1

from cfu-playground.

Ok, looks like the instructions are also missing this for the virtualenv (I found this in the old nMigen Workshop)

pip install --upgrade 'git+https://github.com/nmigen/nmigen.git#egg=nmigen[builtin-yosys]'

after that make prog and make load worked better, although I expected make load to install the binary that just took a while to build, and instead it downloaded tensorflow and built a lot more things, which was both weird and unexpected.

from cfu-playground.

Hi Marc,

I've been using CFU Playground without any type of venv, since the pyrun script sets up the python paths. I can't remember if I had to do anything for requests. But yes, this is far from ideal, since as you say 'which python3' is left up to the system.

Not using venv does allow me to then get SymbiFlow for Xilinx from conda packages -- I set up the environment from my symbiflow-examples installation, then come back to CFU-Playground. But this is not a real solution for users.

from cfu-playground.

Ok, I'm writing more specific instructions, and got 'make' to work, although it's launching renode and expecting it to be there, with no instructions on which renode to get or nothing that installs it (I had renode somewhere, so I added it to $PATH and that worked).
Should I just add 'apt-get install renode' to the instructions?

Next step, Arty A7 adds 2 ttyUSB each time and that confuses make load

[3880517.882385] ftdi_sio 1-6:1.0: FTDI USB Serial Device converter detected
[3880517.882417] usb 1-6: Detected FT2232H
[3880517.882630] usb 1-6: FTDI USB Serial Device converter now attached to ttyUSB0
[3880517.884687] ftdi_sio 1-6:1.1: FTDI USB Serial Device converter detected
[3880517.884704] usb 1-6: Detected FT2232H
[3880517.884895] usb 1-6: FTDI USB Serial Device converter now attached to ttyUSB1

(CFU-Playground) sauron:~/fpga/CFU-Playground/proj/proj_template$ make load
Error: could not determine unique TTY
TTY possibilities: /dev/ttyUSB0 /dev/ttyUSB1

I can force the tty in ../proj.mk", although I tried both ttyUSB0 and ttyUSB1 and it the last line I now see is

/home/merlin/fpga/CFU-Playground/soc/bin/litex_term --speed 3686400  --kernel /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin /dev/ttyUSB0

or the same with ttyUSB1
Does not look like anything actually flashes.

I tried a flasher that's worked for me before, but I guess the .bin is not a compatible file for it
sauron:~/fpga/CFU-Playground/proj/proj_template$ xc3sprog -c nexys4 /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin

from cfu-playground.

Thanks Marc,

#4 is the Renode issue.

Let me check which release the apt-installed package is before we suggest going that route.

from cfu-playground.

About using the Arty board; before our bitstream is loaded, there do appear to be two terminal devices. If you've loaded or flashed a Verilog design, probably neither of them will respond.

If the Arty has the original built-in demo configured (you see different LED patterns when BTN1 or BTN2 is pushed), then one of the TTYs will be active with some very basic printout that will respond to button pushes on the board. Edit: Use 115200 for the speed/baud in this case.

To load the CFU Playground, first you must build and upload the gateware bitstream using "make prog". If it is successful, you will see the LiteX "led chaser" pattern cycling light/dark through four LEDs. At that point, you can try the "make load" target.

from cfu-playground.

Sorry, I was being an idiot and typed 'make load' before 'make prog' (well I had done make prog, but rebooted the device, and it was running the wrong code).
Did make prog again, it worked, and I get the LED chaser pattern.

INFO:SoC:--------------------------------------------------------------------------------
Open On-Chip Debugger 0.11.0-rc2
Licensed under GNU GPL v2
For bug reports, read
	http://openocd.org/doc/doxygen/bugs.html
DEPRECATED! use 'adapter driver' not 'interface'
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
DEPRECATED! use 'adapter speed' not 'adapter_khz'
fpga_program
Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi_tdo_sample_edge falling"
Info : clock speed 25000 kHz
Info : JTAG tap: xc7.tap tap/device found: 0x0362d093 (mfg: 0x049 (Xilinx), part: 0x362d, ver: 0x0)

But make load still seems to hang and do nothing (but indeed it doesn't complain about the 2 ports anymore, now I only have ttyUSB1):

Running interactively on FPGA Board
make -C /home/merlin/fpga/CFU-Playground/soc -f /home/merlin/fpga/CFU-Playground/soc/common_soc.mk load_hook
make[1]: Entering directory '/home/merlin/fpga/CFU-Playground/soc'
MAKEFLAGS=-j8 /home/merlin/fpga/CFU-Playground/scripts/pyrun ./common_soc.py --output-dir build/digilent_arty.proj_template --csr-csv build/digilent_arty.proj_template/csr.csv --cpu-cfu /home/merlin/fpga/CFU-Playground/proj/proj_template/cfu.v --uart-baudrate 3686400 --target digilent_arty --software-load --software-path /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin
make[1]: Leaving directory '/home/merlin/fpga/CFU-Playground/soc'
/home/merlin/fpga/CFU-Playground/soc/bin/litex_term --speed 3686400  --kernel /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin /dev/ttyUSB1

from cfu-playground.

With Renode, I don't even see an apt package any more. So maybe a direct download of their "Renode portable" is how we'll need to go, unless we clone and build from source.

from cfu-playground.

@marcmerlin , two things to try now:

• hit return; if you get a "litex>" prompt, type "serialboot"

or

• press the "RESET" button in the corner of the Arty board

Hopefully you'll see the LiteX banner, and then a progress bar as the software binary is loaded, and finally the CFU Playground menu.

from cfu-playground.

We really should make it easier (or a more automated solution) to detect a FPGA gateware and firmware mismatch.

from cfu-playground.

return didn't work for a long time, but when I came back way later, return worked, but the load didn't work, I just get a basic litex prompt

/home/merlin/fpga/CFU-Playground/soc/bin/litex_term --speed 3686400  --kernel /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin /dev/ttyUSB1
 
litex> 
litex> 1
Command not found
litex> help

LiteX BIOS, available commands:

aaah, but then I typed 'reboot' and it worked:

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin to 0x40000000 (879876 bytes)...
[LXTERM] Upload complete (317.9KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

--============= Liftoff! ===============--
Hello, World!
initTfLite()

CFU Playground
==============
 1: TfLM Models menu
 2: Functional CFU Tests
 3: Project menu
 4: Performance Counter Tests
 5: TFLite Unit Tests
 6: Benchmarks
 7: Util Tests

Is that the expected workflow? If so, I'll do my best to document it and send a PR on the doc page to clear all that up

from cfu-playground.

@marcmerlin , I'm not sure what the contributing factors are --- maybe the board, or the cable, or the host OS --- but sometimes you need these extra steps, sometimes you don't. With my setup, I could not find a way to NOT get to the CFU Playground menu.

But certainly these hints should be added for when the board appears unresponsive. Thank you !

from cfu-playground.

It may be because my board has existing firmware that I burned on it during a linux.conf.au miniconf with @mithro . If I hard reboot the board, it still boots back into whatever firmware we flashed back a few years back.
However, I can confirm that either

1. typing 'reboot'
2. clicking the reset button
   reboots litex and it receives the boot file from

Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[LXTERM] Received firmware download request from the device.
[LXTERM] Uploading /home/merlin/fpga/CFU-Playground/proj/proj_template/build/software.bin to 0x40000000 (879876 bytes)...
[LXTERM] Upload complete (316.1KB/s).
[LXTERM] Booting the device.
[LXTERM] Done.
Executing booted program at 0x40000000

Ok, thanks for all your help, I'll write up some updated instructions and send you a PR

from cfu-playground.

ok, I'm sure it's not perfect but it's better than the existing version ;)
#135

Note that I really wish this were a github wiki, it would be easier to render within github and check if things work , including autolinks.
:doc:vivado-install didn't work, not sure why, so I just put a full web link instead.

I only test rendered the change inside github, I hope it's close enough to how it'll look on readthedocs.io.

from cfu-playground.

Thankyou for the PR!

For render testing, check on the PR #135 page, under the CI test results, there's a link to a (preview on the RTD site)[https://cfu-playground--135.org.readthedocs.build/en/135/]. I find this sufficient for most purposes.

On the RTD site, there's also instructions for (building docs locally)[https://cfu-playground.readthedocs.io/en/latest/docs/index.html], though you might not want to open that can of worms just yet!

from cfu-playground.

Thanks @alanvgreen for the tip. The rendered version looks reasonable.
I'll wait for you or @tcal-x, or whoever appropriate to review it. Thanks :)

from cfu-playground.

Hi Marc, I'd like to understand why the additional nmigen installation was necessary for you. I'll probably try an installation on a clean VM, although I'm not sure when I'll get to that.

from cfu-playground.

@tcal-x PTAL, it was missing and I had to install it manually once, but whatever changed, I did a clean install from scratch and was able to do without that added command, so I removed it from the docs.
They should be what you'd like now, hopefully this is now good to commit as is.

from cfu-playground.

#135 is in, so I think we can close this

from cfu-playground.