Compiling for ESP32 in MSYS2 environment under Windows 10 about lws-esp32-test-server-demos HOT 21 CLOSED

I don't use windows for 15 years or so now. The windows support in lws is all contributed.

So unless it's either contributed or trivial, I'm not going to spend any time on making it work with windows.

My suggestion is use linux in a vm.

Having said that... in case it is trivial... you are doing `make flash_ota monitor" in the top level of the test server project, right? And the results from this won't be useful unless you built and flashed the -factory app first.

from lws-esp32-test-server-demos.

I have considered running the build under a Linux VM but that does have its own issues (file paths no longer match so IDE integration no longer works, mapped folders don't properly symlink inside linux VMs so you end up running smb shares which becomes a pain to manage across development teams, and switching wholesale to linux really isn't an option - it has the worst windowing system and mouse behaviors in existence). Besides the ESP-IDF platform manages to run under windows, to claim ESP-IDF compatibility libwebsockets should really compile under it too.

Anyway, I managed to get it to compile, mostly, by doing the following:

• add -G'MSYS Makefiles' to the cmake command to fix this issue -https://github.com/Alexpux/MINGW-packages/issues/306
• compile genromfs under MSYS2 and add it to the path
• add .exe extension to executables referenced in contrib/corss-esp32.cmake
• add -DLWS_WITH_BUNDLED_ZLIB=0 to cmake command to avoid automatic windows zlib bundling

This gets me to sys/ioctl.h not being found, further debugging seems to indicate the include paths for the cmake compilation are wrong as esp_heap_caps.h is also missing.

from lws-esp32-test-server-demos.

Adding the following to the cross-esp32.cmake fixes the missing includes:

	-I$(IDF_PATH)/components/vfs/include/ \
	-I$(IDF_PATH)/components/soc/include/ \
	-I${IDF_PATH}/components/heap/include \

This now compiles successfully.

I'll investigate how to more legitimately add the exe extension in the cross-esp32.cmake. Currently not sure how to put build platform specific settings into cmake files so will have to investigate further.

Can you explain the genromfs dependency, is this a fundamental part of how libwebsockets work or is this just an artifact of how the resources have been packed for the factory and test apps?

from lws-esp32-test-server-demos.

Nice job... those three includes are needed for esp-idf to build anything here... I added them in the top-level Makefile for -factory and -test-server-demos already... it looks like CFLAGS isn't inherited the same on your setup.

ROMFS isn't part of lws, although lws has its own ROMFS parser.

It's there to enable lws on ESP32 to find assets to serve in the absence of any other filesystem. lws has its own internal VFS and by default the ESP32 platform wires it up to the romfs, so you can serve whatever you put in ./romfs-files (which the Makefile runs genromfs on).

The ROMFS is composed on to the binary along with a JSON manifest of what the image actually is. The factory stuff can read the manifest and confirm it's an OTA image, its name etc.

That is put together using shellscript, which is portable enough to work on eg Mac. But not windows.

genromfs itself is foss and will work on windows https://github.com/chexum/genromfs but the problem is the shellscript I should think.

from lws-esp32-test-server-demos.

So was this decision made because at the time there was no filesystem on the esp32? As I understand it there is now spiffs and fatfs support in the vfs under latest esp-idf.

Just wondering if this dependency can now be removed and replaced with something more esp native.

from lws-esp32-test-server-demos.

Yes, but it was more

• The partitioning is not really fixed on ESP32. Where you VFS storage is or how much of it you want is a variable.

• The things you serve are typically immutable or dynamic. A ROMFS is good for immutable things. If the VFS is writable, it can be trashed.

• lws has its own thin "file ops" vfs

You don't have to use ROMFS, the file ops are set here by default

https://github.com/warmcat/libwebsockets/blob/master/lib/plat/lws-plat-esp32.c#L1707
https://github.com/warmcat/libwebsockets/blob/master/lib/plat/lws-plat-esp32.c#L1341

You can simply override the default fops using the same apis in user code to use provide mapping for open, read, write etc on to the ESP VFS apis.

On nice thing about the lws VFS, it knows how to serve gzip-compressed files directly from inside a ZIP file. I have an ESP32 application here that does that (it requires stock zlib as a component, but that builds without changes).

from lws-esp32-test-server-demos.

I see, that makes sense.

My application will probably require a non-writeable partition and a writeable one, I guess there's nothing stopping me having the static files in the ROMFS partition and a separate FATFS partition for dynamically written files though.

I will investigate updating the genromfs scripts to work on Windows and see if I can get my changes legit so I can file a PR.

Thanks for the help!

from lws-esp32-test-server-demos.

I made some progress getting the compilation to work on windows but be non platform specific. Will file a PR sometime this week.

However, in trying to test it out, I flashed the factory app built with my changes above, this seemed to go well at first. Build resulted in a lws-esp32-factory.bin of 870kb, and a romfs.img of 168kb. Flashed this onto the device and it starts up with the following log:

4: gapss from LWS_GAPSS_INITIAL to LWS_GAPSS_SCAN
I (96) wifi: wifi firmware version: 7a658f3
I (96) wifi: config NVS flash: disabled
I (96) wifi: config nano formating: disabled
I (106) wifi: Init dynamic tx buffer num: 20
I (106) wifi: Init data frame dynamic rx buffer num: 20
I (116) wifi: Init management frame dynamic rx buffer num: 20
I (116) wifi: wifi driver task: 3ffc55e8, prio:23, stack:4096
I (126) wifi: Init static rx buffer num: 20
I (126) wifi: Init dynamic rx buffer num: 20
I (126) wifi: Init rx ampdu len mblock:7
I (136) wifi: Init lldesc rx ampdu entry mblock:4
I (136) wifi: wifi power manager task: 0x3ffce924 prio: 21 stack: 2560
I (166) wifi: mode : sta (24:0a:c4:04:be:f8) + softAP (24:0a:c4:04:be:f9)
4: SYSTEM_EVENT_STA_DISCONNECTED
4: gapss from LWS_GAPSS_SCAN to LWS_GAPSS_SCAN
4: No SSL certs... using port 80
4:  mem: platform fd map:   120 bytes
4: ROMFS length 168KiB
{
 "schema": "lws1",
 "model": "lws",
 "builder": "BEN-DESKTOP",
 "app": "lws-esp32-factory",
 "user": "ben",
 "git": "3f93751-dirty",
 "date": "Mon, Nov  6, 2017  7:13:50 PM",
 "unixtime": "1510024430",
 "file": "lws-esp32-factory-1510024430.bin",
 "factory": "y"
}
4: Creating Vhost 'ap' port 80, 3 protocols, IPv6 off
4: SYSTEM_EVENT_SCAN_DONE
4: scan_finished: count 0
4: setting mount default to factory
4:  heap :190004 (-4964)
4:  heap :189924 (-80)
4: SYSTEM_EVENT_SCAN_DONE
4: scan_finished: count 7
4:  heap :190484 (+560)
4:  heap :190488 (+4)

Looking good, so I connect my phone to the wifi hotspot it brings up, which oddly is named ESP_04BEF9 and not the lws- style ap name suggested in the README. My phone connects to the wifi, pops up the not connected to the internet message. After telling it to stay connected anyway, I navigate to http://192.168.4.1/factory.html and see the request on the log:

I (16436) wifi: n:6 0, o:6 0, ap:6 1, sta:0 0, prof:6
I (16436) wifi: station: ac:37:43:a3:d9:3f join, AID=1, n, 20
4:  heap :189676 (-808)
4: esp32_lws_fops_open: /ap/factory.html
4:  heap :187020 (-2656)
4:  heap :189048 (+2028)

But nothing loads....I wait...still nothing. Sometimes I also get a request for the favicon.ico but nothing.

I then tried connected with a macbook to try and debug the network traffic, same sort of wifi station log appears, but I can't even get the factory.html request to go through this time. Tried connecting from my Windows machine and the wifi immediately disconnects with Cannot connect to network.

Something fishy, not sure what, any ideas?

from lws-esp32-test-server-demos.

I tried this on current factory 3f93751e9eb9266bb5784d6f5b0bfe7184ed8f6a

 $ make erase_flash

to nuke everything back to scratch. then

 $ make flash monitor

I was able to connect my wifi to in my case ESP_05D769 (yes I cannot keep up with esp-idf breaking things, it seems to default to that now), an http://192.168.4.1 comes up with the factory page nice and quickly in Firefox 57 and Chrome 64.

from lws-esp32-test-server-demos.

from lws-esp32-test-server-demos.

Sure... you must have a lot right or it can't find the ROMFS. But make sure it's the same checkout and esp idf is at 0c50b65a34cd6b3954f7435193411a88adb49cb0... with a git submodule update on esp-idf at that checkout

from lws-esp32-test-server-demos.

lws-esp32-factory.zip

from lws-esp32-test-server-demos.

romfs.img.zip

from lws-esp32-test-server-demos.

from lws-esp32-test-server-demos.

Managed to get this to compile under windows 10 and flashed onto the device successfully, with the wifi now coming up and properly delivering pages without issue. So I now have a working Windows build. Will put the PR up this weekend.

My windows PC connection issues were due to the cheap Edimax wifi dongle I believe, I have a better dongle on order and will confirm this is the case before submitting the PR.

from lws-esp32-test-server-demos.

Well, it isn't that simple to port, so good job. If there are any dependencies please also mention them in the README.md. We basically use the esp wlan apis, with the addition of the multi-AP scan state machine on top. So it shouldn't affect compatibility with the peer OS WLAN at all.

I am in the middle of making some big changes to -factory, but they don't affect the build part.

from lws-esp32-test-server-demos.

All the changes are actually in the core libwebsockets library in component.mk and contrib/cross-esp32.cmake this gets the component to compile properly under the esp-idf environment. I'll file a PR against this tomorrow with a bit of luck.

The genromfs script actually works out of the box under MSYS2. I'll add the additional instructions to the libwebsockets/READMEs/README.build.md under the ESP32 section, these are just the steps to install cmake under the MSYS2 shell and build genromfs from source and add it to the path.

Actually very minimal changes to make this work under windows which is awesome!

Besides this I think the project could do with a little more instruction on how to customize the factory build for a specific application and how to use the settings from factory in your own application. As I work through the steps myself I'll try and remember to document them and add them in a PR later. Excellent work though, great to see HTTPS working from a microcontroller!

from lws-esp32-test-server-demos.

Super... it's HTTP/2 too, not just https.

Generally the idea is you just need to change the logo on -factory. Your actual app will have its own config separately that does whatever you need it to do. But presumably every app needs a way to set up certs + wlan + updates, so -factory is designed to do that independent of your main app.

from lws-esp32-test-server-demos.

If it something consumers will see I think it probably needs some simplification, your average user doesn't need to know about certs and updates hopefully. But if its factory only, i.e. consumers never interact with it then that's cool. Do you think we can reduce the ~900kb partition in that case?

from lws-esp32-test-server-demos.

It's already simplified here; the next version will create its own selfsigned cert at runtime by default. But I have another feature to integrate before I update it. If you think it needs to be even more simple after that, well, patches are very welcome.

In the end it will use up its 1MB area and you will have 2.9MB for your app on a typical 4MB serial flash. But your app simply inherits working certs + wlan ap connection, and a story for updating the app image, to make up for it.

from lws-esp32-test-server-demos.

FWIW the stuff I mentioned I intended to do is done and pushed a few days ago.

It's also possible to merge the stuff in -factory with your actual app, most of the work for -factory is done by platform-specific stuff and protocols that are actually in lws.

from lws-esp32-test-server-demos.