liamcottle / meshtastic-map Goto Github PK

Hey @liamcottle, first I wanted to say, awesome project! Was able to get it and running very quickly.

I'm working on a Docker compose implantation right now and had a few quick questions:

1. Do mqtt.js and index.js need to interact with each other in any way? or is it all just mqtt.js writing to DB, and index.js displaying results? (trying to determine if they can be run as separate containers, and if so, any ports that need to be opened between them)

2. Is there any interest in a PR to this repo for a Docker image built off of GitHub Actions and associated docs/docker-compose.yml?

3. Same as 2, but a Kubernetes Helm chart

I understand the implications of merging 2 and 3 into this repo; it becomes 'official supported'. If thats something you're not ready to take on (I'm happy to help on an as-available), totally understand.

Either way, thanks for the awesome contribution to the Meshtastic ecosystem!

Hello Liam and readers
I created a waypoint about 4 weeks ago. It is still visible on the (your) map. But in Android I could not set an expiration date. Since then I wiped the data of the Meshtastic app and now I am not able to edit or delete this waypoint anymore. (ID 4292587809) Also I think there is no persistence MQTT entry for that. So now it only exists in your database or am I wrong ? Any idea how I could edit or delete this waypoint now ?
I am grateful for tips.

Hi, would it be possible to show shortname instead of just a dot?

See this for example: https://meshtastic.taubetele.com/
https://github.com/kkwestt/meshtastic.taubetele.com

Cheers

When running meshtastic-map for long enough, some exceptions like below arise.

Invalid prisma.textMessage.create()` invocation in
meshtastic-map/src/mqtt.js:312:42

309 }
310
311 try {
→ 312 await prisma.textMessage.create(
The provided value for the column is too long for the column's type. Column: text
at In.handleRequestError (meshtastic-map/node_modules/@prisma/client/runtime/library.js:122:6854)
at In.handleAndLogRequestError (meshtastic-map/node_modules/@prisma/client/runtime/library.js:122:6188)
at In.request (meshtastic-map/node_modules/@prisma/client/runtime/library.js:122:5896)
at async l (meshtastic-map/node_modules/@prisma/client/runtime/library.js:127:10871)
at async MqttClient. (meshtastic-map/src/mqtt.js:312:17) {
code: 'P2000',
clientVersion: '5.11.0',
meta: { modelName: 'TextMessage', column_name: 'text' }
`

Two new nodes: Heltec-V3 and WisBlock both with latest FW.

Heltec is Client with Neighbors turned On, WisBlock is Router_Client.

Heltec displays on Map but no Neighbors are showing up that saw Heltec or that Heltec saw.

Heltec connected to MQTT server, WisBlock is not.

Assumed Heltec would find WisBlock even with Neighbors off since both on FW 2.3+

No change even with WisBlock Neighbors On.

Great Map, BTW - good job!

Having duplicate neighbor lines add's complexity to the view.
Can these be collapsed where the to/from are from the same nodes.

Example

Perhaps light blue where heard is only seen one way, dark blue where it's heard both ways, and the hover box shows both info boxes as one box showing both paths?

Alternatively when clicking on the line it splits the lines out into the green/red seen/heard colors? or green/red arrows on the end or similiar?

When refreshing the page, the neighbors checkbox is always cleared for me. Not sure if it's a browser issue.

Also, if I select a node then select view neighbors, either "We heard" or "Heard us", then click the X to close that overlay, the neighbors are always unselected. The neighbors selection should go back to what was selected before.

First of all, thank you for your great work. I'm a thankful every day user of the map.

Back in the day's I've looked at APRS. Some of the bigger APRS sites (https://aprs.fi) do offer tracking of a station automatically. This way you can backtrack where the node has been. It will plot its route by dots and colored lines.

Perhaps this could be a useful option to have on your map?

I do have some requests:

1. For privacy reasons only look back for 1, 2 or 3 hours?
2. Add info to the plotted dots who heard that node on that location. It will help to identify coverage issues.

If using a different randomally generated PSK for primary channel, meshtastic will generate an AES256/32 byte key.
mqtt.js is currently unable to decrypt this packet even when passed the key as an argument. I've added some debugging in - receiving a packet with an AES256 key throws the following:

RangeError: Invalid key length
at Decipheriv.createCipherBase (node:internal/crypto/cipher:121:19)
at Decipheriv.createCipherWithIV (node:internal/crypto/cipher:140:3)
at new Decipheriv (node:internal/crypto/cipher:289:3)
at Object.createDecipheriv (node:crypto:155:10)
at decrypt (/home/ro/meshtastic-map/src/mqtt.js:241:37)
at MqttClient. (/home/ro/meshtastic-map/src/mqtt.js:331:29) {
code: 'ERR_CRYPTO_INVALID_KEYLEN'
}

I'm not overly experienced with javascript, but I was able to solve the problem by adding the following:
if (decryptionKey.length == 24) {
// create aes-128-ctr decipher
decipher = crypto.createDecipheriv('aes-128-ctr', key, nonceBuffer);
} else {
// create aes-256-ctr decipher
decipher = crypto.createDecipheriv('aes-256-ctr', key, nonceBuffer);
}

I'd fork/pull, but I'm not sure that's the most efficient way to do it?

As I wrote in the issue title, it would be smart if the mqtt listener and web UI were decoupled. This is a draft of how it would look:

meshtastic-map
├── docker-compose.yaml
├── mqtt-listener
│  └── dockerfile
├── prisma
└── web
   └── dockerfile

I think it would be also a good time to add a proper vue js build process as you wrote in To Do.
And I think implementing Docker would help to easily self host the project.

Because default MQTT server censored neighbor info frames.... I created my own MQTT server for our mesh, and used this awesome map code :)

Hi,
Ideas for the sidebar. I think it's getting really long, even on a 1440p screen.
Here is what I suggest:
The separator could be a line, comma, table, etc.

• Put Long and short name into one row, with some sort of separator.
• Put Role and firmware into one row
• Put Region and frequency into one row
• Put Modem Preset and Has Default into one row
• Put Lat and Long into one row
• Maybe? Battery level and volt into one row?
• ID and Hex ID into one row
• Neighbors updated and Position updated into one row.

Thanks

Can we get a legend in bottom left of the screen next to the MQTT status legend to describe the colours of the lines?

In example provided below, You can work out Red with Arrow pointing in is "We Heard", but what is the Green Line with no arrows?

yes.

Instead of showing the uptime from DeviceTelemetry in seconds, it would be handy to have it in the days/hours/minutes format.

The voltage graph min could be set to minimum of 3.2v and all measurements. The maximum to the maximum of 4.2 volts and all measurements. The voltage and battery % would look then look similar. (I don't think they follow each other in all cases and the relationship may not be linear)

At the time of writing min v looked to be 0v and the voltage graph was difficult to see as the line was only at the top.

Feature request:
Save the map position, so when refreshing the entire page, it doesn't reset to a globe view. You could add a maximum saved zoom level so it won't be zoomed in too much, maybe max zoom level of 10 or something.

As per email, it would be great to be able to see more detail of the node voltages etc by way of more detailed graphs. Thanks!

None of the nodes show mqtt topics they are seen on,
see this for example: https://meshmap.net/#3662844492

Last seen via on root topic
5min ago self msh/IN/Bir/Alft
5min ago [!da529480] msh/IN/Bir/BU

The same node:
https://meshtastic.liamcottle.net/?node_id=3662844492

Shows:
No packets seen on MQTT

Adding hopStart (added since 2.3) to database -> to have a better neighbor mapping. To make better graph even on node without neighbor info enabled.
Also neighbor info is a pain to collect because it's a low priority frame. With the hopLimit == hopStart trick we can get neighbor and RSSI/SNR from all message or any frames :) and make better link on map, or raw mesh graph like my one https://www.serveurperso.com small homepage made with D3.js showing live Parisian mesh network.

For some reason, a node that has been disabled for over a week remains displayed on the map.
The node is displayed on the map with the information "MQTT Status: Online (3 minutes ago)".
I don't understand why this is the case, the node has really been turned off for over a week.

ID: 3663107308
Hex ID: !da5694ec
https://meshtastic.liamcottle.net/?node_id=3663107308

It's possible for two nodes to have an identical ID (while it means two colliding int32s, it definitely happens -- I have a 3000km link showing for me to a different node that shares an ID with one a few meters away). I'm not sure what the best way to fix this would be, but it would probably be good, if possible, to treat these as several different nodes.

In this case, I think it may be a specific bad ID -- https://meshtastic.liamcottle.net/?node_id=115503498 has multiple cross-country links. Nevertheless, these collisions do seem to happen and it'd improve the map if they could be intelligently detected, probably.

I do also understand this isn't a very easy problem to solve, but perhaps things can be partitioned by uplink/source node to some degree; when a given ID consistently gets similar names and positions from one uplink and gets a different similar set of names and positions from another uplink, the existence of a collision can maybe be inferred.

When nodes are stacked on each other, as noted it's hard to click in to see a node beneath to see details, could calculate if they are within 10meters of each other for example?

Can tell there is more then one node here due to having "Heard" enabled on another node and you still have a blue line next to it.

An another mapping/visual tool I use - https://lukeprior.github.io/nbn-upgrade-map/?suburb=mount-barker&state=SA&commit=main

When there's more then one location there it displays a number on the dot, and when click spreads them out and can click on the individual marker for details.

The MapReport packet contains the number of online local nodes of a certain node. Since when using map reporting nodes in your mesh that do not have MQTT uplink enabled will not be shown, it would be nice to have the MQTT gateway report how many other nodes one can expect in the area.

Really nice map BTW!

Bug:

1. Search for a node and select it
2. It zooms into that node
3. I'm unable to click on "Show Full Details"

If I click elsewhere, then click back on the node, I can click the button.

I tried on Edge and Chrome, it might work elsewhere?

Hello
Just a small issue in the pop-up of a waypoint: It seams that 360 is added to it. Position on the map is fine.

I grabbed the MQTT packet for that one:
{"channel":0,"from":3663091580,"hops_away":0,"id":4292587810,"payload":{"description":"Great viewpoint","expire":2147483647,"id":4292587809,"latitude_i":468602396,"locked_to":0,"longitude_i":75258635,"name":"Chutzen"},"sender":"!da56577c","timestamp":1711963987,"to":4294967295,"type":"position"}

Thanks for the great map ! I did not understand Neighbours yet.

For solar nodes, being able to track battery life over longer than a 24 hour period would be great.
I see that there are other issues about cleaning up the sidebar (e.g. #21 ), but having extended battery stats would be very beneficial for solar node operators due to weather affecting solar power collection.
Could the battery graph be dropped to a new line below battery level and percentage, in order to take up the whole side bar, instead of just the right column of it's row? Then you could show 48+ hours of data hopefully.

Nodes that are offline for days still updating on the map somehow.

I have a node which is shut down on my shelf for 2 days, Android app off on my phone, yet the map says it's updated minutes ago.
Nothing changes, neither name/telemetry just the "Updated" line.

If a node participates in a trace route then this is provide trace route info for that node.This also provides an incoming route, not just an outgoing one.

Also the route discovery packets contain passive route information. (Ignoring the destination)

So a routing layer on the map could be added to show incoming and outgoing routes to a selected node. Including routes the node participated in and route discovery that the node participated in, even if the route was not discovered.

Or a new layer could just display all hops deduced from any route data as a line.

Received the error below:

PrismaClientKnownRequestError: 
Invalid `prisma.deviceMetric.create()` invocation in
src/mqtt.js:544:51

  541 
  542 // create metric if no duplicates found
  543 if(!existingDuplicateDeviceMetric){
→ 544     await prisma.deviceMetric.create(
Value out of range for the type. Out of range value for column 'air_util_tx' at row 1
    at In.handleRequestError (node_modules/@prisma/client/runtime/library.js:122:6854)
    at In.handleAndLogRequestError (node_modules/@prisma/client/runtime/library.js:122:6188)
    at In.request (node_modules/@prisma/client/runtime/library.js:122:5896)
    at async l (node_modules/@prisma/client/runtime/library.js:127:10871)
    at async MqttClient.<anonymous> (src/mqtt.js:544:25) {
  code: 'P2020',
  clientVersion: '5.11.0',
  meta: {
    modelName: 'DeviceMetric',
    details: "Out of range value for column 'air_util_tx' at row 1"
  }
}
PrismaClientKnownRequestError: 
Invalid `prisma.node.updateMany()` invocation in
src/mqtt.js:564:39

  561 // update node telemetry in db
  562 if(Object.keys(data).length > 0){
  563     try {
→ 564         await prisma.node.updateMany(
Value out of range for the type. Out of range value for column 'air_util_tx' at row 1
    at In.handleRequestError (node_modules/@prisma/client/runtime/library.js:122:6854)
    at In.handleAndLogRequestError (node_modules/@prisma/client/runtime/library.js:122:6188)
    at In.request (node_modules/@prisma/client/runtime/library.js:122:5896)
    at async l (node_modules/@prisma/client/runtime/library.js:127:10871)
    at async MqttClient.<anonymous> (src/mqtt.js:564:21) {
  code: 'P2020',
  clientVersion: '5.11.0',
  meta: {
    modelName: 'Node',
    details: "Out of range value for column 'air_util_tx' at row 1"
  }
}

for nodes with fw 2.3+ which don't have neighborinfo enabled, implicit neighbors will be the ones with hop_limit==hop_start

Why does it store node data so long?

Do you have an opt out process?

Really is limited value to anything more than a day old.

I think it would would be interesting to be able to either filter or view nodes by the Uplink Node.

Essentially the ID in the original MQTT topic
msh/UK/2/c/LongFast/!da545934

Position messages on that topic, will just be the nodes that !da545934 is able to hear (even multi-hop).

While neighbours is good, and it is specifically the 'direct' neighbours. This would show the nodes, the particular node received, even multi-hop. which would show more of the mesh coverage. (although would only work if the Uplink node itself is publishing its position!)

Could be kinda like the 'Neighbours' buttons, and just draws a line to every node it has received a position report from.
Maybe Nodes we heard- analogous to the 'neighbours (we heard)' - ie it's not just direct neighbours, but all nodes heard.

Or maybe could be that enter a Node ID in search, and it would only show reports via that Node.

But could also have the reverse. Nodes heard us - shows all the Uplink nodes that have received a position packet from the node.

Ie see how far away a particular node has been heard - even multi-hop. (just with the limitation would only know about uplink nodes that heard the node, and again only if the upload node itself published its position)

(When I say 'Uplink Node' I think in the code they called the 'gateway' node. Saved packets store the gateway_id. So ultimately its viewing nodes by which gateway(s) they connected to)

meshtastic/firmware#4305 has changed/removed the status topic from MQTT which now stops the ability to see if the MQTT is Online or Offline in the map (regardless of using a private MQTT broker)

Is there another method that can be used to determine the status?

If you changed

<div class="w-full space-y-0.5 px-2.5 py-1" v-bind:title="message.created_at" v-html="escapeMessageText(message.text)"></div>

we'd see the time the respective message was sent/forwarded on hovering over the text.

I had the Neighbor Info Module enabled on a node I kept in my car. Nodes found while driving are not true direct neighbors when I get home. Can we have the ability to reset or have the Neighbor Info expire?

Greetings from an area with a growing mesh who are big fans of your work.

We currently have people cycling and hiking around the city to try and make their first connection to a node, and some dedicated people looking at planning links between well-positioned solar power nodes.

One feature that would be helpful to both of these types of users is a display option that limits the nodes displayed by node Role. As the number of nodes grow, we think having the option to choose to display, for example, just ROUTER / ROUTER_CLIENT and exclude CLIENT will give us a nice clean map so we can help the people who want to work with the more 'fixed'/'permanent' nodes.

Initial impressions was that this might best be done in the layers widget.

Open to any and all discussion :)

There were added some Public MQTT server enhancement.

Original post: https://discord.com/channels/867578229534359593/871558934916390953/1260642727335428176

TLDR, these are only allowed portnums on Meshtastic MQTT server:

• NodeinfoApp
• TextMessageCompressedApp
• TextMessageApp
• PositionApp
• TelemetryApp
• MapReportApp
• RoutingApp

Feature request:
Remember options like Neighbors shown, Waypoints, etc. on the map.
Currently those options are not persisted over website refreshes.

The route trace uses the packets to and from data in the header plus the payload that contains additional participants.
So a packet can be sent From: A To: Z asking for a route.
{ From:A To:Z }

Seen by B and "repeated".
{ From:A To:Z Route: [B] }

Seen by C and repeated.
{ From:A To:Z Route: [B,C] }

Seen by Z and so Z replies.
{ From:Z To:A Route: [B,C] }

Repeated without change. across the network.
{ From:Z To:A Route: [B,C] }

Seen by A.
So the route is an outgoing route. But this is a reply so the route starts with the "To" recipient, then the payload and ended at the from recipient.
(To)A [ B , C ] (From) Z
A->B->C->Z
This is called "2 Hops"
A -hop-> B -hop-> C -Rx-> z
So a 2 hop route has a payload of two nodes and a total of four participants in the route. (Plus a gateway node, which is typically just an observer.)
Very very confusing!

I think you are completely missing the "to" field. This is usually to "all", a special node ID.

Hello, I am trying to contribute features to this project, but the Vue app, mqtt.js, and index.js are getting bigger and harder to read and understand. It would be smart to rewrite the front-end into a proper Vue app and use typescript for every aspect.

I am willing to help.

The site currently requests about 2 dozen device images that don't exist each time the site loads. There could be a check to make sure the image exists before trying to load the image.
Or we can add a placeholder image for each device type.
I guess there already is one, but not sure how that is generated.

+1 for your "claim" feature in the readme

After claiming a node, an easy way to add more detail is by uploading the YAML config file, gotten via meshtastic --export-config.
This would parse the settings and provide details that can't be gotten via mesh transport, such as module configs.

Hi
Looks like the neighbour info (heard us / we heard) has not been updated for several days.
Is it disabled now ?
Will you consider enabling again ? Very usefull feature, on my point of view.