question: decoding robot.db about dustcloud HOT 23 OPEN

I managed to create indeed picture out of the data.
https://github.com/marcelrv/XiaomiRobotVacuumProtocol/blob/master/RRMapFile/RRFileFormat.md

I'll try to clean it up a bit an post the code

from dustcloud.

As I wanted the current map during cleaning (the robot.db seems to be updated when cleaning was finished, only), I created the following php script which uses the files /run/shm/*.ppm and /run/shm/SLAM_fprintf.log (both will created and updated during cleaning continuously), to draw the positions on the map.

SLAM_fprintf.log contains the coordinates (x, y & angle)
.ppm was already converted to .png before

<?php
$slam_log = 'SLAM_fprintf.log';
$image = 'navmap20423422.png';

// get image size and calculate center of the image
$s = getimagesize($image);
$center_x = ($s[0] / 2);
$center_y = ($s[1] / 2);

// open jpg image
$im = imagecreatefrompng($image);

// rotate image by -90°
$im = imagerotate($im, -90, 0);

// loop each line of slam log
foreach (file($slam_log) AS $line) {
    // find positions
    if (strstr($line, 'estimate')) {
        $d = explode('estimate', $line);
        $d = trim($d[1]);

        // extract x & y
        list($y, $x) = explode(' ', $d, 3);

        // set x & y by center of the image
        // 20 is the factor to fit coordinates in in map
        $x = $center_x + ($x * 20);
        $y = $center_y + ($y * 20);

        // draw pixel
        imagesetpixel($im, $x, $y, imagecolorallocate($im, 255, 0, 0));
    }
}

// draw current position
imagefilledellipse($im, $x, $y, 8, 8, imagecolorallocate($im, 255, 0, 0));

// rotate image back by 90°
$im = imagerotate($im, 90, 0);

// crop background
$cropped = imagecropauto($im);
imagedestroy($im);

// output image
header('Content-Type: image/png');
imagepng($cropped);
imagedestroy($cropped);
?>

which results in (red dot is the current position):

from dustcloud.

Hi, i took a look at that, but i also have no clue ;) Was not my top priority, tho.

from dustcloud.

I investigated bit further..
It is a .gz file. Once extracted the header starts with RR

looking @ the data, I suspect it is some sort of very simple bitmap format (I see only 00, 01 & FF), followed by a section of the robot path coordinates

from dustcloud.

great

from dustcloud.

@marcelrv: can you write a simple parser for that? e.g. convert to ppm or even jpg?

from dustcloud.

yes, on the same page I posted a simple parser. It is in java though, not in python.
But the format is easy enough. I'm not too familiar with programming graphics, even less in python hence the java flavor
in the mean time I found out also the other pieces of the format (coordinate of charger & movements) , but those have not been implemented.

I will in the next update also add the oneliner to save it as jpg

https://github.com/marcelrv/XiaomiRobotVacuumProtocol/blob/master/RRMapFile/README.md

from dustcloud.

@dgiese see python parser to ppm in #60
Thanks to @marcelrv for file format description

from dustcloud.

Great @jvitkauskas,
Thanks

from dustcloud.

@marcelrv
Do you know what is the correct way to interpret charger position (Data block type 1) data? It does not seem to match map dimensions (i.e. I get charger position x: 25899 y: 25677 on 205x205 pixel map).

from dustcloud.

First, a big thank you for digging up so much about the vacuum! This is really awesome work!

I had a look at the charger positions and path coordinates, however to no success.

Charger position is always something around 25xxx, like @jvitkauskas, or simply (0,0). I wasn't able to come up with any kind of relation between approximate pixel position in the map and the respective values. My thinking was that probably the offset values given in data block 2 must play a role. And rotation of course, however I have no idea with respect to what origin.

The same applies to path coordinates - all of them are either large integers or pairs of large shorts - both way outside the map dimensions. As in some of the navigation logfiles a lot of float values are given, I tried interpreting them as floats, however no luck either.

Any ideas?

from dustcloud.

Hey, looks great. A similar hacky way i used for the aerodust project (https://github.com/dgiese/aerodust). How you write the position continiously?

from dustcloud.

Thanks @CodeKing for your snippet, this helped me to almost figure it out:

It seems, the values for charger and path coordinates are to be interpreted as pairs of shorts. Scaling them suitably (similar to @CodeKing) gives pixel positions in an image.

I modified the charger() and path() routines from @jvitkauskas great script in #60 in the following way to produce an image file that looks a lot like the path the robot took.

def charger(data):
    pos_x = read_int(data)
    pos_y = read_int(data)
    return (pos_x, pos_y)


def path(data, path_len, charger_pos, output_folder, timestamp, map_index):
    set_point_length = read_int(data)
    set_point_size = read_int(data)
    set_angle = read_int(data)
    image_width = (read_short(data), read_short(data))
    print('%d path_len, %d set_point_length, %d set_point_size, (%d, %d) image_width' % (path_len, set_point_length, set_point_size, image_width[0], image_width[1]))

    # extracting path
    path = [(read_short(data), read_short(data)) for _ in range(set_point_length)]

    # rescaling coordinates
    path = [((p[0]) // 50, (p[1]) // 50) for p in path]
    charger_pos = ((charger_pos[0]) // 50, (charger_pos[1]) // 50)

    print(path)
    print(charger_pos)

    # Creating image
    width, height = image_width[0] // 25, image_width[1] // 25

    file_name = make_file_name(output_folder, timestamp, map_index) + '_path.pgm'
    
    pixels = [0] * width * height

    for x, y in path:
        pixels[y * width + x] = 155

    for off_x in range(-2, 2):
        for off_y in range(-2, 2):
            idx = (charger_pos[1] + off_y) * width + charger_pos[0] + off_x
            pixels[idx] = 255

    with open(file_name, 'wb') as file:
        file.write(('P5\n%d %d\n255\n' % (width, height)).encode())
        for h in range(height)[::-1]:
            file.write(bytes(pixels[h * width: h * width + width]))

The result looks like this:

The square matches the charger position and the path looks pretty accurate.

from dustcloud.

can you create a PR for that?

Btw: i believe the Charger is always at 512,512 (pixels)

from dustcloud.

Opened a PR with code posted here #79

from dustcloud.

Sorry, went to bed after posting and haven't found time until now. Thanks @jvitkauskas for creating the PR.

Regarding the charger position: In my set of 10 runs or so, the recorded position was always close but usually not exactly at 512,512. In most cases it was shifted in x-direction for a few pixels.
I guess the center point is the position, where the initial scan is performed. My robot usually leaves the charger towards the left (in above map) before performing the scan and starting the cleaning run.
I haven't tried what happens if you drop the robot somewhere and start cleaning from there. I assume this will then be center (512,512) and the charger is then somewhere else.

from dustcloud.

Looks like something has changed in the map format in the recent firmwares, does anyone know how to parse these? I experimented a bit with the mapextractor, and looks like there's at least a new block type "8", and the path block's format may have been changed.

magic: b'rr'
checksum_ptr: b'H\x08\x01\x00'
header len: 20
version: 1 0
map_idx: 1012 map_seq: 0
block type: 2 (size: 62500)
  unknown (flags?): b'\x18\x00'
block type: 1 (size: 8)
  unknown (flags?): b'\x08\x00'
  charger pos: (25325, 25624)
block type: 8 (size: 8)
  unknown (flags?): b'\x08\x00'
  unknown block type 8 with size 8
    hex: 645900002a5d0000
block type: 3 (size: 5052)
  unknown (flags?): b'\x14\x00'
  got path block
    set_point_length: 1263
    set_point_size: 4
    set_angle: 0
    image_width: (25600, 25600)
block type: 20 (size: 1566956445)
  unknown (flags?): b'\x00\x00'
  unknown block type 20 with size 1566956445
    hex: c0c4303af854707e7d2a96ce2fd4fc74

Any ideas?

edit: added a bit more verbose output.

from dustcloud.

@rytilahti I'm very new to Xiaomi but I'm currently working on a Xiaomi Windows application (without root) cause I wanted to have the map files the bot produced.

For that I needed to parse the RRFiles that are stored in the Xiaomi cloud.

Here are my findings:
Each block starts with unknown 2 bytes.
Followed by 4 bytes which is the length (int32).

• Block 1 - Charger Position

  • 4 bytes (int32) -> X Position
  • 4 bytes (int32) -> Y Position

• Block 2 - Sensor Image

  • 4 bytes (int32) -> Top starting point (in the path matrix when scaled to 1024)
  • 4 bytes (int32) -> Left starting point (in the path matrix when scaled to 1024)
  • 4 bytes (int32) -> Height
  • 4 bytes (int32) -> Width
  • X bytes -> Sensor Data (length defined by block length or width * height) (each byte can be 0x00 (unknown), 0x01 (wall) or 0xff (free))

• Block 3 - Path the robot already took

  • 4 bytes (int32) -> Point length
  • 4 bytes (int32) -> Point size (?) (What's this for?)
  • 4 bytes (int32) -> Set angle (?) (What's this for?)
  • now repeat "Point length" times (Points Position 0 - 51200 each axis ... devide by 50 to scale them to 1024)
    • 2 bytes (uint16) -> Path X Point
    • 2 bytes (uint16) -> Path Y Point

• Block 4 - (I've not been able to produce a case where this type of Block is created so I'm not sure if it even exists)

• Block 5 - Calculated path the robot is taking at the moment (bytes are the same as Block 3)

• Block 6 - Zone Cleaning Zones

  • 4 bytes Number of Zones
  • now repeat "Number of Zones" times.
    (X1 and Y2 are the left top Point of the Rect, X2 and Y2 are the right bottom point of the Rect)
    • 2 bytes (uint16) -> Path X1 Point
    • 2 bytes (uint16) -> Path Y1 Point
    • 2 bytes (uint16) -> Path X2 Point
    • 2 bytes (uint16) -> Path Y2 Point

• Block 7 - Current Target Position

  • 2 bytes (uint16) -> Target X
  • 2 bytes (uint16) -> Target Y

• Block 8 - Current Bot position (bytes are the same as Block 1)

• Block 1024 - I'm not sure about this one ... CRC?

  • X bytes

The RRFileFormat document says that Path contains Width and Height.
As far as I noticed that is wrong (or maybe that changed).
The size of the Path blocks seems to always be 51200x51200 (1024 * 50).
And the Width/Height defined in the doc is actually the first point.
The reason why this still worked is, cause the first point is always on or near the charger if you start the full clean from there and the charger is basicly very close to the center of the 51200x51200 block and while scaling this happens to be the center of the map.

EDIT:
Here is the UWP Windows App (and the map) I created with that informations:

from dustcloud.

@WolfspiritM
How did you get the RRFiles from the Xiaomi cloud? Is there any kind of API? The map is the only reason I rooted my vacuum ;)
Very nice work with the Windows application. Will you publish the source code when you are finished?

from dustcloud.

@WolfspiritM Any progress on your app?

from dustcloud.

@PiotrMachowski Nope sorry. I switched to using Valetudo

from dustcloud.

@WolfspiritM Could you share your code or at least methods used to obtain map data? I would love to recreate something similar for Home Assistant

from dustcloud.

If you are still interested in this topic, I have implemented retrieving map data from Xiaomi cloud and parsing RRFile in python. Code is available here.

Many thanks to @marcelrv for sharing his knowledge and code. My version wouldn't exist without his work.

from dustcloud.