deep-symmetry / crate-digger Goto Github PK

The PIONEER directory contains these files:

• DEVSETTING.DAT
• DJMMYSETTING.DAT
• MYSETTING.DAT
• MYSETTING2.DAT

These apparently allow the device to configure itself according to the preferences of the DJ that the plugged in device belongs to.

Some very basic reverse engineering of the format has been done here: https://github.com/Holzhaus/rekordcrate/pull/27/files#diff-811635f5c64ddc67c4db2a60451b9d96feec1542bb4728595586c7c51ff25003R17-R32

I want to stop discussing this in #5, it deserves its own issue! 😆

@mganss I have updated the online pdf to include my draft so far, if you would be so kind as to review it. I need to add the table of actual phrase ID interpretations, which I have enough information to do already, but I don’t know for certain things like the value of len_header so I hope you can send me a PSSI file to study as well.

The next time work is done on this library, let’s add an explanation for this!

This was reported by @drummerclint at the end of 2019, and thanks to his hunting down of the actual ANLZ000.DAT file, I finally was able to figure out and fix the problem.

Hello, hope you are all well.
First of all I want to say that you did great work.
I tried to parse around 60000 analysis files (.DAT and .EXT files originally created not from Rekordbox but from DJ mixer CDJ 2000 NXS2) using C# code generated from the .ksy file. I have managed to parse most of the files successfully but for around 2500 .DAT files I got the following Exception Unhandled: System.IO.EndOfStreamException: 'requested 400 bytes, but got only 0 bytes' in “public byte[] ReadBytes(ulong count)” method.
The strange thing is that when am using Web IDE to see the structure and visually explore the contents of these files, I get no error and I can see them normally. Also am using the latest version of your code.
Can you please help me on this issue? I can provide you more samples of these .DAT files if you want.
Thanks

DAT FILES.zip

We are using 1.1.4, and it is up to 1.1.6. The big changes came in 1.1.5, which generates better Java code, including the use of real enums rather than interfaces. This will be a backwards-incompatible change for us so it will take a bit of work to adjust to it, but it is probably worth it.

See https://sourceforge.net/p/remotetea/news/

Similar to .EXT files which added new tag types, the .2EX files seem to contain some new tags:

Judging from the tag names and sizes, these seem to be related to waveforms.

Describe the bug
PCPT2 tags (nxs2 extended cue lists) can have entries short enough to omit comments.

Samples
@evanpurkhiser reported this on the Gitter chat with some samples (headers already stripped off, but length as shown):

[
  '00000000: 7c 00 00 00 01 00 01 00  00 00 e8 03 7b 69 04 00 ||.........è.{i..|',
  '00000010: ff ff ff ff 00 00 00 00  00 00 00 00 00 00 00 00 |ÿÿÿÿ............|',
  '00000020: ff ff ff ff b6 54 00 00  00 00 00 00 54 e4 78 00 |ÿÿÿÿ¶T......Täx.|',
  '00000030: 00 00 00 00 42 00 00 00  00 00 00 00 00 00 00 00 |....B...........|',
  '00000040: 00 00 00 00 00 00 00 00  00 00 2c 00 00 00 15 00 |..........,.....|',
  '00000050: ff 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |ÿ...............|',
  '00000060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|',
  '00000070: 00 00 00 00 00 00 00 00  00 00 00 00             |............    |'
]

[
  '00000000: 8c 00 00 00 02 00 01 00  00 00 e8 03 f5 57 02 00 |..........è.õW..|',
  '00000010: ff ff ff ff 00 00 00 00  00 00 00 00 00 00 00 00 |ÿÿÿÿ............|',
  '00000020: ff ff ff ff ff 2c 00 00  00 00 00 00 b9 0e 40 00 |ÿÿÿÿÿ,......¹.@.|',
  '00000030: 00 00 00 00 54 00 00 00  00 00 00 00 00 00 00 00 |....T...........|',
  '00000040: 00 00 00 00 00 00 00 00  12 00 54 00 6f 00 20 00 |..........T.o. .|',
  '00000050: 48 00 6f 00 74 00 2d 00  41 00 00 00 2c 00 00 00 |H.o.t.-.A...,...|',
  '00000060: 38 b3 00 ff 00 00 00 00  00 00 00 00 00 00 00 00 |8³.ÿ............|',
  '00000070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|',
  '00000080: 00 00 00 00 00 00 00 00  00 00 00 00             |............    |'
]

[
  '00000000: 38 00 00 00 00 00 01 00  00 00 e8 03 3d 04 00 00 |8.........è.=...|',
  '00000010: ff ff ff ff 00 00 00 00  00 00 00 00 00 00 00 00 |ÿÿÿÿ............|',
  '00000020: ff ff 01 00 51 00 00 00  00 00 00 00 e0 58 00 00 |ÿÿ..Q.......àX..|',
  '00000030: 00 00 00 00 00 00 00 00                          |........        |'
]

Expected behavior
We should handle these and just return a nil comment, rather than failing with a parse error.

Additional context
We will want to update the analysis document, and the dbserver version of the code in Beat Link as well.

Someone finally has need for figuring out how the History playlists are represented in database exports, so it ist time to figure that out.

Hi. Could you move all the binary files into git lfs? Also I am not sure that storing PDF in the repo is a good approach. I prefer to store only sources in repos, and make CI to generate artifacts like PDF.

Is there anything known about the EDB file format? E.g. the datafile.edb that Rekordbox 5 uses. It seems similar to the PDB files but not quite the same. I haven't been able to find anything about the format anywhere.

In track rows in PDB files, the color ID field is just a single byte (c_id at offset 0x58). The same goes for the color index in Extended (nxs2) Cue List entries of analysis files (c_id at offset 0x1c).

But for PDB color rows, the documentation states that the field is 2 bytes large (id at offset 0x05). Is there a reason to believe that this field is not just also a single byte, and the second byte belongs to u₃ or is yet another unknown byte field?

Hi guys, hope you are all well.

I have already read in the Rekordbox Export Structure Analysis manual that the database export file is the export.pdb. I have created a testing one on Rekordbox using analysed tracks with extra features like colours and expected from export.pdb that the 𝑎𝑛𝑎𝑙𝑦𝑧𝑒_𝑝𝑎𝑡ℎ field will contain/point an .EXT file. Instead, all 𝑎𝑛𝑎𝑙𝑦𝑧𝑒_𝑝𝑎𝑡ℎ fields containing/pointing to .DAT files only.

Am suspecting that exportExt.pdb points to .EXT files. If not then how .EXT files are pointed? Does anyone knows how this works? What is the purpose of exportExt.pdb?

Parsing the exportExt.pdb on Web IDE using the .ksy file seems not working.

Can anyone help please?

Thanks

.EXT files have a section called PQT2. It sounds like it has to do with quantization, but from the data I can't tell what it does 🤷‍♂

Here are some suggestions how to improve your LaTeX:

1. use XeLaTeX . It works with ttf and otf and gives better appearance.
2. \usepackage{cleveref} - automatic crossreferences with links.
3. \usepackage{hyperxmp} - embeds XMP metadata block
4. \usepackage{smartref} - refs with human-readable names
5. set your metadata with \hypersetup
6. use \usepackage[pdfusetitle]{hyperref} to make hyperref to use the title set with \title

I heard from another person who is implementing parsing of PDB files, and he was working with some Russian text, and discovered we were wrong to think these strings are UTF-16LE. Here is what he said, and I validated this by creating a playlist containing the same string in its name:

I could have something off, but here is what I'm seeing about the strings. My PDB includes a Russian song called Покинула чат ("left the chat"). The first letter here is U+041F. All the Cyrillic letters start with 04, but the spacebar between the words is the same U+0020 as in English. Here's how the track name looks in the pdb in hex:

If I skip the 0 and read little endian, I get back the desired "Покинула чат"

If I don't skip and read big endian, I get back the incorrect " окинулаРGат" It gets a lot of the letters right because there is usually a 04 every other byte, but the first letter (which turns out as U+001F "Information separator one") and the characters around the space get messed up (because of the momentary switch from leading 04 to leading 00).
English titles come out right either way, because the leading 00s for each ASCII character in UTF-16 make it forgiving.

I've stumbled upon Deep-Symmetry many times over the last couple of years because I keep wanting to make something that makes my life much easier when it comes to managing my collection. I'm kind of just stuck with modifying the exported XML file for now, but I'm looking to see if the process can be somewhat automated if I can hook into the Rekordbox Database somehow.

The work you've done is amazing! You've found a lot of information on the most over-engineered protocols, and you've done excellent documentation too - something that's far too rare. 🙏

I'm willing to place bets that you've looked into the .edb files as well to see any similarities between the .pdb files, and I'm curious to know if you've found anything?

Looking at its hex/ascii content, it looks somewhat similar to what I would expect, but it's also too far different for me to make any connections in my mind.

Do you reckon it could be reversed? Unfortunately, Ubiquitous AI (the makers of Device SQL - afai could tell) run a pretty tight ship with their SDK.