• Signal-back: v0.1.6-alpha.1
• Binary: darwin_amd64
• OS: MacOS High Sierra 10.13.4
• Backup file size:
  382M Jun 26 01:14 signal-2018-06-26-00-56-36.backup
• Command run:
  ./signal-back_darwin_amd64 format -f XML signal-2018-06-26-00-56-36.backup > backup.xml
• Decrypted file size:
  0B Jun 26 01:36 backup.xml

I've tried using the CSV format as well and the same thing happened. I run the command, then enter my passcode. It runs without error and takes a few seconds, then just shows the cursor waiting at the next prompt.

With the export only supplying the datetime in the filename, there is no way to determine which conversation the media was a part of.

• Operating system: Ubuntu
• Build version (signal-back --version): signal-back v0.1.6-dirty, proto commit: d6610f0

Detailed description

Pretty straight forward, files are named as:

1522082357628.gif

When it would be better if they used <address>_<date>.<ext> such as:

+15555555555_1522082357628.gif

Full relevant output

n/a

I exported the chats of a backup.
In the dumps my own messages are missing in group chats.
That happens in both the csv and the xml export.

hi @xeals ,
I'm really happy you've started the work on this project, as I'm frustrated with the backup/restore functionality of Signal (desktop (hackable), Android (should be better since v4.20), and iOS (missing completely!)).

Would it be in scope of this project to: ?

• build extended functionality of Signal's backup & restore?
• support migration from android/iOS/desktop
• offer signal backup data in a widely supported format (your current plaintext XML, CSV)
• possibility to import from/to other clients (SMS, whatsapp, ...)

With regards to the Desktop backup and restore, I have figured a (manual) way which could well integrate in your CLI:
signalapp/Signal-Desktop#522 (comment)
Eg.:
signal-back backup <path to Signal's folder> <output file name.zip>
signal-back restore <input desktop backup.zip> <path to Signal>

commands tried:

• signal-back format -f XML -o smsbak.xml -p password.txt -v s.backup
• signal-back format -f CSV -o smsbak.csv -p password.txt -v s.backup
• signal-back format -f raw -o smsbak.raw -p password.txt -v s.backup

error: failed to format output: consume [avatar]: can't read attachment of length 0

• Operating system: Win10x64
• Build version: 0.1.7a2

Detailed description

the format commands is always giving me this error.
extracting media files from my encrypted backup using the extract command worked fine.

I would recommend a raw export feature, where just the backup container is decrypted and unpacked.
So one can check if the backup is complete. See issue #26

The decryption process currently gets hung on up a non-matching MAC between what the backup has and what we're expecting.

My major frustration and confusion at the moment is that the Java implementation allows for a 10-byte MAC, but the Rust crypto library produces 32-byte MACs following the same steps; obviously there's not going to be a time when these are equivalent. I'm not sure if it's a fundamental difference in how the Rust and Java libraries work.

I can process what I've written for the decryption process, but something along the line isn't exactly as it should be compared to the Java implementation. I'm diving into source for both languages to see where they differ. If anyone can point me to an open implementation of the javax.crypto library, I'd be grateful (as my version of OpenJDK doesn't have it).

The format XML has this column and data, but the format CSV does not.

I've made some backups of Signal with adb before the new database format was introduced in 4.16.3.
I mean backups like these:

adb backup -apk org.thoughtcrime.securesms

Would you consider supporting these adb backups? It would be sweet to be able to extract the contents out of them without installing old versions of Signal.

I've gotten the core body of the backup decoded, so I can start looking at what output formats are useful to people. I suspect an XML format compatible with stuff like SMS Backup & Restore might be popular. Other things that spring to mind are something IRC-esque where you can review conversations for interest's sake.

..intended, as shown here: https://github.com/xeals/signal-back/blob/master/cmd/format.go#L69

SMS Backup and Restore would mark the exported xml file as an invalid backup.

To fix this:

• Insert newlines after every line (\n). Currently it exports as one single line of text
• Encode the file as UTF-8 (VS Code inferred that the existing encoding was UTF-16LE)

SMS Backup and Restore accepts the modified file as a valid backup. You do lose emojis though. There might be need to be a fix for parsing the emojis

I am from russia.
So, when I encrypte backup file, english text is ok, but all cyrylic text is wrong in hieroglyphs.
And the encrypted file is in UCS-2 Little Endian, not in UTF-8.
What's happening?

Windows 10 64bit.

As the title says, getting a panic on SMS type 24 when trying to format.

According to my novice interpretation of Signal's code repo, it looks like it's choking on the SENT_FAILED type:

protected static final long BASE_INBOX_TYPE                    = 20;
protected static final long BASE_OUTBOX_TYPE                   = 21;
protected static final long BASE_SENDING_TYPE                  = 22;
protected static final long BASE_SENT_TYPE                     = 23;
protected static final long BASE_SENT_FAILED_TYPE              = 24;
protected static final long BASE_PENDING_SECURE_SMS_FALLBACK   = 25;

I'd provide more info about the specific message, if you can let me know how. I'm unsure how to dive into the backup itself.

I have no clue what I've done wrong, I'm very out of my element with all this. I have no coding experience so I'm just throwing darts. I tried to put my password in a txt file and do the "-P password.txt" but it had no difference.

error: failed to open backup file: unable to open backup file: FindFirstFile signal-XXX.backup: The system cannot find the file specified.

Windows 10 64-bit
Build version 4.17.5

This project looks great for exporting signal backups and I notice there is a raw option which worked for me. What I would find really useful is if I could export to raw formats (with all attachments), edit (manually), and then rebuild into a valid achive for import. Essentially all I think I need is to decrypt the bundle as is to disk, edit and then re-encrypt (eg no format conversion). I've got some broken data which I'm trying to fix and I'd like to have a go at fixing this manually in the sql export file and importing that back into signal with all attachments. Many Thanks.

• Operating system: Ubuntu 16.04
• Build version (signal-back --version): v0.1.7-alpha.2-dirty (proto commit: d6610f0)

Detailed description

Extract, check, and analyse commands work as intended, but format fills up the available memory quickly (no matter which format chosen). The backup file size is 1.8 GB, extracted media is 1.6 GB, on a machine with 8 GB of RAM.

Full relevant output

n/a

signal-bak_windows_amd64.exe format -f XML -P pw.txt -o backup.xml ***.backup fails with:

error: failed to format output: failed to write out XML: short write

• Operating system: Windows 10 x64
• Build version (signal-back --version): v0.1.7-alpha2

Detailed description

Extraction of media works fine. Can't get xml output on either Windows or debian-based Linux with the precompiled binaries. Signal backup from 03-16-2019.

On Linux simply results in a "Killed" message and process terminates. Same with manual password entry instead of using the password file.

• Operating system: OS 10.14.3
  2015 15" macbook pro, 2.5g intel i7, 16gb ram
  Backup file size 386mb
  Output xml file size 0kb

• Build version (signal-back --version): signal-back_darwin_amd64
  I put the above file and the backup file in the same folder and navigated there in terminal and pasted this:
  chmod +x signal-back_darwin_amd64
  ./signal-back_darwin_amd64 format -f XML -o backup.xml signal-2019-04-16-20-28-42.backup
  I didn't download the .sha256 file, I'm not sure if I needed that.

Detailed description

When I first run in terminal and enter the password I get this error
Password: error: failed to format output: consume [avatar]: can't read attachment of length 0

However, if I run it again and just hit return for the password without entering it, it creates a file that is empty. Below is what I see in terminal

NEWs-MBP:Signal new$ ./signal-back_darwin_amd64 format -f XML -o backup.xml signal-2019-04-16-20-28-42.backup
Password: goroutine 1 [running]:
runtime/debug.Stack(0xc4200900f0, 0x2, 0xc42041c3c0)
/home/travis/.gimme/versions/go1.10.7.linux.amd64/src/runtime/debug/stack.go:24 +0xa7
runtime/debug.PrintStack()
/home/travis/.gimme/versions/go1.10.7.linux.amd64/src/runtime/debug/stack.go:16 +0x22
github.com/xeals/signal-back/cmd.XML.func2.1(0xc42001c800)
/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:169 +0x142
panic(0x11c1f40, 0x130a250)
/home/travis/.gimme/versions/go1.10.7.linux.amd64/src/runtime/panic.go:502 +0x229
github.com/xeals/signal-back/cmd.XML.func2(0xc42001c800, 0x0, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:175 +0x92
github.com/xeals/signal-back/types.(*BackupFile).Consume(0xc42000e150, 0xc420088870, 0xc42005c210, 0xc42000a020, 0x0, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/types/backup.go:260 +0x14d
github.com/xeals/signal-back/cmd.XML(0xc42000e150, 0x121c8a0, 0xc42000c008, 0x3, 0x1)
/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:203 +0x24b
github.com/xeals/signal-back/cmd.glob..func4(0xc42009c420, 0x0, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:70 +0x40f
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x11b7a00, 0x1203488, 0xc42009c420, 0xc420068100, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x11f70b7, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x11fd1e2, 0x1f, 0x1202719, ...)
/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0xa36
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc42008c340, 0xc42000e070, 0x7, 0x7, 0x0, 0x0)
/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x6a0
main.main()
/home/travis/gopath/src/github.com/xeals/signal-back/main.go:52 +0x317
NEWs-MBP:Signal new$

Full relevant output

• Operating system: all
• Build version (signal-back --version): all

Detailed description

signal-back allows to extract messages (of type sms, i.e., normal messages, and mms, i.e. captions to media files) using the format subcommand. For instance, $ signal-back_linux_amd64 format --format csv --pwdfile signal_passphrase.txt signal-2018-11-10-19-30-15.backup yields:

ID,THREAD_ID,ADDRESS,ADDRESS_DEVICE_ID,PERSON,DATE_RECEIVED,DATE_SENT,PROTOCOL,READ,STATUS,TYPE,REPLY_PATH_PRESENT,DELIVERY_RECEIPT_COUNT,SUBJECT,BODY,MISMATCHED_IDENTITIES,SERVICE_CENTER,SUBSCRIPTION_ID,EXPIRES_IN,EXPIRE_STARTED,NOTIFIED,READ_RECEIPT_COUNT,UNIDENTIFIED
1,1,+1123456789,1,,1483906520819,1483906520818,,1,-1,10485783,,1,,How are you?,,,-1,0,0,0,0
2,1,+1123456789,1,,1483906614283,1483906738206,31337,1,-1,10485780,1,0,,Fine! How about you? ,,GCM,-1,0,0,0,0
...

Now, the extracted messages can be grouped by "THREAD_ID" and sorted by "DATE_SENT" to restore the original message history. But, the attachemts are missing!

Using the extract subcommand, signal-back allows to extract all attachments at once . The extracted attachments are saved as attachment_id.extension. But then, the link to the original "THREAD_ID" and the "DATE_SENT" is missing and it is not possible to map an attachment to the thread it belongs to.

So, it is not possible to restore the original message history completely by hand (without using the Signal app). This is required for example if you would like to create a book of your Signal chats using online services like https://www.zapptales.com/en/.

They expect the message history as plain text in following format (and the attachments with the referenced filename in a separate directory):

08.07.17, 16:52 - John Doe: How are you?
08.07.17, 20:01 - Jane Doe: Fine! How about you?
08.07.17, 20:02 - John Doe: IMG-20170708-WA0000.jpg <- This information is missing when using signal-back

Therefore, it would be useful to have the mapping between an attachment and its thread id (including the sent date).

Full relevant output

I have already looked up the method ExtractAttachments() in cmd/extract.go, but I have noticed that an attachment only has the attributes RowId, AttachmentId, Length and XXX_unrecognized. Unfortunately, I have no details about the enclosed BackupFile.Frame. Maybe, this Frame offers the "THREAD_ID" for an attachment.

CSV records have a variable number of fields. I assume the missing fields for any given record are suppose to be empty fields.

For example records with type 20 and 23 have a differing number of fields. And neither of them have the 22 fields I would expect based upon the header.

365,1,+1XXXXXXXXXX,1,97,1522112554906,1522112554906,0,1,-1,20,0,0,It's ok. Not too bad ,+12819659999,-1,0,0,0,0
366,1,+1XXXXXXXXXX,1,1522111363707,1522111363707,1,-1,23,0,How is the job going?,-1,0,0,0,0

I did a reinstall of my Signal and could not restore my backup using the app. So I turned to your work to be able to recover some old chat history.

I get the following error when running the program

> ./signal-back_linux_amd64 format -p [pass] signal-[date].backup > out.xml
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x559e6c]

goroutine 1 [running]:
github.com/xeals/signal-back/types.(*BackupFile).DecryptAttachment(0xc4200ec0e0, 0xc42009b230, 0x5e42a0, 0x6e0048, 0x550c7dc3243185be, 0x80deb1fe72be5d74)
	/home/travis/gopath/src/github.com/xeals/signal-back/types/backup.go:136 +0x12c
github.com/xeals/signal-back/cmd.XML(0xc4200ec0e0, 0x5e40a0, 0xc420094008, 0x3, 0x3)
	/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:153 +0x3dd
github.com/xeals/signal-back/cmd.glob..func3(0xc4200ae420, 0x0, 0xc4200ae420)
	/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:71 +0x246
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x587380, 0x5ccf48, 0xc4200ae420, 0xc4200a8000, 0x0)
	/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x5c171c, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5c7567, 0x1f, 0x5cc2b1, ...)
	/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0xa36
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc4200aa1a0, 0xc4200960f0, 0x5, 0x5, 0x0, 0x0)
	/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x6a0
main.main()
	/home/travis/gopath/src/github.com/xeals/signal-back/main.go:51 +0x2d8

The program manages to successfully recover a backup that I did after the reinstall. Does this mean that I have written down my encryption code for the old backup incorrectly? Or is something strange with the older backup file.

Thanks for your work!

• Operating system: Linux
• Build version: from source, today

Detailed description

Hi,
I wanted to implement a signal backup reader/writer in Go, so it was a delight to see it here :)
Trying to decrypt an archive from 2018-04-01, I got a panic accessing an out of range slice item.
Another archive from 2018-06-18 (on another phone) was fine.

Full relevant output

$ signal-back format -v -o signal-s8.xml signal-2018-04-01-11-48-29.backup 
Password: 2019/05/24 03:25:04 Unexpected error: runtime error: index out of range
2019/05/24 03:25:04 TEMP: statement is statement:"INSERT INTO part VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)" parameters:<integerParameter:1 > parameters:<integerParameter:1 > parameters:<integerParameter:0 > parameters:<stringParamter:"image/png" > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<nullparameter:true > parameters:<integerParameter:0 > parameters:<stringParamter:"/data/user/0/org.thoughtcrime.securesms/app_parts/part1279258965.mms" > parameters:<integerParameter:98833 > parameters:<nullparameter:true > parameters:<stringParamter:"/data/user/0/org.thoughtcrime.securesms/app_parts/part1197476668.mms" > parameters:<doubleParameter:1 > parameters:<integerParameter:1512410589211 > parameters:<nullparameter:true > parameters:<stringParamter:"2936884363452528202" > parameters:<integerParameter:0 > parameters:<nullparameter:true > parameters:<nullparameter:true > 
2019/05/24 03:25:04 TEMP: statement is &signal.SqlStatement{Statement:(*string)(0xc00295ac60), Parameters:[]*signal.SqlStatement_SqlParameter{(*signal.SqlStatement_SqlParameter)(0xc00085a550), (*signal.SqlStatement_SqlParameter)(0xc00085a5a0), (*signal.SqlStatement_SqlParameter)(0xc00085a5f0), (*signal.SqlStatement_SqlParameter)(0xc00085a640), (*signal.SqlStatement_SqlParameter)(0xc00085a690), (*signal.SqlStatement_SqlParameter)(0xc00085a6e0), (*signal.SqlStatement_SqlParameter)(0xc00085a730), (*signal.SqlStatement_SqlParameter)(0xc00085a780), (*signal.SqlStatement_SqlParameter)(0xc00085a7d0), (*signal.SqlStatement_SqlParameter)(0xc00085a820), (*signal.SqlStatement_SqlParameter)(0xc00085a870), (*signal.SqlStatement_SqlParameter)(0xc00085a8c0), (*signal.SqlStatement_SqlParameter)(0xc00085a910), (*signal.SqlStatement_SqlParameter)(0xc00085a960), (*signal.SqlStatement_SqlParameter)(0xc00085a9b0), (*signal.SqlStatement_SqlParameter)(0xc00085aa00), (*signal.SqlStatement_SqlParameter)(0xc00085aa50), (*signal.SqlStatement_SqlParameter)(0xc00085aaa0), (*signal.SqlStatement_SqlParameter)(0xc00085aaf0), (*signal.SqlStatement_SqlParameter)(0xc00085ab40), (*signal.SqlStatement_SqlParameter)(0xc00085ab90), (*signal.SqlStatement_SqlParameter)(0xc00085abe0), (*signal.SqlStatement_SqlParameter)(0xc00085ac30), (*signal.SqlStatement_SqlParameter)(0xc00085ac80), (*signal.SqlStatement_SqlParameter)(0xc00085acd0)}, XXX_unrecognized:[]uint8(nil)}
goroutine 1 [running]:
runtime/debug.Stack(0xc0000a0000, 0x2, 0xc000959200)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
	/usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/xeals/signal-back/cmd.XML.func2.1(0xc00281d5c0)
	/home/dolanor/go/src/github.com/xeals/signal-back/cmd/format.go:169 +0x142
panic(0x5e5820, 0x7bc940)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/xeals/signal-back/types.ParametersToPart(0xc000523c00, 0x19, 0x20, 0x47720)
	/home/dolanor/go/src/github.com/xeals/signal-back/types/raw.go:396 +0x774
github.com/xeals/signal-back/types.StatementToPart(...)
	/home/dolanor/go/src/github.com/xeals/signal-back/types/raw.go:362
github.com/xeals/signal-back/types.NewPartFromStatement(0xc00281d5c0, 0x61ce45, 0x10, 0x101, 0x9)
	/home/dolanor/go/src/github.com/xeals/signal-back/types/sms.go:313 +0x64
github.com/xeals/signal-back/cmd.XML.func2(0xc00281d5c0, 0x0, 0x0)
	/home/dolanor/go/src/github.com/xeals/signal-back/cmd/format.go:192 +0x109
github.com/xeals/signal-back/types.(*BackupFile).Consume(0xc0004ba070, 0xc000084b70, 0xc00005e230, 0xc00000e020, 0x0, 0x0)
	/home/dolanor/go/src/github.com/xeals/signal-back/types/backup.go:260 +0x14d
github.com/xeals/signal-back/cmd.XML(0xc0004ba070, 0x65c7c0, 0xc000010008, 0x3, 0x1)
	/home/dolanor/go/src/github.com/xeals/signal-back/cmd/format.go:203 +0x201
github.com/xeals/signal-back/cmd.glob..func4(0xc000092420, 0x0, 0x0)
	/home/dolanor/go/src/github.com/xeals/signal-back/cmd/format.go:70 +0x457
github.com/urfave/cli.HandleAction(0x5da720, 0x628998, 0xc000092420, 0x0, 0xc0000124e0)
	/home/dolanor/go/src/github.com/urfave/cli/app.go:501 +0xc8
github.com/urfave/cli.Command.Run(0x61ae19, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x621803, 0x1f, 0x6272b5, ...)
	/home/dolanor/go/src/github.com/urfave/cli/command.go:165 +0x487
github.com/urfave/cli.(*App).Run(0xc0000d41c0, 0xc0000121e0, 0x6, 0x6, 0x0, 0x0)
	/home/dolanor/go/src/github.com/urfave/cli/app.go:259 +0x6e3
main.main()
	/home/dolanor/go/src/github.com/xeals/signal-back/main.go:52 +0x2cf

Firstly, kudos for developing and maintaining this project! It is of great help. 👍

Environment

• Operating system: macOS 10.14.2 (Mojave), amd64
• Build version (signal-back --version): v0.1.7-alpha.2-dirty, proto commit: d6610f0

Detailed description

When we receive messages from a given contact, both directly and as a part of a group, how do we tell such messages apart? How do we identify which messages were part of the group, and which were direct?

The problem is exacerbated in the presence of multiple groups in which the contact participates.

Thank you!

It would be nice if you could disable password echo.

I just downloaded your prebuild binary for version v0.1.6 "signal-back_linux_amd64" and tried analyse & check commands, and saw that it echoed my backup password back to me.

Industry standard seems to me to either disable echo for password completely or echo stars instead to show how many characters have been entered.

• Operating system: Linux Mint 19 Kernel 4.15.0-39
• Build version (signal-back --version): signal-back v0.1.6-dirty, proto commit: d6610f0

I made a Signal backup (554MB) today, then attempted to use:
v0.1.6-alpha.1 binary
v0.1.5 stable binary
git clone compiled code.

All produced a panic when using commands extract or analyse.

panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                            
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x566634]                                                                                                                                             

goroutine 1 [running]:
github.com/xeals/signal-back/cmd.ExtractAttachments(0xc4200dc0e0, 0x6, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/cmd/extract.go:68 +0x134
github.com/xeals/signal-back/cmd.glob..func2(0xc420088420, 0x0, 0xc420088420)
        /home/travis/gopath/src/github.com/xeals/signal-back/cmd/extract.go:43 +0x8c
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x5939a0, 0x5dacf0, 0xc420088420, 0xc420082100, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x5cf178, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5d66e7, 0x24, 0x5d6b6a, ...)
        /home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0xa36
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc4200841a0, 0xc4200760f0, 0x5, 0x5, 0x0, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x6a0
main.main()
        /home/travis/gopath/src/github.com/xeals/signal-back/main.go:51 +0x2e1

Using Arch Linux 4.17.3-1-ARCH, golang version 1.10.3.

The approach of first reading the entire SMS database into a slice of structs in memory, then marshalling the entire slice into XML leads to uncontrollable memory consumption, making it almost impossible to use the program on a relatively large message database (parsing encrypted export file of ≈800 MB leads to peak memory consumption ≈ 6 GB on Windows).

XML export should be implemented on a per-message basis (decode -> marshal -> write to file).

I am running the latest version after the new build that came after my issue here #38 and I am getting an error for a correct password as follows:

Command
signal-back_windows_amd64.exe format -f XML -o backup.xml signal-2018-12-15-14-23-41.backup

Error

Password: panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x5b25a1]

goroutine 1 [running]:
github.com/xeals/signal-back/cmd.XML.func2(0xc000046680, 0xc000138d70, 0x0)
        /home/xeal/go/src/github.com/xeals/signal-back/cmd/format.go:164 +0x61
github.com/xeals/signal-back/types.(*BackupFile).Consume(0xc000480070, 0xc0000d11a0, 0xc00003e210, 0xc0000443e0, 0x0, 0x0)
        /home/xeal/go/src/github.com/xeals/signal-back/types/backup.go:260 +0x15d
github.com/xeals/signal-back/cmd.XML(0xc000480070, 0x651500, 0xc000072018, 0x3, 0x1)
        /home/xeal/go/src/github.com/xeals/signal-back/cmd/format.go:192 +0x229
github.com/xeals/signal-back/cmd.glob..func4(0xc00008a580, 0x0, 0x0)
        /home/xeal/go/src/github.com/xeals/signal-back/cmd/format.go:69 +0x403
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x5dd6c0, 0x62a308, 0xc00008a580, 0xc000084100, 0x0)
        /home/xeal/go/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xcf
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x61bf7b, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x623b9c, 0x1f, 0x629597, ...)
        /home/xeal/go/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0x997
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc0000861a0, 0xc00007a000, 0x7, 0x8, 0x0, 0x0)
        /home/xeal/go/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x68e
main.main()
        /home/xeal/go/src/github.com/xeals/signal-back/main.go:52 +0x2d7

• Operating system: Windows 10 Pro x64
• Build version (signal-back --version): signal-back v0.1.7-alpha.1 / proto commit: d6610f0

Quick little issue. You should fix that link!

Thanks for working on signal, my main messenger :)

Now to the issue: Currently backing up signal is a bit of a pain; I cannot update it along with my other android applications using oandbackup; initially it seems to work but, when I try to restore signal on the same phone, I just get error messages.

This is a major pain, mostly because it comes so unexpected; there is no warning during backup, it just fails on restore.

The current solution is to use the signal backup feature; unfortunately this still falls short, for to reasons: (1) it is a bit of a pain to have to backup signal separately (2) the actual encryption/authentification keys are not backed up.

The latter is an issue, because at least my contacts have by now been trained to simply disregard notifications of my security number changing, because I had two phones breaking this year.

In order to remedy this situation, I suggest introducing a restore key/password; this key could be user chosen or could be issued on the first install. I could safely store it in my gpg encrypted password manager on my laptop and just use that to unlock my signal instance after any backup.

SMS Backup & Restore is a popular app for backing up the system SMS database, and Signal used to be able to produce outputs compatible with it. This will probably be my first endeavour.

• Parse SMS and MMS transactions
• Write SMS instances
• Write MMS instances
• Write MMS attachments?
• SMS Backup & Restore can actually fully restore the SMS database with it

Hello,

I'm experiencing an issue where executing the Darwin build against a 913MB .backup file throws:

panic: runtime error: makeslice: len out of range

heres the rest of the gooblyglob:

goroutine 1 [running]:
github.com/xeals/signal-back/types.(*BackupFile).Frame(0x106ea080, 0x1730a540, 0x0, 0x0)
	/home/travis/gopath/src/github.com/xeals/signal-back/types/backup.go:94 +0xc4
github.com/xeals/signal-back/cmd.XML(0x106ea080, 0x1a0960, 0x1061c0e0, 0x3, 0x3)
	/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:141 +0xb0
github.com/xeals/signal-back/cmd.glob..func3(0x10692370, 0x0, 0x0)
	/home/travis/gopath/src/github.com/xeals/signal-back/cmd/format.go:66 +0x205
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x150840, 0x1889f8, 0x10692370, 0x1061ed00, 0x0)
	/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0x9c
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x17cfb1, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x182f53, 0x1f, 0x187e07, ...)
	/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0x8db
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0x10610b60, 0x1061e060, 0x5, 0x6, 0x0, 0x0)
	/home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x585
main.main()
	/home/travis/gopath/src/github.com/xeals/signal-back/main.go:51 +0x1f7

Steps to reproduce:

1. get your Signal-xx.backup, and signal-back_darwin_386 all in the same directory
2. do the thing where you chmod and give executable permission to the darwin file
3. execute ./signal-back_darwin_386 format -f xml > backup.xml signal-2018-06-23-11-41-13.backup
4. enter the pw you got from signal
5. wait for the error

version used:
signal-back v0.1.5-dirty proto commit: d6610f0

I'm assuming this is because the backup file is just huge (not sure why its so big esp when my plaintext exports werent nearly as big), but if not what else do you think would cause the error?

Thanks, Nick

I'm trying to extract a backup made on Signal 4.18.3 on a Samsung Galaxy Alpha.

The backup completed (seemingly) successfully on the phone and the backup file was then transferred to a MacBook Pro using Google's "Android File Transfer" app.

Running signal-back with the "analyse" option on this file seems to work just file, which leads me to believe the backup file is good, when I try to run an extract I get this error:

./signal-back_darwin_amd64 extract -p [pass] signal-[date].backup 
error: failed to extract attachment: encoding `` not recognised. create a PR or issue if you think it should be

I can't figure out if this is an error related to the backup file or signal-back, or how to further diagnose this.

When running signal-back format signal-*.backup > backup.xml, Password: is written to stdout.

Prompts should always be written to stderr so that they are not redirected to a file, where only the actual command output should go.

• Operating system: Windows 8,1
• Build version (signal-back --version): v0.1.6

Detailed description

While running the sript after entering the password i get the error. Out of memory : cannot allocate 80936960-byte block (159844766 in use) fatal erro: out of memory.

Full relevant output

Before having encrypted backups Signal did not allow proper transition from an old phone to a new one, it only allowed the text export and import and media were lost.

So I kept the old phone until Signal introduced the encrypted backups. In the meantime I have used Signal on the new phone. Now that Signal has proper backups, I made an encrypted backup from my old phone. Now I would like to merge my old backup with the backup of my new phone.

I guess this request is related to #44.

Would be great if you could manage to do that, I would consider a donation.

from #29

• Operating system: Windows 10 Pro x64
• Build version (signal-back --version): 0.1.6

Detailed description

While I previously successfully decrypted a first generation Signal backup file, when I tried with one that was created just days ago, it fails with this error.

Full relevant output

runtime: VirtualAlloc of 2955739136 bytes failed with errno=1455
fatal error: out of memory

runtime stack:
runtime.throw(0x5e3947, 0xd)
        /usr/lib/go-1.10/src/runtime/panic.go:616 +0x88
runtime.sysMap(0xc1098b0000, 0xb02d0000, 0x6f1e01, 0x70a138)
        /usr/lib/go-1.10/src/runtime/mem_windows.go:122 +0x13b
runtime.(*mheap).sysAlloc(0x6f16e0, 0xb02d0000, 0x0)
        /usr/lib/go-1.10/src/runtime/malloc.go:470 +0xdb
runtime.(*mheap).grow(0x6f16e0, 0x58168, 0x0)
        /usr/lib/go-1.10/src/runtime/mheap.go:907 +0x67
runtime.(*mheap).allocSpanLocked(0x6f16e0, 0x58168, 0x70a148, 0x31cfe40)
        /usr/lib/go-1.10/src/runtime/mheap.go:820 +0x308
runtime.(*mheap).alloc_m(0x6f16e0, 0x58168, 0xffffffffffff0101, 0x31cfe70)
        /usr/lib/go-1.10/src/runtime/mheap.go:686 +0x126
runtime.(*mheap).alloc.func1()
        /usr/lib/go-1.10/src/runtime/mheap.go:753 +0x54
runtime.(*mheap).alloc(0x6f16e0, 0x58168, 0x3010101, 0x41390a)
        /usr/lib/go-1.10/src/runtime/mheap.go:752 +0x91
runtime.largeAlloc(0xb02ce008, 0x440101, 0x1b06a8)
        /usr/lib/go-1.10/src/runtime/malloc.go:826 +0x9b
runtime.mallocgc.func1()
        /usr/lib/go-1.10/src/runtime/malloc.go:721 +0x4d
runtime.systemstack(0x0)
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:409 +0x7e
runtime.mstart()
        /usr/lib/go-1.10/src/runtime/proc.go:1175

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:363 fp=0xc042033050 sp=0xc042033048 pc=0x44f300
runtime.mallocgc(0xb02ce008, 0x59eaa0, 0x1, 0x0)
        /usr/lib/go-1.10/src/runtime/malloc.go:720 +0x8e1 fp=0xc0420330f0 sp=0xc042033050 pc=0x40fda1
runtime.makeslice(0x59eaa0, 0xb02ce008, 0xb02ce008, 0x4, 0x4, 0x4)
        /usr/lib/go-1.10/src/runtime/slice.go:61 +0x7e fp=0xc042033120 sp=0xc0420330f0 pc=0x43b62e
github.com/xeals/signal-back/types.(*BackupFile).Frame(0xc0420ecfc0, 0xc0b173a410, 0x0, 0x0)
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/types/backup.go:94 +0xf7 fp=0xc0420331d0 sp=0xc042033120 pc=0x56bcc7
github.com/xeals/signal-back/cmd.XML(0xc0420ecfc0, 0x606e40, 0xc042072008, 0x3, 0x1)
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/cmd/format.go:145 +0x119 fp=0xc042033998 sp=0xc0420331d0 pc=0x57b979
github.com/xeals/signal-back/cmd.glob..func4(0xc042088420, 0x0, 0x0)
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/cmd/format.go:67 +0x416 fp=0xc042033a38 sp=0xc042033998 pc=0x57d486
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x5a6140, 0x5ef668, 0xc042088420, 0xc042082100, 0x0)
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xcf fp=0xc042033a60 sp=0xc042033a38 pc=0x504b5f
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x5e20ed, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5e96cb, 0x1f, 0x5ee97c, ...)
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0xa3d fp=0xc042033cd0 sp=0xc042033a60 pc=0x505ddd
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc0420841a0, 0xc04207a000, 0x5, 0x8, 0x0, 0x0)
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x6a7 fp=0xc042033ea8 sp=0xc042033cd0 pc=0x502ea7
main.main()
        /home/xeal/.local/share/go/src/github.com/xeals/signal-back/main.go:52 +0x31e fp=0xc042033f88 sp=0xc042033ea8 pc=0x57e2fe
runtime.main()
        /usr/lib/go-1.10/src/runtime/proc.go:198 +0x20e fp=0xc042033fe0 sp=0xc042033f88 pc=0x42c4be
runtime.goexit()
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc042033fe8 sp=0xc042033fe0 pc=0x451d81

Never mind, I managed to get the program built. However, I'm not opposed to you running a build_all.sh and uploading the results to github anyway. ;)

"Enter your 30-digit password at the prompt (with or without spaces, doesn't matter)."

At the very least, on Windows 7's command prompt, due to the "XML > " forwarding of the command, there is no prompt for a password as the prompt is being sent to the file!

Also this means that "Password: " is at the start of the xml file, possibly corrupting the backup file altogether. I'm not sure this forwarding command is the best way to output when the script is sending prompts into it that the user misses out on seeing.

• Export SMS database
• Export MMS database

Added in 236af7d. The format basically just dumps the entire database(s). Plug it into whatever your favourite analysis tool is.

As the title says, the program fails with that error.
I worked around it by modifying format.go to simply skip any SMS where it couldn't assign the type number to a meaningful variable.

As an aside, after doing that, this program worked perfectly! I copy-pasted the xml into an online xml to json converter, and that worked just fine. From what I can tell, it didn't miss any SMS messages (though I didn't look that hard).

When attempting to verify a large Signal backup I get the above error message.

For analyse:

$ ./signal-back_darwin_amd64 analyse -v -e signal-2018-08-02-12-31-52.backup
Password: xxxxx xxxxx xxxxx xxxxx xxxxx xxxxx
2018/08/02 14:21:39 error reading: initialising frame: unexpected EOF

For format:

$ ./signal-back_darwin_amd64 format -v -f XML signal-2018-08-02-12-31-52.backup
Password: xxxxx xxxxx xxxxx xxxxx xxxxx xxxxx
2018/08/02 14:23:49 MMS export not yet supported
... (repeated many times)
2018/08/02 14:23:56 error reading: initialising frame: unexpected EOF

The extract command works and extracts all images and media which indicates that the password is correct.

Deliberately entering an incorrect password gives a different error message:

$ ./signal-back_darwin_amd64 analyse -v -e signal-2018-08-02-12-31-52.backup
Password: 12345 12345 12345 12345 12345 12345      
Panicked: runtime error: invalid memory address or nil pointer dereference
[slurping 0]
$ ./signal-back_darwin_amd64 format -v -f XML signal-2018-08-02-12-31-52.backup
Password: xxxxx xxxxx xxxxx xxxxx xxxxx xxxxy
Panicked during formatting: runtime error: invalid memory address or nil pointer dereference

The backup file size is 1.1 GB. Using a smaller backup file (233 MB from my son's phone) works without problems. Reducing my message history to the last 500 messages in each thread thus reducing the backup file size to 569 MB does not eliminate the problem.

Decoding and generating the file seems to work, and I when viewing the file in Sublime Text the format seems okay.

When trying to restore with SMS Backup & Restore, though, the file is not shown to you as an option to restore from.

When browsing backup files within the app the file is given a label of "Invalid File - 22.8MB".

Any ideas on what might be causing this issue? Is it likely to be a formatting issue, or an incompatibility with some of my messages?

As mentioned in this comment, the Windows builds do not have the ".exe" extension.

While this was an easy thing to guess for myself and @IslandFun73, it would not necessarily be easy for others.

If I run signal-back on my backup (kind of large ~3.5GB) it stops with the following error message:
signal-back v0.1.6-alpha.1:

./signal-back_linux_amd64 extract -o output signal-2018-07-11-02-19-01.backup 
Password: XXXX
Panicked: runtime error: slice bounds out of range
[frame: starting at 3864515, size 0]

signal-back v0.1.5:

./signal-back_linux_amd64 extract -o output signal-2018-07-11-02-19-01.backup 
Password: XXXX
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
github.com/xeals/signal-back/types.(*BackupFile).Frame(0xc4200f20e0, 0xc4c7293090, 0x0, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/types/backup.go:97 +0x4d1
github.com/xeals/signal-back/cmd.ExtractAttachments(0xc4200f20e0, 0x6, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/cmd/extract.go:56 +0xcc
github.com/xeals/signal-back/cmd.glob..func2(0xc4200a0420, 0x0, 0xc4200a0420)
        /home/travis/gopath/src/github.com/xeals/signal-back/cmd/extract.go:43 +0x8c
github.com/xeals/signal-back/vendor/github.com/urfave/cli.HandleAction(0x5939a0, 0x5dacf0, 0xc4200a0420, 0xc42009a100, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/xeals/signal-back/vendor/github.com/urfave/cli.Command.Run(0x5cf178, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5d66e7, 0x24, 0x5d6b6a, ...)
        /home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/command.go:210 +0xa36
github.com/xeals/signal-back/vendor/github.com/urfave/cli.(*App).Run(0xc42009c1a0, 0xc4200900f0, 0x5, 0x5, 0x0, 0x0)
        /home/travis/gopath/src/github.com/xeals/signal-back/vendor/github.com/urfave/cli/app.go:255 +0x6a0
main.main()
        /home/travis/gopath/src/github.com/xeals/signal-back/main.go:51 +0x2e1

I'm not sure, but it could be related to #19

If I can help with any additional information just let me know.

Hi,
The README talks about PASS but does not explain where does the password comes from. I am using Signal for years and did not set any password within Signal for Android. Do I have an empty password, just requiring to press Return when asked for the PASS?

(I can't confirm that an empty password works since I got panic errors in any case which I'd write into a separate issue, if they are not related to my password issue.)

The filenames of images are numbers such as 1497235121791.jpg
I assume it is a unix timestamp?
As most of these files don't have exif tags, it would be cool if the actual file modification date and file creation date would be set from the filename.
While it is possible to do that afterwards in a helper script, it would be comfortable to have this built-in.

It is a known fact that the CSV export does not (by its very nature) contain the images of a conversation.

However, looking at the messages in detail, I realized that for messages that contain an image and text, the text is missing as well. The message just plain doesn't exist in the output.
That obviously leads to missing parts of the conversation.

I assume the algorithm generating the filenames in extract mode is deterministic.
Would it be possible to include all messages with media in CSV mode, but replacing the actual image with the text [attachment:$filename], so that one could check the extracted attachments for that file name?

Either way, at least the text of such a message should be exported, regardless of attachment handling.

Edit: Affects raw export as well. Can't test XML because of #16.

Trying to output an XML format generates a 'panic: undefined SMS type: 0x57' error. Looking through the cvs output of the backup I found the message. It is the result of selecting Invite in the conversation menu.

First there is a message with the body "Let's switch to Signal: https://sgnl.link/1KpeYmF" and a type of 23 (sent). Immediately following is the message with type 87 (0x57) with the same body.