Code Monkey home page Code Monkey logo

go-jwlm's Introduction

Coverage Status

go-jwlm

A command-line tool to easily merge JW Library backups, written in Go. For the iOS version, visit ios-jwlm.

go-jwlm allows you to merge two .jwlibrary backup files, while giving you control of the process - your notes are precious, and you shouldn‘t need to trust a program solving possible merge conflicts for you.

I created this project with the goal of having a tool that is able to work on multiple operating systems, and even allowing it to be incorporated in other programs as a library (like an iOS app). It is - and will be for quite some time - a work-in-progress project, so I‘m always open for suggestion and especially reports if you encounter an unexpected behavior or other bugs.

The usage is pretty simple: you have one command, you name your backup files - and press enter. The tool will merge all entries for you. If it encounters a conflict (like the same note with different content or two markings that overlap), it will ask you for directions: should it choose the left version or the right one? After that is finished, you have a nicely merged backup that you can import into your JW Library App. The first merge process might take some time because of the number of possible conflicts, depending on how far apart you backups are. But if you merge them regularly, it should be a matter of seconds :)

Usage

go-jwlm merge <left-backup> <right-backup> <merged-backup>

If a conflict occurs while merging, the tool will ask for directions: should it choose the left version or the right one. For that, it shows you the actual entries (I‘m planning to improve that view and add more information, especially about publications, in the future). If you are not sure what to do, press ? for help.

Resolve conflicts automatically

Currently, there are three solvers you can use to automatically resolve conflicts: chooseLeft, chooseRight, and chooseNewest (though the last one is only usable for Notes). As their names suggest, chooseLeft and chooseRight will always choose the same side if a conflict occurs, while chooseNewest always chooses the newest entry.

You can enable these solvers with the --bookmarks, --markings, --notes, and --inputFields flags:

go-jwlm merge <left-backup> <right-backup> <merged-backup> --bookmarks chooseLeft --markings chooseRight --notes chooseNewest --inputFields chooseLeft

The conflict resolvers are helpful for regular merging when you are sure that one side is always the most up-to-date one. For a first merge, it is still recommended to manually solve conflicts, so you don't risk accidentally overwriting entries.

Compare two backups

To quickly compare two backup files and check if their content is equal, you can use the go-jwlm compare <left-backup> <right-backup> command. This one is mainly used for validation, but might be helpful in other situations :)

Installation

You can find the compiled binaries for Windows, Linux, and Mac under the Release section.

Installation using Homebrew (Mac and Linux)

go-jwlm can be easily installed using Hombrew:

brew install andreassko/homebrew-go-jwlm/go-jwlm

See the instructions on how to install Homebrew at https://brew.sh

Mobile version

If you want to merge backups using your iPhone or iPad, have a look at JWLM. It uses the whole merge logic of go-jwlm, but wraps it in a nice and easy to use iOS app. It is already available on the App Store.

There might come an Android version at some point, but as I personally don't use any Android devices, it is - unfortunately - not the highest priority for me right now. If you are interested in helping with this project, feel free to contact me or just start it on your own :)

Technical background

Gomobile enables to include Go packages within a mobile application. As there are still some limitations, I built a wrapper for most structs and methods. To build the binding on your own, install Gomobile, change into the gomobile directory of this repo and run gomobile bind -target <ios or android>.

A word of caution

It took me a while to trust my own program, but I still keep backups of my libraries - and so should you. Go-jwlm is still in beta-phase, so there is a possibility that something might get lost because of a yet-to-find bug. So please keep that in mind and - again - if you found a bug, feel free to open an issue.

Need help?

Something is unclear, you have suggestions for documentation or you found a bug? Feel free to open an issue. I‘m happy to help, though please be patient if it takes a while for me to respond :)

go-jwlm's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-jwlm's Issues

Position of TagMap entries are non-deterministic

When running the same merge multiple times, it can happen that the position of TagMap entries change. Maybe it is possible
to introduce some sorting in the merge process of TagMaps to make it deterministic.

Steps to reproduce:

  • Create a merged backup from the left and right test cases in merge_test.go, select right side if a conflict happens
  • Run go-jwlm merge left.jwlibrary right.jwlibrary merged.jwlibrary --bookmarks chooseRight --notes chooseRight --markings chooseRight && go-jwlm compare merged.jwlibrary <other-merged-db> multiple times until ❌ Backups are NOT equal is returned

Minor bug with merged export

78F760FE-DE3D-43FD-BD6E-7C70AD9F6E33
Hi Andreas,

looks like I found a minor bug while trying to export a merged file.
Actually I’m not really understanding the error message…do you have an idea? Hast ne Idee?

Best regards,
Alex

Heuristic to decide if NwtstyMigration necessary is unreliable

Currently, the heuristic to decide if we need to migrate locations from the regular bible translation to the study one simply checks if the majority on one side has already been migrated (see

func decideMigration(langCounts map[int]*bibleEditionCounts) map[int]MergeSide {
). This can be unreliable as reported in #97. It would be better to check for collisions of the same userMarkGUID or NoteGUID to determine if one side has been upgraded.

Testing: Test if LocationIDs are properly updated when merging

While working on #52 I noticed that for Notes and TagMaps the merging tests don't notice if

merger.UpdateLRIDs(left.Note, right.Note, "LocationID", locationIDChanges)
and
merger.UpdateLRIDs(left.TagMap, right.TagMap, "LocationID", locationIDChanges)
are commented out. There is a similar issue in:
merger.UpdateLRIDs(dbw.leftTmp.Note, dbw.rightTmp.Note, "LocationID", locationIDChanges)
.
The tests should be updated to fail if LocationsIDs are not properly updated.

Use catalog.db for better insights into publications when merging

Sometimes it can be hard to figure out what publication a bookmark or marking belongs to when only looking at the KeySymbol or DocumentID. We could benefit from the catalog that JWLibrary is using to show all available publications.

Proposed approach:
Create an additional package that is responsible for looking up the publication a KeySymbol or DocumentID is belonging to. We are able to use API for that:

  1. Fetch the current manifest from https://app.jw-cdn.org/catalogs/publications/v4/manifest.json
  2. Download the current catalog from https://app.jw-cdn.org/catalogs/publications/v4/{MANIFEST}/catalog.db.gz
  3. Cache it locally and only fetch it again if we are not able to lookup a publication (because it may be too new)
  4. Query the DB for the KeySymbol or DocumentID (using the PublicationDocument table).

Importing TagMap: „Converting NULL to int is unsupported“

The following error was reported when importing a backup:

Error while scanning results from SQLite database: sql: Scan error on column index 4, name "Tagld": converting NULL to int is unsupported

This looks to be an issue when importing TagMaps.

Make `sortedSolutionKeys` more efficient by only running Regex once

func sortedSolutionKeys(conflictSolution map[string]MergeSolution) []string {

The function parses each key for every iteration of sort, so it does a lot of work multiple times. This results in the function taking almost 500ms alone! We could improve this by pulling out the number infront of the uniqueKey first, running the sort on this and later put it back together. This migth speed up the function quite a bit.

Implement Playlist

PlaylistItem, PaylistItemChild, and PlaylistMedia are not implemented yet and would get lost currently. Has to be implemented before getting out of alpha-phase!

Merge latest Beta to Releases for Windows

Dear Andreas. Pleased to see a way that we can still merge our backup files. Is there any chance you can release a binary of your latest so we can use in windows? Tx so much

Not sure how to make this work in windows - Novice

I am so excited that I may have an option to use this app since jwmerge is no longer working. However I am not even close to to programmer so I am not sure the steps that show "Usage" are clear to me. I have downloaded the .msi for go-jwlm but everything I try at the command prompt gives me a series of errors. I don't know how to choose any files using this application. can someone help me with a step by step tutorial or video to show me how to use this tool?

I would appreciate it.

Thanks... BL

"The '<' operator is reserved for future use"

Hi.

Working with windows 11; powershell (latest).

I get this information below after running go-jwlm merge

"The '<' operator is reserved for future use".

Thanks for your hints.

Have a great day2day - Roland

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

  • Update golang.org/x/mobile digest to 1ad2df2
  • Update module github.com/sergi/go-diff to v1.3.1
  • Update module github.com/sirupsen/logrus to v1.9.3
  • Update actions/checkout action to v4
  • Update actions/setup-go action to v5
  • Update module github.com/MakeNowJust/heredoc to v2
  • 🔐 Create all rate-limited PRs at once 🔐

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.


Warning

Renovate failed to look up the following dependencies: Could not determine new digest for update (go package github.com/cavaliercoder/grab).

Files affected: go.mod


Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

github-actions
.github/workflows/coverage.yml
  • actions/setup-go v3
  • actions/checkout v3
  • jandelgado/gcov2lcov-action v1.0.0
.github/workflows/main.yml
  • actions/checkout v3
  • reviewdog/action-golangci-lint v2
  • actions/setup-go v3
  • actions/checkout v3
  • actions/setup-go v3
  • actions/checkout v3
.github/workflows/release.yml
  • actions/checkout v3
gomod
go.mod
  • go 1.20
  • github.com/AlecAivazis/survey/v2 v2.3.2
  • github.com/DATA-DOG/go-sqlmock v1.5.0
  • github.com/MakeNowJust/heredoc v1.0.0
  • github.com/Netflix/go-expect v0.0.0-20210722184520-ef0bf57d82b3@ef0bf57d82b3
  • github.com/buger/goterm v1.0.1
  • github.com/cavaliercoder/grab v1.0.1-0.20201108051000-98a5bfe305ec@98a5bfe305ec
  • github.com/codeclysm/extract/v3 v3.0.2
  • github.com/davecgh/go-spew v1.1.1
  • github.com/hinshun/vt10x v0.0.0-20180809195222-d55458df857c@d55458df857c
  • github.com/jedib0t/go-pretty v4.3.0+incompatible
  • github.com/klauspost/compress v1.15.1
  • github.com/mattn/go-sqlite3 v1.14.8
  • github.com/mitchellh/go-homedir v1.1.0
  • github.com/mitchellh/go-wordwrap v1.0.1
  • github.com/pkg/errors v0.9.1
  • github.com/sergi/go-diff v1.2.0
  • github.com/sirupsen/logrus v1.8.1
  • github.com/spf13/cobra v1.4.0
  • github.com/spf13/viper v1.11.0
  • github.com/stretchr/testify v1.7.1
  • golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028@d2bd2a29d028

  • Check this box to trigger a request for Renovate to run again on this repository

Schema Version

failed to import left backup: Schema version is incompatible. Should be 13 is 14. You might need to upgrade to a newer version of JW Library first

Not a bug but want to contribute

Hi Andreas!

I just want to say thanks for this utility.

I want to be able to contribute as well. I’m not sure if you have enough time for open source contributions. If you do, do you have any guidelines?

Run `VACUUM` before exporting Database

The VACUUM command optimizes and cleans up the SQLite DB, resulting in a decreased DB size. We should simply run VACUUM; before finally exporting the DB into the .jwlibrary file.

Bookmark "conflicts" refer to completely different bookmarks

The example image shows two very different bookmarks shown as a conflict.
I am assuming that in the backup files they share an ID of some sort, but from a users perspective, if they refer to different publications or location in a publication, they are independent and should not be in conflict.

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.