Build tools dependencies "Python executable not found" / leveldown about hoodie-plugin-store-crypto HOT 17 CLOSED

@Terreii
I think I found the problem and it is probably an issue in the used crypto library respectively its dependencies (native-crypto, pbkdf2 and normalize).

But fist once again the initial situation:
I use hoodie-plugin-store-crypto and already create some data in the local db (password and salt defined by me). I have also some functionality to export and import the local db to a file.
I also have now a “real” hoodie backend sync (thank you to merge my pouchDB7 PR). Everything runs local in a non https environment. (Hint: non https is probably on root cause of the different behavior... explanation later.)

The Problem:
When I access my app in the browser (chrome) by http://127.0.0.1:port... (respectively locahlhost) Everything works perfectly. I can import/ export also with the crypto-plugin and also the hoodie backend sync works.

When I access the same thing over the machines network ip http://192.168.0..... (regardless if I’m on the same machine or another device in the lan) I can’t access/ unlock the local db ( 'Name or password is incorrect' exception). Also if the db is synced by hoodie (without any of my import/export stuff I can’t unlock the db, even though the sync works perectly)

The code:
I unlock the store with hoodie.cryptoStore.unlock.

• Localhost case works -> no problem

• Machine IP case: The unauthorized exception comes from the createKey function. The res.key in the promise is empty. (this is not when accessing by localhost).
  

The exception thrown here by undelaying crypto lib is :

Maybe it would be good if you attach the “real exception” to your “unauthorized” exception)

So I dig into the crypto libs:
In the pbkdf2 is a “checkNative” function when “localhost” and “machine IP” goes different ways. In localhost the checkNative returns true and everything is ok, because there is a check if (global.crypto) is available. It looks like chrome provide this native api only in “secure origins”. Localhost or 127.0.0.1 even without https (!) is probably handled as secure origin, everything else not. (Here is an emotional discussion about this: w3c/webcrypto#28 )

So because there is no native crypto api we move one to “compat.pbkdf2” an here is the real bug:
There is a string normalization that converts “sha256” to “SHA256” (convert to capital letters).

After this, there is a part that try go get the right size/length for the algo out of an defined array…

unfortunately the algos there are written in lower case and so the length for “SHA256” is undefined. The rest fails and the returned key is an empty array.

If I hard overwrite the “SHA256” back to “sha256” everything works like a charm also in “non secure origins”

I can’t imagine that I’m the first one who trapped in this issue.. could you check if the used crypto libs are outdated and look if this bug has already been fixed?

from hoodie-plugin-store-crypto.

No problem!

Oh, yes! I did stumble over this bug, too! It is fixed in 7fd0624. And in v2.2.1 (just released)!

from hoodie-plugin-store-crypto.

Nice, with the new release it works like a charm!
Thanks for the great work.

from hoodie-plugin-store-crypto.

Oh no! Sorry that you had to do my work 😔. Thank you!
I didn't test it on Chrome on a remote address! Firefox still supports crypto.subtle on insecure origins.
The main reasons for my usage of native-crypto is to use the same API everywhere and that it is used in crypto-pouch.
But it looks like browserify's pbkdf2 also uses crypto.subtle if present. And running on node is only used in the tests. Sadly directly "using" nodes crypto won't work, because it looks like rollup doesn't include the crypto plugin by default.
So I'm going to switch to crypto-browserify/pbkdf2.

For encrypting and decrypting I have to do some more digging.

from hoodie-plugin-store-crypto.

🎉 This issue has been resolved in version 3.0.1 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

from hoodie-plugin-store-crypto.

Sorry, for taking so long to answer.

I did look in the (production) dependencies, and as far as I have looked, none require leveldown and only rsa-keygen and raw-ecdsa use C-extensions (they are optional and not used).

The devDependencies on the other hand require leveldown.

So I have to ask: do you install this plugin as a dependency of a project or do you want to contribute to it?
If the later is the case: first of all, thank you! And second: you could try to remove hoodie from the devDependencies-list in package.json. It is only there to have an running example. It uses leveldown.

If you use it as a dependency of a project: I might need more info:

• Do you use the full hoodie-package?
• What package was before every node-gyp rebuild line? There are probably multiple packages that call node-gyp rebuild on install.

But you could try to install pouchdb-browser. It doesn't come with leveldown. (This option only works if you don't use the full hoodie-package).
At the moment I don't get my Windows PC to run and I only have Macs, and they come with Python installed. Which might mean that you have to install python as a last option ...

from hoodie-plugin-store-crypto.

Thanks for your response.

I try to use it as a dependency in a project to take my first steps whit hoodie.
There I use hoodie/store-client and pouchdb-browser for client/ offline storage. (The pouchdb backend sync I’ll probably use late but not in my first evaluations)
And because I don’t want to store data readable in in the indexedDB , I came across your project and try to install is.

The first output after “npm install --save hoodie-plugin-store-crypto” is:

[email protected] install C:...\node_modules\raw-ecdsa
node-gyp rebuild

C:...\node_modules\raw-ecdsa>if not defined npm_config_node_gyp (node C:...\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node "C:...\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "C:....windows-build-tools\python27\python.exe", you can set the PYTHON env variable.
gyp ERR! stack at PythonFinder.failNoPython (C:...\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:484:19)
gyp ERR! stack at PythonFinder. (C:...\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:509:16)
gyp ERR! stack at C:...\AppData\Roaming\npm\node_modules\npm\node_modules\graceful-fs\polyfills.js:282:31
gyp ERR! stack at FSReqWrap.oncomplete (fs.js:153:21)
gyp ERR! System Windows_NT 10.0.17763
gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\...\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:...\node_modules\raw-ecdsa
gyp ERR! node -v v10.15.3
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok

[email protected] install C:...\node_modules\rsa-keygen
node-gyp rebuild
...

I use node.js v10.15.3 and npm 6.9.0

from hoodie-plugin-store-crypto.

I get similar errors while using node v10. With the difference of not python being the problem, but raw-ecdsa and rsa-keygen not supporting node v10.
As long as the installation finish and hoodie-plugin-store-crypto appears in your package.json it should work.
raw-ecdsa and rsa-keygen are not used and the C-code, that causes the error, is for node and isn't used in the browser.

The hoodie-plugin-store-crypto setup-function expects an object with the store and account.
If you only use hoodie/store-client you must warp it in an object:

const h = {
  store: yourHoodieStoreInstance,
  account: {
    on: function () {} // this is for auto-locking on sign-out
  }
}
require('hoodie-plugin-store-crypto')(h)
// The cryptoStore will be added to the object/hoodie-instance.
// This is done be compatible with the hoodie-plugin-system.
const cryptoStore = h.cryptoStore

from hoodie-plugin-store-crypto.

Does it work?

from hoodie-plugin-store-crypto.

Sorry for my late respons.

Basically yes. Even with the installation-errors it looks like, the crypto-plugin is loaded and should work.

But I have some trouble on accessing the db and so I must do some more investigating.
Like in your example I create a DB with store.setup().
It seems to work and in the console the warning Couldn't pull docs from remote. Creating new salt! is shown and the then-promis is hit.
When I try to unlock with store.unlock() in this promise I see a net::ERR_CONNECTION_REFUSED (it try to connect to the remote pouchdb, but of course it's not there because I currently don’t want to sync) So unlock returns a CustomPouchError:

result:
	doc_write_failures: 0
	docs_read: 0
	docs_written: 0
	end_time: "..."
	errors: []
	last_seq: 0
	ok: false
	start_time: "..."
	status: "aborting"
	__proto__: Object
	__proto__: Error

When I try to access something in the db I get an unauhorized "Name or password is incorrect."

from hoodie-plugin-store-crypto.

I have one more question:
Is it possible Export/Import (e.g. as json) an encrypted store locally and import it to another machine (not speaking about the pouch sync.. more like a local backup)

1. I set up a database with a password and salt. (Write and read to the local indexdb works perfectly)
2. After this I've done a dump with 'pouchdb-replication-stream' and save all (the encrypted data) as json to a text file.
3. I drop my db an import every thing back with 'pouchdb-load'.

If I do this on the same machine everything works smooth. But if I try to import this stuff from on to another machine, I get the 'Name or password is incorrect' again. (In chrome dev tools I can see my import recreated the db and the structure looks ok)

As I said I use pouchdb-browser and @hoodie/store-client and no backend sync so I didn't provide any account/login. Is this my problem or is there something other machine/ session specific that will break import/export experiment?

from hoodie-plugin-store-crypto.

Interesting! It should work!
But I have to look into it …

from hoodie-plugin-store-crypto.

Here are some code snippets (using typescript).. maybe you can see an error:

It works on the same machine, even from a non-incognito chrome into a incognito session.
But not to an chrome on another pc.

Set up the DB:

const PouchDB = require('pouchdb-browser');
const hoodieStore = require('@hoodie/store-client');
const pouchdb = PouchDB.default.defaults({ revs_limit: 1, auto_compaction: true });

const store = hoodieStore('dbname', {
    PouchDB: pouchdb,
    remote: this.remoteUrl
});

//setup crypto
const cryptoStore = {
    store,
    account: {
        on: () => { } 
    }
}

require('hoodie-plugin-store-crypto')(cryptoStore);
this.store = (cryptoStore as any).cryptoStore;

this.settingsStore = this.store.withIdPrefix('settings/');
...
const resetKeys = await this.store.setup(password, salt)

The Export and JSON Download

const PouchDB = require('pouchdb-browser');
const replicationStream = require('pouchdb-replication-stream');
const MemoryStream = require('memorystream');
const pouchdb = PouchDB.default.defaults({ revs_limit: 1, auto_compaction: true });
pouchdb.plugin(replicationStream.plugin);
pouchdb.adapter('writableStream', replicationStream.adapters.writableStream);

const stream = new MemoryStream();
let dumpedString = '';
stream.on('data', (chunk: any) => {
    dumpedString += chunk.toString();
});

const outer = this;
const db = pouchdb('dbname');
db.dump(stream).then(() => {
    ...
    //download as json
    const dataStr = "data:text/json;charset=utf-8," + dumpedString;
    const downloadAnchorNode = document.createElement('a');
    downloadAnchorNode.setAttribute("href", dataStr);
    downloadAnchorNode.setAttribute("download", exportName + ".json");
    document.body.appendChild(downloadAnchorNode);
    downloadAnchorNode.click();
    downloadAnchorNode.remove();
}

The Import

const PouchDB = require('pouchdb-browser');
const pouchdb = PouchDB.default.defaults({ revs_limit: 1, auto_compaction: true });
const load = require('pouchdb-load');
pouchdb.plugin(load);

const db = new pouchdb('dbName');
db.load(dbJson)

from hoodie-plugin-store-crypto.

Sorry, for not answering. I had some busy weeks. Next week I will look into it.

from hoodie-plugin-store-crypto.

No Problem, we all have a “job” :-)
I’ve done some more tests and just now I noticed the following behavior:

I serve my progressive web app by the „http-server“ node module, to test some manifest/ chrome pwa installations stuff… pouch/hoodie are still offline, so no backendsync. (so just hoodie in local browser)

“http-server” provides the app on multiple ip addresses (localhost, machines network ip, etc).

If I access it over localhost 127.0.0.1 everything works perfect. If I move the http-server to another machine and access it there also over localhost I can import my data from the other machine!

But if I access my app e.g. over the machines own ip (or another remote ip) I get the “wrong password” error when I try to import data. (Creating a new hoodie/crypto database and store data to it works normally also in this scenario)

from hoodie-plugin-store-crypto.

Sorry that I didn't respond again.

I tried to make your examples work, but couldn't, which resulted in me procrastinating. 😔

But your example looks like you call cryptoStore.setup right away. Is this correct?
setup tries to fetch new versions of the required document (hoodiePluginCryptoStore/salt). It uses hoodie-store for it (This is also the original bug). If this fails or there is none then it will create a new document. But fails if there are some.
The salt is randomly created. If then the other version comes in, it will create a conflict. Resulting in one of the two to be the new version. You can read more about it in PouchDB's conflict tutorial.
Also the plugin will not listen to changes to the salt-doc while it is unlocked!

You can test if there is a conflict if you use:

// db is the PouchDB instance that hoodie-store uses.
hoodie.store.db.get('hoodiePluginCryptoStore/salt', {
  conflicts: true
})
  .then(doc => {
    const text = doc._conflicts.length > 0
      ? 'doc has conflict'
      : 'doc has no conflict'
    console.log(text)
  })

If this is the case: then it will help if you call setup after importing the data for the first time. And call lock and unlock every time a new version of hoodiePluginCryptoStore/salt is imported.

Other than that I have currently no idea.

from hoodie-plugin-store-crypto.

Hi, currently I have some more issues with hoodie/pouchdb-browser and so for the moment I temporarily stop my crypto-store implementation to minimize the issues with the hoodie backend sync.
I’ll try this again later and will respond here.

from hoodie-plugin-store-crypto.