Backend answers for exchange rates in a naive way.

Function to improve: BlockchainController.GetExchangeRatesAsync.

Outline of performance increases:

// ToDo: Implement caching for instant answers.
// ToDo: Implement redundancy, call another API if SmartBit fails, if that fails, call another one.

I added Nicolas's NodeBuilder (the one that's used in NTumbleBit, so it works on Linux and Windows) and I added some basic tests for the IndexFilterBuidler:

dc5d694#diff-4b985814ee76663ced0a1d3899107e09R202

More tests are appreciated, especially a reorg test. I can test it with regtest rpc's invalidateblock, so I made sure reorgs work properly, yet unit test may be also needed here for further code changes.

Reorgs will work properly after this PR is merged, now they only half working: #48

It's the most critical part of the wallet. Incorrectly built transaction is a disaster.

At the time of writing, this function is not yet implemented.

Use the NBitcoin.TestFramework instead of our own copy of the NodeBuilder class.

#38

Its server side pair is the IndexFilterBuilder. Sibling issue: #51

For reference basic tests for this service can be found here:

Functionalities:
Increment
Decrement
operator +
operator -

And basically any operator that you can figure and makes sense.

For this you must understand the Height struct. You may want to start with the xml comments, that helps you in the understanding. It's a quite handy class. It's basically an int, plus it has two special values: Height.Unknown and Height.Mempool

Go through what RPC-s are used in the tesoftware and if the RPC is used manually with SendCommandAsync, then implement and add them to NBitcoin. Try doing full job, ergo don't do things, like throwing away an RPC response, because that doesn't seem relevant.

Also add tests.

It's implemented to int-s only at the moment.
Maybe it could even be implemented for double, decimal, Money and whatever other often used types there are that makes sense.

#38

I implemented a working filtering, but it is rarely using @lontivero your concepts from previous PR.

If you think some concepts/code can be replaced, then please do it.

Issue opened, and reproduced here: MetacoSA/NBitcoin#345

This is why the OSX test is failing currently.

In MemPoolBehavior:

foreach (var inv in invPayload.Inventory.Where(inv => inv.Type.HasFlag(InventoryType.MSG_TX)))
{
	// if we already have it continue;
	// ToDo: Make sure TransactionHashes don't overflow
	if(!MemPoolService.TransactionHashes.Add(inv.Hash))
	{
		continue;
	}

	payload.Inventory.Add(inv);
}

It says

No test is available in /Users/travis/build/nopara73/MagicalCryptoWallet/MagicalCryptoWallet.Tests/bin/Debug/netcoreapp2.0/MagicalCryptoWallet.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

https://travis-ci.org/nopara73/MagicalCryptoWallet/jobs/340936221

@lontivero Could you create new branches with the traditional upstream-origin GitHub way? Right now you are working inside my repository, since you have a commit access. No worry about the existing branches, just keep in mind for new ones, ok?

https://github.com/nopara73/MagicalCryptoWallet/blob/master/MagicalCryptoWallet.Backend/README.md

Whenever I'm using Interlocked, I always end up using it with a long, due to it's the only type that Interlocked.Read() is implemented for.

Is it ok to write an extension method for int, too?

Moving on to client implementation there are different services/jobs the client must implement, some of it can be done without backend, some of it cannot, some of it which can be brought over from HiddenWallet, some of it which cannot.

• Connection maintainer service to the Bitcoin peer to peer network.
• Tracker/Indexer service, that tracks balances, transactions, etc...
• Mempool service.
• Block downloading service.
• Block store service.
• Filter downloading service.
• FeeRate service - not sure this is needed.

The services are also not terribly well thought out by me, I roughly sketched up what HiddenWallet does, we'll make decisions in implementation time, because it was quite slow (though it's possible, because of the millions of blocks parsed and indexed real time.) I'll try to make sure it'll be a cleaner design this time.

Issue opened here: MetacoSA/NBitcoin#346

#73

MetacoSA/NBitcoin#377

#38

I implemented a working filtering, but it is not optimized.

#38

Right now, I'm just periodically asking for block from the RPC, and if the block height doesn't exist, then I wait 1 sec and continue. With waitforblock it'd be more elegant.

Current code:

Block block = null;
try
{
	block = await RpcClient.GetBlockAsync(height);
}
catch (RPCException) // if the block didn't come yet
{
	// ToDO: If this happens, we should do `waitforblock` RPC instead of periodically asking.
	// In that case we must also make sure the correct error message comes.
	await Task.Delay(1000);
	continue;
}

#38

Right now, when a block is queried from the RPC and if its prevhash is not the hash of our last index's hash, then we delete the last index and continue. The problem, we need to keep track of an utxoset, too, so we have to reverse the last operation from that, too. Which is a pain in the ass. The current code:

// ToDo: IMPORTANT! The Bech32UtxoSet MUST be recovered to its previous state, too!
if (prevHash != block.Header.HashPrevBlock) // reorg
{
	using (await IndexLock.LockAsync())
	{
		Index.RemoveAt(Index.Count - 1);
	}

	// remove last line
	var lines = File.ReadAllLines(IndexFilePath);
	File.WriteAllLines(IndexFilePath, lines.Take(lines.Length - 1).ToArray());
	continue;
}

Backend currently estimates fees in a naive way. Avoiding redundant RPC commands can lower resources as much as 99%.

Function to improve: BlockchainController.GetFeesAsync.

Outline of performance increases:

// ToDo: This is the most naive way to implement this.
// 1. Use the sanity check that under 5 satoshi per bytes should not be displayed.
// 2. Use the RPCResponse.Blocks output to avoid redundant RPC queries.
// 3. Implement caching.

I have a general encrypy/decrypt function: StringCypher.

It fails once in a while. Appveyor Windows logs:

[xUnit.net 00:00:06.4325447]     CipherTests [FAIL]
[xUnit.net 00:00:06.4328704]       Assert.Throws() Failure
[xUnit.net 00:00:06.4329209]       Expected: typeof(System.Security.Cryptography.CryptographicException)
[xUnit.net 00:00:06.4329547]       Actual:   (No exception was thrown)
[xUnit.net 00:00:06.4338087]       Stack Trace:
[xUnit.net 00:00:06.4349705]         C:\projects\magicalcryptowallet\MagicalCryptoWallet.Tests\CryptoTests.cs(44,0): at MagicalCryptoWallet.Tests.CryptoTests.CipherTests()
Failed   CipherTests
Error Message:
 Assert.Throws() Failure
Expected: typeof(System.Security.Cryptography.CryptographicException)
Actual:   (No exception was thrown)
Stack Trace:
   at MagicalCryptoWallet.Tests.CryptoTests.CipherTests() in C:\projects\magicalcryptowallet\MagicalCryptoWallet.Tests\CryptoTests.cs:line 44

Possible way to reproduce it:

1. Write a test that assert throwing the CryptographicException many times and hope it'll fail once in a while, too.

It's a straightforward improvement I believe.

It'd be really good if CI would be done on different platforms in NBitcoin. I'm afraid, there'd be some really nasty issues and it'd take a lot of time, I'm not sure it worth it. Maybe try spending 3 days on it and if the tests still don't pass, then leave it?

https://twitter.com/nopara73/status/968381548310429696

@lontivero Could you take a quick look into this? I've never heard of it. It may be simpler than the notification script we are using.

@lontivero In the AuthenticateMessageTest there is 10000 of this message if you run the tests in debug mode in Visual Studio: Exception thrown: 'System.Security.Cryptography.CryptographicException' in MagicalCryptoWallet.dll.

Is there any way to make it not write the exception to Debug output that without turning off the test?
If there is no, I think we should lower the cycle to run 3 times only.

The Electrum Wallet is distributed with the privacy-oriented Tails operating system and we could do the same.

• ActionHistoryHelper should be turned off while initial index building is in process. It be turned on from best height - 100 blocks maturity.
• Similarly any Index reorgs should not happen initial index building phase. Eg, this if condition should check for IIB (Initial Index Building, I'm abbreviating it similarly to IBD- Initial Blockchain Downloading.)

// In case of reorg:
if (prevHash != block.Header.HashPrevBlock)
{
	Logger.LogInfo<IndexBuilderService>($"REORG Invalid Block: {prevHash}");
	// 1. Rollback index

• ActionHistoryHelper should drop its oldest history record, when it hits the number 100. (To prevent memory leak.)

This way we can do a couple of things before the user logs in (like establishing connections)

#38

It takes like half a day right now. If possible, it should be made faster.

Both on main and testnet. And update the readme of Backend:

At initial syncronization the wallet must specify the hash of the first block that contains native segwit output. This hash must be hard coded into the client. (ToDo: find the hash.)

https://github.com/nopara73/MagicalCryptoWallet/blob/master/MagicalCryptoWallet.Backend/README.md

Utxo History is reorgable until 100 blocks. This is currently not tested. The reorg test only tests the Index reorg (which is currently reorgable until forever)

The UTXO history reorgability was fully implemented here: 0b5ce8a

It will speed up the tests.
On travis the datadir is in home .magicalcryptowallet.
On appveyor the datadir is in appdata MagicalCryptoWallet.

I intend to form the folder structure of this working directory the following way:
datadir/{project name}
For example for the backend project on Windows it'll be: MagicalCryptoWallet\Backend.
Tests are even more structured, I plan to put every test data into different directory, eg:
MagicalCryptoWallet\Tests\FooTestAsync.

The main problem is, there is no cross platform GUI for .NET Core, like Windows Forms or WPF.

HiddenWallet uses Electron, which is basically a "lean" chrome browser. Therefore the architecture looks like having a local http api running (daemon written in ASP.NET Core) and the GUI was written in HTML/CSS/JavaScript, which communicates with the daemon.

Since then, Electron may have been somehow integrated into .NET Core with Electron.NET: https://github.com/ElectronNET/Electron.NET so if we want to keep working with it we must go with Electron.NET.

To drop it here, there is also another framework called Avalonia, which is "a WPF-inspired cross-platform XAML-based UI framework." https://github.com/AvaloniaUI/Avalonia
This had no documentation at the time, but now it has decent amount. Also it looks like much more popular than Electron, so this may be a better choice?

Related issue: zkSNACKs/WalletWasabi#96

Or just simulate to work with it (eg. fees cannot be queried.)

The way I'd go about it:

1. Change to regtest in the Backend's Config.json.
2. Start bitcoind in regtest.
3. Start backend and go to swagger.
4. Go through all the requests in swagger.
5. Fix what's breaking.

@molnard told me about this idea, it could improve performance a lot. I always used ==.

This is related, but not limited to: #76

Just skim it through, and say "Go, No-Go" or to modify something.
https://github.com/nopara73/MagicalCryptoWallet/blob/master/README.md

@molnard It turns out, right now it only works up to 1 block reorg.

The code was taken from Nicolas and it doesn't comply with the best practices.

The code: https://github.com/nopara73/MagicalCryptoWallet/tree/master/MagicalCryptoWallet.Tests/NodeBuilding

Async/Await - Best Practices in Asynchronous Programming: https://msdn.microsoft.com/en-us/magazine/jj991977.aspx

Test name: TestServicesAsync. Have to do according to the ToDos.

while (Interlocked.Read(ref _nodeCount) < 3)
{
	await Task.Delay(100); // ToDo: timeout after a few minutes
}

while (Interlocked.Read(ref _mempoolTransactionCount) < 7)
{
	await Task.Delay(100); // ToDo: timeout after a few minutes
}

Since it got merged into NBitcoin.