Actual:
This error prints repeatedly on startup:

mirror-node-record-download-parse_1  | WARNING: An illegal reflective access operation has occurred
mirror-node-record-download-parse_1  | WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/MirrorNodeCode/lib/protobuf-java-3.5.1.jar) to field java.nio.Buffer.address
mirror-node-record-download-parse_1  | WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
mirror-node-record-download-parse_1  | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
mirror-node-record-download-parse_1  | WARNING: All illegal access operations will be denied in a future release
mirror-node-record-download-parse_1  | Downloading Done
mirror-node-record-download-parse_1  | 2019-08-09 20:31:07.326 ERROR  241  recordfileparser - Exception file /MirrorNodeData/recordStreams/valid does not exist
mirror-node-record-download-parse_1  | Parsing Done
mirror-node-record-download-parse_1  | Downloading
mirror-node-record-download-parse_1  | Downloading Done

Expected:
All necessary directories are created by the app on startup.

In EventStreamParser, we should create the eventStreams/valid/ directory if it doesn't exist

mirror-node-balance-parser_1         | 2019-08-09 18:41:25.749 ERROR  139  balancelogger - /MirrorNodeData/accountBalances/valid does not exist.
mirror-node-balance-parser_1         | Exception in thread "main" java.lang.NullPointerException
mirror-node-balance-parser_1         | 	at com.hedera.balanceFileLogger.BalanceFileLogger.processAllFilesForHistory(BalanceFileLogger.java:162)
mirror-node-balance-parser_1         | 	at com.hedera.balanceFileLogger.BalanceFileLogger.main(BalanceFileLogger.java:156)

Expected:
It should create the directory on startup if it doesn't exist.

Add an EventStream Parser to parse new version EventStream files.
format:

Actual:
Occurs on startup with a clean db and filesystem:

mirror-node-balance-downloader_1     | WARNING: An illegal reflective access operation has occurred
mirror-node-balance-downloader_1     | WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/MirrorNodeCode/lib/protobuf-java-3.5.1.jar) to field java.nio.Buffer.address
mirror-node-balance-downloader_1     | WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
mirror-node-balance-downloader_1     | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
mirror-node-balance-downloader_1     | WARNING: All illegal access operations will be denied in a future release
mirror-node-balance-downloader_1     | Exception in thread "main" java.lang.NullPointerException
mirror-node-balance-downloader_1     | 	at com.hedera.downloader.AccountBalancesDownloader.verifySigsAndDownloadBalanceFiles(AccountBalancesDownloader.java:88)
mirror-node-balance-downloader_1     | 	at com.hedera.downloader.AccountBalancesDownloader.main(AccountBalancesDownloader.java:47)

Expected:
No NPE

Currently the architecture is complex with 5 java processes starting up each with only a single thread. We should also support starting up in a single process with each of the standalone processes now running in a separate thread. Wouldn't take much work, just need to consolidate to a single main() and have each of the previous mains implement a common interface. A flag would allow you to pick and choose which module you wanted to run on startup.

Related to issue: swirlds/services-hedera/issues/1514

Right now In version1, hashing is calculated as follows:
h[i] = hash(h[i-1] || c[i])

In version2 , Incorporate the change of hashing formula to:
h[i] = hash(h[i-1] || hash(c[i]))

The state of the last balance file received with the actual data downloaded can get out of sync. When this occurs the balance downloader will never download any new files indefinitely. In practice, I'm not sure how the balances.json file got behind the data. This happened when I was playing around with adding sleep. But it could potentially occur if the balances.json fails to write or gets accidentally wiped out.

To reproduce, you can set your balances.json to a file in the past (pick something past maxDownloadItems in the past). For example, download up to date then update balances.json:

{
  "lastValidBalanceFileName": "2019-07-11-10-05.csv"
}

You'll just see it trying the same files over and over in the logs and saying they exist:

2019-08-22 11:28:31,818 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-10.csv
2019-08-22 11:28:31,818 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-15.csv
2019-08-22 11:28:31,818 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-20.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-25.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-30.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-35.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-40.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-45.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-50.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-10-55.csv
2019-08-22 11:28:31,819 TRACE [main  ] c.h.d.AccountBalancesDownloader File exists: ./MirrorNodeData/accountBalances/valid//2019-07-11-11-00.csv

Extract the keys, keylist and threshold keys from keys and adminKeys in t_entities.

Option 1: Create a database structure to mirror keys, lists and threshold lists. Consider searches must be able to find an entity for a given key regardless of depth.

Option 2: Simply extract a distinct list of keys to attach to an entity such that an entity can be found by searching for the key

Use address book for list of nodes rather than nodes info, this keeps the list of nodes up to date and removes a configuration step.

• Proxy
  Look at list of nodes from address book.

• Mirror node downloads
  -Not automating this does allow for some freedom in where to download from, at the expense of trust and reliability
  -Automating forces mirror node operators to download everything

There are several .env files, confusing environment variable names, etc...
We should consolidate as much as possible to reduce the likelihood of errors.

When any of the download or mirror processes start, check the address book file is present in ./config.
If not, attempt to download using parameters from config.json or .env (NODE_ID, NODE_ADDRESS, OPERATOR_ID and OPERATOR_KEY).
If the address book fails to download of those parameters are missing, exit with error message.

Add an EventStream Downloader, and verify the signature files.

Modify MirrorNode to be able to parse EventStream files with new version number 3.
Related issue: https://github.com/swirlds/platform-swirlds/issues/1369

Difference between version3 and version2:

1. the way of generating file Hash

file[i] = p[i] || h[i] || c[i]

version2:
h[i] = hash(p[i-1] || h[i-1] || c[i-1])

version3:
h[i] = hash(p[i-1] || h[i-1] || hash(c[i-1]))

2. in version2, each time the nodes restart, the prevFileHash contained in the first EventStream file which is generated after restarting would be new byte[48];
   in version3, while starting the platform, we read the FileHash contained in the last .evts_sig file in the directory, and use it as prevFileHash when we start to write the first EventStream file; if such file doesn't exist, we use new byte[48] as prevFileHash.

add a file status table in database;
add a column fileId in t_events table;
if a file was partially parsed and the parsing stoped, we should first delete the events in t_events which are contained in this file, then reimport this eventstream file without producing duplicate rows.

Account balances only occur every 15 minutes, but the current downloader will repeatedly poll S3 for new files without any sleep in between. We should institute some amount of sleeping to reduce CPU usage and S3 network costs. There are two scenarios to consider:

1. Initial startup and catching up: There should be no sleep in between queries when there were any downloads last round.
2. Up to date: There should be a sleep. It could be anywhere from a dumb 1s sleep to a smart 15min from last download sleep.

Just examining the code, the java.sql.Connection is opened once and reused throughout the lifecycle of the app. However, it's possible that the database will close the connection during some fatal errors. The app should be resilient to such closures.

Another issue is that the Connection object is not thread safe. A thread pool like HikariCP should be used to provide separate connections per thread and handle automatically reopening closed connections.

Actual:
Account balancer parser uses too much CPU even with no files to process. I assume it will also use a lot of CPU even with files.

$ docker stats
CONTAINER ID        NAME                                         CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
7d58db1a8885        docker_mirror-node-balance-parser_1          20.48%              98.04MiB / 3.855GiB   2.48%               2.95kB / 1.28kB     0B / 0B             19
2ea1372c35b4        docker_mirror-node-record-download-parse_1   2.15%               202.5MiB / 3.855GiB   5.13%               12.6MB / 8.35MB     0B / 0B             21
2270524615a9        docker_mirror-node-postgres_1                0.49%               4.312MiB / 3.855GiB   0.11%               842kB / 779kB       0B / 4.1kB          7
0141de8d8535        docker_mirror-node-rest-api_1                0.00%               52.38MiB / 3.855GiB   1.33%               600kB / 416kB       0B / 16.4kB         24

$ docker logs -f 7d58db1a8885
2019-08-09 21:25:10,364 INFO  [main  ] balancelogger: Balance History processing done
2019-08-09 21:25:10,365 INFO  [main  ] balancelogger: No balance file to parse found
2019-08-09 21:25:10,366 INFO  [main  ] balancelogger: Last Balance processing done
2019-08-09 21:25:10,367 INFO  [main  ] balancelogger: Balance History processing done
2019-08-09 21:25:10,369 INFO  [main  ] balancelogger: No balance file to parse found
2019-08-09 21:25:10,369 INFO  [main  ] balancelogger: Last Balance processing done
2019-08-09 21:25:10,370 INFO  [main  ] balancelogger: Balance History processing done
2019-08-09 21:25:10,372 INFO  [main  ] balancelogger: No balance file to parse found
2019-08-09 21:25:10,372 INFO  [main  ] balancelogger: Last Balance processing done
2019-08-09 21:25:10,374 INFO  [main  ] balancelogger: Balance History processing done
2019-08-09 21:25:10,376 INFO  [main  ] balancelogger: No balance file to parse found
2019-08-09 21:25:10,376 INFO  [main  ] balancelogger: Last Balance processing done
2019-08-09 21:25:10,377 INFO  [main  ] balancelogger: Balance History processing done
2019-08-09 21:25:10,379 INFO  [main  ] balancelogger: No balance file to parse found

Expected:
The process uses less CPU.

The problem is there's a while (true) {someWork();} without there being any sleep after processing. Alternatively, it would be better to use Java's FileWatcher API to be asynchronously notified of file changes.

Actual:

2019-08-15 15:10:37,094 INFO  [main  ] configloader Loading configuration from ./config/config.json
2019-08-15 15:11:17,133 INFO  [main  ] balancelogger No balance file to parse found
2019-08-15 15:11:17,133 INFO  [main  ] balancelogger Last Balance processing done
2019-08-15 15:11:17,133 INFO  [main  ] balancelogger Balance History processing done
2019-08-15 15:12:17,209 ERROR [main  ] balancelogger File ./MirrorNodeData/accountBalances/valid/foo.csv is not named as expected, should be like 2019-06-28-22-05.csv
2019-08-15 15:12:17,221 INFO  [main  ] balancelogger Last Balance processing done
2019-08-15 15:12:17,222 ERROR [main  ] balancelogger File ./MirrorNodeData/accountBalances/valid/foo.csv is not named as expected, should be like 2019-06-28-22-05.csv
2019-08-15 15:12:17,223 ERROR [main  ] filewatcher Exception : {}
java.lang.StringIndexOutOfBoundsException: begin 0, end 10, length 7
	at java.lang.String.checkBoundsBeginEnd(String.java:3319) ~[?:?]
	at java.lang.String.substring(String.java:1874) ~[?:?]
	at com.hedera.utilities.Utility.moveFileToParsedDir(Utility.java:499) ~[classes/:?]
	at com.hedera.balanceFileLogger.BalanceFileLogger.processAllFilesForHistory(BalanceFileLogger.java:162) ~[classes/:?]
	at com.hedera.balanceFileLogger.BalanceFileLogger.onCreate(BalanceFileLogger.java:149) ~[classes/:?]
	at com.hedera.fileWatcher.FileWatcher.watch(FileWatcher.java:61) [classes/:?]
	at com.hedera.balanceFileLogger.BalanceFileLogger.main(BalanceFileLogger.java:143) [classes/:?]

Expected:
Reject invalid file and move on

Actual:
On startup with a clean filesystem and db:

mirror-node-event-download-parse_1   | Exception in thread "main" java.lang.NullPointerException
mirror-node-event-download-parse_1   | 	at com.hedera.parser.EventStreamFileParser.loadEventStreamFiles(EventStreamFileParser.java:363)
mirror-node-event-download-parse_1   | 	at com.hedera.parser.EventStreamFileParser.parseNewFiles(EventStreamFileParser.java:459)
mirror-node-event-download-parse_1   | 	at com.hedera.downloader.DownloadAndParseEventFiles.main(DownloadAndParseEventFiles.java:48)

Expected:
No crash

Begin evaluating tooling and implementing performance tests, to be run from CI (nightly test?).

• spin up environment to test the performance of the mirror-node (parser and rest-api), likely nightly build.
• get results for:
  • rest-api performance numbers per transaction
  • insert/processing performance numbers from the parser
    ...

cc @atul-hedera

BalanceFileLogger uses the Java file watching API to be notified of creates and updates of files that the downloader writes. However, if BalanceFileLogger is down while downloader writes them then there will be no notification. We need to look for new files on startup.

Generate a CSV file with 5M records (starting at 1000 for the account number) and load into the database (place the file in the valid folder and run balanceParse).

Generate a second CSV file with different balances and load into the database.

Note: Set the config.json parameter "validatebalancesignatures" to false before running the test, else signatures will be required for all nodes on this file.

Creating network "docker_default" with the default driver
Creating docker_mirror-node-rest-api_1              ... done
Creating docker_mirror-node-balance-downloader_1    ... error
Creating docker_mirror-node-postgres_1           ... done
Creating docker_mirror-node-102-file-update_1    ... done
Creating docker_mirror-node-balance-parser_1     ... 
Creating docker_mirror-node-record-download-parse_1 ... 

Creating docker_mirror-node-balance-parser_1        ... done
Creating docker_mirror-node-record-download-parse_1 ... done

ERROR: for mirror-node-balance-downloader  Cannot start service mirror-node-balance-downloader: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/usr/bin/java\": stat /usr/bin/java: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project.

Manually running container shows java is not there in path:

docker run -it --rm docker_mirror-node-balance-downloader /bin/sh
# java
/bin/sh: 1: java: not found
# /bin/java 
/bin/sh: 2: /bin/java: not found

Currently there are some json files that contain the state of the system like the below:

• balance.json
• records.json
• events.json
• loggerStatus.json

All system state should be consolidated to the database or constructed dynamically from existing tables. For example, the lastValidRcd* state could be reconstructed by getting the max timestamp of records processed on startup or by manually storing the last valid processed in a separate table.

Reasons for using the database:

• State is stored in one spot instead of spread around
• Can be migrated between versions when new fields added/removed
• Transactions provide atomic updates, ensuring no race conditions between multiple threads/processes reading/writing to same file
• Current files hardcode paths with / in config and thus won't work on Windows
• Better data type validation, foreign keys and cascading delete

Current resolution is seconds in the file name. We should include nanoseconds in the database once the testnet is upgraded.

• Transactions - use consensus_ns as primary key
• Balances - use entities instead of normalising data into t_account_balances see #86
• Remove unnecessary columns and indices

There are several fields returned by the REST API that are returned as JSON strings (quoted) instead of JSON numeric values.

Simple cleanup of this (to return numerics) before the API is released would be helpful.

transaction response fields:

charged_tx_fee
transfers.amount
balances - I'm not sure (still trying to get this working locally)

Fields that are numeric (decimal) where using a string is an ok tradeoff to avoid loss of precision (we don't need to change these, necessarily):

valid_start_timestamp
consensus_timestamp

Entity keys (key for accounts and files and admin_key for contracts) are currently stored in protobuf format.

1. Add column to store key in plain text
   1.1 Consolidate key and admin_key into a single column
2. Modify java code to extract the ED25519 key from protobuf at the time of parsing and store in that column
3. Ignore keys that contain keylist or threshold keys
4. Optionally implement a migration for the entities that already exist in the database.

Purpose is to reduce the amount of updates to existing rows.

-Insert balance data into new table
-Drop t_account_balances table
-Rename new table to t_account_balances
-Rebuild indices / foreign keys

mirror-node-record-download-parse_1  | 2019-08-09 18:41:27.870 ERROR  286  utility - getBytes() failed, Exception: {}
mirror-node-record-download-parse_1  | java.io.FileNotFoundException: ./config/0.0.102 (No such file or directory)
mirror-node-record-download-parse_1  | 	at java.io.FileInputStream.open0(Native Method) ~[?:?]
mirror-node-record-download-parse_1  | 	at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
mirror-node-record-download-parse_1  | 	at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
mirror-node-record-download-parse_1  | 	at com.hedera.utilities.Utility.getBytes(Utility.java:283) [mirrorNode.jar:?]
mirror-node-record-download-parse_1  | 	at com.hedera.signatureVerifier.NodeSignatureVerifier.<init>(NodeSignatureVerifier.java:49) [mirrorNode.jar:?]
mirror-node-record-download-parse_1  | 	at com.hedera.downloader.RecordFileDownloader.verifySigsAndDownloadRecordFiles(RecordFileDownloader.java:137) [mirrorNode.jar:?]
mirror-node-record-download-parse_1  | 	at com.hedera.downloader.RecordFileDownloader.downloadNewRecordfiles(RecordFileDownloader.java:37) [mirrorNode.jar:?]
mirror-node-record-download-parse_1  | 	at com.hedera.downloader.DownloadAndParseRecordFiles.main(DownloadAndParseRecordFiles.java:35) [mirrorNode.jar:?]
mirror-node-record-download-parse_1  | Exception in thread "main" java.lang.NullPointerException
mirror-node-record-download-parse_1  | 	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:209)
mirror-node-record-download-parse_1  | 	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:214)
mirror-node-record-download-parse_1  | 	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
mirror-node-record-download-parse_1  | 	at com.hederahashgraph.api.proto.java.NodeAddressBook.parseFrom(NodeAddressBook.java:237)
mirror-node-record-download-parse_1  | 	at com.hedera.signatureVerifier.NodeSignatureVerifier.<init>(NodeSignatureVerifier.java:49)
mirror-node-record-download-parse_1  | 	at com.hedera.downloader.RecordFileDownloader.verifySigsAndDownloadRecordFiles(RecordFileDownloader.java:137)
mirror-node-record-download-parse_1  | 	at com.hedera.downloader.RecordFileDownloader.downloadNewRecordfiles(RecordFileDownloader.java:37)
mirror-node-record-download-parse_1  | 	at com.hedera.downloader.DownloadAndParseRecordFiles.main(DownloadAndParseRecordFiles.java:35)

mirror-node-balance-downloader_1     | 2019-08-09 18:41:23.922 ERROR  286  utility - getBytes() failed, Exception: {}
mirror-node-balance-downloader_1     | java.io.FileNotFoundException: ./config/0.0.102 (No such file or directory)
mirror-node-balance-downloader_1     | 	at java.io.FileInputStream.open0(Native Method) ~[?:?]
mirror-node-balance-downloader_1     | 	at java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
mirror-node-balance-downloader_1     | 	at java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
mirror-node-balance-downloader_1     | 	at com.hedera.utilities.Utility.getBytes(Utility.java:283) [mirrorNode.jar:?]
mirror-node-balance-downloader_1     | 	at com.hedera.signatureVerifier.NodeSignatureVerifier.<init>(NodeSignatureVerifier.java:49) [mirrorNode.jar:?]
mirror-node-balance-downloader_1     | 	at com.hedera.downloader.AccountBalancesDownloader.verifySigsAndDownloadBalanceFiles(AccountBalancesDownloader.java:74) [mirrorNode.jar:?]
mirror-node-balance-downloader_1     | 	at com.hedera.downloader.AccountBalancesDownloader.main(AccountBalancesDownloader.java:47) [mirrorNode.jar:?]
mirror-node-balance-downloader_1     | Exception in thread "main" java.lang.NullPointerException
mirror-node-balance-downloader_1     | 	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:209)
mirror-node-balance-downloader_1     | 	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:214)
mirror-node-balance-downloader_1     | 	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
mirror-node-balance-downloader_1     | 	at com.hederahashgraph.api.proto.java.NodeAddressBook.parseFrom(NodeAddressBook.java:237)
mirror-node-balance-downloader_1     | 	at com.hedera.signatureVerifier.NodeSignatureVerifier.<init>(NodeSignatureVerifier.java:49)
mirror-node-balance-downloader_1     | 	at com.hedera.downloader.AccountBalancesDownloader.verifySigsAndDownloadBalanceFiles(AccountBalancesDownloader.java:74)
mirror-node-balance-downloader_1     | 	at com.hedera.downloader.AccountBalancesDownloader.main(AccountBalancesDownloader.java:47)

It would be better to fallback to nodesConfig.json if not available, I would think. Or possibly constructing it from the files in S3.

Also, there is a race condition if the 102 updater doesn't run before the other processes that require the file run they will still fail to start.

Enable CI on the repository for build.

Sharing config.json across multiple processes results in changes made by one process to the config.json file to be overwritten by another.

Basic Network Health Analytics Based on Events

1. Count Created Events Per time window
2. Count Reached Consensus Events Per time window
3. Count Created Events Per time window Per Node
4. Count Reached Consensus Events Per time window Per Node
5. Count of System & User Transactions in Reached Consensus Events Per time window Per Node
6. query for the min, median, and max latency (consensusTimestamp - timeCreated)for all events that reached consensus in a given time window, per node.
7. get the latency of events that reached consensus in a given time window

t_events Table

In order to reduce data storage requirements, only store the balance of an account in history if it has changed since the last time it was recorded.

Due to the nature of downloading to files and processing the files in separate processes, it's possible that processing could occur on a partially downloaded file. Whether it's polling for new files manually or using java's WatcherService, it's a possibility that the S3 downloader created the file, wrote some bytes but has not finished writing all bytes by the time the file is picked up by the parsers (especially for larger files).

Possible solutions:

1. Download files to a different directory and move them once complete. Move is an atomic operation.
2. Combine downloader and parser into a single process and use S3's GetObject to download to a ByteArrayOutputStream instead of a file.

Pros to 1 is its quickest.

Pros to 2 is it's a simpler architecture and avoids the penalty of writing files and coordinating processes. Due to the serial nature of processing these file streams, the only benefit of having separate processes is for separation of failures. But parser would be effectively down anyway if there's no new files being downloaded. Con is it's more work and not sure how large files get to be loaded into memory.

When the testnet is updated, re-enable balance file signature verification.
Also store the last valid balance file in a file other than config.json.

Problem

Currently, the importer is configured to either download from AWS or GCS. In the event of a connection failure, the mirror node should support failing over to one or more other cloud providers.

Solution

• Add a hedera.mirror.importer.downloader.sources to CommonDownloaderProperties:

    private List<StreamSourceProperties> sources = new ArrayList<>();

    public static class StreamSourceProperties {
        private StreamCredentials credentials;
        private String projectId; // maps to gcpProjectId
        private StreamSourceType type; // Renamed from CloudProvider
        private URI uri; // maps to endpointOverride
    }

    public static class StreamCredentials {
        private String accessKey;
        private String secretKey;
    }

• Keep existing bucket properties as is. On startup, create a StreamSourceProperties from the existing properties and push to the front of the sources list.
• Add a class to abstract away the source of stream files

public interface StreamFileProvider {

    Mono<StreamFileData> get(ConsensusNode node, StreamFilename streamFilename);

    Flux<StreamFileData> list(ConsensusNode node, StreamFilename lastFilename);
}

• Add a S3StreamFileProvider that uses S3AsyncClient
  • A GcsStreamFileProvider is not necessary since we can reuse S3StreamFileProvider for both since they implement the same API.
• Add a CompositeStreamFileProvider to handle the failover functionality
  • Iterate over the sources always starting with the first
  • If the selected source throws an exception, and it's not the last functioning source, remove it from the list of valid sources
  • Don't remove if it's a common, transient exception like NoSuchKeyException, etc. Check historical logs to verify which ones.
  • Periodically retry bad sources to see if they've become healthy again and re-add to valid sources

Currently, for the Queries which don't contain a payment, we send them to a default node.

For load balancing and getting rid of single point failure, each time we should pick a random node that have worked recently and send a Query to it.

Reference: wallets and micropayment server

As Leemon said: In our wallets and micropayment server etc, we have code to remember when a node fails to respond, and we then mark it to not be randomly called in the near future. So mostly we choose randomly from the nodes that have worked recently, and only occasionally call a node that used to be down. That kind of behavior should eventually be implemented here, too.

Currently, database queries are performed directly in the business logic layers. Most enterprise applications separate these two so the database operations can be tested in isolation and reused among multiple components. This pattern is call the DAO or Repository pattern. Libraries like Hibernate and Spring Data help automate most of this so you get the basic operations for free.

We should consider at minimum separating into separate classes and potentially using Spring Data in the future. Should also create domain objects populated by repository to pass between business layer and repo layer.

Currently we loop through all node IDs and download any new files serially. To optimize this, we should start a new thread per node ID (maybe up to some max threads) and download them in parallel. This way we can utilize network and CPU resources more efficiently and enable parser to get the 2/3 of nodes signatures quicker.

When a transaction to update the address book is detected by mirror node, the address book file is updated.
Need to test this scenario with an actual address book update transaction.

In the event that an address book update transaction itself cannot be tested, a test with another file to verify contents are updated would be a good first confidence test.

1. Create a file on the file system containing some data
2. Create the same file on the network via a transaction
3. Issue an update transaction for the same file with new contents
4. Check that the 0.0.102 file contains the new contents.