davidkel / fabric-chaos-testing Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
{"component":"CLIENT","timestamp":"2021-11-01T18:21:08.487Z","txnId":"6ae19d4db7e679a20dc01875eae813ee395b86158da20edc1845067bea7f7431","stage":"Failed","message":"10 ABORTED: evaluate call to endorser returned an error response, see attached details for more info"}
Current logging options are
'logOnlyOnFailure', 'AllPoints', 'Failure&Success'
which isn't very consistent. I think we should rename them to
Note that All is not equal to Failure and Success
Failure and Success outputs only all log points for a txn if the txn fails or the single success message. All will output all log points for a successful transaction as well.
The report creator must be able to recognise and ignore or report as expected problems which we know would happen under certain conditions
Actions are going to cause problems, eg
These will generate errors, but what we are looking for is client recovery without having the re-start the client or the gateway peer
An error of
{"component":"CLIENT","timestamp":"2021-10-28T14:06:24.664Z","txnId":"531544e8def41638475181253759e80748377f3d47181d12d9fa11fe869337a6","stage":"Failed","message":"10 ABORTED: failed to endorse transaction, see attached details for more info"}
says there are more details, but the details aren't captured. We should capture these
Useful to know that the client is still running
Apart from business transactions we should also be able to test the effect of config transactions happening at the same time to see the effect on the system
Unsure whether to make this part of the client or have a separately running app
Config transactions that would have an effect
Not specific to config transactions but might also want to consider
This should provide the capability to add Org4 to the existing 3 Org network
This is a manual activity to determine what is required to make use of fabric-test operator, specifically
This would add orderer6 to the current network environment
Issues raised
blocked by
github.com/hyperledger/fabric#2985
tested by
At the moment if the client exits when it just so happens that the last set of stats recorded that all failed then it will exit with 2 saying the client/network didn't recover. We mitigate this by ensuring the chaos engine has shutdown for a period of time to allow for successful transactions. Would a counter for number of sequential failures be useful to output as stats ?
We could do something similar for no transactions being submitted/evaluated but this time we could also have a threshold value as well. If at termination we have exceeded that threshold then we will exit with 3 to indicate that the gateway peer is likely to be down. This threshold value would have to match up with when the chaos engine terminated and how long the client was left running for after to ensure that if the final scenario in chaos kills the gateway peer permanently that this will be detected.
The testing here needs to be split into 2 separate tasks
Endorsement policies to test
This is going to be more tricky to test, because if the chaos engine runs in evaluate mode only and the scenarios always leave the gateway peer up, then the gateway peer is always going to be chosen for the evaluate, the only slight change is that it may try the other peer in the org first if the 2 have the same block heights, but no guarantee (Would need to confirm with Andy but I suspect that the gateway peer will always be the first peer to try if the block heights are identical to cut down on any unnecessary network traffic)
This would allow us to test the following scenarios
Test with a majority implicit EP only, don't see a need to do others although if time will do so
To note, if you kill the leader then that will cause problems to occur
For this test I turned of submit timeouts because of the potential time it takes to move to try another orderer if 1 goes down (upto 20 seconds has been seen).
We should move our test client to use the official timeouts. What would be good to test as well would be that the client doesn't remain hanging on exit (it would mean changing the client to just exit the loop when ctrl-c is received but not doing a process.exit() ) if timeouts or problems are hit
The current proposal for fabric test is 2 build runs
If the gateway peer goes down and stays down then the client just loops with the message
{"component":"CLIENT","timestamp":"2021-12-24T11:25:05.608Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:10.608Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:15.610Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:20.610Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:25.616Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:30.616Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:35.617Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:40.621Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:45.626Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:50.626Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:25:55.628Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:26:00.628Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
request to terminate received, stopping......
{"component":"CLIENT","timestamp":"2021-12-24T11:26:05.629Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:26:10.630Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:26:15.631Z","stage":"STATS","message":"WARNING: Client/Network may have stalled, no new transactions are being evaluated or endorsed"}
{"component":"CLIENT","timestamp":"2021-12-24T11:26:16.976Z","stage":"FINAL-STATS","message":"Submit: good=200, bad=5. Evals: good=310, bad=10"}
I assume this is down to the waitForReady call we have and the reason it doesn't terminate straight away is because wait for a timeout most likely the wait for ready grpc timeout.
Issue being raised to capture this behaviour so it's documented and also a consideration as to whether it could be reported as an exit code
need to test
linked to #8
I was thinking along the lines of
When the client terminates it should output final stats
if an env variable such as FailOnExit is set to true then the client should also terminate with a non-zero exit code if any failures were detected
Apparently this could be done by not setting the CORE_PEER_GOSSIP_EXTERNALENDPOINT
means they should not be used to endorse for evaluate or submit, but will just commit
In the client if the gateway peer is restarted, the chaincode event listener stops completely, we need to implement an automatic recovery of this so that the chaos engine can continue to test, this needs to be good enough to not miss events but at the same time not flood the client with constant repeats of checking
linked to #79
{"component":"CLIENT","timestamp":"2021-11-01T17:50:32.106Z","txnId":"bdb747da265d8a34bc34b5b3257458883abbf1d2679bc1499e52daa21a67d4ac","stage":"Evaluated","message":""}
Txn name, params etc similar to submit
Add Support to dynamically change the endorsement policy and test the following scenarios
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.