Resilient Database: Scalability, High Availability and Disaster Recovery about nz-orcid-hub HOT 6 CLOSED

the configuration for the backend store comprises a master DB deployed on the same EC2 node the application container is deployed, and a standby "slave" DB deployed on separate EC2 node (backup.orcidhub.org.nz).

Standby DB is set up as a read-only DB that keeps continuously "recovering" using:

• compressed and archived WAL logs shipped from the master DB using rsync;
• streaming replication providing near instantaneous data change replication from the master to the stand-by "slave" DB;

The backup server (node) is used also for storing DB backup file archives. The DB can be brought back from the backup file using a full backup file and a set of WAL log archive files created after the full backup.

In case if the master DB becomes unresponsive, the application would initiate fail-over procedure switching to the standby DB and promoting it to the master DB.

from nz-orcid-hub.

^^In terms of the metrics, in a passing discussion today we considered for the New Zealand research, science, and innovation domain that we're looking at something like, in very rough and assumptive terms:

== research institutiones ==

• maximum of ninety-nine
• half are onboarded to the hub in the first month
• the remainder are onboarded to the hub in equal instalments over the following year

== researchers ==

• 10% of research institutions have thousands of researchers (i.e., more than 1,000)
• 30% of research institutions have hundreds of researchers (i.e., between 100 and 999)
• 60% of research institutions have tens of researchers (i.e., between 1 and 99)

...and that the peak load for the Hub, which is a transactional engine expecting to see each researcher only once every few years, will be composed of a combination of the research institution onboarding and the researcher enrolment — and that the peak load will therefore be modest at best.

from nz-orcid-hub.

^^In terms of the metrics, in a passing discussion today we considered for the New Zealand research, science, and innovation domain that we're looking at something like, in very rough and assumptive terms:

== research institutiones ==

• maximum of ninety-nine
• half are onboarded to the hub in the first month
• the remainder are onboarded to the hub in equal instalments over the following year

== researchers ==

• 10% of research institutions have thousands of researchers (i.e., more than 1,000)
• 30% of research institutions have hundreds of researchers (i.e., between 100 and 999)
• 60% of research institutions have tens of researchers (i.e., between 1 and 99)

...and that the peak load for the hub, which

from nz-orcid-hub.

^^Discussions around the reslience and recovery characteristics of the ORCID Hub have generally been of a view that high-availability (e.g., five 9s) is not the priority here, but that losing any data would be a bad thing. As a working (and to be validated with Jason Gush and with the IT Advisory Group, the current thinking is that the ORCID Hub must be designed with an architecture that supports:

• RTO = Recovery Time Objective = of not more than twenty-four hours.
• RPO = Recovery Point Objective = of less than one minute.

by jken016

from nz-orcid-hub.

Jason, I have updated the user story. Technically it's not a story, but rather a non-functional requirement, I reckon , we can use "Data Integrity" issue type for that.

I will go the metrics to evaluate the workload in terms of kinds, frequencies, volumes and complexity of the data exchange. It would be also very helpful to define and quantify overall orcidhub scalability requirements for performance testing.

from nz-orcid-hub.

I love how this project continues to introduce me to new terminology.

What kinds of exchanges are we expecting to see, and at what expected frequencies? Is the size and complexity of the hubs data going to need this kind of management approach?

from nz-orcid-hub.