lassal / dbdiff Goto Github PK

Problem: the parallel serializer is commiting the repository changes to a environment branch before the serializer process has finish all tasks for that environment.

Cause: the current implementation in ParallelDBVersionCommand.takeDatabaseSchemaSnapshotVersion() is waiting for the last action instead of verify that all actions were completed.

When schemas are informed for DB Versioning they are not checked before processing.
It is best to verify if the schema exists before start all versioning tasks.

Currently Oracle is giving the name for the function return parameter of OracleRepository.RETURN_PARAMETER_NAME.
MySQL doesn't seen to use any special treatment for the name; I don't know if it is NULL or if MySQL provides a special name for it.

Create builder to instantiate ParallelDBVersionCommand.
Use fluent API style.

Main methods

• Constructor: pass VCS repository, Base branch, local repo path, tmp relative path
• Set Schemas : db schemas to be serialized
• Add DB Environment: add DB Environment to be processed : Name, jdbcUrl, credentials

Create Maven goal to take database snapshot of the DB environments configured

• Create specific object configuration for
  • VCS repository
  • Database
• Uses builder implemented in #21

Create unit tests for the children of DBModelSerializer

• TableSerializer
• ViewSerializer
• TriggerSerializer
• RoutineSerializer

The tests should cover:

• Assemble of objects, using mock repositories
• Serialization
• Deserialization, comparing with the original

Create integrated test for DBVersionCommand.

The tests should cover:

• Load metadata from Oracle
• Load metadata from MySQL
• Serialize all info and store each database in a different branch
• Commit changes
• Send changes to remote server Git

A serialization process should proceed even if some object types in some schemas fail. If at least one object type in a schema is able to be serialized it should serialize this information and register in the VCS.

The commit report should inform what object types fail for each schema, and also bring a extra session with the error generated in each serialization process per object and schema.

Requirements:

• Same behaviour for parallel and serial serialization
• Show the exception for each serialization error
• Serialize what is possible in the unit schema + object type

Expected report were indexes failed in the schema ATZ

 ENVIRONMENT NAME: DEV
  JDBC URL: jdbc:oracle:thin:@10.85.5.11:1521/BDTDPRC

  OBJECTS SERIALIZED BY SCHEMA:

Schema: [                 ATZ]  // inde
  :: Tables: 00074 | Views: 00000 | Indexes: _FAIL | Triggers: 00000 | Routines: 00000
Schema: [                 CCR] 
  :: Tables: 00144 | Views: 00001 | Indexes: 00278 | Triggers: 00000 | Routines: 00011
Schema: [                 CEM] 
  :: Tables: 00120 | Views: 00000 | Indexes: 00188 | Triggers: 00000 | Routines: 00001
-------
ERRORS
 - Schema: ATZ, serializing indexes
 <error message | limit message size >

Problem: In the Parallel serializing command in the step to copy the generated files it is generating an exception DirectoryNotEmptyException during the copy of files.

How to reproduce: : Run test br.lassal.dbvcs.tatubola.integration.DBVersionCommandTest.versionSingleOracleDBParallel() with /testresources/log4j-test.yaml logger "br.lassal.dbvcs.tatubola", level = TRACE.

Entry point: https://github.com/Lassal/dbdiff/blob/master/core-library/src/main/java/br/lassal/dbvcs/tatubola/ParallelDBVersionCommand.java

Method: copyFullFolderStructure(...)

Test the behavior of VersionCommand using mocked repositories and VCS.
This test will cover

• Object serialization
• Serial tasking
• Integration with VCS
• Main comportment of the class: versioning database

This test covers the same tests covered by #29 but using the serial implementation

This test complements the component tests of DBModelSerializer and integrated tests of GitController, and it was mapped in #14.

List all intermediate components that need to fully covered by Unit Tests.

Use Mockito to create the mocks.

Components to be covered:

• DBModelSerializer and all its children
• ParallelSerializer
• DBVersionCommand
• ParallelDBVersionCommand
• JacksonYamlSerializer : ? test all model serialization ?

Show by object type the number of objects that were loaded from the database and add this info in the commit message of that branch.

Database credentials for the database connection should not be kept plain in the POM file, it should be stored in a key store.

Possible solutions:

• use a java keystore to save connection credentials. A credential workflow will need to be configured

Create integrated tests for GitController. It should create a local repository in a path relative to the test project and cover all the methods of the interface.

Tests item defined in item #9

Implement a interface to list all the schemas in the database in the interface RelationalDBRepository and implement these methods in Oracle and MySQL implementations

The ParallelDBVersionCommand uses default exception handler and it is not raising exception found in the tasks runned by it.

Please verify : ForkJoinPool docs

How to reproduce the problem
Change InMemoryTestRepository.loadTableColumns(...)

// CORRECT
        Map<String, Table> tableColumns = new HashMap<>();

        if(this.schemas.containsKey(schemaName)){
            Schema schema = this.schemas.get(schemaName);

// THROW EXCEPTION
        Map<String, Table> tableColumns = new HashMap<>();

        if(this.schemas.containsKey(schemaName)){
            Schema schema = this.schemas.get(schemas);

Even if all serialization tasks (schema + object type) fail for an environment we should be able to know it. We should register the problem and the result of this failed serialization process in the VCS, to keep track of what happen and when happen.

The reasons could be several, the most trivial can be a DB password change that has became invalid.

It is also import to make the process fail after all processing (or create a flag to allow it) once it can be used in CI/CD pipelines that will not know that everything is wrong otherwise.

Requirements:

• Handle the error independently by environment
• Same results for serial and parallel processing
• Create feature flag to allow the user decide if the snapShot task should raise an error if an environment fail
• Register all error details by environment in the VCS; the register will not have any changed file

The current solution is trying to open all connections for all environments in a sequential way. This is taking a long time, around 30 seconds to open the connections to each environment in a sequential way.

This time is being taken by the Hikari connection pool that currently is trying to open the connections on the startup.

Possible solution: change the default initialization parameter in RelationalDBVersionFactory.createConnectionPool(..., openConnectionsOnInitialization=false)

Test the behavior of ParallelDBVersionCommand using mocked repositories and VCS.
This test will cover

• Object serialization
• Parallel tasking
• Integration with VCS
• Main comportment of the class: versioning database

This test complements the component tests of DBModelSerializer and integrated tests of GitController, and it was mapped in #14.

IMPORTANT: this test will not use the ParallelDBVersionCommandBuilder because we will not be able to mock the repository and VCS using it, but we will apply the same building logic in the test.

The VCS Controller is responsible to define the common interface to be provided by the Version Control Systems (in the default implementation Git).

Responsabilities:

• Setup of local VC repository
• Commit changes locally
• Push changes to central server

In the commit message for dbVersionSnapshot show the date and time of the message generation in UTC format.

Create integrated tests covering all methods defined at

br.lassal.dbvcs.tatubola.relationaldb.repository.RelationalDBRepository

All tests should be validated against a test database created at #issue???

Create integrated tests covering all methods defined at

br.lassal.dbvcs.tatubola.relationaldb.repository.RelationalDBRepository

All tests should be validated against a test database created at #issue???

Create plugin mechanism to allow the usage of custom repositories to load database metadata, instead of the provided in the project.

In the issue #32 we minimally exposed and handle exceptions generated during parallel serialization but the code still not very clear and basically we only abort the whole serialization process.

OBJECTIVE:
Define and create proper tasks to allow to commit the changes detected in the schemas and environments without error.

• ? how about to commit all schemas and objects that are ok and mention in the commit message what schemas + object types present any problem during the serialization
• ? Even environment that no information could be retrieved could be commited with this error information to keep a history of what happen

STARTING POINTS

ParallelDBVersionCommand.waitEndSerializationAndCheckEnvBranch(....)

//try to change any serialization process to break the process

Create tag and publish version to repository

Create builder for DBVersionCommand similar to builder implemented in #21 .

Create main factory class to enforces Dependency Injection in the solution.

• Build FS serializer
• Build Database serializer
• Buid database connection pool

A VCS controller class responsible to implement the interface defined in feature #6 for Git.

Implement the interface :

br.lassal.dbvcs.tatubola.versioncontrol.VersionControlSystem

Create a main class responsible to orchestrate all the process of load objects from the database, serialize them in text files and versioning this files in a VCS System.

This class will implement the following behaviour

• Download/setup local VCS repository from remote
  • For each [DB Environment]
    • For each [DB Schema]
      • For each object type
        • Load object definition from database
        • Serialize object definition as text file
  • Commit changes for each [DB Environment] in a different VCS branch
• Push changes to the VCS server

Test serialization process for Indexes. The following items should be tested

• Order of the properties are serialized
• Object assembly and serialization process
• Output path
• Serialization output YAML

It relates to issue #14

Test all public methods and outputs of the class

br.lassal.dbvcs.tatubola.fs.DBModelFS

The tests should cover all naming and filesystem structure used.
Serialization don't need to be tested here.

Create a test project to concentrate all integration tests.

In this project all tests with external elements should be placed

• Databases
• VCS Repositories: Git, SVN | Local and remote
• File system

Define how the maven plugin will connect to Git. We can use the current git mechanisms or include these credentials in the same database credentials defined here #2

Create parallel implementation using ForkJoinThreadPool to serialize objects in parallel of the different environments and schemas

Verify if after implementing #35 and #36 the code is easy to read and allow any error created in future development to be captured during the execution of the unit tests.

The parallel serialization should have an exception handler in the main process to cater for any possible problem that may happen.