Code Monkey home page Code Monkey logo

dbdiff's Introduction

dbdiff

Compares two databases and prints SQL commands to modify the first one in order to match the second one.

It does NOT execute the statements. It only prints the statements.

It supports PostgreSQL and MySQL.

Installing

Install globally with npm

npm install dbdiff -g

CLI Usage

dbdiff \
  -l safe
  dialect://user:pass@host[:port]/dbname1 \
  dialect://user:pass@host[:port]/dbname2

Where dialect can be either postgres or mysql. The first database url denotes the target, the second the source, the sql queries will allow target to be updated to source state.

The flag -l or --level indicates the safety of the SQL. Allowed values are safe, warn and drop

Safety level

Some statements may fail or may produce data loss depending on the data stored in the target database.

  • When the safe level is specified, only SQL statements that are guaranteed to preserve existing data will be printed. Any other command will be commented out.
  • When the warn level is specified also SQL statements that may fail because of existing data will be printed. These commands are for example: changes in data types or dropping a NOT NULL constraint.
  • When the drop level is specified all SQL statements are printed and this may contain DROP COLUMN or DROP TABLE statements.

Dropping a sequence or dropping an index is considered safe.

Changing the data type of existing columns

Sometimes Postgresql won't be able to change the existing data to the new data type. In that case you will get an error similar to this:

ERROR:  column "column_name" cannot be cast automatically to type integer
HINT:  Specify a USING expression to perform the conversion.

You can manually specify a USING expression to perform de conversion. For example to convert text to integers:

ALTER TABLE table_name
  ALTER column_name TYPE data_type USING column_name::integer

Usage as a library

You can use dbdiff as a library:

var dbdiff = require('dbdiff')

dbdiff.describeDatabase(connString)
  .then((schema) => {
    // schema is a JSON-serializable object representing the database structure
  })

var diff = new dbdiff.DbDiff()
// Compare two databases passing the connection strings
diff.compare(conn1, conn2)
  .then(() => {
    console.log(diff.commands('drop'))
  })

// Compare two schemas
diff.compareSchemas(schema1, schema2)
console.log(diff.commands('drop'))

You can pass connection strings such as postgres://user:pass@host:5432/dbname1 or objects to these methods. For example:

dbdiff.describeDatabase({
  dialect: 'postgres', // use `mysql` for mysql
  username: 'user',
  password: 'pass',
  database: 'dbname1',
  host: 'localhost',
  dialectOptions: {
    ssl: false
  }
})
.then((schema) => {
  // ...
})

Example of .describeDatabase() output

{
  "tables": [
    {
      "name": "users",
      "schema": "public",
      "indexes": [],
      "constraints": [
        {
          "name": "email_unique",
          "schema": "public",
          "type": "unique",
          "columns": [
            "email"
          ]
        },
        {
          "name": "users_pk",
          "schema": "public",
          "type": "primary",
          "columns": [
            "id"
          ]
        }
      ],
      "columns": [
        {
          "name": "id",
          "nullable": false,
          "default_value": "nextval('users_id_seq'::regclass)",
          "type": "integer"
        },
        {
          "name": "email",
          "nullable": true,
          "default_value": null,
          "type": "character varying(255)"
        }
      ]
    },
    {
      "name": "items",
      "schema": "public",
      "indexes": [],
      "constraints": [
        {
          "name": "items_fk",
          "schema": "public",
          "type": "foreign",
          "columns": [
            "user_id"
          ],
          "referenced_table": "users",
          "referenced_columns": [
            "id"
          ]
        }
      ],
      "columns": [
        {
          "name": "id",
          "nullable": false,
          "default_value": "nextval('items_id_seq'::regclass)",
          "type": "integer"
        },
        {
          "name": "name",
          "nullable": true,
          "default_value": null,
          "type": "character varying(255)"
        },
        {
          "name": "user_id",
          "nullable": true,
          "default_value": null,
          "type": "bigint"
        }
      ]
    }
  ],
  "sequences": [
    {
      "data_type": "bigint",
      "numeric_precision": 64,
      "numeric_precision_radix": 2,
      "numeric_scale": 0,
      "start_value": "1",
      "minimum_value": "1",
      "maximum_value": "9223372036854775807",
      "increment": "1",
      "schema": "public",
      "name": "users_id_seq",
      "cycle": false
    },
    {
      "data_type": "bigint",
      "numeric_precision": 64,
      "numeric_precision_radix": 2,
      "numeric_scale": 0,
      "start_value": "1",
      "minimum_value": "1",
      "maximum_value": "9223372036854775807",
      "increment": "1",
      "schema": "public",
      "name": "items_id_seq",
      "cycle": false
    }
  ]
}

dbdiff's People

Contributors

gimenete avatar spencerdcarlson avatar tjunnone avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dbdiff's Issues

making dbdiff -l safe user:password@localhost/db1 user:password@localhost/db2 present error /usr/local/lib/node_modules/dbdiff/dbdiff.js:6 class DbDiff { ^^^^^

/usr/local/lib/node_modules/dbdiff/dbdiff.js:6
class DbDiff {
^^^^^

SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/usr/local/lib/node_modules/dbdiff/index.js:3:31)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)

Anable to launch !

dbdiff --help
/usr/local/lib/node_modules/dbdiff/dbdiff.js:6
class DbDiff {
^^^^^

SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:374:25)
at Object.Module._extensions..js (module.js:417:10)
at Module.load (module.js:344:32)
at Function.Module._load (module.js:301:12)
at Module.require (module.js:354:17)
at require (internal/module.js:12:17)
at Object. (/usr/local/lib/node_modules/dbdiff/index.js:3:31)
at Module._compile (module.js:410:26)
at Object.Module._extensions..js (module.js:417:10)

Does not seem to handle FK ON DELETE CASCADE change

Source:
TABLE "systemrole" CONSTRAINT "fk474jpkeb2oe23px6stixe16wh" FOREIGN KEY (id) REFERENCES role(id) ON DELETE CASCADE

Target:
TABLE "systemrole" CONSTRAINT "fk474jpkeb2oe23px6stixe16wh" FOREIGN KEY (id) REFERENCES role(id)

Command:

dbdiff -l drop $TARGET $SOURCE

Result:

Alter table statements containing not mention of ON DELETE CASCADE

Expected Result:

Alter table statements containing ON DELETE CASCADE changes (drop / recreate for instance)

PG.connect deprecation

When running dbdiff on a postgres database I get the following message and nothing else happens:

(node:9660) DeprecationWarning: PG.connect is deprecated - please see the upgrade guide at https://node-postgres.com/guides/upgrading

I found a fork of this repo that seems to have updated to PG7 (https://github.com/contra/dbdiff) and installed that one, but that gives me an authentication error. Strange thing is though that the username in the authentication error is not the username that I entered in any of the connection info for dbdiff.

update
I applied the pg7 changes to the code of this repo as well, and get the same authentication error. This is the command I issue:

dbdiff postgres://testuser:[email protected]:5432/test postgres://testuser:[email protected]:5432/test2

and this is the error:

error: password authentication failed for user "Ron" at Connection.parseE (C:\Users\Ron\AppData\Roaming\npm\node_modules\dbdiff\node_modules\pg\lib\connection.js:567:11) at Connection.parseMessage (C:\Users\Ron\AppData\Roaming\npm\node_modules\dbdiff\node_modules\pg\lib\connection.js:391:17) at Socket.<anonymous> (C:\Users\Ron\AppData\Roaming\npm\node_modules\dbdiff\node_modules\pg\lib\connection.js:129:22) at emitOne (events.js:116:13) at Socket.emit (events.js:211:7) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Socket.Readable.push (_stream_readable.js:208:10) at TCP.onread (net.js:607:20)

describeDatabase hangs

dbdiff.describeDatabase(conn1) .then((schema) => { console.log(schema) })

prints the schema, but hangs in the terminal

No single quote around default value.

When a table has attribute default value the final create table statement does not include a single quote around the default value, as a result when the query is run in the database it throws a parsing error.

I created a PR for this #24

Test fails for constraint

Hi, i'm looking into contributing to and using this library as a way of not writing migrations. So far, i've started by running the test suite and ran into this issue:

  32 passing (5s)
  1 failing

  1) Postgresql should support all constraint types:

      AssertionError [ERR_ASSERTION]: 'CREATE SEQUENCE "public"."items_id_seq" INCREMENT 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 NO CYCLE;\n\nCREATE SEQUENC == 'CREATE SEQUENCE "public"."users_id_seq" INCREMENT 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 NO CYCLE;\n\nCREATE SEQUENC
      + expected - actual

      +CREATE SEQUENCE "public"."users_id_seq" INCREMENT 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 NO CYCLE;
      +
       CREATE SEQUENCE "public"."items_id_seq" INCREMENT 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 NO CYCLE;

      -CREATE SEQUENCE "public"."users_id_seq" INCREMENT 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 NO CYCLE;
      -
       CREATE TABLE "public"."users" (
         "id" integer DEFAULT nextval('users_id_seq'::regclass) NOT NULL,
         "email" character varying(255) NULL
       );

      at runCommands.then.then (test/utils.js:35:28)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)



npm ERR! Test failed.  See above for more details.

Could you tell me which version of PG you're testing against? I'm on 9.6. Thanks!

Index statements not printed

Nice work! From the code, it looks like there's support for CREATE INDEX and DROP INDEX statements, but it doesn't work for me:

DB 1, table A:
Indexes:
"Blueprints_pkey" PRIMARY KEY, btree (id)
"Blueprints_data" gin (data jsonb_path_ops)

DB2, table A:
Indexes:
"Blueprints_pkey" PRIMARY KEY, btree (id)
"Blueprints_data" gin (data jsonb_path_ops)
"CreatorId_index" btree ("CreatorId")

I expected an output of CREATE INDEX for CreatorId_index, but got nothing. Column diffs are however correctly printed out. The type of CreatorId column is uuid.

diff two schema files

Is there an easy way to modify this script to diff two sql files containing a schema definition?

I'm not too much of a database guy generally, but if you could point me in the right direction I bet I could figure it out from there.

Thanks!

diff installed extensions

  • Installs if needed via CREATE EXTENSION
  • Updates if needed via UPDATE EXTENSION
  • Removes if needed via DROP EXTENSION

Deprecate global install

Instead of install dbdiff with -g (which requires sudo privileges for many of us), how about this?

  • mkdir somedir
  • cd somedir
  • npm init -y
  • npm install dbdiff
  • npx dbdiff -l safe ...

Note that is npx, not npm, on the last line.

Inform: not handle case Error : BLOB/TEXT column {{columnname}} used in key specification without a key length

for ex:
this command below is solution after diff.
ALTER TABLE table_name ADD INDEX matchID USING BTREE (column_text);
but i get error "BLOB/TEXT column {{columnname}} used in key specification without a key length" because that column defined "TEXT" type.

must be like command below
ALTER TABLE table_name ADD INDEX matchID USING BTREE (column_text(20));

How should I do?, please suggest.

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.