The following text specifies the format and the structure of the data that is to be deliver by the integrating party. The party can use the provided validation script to ensure that the data is valid before submitting it.

The data is delivered as a single UTF-8 encoded jsonl file. Each line contains a single JSON representation of an account. The line is ended with UNIX style line separator (\n).

{"original_id": "12312312", "email": "[email protected]"}\n
{"original_id": "12312313", "email": "[email protected]"}\n

Each line contains a JSON document of an account. The fields described below are required and will cause an error if missing. Additionally the validation script will produce an error for unknown fields.

{
  "original_id": "12312312",
  "email": "[email protected]",
  "email_verified_at": "2017-06-21T12:11:54.463Z",
  "nickname": "Morty",
  "first_name": "Beth",
  "last_name": "Smith",
  "gender": "female",
  "preferred_language": "en",
  "phone_number": "800123456",
  "phone_number_verified_at": "2017-06-22T12:11:54.463Z",
  "phone_number_verified_by": "operator",
  "birthdate": "2017-06-01T12:01:23.456Z",
  "birthdate_verified_at": "2017-06-01T12:01:23.456Z",
  "birthdate_verified_by": "operator",
  "address": {
    "street": "Platz der Republik 1",
    "city": "Berlin",
    "postal_code": "11011",
    "state": "Brandenburg",
    "country": "de"
  },
  "password_digest_name": "bcrypt",
  "password_digest": "$2a$10$ekiHPhlSKTAI4svvLrBgwefJ6/XlcOmUlPel1ovsGnfN2R6rC9QgS",
  "password_salt": "E1F53135E559C253",
  "created_at": "2017-06-01T12:01:23.456Z"
}

PHP bcrypt digests: PHP had faulty $2a$ implementation and to differentiate fixed hashes from the wrong ones the prefix is $2y$. It is necessary to just replace it with $2a$.

Good explanation of different BCrypt prefixes can be found on StackOverflow.

We currently support only $2a$ digests.

A validation script is available to verify the structure of the export file.

• clone the repository and enter its directory
• make sure the recent enough Node.js is installed

  $ nvm install

• install it globally for the given version of Node.js

  $ npm install -g

• the validation script is now available as migrations-validate executable

Internally the validation script can be installed as simple as:

npm install -g @pass/migrations-validate

$ migrations-validate --help
...
$ migrations-validate examples/*.jsonl
2017-11-08T13:22:59.614Z Processing 'examples/invalid.jsonl'...
2017-11-08T13:22:59.628Z Report for 'examples/invalid.jsonl':
    processed: 4
    unsupported bcrypt password digest scheme, please substitute $2y$ prefix with $2a$: 1
    emailNotLowerCase: 2
    suspicious bcrypt password digest: 2, 3
    invalidPasswordDigest: 4
    duplicateEmail: [1,3], [2,4]
2017-11-08T13:22:59.629Z Processing 'examples/valid.jsonl'...
2017-11-08T13:22:59.631Z Report for 'examples/valid.jsonl':
    processed: 2
2017-11-08T13:22:59.631Z Finished

The script will read the file line by line, it will not read the whole file to memory at once. The script will print the current status (report) every 5 seconds and at the very end. The processed field contains the number of rows that have been processed so far. Other field names indicate the type of error encountered and the value is an array of corresponding line numbers. Note that the array of line numbers is capped to the first 50 elements for each error. Also note that the email duplicity check gathers all emails in a hash as the file is being processed so that it can determine email duplicities when the whole file is parsed. This can be memory intensive for very large files and it can be disabled via parameter.