greenstand / bulk-pack-transformer-v2 Goto Github PK

0.0 7.0 2.0 866 KB

License: GNU General Public License v3.0

Shell 0.81% Dockerfile 0.34% JavaScript 98.85%

bulk-pack-transformer-v2's Introduction

Name of this microservice

Description of this microservice

Development toolkit

This repository was created from Greenstand's template for microservice projects. This means it comes with many development tools that we use for development and deployment. As a contributor to this repository, you should learn and use these tools. They are outlined below.

db-migrate

db-migrate is used to create database tables and other database objects

Conventional Commits

https://www.conventionalcommits.org/en/v1.0.0/

husky

husky will check commit messages to make sure they follow the required format

prettier / lint

github actions

mocha

Getting Started

Project Setup

Open terminal and navigate to a folder to install this project:

git clone https://github.com/Greenstand/treetracker-repository-name.git

Install all necessary dependencies:

npm install

Database Setup

This repository using db-migrate to manage database migrations for its schema.

cd database/
db-migrate --env dev up

If you have not installed db-migrate globally, you can run:

cd database/
../node_modules/db-migrate/bin/db-migrate --env dev up

Documentation for db-migrate: https://db-migrate.readthedocs.io/en/latest/

Architecture of this project

This project use multiple layer structure to build the whole system. Similar with MVC structure:

Protocol layer

Wallet API offers RESTFul API interace based on HTTP protocol. We use Express to handle all HTTP requests.

The Express-routers work like the controller role in MVC, they receive the requests and parameters from client, and translate it and dispatch tasks to appropriate business objects. Then receive the result from them, translate to the 'view', the JSON response, to client.

Service layer

Both service layer and model layer are where all the business logic is located. Comparing to the Model , service object don't have state (stateless).

Please put business logic code into service object when it is hard to put them into the Model object.

Because we didn't use Factory or dependency injection to create object, so service layer also can be used as Factory to create model object.

Model layer

The business model, major business logic is here. They are real object, in the perspective of object oriented programming: they have states, they have the method to do stuff.

There are more discussion about this, check below selection.

Repository layer

Repository is responsible for communicate with the real database, this isolation brings flexibility for us, for example, we can consider replace the implementation of the storage infrastructure in the future.

All the SQL statements should be here.

How to test

Unit test

To run the unit tests:

npm run test-unit

Integration test

All the integration tests are located under folder __tests__

To run the integration test:

Run tests:

npm run test-integration

Database seeding test

In order to efficiently run our integration tests, we rely on automated database seeding/clearing functions to mock database entries. To test these functions, run:

npm run test-seedDB

Suggestion about how to run tests when developing

There is a command in the package.json:

npm run test-watch

By running test with this command, the tests would re-run if any code change happened. And with the bail argument, tests would stop when it met the first error, it might bring some convenience when developing.

NOTE: There is another command: test-watch-debug, it is the same with test-watch, except it set log's level to debug.

Postman

Can also use Postman to test the API manually.

To run a local server with some seed data, run command:

npm run server-test

This command would run a API server locally, and seed some basic data into DB (the same with the data we used in the integration test).

bulk-pack-transformer-v2's People

Contributors

Watchers

Forkers

kpoke ibnmajah

bulk-pack-transformer-v2's Issues

Message resource support into dev env

Message resources only exist in bulk back v2, so the processor only needs to send to transformer-v2, and they only need to be inserted into the message table in the messaging schema.

Alerting for bulk pack services

Add more detail to external service failure error message

When an orchestration layer API has a failure downstream, log more details about the error so we can debug faster

body: {
code: 500,
message: 'Unknown error occured with external service call'
},

Catch these errors and return with a HTTP code that can mean 'try again later'

We can have a totally valid situation where raw_captures can't ingest yet because the associated sessions did not make it into the database yet. We should not through a 500 error and does not need to dump the entire trace to the log file. We would like to log that a foreign key doesn't exist yet, and tell the client to retry. We could use 409 or 503 for the error code, I'm not sure right now

   message: 'Unknown error (with "createdRawCapture" as (insert into "raw_capture" ("abs_step_count", "captured_at", "created_at", "delta_step_count", "extra_attributes", "gps_accuracy", "id", "image_url", "lat", "lon", "note", "reference_id", "rotation_matrix", "session_id", "status", "updated_at") values ($1, $2, $3, $4, $5, DEFAULT, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) returning *) select "createdRawCapture"."id", "createdRawCapture"."reference_id", "createdRawCapture"."image_url", "createdRawCapture"."lat", "createdRawCapture"."lon", "createdRawCapture"."gps_accuracy", "createdRawCapture"."note", "createdRawCapture"."abs_step_count", "createdRawCapture"."delta_step_count", "createdRawCapture"."rotation_matrix", "createdRawCapture"."session_id", "createdRawCapture"."rejection_reason", "device_configuration"."device_identifier", "wallet_registration"."grower_account_id", "wallet_registration"."wallet", "wallet_registration"."user_photo_url", "createdRawCapture"."extra_attributes", "createdRawCapture"."status", "createdRawCapture"."created_at", "createdRawCapture"."updated_at", "createdRawCapture"."captured_at" from "createdRawCapture" left join "session" on "createdRawCapture"."session_id" = "session"."id" left join "wallet_registration" on "session"."originating_wallet_registration_id" = "wallet_registration"."id" left join "device_configuration" on "session"."device_configuration_id" = "device_configuration"."id" - insert or update on table "raw_capture" violates foreign key constraint "raw_capture_session_id_fkey")'
  },

npm ERR! cpu-features not accessible from ssh2

Validate message payload before sending

Message payload must have id, composed_at and other fields before sending to message endpoint
(message endpoint itself does not require these fields)

Error receiving v1 bulk pack

2020.02.26.00.29.15_64064f11-2247-4f5c-98a3-085a583aa155_9cfc31605512baf361fee55f6a7567f0
bulk pack format version 1 detected
processing v1 trees





===================================
{
  transformerUrl: 'http://bulk-pack-transformer-v2.bulk-pack-services/v1/tree',
  data: {
    lat: 37.84495902,
    lon: -122.25719742,
    note: '',
    uuid: '62e8258b-1f50-4275-ad72-9da256f4924b',
    user_id: 1,
    image_url: 'https://treetracker-production-images.s3.eu-central-1.amazonaws.com/2020.02.26.00.29.13_37.84495902_-122.25719742_9780fcdf-9006-471e-b0d3-fd51f2b1b1fb_IMG_20200226_002554_1723951794203185136.jpg',
    timestamp: 1582705560587,
    attributes: [],
    sequence_id: 1,
    gps_accuracy: 10,
    planter_photo_url: 'https://treetracker-production-images.s3.eu-central-1.amazonaws.com/2020.02.26.00.29.08_37.84495843_-122.25719593_e1c445c9-7c9d-4086-906f-8cfc3212ef32_IMG_20200226_002537_2924224038665621192.jpg',
    planter_identifier: '3132687546'
  },
  response: { code: 500, message: 'Unknown error (Value must be string)' }
}
===================================

location field for grower_account

treetracker.grower_account needs to have a location field added, of type public.geometry(Point,4326). This value for this field should be populated on ingestion with a coordinate generated from lat,lon in the wallet_registration table. The value stored in the grower_account record should be UPDATED with the latest lat,lon as new wallet_registration records for the same grower_account arrive.

field data not getting processed correctly !!!

A lot of the packs are not getting processed.

v1 planter record is missing data

===================================
{
  transformerUrl: 'http://bulk-pack-transformer-v2.bulk-pack-services/v1/planter',
  data: {
    lat: 37.8449304,
    lon: -122.25714411,
    last_name: 'test',
    first_name: 'zzven',
    organization: '',
    device_identifier: '4f4d6a8b6433e4ca',
    planter_identifier: '3132687669'
  },
  response: {
    microserviceUrl: 'http://treetracker-api.treetracker-api/grower_accounts',
    data: '{"wallet":"3132687669","first_name":"zzven","last_name":"test","email":null,"phone":"3132687669","lat":37.8449304,"lon":-122.25714411,"first_registration_at":"2022-03-01T23:00:08.713Z"}',
    code: 422,
    message: '"image_url" is required'
  }
}

Error when deployed


TypeError: res.status is not a function
11:40:54.785
    at exports.errorHandler (/app/server/utils/utils.js:56:9)
11:40:54.785
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
11:40:54.785
    at trim_prefix (/app/node_modules/express/lib/router/index.js:317:13)
11:40:54.785
    at /app/node_modules/express/lib/router/index.js:284:7
11:40:54.785
    at Function.process_params (/app/node_modules/express/lib/router/index.js:335:12)
11:40:54.785
    at next (/app/node_modules/express/lib/router/index.js:275:10)
11:40:54.785
    at /app/node_modules/express/lib/router/index.js:635:15
11:40:54.785
    at next (/app/node_modules/express/lib/router/index.js:260:14)
11:40:54.785
    at Function.handle (/app/node_modules/express/lib/router/index.js:174:3)
11:40:54.785
    at router (/app/node_modules/express/lib/router/index.js:47:12)

bulk pack v2 test env deployment configuration

bulk pack v2 prod env deployment configuration

Organization should not be required

===================================
{
  transformerUrl: 'http://bulk-pack-transformer-v2.bulk-pack-services/sessions',
  data: {
    id: 'ddad6f74-015b-404a-ae82-f111d3f3c555',
    organization: '',
    target_wallet: '6333333333',
    device_configuration_id: 'af370725-5172-46de-8b87-299269610092',
    originating_wallet_registration_id: '55ea221d-7941-45e0-bd16-27fa9496958d'
  },
  response: {
    microserviceUrl: 'http://treetracker-field-data.field-data-api/session',
    data: '{"id":"ddad6f74-015b-404a-ae82-f111d3f3c555","device_configuration_id":"af370725-5172-46de-8b87-299269610092","originating_wallet_registration_id":"55ea221d-7941-45e0-bd16-27fa9496958d","target_wallet":"6333333333","organization":""}',
    code: 422,
    message: '"organization" is not allowed to be empty'
  }
}

Copy properties to POST objects rather than sending along object from bulk pack

The API currently updates and forwards objects received to downstream APIs. However, if the mobile team decides to add fields to the these objects, these fields are also forwarded and this triggers validation errors on downstream APIs.

Instead, all objects should be built up as new objects, with properties copied from the received POSTed object into the object to be POSTed downstream. This way the mobile team can add fields to objects for various reasons without breaking the ingestion of data.

Support extra attributes in bulk pack v1 for kisiki hai treetracker

Emailed Beau to see if this is still necessary.