vdandugc / sfintegrationpoc Goto Github PK

App will sync Account data with Salesforce via its Change Data Capture(CDC) , On Account creation event it will generate a unique code and update Salesforce via API

This is a POC. Items skimped

• Ordering and Buffering (Transaction Based Replication)
• Authentication
• Security
• Idiomatic Code
• Deletion of record on SF

Listen to events on Topic - /data/AccountChangedEvent which Salesforce publishes change data capture events on PubSub when Account Objects are created/updated/deleted to build the Account data on our side.

Initial thoughts on how to approach (can improve as we move forward). Move a pipeline which subscribes on Pub-Sub, store the parse the data and enrich and save the data.

subscribe -> store -> parse (group by transactionKey and order by sequence) -> enrich -> save

There are other cases which need to be considered and partially considered here.

1. ordering and buffering
2. message reliablity
3. error handling (replayId)

The order of change events stored in the event bus corresponds to the order in which the transactions corresponding to the record changes are committed in Salesforce. If a transaction includes multiple changes, like a lead conversion, a change event is generated for each change with the same transactionKey but different sequenceNumber in the header. The sequenceNumber is the order of the change within the transaction.

There is a temporary storage in event bus, we also propse to save on our side. Save replayId to replay events in case events are missed, this makes system more resilient. Link

Refer this for error handling.

To run this

• Set up Salesforce dev account and CLI app (to use the auth token)
• Set up the database schema
• Run

• Setup account in https://developer.salesforce.com/
• Setup a CLI app (instructions)
• Enable Change Data Capture in Salesforce settings, instructions here More on Salesforce Change Data Capture here
• Update values(AccessToken and OAuthEndpoint) in common.go

Create database with schema

>> createdb testdb
>> psql -d testdb
psql >> CREATE USER foouser WITH PASSWORD 'foopassword' CREATEDB CREATEROLE;
psql >> exit

psql -d testdb -U foouser --password

CREATE TABLE ACCOUNTS(
  id TEXT NOT NULL PRIMARY KEY,
  account_number text,
  name text,
  signup_code text
)

CREATE TABLE EVENTS(
  id TEXT NOT NULL PRIMARY KEY,
  schema_id TEXT,
  payload bytea,
  replay_id bytea
)

go run subscribe/subscribe.go