• Define IO for writing bounded and unbounded collections to GCS
• Reuse IO in Dead Letter Queue (in a similar way the Diagnostic reuse BigQuery IO)

Show unified batch and streaming for TollApplication.

• build flex templates for batch and streaming during CI
• prepare scripts for running jobs

Unify diagnostic output for all IOs. Typical use-case: handling corrupted Pubsub messages if the message couldn't be decoded into raw case class. Currently handled explicitly in toll-application:

    SCollection
      .unionAll(
        Seq(
          boothEntriesRawDlq.map(x => IoDiagnostic(x.id, x.error)),
          boothExitsRawDlq.map(x => IoDiagnostic(x.id, x.error)),
          vehicleRegistrationsRawUpdatesDlq.map(x => IoDiagnostic(x.id, x.error))
        )
      )
      .keyBy(_.key)
      .writeDiagnosticToBigQuery(IoDiagnosticTableIoId, config.ioDiagnosticTable)

Re-think diagnostic output:

• how to corelate with specific domain case in type safe manner?
• what kind of statistics it should provide: counter, histograms, etc?
• how to model BigQuery table schema for convinient use?

1. Scio CustomOutput doesn't work because it returns ClosedTap and we need dead letters.
2. TransformOverride is a hack but how to easily test output?
3. In general all methods in the infrastructure module should be implemented as single transformation, easy to stub in the job tests.

TimestampedMatchers delegates to ScioCollectionMatcher but they should be tested anyway

Generated payloads for BigQuery should show what is accepted and what is not.
Domain types like dates and numbers might accept values out of valid ranges.

#146

Split shared package into shared (allowed as domain dependency) and infrastructure (allowed only as application dependency)

• add historical vehicle registrations (BigQuery)
• add event time to realtime vehicle registration updates (Pubsub) - does it make any sense if vehicle registration is defined as side input and joined using processing time?
• add invalid realtime vehicle registration event
• add corrupted realtime vehicle registration event
• consider side input in finite window (e.g. 1 day) instead of global window

Report vehicles with expired registration using side input as registration lookup table.

SELECT EntryStream.EntryTime, EntryStream.LicensePlate, EntryStream.TollId, Registration.RegistrationId
FROM EntryStream TIMESTAMP BY EntryTime
JOIN Registration
ON EntryStream.LicensePlate = Registration.LicensePlate
WHERE Registration.Expired = '1'

How to update state of the streaming job using BigQuery?

Right now the jobs can be deployed on Dataflow but there is no test data.

It's a great opportunity to use Dataflow data generator template :)
https://cloud.google.com/dataflow/docs/guides/templates/provided/streaming-data-generator