This is important (for the v0.2.0?) so Snowplow BigQuery usage can be attributed to us.

/cc @yalisassoon

BigQuery client supports mutating schema on-fly with standard Load job API, however it is not available in Beam yet.

https://issues.apache.org/jira/browse/BEAM-876

To start with, we're interested to know:

• How many events were loaded?
• What was the latency?

If user defines a context with array root type, then in the resulting LoaderRow we get following structure:

"contexts_com_acme_mycontext_1_0_0": [
  [
    {
      "key": "foo",
      "value": "bar"
    },
    {
      "key": "one",
      "value": "two"
    }
  ]
]

First-level array because contexts always have zero-or-more cardinality and second-level array is actual context.

This structure is not supported by BigQuery, which means we have to flatten this array into single-level.

• BINTRAY_SNOWPLOW_DOCKER_USER
• BINTRAY_SNOWPLOW_DOCKER_API_KEY

SLF4J: Defaulting to no-operation (NOP) logger implementation

Repeater checks every time if event is old enough to make an insertion attempt. However, it checks it after every pull which happens very often. As a result we have messages like:

DEBUG com.snowplowanalytics.snowplow.storage.bigquery.repeater.Repeater - Event fcbcc0b1-e51b-488b-9a90-7403cdfb19f4/2020-02-19T21:53:36.943Z is not ready yet. Nack

Printed every 300 milliseconds.

However, this is logged with DEBUG level, which means user can technically get rid of it.

• BINTRAY_SNOWPLOW_DOCKER_USER
• BINTRAY_SNOWPLOW_DOCKER_API_KEY

Settings should be specified and exposed like in: https://github.com/snowplow/snowplow-scala-project.g8/blob/app/src/main/g8/project/BuildSettings.scala

• BINTRAY_SNOWPLOW_GENERIC_USER (unencrypted?)
• BINTRAY_SNOWPLOW_GENERIC_API_KEY

Important note: we'll be publishing tar.gz archives with executables there, not fatjar, because scio/beam does not work with sbt-assembly.

Probably we'll also need docker registry credentials as me and @BenFradet thinking about using sbt-native-packager (?) and its Docker integration and publish an image straight away.

We need to migrate documentation from wiki to docs.snowplowanalytics.com/

I don't know how to create an executable on of the git repo? I am clearly missing something but after cloning and calling the given:

./snowplow-bigquery-loader \
    --config=$CONFIG \
    --resolver=$RESOLVER

It doesn't find it, which makes sense because I don't see this.

We are using Base64.getDocoder().decode which does not allows for mentioned characters.
We should investigate using Base64.getUrlDocoder().decode which would allow for these.

source: https://discourse.snowplowanalytics.com/t/illegal-base64-character-when-running-bigquery-mutator-with-docker/3662

The most reliable way is to try to achieve it as a SQL function, but doesn't look like streaming inserts provide this kind of API.

Should bring important bug fixes:

• snowplow/iglu#446
• snowplow/iglu#461

I have configured to send data every 100 seconds, I can see the records in DataFlow heading into the BigQuery write job, however, they never get written to BigQuery. Why would this be? If you need further information to help, please ask.

Thanks

Currently Repeater's statistics message looks like following:

Statistics: 43 rows inserted, 114 rows rejected

This message could be totally fine if repeater is running for a month and rejected just 114 rows, but can be also alarming if it reject 114 rows in an hour. In order to figure out how long it is running we need to manually check when it was launched and then substract it from log entry timestamp.

It would be really nice if it was

Statistics: 43 rows inserted, 114 rows rejected in last 4 hours

Migrated from snowplow/snowplow#3934

At first I was thinking about using description to store metadata such as Iglu URI, but in the end I agree with @colmsnowplow - this should be something human-readable and if user changes the description - it should not affect the loading.

Given that BigQuery now has some geography support does it make sense to convert the latitude / longitude columns to a single GEOGRAPHY column of type ST_GEOGPOINT?

When deploying more than 1 pipeline on the same GCP project there is no way to know where the costs are coming from.

GCP supports labeling the resources and to export the billing information into BigQuery. The cost can then be breakdown by using SQL queries that filter by label.

For reference, the google documentation on labels: https://cloud.google.com/compute/docs/labeling-resources

The plan is to label all the resources with "sp_env=prod1" or "sp_env=qa1"

BigQuery columns/fields can only contain alphanumeric or underscore characters, however there are use cases whereby we have no control over different characters being in field names - for example, when integrating data from a vendor like branch, users can't always rename fields with different characters.

We might strip special characters, or maybe replace them with underscores to avoid clashes with other column names perhaps.

To add the plugin:

addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.3.2")

And a .scalafmt.config file can be found here.

• add repeater test example
• update copyright

java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: Failed to create load job with id prefix <REDACTED>, reached max retries: 3, last failed load job

Once this error occurs, no more data is inserted, it seems to block any further inserts, should this gracefully fail and move onto the next file?

These two operations:

By replacing some of the current transaction FLOAT types with NUMERIC we avoid any floating point arithmetic issues that we can run into if using BigQuerys FLOAT64.