Feature Request: Requires Delta Data to be retained in incremental Snapshots about fhir-data-pipes HOT 1 OPEN

Following the offline conversions, here are a few points about this for posterity:

• Our Parquet file generation used to be "append only"; this would mean that if a FHIR resource was changed, two versions of it existed in the DWH and a deduplication should be done at query time.
• The above situation was not ideal for large resource tables, e.g., Observations; IOW this can put a performance burden at query time.
• Because of this reason, we added the merger in #364. The performance of the merger should be good because it reads from filesystem and writes to it (no FHIR-server/DB interaction). That said, if for some resources we know that deduplication is not necessary, for example we know that an Observation will not be modified after it first was created, then we can add an option to disable merge for certain resource types.
• This obviously leave the responsibility to the user to make sure if any deduplication is needed it is taken care of at query time.
• Finally, this is also related to #1012; i.e., if we want to keep historical information for a resource type, we should also disable merge/deduplication for that and change queries to take care of that.

To summarize: The work to be done here is to add the feature to disable merge for some resource types and also do not copy their Parquet files in DWH snapshots (instead re-use one copy).

from fhir-data-pipes.