Comments (1)
Following the offline conversions, here are a few points about this for posterity:
- Our Parquet file generation used to be "append only"; this would mean that if a FHIR resource was changed, two versions of it existed in the DWH and a deduplication should be done at query time.
- The above situation was not ideal for large resource tables, e.g., Observations; IOW this can put a performance burden at query time.
- Because of this reason, we added the merger in #364. The performance of the merger should be good because it reads from filesystem and writes to it (no FHIR-server/DB interaction). That said, if for some resources we know that deduplication is not necessary, for example we know that an Observation will not be modified after it first was created, then we can add an option to disable merge for certain resource types.
- This obviously leave the responsibility to the user to make sure if any deduplication is needed it is taken care of at query time.
- Finally, this is also related to #1012; i.e., if we want to keep historical information for a resource type, we should also disable merge/deduplication for that and change queries to take care of that.
To summarize: The work to be done here is to add the feature to disable merge for some resource types and also do not copy their Parquet files in DWH snapshots (instead re-use one copy).
from fhir-data-pipes.
Related Issues (20)
- Various documentations
- Feature Request: Options to rerun pipelines for selective date ranges HOT 2
- Feature Request: Pipeline Failure Notifications
- Relook at the `recursiveDepth` parameter
- Fix the number of parquet files generation for Direct Runner
- Create a Util Function for testing Resources in the FHIR store
- Add ability to turn on /off parquet file generation in case of syncying fhir to fhir server
- Support BULK_EXPORT mode of fetching FHIR resources for Incremental Run
- Controller maven module unit tests in `Junit 4` does not report coverage correctly
- Make sure all `DoFn`s are idempotent
- Migrate all unit test cases to Junit 5 platform HOT 6
- Change the default for batchSize to 3000
- FhirEtl misses resources HOT 9
- Support Parquet as an option for output flat views
- Extra Documentation for mapping ViewDefinitions to DB tables
- Reduce CI build time
- Table Not Found HOT 2
- Implement CLI for Controller HOT 1
- Resource Table Missing Extension Column.
- As a Java Developer, facing error(s) when trying to run the `controller` locally via `mvn spring-boot:run`. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fhir-data-pipes.