Implement FHIR bulk paging logic inside the analytics engine codebase about fhir-data-pipes HOT 10 CLOSED

To add more context to this: Currently the batch pipeline relies on FHIR search API and the specific way HAPI implements paging. This means that for very large DBs, the initial query for creating the list of IDs for resources to return can take very long. The idea is to implement a way that avoids such long DB queries and instead reads the list of IDs in segments from the DB directly.

That said, one benefit of the current implementation is that it supports general FHIR search URLs, e.g., with filters as mentioned here so it would be great to keep the current functionality while adding a direct DB based implementation too.

from fhir-data-pipes.

@kimaina That seems like a sensible way to implement this to me!

from fhir-data-pipes.

Thanks @bashir2 for adding more context!

from fhir-data-pipes.

I am almost done working on this, just a few design issue and a possible solution that needs to be discussed,

some tables do not have the UUID column necessary for FHIR extraction (e.g patient table), to resolve this, I have added a new field in the JSON config schema that solves this problem. What do you think? @bashir2 @ibacher

 "patient": {
      "enabled": "true",
      "title": "Patient",
      "uuidTable": "person",
      "linkTemplates": {
        "rest": "/ws/rest/v1/patient/{uuid}?v=full",
        "fhir": "/Patient/{uuid}"
      }
    },

from fhir-data-pipes.

@kimaina My question here would be how do we know how to join to the person table in that instance? For instance to find the UUID for a patient, you need to match the patient_id field to the corresponding person record where person.person_id = patient_id; however, to do the same for, e.g., drug orders, we need to match order record where order.order_id = drug_order.order_id.

from fhir-data-pipes.

Thanks @ibacher! For the batch case, we do not need to do any joins since we can directly fetch UUIDs from parent tables. However, I see how this approach can be vital for streaming mode. I guess we need another field to indicate how we do the join for the streaming bit:

 "patient": {
      "enabled": "true",
      "title": "Patient",
      "parentTable": "person",
     "joinClause": "person.person_id = patient_id"
      "linkTemplates": {
        "rest": "/ws/rest/v1/patient/{uuid}?v=full",
        "fhir": "/Patient/{uuid}"
      }
    },

WDYT?

@kimaina My question here would be how do we know how to join to the person table in that instance?

from fhir-data-pipes.

@ibacher thanks for the suggestion! Thinking about it carefully, we still need to do join even for the batch case. So the same suggestion applies! We will need to create a ticket for this!

For the batch case, we do not need to do any joins since we can directly fetch UUIDs from parent tables.

from fhir-data-pipes.

I see we already to have a ticket for this: #46

from fhir-data-pipes.

@kimaina can we close this issue now that PR #72 is submitted or are there any pieces that are still left?

from fhir-data-pipes.

We can mark this as "done" and track specific issues/bugs in separate tickets.

from fhir-data-pipes.