Comments (7)
Thanks @vbothe23 for filing this issue; this is a known problem and the ultimate fix is tracked under #560. That said, extensions are currently half-supported if you are willing to hard-code the extension URL and recompile the code. An example for this is described in PR #562.
from fhir-data-pipes.
@bashir2 can u help me like in where is our data from fhir server is getting transformed in batch
i got some extended keys in my subject like group id, identifier, with nulls
from fhir-data-pipes.
@bashir2 can u help me like in where is our data from fhir server is getting transformed in batch i got some extended keys in my subject like group id, identifier, with nulls
Sorry @LovjeetVyas for the late reply; can you please elaborate more on your issue? Are you saying that you have some FHIR extensions and you are not getting those values in the Parquet files? If that is the case, then this is expected behavior unless if you hardcode the extension URL (as described in #562). The fix for this is in #924 which @chandrashekar-s is working on and is expected to be merged this week.
from fhir-data-pipes.
Hello @chandrashekar-s, @bashir2 I trust this message finds you well. I am reaching out to seek your assistance regarding a matter concerning the compatibility of Google FHIR Data Pipes with the Microsoft FHIR server. Our team has been conducting research and encountering challenges in establishing a connection between the two platforms. I would greatly appreciate your guidance on the requirements for connecting with the Microsoft FHIR server. Additionally, I am attaching a screenshot illustrating the specific issue we are currently facing. Your prompt response and insights would be highly valuable. Thank you, and I look forward to your reply.
BELOW IS THE COMMAND WE ARE USING:
PS D:\ccare\datapipes\fhir-data-pipes> java -jar ./pipelines/batch/target/batch-bundled.jar --fhirServerUrl=https://tenantfhirwrapperdata-tenantfhirservice.fhir.azurehealthcareapis.com --fhirServerOAuthTokenEndpoint=https://login.microsoftonline.com/xysd-sdasdan-xsdasd/oauth2/token --fhirServerOAuthClientId=fhune-s3434n3-sd3dgf45 --fhirServerOAuthClientSecret=uib3b4_34onin23 --resourceList=Patient --batchSize=10 --targetParallelism=2 --fhirSinkPath="" --sinkUserName="" --sinkPassword="" --outputParquetPath=/DATA_WAREHOUSE --jdbcModeEnabled=false --jdbcMaxPoolSize=50 --fhirDatabaseConfigPath=/workspace/utils/hapi-postgres-config.json --jdbcInitialPoolSize=10 --jdbcFetchSize=10000 --jdbcModeHapi=false --runner=DirectRunner --activePeriod="" --since=""
[above is the cli command we are using for connection'
13:07:49.074 [main] INFO com.google.fhir.analytics.FhirEtl com.google.fhir.analytics.FhirEtl.main:389 - Flags: Current Settings:
activePeriod:
appName: FhirEtl
batchSize: 10
fhirDatabaseConfigPath: /workspace/utils/hapi-postgres-config.json
fhirServerOAuthClientId: #############################
fhirServerOAuthClientSecret: #############################
fhirServerOAuthTokenEndpoint: https://login.microsoftonline.com/#######################/oauth2/token
fhirServerUrl: https://ccarefhirwrapperdata-ccarefhirservice.fhir.azurehealthcareapis.com
fhirSinkPath:
jdbcFetchSize: 10000
jdbcInitialPoolSize: 10
jdbcMaxPoolSize: 50
jdbcModeEnabled: false
jdbcModeHapi: false
optionsId: 0
outputParquetPath: /DATA_WAREHOUSE
resourceList: Patient
runner: class org.apache.beam.runners.direct.DirectRunner
since:
sinkPassword:
sinkUserName:
stableUniqueNames: WARNING
targetParallelism: 2
13:07:49.119 [main] INFO ca.uhn.fhir.util.VersionUtil ca.uhn.fhir.util.VersionUtil.initialize:84 - HAPI FHIR version 6.6.2 - Rev 8d55781507
13:07:49.127 [main] INFO ca.uhn.fhir.context.FhirContext ca.uhn.fhir.context.FhirContext.:210 - Creating new FHIR context for FHIR version [R4]
13:07:49.200 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-resources.xml
13:07:49.225 [main] INFO ca.uhn.fhir.util.XmlUtil ca.uhn.fhir.util.jar.DependencyLogImpl.logStaxImplementation:74 - FHIR XML procesing will use StAX implementation 'null' version 'null'
13:07:51.619 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-types.xml
13:07:51.727 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/profile/profiles-others.xml
13:07:51.935 [main] INFO c.u.f.c.s.DefaultProfileValidationSupport c.u.f.c.s.DefaultProfileValidationSupportBundleStrategy.loadStructureDefinitions:371 - Loading structure definitions from classpath: /org/hl7/fhir/r4/model/extension/extension-definitions.xml
13:07:52.736 [main] INFO com.google.fhir.analytics.FetchUtil com.google.fhir.analytics.FetchUtil.:82 - Fetching access tokens from https://login.microsoftonline.com/2466764f-5f33-4d63-be86-052226a72c9a/oauth2/token
13:07:53.567 [main] INFO c.g.fhir.analytics.FhirSearchUtil com.google.fhir.analytics.FhirSearchUtil.createSegments:221 - Fetching first batch of Patient
13:07:54.383 [main] ERROR com.google.fhir.analytics.FhirEtl com.google.fhir.analytics.FhirEtl.buildFhirSearchPipeline:141 - Either the date format in the active period is wrong or none of the resources support 'date' featurekey expires_in
Exception in thread "main" java.lang.IllegalArgumentException: key expires_in
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:360)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:335)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79)
at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73)
at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:460)
at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:324)
at com.google.fhir.analytics.FetchUtil$ClientCredentialsAuthInterceptor.requestAccessToken(FetchUtil.java:196)
at com.google.fhir.analytics.FetchUtil$ClientCredentialsAuthInterceptor.getToken(FetchUtil.java:172)
at com.google.fhir.analytics.FetchUtil$ClientCredentialsAuthInterceptor.interceptRequest(FetchUtil.java:188)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService$HookInvoker.invoke(BaseInterceptorService.java:517)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.doCallHooks(BaseInterceptorService.java:281)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.callHooks(BaseInterceptorService.java:269)
at ca.uhn.fhir.interceptor.executor.BaseInterceptorService.callHooks(BaseInterceptorService.java:63)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:319)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:538)
at ca.uhn.fhir.rest.client.impl.GenericClient$FetchConformanceInternal.execute(GenericClient.java:830)
at ca.uhn.fhir.rest.client.impl.RestfulClientFactory.validateServerBase(RestfulClientFactory.java:320)
at ca.uhn.fhir.rest.client.impl.RestfulClientFactory.validateServerBaseIfConfiguredToDoSo(RestfulClientFactory.java:283)
at ca.uhn.fhir.rest.client.impl.BaseClient.invokeClient(BaseClient.java:245)
at ca.uhn.fhir.rest.client.impl.GenericClient$BaseClientExecutable.invoke(GenericClient.java:538)
at ca.uhn.fhir.rest.client.impl.GenericClient$SearchInternal.execute(GenericClient.java:1997)
at com.google.fhir.analytics.FhirSearchUtil.createSegments(FhirSearchUtil.java:224)
at com.google.fhir.analytics.FhirEtl.buildFhirSearchPipeline(FhirEtl.java:139)
at com.google.fhir.analytics.FhirEtl.setupAndBuildPipelines(FhirEtl.java:377)
at com.google.fhir.analytics.FhirEtl.main(FhirEtl.java:395)
Caused by: java.lang.IllegalArgumentException: key expires_in, field private java.lang.Long com.google.api.client.auth.oauth2.TokenResponse.expiresInSeconds
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900)
at com.google.api.client.json.JsonParser.parse(JsonParser.java:451)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787)
... 29 more
Caused by: java.lang.IllegalArgumentException: number field formatted as a JSON string must use the @JsonString annotation at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:47)
at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:863)
... 31 more
from fhir-data-pipes.
Hi @LovjeetVyas, sorry for the late response. The auth implementation that is followed OAuth2 standards and hence a successful auth response must receive a token in this format, i.e. the expires_in
field must be a Long
. Even the microsoft implementation follows the same.
Also, can you please create a new issue for this, as the context for this is different.
from fhir-data-pipes.
Support for extensions has been provided with the PR #924, now the fhir profile structure definitions can be configured with the parameter structureDefinitionsDir and the pipelines will use these extensions during conversion to Parquet format. By default, the US Core profile FHIR extensions is supported.
One limitation with the current solution is that, if multiple fhir profiles are configured for the same resource type, then at any given point in time only one profile is applied. A more generic solution is being tracked as part of this issue #980
from fhir-data-pipes.
hey @chandrashekar-s , sorry for this late response I was indulged in some activities and didnt got the time
but so far from our side to authenticate our request we made some changes for that string value that we were getting
and we were getting a url in resource field too that was required for authentication that was not in fhir-datapipes auth files so added that too
and its working fine
another issue that we faced was with pagination
casue ms fhir doesnt support the pagination as hapi fhir and none was specified in the repo and "fhirsearchutil" thus we made some changes into that too
now we are facing a problem with the extension field and currently trying to resolve that too
thankyou for your response and help earlier, if again i got stuck in something will definitely reach out to you guys
from fhir-data-pipes.
Related Issues (20)
- FhirEtl with DataflowRunner does not produce Parquet files HOT 7
- Document current parquet schema for the sql-on-FHIR v1 and come up with a plan to standardize it with the community
- NPE while trying to find StructureDefinition for a resource extensions that is missing.
- Integrate FHIR Bulk export API with data-pipes
- Add config to change the pipeline controller cron schedule timezone HOT 1
- add config to override the default 100 batch size for server queries HOT 1
- Add last run time and purging schedule to the pipeline contoller UI
- Various documentations
- Feature Request: Options to rerun pipelines for selective date ranges HOT 2
- Feature Request: Pipeline Failure Notifications
- Relook at the `recursiveDepth` parameter
- Fix the number of parquet files generation for Direct Runner
- Create a Util Function for testing Resources in the FHIR store
- Add ability to turn on /off parquet file generation in case of syncying fhir to fhir server
- Support BULK_EXPORT mode of fetching FHIR resources for Incremental Run
- Controller maven module unit tests in `Junit 4` does not report coverage correctly
- Make sure all `DoFn`s are idempotent
- Migrate all unit test cases to Junit 5 platform HOT 5
- Change the default for batchSize to 3000
- FhirEtl misses resources HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fhir-data-pipes.