Code Monkey home page Code Monkey logo

smolder-solacc's Introduction

Smolder: A package for ingesting HL7 messages to Delta Lake

In this notebook, we demonstrate how Smolder can be used to parse HL7v2 message into your lakehouse. Smolder defines a very simple Spark file format that loads one HL7 message per file, and parses it into a dataframe. Here, we'll demonstrate this using a stream of HL7 ADT messages generated from a simulated dataset with Synthea


© 2022 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License [https://databricks.com/db-license-source]. All included or referenced third party libraries are subject to the licenses set forth below.

Library Name Library License Library License URL Library Source URL
Spark Apache-2.0 License https://github.com/apache/spark/blob/master/LICENSE https://github.com/apache/spark
Smolder Apache-2.0 License https://github.com/databrickslabs/smolder https://github.com/databrickslabs/smolder/blob/master/LICENSE

To run this accelerator, clone this repo into a Databricks workspace. Attach the RUNME notebook to any cluster running a DBR 11.0 or later runtime, and execute the notebook via Run-All. A multi-step-job describing the accelerator pipeline will be created, and the link will be provided. Execute the multi-step-job to see how the pipeline runs.

The job configuration is written in the RUNME notebook in json format. The cost associated with running the accelerator is the user's responsibility.

Disclaimers

Databricks Inc. (“Databricks”) does not dispense medical, diagnosis, or treatment advice. This Solution Accelerator (“tool”) is for informational purposes only and may not be used as a substitute for professional medical advice, treatment, or diagnosis. This tool may not be used within Databricks to process Protected Health Information (“PHI”) as defined in the Health Insurance Portability and Accountability Act of 1996, unless you have executed with Databricks a contract that allows for processing PHI, an accompanying Business Associate Agreement (BAA), and are running this notebook within a HIPAA Account. Please note that if you run this notebook within Azure Databricks, your contract with Microsoft applies.

All names, last names and places in this notebook have been randomly generated. No identification with actual persons (living or deceased), places, buildings, and products is intended or should be inferred.

smolder-solacc's People

Contributors

danielsparing avatar dbbnicole avatar

Stargazers

 avatar  avatar

Watchers

 avatar

smolder-solacc's Issues

Undocumented syntax: DESC HISTORY

@dbbnicole

Hi, I notice you contributed to code in this repo (4240e4e)

In the above commit, line 183 (now line 174), there is the following SQL command:

DESC HISTORY hl7_adt_stream

This command fails parsing using industry tools such as sqlglot. (I discovered that while working on ucx)

Looking at Databricks docs, I find a DESCRIBE HISTORY command, but it says nothing about a 'DESC' shortcut for 'DESCRIBE'.

Is that a typo ?

Or is this an undocumented syntax that should be supported (and documented) ?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.