nhsx / au-data-engineering Goto Github PK

.. image:: _static/img/screenshot.png
  :width: 600
  :alt: text describing screenshot
*Figure 1: text describing screenshot*

Pipeline to ingest raw data in a time-stamped folder to Azure Datalake blob storage from a SQL database.

 1. Lookups the JSON configuration file for this pipeline
 2. Source:

    a. Sets the source database owner (dbo)
    b. Sets the source table
    c. Sets the SQL query

 3. Sink:

    a. Sets the file system
    b. Sets the sink path
    c. Sets the sink file

 4. Copy activity copies the data returned from the SQL query as either a .csv file or a .parquet file.

Below are some test paragraphs.

1. This is a paragraph detailing a higher level step, with 3 sub-steps.

   a. This is a sub-bullet, with an example::

         ls -l foo.txt

   b. I can get this line to display as part of the code block!
      This is another sub-bullet, with an example::

         git add bar.txt

2. This is step 2.