See instructions here.

We have moved away from 2 VMs for processing AQUA data to a single VM and the diagram needs to be updated to reflect this.

AOGS sends data downlinked from satellites to a specific IP and Port. This feature request is for a service that listens on an arbitrary port, and writes the data stream to Azure Storage. This will enable using Azure Storage as a repository of satellite raw data that can be easily made available to other applications for further processing.

Since the BlobDownloadService is a compilable artifact we should add its build to the GitHub Actions workflow to maintain build validity.

Add missing project descriptions in readme files and revise description to reflect current state.

Currently we are using filename for correlations, but not all events will have a filename reference. It would be good to generate a correlationId and tag all events for a given session with that id so we could easily correlate.

Once we have #43 completed, we need to provide documentation on how to setup IPOPP and watch a directory.

Currently TCP to BLOB users are responsible for creating a contact profile that targets the TCP to BLOB endpoint. This is additional work that increases the risk of user error during initial setup as well as increasing the risk of the contact profile becoming non-functional if/when the TCP to BLOB configuration changes.

DoD: All TCP to BLOB deployment mechanisms create/update an appropriately configured Orbital contact profile including:

• AZ CLI deployer
• Bicep deployer
• ADO pipeline deployer
• Github actions deployer (when available)

Missing appsettings.json docs and should include these for usability

Make it more clear in the docs the default location where blob data and RT-STPS data lands

Currently tcp-to-blob and processor log to their respective Log Analytics Workspaces. This improvement would bring the 2 workspaces together into a single ADX instance. This would be optional for users that want to bring their logging together in a single location.

To bring the 2 workspaces together, we would need to follow these steps:

1. Completion of: #19
2. Create and setup an Event Hub with 2 hubs (AppTraces and AppExceptions) to receive log exports from Log Analytics Workspaces
3. Create an ADX instance
4. Create a new database, tables, mappings, update policies and data connections in ADX
5. Configure the 2 workspaces to export AppTraces and AppExceptions to the Event Hub in step number 2
6. Validate data flowing into the 2 new tables in ADX (log ingestion can be up to 10min behind)

We need to update the deployment piece for AppExceptions to properly expand these values.

We should move ignores from ./blob-download-service/.gitignore into the root .gitignore.

Currently the Event Hub receiver leverages the contact data storage account for storing checkpoints. This can cause unwanted behavior if you do not have your Event Grid subscription setup appropriately. We should give the option for users to use a separate storage account if desired.

Current formatting of the pull request template does not auto-populate a link to the issue and does not correctly display checkboxes in the checklist. The following format works well.

# Description
Fixes #nn

Add description here. 

## Dependencies affected:
- 


## Checklist before merging
- [] Properly labeled PR 
- [] Licensing statement added to new files 
- [] Downstream dependencies have been addressed
- [] Corresponding changes to the documentation have been made
- [] Issue is linked under the development section

Once we have BlobDownloadService ready to use, we will need to integrate it with the solution, specifically it will be running as a service on the Aqua processing VM.

1. Update the deployment of the Aqua processing VM to use BlobDownloadService
2. Enable log export in the central-logging component

Once we have "Fully integrate BlobDownloadService #28", we will need to integrate integrate inotify and RT-STPS

Leverage inotify to trigger RT-STPS

The filename on property is missing for blob-write log events. Update code to ensure filename is logged.

Summary

Enable Enable Shared Access Signature (SAS) support for securing access to storage accounts.

Use case

Given

1. Two teams/organizations:
   1. Data provider: Owns the subscription/resource group that is on-boarded to Azure Orbital and contains the spacecraft (spacecraft-p) and contact profile (cp-p).
   2. Data recipient: Owns the subscription/resource group containing the storage account (storage-account-r) to which Orbital contact data is to be delivered.
2. Data Provider owned TCP to BLOB instance (tcp-to-blob-p) configured with the data recipient's SAS token for storage-account-r.
3. Data Provider owned contact profile (cp-p) pointing to tcp-to-blob-p endpoint.

When

Data provider owned contact for spacecraft-p hits cp-p.

Then

tcp-to-blob-p will deliver contact data to a BLOB in storage-account-r.

Problem

The TLE for an Orbital spacecraft must be specified during initial creation resulting in a static TLE setting. Over time the actual data the TLE is meant to represent drifts eventually resulting in contact failures if the spacecraft TLE setting is not updated.

1. Orbital users are likely not to be aware of TLE drift until their contacts fail.
2. Orbital users are responsible for establishing a process or system for periodically updating the TLE for each of their spacecrafts. Any failure of this process may result in missing data for all contacts until the TLE is properly updated.

DoD

Create an AKS service and/or Azure function that periodically updates the TLE for one or more configurable spacecrafts.

Currently the storage account connection string is stored as a Kubernetes secret.

DoD: Storage account connection string or SAS token is stored in Azure Key Vault where it it retrieved by TCP to BLOB.

We need to fix this behavior to make querying work easier and more efficient

Currently we are leveraging the built in logging that AKS provides, but we should deploy a separate Log Analytics Workspace and a separate Application Insights and point AKS to use the new separate workspace for logging.

Application Insights might not be required if AKS allows linking directly to a workspace.

MacBooks with ARM chips do not build docker images that can be used on AKS by default. Fix this by specifically targeting the linux os using the platform tag.

Make it a little more clear how BDS integrates with aqua-processor

Currently TCP to BLOB requires a bash-like shell for environment configuration which may be problematic for some environments. Additionally, lingering environment variables set in the shell have cause confusion/unexpected behavior for some users.

DoD: TCP to BLOB leverages Dotenv for environment configuration to enable cross-platform configuration in a standard way.

To start with a simple foundation, create a workflow that does linting. Depending on when this issue is put a code coverage threshold as well as a job to run unit tests.

We need to update the processor component (aqua-processor) to reflect how we are deploying tcp-to-blob using bicep. Simplify variable declarations (not everything needs to be adjusted) and adopt the .env pattern.

Add support for yarn workspaces for simple, fast and efficient installation.

Add prettier NPM script to format code.

In order to better control who has authority over PRs we should add a CODEOWNERS file https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners

Create NodeJS package that makes it easier to interact with Orbital API.

Currently we have the total end to end time from when a connection starts sending data to tcp-to-blob to when it is finished uploading. It would be good to have this broken down to separate tcp-to-blob receive time vs tcp-to-blob upload time.

Set up an event to trigger an ADO pipeline deployment based upon an agreed state in GitHub , most likely a named GitHub release.

Currently two VMs are used for the ground processing. In order to simplify the architecture, we should use the same VM for both steps, defaulting to whatever the config was for the most resource intensive task.

Update architecture diagram in the main README to show the new components and the correct flow.

Create a pipeline in GitHub to deploy TCP-to-Blob. If possible (and if it makes sense) we should leverage existing scripts that are used with other deployment methods to avoid having to maintain duplicate scripts.

Currently we have custom code to build environment files. We should update to use envsubst, awk or sed to handle token replacement.

Some users will want to use the Bicep templates without the parameters.json files. We should add reasonable default values directly in the Bicep templates.

RT-STPS 6.0 no longer includes scripts, just data. Update README to instruct users to use RT-STPS 7.0.

testdata/input/rt-stps_npp_testdata.dat is not available in version 7.0. Update readme with new sample testdata instructions.

We should move ignores from ./orbital-helper/.gitignore into the root .gitignore.

For any reason that there is a message in the queue that points to a deleted/non-existent blob, BlobDownloadService always creates the tmp file first before trying to download and this will result in incomplete/lingering files. We should clean these up if blob downloads fail for any reason.

IPOPP requires a large amount of disk space for storage and performance. Add a 1TB dataDisk under storageProfile.

Currently we rely on cron jobs to check a storage account for new blobs on a regular cadence. It would be ideal to have a service that listened for Event Grid events via Service Bus that would then download the blob for processing. Also if we were to use this strategy, we could trigger resources to startup and start processing, allowing resources to be deallocated when not in use.

With the recent changes to add Event Grid and Service Bus for contact data notifications to Bicep, we need to bring the az cli deployment piece up to parity so that it provides the same deployment experience.

Currently BlobDownloadService supports only queues for the Service Bus receiver. We should also support topics.

DoD:

• Note that user must create SAS and will need to rotate it prior to expiration.