We want to publish the pyPI package indicating it is a alpha release for now.

The Version specifiers based on (PEP 440) see Python Packaging User Guide requires us to use a releases that MUST be numbered in a consistently increasing fashion, otherwise automated tools will not be able to upgrade them correctly.

As such we would like to publish for now in a X.YaN which indicates an alpha release and keep options open to adopt semantic versioning, etc. later one.

The easiest and a consistent approach is to use 0.0.0.a${{ github.run_number}} for now, and evolve from there.

Parent Task: #22
Goal: Get acquainted with the framework, test it and provide feedback.

Success criteria:

• Test the framework with the examples provided in the repo.
• Get acquainted with the code
• Provide feedback (document it in the issue) and create issues to address items

Fabric's git integration and workspace flow is still evolving. We need to add some guideance for setup and link potential Microsoft docs to enable the user to understand the workflow using PRs and Azure Devops.

We can separate the work in the following groups. Please order by amount of work and likelihood of usage:

Goal: improve the guidance, by adding a step-by-step on the things that need to be considered when starting to test a new pipeline.

Success criteria:

• Add guidance (might include visuals) on how to approach the framework for new pipelines (that are not in the examples)
• Add more examples to include often used tasks like the Copy activity for example

Read current ReadMe e2e and improve quality of the guidance.

Add separated md when necessary.

Goal: Investigate how to publish official Microsoft NuGets and if possible find people that had experienced the process.

Success Criteria:

• Identify official documentation to publish Microsoft NuGets that follow best practices and guidelines
• Identify potential point of contact in the internal team to ask questions when needed

Hi, project descriptions says:

A stand-alone test framework that allows to write unit tests for Data Factory pipelines on Microsoft Fabric and Azure Data Factory.

Are you thinking about adding support for Synapse Pipelines as well?

Create and describe steps for PR creation for the repo.

Fabric's git integration stores pipeline-content.json currently as UTF-16-LE.
The testing framework fails to read these fails as:

opens file with r and thus tries to utf-8 encode.

System variables are not replaced upon evaluation (https://learn.microsoft.com/en-us/azure/data-factory/control-flow-system-variables).

Requirements:

• Build a new implementation of BaseExpression that detects and evaluates system variables
• Add system variables to the state

Nice to have:

• Improve expression detection and replacement mechanism

The current grammar does not allow to access complex objects being returned by:

• parameters
• variables
• item()

Goal: create clarity on the ReadMe on expression validation

Success Criteria:
[ ] Add in the README the assumption that the json file is valid
[ ] Add note on npm package can be used on the CI pipelines to do that validation
[ ] Remove expression example on the README file

The current Github workflow builds and publishes within a single job (and thus creates a deployment on PR workflows).

This repository is currently missing a LICENSE file.

A license helps users understand how to use your project in a compliant manner. You can find the standard MIT license Microsoft uses at: https://github.com/microsoft/repo-templates/blob/main/shared/LICENSE.

If you would like to learn more about open source licenses, please visit the document at https://aka.ms/license (Microsoft-internal guidance).

Parent Task: #22
Goal: Get acquainted with the framework, test it and provide feedback.

Success criteria:

• Test the framework with the examples provided in the repo.
• Get acquainted with the code
• Provide feedback (document it in the issue) and create issues to address items

Goal: Allows to validate whether different pipelines work well together

Success criteria:

• Can be enabled or disabled on a per test basis
• If enabled, upon evaluation of an ExecutePipelineActivity by the TestFramework, it returns activities of the referenced child pipeline
• Execution of child pipeline should have its own scope

Goal: investigate PyPi package publication through Microsoft organization.

Success criteria:
[ ] Find documentation and guidelines to publish the PyPi package to Microsoft organization
[ ] Define next steps based on investigation

Goal: Create Fabric pipelines to use as examples in the repo

Success criteria:
[ ] Create Fabric pipelines (minimum 2) with typical activities to be used as examples in the repo

• Share with Data champs so that they can look out for us
• Check the engagement pipeline to spot Data Factory, Synapse and perhaps even Fabric pipelines

Parent Task: #22

Goal: Get acquainted with the framework, test it and provide feedback.

Success criteria:

• Test the framework with the examples provided in the repo.
• Get acquainted with the code
• Provide feedback (document it in the issue) and create issues to address items

Parent Task: #22
Goal: Get acquainted with the framework, test it and provide feedback.

Success criteria:

• Test the framework with the examples provided in the repo.
• Get acquainted with the code
• Provide feedback (document it in the issue) and create issues to address items

Goal: Create Fabric pipelines to use as examples in the repo

Success criteria:
[ ] Create Fabric pipelines with typical activities to be used as examples in the repo

I have a pipeline with some expressions containing formatDateTime function as shown below and always get an ExpressionParsingError exception:

"File_Name": { "value": "@concat(item().TABLESPACE_NAME, '__', item().TABLE_NAME,'.v', formatDateTime(utcnow(), 'yyyyMMdd_HHmmss'), '.parquet')", "type": "Expression" }

Looks just like a typo in the framework code, there I find formatDataTime instead of formatDateTime.

P.S.: Very nice framework you created here for ADF testing. Thanks 👍

Pre-commit hooks fail on commitlint:

commitlint...............................................................Failed
- hook id: commitlint
- exit code: 1

⧗   input: refactor: make functions internal
✖   Please add rules to your `commitlint.config.js`
    - Getting started guide: https://commitlint.js.org/#/?id=getting-started
    - Example config: https://github.com/conventional-changelog/commitlint/blob/master/%40commitlint/config-conventional/index.js [empty-rules]

✖   found 1 problems, 0 warnings
ⓘ   Get help: https://github.com/conventional-changelog/commitlint/#what-is-commitlint

Related issue alessandrojcm/commitlint-pre-commit-hook#63

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Goal: The goal of this spike is to check how feasible it is to convert the framework to Python and investigate what the different options are

Outcome:

• What's the expected effort to convert this framework to Python?
• Can we re-use reflection and generic-like features within Python? If not, what does this mean?
• Can we generate models with dfe (data factory expression) types?

Currently we accept any type of value in test_framework.evaluate_pipeline(...). However, supported are only primitives and dict.

If the user, for example passes:

state = [
RunParameter(
    RunParameterType.Pipeline, "Example", {},
)
]

vs

state = [
RunParameter(
    RunParameterType.Pipeline, "Example", {}
)
]

python constructs a tuple and not a dict, causing error in the evaluation mechanism.

Batch Job example for Fabric link is broken

Add devcontainer capability to the repository for examples

Hello,

First time opening a bug, so please share any feedback if something is not right.

I installed the library from PyPi both locally and on a Databricks cluster with no issues.
pip install data-factory-testing-framework

When I'm trying to import TestFramework, I get the following error both locally and on ADB.
from data_factory_testing_framework import TestFramework
ImportError: cannot import name 'TestFramework' from 'data_factory_testing_framework' (/opt/homebrew/lib/python3.11/site-packages/data_factory_testing_framework/__init__.py)

If I run something like import data_factory_testing_framework it doesn't complain.

Not sure how to proceed.

Thanks.

Add best practices for the framework usage in the Readme

Goal: describe the testing terminology assumed in this repo: unit testing and functional testing in the context of data pipelines.

Success criteria:

• Add testing terminology to the main ReadMe.

Goal: Get acquainted with the framework, test it and provide feedback.

Success criteria:

• Test the framework with the examples provided in the repo.
• Get acquainted with the code
• Provide feedback (document it in the issue) and create issues to address items

Example should only evaluate left hand side if lateUpdatedDatalake is '':
@or(equals(item().lastUpdatedDatalake,''),greaterOrEquals(item().lastUpdatedSourceSystem,addSeconds(item().lastUpdatedDatalake,-1)))