DataWeave CLI

DataWeave CLI is a command-line interface that allows querying, filtering, and mapping structured data from different data sources like JSON, XML, CSV, YML to other data formats. It also allows to easily create data in such formats, all through the DataWeave language. For example:

dw 'output json --- { message: ["Hello", "world"] joinBy " "}'

The DataWeave language is in the process of being open-sourced. You can read our announcement here. Our journey has just begun and it will take some time for the code to be available. In the meantime, we want to start engaging with our community to understand how DataWeave could be used and integrated.

If you are interested on leveraging DataWeave:

Join our community Slack
Join the #opensource channel

For more news and all things DataWeave, visit our site

What is Included?

The binary distribution already ships with a set of modules and data formats that makes this useful for a very interesting and broad set of use cases.

Included Modules

DataWeave Standard Library

Supported Data Formats

MIME Type	ID	Supported Formats
`application/csv`	`csv`	CSV Format
`application/json`	`json`	JSON Format
`application/octet-stream`	`binary`	Binary Format
`application/xml`	`xml`	XML Format
`application/x-ndjson`	`ndjson`	Newline Delimited JSON Format (ndjson)
`application/x-www-form-urlencoded`	`urlencoded`	URL Encoded Format
`application/yaml`	`yaml`	YAML Format
`multipart/form-data`	`multipart`	Multipart Format
`text/plain`	`text`	Text Plain Format
`text/x-java-properties`	`properties`	Text Java Properties

Installation

Homebrew (Mac)

brew tap mulesoft-labs/data-weave
brew install dw

Manual Installation

Download the latest release version according to your OS.
Unzip the file on your <user.home>/.dw
Add <user.home>/.dw/bin to your PATH

Build and Install

To build the project, you need to run gradlew with the graalVM distribution based on Java 11. You can download it at https://github.com/graalvm/graalvm-ce-builds/releases Set:

export GRAALVM_HOME=`pwd`/.graalvm/graalvm-ce-java11-22.3.0/Contents/Home
export JAVA_HOME=`pwd`/.graalvm/graalvm-ce-java11-22.3.0/Contents/Home

Execute the gradle task nativeCompile

./gradlew native-cli:nativeCompile

It takes several minutes so good time to take and refill your mate.

Once it finishes you will find the dw binary in native-cli/build/native/nativeCompile/dw

How to Use It

If the directory containing the dw executable is in your PATH, you can run dw from anywhere.

If it is not, go to the bin directory referenced in the installation instructions and run dw from there.

The following example shows the DataWeave CLI documentation

dw --help

usage: dw [--eval] [-f <file-path>] [--help] [-i <input-name input-path>]
       [--migrate <dw1-file-path>] [-o <output-path>] [-p <param-name
       param-value>] [--privileges <privileges>] [--silent]
       [--untrusted-code] [-v] [--version] [--add-wizard <wizard-name>]
       [--list-spells] [--local-spell <spell-folder>] [--new-spell
       <spell-name>] [-s <spell-name>] [--update-grimoires]


.........................................................................
.%%%%%....%%%%...%%%%%%...%%%%...%%...%%..%%%%%%...%%%%...%%..%%..%%%%%%.
.%%..%%..%%..%%....%%....%%..%%..%%...%%..%%......%%..%%..%%..%%..%%.....
.%%..%%..%%%%%%....%%....%%%%%%..%%.%.%%..%%%%....%%%%%%..%%..%%..%%%%...
.%%..%%..%%..%%....%%....%%..%%..%%%%%%%..%%......%%..%%...%%%%...%%.....
.%%%%%...%%..%%....%%....%%..%%...%%.%%...%%%%%%..%%..%%....%%....%%%%%%.
.........................................................................

    --eval                                 Executes the script but it
                                           doesn't use the writer. This is
                                           useful when launching a
                                           webserver.
 -f,--file <file-path>                     Specifies the DataWeave file
                                           path to execute.
    --help                                 Shows the help.
 -i,--input <input-name input-path>        Declares a new input.
    --migrate <dw1-file-path>              Migrates a DW1 file to DW2 and
                                           outputs the result.
 -o,--output <output-path>                 Specifies output file for the
                                           transformation if not standard
                                           output will be used.
 -p,--parameter <param-name param-value>   Parameter to be passed. All
                                           input parameters are accessible
                                           through the variable `params`
                                           of type object.
    --privileges <privileges>              A comma separated set of the
                                           privileges for the script
                                           execution.
    --silent                               Executes the script in silent
                                           mode, where all info messages
                                           is not going to be shown.
    --untrusted-code                       Run the script as untrusted,
                                           which means that the script has
                                           no privileges.
 -v,--verbose                              Enable verbose mode.
    --version                              The version of the CLI and
                                           Runtime.
    --add-wizard <wizard-name>             [Experimental] Downloads wizard
                                           grimoire so that its spell are
                                           accessible.
    --list-spells                          [Experimental] List all the
                                           available spells.
    --local-spell <spell-folder>           [Experimental] Executes a local
                                           folder spell.
    --new-spell <spell-name>               [Experimental] Create a new
                                           spell.
 -s,--spell <spell-name>                   [Experimental] Runs a spell.
                                           Use the <spellName> or
                                           <wizard>/<spellName> for spells
                                           from a given wizard.
    --update-grimoires                     [Experimental] Update all
                                           wizard grimoires.


 Example:

 dw -i payload <fullPathToUser.json> "output application/json --- payload
filter (item) -> item.age > 17"

 Documentation reference:

 https://docs.mulesoft.com/dataweave/latest/

DataWeave CLI Environment Variables

The following are the DataWeave CLI environment variables that you can set in your operating system:

Environment Variable	Description
`DW_HOME`	The directory where the home will be found if not defined `~/.dw` will be used.
`DW_DEFAULT_INPUT_MIMETYPE`	The default `mimeType` that is going to be used for the standard input. If not defined `application/json` will be used.
`DW_DEFAULT_OUTPUT_MIMETYPE`	The default output `mimeType` that is going to be if not defined. If not defined `application/json` will be used.

Dependency Manager

In order for a spell to depend on a library it can include a library it can use the dependencies.dwl to specify the list of dependencies that it should be included and download

%dw 2.0
var mavenRepositories = [{
    url: "https://maven.anypoint.mulesoft.com/api/v3/maven"
}]
---
{
  dependencies: [
    {
      kind: "maven",
      artifactId: "data-weave-analytics-library",
      groupId: "68ef9520-24e9-4cf2-b2f5-620025690913",
      version: "1.0.1",
      repositories: mavenRepositories // By default mulesoft, exchange and central are being added
    }
  ]
}

Querying Content From a File

Giving the following input file users.json

[
  {
    "name": "User1",
    "age": 19
  },
  {
    "name": "User2",
    "age": 18
  },
  {
    "name": "User3",
    "age": 15
  },
  {
    "name": "User4",
    "age": 13
  },
  {
    "name": "User5",
    "age": 16
  }
]

Let's query users old enough to drink alcohol:

dw -i payload <fullpathToUsers.json> "output application/json --- payload filter (item) -> item.age > 17"

Output

[
  {
    "name": "User1",
    "age": 19
  },
  {
    "name": "User2",
    "age": 18
  }
]

Query Content From Standard Input

cat <fullpathToUser.json> | dw "output application/json --- payload filter (item) -> item.age > 17"

Redirecting the Output to a File

dw "output application/xml --- users: {( 1 to 100 map (item) -> {user: "User" ++ item} )}" >> out.xml

CURL + DataWeave => Power API Playground

An interesting use case for the DataWeave CLI is to combine it with curl

Query a GET Response

We can use the GitHub API to query commits of a repository.

We can easily get the first commit by doing:

curl "https://api.github.com/repos/mulesoft/mule/commits?per_page=5" | dw "payload[0]"

or we can get the message by doing:

curl "https://api.github.com/repos/mulesoft/mule/commits?per_page=5" | dw "{message: payload[0].commit.message}"

Generate a Request with Body

This example uses the jsonplaceholder API to update a resource.

Steps:

Search the post resource with the id = 1.
Use DataWeave CLI to create a JSON output changing the post title My new title.
Finally, update the post resource.

curl https://jsonplaceholder.typicode.com/posts/1 | dw "output application/json --- { id: payload.id, title: 'My new title', body: payload.body, userId: payload.userId }" | curl -X PUT -H "Content-type: application/json; charset=UTF-8" -T "/dev/stdin" https://jsonplaceholder.typicode.com/posts/1 -v

Output

{
  "id": 1,
  "title": "My new title",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto",
  "userId": 1
}

Using parameters

Using the internal map params, we can access injected parameters in the command line with the -p option

dw -p myName Julian "output json --- { name : params.myName }"

Output

{
  "name": "Julian"
}

Contributions Welcome

Contributions to this project can be made through Pull Requests and Issues on the GitHub Repository.

Before creating a pull request review the following:

When you submit your pull request, you are asked to sign a contributor license agreement (CLA) if we don't have one on file for you.

omkarium / data-weave-cli Goto Github PK

data-weave-cli's Introduction

DataWeave CLI

What is Included?

Included Modules

Supported Data Formats

Installation

Homebrew (Mac)

Manual Installation

Build and Install

How to Use It

DataWeave CLI Environment Variables

Dependency Manager

Querying Content From a File

Output

Query Content From Standard Input

Redirecting the Output to a File

CURL + DataWeave => Power API Playground

Query a GET Response

Generate a Request with Body

Output

Using parameters

Output

Contributions Welcome

data-weave-cli's People

Contributors

Recommend Projects

Recommend Topics

Recommend Org