From @sabbyanandan on May 6, 2016 22:32

From @sabbyanandan on November 20, 2015 20:0

As a developer, I'd like to move gpload module from XD to s-c-t-m repo, so I can use it as task to build batch pipelines.

Acceptance:

• gpload as task module (Spring Boot App) is available in spring-cloud-task-modules repo
• maven coordinates include gpload with respective group/artifact/version

Copied from original issue: spring-cloud/spring-cloud-task-modules#7

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#12

Identify the environment and resources necessary to run Composed task runner
Run Acceptance tests in CF
Record results and notes on what was found.

Currently if the child task being executed by the TaskLauncherTasklet returns a non zero return code it considers the ChildTask execution a failure. This is not always the case. Users should be able to add a property to the CTR specifying a comma delimited set of legal exit codes that can be returned by the tasks.

If a user attempts to set the command line properties property in the tasklaunchrequest-transform using the --properties option at stream deployment time, the CTR fails to process any entry that contains a --. For example in the following stream:
time --fixed-delay=30 | tasklaunchrequest-transform --uri=file:///Users/glennrenfro/project/spring-cloud-task-app-starters/composed-task-runner/apps/composedtaskrunner-task/target/composedtaskrunner-task-1.1.0.BUILD-SNAPSHOT.jar | task-launcher-local

if the user tries to add an argument with the --
prefix in the command-line-args like the following:

stream deploy doodle3 --properties "app.tasklaunchrequest-transform.command-line-arguments='--graph=taskfun-AAA&&taskfun-BBB --increment-instance-enabled=true --spring.datasource.url=jdbc:mariadb://localhost:3306/practice --spring.datasource.username=root --spring.datasource.password=password --spring.datasource.driverClassName=org.mariadb.jdbc.Driver' ,app.tasklaunchrequest-transform.uri=file:///Users/glennrenfro/project/spring-cloud-task-app-starters/composed-task-runner/apps/composedtaskrunner-task/target/composedtaskrunner-task-1.1.0.BUILD-SNAPSHOT.jar"

we get this stack track:

org.springframework.cloud.dataflow.core.dsl.CheckPointedParseException: 168E:(pos 25): arguments not allowed unless parser is in app mode
taskfun-AAA&&taskfun-BBB --increment-instance-enabled=true --spring.datasource.url=jdbc:mariadb://localhost:3306/practice --spring.datasource.username=root --spring.datasource.password=password --spring.datasource.driverClassName=org.mariadb.jdbc.Driver
                        *^

	at org.springframework.cloud.dataflow.core.dsl.Tokens.raiseException(Tokens.java:276) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.dataflow.core.dsl.TaskParser.eatTaskApp(TaskParser.java:215) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.dataflow.core.dsl.TaskParser.parseTaskNode(TaskParser.java:163) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.dataflow.core.dsl.TaskParser.parseFlow(TaskParser.java:173) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.dataflow.core.dsl.TaskParser.parseTaskNode(TaskParser.java:166) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.dataflow.core.dsl.TaskParser.eatSequences(TaskParser.java:115) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.dataflow.core.dsl.TaskParser.parse(TaskParser.java:89) ~[spring-cloud-dataflow-core-1.2.3.RELEASE.jar!/:1.2.3.RELEASE]
	at org.springframework.cloud.task.app.composedtaskrunner.StepBeanDefinitionRegistrar.getTaskApps(StepBeanDefinitionRegistrar.java:132) ~[spring-cloud-starter-task-composedtaskrunner-1.1.0.BUILD-SNAPSHOT.jar!/:1.1.0.BUILD-SNAPSHOT]
	at org.springframework.cloud.task.app.composedtaskrunner.StepBeanDefinitionRegistrar.registerBeanDefinitions(StepBeanDefinitionRegistrar.java:55) ~[spring-cloud-starter-task-composedtaskrunner-1.1.0.BUILD-SNAPSHOT.jar!/:1.1.0.BUILD-SNAPSHOT]
	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsFromRegistrars(ConfigurationClassBeanDefinitionReader.java:359) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:143) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitions(ConfigurationClassBeanDefinitionReader.java:116) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:320) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:228) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:270) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:93) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:687) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:525) ~[spring-context-4.3.10.RELEASE.jar!/:4.3.10.RELEASE]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693) [spring-boot-1.5.6.RELEASE.jar!/:1.5.6.RELEASE]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360) [spring-boot-1.5.6.RELEASE.jar!/:1.5.6.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) [spring-boot-1.5.6.RELEASE.jar!/:1.5.6.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.6.RELEASE.jar!/:1.5.6.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.6.RELEASE.jar!/:1.5.6.RELEASE]
	at org.springframework.cloud.task.app.composedtaskrunner.ComposedtaskrunnerTaskApplication.main(ComposedtaskrunnerTaskApplication.java:29) [classes!/:1.1.0.BUILD-SNAPSHOT]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_144]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_144]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [composedtaskrunner-task-1.1.0.BUILD-SNAPSHOT.jar:1.1.0.BUILD-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [composedtaskrunner-task-1.1.0.BUILD-SNAPSHOT.jar:1.1.0.BUILD-SNAPSHOT]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [composedtaskrunner-task-1.1.0.BUILD-SNAPSHOT.jar:1.1.0.BUILD-SNAPSHOT]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [composedtaskrunner-task-1.1.0.BUILD-SNAPSHOT.jar:1.1.0.BUILD-SNAPSHOT]

User wants an app-starter that can consume a Composed Task DSL and execute the composed task.

From @sabbyanandan on May 6, 2016 22:32

From @sabbyanandan on November 20, 2015 20:2

As a developer, I'd like to move filepollhdfs module from XD to s-c-t-m repo, so I can use it as task to build batch pipelines.

Acceptance:

• filepollhdfs as task module (Spring Boot App) is available in spring-cloud-task-modules repo
• maven coordinates include filepollhdfs with respective group/artifact/version

Copied from original issue: spring-cloud/spring-cloud-task-modules#10

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#14

To get started using the task app starters, just need to download them and run.

Should link to the ref guide for building from scratch or creating new ones.

Tests currently fail after Parser Update.

The parser was updated to give more detailed error messages for labels.

We can add the Microsoft SQL Server JDBC driver to the build for individual task apps. There is a MIT licensed Maven artifact now: com.microsoft.sqlserver:mssql-jdbc :6.1.0.jre8

As a developer, I'd like to update Spring Boot (1.5.4), Spring Cloud (Dalston.SR1), and SCT (1.2.1) to latest GA releases.

Acceptance:
All the core dependencies are up-to-date.

From @ericbottard on October 13, 2016 14:18

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#66

• Switch master to 2.1
• Upgrade to Spring Cloud 2.1
• Deal with breaking changes

Currently when running a ComposedTaskRunner there are cases where a task execution fails before it starts on CF. In these cases the CTR waits forever (using defaults). We should be able to stop the CTR through the UI vs going to the CF Console and terminating the execution.

As a developer, I'd like to add Spring Boot 2.0 compatibility for the OOTB apps.

Acceptance:

• Ensure Spring Boot, Spring Cloud, Spring Cloud Task and the ecosystem is on 2.0 line
• Ensure README/docs are up-to-date
• Ensure tests run successfully

Targeted Apps:

• timestamp @cppwfs PR Submited
• timestamp-batch @sobychacko PR Submited
• composed-task-runner @cppwfs PR Submited
  - [ ] jdbchdfs-local @jvalkeal
  - [ ] spark-client
  - [ ] spark-cluster
  - [ ] spark-yarn

As a developer, I'd like to refactor applicaion READMEs for clarity.

Add Build section

== Build
$ ./mvnw clean install -PgenerateApps
$ cd apps

You can then cd into one one of the folders and
build it:

$ ./mvnw clean package

Notes:

• Whatever we add/polish at each READMEs would directly feed the SCDF reference-docs stories targeted for 1.2.3.RELEASE.
• it is an anti-goal to include SCDF DSL samples in each README

Targeted Apps:

• timestamp - Glenn
• timestamp-batch - Glenn
• jdbchdfs-local - Glenn
• composed-task-runner - Glenn
• spark-client - Thomas
• spark-cluster - Thomas
• spark-yarn - Thomas

From @trisberg on June 1, 2016 13:57

To be consistent with other projects we should use tab characters for indentation in POM and XML docs

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#31

From @sabbyanandan on May 6, 2016 22:32

From @sabbyanandan on November 20, 2015 20:1

As a developer, I'd like to move hdfsmongodb module from XD to s-c-t-m repo, so I can use it as task to build batch pipelines.

Acceptance:

• hdfsmongodb as task module (Spring Boot App) is available in spring-cloud-task-modules repo
• maven coordinates include hdfsmongodb with respective group/artifact/version

Copied from original issue: spring-cloud/spring-cloud-task-modules#8

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#13

As a developer, I'd like to update the relevant BOM dependencies and prep the Celcius release-train for RC release.

Doesn't need the un-necessary coupling with the BOM in core.

For demo and testing purposes we need a simple batch app that can be executed without requiring of other resources.

It looks like https://github.com/spring-cloud/spring-cloud-task-app-starters/tree/master/spring-cloud-task-jdbchdfs-common wasn't migrated from the old repository.

CTR should pick up at failed step upon a restart

From @sabbyanandan on May 6, 2016 22:32

From @sabbyanandan on February 16, 2016 15:45

As a developer, I'd like to add Sqoop docs, so it can be used as a reference for anyone wanting to run Sqoop jobs.

Acceptance:

• Ref. guide includes sqoop section
• Example included

Copied from original issue: spring-cloud/spring-cloud-task-modules#26

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#15

When using TaskLauncher, if the user has specified an artifact that is not in the local maven repository the aether does not pull it from a parent repository.

As a developer, I'd like to create a JAR artifact for each task app-starter, so I embed the application metadata and the whitelisted properties in it.

Acceptance:

• the JAR artifact is automatically generated at the build time
• the companion artifact can be used by other clients such as SCDF's shell, dsl and UI
• unit-tests verify the creation and existence of the companion artifact for each task app-starter

As a developer, I'd like to add an OOTB s3jdbc task application.

Following artifacts/apps need to be released as part of Belmont RC1 release train.

• app starters core - 1.2.0.RC1
• timestamp - 1.2.0.RC1
• jdbchdfs-local - 1.2.0.RC1
• composed-task-runner - 1.0.0.RC1
• spark-client - 1.2.0.RC1
• spark-cluster- 1.2.0.RC1
• spark-yarn- 1.2.0.RC1
• app-starters-release - Belmont.RC1

From @sabbyanandan on May 6, 2016 22:31

From @sabbyanandan on November 20, 2015 20:0

As a developer, I'd like to move ftphdfs module from XD to s-c-t-m repo, so I can use it as task to build batch pipelines.

Acceptance:

• ftphdfs as task module (Spring Boot App) is available in spring-cloud-task-modules repo
• maven coordinates include ftphdfs with respective group/artifact/version

Copied from original issue: spring-cloud/spring-cloud-task-modules#6

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#11

Update:

• SCT 2.1 GA
• Spring Cloud Build - Greenwich

From @sabbyanandan on May 6, 2016 22:30

From @sabbyanandan on November 20, 2015 19:58

As a developer, I'd like to move hdfsjdbc module from XD to s-c-t-m repo, so I can use it as task to build batch pipelines.

Acceptance:

• hdfsjdbc as task module (Spring Boot App) is available in spring-cloud-task-modules repo
• maven coordinates include hdfsjdbc with respective group/artifact/version

Copied from original issue: spring-cloud/spring-cloud-task-modules#3

Copied from original issue: spring-cloud/spring-cloud-task-app-starters#9