marklogic / ml-gradle Goto Github PK

View Code? Open in Web Editor NEW

72.0 31.0 78.0 2.73 MB

Gradle plugin for automating everything involving MarkLogic

License: Other

Groovy 97.36% Java 2.64%

gradle corb marklogic gradle-plugin example examples sample samples mark logic

ml-gradle's People

Contributors

Stargazers

Watchers

Forkers

nikos lewisap mariannemyers daldei xquery sastafford grechaw hansenmc luiz-n ashutosha tzejohn derms ebenhabi jmakeig ucal sebastiankalinowski sanderson-sfdc ericputnam divino jmeekhof aclavio miguelrgonzalez millerbill3 patrickmcelwee jamesagardner vivekmuniyandi jkerr5 arulsiva21 kghmanuel iowusu davidbcollins praneet18 freshie ehennum stevenbrockman xtrmwyz lukasz1234567890 ryanjdew bradmann mthahir akshaysonvane florismeijer jonesyface leond08 sulthan007 aajacobs prestonmcgowan tomasz9248 lifan0127 omeshkumaryadav dmcassel haritrichy instructorjason msimons joseespinoza anu3990 bgeorge11 sfouzi khadijaqaraqe ramavala halftermeyer pam2202 lgh521 brent-hartwig mandlavasavi20182018 ruchika-12 vnayfeld ajith0888 abhinavmishra14 virudwivedi peetkes billfarber tsv19su254052 4vservices airkhan41 ramgopalp9 ronaksomaiya

ml-gradle's Issues

Provide example of converting Roxy project to ml-gradle

Add task for deploying groups

Need to add DeployGroupsCommand to mlDeploy too

Configure security resources

Should use ml-app-deployer to configure users/roles/amps/etc. This should happen as part of mlDeploy.

Updating amp fails

Having issue at Boeing with this - URL looks okay, but getting a 404, and the modules database is in fact there

Sample project database configuration

The sample project has an example CPF pipeline but the sample content database "maintain last modified" property is set to FALSE. This should default to TRUE so CPF will work correctly.

gradle mlWatch not working

Error: Could not find or load main class com.marklogic.client.modulesloader.ModulesWatcher

WatchTask.groovy line 16

wrong classpath referenced

Deploy all servers

Need a task for updating all servers - or at least all the non-REST API ones, but might as well include that one too.

Provide easy way of configuring replica forests

Task should utilize ConfigureReplicaForestsCommand in ml-app-deployer

build failed : Unsupported major.minor version 51.0

Hi Rob,

not sure if this is the right place to log this but I have the obi-ui ez app and want to install in on a vm.

I used Geert Josten his vagrant file to create a ml8 vm...

now Itry to do ./gradlew mlinstall and bump into this :

FAILURE: Build failed with an exception.

What went wrong:
A problem occurred configuring root project 'obi-ui'.

Could not open no_buildscript class cache for build file '/space/projects/obi/obi-ui/build.gradle' (/home/vagrant/.gradle/caches/2.1/scripts/build_2tv2dt3jfe4uuce4od0jn6fp28/ProjectScript/no_buildscript).
com/marklogic/gradle/task/MlcpTask : Unsupported major.minor version 51.0
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 37.079 secs
[vagrant@obi-ml8-ml1 obi-ui]$

please advise

Added redeploy task

mlRedeploy = mlDeploy + mlReloadModules (or possibly vice versa)

docs should reference latest version in gradle dependencies

buildscript {
  repositories {
    mavenCentral()
    maven {url "http://developer.marklogic.com/maven2/"}
    maven {url "http://rjrudin.github.io/marklogic-java/releases"}
  }
  dependencies {
    classpath "com.marklogic:ml-gradle:2.0-alpha-4"
  }
}

should be

buildscript {
  repositories {
    mavenCentral()
    maven {url "http://developer.marklogic.com/maven2/"}
    maven {url "http://rjrudin.github.io/marklogic-java/releases"}
  }
  dependencies {
    classpath "com.marklogic:ml-gradle:2.0a8"
  }
}

Document how to use CreateTriggersTask

Can improve this task too, at least by using getDefaultXccUrl

Exclude URIs when clearing a modules database

Needed when a project has an mlRestApi dependency and a custom rewriter; when it does an mlReloadModules, it will remove the custom rewriter and then not be able to load any of the modules in the mlRestApi dependency folder.

Application-level auth not supported

Some of the gradle tasks (e.g., mlLoadModules) aren't supported with application-level authentication on the appserver. The workaround currently is to give the default user the rest-admin role, but it would be nice to have ml-gradle handle this type of server configuration automatically.

Need support for MLCP streaming

See here: https://docs.marklogic.com/guide/ingestion/content-pump#id_16000

I need to be able to specify -streaming

I would consider a pull request but I'm not 100% sure how to test it.

Execute a corb gradle task type

As a developer
I want to TITLE
So that it is easier to execute a corb job. Right now I have to run a JavaExec task which forces me to remember all the corb parameters. This would help new ml-gradle users run a corb job.

getting sample project to work ...

I downloaded ml_gradle and ran gradle mlDeploy and it worked perfectly (the databases were created etc) so to go further, following the docs I cloned sample-project did the same thing and its failing ...

Any ideas ?
THis is running against a 8.0.1 release build of ML.

------> Not working
DLEE@DLEE-HP /cygdrive/c/Work/MarkLogic/Projects/mlui/gradle/test1/sample-project
$ gradle mlInstalLPackages
App name: sample-project
App host: vmnex
App username: admin
App REST port: 8100
App XDBC port: 8101
Manage app host: vmnex
Manage app username: admin
Couldn't find local.gradle; you can create this gitignore'd file to override properties if necessary
:mlPrepareRestApiDependencies
No mlRestApi dependencies found, so no preparation of dependencies required

:mlInstallPackages
Installing database and appserver packages
GET request for "http://vmnex:8002/manage/v2/packages/sample-project-package" resulted in 404 (Not Found); invoking error handler
POST request for "http://vmnex:8002/v1/rest-apis" resulted in 400 (Bad Request); invoking error handler
:mlInstallPackages FAILED

FAILURE: Build failed with an exception.

What went wrong:
Execution failed for task ':mlInstallPackages'.

400 Bad Request
Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 12.299 secs

DLEE@DLEE-HP /cygdrive/c/Work/MarkLogic/Projects/mlui/gradle/test1/sample-project

---> Working

DLEE@DLEE-HP /cygdrive/c/Work/MarkLogic/Projects/mlui/gradle/test1
$ gradle mlInstalLPackages
App name: quick-start
App host: vmnex
App username: admin
App REST port: 8200
App XDBC port: 8201
Manage app host: vmnex
Manage app username: admin
:mlPrepareRestApiDependencies
No mlRestApi dependencies found, so no preparation of dependencies required

:mlMergeDatabasePackages
No mergePackageFilePaths specified, will not produce a new database package file
:mlMergeHttpServerPackages
No mergePackageFilePaths specified, will not produce a new HTTP server package file
:mlInstallPackages

Deleting package with quick-start-package
Sending a 'DELETE' to 'http://vmnex:8002/manage/v2/packages/quick-start-package'
Sending a 'GET' to 'http://vmnex:8002/manage/v2/packages/quick-start-package?format=json'

Creating package quick-start-package
Sending a 'POST' to 'http://vmnex:8002/manage/v2/packages?format=json&pkgname=quick-start-package'
No content database package file found, so not installing a content database

Checking to see if REST API with name quick-start exists...
Sending a 'GET' to 'http://vmnex:8002/v1/rest-apis/quick-start'
Creating new REST API server: {rest-api: {name: "quick-start", database: "quick-start-content", port: 8200}}
Sending a 'POST' to 'http://vmnex:8002/v1/rest-apis'

Adding server quick-start-content-xdbc in group Default to package quick-start-package
Sending a 'POST' to 'http://vmnex:8002/manage/v2/packages/quick-start-package/servers/quick-start-content-xdbc?group-id=Default'

Installing package quick-start-package
Sending a 'POST' to 'http://vmnex:8002/manage/v2/packages/quick-start-package/install?format=json'
Finished installing application quick-start

BUILD SUCCESSFUL

Total time: 19.348 secs

DLEE@DLEE-HP /cygdrive/c/Work/MarkLogic/Projects/mlui/gradle/test1
$

Check for updates and tell the user if a newer ml-gradle exists

Not sure how this gets done, but I've seen ml-gradle have about 3 new builds since I started using it 2 weeks ago. An automagic way to check and tell me would be so awesome.

Restart cluster

Based on Jim's tip:

curl -v -X POST --anyauth --user admin:admin --header "Content-Type:application/json" -d'{"operation": "restart-local-cluster"}' http://127.0.0.1:8002/manage/v2

Should be "mlRestartCluster"

Supporting loading data

As part of integrating into samplestack, need a replacement for MarkLogicSlurpTask. This loads documents via the REST API, which is one option - which also includes support for loading triple files.

Can do REST API initially, though mlcp will of course scale better.

Fix ordering of deploying amps

This needs to happen much later in the cycle, as the modules database referenced by an app must exist, else you get a 404

Add sample project that only shows mlcp being used

This will be a good example of easing mlcp usage without all the other ml-gradle bells and whistles.

Generate project structure

Can't use https://docs.gradle.org/current/userguide/init_scripts.html, as that always runs. Could instead have a task in ml-gradle for initializing a project - e.g. mlProjectInit. That could then do the following:

Don't need to touch build.gradle, which will already depend on ml-gradle
Stub out src/main/ml-config with a database file and a REST API file, and an app role and an app user
Stub out src/main/ml-modules with an /ext/(mlAppName) directory with a sample module, and then a basic search options file, a basic transform, and a basic service

ml-gradle init messages should be quiet if not running ml-gradle

when not running an ml-gradle task, consider suppressing gradle init messages.

example of console trace;

Initializing ml-gradle
App name: quick-start
App host: localhost
App username: admin
App REST port: 8200
Manage host: localhost
Manage username: admin
Admin host: localhost
Admin username: admin
Initializing ManageClient with manage config of: [host: localhost, port: 8002, username: admin]
Initialized ManageClient with base URL of: http://localhost:8002
Finished initializing ml-gradle

rationalise targets

careful target naming in gradle reduces cognitive load.

for example the current list of targets are ;

mlAppDeploy
mlAppUndeploy
mlClearContentDatabase
mlClearModules
mlCpfDeploy
mlCpfLoadDefaultPipelines
mlCreateResource
mlCreateTransform
mlDeleteModuleTimestampsFile
mlDeploy
mlInit
mlInstallAdmin
mlLoadModules
mlPostDeploy
mlPostUndeploy
mlPrepareRestApiDependencies
mlReloadModules
mlScaffold
mlUndeploy
mlUpdateContentDatabase
mlUpdateRestApiServers
mlWatch

I would suggest rationalising this list and thinking about overlap between targets.

MLCP Export Task doesn't seem to work due to invaild host

I am running a virtual host on my laptop and it is possible that might be the reason. But I can pass my host via command line and it is still not being picked up.

task exportGeonamesTaxonomy(type: com.rjrudin.marklogic.gradle.task.MlcpTask) {
classpath = configurations.mlcp
command = "EXPORT"
database = testFlag.equals("true") ? mlAppConfig.testContentDatabaseName : mlAppConfig.contentDatabaseName
port = testFlag.equals("true") ? mlAppConfig.testRestPort : mlAppConfig.restPort
args = ['-collection_filter', 'org.geonames', '-output_file_path', '$projectDir/src/main/resources/geonames/archive', '-output_type', 'archive']
}

15/09/06 06:23:03 ERROR contentpump.LocalJobRunner: Error running task:
java.lang.IllegalArgumentException: Default provider - Not a usable net address:localhost.localdomain:8100
at com.marklogic.xcc.ContentSourceFactory.defaultConnectionProvider(Cont
entSourceFactory.java:331)
at com.marklogic.xcc.ContentSourceFactory.newContentSource(ContentSource
Factory.java:219)
at com.marklogic.xcc.ContentSourceFactory.newContentSource(ContentSource
Factory.java:243)

Add SSL sample project

Should extract this out of sample-project

Generate signed certificate

When associating an app server with a certificate template via the mgmt API, the certificate is not being signed until a user manually reassigns the app server to the template.

Make it easy to override module permissions

Provide easy way of deleting replica forests

Waiting for restart trips over NoHttpResponseException

I was undeploying, and the check to see if a restart finished threw an error:

:mlAppUndeploy
Caught exception while waiting for MarkLogic to restart: I/O error on GET request for "http://ml8-ml1:8001/admin/v1/timestamp":ml8-ml1:8001 failed to respond; nested exception is org.apache.http.NoHttpResponseException: ml8-ml1:8001 failed to respond
:mlAppUndeploy FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':mlAppUndeploy'.
> I/O error on GET request for "http://ml8-ml1:8002/manage/v2/databases":ml8-ml1:8002 failed to respond; nested exception is org.apache.http.NoHttpResponseException: ml8-ml1:8002 failed to respond

Configure scheduled tasks

Use ml-app-deployer's new support for configuring schedule tasks. Should happen as part of mlDeploy.

WatchTask must support SSL and authentication options

Getting rid of ModulesWatcher in ml-javaclient-util, just going to implement this completely in a task

Show example of disconnected deployments in sample project

Add support for flexrep commands in ml-app-deployer

Need to leverage the results of marklogic/ml-app-deployer#29

Add tasks for deploying individual security resources

Create REST API server without using /v1/rest-apis

A problem with /v1/rest-apis is that I don't think there's a way to customize the forests that are auto-created - e.g. if you want to setup a fast data directory.

Special characters for the admin password are not being recognized

Given mlAdminPassword=[some character string with a $ or other special character]
When I run gradlew -PmlAdminPassword=abc$123 mlDeploy
Then I get a 401 Error when trying to create the roles

run mlDeploy twice

I've had to run mlDeploy target twice ... first time forests/databases/users gets deployed, 2nd time the rest server.

also getting strange error

9:40:06 AM: Executing external task 'mlDeploy'...

Initializing ml-gradle
Setting config dir to: src/ml-gradle/ml-config
App name: marklogic-sesame-test
App host: localhost
App username: admin
App REST port: 8200
Manage host: localhost
Manage username: admin
Admin host: localhost
Admin username: admin
Finished initializing ml-gradle

:mlDeleteModuleTimestampsFile
:mlPrepareRestApiDependencies
No mlRestApi dependencies found, so no preparation of dependencies required

:mlAppDeploy
POST request for "http://localhost:8002/v1/rest-apis" resulted in 400 (Bad Request); invoking error handler
:mlAppDeploy FAILED

FAILURE: Build failed with an exception.

my gradle.properties looks like

mlConfigDir=src/ml-gradle/ml-config
mlHost=localhost
mlUsername=admin
mlPassword=admin
adminUser=admin
adminPassword=admin
writerUser=rest-writer
writerPassword=x
readerUser=rest-reader
readerPassword=x
validUser=rest-writer
validPassword=x
mlAppName=marklogic-sesame-test
mlRestPort=8200

if I get a chance will peek into ml-gradle to see what is up, but thought I would mention.

Add task for viewing all the commands in the deployer

This will be helpful for determining what commands can be configured

mlLogCommands or mlPrintCommands or just mlCommands

Don't require ext for assets loaded via mlcp

LoadAssetsViaMlcpTask currently assumes that the modules in the "consolidated assets" directory will be stored under "/ext". This doesn't need to be true - a developer may be using this task to load assets that purposefully are not exposed via the REST API.

Execute mlPostDeploy task as mlManageUsername

As a developer

I want to TITLE

So that I can avoid using the default permissions assigned to mlRestAdminUsername

I am using the mlPostDeploy task to create a few triggers but the create triggers task uses the default permissions of the user (mlRestAdminUserName). As a result, I've had to assign the createTriggers execute priv to that user. I don't think we want that user creating triggers. One way to address this problem would be to change the createTriggersTask to not use xdmp:default-permissions(). But it may be a better solution to have mlManageUsername run the postDeploy task.

property for setting ml-config and ml-modules dir

might exist already ?

Document example of overriding MarkLogic modules

sample-project should have a task that loads modules under the root path of the modules database, not under /ext, in the event that a developer wishes to override a MarkLogic module

Execute roxy commands that will run a ml-gradle task

As a developer used to roxy commands
I want to use TITLE
So that I decrease my learning curve to using ml-gradle

This is a low priority nice-to-have feature. If we want to get serious about our consultants using ml-gradle this might prove to be a nice way to transition.

Add support for IDEA

Including the IntelliJ IDEA gradle plugin allows also to create project and module files easily.

sample project maven entry

contains this
maven {url "../../gh-pages-marklogic-java/releases"}

probably want to sanitize that

Including creation of a trigger in sample-project

Loading Scheduled Tasks a second time fails

This may be a Management API issues.

Trying to load the following scheduled task a second time and I get a 500 error when it hits PUT /manage/v2/tasks/.

If I remove the task-user property then it works.

{
"task-enabled": false,
"task-path": "/ext/isearch/task/getDeltaObjects.xqy",
"task-root": "/",
"task-type": "weekly",
"task-period": 2,
"task-day": ["tuesday"],
"task-start-time": "12:00:00",
"task-database": "%%DATABASE%%",
"task-modules": "%%MODULES_DATABASE%%",
"task-user": "isearch-user"
}

mlRedeploy doesn't run mlLoadModules twice

Need a different task that calls mlLoadModules