Code Monkey home page Code Monkey logo

drat's Introduction

Apache Distributed Release Audit Tool (DRAT)™

A distributed, parallelized (Map Reduce) wrapper around Apache RAT™ (Release Audit Tool). RAT is used to check for proper licensing in software projects. However, RAT takes a prohibitively long time to analyze large repositories of code, since it can only run on one JVM. Furthermore, RAT isn't customizable by file type or file size and provides no incremental output. This wrapper dramatically speeds up the process by leveraging Apache OODT™ to parallelize and workflow the following components:

  1. Apache Solr™ based exploration of a CM repository (e.g., Git, SVN, etc.) and classification of that repository based on MIME type using Apache Tika™.
  2. A MIME partitioner that uses Apache Tika™ to automatically deduce and classify by file type and then partition Apache RAT™ jobs based on sets of 100 files per type (configurable) -- the M/R "partitioner"
  3. A throttle wrapper for RAT to MIME targeted Apache™ RAT. -- the M/R "mapper"
  4. A reducer to "combine" the produced RAT logs together into a global RAT report that can be used for stats generation. -- the M/R "reducer"

See the wiki for more information on installing and running DRAT:

You can clone the wiki by running
git clone https://github.com/apache/drat.wiki.git

Visit our new website drat.apache.org at the Apache Software Foundation.

Current build status: Build Status

drat's People

Contributors

ahmedifhaam avatar brenopolanski avatar buggtb avatar chrismattmann avatar dalerasrorov avatar darth-pr avatar dependabot[bot] avatar jihyunoh avatar karanjeets avatar lewismc avatar mjjoyce avatar ottlinger avatar snyk-bot avatar solomax avatar tbpalsulich avatar thejanw avatar wikier avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

ericwhyne lewismc imclab tbpalsulich mjjoyce jweaver darth-pr pree011235 pombredanne jihyunoh sfrancus wikier karanjeets spiculedata solomax thejanw kinarashah nipurndoshi djkevincr bolaben ahmedifhaam ims94 saikuanalytics brenopolanski dd-guo ngimhana isabella232

drat's Issues

Unable to serialize class: org.apache.oodt.cas.workflow.structs.WorkflowInstance

Is there any significance of this error ? This seems to be appearing at regular intervals in tomcat logs

STATUS IS PGE EXEC
ERROR - Objects                    - Error serializing object class org.apache.oodt.pcs.opsui.WorkflowInstanceViewerPage [object=[Page class = org.apache.oodt.pcs.opsui.WorkflowInst
anceViewerPage, id = 0, version = 0]]
org.apache.wicket.util.io.SerializableChecker$WicketNotSerializableException: Unable to serialize class: org.apache.oodt.cas.workflow.structs.WorkflowInstance
Field hierarchy is:
  0 [class=org.apache.oodt.pcs.opsui.WorkflowInstanceViewerPage, path=0]
    private java.lang.Object org.apache.wicket.MarkupContainer.children [class=[Ljava.lang.Object;]
      private org.apache.wicket.markup.html.link.PopupSettings org.apache.wicket.markup.html.link.Link.popupSettings[9] [class=org.apache.oodt.cas.webcomponents.workflow.instance.Wo
rkflowInstancesViewer, path=0:instance_viewer]
        private java.lang.Object org.apache.wicket.MarkupContainer.children [class=[Ljava.lang.Object;]
          java.lang.Object org.apache.wicket.Component.data[4] [class=org.apache.oodt.cas.webcomponents.workflow.instance.WorkflowInstancesViewer$2, path=0:instance_viewer:workflow_
insts]
            java.lang.Object org.apache.wicket.Component.data [class=org.apache.wicket.model.util.ListModel]
              private java.lang.Object org.apache.wicket.model.util.GenericBaseModel.object [class=java.util.Vector]
                private java.lang.Object org.apache.wicket.model.util.GenericBaseModel.object[write:1] [class=[Ljava.lang.Object;]
                  private java.lang.Object org.apache.wicket.model.util.GenericBaseModel.object[write:1][0] [class=org.apache.oodt.cas.workflow.structs.WorkflowInstance] <----- fiel
d that is not serializable
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:386)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:432)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.access$500(SerializableChecker.java:64)
    at org.apache.wicket.util.io.SerializableChecker$1InterceptingObjectOutputStream.replaceObject(SerializableChecker.java:535)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1144)
    at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
    at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
    at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
    at java.util.Vector.writeObject(Vector.java:1077)
    at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:542)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:432)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:432)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.checkFields(SerializableChecker.java:656)
    at org.apache.wicket.util.io.SerializableChecker.internalCheck(SerializableChecker.java:579)
    at org.apache.wicket.util.io.SerializableChecker.check(SerializableChecker.java:365)
    at org.apache.wicket.util.io.SerializableChecker.writeObjectOverride(SerializableChecker.java:725)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
    at org.apache.wicket.util.io.IObjectStreamFactory$DefaultObjectStreamFactory$2.writeObjectOverride(IObjectStreamFactory.java:129)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)
    at org.apache.wicket.util.lang.Objects.objectToByteArray(Objects.java:1124)
    at org.apache.wicket.protocol.http.pagestore.AbstractPageStore.serializePage(AbstractPageStore.java:203)
    at org.apache.wicket.protocol.http.pagestore.DiskPageStore.storePage(DiskPageStore.java:840)
    at org.apache.wicket.protocol.http.SecondLevelCacheSessionStore$SecondLevelCachePageMap.put(SecondLevelCacheSessionStore.java:332)
    at org.apache.wicket.Session.requestDetached(Session.java:1435)
    at org.apache.wicket.RequestCycle.detach(RequestCycle.java:1176)
    at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1462)
    at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
    at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
    at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.NotSerializableException: org.apache.oodt.cas.workflow.structs.WorkflowInstance
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
    at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
    at java.io.ObjectOutputStream.access$300(ObjectOutputStream.java:162)
    at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1707)
    at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
    at java.util.Vector.writeObject(Vector.java:1077)
    at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
    at org.apache.wicket.Component.writeObject(Component.java:4700)
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at org.apache.wicket.util.io.IObjectStreamFactory$DefaultObjectStreamFactory$2.writeObjectOverride(IObjectStreamFactory.java:121)
    ... 28 more

Error [urn:drat:MimePartitioner] and [urn:drat:RatAggregator] is not defined

Hi,
I have issue with running DRAT
First two parts (crawling and indexing) seems working fine
However, last two part is not working with this error

So, first two parts completed and here is how my next two parts looks like

Firing off the MapReduce mapper
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException: Dynamic workflow task: [urn:drat:MimePartitioner] is not defined!
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.executeDynamicWorkflow(XmlRpcWorkflowManagerClient.java:104)
at org.apache.oodt.cas.workflow.cli.action.DynWorkflowCliAction.execute(DynWorkflowCliAction.java:50)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.main(XmlRpcWorkflowManagerClient.java:692)
ERROR: Failed to submit dynamic workflow for taskIds [urn:drat:MimePartitioner] with metadata {} : java.lang.Exception: org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException: Dynamic workflow task: [urn:drat:MimePartitioner] is not defined!
Navigate to http://localhost:8080/opsui/ to view the OODT browser and http://localhost:8080/solr to view the Solr catalog.

Waiting for the mapping and partitioning to finish...Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:395)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:416)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:112)
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2942)
at java.net.URLConnection.getContentType(URLConnection.java:512)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:380)
... 2 more

Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:395)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:416)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:112)
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2942)
at java.net.URLConnection.getContentType(URLConnection.java:512)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:380)
... 2 more
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException: Dynamic workflow task: [urn:drat:RatAggregator] is not defined!
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.executeDynamicWorkflow(XmlRpcWorkflowManagerClient.java:104)
at org.apache.oodt.cas.workflow.cli.action.DynWorkflowCliAction.execute(DynWorkflowCliAction.java:50)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.main(XmlRpcWorkflowManagerClient.java:692)
ERROR: Failed to submit dynamic workflow for taskIds [urn:drat:RatAggregator] with metadata {} : java.lang.Exception: org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException: Dynamic workflow task: [urn:drat:RatAggregator] is not defined!
Navigate to http://localhost:8080/opsui/ to view the OODT browser and http://localhost:8080/solr to view the Solr catalog.

Seems like [urn:drat:MimePartitioner] and [urn:drat:RatAggregator] are not defined so it makes this error
How can I fix this?

Thank You

Run DRAT on all of Apache SVN

Let's run DRAT on the whole Apache SVN repository and see how long it takes to complete and then come up with some stats on what worked, what didn't, what errors were seen, etc.

http://svn.apache.org/repos/asf/

The output of this task should be:

  1. All DRAT logs for the code bases analyzed
  2. The output Solr catalog of the code
  3. A histogram analysis of the code types, etc.

Connect to OODT REST services from Proteus

We have a question regarding on Apache CFX and OODT REST
Can we get the processing status of 'map' and 'reduce' by OODT REST?
I thought map and reduce are not about OODT so cannot get process status for those two.

Also, I looked at 'The OODT PCS Health Monitor service' https://cwiki.apache.org/confluence/display/OODT/OODT+REST+Services#OODTRESTServices-TheOODTPCSHealthMonitorservice

and it's output includes

"crawlerStatus"
"deamonStatus"
"ingestHealth"
"jobHealth"
and even
"LatestFiles"

Question is. Isn't it pretty much all information for Crawling?
(Assuming LatestFiles are the files that are ingested recently)

If so, we don't need any more REST request for crawling?

Lastly, I also want to know what REST request related for indexing...
It would be very helpful.

Create Web Interface

We should create a simple web interface for DRAT. Go to the page, enter a URL (maybe a local file, a git repo, an svn repo, or something else), run DRAT, give a pretty status page of what DRAT is doing, and show a pretty results page.

Getting CSV file with workflow ID

Started dynamic workflow with id 'f18d24f2-80f5-11e5-85f5-b129ee5c57a9'

Navigate to http://localhost:8080/opsui/ to view the OODT browser and http://localhost:8080/solr to view the Solr catalog.
Started dynamic workflow with id 'f9171374-80f5-11e5-85f5-b129ee5c57a9'

Navigate to http://localhost:8080/opsui/ to view the OODT browser and http://localhost:8080/solr to view the Solr catalog.

So I think I can get csv file (# of notes/binaries/archives/standards/apache.. etc) by using workflow id above.
However, looks like http://host/fmprod/data?productID=3abc2ec0-721e-11de-831c-abb3b8e32d26 (Download a product (by ID) from the CAS REST API) request
is nothing related to workflow id. How can we get REST request for workflow id?
Moreover, which REST request gives us the workflow id (ex: Started dynamic workflow with id 'f18d24f2-80f5-11e5-85f5-b129ee5c57a9')

javax.servlet.ServletException: java.lang.Exception: java.lang.NullPointerException on hitting fmprod (File Manager API) endpoint

Hey,
So I'm trying to download individual (and aggregate) RatLogs via hitting the following OODT REST endpoint (http://localhost:8080/fmprod/dataset?typeID=urn:drat:RatLog) [replace RatLog with AggregateRatLog for the aggregate]. However, when I go to that endpoint in my browser I receive this 500 error (taken from deploy/tomcat/logs/catalina.out)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:745)
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: java.lang.NullPointerException
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.getFirstPage(XmlRpcFileManagerClient.java:489)
at org.apache.oodt.cas.product.data.DatasetDeliveryServlet.doPut(DatasetDeliveryServlet.java:136)
at org.apache.oodt.cas.product.data.DatasetDeliveryServlet.doGet(DatasetDeliveryServlet.java:84)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:745)

Anyone know what could be the cause? Thanks.

API for getting license type and mime type of repository in real-time (aka while DRAT is running)

Hey guys,
I was wondering what the best method for getting a breakdown of Mime Types/License Types in a repo are, especially which DRAT is running. I'm part of the Proteus team (stevenfrancus/drat is the Proteus project... for now), developing a GUI for DRAT. We want to show in real time the licenses and mime types in the repo as DRAT analyzes them.

From what I understand, getting the mime types would be plausible during the Crawling step, while license might not be available until DRAT has finished analyzing. I saw using http://localhost:8080/localhost:8080/fmprod/data?productID={RAT_LOG_ID} that I can download a .log file of each RAT instances' operations, including the license types and mime types.

However:

  1. Downloading may be prohibitably slow and expensive if it's a large repo
  2. It could only really be done once (not in real time)

Is there a REST API that can give me a breakdown of mime-types in the repo/license types?

Thanks,
Steven

java.lang.RuntimeException: Pge didn't finish successfully

I am getting a runtime exception and subsequent failure of the entire go task using master branch as of today.
@jihyunoh can you please review and see if it is/was the same or similar issue to that you were getting a few weeks ago...

INFO: Loading PgeConfig file '/usr/local/drat/deploy//pge/policy/PgeConfig_MimePartitioner.xml'
Jun 02, 2015 10:44:30 AM org.apache.oodt.cas.pge.PGETaskInstance createExeDir
INFO: Creating PGE execution working directory: [/usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660]
Jun 02, 2015 10:44:30 AM org.apache.oodt.cas.pge.PGETaskInstance updateStatus
INFO: Updating status to workflow as BUILDING CONFIG FILE
Jun 02, 2015 10:44:30 AM org.apache.oodt.cas.pge.PGETaskInstance updateStatus
INFO: Updating status to workflow as PGE EXEC
Jun 02, 2015 10:44:30 AM org.apache.oodt.commons.io.LoggerOutputStream flush
INFO: OUTPUT: Creating working dirs
OUTPUT: Running MIME partitioner

Jun 02, 2015 10:44:30 AM org.apache.oodt.commons.io.LoggerOutputStream flush
SEVERE: ERROR:   File "/usr/local/drat/deploy//pge/bin/mime_partitioner/mime_rat_partitioner.py", line 111
ERROR: SyntaxError: Non-ASCII character '\xe2' in file /usr/local/drat/deploy//pge/bin/mime_partitioner/mime_rat_partitioner.py on line 111, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details

Jun 02, 2015 10:44:30 AM org.apache.oodt.commons.io.LoggerOutputStream flush
INFO: OUTPUT: Creating working dirs
OUTPUT: Running MIME partitioner

java.lang.RuntimeException: Pge didn't finish successfully
    at org.apache.oodt.cas.pge.PGETaskInstance.runPge(PGETaskInstance.java:322)
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:451)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException: PGETask failed : Exception when executing PGE commands '[export PATH=$HOME/bin/:${PATH}, shopt -s expand_aliases, echo "Creating working dirs", mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/input ; mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/output; mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/logs, echo "Running MIME partitioner", /usr/local/drat/deploy//pge/bin/mime_partitioner/mime_rat_partitioner.py -u http://localhost:8080/solr/drat -c 100 -w http://localhost:9001 -t urn:drat:RatCodeAudit > /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/logs/mime-partitioner-1433267070660.log]' : Pge didn't finish successfully
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:456)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Exception when executing PGE commands '[export PATH=$HOME/bin/:${PATH}, shopt -s expand_aliases, echo "Creating working dirs", mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/input ; mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/output; mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/logs, echo "Running MIME partitioner", /usr/local/drat/deploy//pge/bin/mime_partitioner/mime_rat_partitioner.py -u http://localhost:8080/solr/drat -c 100 -w http://localhost:9001 -t urn:drat:RatCodeAudit > /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/logs/mime-partitioner-1433267070660.log]' : Pge didn't finish successfully
    at org.apache.oodt.cas.pge.PGETaskInstance.runPge(PGETaskInstance.java:334)
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:451)
    ... 4 more
Jun 02, 2015 10:44:30 AM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread executeTaskLocally
WARNING: Exception executing task: [MimePartitioner] locally: Message: PGETask failed : Exception when executing PGE commands '[export PATH=$HOME/bin/:${PATH}, shopt -s expand_aliases, echo "Creating working dirs", mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/input ; mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/output; mkdir /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/logs, echo "Running MIME partitioner", /usr/local/drat/deploy//pge/bin/mime_partitioner/mime_rat_partitioner.py -u http://localhost:8080/solr/drat -c 100 -w http://localhost:9001 -t urn:drat:RatCodeAudit > /usr/local/drat/deploy//data/jobs/mimepartitioner/1433267070660/logs/mime-partitioner-1433267070660.log]' : Pge didn't finish successfully
Jun 02, 2015 10:44:36 AM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread checkTaskRequiredMetadata
INFO: Task: [RatAggregator] has no required metadata fields
Jun 02, 2015 10:44:36 AM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread executeTaskLocally
INFO: Executing task: [RatAggregator] locally
Jun 02, 2015 10:44:36 AM org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder buildImports
INFO: Loading PgeConfig file '/usr/local/drat/deploy//pge/policy/PgeConfig_RatAggregator.xml'
Jun 02, 2015 10:44:37 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog paginateQuery
WARNING: Query: [q=] for Product Type: [urn:drat:RatLog] returned no results
java.lang.NullPointerException
    at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFileManager.java:607)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
    at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
    at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
    at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
    at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
    at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
    at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
    at java.lang.Thread.run(Thread.java:745)
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null
    at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
    at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
    at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
    at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:1003)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:322)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:313)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.getCustomMetadata(XmlFilePgeConfigBuilder.java:180)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:109)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.buildImports(XmlFilePgeConfigBuilder.java:101)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:69)
    at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:133)
    at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:98)
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
java.lang.Exception: Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:326)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:313)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.getCustomMetadata(XmlFilePgeConfigBuilder.java:180)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:109)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.buildImports(XmlFilePgeConfigBuilder.java:101)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:69)
    at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:133)
    at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:98)
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null
    at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:1009)
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:322)
    ... 12 more
java.io.IOException: Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
    at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:75)
    at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:133)
    at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:98)
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException: PGETask failed : Failed to instanciate PGETaskInstance : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:456)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
    at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InstantiationException: Failed to instanciate PGETaskInstance : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
    at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:116)
    at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
    ... 4 more
Jun 02, 2015 10:44:37 AM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread executeTaskLocally
WARNING: Exception executing task: [RatAggregator] locally: Message: PGETask failed : Failed to instanciate PGETaskInstance : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
Jun 02, 2015 10:45:25 AM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog paginateQuery
WARNING: Query: [q=] for Product Type: [urn:drat:RatAggregateLog] returned no results

Proteus UI reports numbers different from DRAT backend

The numbers reported in Proteus are different than the DRAT backend:

  1. Counts of files are off - there is a disagreement between GenericFile count and count reported by Proteus for total files.
    1. Indexing is off - for example only when running DRAT with Proteus only 463 files were index in Solr rather than 4.5k or 6.5k.
    2. Only ran 5 mappers instead of 7

DRAT Mapper - Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1

While running mapper, DRAT breaks the current_pges loop and runs reduce job. This issue is intermittent and might related to the below mentioned error.

DRAT Logs

Waiting for the mapping and partitioning to finish.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:395)
    at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:416)
    at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:112)
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2942)
    at java.net.URLConnection.getContentType(URLConnection.java:512)
    at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:380)
    ... 2 more

There are still MapReduce mappers running! It is reccomended you wait for them to finish, 
then try to run '$DRAT_HOME/bin/drat reduce' again later.
Stopping OODT...
/gpfs/flash/users/tg830544/final/drat/drat_master/deploy/bin /gpfs/flash/users/tg830544/final/logs
/gpfs/flash/users/tg830544/final/logs

Tomcat Logs

Nov 27, 2015 1:13:03 PM org.apache.jasper.compiler.JspRuntimeContext checkUnload
FINER: Checking JSPs for unload in context [/pcs], JSP count: 0 queue length: -1
STATUS IS PGE EXEC
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: java.lang.NullPointerException
    at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
    at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
    at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
    at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.paginateWorkflowInstances(XmlRpcWorkflowManagerClient.java:213)
    at org.apache.oodt.cas.webcomponents.workflow.WorkflowMgrConn.safeGetWorkflowInstPageByStatus(WorkflowMgrConn.java:146)
    at org.apache.oodt.cas.webcomponents.workflow.instance.WorkflowInstancesViewer.<init>(WorkflowInstancesViewer.java:89)
    at org.apache.oodt.pcs.opsui.WorkflowInstanceViewerPage.<init>(WorkflowInstanceViewerPage.java:43)
    at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:188)
    at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:89)
    at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.newPage(BookmarkablePageRequestTarget.java:305)
    at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.getPage(BookmarkablePageRequestTarget.java:320)
    at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.processEvents(BookmarkablePageRequestTarget.java:234)
    at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
    at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1250)
    at org.apache.wicket.RequestCycle.step(RequestCycle.java:1329)
    at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1436)
    at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
    at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
    at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
ERROR - RequestCycle               - Can't instantiate page using constructor public org.apache.oodt.pcs.opsui.WorkflowInstanceViewerPage(org.apache.wicket.PageParameters) and argum
ent status = "PGE EXEC" pageNum = "1"
org.apache.wicket.WicketRuntimeException: Can't instantiate page using constructor public org.apache.oodt.pcs.opsui.WorkflowInstanceViewerPage(org.apache.wicket.PageParameters) and 
argument status = "PGE EXEC" pageNum = "1"
    at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:212)
    at org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java:89)
    at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.newPage(BookmarkablePageRequestTarget.java:305)
    at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.getPage(BookmarkablePageRequestTarget.java:320)
    at org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.processEvents(BookmarkablePageRequestTarget.java:234)
    at org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(AbstractRequestCycleProcessor.java:92)
    at org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:1250)
    at org.apache.wicket.RequestCycle.step(RequestCycle.java:1329)
    at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1436)
    at org.apache.wicket.RequestCycle.request(RequestCycle.java:545)
    at org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:486)
    at org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:319)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.wicket.session.DefaultPageFactory.createPage(DefaultPageFactory.java:188)
    ... 28 more
Caused by: java.lang.NullPointerException
    at org.apache.oodt.cas.webcomponents.workflow.instance.WorkflowInstancesViewer.computeStartEndIdx(WorkflowInstancesViewer.java:205)
    at org.apache.oodt.cas.webcomponents.workflow.instance.WorkflowInstancesViewer.<init>(WorkflowInstancesViewer.java:92)
    at org.apache.oodt.pcs.opsui.WorkflowInstanceViewerPage.<init>(WorkflowInstanceViewerPage.java:43)
    ... 32 more

Release DRAT 1.0

Few changes here and there. A number of bugs fixed. @chrismattmann what do you think about a release and possibly getting a Github website?

Environment Variables Missing when Attempting to hit OODT File Manager API

I worked with @chrismattmann on this last night but when I tried to connect first to the OODT Product Service API using DRAT, I ran into two issues:

  1. The Wiki page seems to be out of date (https://cwiki.apache.org/confluence/display/OODT/File+Manager+REST+API) since it says the API URLs are http://host/prodserver, when they now are http://host/fmprod

  2. Several environment variables are missing by default, which cause 500 errors when hitting these URLs:
    To use the Product Service APIs for File Manager, you need to set $FMPROD_HOME, which is defined as $DRAT_HOME/webapps/fmprod.

DRAT does not identify different files with same name

Say for example I have 4 files called index.html e.g.

lmcgibbn@LMC-032857 /usr/local/incubator-usergrid(master) $ find . -name "index.html"
./sdks/android/doc/index.html
./sdks/html5-javascript/index.html
./sdks/html5-javascript/tests/mocha/index.html
./sdks/java/doc/index.html

The output of DRAT is as follows

  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/index.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/insecure-banner.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/ios.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/javascript.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/loading.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/login.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/logout.html

Here you can see that only one index.html has been specified! This is a bug and it should be resolved.
A possibly mechansim for actually adding more context to the location of 'Unapproved licenses', would be for us to alter the DRAT output to something the following

Unapproved licenses:

  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/account.html            --> /path/to/account.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/activities.html           --> /path/to/activities
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/all-calls.html             --> etc
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/android.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/apigee_test.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/app-overview.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/appMenu.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/data.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/display-generic.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/display-roles.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/display-users.html
  /usr/local/drat/deploy/data/jobs/rat/1407484852545/input/dogs.html

DRAT file copy script misses all sorts of file naming issues

We really need a better solution than the file copy used by the RAT audit tool. Undeniably, it's very interesting that we have a great sed command and so forth to handle files, but we need to support all sorts of files, with '$' in the name; with ' ' whitespace, etc. #93 exposed this issue.

org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.getTopNProducts(XmlRpcFileManagerClient.java:672) on calling fmprod

Hey all,
I'm getting this error when I try to hit the endpoint fmprod/viewRecent?channel=RatLog&id=urn:drat:RatLog on http://drat.dyndns.org:8080/proteus (it's the Proteus GUI). Anyone know what could be the cause? I can't reproduce it on my local machine.

drat 'go' command stalls at reduce prompt

BTW the go command is dynamite. Really easy to get up and running.
I've recently been running DRAT on Apache Usergrid (Incubating) [0] in an attempt to add missing license headers, detect binaries and also set up up for a release as we have yet to release within the Incubator.

I see that after we've crawler, indexed and fired off map task(s), we are immediately prompted to fire of reducers e.g.

Waiting for the mapping and partitioning to finish...
There are still MapReduce mappers running! It is reccomended you wait for them to finish.
Are you sure you wish to continue? [yN]

This therefore essentially stalls the 'go' process. I would imagine for a user it would be be a catch22 situation as to whether we answer yes or no to this prompt.

I think the behavhiour should find another mechanism for firing off reduce tasks which do not require user input to complete (or destroy) the current DRAT job.

I need ot delve more closely into the mechanics of getting this working but will try my best over the next wee while.

[0] https://git-wip-us.apache.org/repos/asf?p=incubator-usergrid.git;a=tree;h=refs/heads/master;hb=master

org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null

Hello,

I am getting the below error while running DRAT using resource manager. Is there any known solution for this ?

Note - The resource manager was earlier looking for cas-filemgr-0.10.jar. I added that and the error shifted to the below one.

INFO: Loading PgeConfig file '/root/drat/deploy/pge/policy/PgeConfig_RatAggregator.xml'
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null
        at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
        at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
        at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
        at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
        at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:1003)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:322)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:313)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.getCustomMetadata(XmlFilePgeConfigBuilder.java:180)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:109)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.buildImports(XmlFilePgeConfigBuilder.java:101)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:69)
        at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:133)
        at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:98)
        at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
        at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
        at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
        at java.lang.Thread.run(Thread.java:745)
java.lang.Exception: Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:326)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:313)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.getCustomMetadata(XmlFilePgeConfigBuilder.java:180)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:109)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.buildImports(XmlFilePgeConfigBuilder.java:101)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:69)
        at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:133)
        at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:98)
        at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
        at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
        at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : null
        at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:1009)
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.fillIn(XmlFilePgeConfigBuilder.java:322)
        ... 11 more
java.io.IOException: Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
        at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:75)
        at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:133)
        at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:98)
        at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
        at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
        at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
        at java.lang.Thread.run(Thread.java:745)
org.apache.oodt.cas.resource.structs.exceptions.JobInputException: Failed to run task
        at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:104)
        at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException: PGETask failed : Failed to instanciate PGETaskInstance : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
        at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:456)
        at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
        ... 2 more
Caused by: java.lang.InstantiationException: Failed to instanciate PGETaskInstance : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}
        at org.apache.oodt.cas.pge.PGETaskInstance.initialize(PGETaskInstance.java:116)
        at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:449)
        ... 3 more
Oct 16, 2015 8:10:35 PM org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub genericExecuteJob
INFO: stub attempting to execute class: [org.apache.oodt.cas.workflow.structs.TaskJob]

Integrate dratstats.py into DRAT

dratstats.py is an advanced version of drat "go" and hence should be natively integrated with the DRAT run process.

We can call this python script from the drat bash script when it has done scanning and indexing of the project repository.

Upgrade to OODT 0.8

I'm using DRAT right now and thought I may as well get my sh*t together, do the good thing and send a PR for upgrade to OODT. PR coming very shortly.

Duplicate files detected and timestamped

Whilst reviewing recent htrace 4.0 incubating RC I am getting results like the following

...
  /usr/local/drat/deploy/data/jobs/rat/1441639423619/input/dependency-reduced-pom.xml
  /usr/local/drat/deploy/data/jobs/rat/1441639423619/input/dependency-reduced-pom.xml_09072015_1123

This is the same file... seemingly being strangely timestamped and analyzed twice. I think this is a bug in drat master.

Add capability for python file detection

A simple addition to the filemgr etc/mime-types.xml file makes file manager aware of Python files:

  <mime-type type="text/x-python">
    <_comment>Python script</_comment>
    <magic priority="50">
      <match value="#!/bin/python" type="string" offset="0"/>
      <match value="#! /bin/python" type="string" offset="0"/>
      <match value="eval &quot;exec /bin/python" type="string" offset="0"/>
      <match value="#!/usr/bin/python" type="string" offset="0"/>
      <match value="#! /usr/bin/python" type="string" offset="0"/>
      <match value="eval &quot;exec /usr/bin/python" type="string" offset="0"/>
      <match value="#!/usr/local/bin/python" type="string" offset="0"/>
      <match value="#! /usr/local/bin/python" type="string" offset="0"/>
      <match value="eval &quot;exec /usr/local/bin/python" type="string" offset="0"/>
      <match value="/bin/env python" type="string" offset="1"/>
    </magic>
    <glob pattern="*.py"/>
    <sub-class-of type="text/plain"/>
  </mime-type>

javax.servlet.ServletException: Resource class org.apache.oodt.pcs.services.HealthResource can not be instantiated due to InvocationTargetException

When running DRAT master branch on the recent Apache OODT 0.10 RC#1 I am coming across some problems running drat go.
A stack trace can be located within logs/cas_resource1.log

SEVERE: Servlet /pcs threw load() exception
javax.servlet.ServletException: Resource class org.apache.oodt.pcs.services.HealthResource can not be instantiated due to InvocationTargetException
    at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:247)
    at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.getResourceProviders(CXFNonSpringJaxrsServlet.java:211)
    at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.loadBus(CXFNonSpringJaxrsServlet.java:92)
    at org.apache.cxf.transport.servlet.AbstractCXFServlet.init(AbstractCXFServlet.java:84)
    at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1139)
    at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:966)
    at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3956)
    at org.apache.catalina.core.StandardContext.start(StandardContext.java:4230)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:760)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:740)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:544)
    at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:920)
    at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:883)
    at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:492)
    at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1138)
    at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:311)
    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:120)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1022)
    at org.apache.catalina.core.StandardHost.start(StandardHost.java:736)
    at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1014)
    at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
    at org.apache.catalina.core.StandardService.start(StandardService.java:448)
    at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)

Aug 31, 2015 9:26:31 PM org.apache.jasper.compiler.JspRuntimeContext <init>
FINE: Parent class loader is: WebappClassLoader
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
org.apache.catalina.loader.StandardClassLoader@6220ac02

... investigating

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.