Error [urn:drat:MimePartitioner] and [urn:drat:RatAggregator] is not defined about drat HOT 15 CLOSED

Which version of DRAT are you using?

On Tue, Sep 22, 2015 at 4:42 PM, junsuhle [email protected] wrote:

Hi,
I have issue with running DRAT
First two parts (crawling and indexing) seems working fine
However, last two part is not working with this error

So, first two parts completed and here is how my next two parts looks like

Firing off the MapReduce mapper
org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException:
Dynamic workflow task: [urn:drat:MimePartitioner] is not defined!
at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at
org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.executeDynamicWorkflow(XmlRpcWorkflowManagerClient.java:104)
at
org.apache.oodt.cas.workflow.cli.action.DynWorkflowCliAction.execute(DynWorkflowCliAction.java:50)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at
org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.main(XmlRpcWorkflowManagerClient.java:692)
ERROR: Failed to submit dynamic workflow for taskIds
[urn:drat:MimePartitioner] with metadata {} : java.lang.Exception:
org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException:
Dynamic workflow task: [urn:drat:MimePartitioner] is not defined!
Navigate to http://localhost:8080/opsui/ to view the OODT browser and
http://localhost:8080/solr to view the Solr catalog.

Waiting for the mapping and partitioning to finish...Exception in thread
"main" java.io.IOException: Server returned HTTP response code: 500 for
URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:395)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:416)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:112)
Caused by: java.io.IOException: Server returned HTTP response code: 500
for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at
sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2942)
at java.net.URLConnection.getContentType(URLConnection.java:512)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:380)
... 2 more

Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 500 for URL:
http://localhost:8080/opsui/instances/PGE%20EXEC/1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1889)
at
sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1884)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1883)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1456)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:395)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:416)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:112)
Caused by: java.io.IOException: Server returned HTTP response code: 500
for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at
sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:2942)
at java.net.URLConnection.getContentType(URLConnection.java:512)
at org.apache.tika.io.TikaInputStream.get(TikaInputStream.java:380)
... 2 more
org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException:
Dynamic workflow task: [urn:drat:RatAggregator] is not defined!
at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at
org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.executeDynamicWorkflow(XmlRpcWorkflowManagerClient.java:104)
at
org.apache.oodt.cas.workflow.cli.action.DynWorkflowCliAction.execute(DynWorkflowCliAction.java:50)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at
org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.main(XmlRpcWorkflowManagerClient.java:692)
ERROR: Failed to submit dynamic workflow for taskIds
[urn:drat:RatAggregator] with metadata {} : java.lang.Exception:
org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException:
Dynamic workflow task: [urn:drat:RatAggregator] is not defined!
Navigate to http://localhost:8080/opsui/ to view the OODT browser and
http://localhost:8080/solr to view the Solr catalog.

Seems like [urn:drat:MimePartitioner] and [urn:drat:RatAggregator] are not
defined so it makes this error
How can I fix this?

Thank You

—
Reply to this email directly or view it on GitHub
chrismattmann/drat#47.

Lewis

from drat.

I just installed, so it may be the latest version. I followed the instruction posted here

from drat.

OK I assume your talking about master branch.
Do you have a stored version of your terminal e.g. a history so we can see the steps you've executed? Are you sure that you've built and installed are services and that they are successfully running?

from drat.

Here is what i did.
Initiate environmental variable DRAT_HOME JAVA_HOME and GANGLIA_URL
and did this

mkdir -p ~/drat/deploy
mkdir -p ~/drat/src
cd ~/drat/src
git clone https://github.com/chrismattmann/drat.git .
mvn install
tar -C ../deploy/ -xvzf distribution/target/dms-distribution-0.1-bin.tar.gz
(everything successful)

and i delete pid files from workflow/run filemgr/run

and
$DRAT_HOME/bin/oodt start
(all three are successful)

and I run 'go' function
$DRAT_HOME/bin/drat go $HOME/your-repo
I just put repo as Drat folder
but I also tried other folders as well

First two steps are working fine
but reducing and partitioning..

Also tried Manual method
but result is the same
Crawl the repository of interest, e.g., $HOME/your-repo:
$DRAT_HOME/bin/drat crawl $HOME/your-repo

Index the crawled repo in Apache™ SOLR:
$DRAT_HOME/bin/drat index $HOME/your-repo

these two are finished successfully

but Fire off the partitioner and mappers:
$DRAT_HOME/bin/drat map

Fire off the reducer:
$DRAT_HOME/bin/drat reduce

weren't run because of the error I mentioned above

from drat.

ahh I think you need to source the environment in order to run DRAT. Try setting the environments variables mentioned here before doing $DRAT_HOME/bin/oodt start.

from drat.

Still not working.
Is my JAVA_HOME is wrong?
I was first trying what the instruction says
export JAVA_HOME="readlink -f /usr/bin/java | sed "s:bin/java::""
but this wasn't work
so I set up it as what my friend recommended, which was
export JAVA_HOME=$(/usr/libexec/java_home)

from drat.

Can you run the following commands and paste the output here:

java -version
javac -version

which java
ls -l "Path returned from the above command"
ls -l "Path returned from the above command"

echo $JAVA_HOME
export -p | grep -i java
env | grep -i java

Note - There are two "ls -l" commands above and it's a not a mistake.

You might need to do "mvn clean install" if you have all valid values for the above mentioned commands.

from drat.

yep good idea @karanjeets @junsuhle please try that :) 👍

from drat.

java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

javac -version
javac 1.8.0_51

which java
/usr/bin/java

ls -l /usr/bin/java
lrwxr-xr-x 1 root wheel 74 May 28 21:05 /usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java

ls -l /usr/bin/java
lrwxr-xr-x 1 root wheel 74 May 28 21:05 /usr/bin/java -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java

echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home

export -p | grep -i java
declare -x JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home"

env | grep -i java
JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_51.jdk/Contents/Home

from drat.

What errors are you getting when running drat

from drat.

Firing off the MapReduce mapper
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException: Dynamic workflow task: [urn:drat:MimePartitioner] is not defined!

and

Waiting for the mapping and partitioning to finish...Exception in thread "main" java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/opsui/instances/PGE%20EXEC/1

org.apache.oodt.cas.workflow.structs.exceptions.RepositoryException: Dynamic workflow task: [urn:drat:RatAggregator] is not defined!

basically last two parts. First two (crawling indexing) works fine

from drat.

@jihyunoh did you encounter this previously when we met @jpl?

from drat.

@junsuhle what happens when you type:

env | grep WORKFLOW
env | grep OODT

What are the outputs there?

from drat.

The issue is resolved now.
This error was due to some zombie processes of CAS Resource Manager sitting on the port 9002 and stopping any new process to run. The resolution is as below:

1. Stop OODT
   ${DRAT_HOME}/bin/oodt stop
2. Check for any zombie process
   ps -ef | grep -i 9002
3. Kill the process which corresponding to CAS Resource Manager
   kill -9 <pid>
4. Start OODT
   ${DRAT_HOME}/bin/oodt start
5. Run DRAT and check.

from drat.

Great yep I think it was an env var thing and I was going to ask if there was OODT running anywhere too. Great work!

from drat.