nathanmarz / cascalog-contrib Goto Github PK

We should convert the example module into a template that uses https://github.com/Raynes/lein-newnew for instantiation.

The following code doesn't work:

(with-job-conf {"foo" "bar"}
(workflow ...))

with-job-conf updates thread-local bindings, but each workflow step runs in a new thread which gets original binding.

Would be great to have an option to write headers for delimited taps.
That would improve consistency with other classes and examples in Cascading.

When I have ...

(workflow [tmp-path]
a-step ()
b-step ([:deps nil] ...)
c-step ([:deps [a-step b-step]] ...))

I get the exception below. I changed nil to [a-step] and it works fine.

java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: cascading.flow.FlowException: internal error during mapper configuration
at cascading.flow.FlowMapper.configure(FlowMapper.java:65)
... 22 more
Caused by: com.esotericsoftware.kryo.SerializationException: Invalid class name reference: 16383
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:441)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:561)
at com.esotericsoftware.kryo.ObjectBuffer.readClassAndObject(ObjectBuffer.java:206)
at cascalog.KryoService.deserialize(KryoService.java:28)
at cascalog.ClojureCascadingBase.prepare(ClojureCascadingBase.java:51)
at cascading.pipe.Each$EachHandler.prepare(Each.java:499)
at cascading.flow.stack.EachMapperStackElement.prepare(EachMapperStackElement.java:67)
at cascading.flow.stack.StackElement.open(StackElement.java:178)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.StackElement.open(StackElement.java:182)
at cascading.flow.stack.FlowMapperStack.(FlowMapperStack.java:88)
at cascading.flow.FlowMapper.configure(FlowMapper.java:58)
... 22 more