hadoop-streaming dnloader is breaking because the mapper can't get the AWS credentials about eggo HOT 4 CLOSED

I think this is happening because AWS_ACCESS_KEY_ID is not set when using a non-interactive login shell (ssh localhost 'echo $AWS_ACCESS_KEY_ID' doesn't print a value).

One option would be to use the hadoop command for all S3 operations, since the S3 credentials are set in /root/ephemeral-hdfs/conf/core-site.xml. The problem would then reduce to #34. The other nice thing about getting this to work is that we are not tied to S3, which may be useful - e.g. using eggo to populate files in HDFS.

from eggo.

You may be able to get around this by appending the env vars to /etc/environment instead of ~/.bash_profile. This worked for me when SunGridEngine nodes were running in non-interactive mode

from eggo.

Thanks for the suggestion @jfeala. I was able to fix this in the end by specifying -cmdenv via the streaming_args parameter in Luigi. I committed this to unblock eggo and get master working again.

from eggo.

Awesome! For the record, I think the luigi S3 client either does or can
wrap the hadoop CLI.
On Apr 21, 2015 21:23, "Tom White" [email protected] wrote:

Thanks for the suggestion @jfeala https://github.com/jfeala. I was able
to fix this in the end by specifying -cmdenv via the streaming_args
parameter in Luigi. I committed this to unblock eggo and get master working
again.

—
Reply to this email directly or view it on GitHub
#35 (comment).

from eggo.