mids-scaling-up / coursework Goto Github PK

Hello
When running the command
' salt-cloud -p sl_centos7_small mytestvs'

I am running into this error:
Invalid keyword 'str' for variable 'master_type'

Anyone else having the same problem?

We need to remove the "Explore the VLAN Ordering Form" section because SL doesn't offer this option any longer.

Need to include instruction on which directory to run from. Need to cd to README dir before running spark shell

Hi all -

I was able to get this running successfully. A few points that may help other people out:

1. I didn't have a masters file in the /usr/local/hadoop/etc/hadoop folder. Creating the file and adding the master line worked just fine.
2. mapred-site.xml needs to be copied from mapred-site.xml.template before making any edits
3. There's a broken line on the config text for yarn-site.xml, if you copy paste and don't join the line you'll get an error starting the namenode services:

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</
name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

Should be:

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>

• the link '(see Hadoop v1 Installation)' is broken (it points to https://github.com/MIDS-scaling-up/coursework/blob/master/week5/hw/version_1/README.md) which is a 404
• after setup, /usr/local/hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar (in Part 2: Load data) is not valid---maybe this should be updated to /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.1.jar?
• when I fix the previous issue with the suggested workaround, and then do 'Part 2: Load data', I get a lot of errors like the following:

  Container [pid=32025,containerID=container_1443213849013_0005_01_000002] is running beyond virtual memory limits. Current usage: 159.9 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

  ...

  ERROR streaming.StreamJob: Job not successful!
  Streaming Command Failed!

The slcli create command in lab 1 should have CENTOS_7_64 as the OS.

There is no submission information for hw5. We need to update them to reflect the deliverables.

Students are getting errors related to 'RSA', for example per https://ucbischool.slack.com/archives/C1VFF2Y3U/p1495496191162070

[root@saltmaster ~]# salt-cloud -p sl_centos7_small mytests
[INFO    ] salt-cloud starting
[INFO    ] POST https://api.softlayer.com/xmlrpc/v3.1/SoftLayer_Account
[ERROR   ] There was a profile error: global name 'RSA' is not defined

Resolved the issue by installing dependencies first, code below should be added to instructions

 yum install -y python-pip && pip install SoftLayer apache-libcloud
 yum install openssl-devel
 yum install gcc
 yum install python-devel
 pip install M2Crypto
 curl -o /tmp/install_salt.sh -L https://bootstrap.saltstack.com && sh /tmp/install_salt.sh -Z -M git 2015.5

https://github.com/MIDS-scaling-up/coursework/tree/master/week2/labs/salt_states_and_docker-ELK/salt_seed

SSL generation should happen before changing the ssl.sls file.

The Spark installation indicates installing JDK 1.8, but apparently the latest CENTOS 7 x64 image / RPMs installs both 1.7 and 1.8 and sets the default to 1.7. To correct this, issue the following command:

[root@spark1 ~]# alternatives --config java

There are 2 programs which provide 'java'.

Selection Command

1 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre/bin/java
*+ 2 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91-2.6.2.1.el7_1.x86_64/jre/bin/java

Enter to keep the current selection[+], or type selection number: 1

Once done, do java --version to check that the correct version is selected.

The output is buried in the sea of log messages. Tried having a log4j.properties files in the project directory with spark log level set to WARN but doesn't seem to work.

log4j.logger.org.apache.spark=WARN

Getting authentication error caused by using the wrong username for slcli

Instructions should specify to use the SL id displayed when viewing API key.

We need to update HW5 to specify what the students turn in.

Homework 3 requests that IP addresses are created, but are never used.