objectrocket / ansible-hadoop Goto Github PK

If you run the task "Register the alert_targets with the Ambari server" more than once you get a 500 error like the following on subsequent runs:

failed: [master03] => {"content": "{\n "status": 500,\n "message": "Exception [EclipseLink-4002](Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint \"alert_target_target_name_key\"\nError Code: 0\nCall: INSERT INTO alert_target (target_id, DESCRIPTION, is_global, notification_type, PROPERTIES, target_name) VALUES (?, ?, ?, ?, ?, ?)\n\tbind => [6 parameters bound]\nQuery: DataModifyQuery(sql=\"INSERT INTO alert_target_states (target_id, alert_state) VALUES (?, ?)\")"\n}", "content_length": "622", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "redirected": false, "server": "Jetty(7.6.7.v20120910)", "set_cookie": "AMBARISESSIONID=xxx;Path=/", "status": 500}
msg: Status code was not [200, 201, 202]

Hi,

Please consider reverting #87

The removed memory logic for namenodes now generates a JAVA out of heap error upon deployment, and the cluster fails to build.

e.g.

[root@master-01 log]# less hadoop/hdfs/hadoop-hdfs-namenode-master-01.uumo2ahfoh8oh0eijoh4.out
#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError=""/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node"
"/usr/hdp/current/hadoop-hdfs-namenode/bin/kill-name-node""

add dynamic bonding based on interfaces provided

bootstrap_static.sh is failing for me with the Destination /etc/sysconfig/iptables does not exist error message on Vagrant with the bento/centos-6.7 box. Indeed /etc/sysconfig/iptables does not exist on this system.

(This Vagrant box is expected to be fairly faithful to the real systems where we would want to deploy Hadoop, possibly with your playbooks.)

The site_facts.yml playbook fails on single-node deployments because there are no nodes in the slave-nodes group to reference. Instead the first/only master node should be referenced in this situation. Additional changes may be necessary to ensure memory on the single node isn't over-allocated with both master and slave services on the same node.

Plays are failing due to:

fatal: [172_23_0_203]: FAILED! => {"changed": false, "failed": true, "msg": "Package at https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm could not be installed", "rc": 1}

Updating to https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Grafana is hard coded into the blueprints but it's not supported prior to Ambari 2.2.2. We should have a conditional to verify that ambari_version is >= 2.2.2 before including METRICS_GRAFANA in the blueprints.

If the "Fail if the cluster create task is in an error state" task in the ambari-server role fails because the cluster create is in a "FAILED" status we should retry the build on subsequent runs of the playbook.

It looks like this might be a race condition since the component that causes the error seems to change. In this case it was MAPREDUCE2, but I've also seen it for ZOOKEEPER.

TASK: [ambari-server | Create the cluster instance] ***************************
failed: [hadoop-node3] => {"content": "{\n "status" : 500,\n "message" : "An internal system exception occurred: Invalid transition for servicecomponenthost, clusterName=hadoop-poc, clusterId=2, serviceName=MAPREDUCE2, componentName=HISTORYSERVER, hostname=hadoop-node2, currentState=INIT, newDesiredState=STARTED"\n}", "content_length": "277", "content_type": "text/plain", "expires": "Thu, 01 Jan 1970 00:00:00 GMT", "failed": true, "redirected": false, "server": "Jetty(7.6.7.v20120910)", "set_cookie": "AMBARISESSIONID=xxx;Path=/", "status": 500}
msg: Status code was not [200, 201, 202]

FATAL: all hosts have already failed -- aborting