jaytaylor / ansible-kafka Goto Github PK
View Code? Open in Web Editor NEWAnsible Kafka role
Home Page: https://galaxy.ansible.com/jaytaylor/kafka/
License: Other
Ansible Kafka role
Home Page: https://galaxy.ansible.com/jaytaylor/kafka/
License: Other
looks like it's only used in check-env.yml. should it be removed?
reserved.broker.max.id = 1000
[2016-01-06 10:46:40,824] FATAL (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: broker.id must be equal or greater than -1 and not greater than reserved.broker.max.id
at scala.Predef$.require(Predef.scala:219)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:945)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:942)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:691)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)
Would have been nice to specify JMX_PORT in the role. Seems like Kafka won't start the JMX service as default if the variable isn't set, and if it's not specified in ZK.
There is an unsupported syntax in /etc/init/kafka.conf
serge:~$ cat /etc/centos-release
CentOS release 6.8 (Final)
serge:~$ initctl version
init (upstart 0.6.5)
serge:~$ initctl status kafka
initctl: Unknown job: kafka
serge:~$ initctl start kafka
initctl: Unknown job: kafka
serge:~$ initctl list | grep kafka ; echo $?
1
serge:~$ initctl log-priority debug && initctl --verbose reload-configuration
serge:~$ tail -n 4 -f /var/log/messages
Jun 2 03:42:18 serge-opo-1 init: Reloading configuration
Jun 2 03:42:18 serge-opo-1 init: Loading configuration from /etc/init.conf
Jun 2 03:42:18 serge-opo-1 init: Loading configuration from /etc/init
Jun 2 03:42:18 serge-opo-1 init: /etc/init/kafka.conf:9: Unknown stanza
In the ansible-kafka/tasks/system.yml
, we have the following task:
- name: "Add kafka utilities to /etc/profile* so they'll be available in $PATH"
lineinfile: dest=/etc/profile.d/Z99-kafka.sh regexp="^export PATH=.*$" line='export PATH="${PATH}:/usr/local/lib/kafka/bin"' state=present create=yes
sudo: yes
Please bear in mind that /usr/local/lib/kafka/bin
is wrong. It should be /usr/local/kafka/bin
instead.
TASK: [jaytaylor.kafka | Extract downloaded kafka archive] ********************
ok: [52.31.200.214] => {"changed": false, "check_results": {"cmd": "/bin/tar -C \"/usr/local/\" --diff -zf \"/tmp/kafka.tar.gz\"", "err": "", "out": "", "rc": 0, "unarchived": true}, "dest": "/usr/local/", "gid": 0, "group": "root", "handler": "TgzArchive", "mode": "0755", "owner": "root", "size": 4096, "src": "/tmp/kafka.tar.gz", "state": "directory", "uid": 0}
ok: [52.48.44.129] => {"changed": false, "check_results": {"cmd": "/bin/tar -C \"/usr/local/\" --diff -zf \"/tmp/kafka.tar.gz\"", "err": "", "out": "", "rc": 0, "unarchived": true}, "dest": "/usr/local/", "gid": 0, "group": "root", "handler": "TgzArchive", "mode": "0755", "owner": "root", "size": 4096, "src": "/tmp/kafka.tar.gz", "state": "directory", "uid": 0}
ok: [52.48.12.7] => {"changed": false, "check_results": {"cmd": "/bin/tar -C \"/usr/local/\" --diff -zf \"/tmp/kafka.tar.gz\"", "err": "", "out": "", "rc": 0, "unarchived": true}, "dest": "/usr/local/", "gid": 0, "group": "root", "handler": "TgzArchive", "mode": "0755", "owner": "root", "size": 4096, "src": "/tmp/kafka.tar.gz", "state": "directory", "uid": 0}
TASK: [jaytaylor.kafka | Link /usr/local/kafka to the right version] **********
failed: [52.48.44.129] => {"failed": true, "path": "/usr/local/kafka", "src": "/usr/local/kafka_2.11-0.9.0.0", "state": "absent"}
msg: src file does not exist, use "force=yes" if you really want to create the link: /usr/local/kafka_2.11-0.9.0.0
failed: [52.48.12.7] => {"failed": true, "gid": 0, "group": "root", "mode": "0777", "owner": "root", "path": "/usr/local/kafka", "size": 29, "src": "/usr/local/kafka_2.11-0.9.0.0", "state": "link", "uid": 0}
msg: src file does not exist, use "force=yes" if you really want to create the link: /usr/local/kafka_2.11-0.9.0.0
failed: [52.31.200.214] => {"failed": true, "gid": 0, "group": "root", "mode": "0777", "owner": "root", "path": "/usr/local/kafka", "size": 29, "src": "/usr/local/kafka_2.11-0.9.0.0", "state": "link", "uid": 0}
msg: src file does not exist, use "force=yes" if you really want to create the link: /usr/local/kafka_2.11-0.9.0.0
I had 0.8.2.0 installed and tried to install 0.9.0.0 on top of that. The unpack module was unable to change state.
Hi,
I had to override server.advertised_listeners, which is defined in the default role variables, and it was quite a difficult task.
I asked a coworker which knows Ansible much better than I and the best method we found so far is to use pre_tasks
as bellow:
- hosts: kafka
roles:
- role: ansible-kafka
kafka_hosts: "{{ ansible_play_batch }}"
kafka_version: 1.0.0 # Kafka version override.
vars:
zookeeper_hosts: "kafka1-1:2181,kafka1-2:2181,kafka1-3:2181"
kafka_zookeeper_hosts: ['kafka1-1', 'kafka1-2', 'kafka1-3']
pre_tasks:
- name: set advertised listeners
set_fact:
server: '{{ server | combine({"advertised_listeners": "PLAINTEXT://"+ansible_hostname+":9092" }) }}'
The thing is we either need to completely redefine the server
dict in the ansible play, or change the Ansible merge strategy in ansible.cfg (which is generally a bad idea) or use this set_fact
trick.
Are you aware of a better method to redefine those variables ? If not, would you like to add an example in the role documentation?
Cheers,
Problem is caused by fact that tr on empty string is ok (command return 0). So tr -d '-' || echo '0' behaves different then expected.
Proper order of commands below
i think that means:
File : kafka-cfg.yml
task name: "Generate generic unique machine id integer"
shell command : ( test -r /etc/fstab && ls -l /dev/disk/by-uuid/ | grep $(mount | grep ' / ' | cut -d' ' -f1 | cut -d'/' -f3) | grep --ignore-case --only-matching --extended-regexp --max 1 '[0-9a-f]{3,}[0-9a-f-]+' | tr -d '-' || echo '0' ; ifconfig | grep --ignore-case --only-matching --extended-regexp '([0-9a-f]{2}:){5}[0-9a-f]{2}' | tr -d ':' | tr -d '\n') | python -c 'import sys; x, y = sys.stdin.read().split(chr(10))[0:2]; x = int(x, 16); y = int(y, 16); sys.stdout.write((str(x + y)[-9:])); sys.exit(1 if x == 0 and y == 0 else 0)'
When I use a non-LVM partition, it works because of :
$ mount | grep ' / ' | cut -d' ' -f1
/dev/sda2
And this pattern matches with UUID symlink, so you can pick its UUID :
$ ls -l /dev/disk/by-uuid/ | grep $(mount | grep ' / ' | cut -d' ' -f1 | cut -d'/' -f3) ; echo $?
lrwxrwxrwx 1 root root 10 avril 28 11:48 5c0dc710-1247-402a-b3c7-b6a263c02cc3 -> ../../sda2
0
Output code is 0, and that's okay
But with LVM partitioning :
$ mount | grep ' / ' | cut -d' ' -f1
/dev/mapper/vg_covfefe-lv_root
$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 28 avril 09:12 1815e505-77d6-47e5-aecf-653404cb8933 -> ../../sda1
lrwxrwxrwx 1 root root 10 28 avril 09:12 9b40461e-6840-4225-8899-421c0e5dcc39 -> ../../dm-0
lrwxrwxrwx 1 root root 10 28 avril 09:12 e63ff395-4c7b-4e14-b690-1cb5c7da371e -> ../../dm-1
$ ls -l /dev/disk/by-uuid/ | grep $(mount | grep ' / ' | cut -d' ' -f1 | cut -d'/' -f3) ; echo $?
1
Output code is 1, and string is empty.
So here is the result of the whole shell command with LVM partitioning :
$ ( test -r /etc/fstab && ls -l /dev/disk/by-uuid/ | grep $(mount | grep ' / ' | cut -d' ' -f1 | cut -d'/' -f3) | grep --ignore-case --only-matching --extended-regexp --max 1 '[0-9a-f]{3,}[0-9a-f-]+' | tr -d '-' || echo '0' ; ifconfig | grep --ignore-case --only-matching --extended-regexp '([0-9a-f]{2}:){5}[0-9a-f]{2}' | tr -d ':' | tr -d '\n') | python -c 'import sys; x, y = sys.stdin.read().split(chr(10))[0:2]; x = int(x, 16); y = int(y, 16); sys.stdout.write((str(x + y)[-9:])); sys.exit(1 if x == 0 and y == 0 else 0)'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: need more than 1 value to unpack
And here is the error when playing role :
TASK [jaytaylor.ansible-kafka : Generate generic unique machine id integer] ****
fatal: [sh-opo-1]: FAILED! => {"changed": false, "cmd": "( test -r /etc/fstab && ls -l /dev/disk/by-uuid/ | grep $(mount | grep ' / ' | cut -d' ' -f1 | cut -d'/' -f3) | grep --ignore-case --only-matching --extended-regexp --max 1 '[0-9a-f]{3,}[0-9a-f-]+' | tr -d '-' || echo '0' ; ifconfig | grep --ignore-case --only-matching --extended-regexp '([0-9a-f]{2}:){5}[0-9a-f]{2}' | tr -d ':' | tr -d '\\n') | python -c 'import sys; x, y = sys.stdin.read().split(chr(10))[0:2]; x = int(x, 16); y = int(y, 16); sys.stdout.write((str(x + y)[-9:])); sys.exit(1 if x == 0 and y == 0 else 0)'", "delta": "0:00:00.018923", "end": "2017-05-28 08:49:15.438258", "failed": true, "rc": 1, "start": "2017-05-28 08:49:15.419335", "stderr": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\nValueError: need more than 1 value to unpack", "stdout": "", "stdout_lines": [], "warnings": []}
Hope it helps you to solve this issue.
Regards,
Serge
Hi,
It seems that the current link to galaxy in the project description is outdated. Looks like it should now be https://galaxy.ansible.com/jaytaylor/kafka/
.
I cannot make a PR for this one ;)
Regards,
This role fails with kafka_version: 0.10.1.0
, because it has been signed with another key than the one specified in defaults.yml (kafka_sig_id: E0A61EEA
).
We should instead follow the instructions at the bottom of https://www.apache.org/dyn/closer.cgi and download and import the KEYS
file.
this role fails on latest version unless you change the keys url to:
http://kafka.apache.org/KEYS
This file contains one extra key:
diff KEYS KEYS-NEW
101d100
<
544a544
>
604,713d603
< pub 4096R/BE7EFC73 2015-01-14
< uid [ultimate] Ewen Cheslack-Postava <[email protected]>
< sig 3 BE7EFC73 2015-01-14 Ewen Cheslack-Postava <[email protected]>
< sub 4096R/08EA97A4 2015-01-14
< sig BE7EFC73 2015-01-14 Ewen Cheslack-Postava <[email protected]>
<
< pub 4096R/5234D94F 2014-08-22
< uid [ultimate] Ewen Cheslack-Postava <[email protected]>
< sig 3 5234D94F 2014-08-22 Ewen Cheslack-Postava <[email protected]>
< sub 4096R/964F16A8 2014-08-22
< sig 5234D94F 2014-08-22 Ewen Cheslack-Postava <[email protected]>
<
< -----BEGIN PGP PUBLIC KEY BLOCK-----
< Version: GnuPG v2
<
< mQINBFS2qt4BEACksfgjdHfD2io0WZsc8jXHzx6oXE6lWnARo4hLgf1q32adgeRz
< ofGV5AdiP1N5Pwus0o8PEa2UN9/rOIXavw4W0xMlWRiZlavYw6wR7zOGuFINRZJm
< LFvPyIuTtAwdrsuUdOqAcw4hrz0j/V7xg7l79CalVpzO4JKnBYRtNSbg0FeVpk+0
< AEt6e3nlA/5WZLRnNwJTkSpwVkgRgmIjJh889xCtXtxRkjNoTq0H+ks12xdQyruL
< nFRzQmrBCsIS9C+VCmVu8r/OBVVtwulaIYCcIxu213a23XmTmlfmHQXnvwdMPBT3
< me/2ubm6Xqon92Nc08AhxzzWp/rh9SceNWBkFzMdYnXpCIwWGRcvzJ4M+WhL5ho2
< SjaPoKj1LqFhzpeNA5F4SRPOfqvkEkGIl3L3qxXlrax2yOuJMv9sb6op01JIrQY6
< c0oQCzX5vxRdDimZfHnrcZnlbn7Wzs3N8hNwPn0+0ZGLTn//5/esNWtUkf/5cK8R
< afqe0bntiNFLxvR1ysMYJf02FXRJ7rVaS02KZISrc/64bpE5GG5v6iklHrx3SJ4B
< HykbITfN9n0uO2B4iivb86OnV/X5bjv52aJ3llwmsQtTyMrr+JOeccjYv+bw6SK+
< xi/t8BWlwfxa7+//5XrTgNaINE96R9EtDlMMnEU+zk0EFFQxXmTx2384DwARAQAB
< tClFd2VuIENoZXNsYWNrLVBvc3RhdmEgPGV3ZW5AY29uZmx1ZW50LmlvPokCOAQT
< AQIAIgUCVLaq3gIbAwYLCQgHAwIGFQgCCQoLBBYCAwECHgECF4AACgkQk07sfb5+
< /HNJIg/9GW2s+0EpKyIcvyp46Hw4y0ruOKz++JZs/vbHGWCbYQbC80gKDHxCQxvH
< FgK+s4xFe9hsbD/36FBKE9Jbit581F983/KPgoD0bvX2yum6X6T/TaGFSfPkO6hw
< qCbLSswLo5QJV527yrvbKe2dpL7v6E6ZOLZiP8vYJMIS30UyV40gPJp7q6yyT8gH
< xEKW5aGDuCHCzRlRB+jNFUzsB6Q5MeOTVHfTvjPhHLbFhGBmU7buqxtT7EREAze5
< 8I8JUFJkHtdf32a463a+2qjll6AEkWt2iBVuRilkgOnxuX/xNdb7H/6iFCT6UNCb
< pMrM3RLzh0iklT5BNeZ5mMXY8FnOqBy/fFGvWedb6T9Im4ERgWrVq14X8b+uUaP9
< URRvBczXvTfSb4Lajt5BvOvgleOF7iQhzFO1byzo2HD5CArv0HJEIP7GYNDnp14g
< GjQehaqmHP7/RBRDw/Hg0gy828KmfwD3K6VSWpwHOPaas84O6waxRsCKW6Ng/PpM
< FWF5NfPm4sVsuK/QiCpc/hpDzpDVxLL9Z7cOG6APZKFv+g9isUTtP2sn7tDoas0Y
< BvgpPz9gpASF2Y3/3wgyitzY7YO1w36jwgSqkTud8Lknc6tGOStGqclqmt4Ep/OB
< 43i7nbYTo12Udn37WotuDstOm8NVZWxsjXVBb7QW8Rn/fJnolZq5Ag0EVLaq3gEQ
< AKN94fP55s47wo31KQhRgQH7AsLQcMh2bWcJDchIvMsF6oB582T7uHqU3AYFvJNP
< FSkPc0k4XetyZTBTtbnw0bRM1UQECDvQnB4M+HpqlAM3LJce3oPzyVYZ0vxpQRcc
< Y/bCyEc9j3U2BxtbHfZgCTv+keSNsE6i5dpZ92KRw/PQAyth6fDDM40DR8SiXJR2
< tIaTmXoa0JOoECPwDLQqccXmpL+5Vl4xcjrPBZ9BIiA0WBm2dvSk4Nt+8xz8FjNh
< M+slsTUiI20yfKEXN9mM9pPsS7gnngZR96QJBWOBersf3L/aO1z1ywZORyYFED86
< U+I67IIHHHYVC+CsRTOLHEeSNGnYKAmuY9aQnnFkE5fkGQ46mx2eE05Od9xiciy7
< U+XKJ0Pz/yBaiSoLg5TMHo+33giHCh/ESrc+aiohjXKA6Z6cqkmQIg19sisNmZqJ
< 0IBNZrvNJHCxolN8VSvz1lfAyEkbYQ6L4JGdnp5Gu0DWaaJcQ2BoE/i2uD0xjNCv
< o+GuEF66YI2Gm0d1oePsy+jkgrw0sZetaCccX08dJOcNklKz6nQYjFT2C43zSlzP
< jkDEA9sgEabDN0SyYAnbDR+ZW0g+/B/VrWS+j+apNQAckYWAIQCzyMSbkrYJRAdO
< vEXKo+rQxm/LyBIq5pMwqy1nlx87LP9h/AgtGJN6Qc0PABEBAAGJAh8EGAECAAkF
< AlS2qt4CGwwACgkQk07sfb5+/HOqHw/+MhfNz5I8n7xmBvVln7sujPkiA6heWbxe
< Ga+zZnqRTFFsULaMgz4dIdXwXI2Sgym3WKangiJZr3OEpLPlXSM/FS+GAruRH+/y
< 2CPfkqxd1jG13Q1/BWBHbtxJizWuZ+UtK+/wqWc099/4oTnkckchtZBJQ5gdGoio
< swqZdj9rguXROAtW2Fmc/5K+vxgGTuE8cu4TxjA0ZEWdMaJYI6dlLg9LjPpGzs4L
< jDn2dItasnsZ6irhXISYhV9WJbus2H9kYMjZxdJhY6HzkWVaUVrdtiGlWjf6fQ9N
< e0X6DN9kd3YMNhiJ8JExxrB//2JPr/3B5nc/Z0zKmwn1znUz04WWKQ1QM9Irszrr
< 5fWAFy0LgsiT0922hH0f9puss5+irQUSIXDaW2iaCdNszZiTZxNwJDYu8m0pj7eP
< KgkKutptC/1on/mE9IeOkQ5acj2XfgKvljMP9VBX5PrRoRjJ/PMxinZPbzAHZy20
< 7TYMbJH5dsqFIeh66jycn2IDKElvk7A9Wk98XbaxEFq8KhxblG+glV+kz2NWm3x6
< wJ/WFx5uXvKXuXrir92HGgzC6hy+v7m1DPCBlqM3+qrXTcyhbqeC52pjj4AAFnaF
< G9VPAuAnKuZSCkxGuiEgxxjDdikTkYmxbqpel+m0T6DLLgNX58M2yURAIaDajLW/
< gozBq5tLHNiZAg0EU/a4MQEQAMohxV5xUHkknnH4h3xr6tOczInCW3xObdUlsZh9
< mLhiU4+0kzQtNbjKK2qgNgmQzL1p+VYuQ2XQJmhxynXAcc9sFag7kOnzrVvC+HGb
< gZRr79YrtGqEFwU/a3YommEVAFdVvBe0HM+24pbBMoiHV1OaabnSmAYX2Fmpuh+t
< xo7FvPZL6/XqPs3lNQiOR10HU5ZaRajfjok8X5w10X5nOUr0jlj3hhE2usb5/tII
< 6WX2LaYwiC0nSu61TlbijN4m0ikbY3L1ycZirGGbn9eziTfK36itznrNF3DaARSW
< 2rodDKiISiln92QGXnV9rA+C/+bcSUpd8KFsswjfuWCUVzhS3h3xnRZ9UBhvex69
< soLa0TAZPzGqyoRwQF0gTTmRDkwOebzop/t1ZfaQWDaNfEqAK7MD72fmqGQX43dj
< DdgMg9dpmxi3EMSLuk082NqffL1bTv3wW6SFpyra3iRLfz1TJISuszU/Z+TNRZTi
< 8v/U5X5VH5TBRLSRaLGbAfq3z0Ro1XmK+opccbwKMpQ1j40KXTy1kWgt/RYNQWZ/
< nSZ4sNSwko6sFN8F6pvLcjyHZCFpc3aSdTdsv/c539qgEF+2+b/Epus9GEiYvyE+
< mxUqxR37jLXF6lnkHOtJBc1FEiUNDLupdzGU5XRvXDmpxJQY8tgSALloiwneVhpL
< U8E/ABEBAAG0JUV3ZW4gQ2hlc2xhY2stUG9zdGF2YSA8bWVAZXdlbmNwLm9yZz6J
< AjgEEwECACIFAlP2uDECGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEN2q
< NFJSNNlPvpwP/1wypIiAYYn32A/FEbRP6XZ9IjTOBL3HXUqj1FpZm+kNorkUoBk5
< EyMV9O4emzhrxqBlvsgHG9OM+Zw+oHNcdw3bcSI08+ckIFIRqVgRYe9ghEZZgbZs
< SmupaW/wr0jcBQSWLW7VEMlFleTZjrGhL6rP0sIXm4cqTiJJAzDF6Ngf1slfZ2xd
< /eZxHII7mNizFLtLuA57+aBhekxCX2i7n13vM8l7cr+9uXUeeUwxh0HFlpy+24sf
< yqrm261jE186YPGkrB6oFTa3XfP1JsLx1yTO2dXUqYzroMzh7jM8Ohv16U/2PWIf
< 5UKskEnQqv9sqPOKWDhWoWtpk92bYWj4mglPpqk0s7A+gIgCq8ODICNMFKIYl/EO
< +zKftuGLZnAOOqJ6yUz+ZWOQHbDpKNCPmtTxj5QZtaWeZEghCtysITFTvZNLkex3
< xAJo659/k3LJqphHn3RXR9u694ELxNoHCD5ZPSO4nW8zcBcqway4lEZV6LBxE+A8
< oKNKzWOpdnxwussK120PXIR4rO+V/x6rEh4pcQzzgD1dmkaXXtBsFh/dykgT0Abd
< M0qHQO0uJmZkhh/H80mH9IBGrU5QWxspwmfcLOZfx0lHsUkXZJcW1EhPSRC2hIGm
< gYm3wHrBcbXHHKH4f0XsBQSM6MASC1GseZJOR6qcs0oqbAfK3sQDFoyIuQINBFP2
< uDEBEAC3lnj+gp6swX14ZG/vHBmC8TrSi0AdHQRA5p3eYwYCo5lxjdYfzhYSXW8p
< /5M9i4x3M7MQodKqNTmG7w8w7EJhDUxR6j5dUcIRtt1oPbKcjazzYt3GOChyY8lI
< dt6eXsdZ+s0lFujDu7lyJXVbR3A7/HSjy75ExH03Dtct20yawO8yibZOvs9A2QeR
< B689GSp6d1xp2iO8g66zrRGWjiJka7Uxy9hlr0TQ1/9uovhlcj+3ISJwDs3J/ZUc
< dra7fKnr3aanGBJ6fVeKh6ymz8AfuecXRrIza1tx0Hp2K+JcZDcNa+gYjfjWjV0l
< W8M/cxDJpusLbu0szP4B5lPqMGzU/NMjVkV3BOMHRGZDRmGxP/GeTEmjjmLiiVz/
< nsg1n6sGIVwKhpy/c7kcxzJUTA+/+pSLT6H7TvKo6K9GYmb/EJ/GLwobZePo09h0
< pH5uVjKEKCoa83Nr0ZYyanN3GHQIgm5ZbsjEUeiRUGnDSo6dHfJzvPUFFHwpaYHa
< kfSg+kBmti+7igCbqbF2jMAaaiChSQDc4BXsgj77RDM81e10ZqK0Ju3sKyVvwBK2
< KjrQXqJcesxRPpg8GjLbCDw5W7W1yOvtRyhhxKXR4hGMSFVxGdQ1XAbdVb0BQEHb
< rPDh4VanzX9ih0q1bLLdXlRMy88AqW2pOa3x92amDgTEXpi+RQARAQABiQIfBBgB
< AgAJBQJT9rgxAhsMAAoJEN2qNFJSNNlPHRIP/336jYm0s4n5usCiGj3sI8e9EQXh
< QBm0lvScLZAwFESkGIWk1Zs09E5hy9iZlMENQnOrMLCmvwj7T7YJq8ctMz+5R2tB
< a5xCMvmc+D2WfFQzHY4b9AsBddkEAt88izZEpZZszR3ikAtMzO2pw+HEd6ATW7pI
< cBGeJ9INInpwz9wjKW6mjWtGuSDAsVtSvnDShjxOnbw0/lJvGUNRvqkJ7TFeU0wl
< YkrLuQgry2tzAqaYzTq6cA7P9aYQd0dVjjhnAIv5ZU1Bq5vYQnoF6cJZRtnmBNaq
< GwHGE/wpkLD/Mxu/gCvjB1eT7UqMPsdxTqAKlyFKPDqpggLqNjcUVVckUXO94P67
< 3R/fOWSCvPtwEioVSUaT3Y2EQcdO7nyGzydSYFhlvJf1foNhmabXHebyZ7SzwDX2
< EW/9IzyvZCeiduCu3zCiRD5GHiKpHwTUSsHxg/evmhZQlq8P6HGme94IP1VZzxAy
< jqUFRnVEc12qWtYKXROF5VsZ8HAyil8O4rlzstv5eFzB1HHBUOMONFsVyMqSHybW
< EbtR59HTeakJ/3W1aNPK0niadVYhR583XHoOZlKLXq2AVd5PeSgRANI3c9847zsY
< U0PdwcFXT1sU3rtuhREDUTagZ3FmwIOse/NXbN5dHcnnZMWn+qqzMxiutb4MuN01
< qUAdpxxKffByAEiV
< =LJo0
< -----END PGP PUBLIC KEY BLOCK-----
i'm trying to run the ansible-kafka playbook clone from the github repo and happen to be receiving below 'dict object' has no attribute 'por
playbook.yml
---
- hosts: dev
roles:
- ansible-kafka
vars:
zookeeper_hosts: "kafka-d1:2181,kafka-d2:2181,kafka-d3:2181"
kafka_zookeeper_hosts: [ 'kafka-d1','kafka-d2','kafka-d3' ]
kafka_zookeeper_port: 2181
kafka_hosts:
- dev
kafka_healthcheck_address: "kafka-d1:2181,kafka-d2:2181,kafka-d3:2181/kafka"
kafka_server_port: 9092
kafka_connection_string: [ 'kafka-d1','kafka-d2','kafka-d3' ]
TASK [ansible-kafka : Generate the kafka hosts connection string] ******************************************************************************************** fatal: [kafka-d1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'port'\n\nThe error appears to have been in '/home/ahshan.md/ansible/ansible-kafka/tasks/kafka-cfg.yml': line 31, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears
I think the Vagrant VM is irrelevant, or at least much less relevant than updating apt's cache before trying to install the java package.
$ vagrant provision
==> deployml: Running provisioner: ansible...
deployml: Running ansible-playbook...
statically included: /usr/local/etc/ansible/roles/ansible-kafka/tasks/check-env.yml
statically included: /usr/local/etc/ansible/roles/ansible-kafka/tasks/java.yml
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and
make sure become_method is 'sudo' (default).
This feature will be removed in a
future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
statically included: /usr/local/etc/ansible/roles/ansible-kafka/tasks/system.yml
statically included: /usr/local/etc/ansible/roles/ansible-kafka/tasks/limits.yml
statically included: /usr/local/etc/ansible/roles/ansible-kafka/tasks/kafka-install.yml
statically included: /usr/local/etc/ansible/roles/ansible-kafka/tasks/kafka-cfg.yml
PLAY [Apply configuration to virtual machine] **********************************
TASK [setup] *******************************************************************
ok: [deployml]
TASK [ansible-kafka : Check 'kafka_zookeeper_hosts' variable] ******************
skipping: [deployml]
TASK [ansible-kafka : Check 'kafka_hosts' variable] ****************************
skipping: [deployml]
TASK [ansible-kafka : Check 'kafka_generate_broker_id' variable] ***************
skipping: [deployml]
TASK [ansible-kafka : Check if Java is installed] ******************************
fatal: [deployml]: FAILED! => {"changed": false, "cmd": "command -v java", "delta": "0:00:00.001418", "end": "2016-11-03 20:17:58.563578", "failed": true, "rc": 127, "start": "2016-11-03 20:17:58.562160", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}
...ignoring
TASK [ansible-kafka : Install OpenJDK] *****************************************
fatal: [deployml]: FAILED! => {"cache_update_time": 0, "cache_updated": false, "changed": false, "failed": true, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'openjdk-7-jre-headless'' failed: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata-java_2016f-0ubuntu0.14.04_all.deb 404 Not Found [IP: 91.189.91.23 80]\n\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n", "stderr": "E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata-java_2016f-0ubuntu0.14.04_all.deb 404 Not Found [IP: 91.189.91.23 80]\n\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n", "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n acl at-spi2-core colord dconf-gsettings-backend dconf-service fontconfig\n hicolor-icon-theme libasound2 libasound2-data libatk-bridge2.0-0 libatk1.0-0\n libatk1.0-data libatspi2.0-0 libcairo-gobject2 libcairo2 libcanberra-gtk3-0\n libcanberra-gtk3-module libcanberra0 libcolord1 libcolorhug1 libdatrie1\n libdconf1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libexif12 libfontenc1\n libgd3 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgl1-mesa-dri\n libgl1-mesa-glx libglapi-mesa libgphoto2-6 libgphoto2-l10n libgphoto2-port10\n libgraphite2-3 libgtk-3-0 libgtk-3-bin libgtk-3-common libgudev-1.0-0\n libgusb2 libharfbuzz0b libice6 libieee1284-3 libjasper1 libjbig0 liblcms2-2\n libllvm3.4 libltdl7 libnotify-bin libnotify4 libpango-1.0-0\n libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpixman-1-0 libsane\n libsane-common libsm6 libtdb1 libthai-data libthai0 libtiff5\n libtxc-dxtn-s2tc0 libv4l-0 libv4lconvert0 libvorbisfile3 libvpx1\n libwayland-client0 libwayland-cursor0 libx11-xcb1 libxaw7 libxcb-dri2-0\n libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shm0\n libxcb-sync1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxfont1\n libxi6 libxinerama1 libxkbcommon0 libxkbfile1 libxmu6 libxpm4 libxrandr2\n libxrender1 libxshmfence1 libxt6 libxtst6 libxxf86vm1 notification-daemon\n sound-theme-freedesktop x11-common x11-xkb-utils xfonts-base\n xfonts-encodings xfonts-utils xserver-common xserver-xorg-core\nUse 'apt-get autoremove' to remove them.\nThe following extra packages will be installed:\n ca-certificates-java java-common libasyncns0 libflac8 libnspr4 libnss3\n libnss3-nssdb libpcsclite1 libpulse0 libsctp1 libsndfile1 libvorbisenc2\n lksctp-tools tzdata-java\nSuggested packages:\n default-jre equivs pcscd pulseaudio icedtea-7-jre-jamvm libnss-mdns\n sun-java6-fonts fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho\n ttf-wqy-microhei ttf-wqy-zenhei ttf-indic-fonts-core ttf-telugu-fonts\n ttf-oriya-fonts ttf-kannada-fonts ttf-bengali-fonts\nThe following NEW packages will be installed:\n ca-certificates-java java-common libasyncns0 libflac8 libnspr4 libnss3\n libnss3-nssdb libpcsclite1 libpulse0 libsctp1 libsndfile1 libvorbisenc2\n lksctp-tools openjdk-7-jre-headless tzdata-java\n0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 69.6 kB/41.4 MB of archives.\nAfter this operation, 66.0 MB of additional disk space will be used.\nErr http://archive.ubuntu.com/ubuntu/ trusty-updates/main tzdata-java all 2016f-0ubuntu0.14.04\n 404 Not Found [IP: 91.189.88.162 80]\nErr http://security.ubuntu.com/ubuntu/ trusty-security/main tzdata-java all 2016f-0ubuntu0.14.04\n 404 Not Found [IP: 91.189.91.23 80]\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " acl at-spi2-core colord dconf-gsettings-backend dconf-service fontconfig", " hicolor-icon-theme libasound2 libasound2-data libatk-bridge2.0-0 libatk1.0-0", " libatk1.0-data libatspi2.0-0 libcairo-gobject2 libcairo2 libcanberra-gtk3-0", " libcanberra-gtk3-module libcanberra0 libcolord1 libcolorhug1 libdatrie1", " libdconf1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libexif12 libfontenc1", " libgd3 libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgl1-mesa-dri", " libgl1-mesa-glx libglapi-mesa libgphoto2-6 libgphoto2-l10n libgphoto2-port10", " libgraphite2-3 libgtk-3-0 libgtk-3-bin libgtk-3-common libgudev-1.0-0", " libgusb2 libharfbuzz0b libice6 libieee1284-3 libjasper1 libjbig0 liblcms2-2", " libllvm3.4 libltdl7 libnotify-bin libnotify4 libpango-1.0-0", " libpangocairo-1.0-0 libpangoft2-1.0-0 libpciaccess0 libpixman-1-0 libsane", " libsane-common libsm6 libtdb1 libthai-data libthai0 libtiff5", " libtxc-dxtn-s2tc0 libv4l-0 libv4lconvert0 libvorbisfile3 libvpx1", " libwayland-client0 libwayland-cursor0 libx11-xcb1 libxaw7 libxcb-dri2-0", " libxcb-dri3-0 libxcb-glx0 libxcb-present0 libxcb-render0 libxcb-shm0", " libxcb-sync1 libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxfont1", " libxi6 libxinerama1 libxkbcommon0 libxkbfile1 libxmu6 libxpm4 libxrandr2", " libxrender1 libxshmfence1 libxt6 libxtst6 libxxf86vm1 notification-daemon", " sound-theme-freedesktop x11-common x11-xkb-utils xfonts-base", " xfonts-encodings xfonts-utils xserver-common xserver-xorg-core", "Use 'apt-get autoremove' to remove them.", "The following extra packages will be installed:", " ca-certificates-java java-common libasyncns0 libflac8 libnspr4 libnss3", " libnss3-nssdb libpcsclite1 libpulse0 libsctp1 libsndfile1 libvorbisenc2", " lksctp-tools tzdata-java", "Suggested packages:", " default-jre equivs pcscd pulseaudio icedtea-7-jre-jamvm libnss-mdns", " sun-java6-fonts fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho", " ttf-wqy-microhei ttf-wqy-zenhei ttf-indic-fonts-core ttf-telugu-fonts", " ttf-oriya-fonts ttf-kannada-fonts ttf-bengali-fonts", "The following NEW packages will be installed:", " ca-certificates-java java-common libasyncns0 libflac8 libnspr4 libnss3", " libnss3-nssdb libpcsclite1 libpulse0 libsctp1 libsndfile1 libvorbisenc2", " lksctp-tools openjdk-7-jre-headless tzdata-java", "0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.", "Need to get 69.6 kB/41.4 MB of archives.", "After this operation, 66.0 MB of additional disk space will be used.", "Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main tzdata-java all 2016f-0ubuntu0.14.04", " 404 Not Found [IP: 91.189.88.162 80]", "Err http://security.ubuntu.com/ubuntu/ trusty-security/main tzdata-java all 2016f-0ubuntu0.14.04", " 404 Not Found [IP: 91.189.91.23 80]"]}
NO MORE HOSTS LEFT *************************************************************
to retry, use: --limit @/vagrant_setup.retry
PLAY RECAP *********************************************************************
deployml : ok=6 changed=0 unreachable=0 failed=1
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
I believe this issue can easily be fixed by editing this step in tasks/java.yml
:
- name: "Install OpenJDK"
apt: name=openjdk-7-jre-headless state=present update_cache=yes cache_valid_time=3600
sudo: yes
when: check_java.rc != 0
tags:
- kafka-install
- java
kafka_log_dir
, kafa_user
, kafa_group
not referenced at tasks/kafka-cfg.yml:53
Hi. Thanks for perfect role, like it a lot, however:
This is my playbook:
- hosts: kafka
vars:
zookeeper_hosts: 172.31.32.93
kafka_zookeeper_hosts:
- 172.31.32.93
kafka_hosts:
- 172.31.46.135
- 172.31.47.248
roles:
- {role: ansible_kafka ,kafka_hosts: "{{ groups.kafka | list }}", kafka_zookeeper_hosts: "{{ zookeeper_hosts | list }}", kafka_version: "0.11.0.2", kafka_scala_serverion: "2.10" }
There is a line in kafka-cfg.yml
which suppose to create zookeeper connection string.
set_fact: kafka_zookeeper_connection_string="{{ kafka_zookeeper_hosts | join(':' ~ kafka_zookeeper_port ~ ',') }}:{{ kafka_zookeeper_port }}"
Instead of creating string like 172.31.32.93:2181
it creates something like
1:2181,7:2181,2:2181,.:2181,3:2181,1:2181,.:2181,3:2181,2:2181,.:2181,9:2181,3:2181
Could You please suggest a fix?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.