jhorey / ferry Goto Github PK

setup.py chokes on installing docker/docker-ferry because it's unexpectedly a binary file.

running build_scripts
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/tokenize.py", line 375, in find_cookie
    line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa8 in position 40: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 42, in <module>
    """
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 930, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 949, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/command/build.py", line 126, in run
    self.run_command(cmd_name)
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 949, in run_command
    cmd_obj.run()
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/command/build_scripts.py", line 50, in run
    self.copy_scripts()
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/command/build_scripts.py", line 82, in copy_scripts
    encoding, lines = tokenize.detect_encoding(f.readline)
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/tokenize.py", line 416, in detect_encoding
    encoding = find_cookie(first)
  File "/usr/local/Cellar/python3/3.3.5/Frameworks/Python.framework/Versions/3.3/lib/python3.3/tokenize.py", line 380, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for 'docker/docker-ferry'

reading the (docs)[http://docs.python.org/3.3/distutils/setupscript.html#installing-scripts] it seems that scripts are expected to be python scripts. Perhaps using data_files= would be a better choice, maybe.

A nice to have feature would be to have R support in computing nodes so the hadoop platform could be used from R scripts.
This would require a functional R environment
Optionally the rhadoop library rmr2 could be added to the mix (it makes easier to use hadoop from R)

Another nice to have feature would be to somehow add extension capabilities on the backend images in a way that a user could add some additional libraries over the already built base images.
Essentiallty the ability to support customized images, either by replacing the base ones or by extending them

Expose Ferry CLI capabilities via nice web interface. This would also entail multi-user support, authentication, etc.

Hi,
Thanks for great effort building Ferry!
During ferry install I've had to install lxc due to this check: https://github.com/opencore/ferry/blob/master/ferry/install.py#L71

For some time Docker does not depend on lxc and uses libcontainer as a default.
Is there anything specific preventing Ferry to switch to pure Docker?

I installed docker in ubuntu 12.04 ( kernel 3.13.0-35-generic) . Docker works fine. Ferry installation fails with the following error, though pip installation of ferry and dependent package were successful.

sudo ferry install
Traceback (most recent call last):
File "/usr/local/bin/ferry", line 5, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2707, in
working_set.require(requires)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: sphinxcontrib-httpdomain>=1.2.0

The following plan causes the Hadoop client configuration to throw an exception since apparently the "hdfs_type" parameter is being lost somewhere.

backend:

• storage:
  personality: "mongodb"
  instances: 1
• storage:
  personality: "hadoop"
  instances: "{numhadoop}"
  layers:
  - "hive"
  compute:
  - personality: "spark"
  instances: "{numspark}"
  connectors:
• personality: "ferry/spark-client"

Receive this traceback when trying to start a 3x3 hadoopxspark application instance:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/threading2/t2_base.py", line 331, in run
self_run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(_self.__args, *_self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 48, in _stack_worker
_allocate_new_worker(payload["_uuid"], payload)
File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 476, in _allocate_new_worker
new_stack=True)
File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 215, in _allocate_backend
replace = replace)
File "/usr/local/lib/python2.7/dist-packages/ferry/docker/manager.py", line 1201, in allocate_storage
args)
File "/usr/local/lib/python2.7/dist-packages/ferry/docker/configfactory.py", line 129, in generate_storage_configuration
return self._generate_configuration(uuid, container_info, service)
File "/usr/local/lib/python2.7/dist-packages/ferry/docker/configfactory.py", line 76, in _generate_configuration
return config_factory.apply(config, container_info)
File "/usr/local/lib/python2.7/dist-packages/ferry/config/hadoop/hadoopconfig.py", line 513, in apply
hadoop_dirs, hadoop_entry = self._apply_hadoop(config, hadoop_containers)
File "/usr/local/lib/python2.7/dist-packages/ferry/config/hadoop/hadoopconfig.py", line 387, in _apply_hadoop
self._generate_yarn_site(yarn_master, new_config_dir)
File "/usr/local/lib/python2.7/dist-packages/ferry/config/hadoop/hadoopconfig.py", line 249, in _generate_yarn_site
cores = self.system.get_num_cores() / 2
File "/usr/local/lib/python2.7/dist-packages/ferry/config/system/info.py", line 43, in get_num_cores
return int(output.strip())
ValueError: invalid literal for int() with base 10: '4\n4\n4\n4'

It would be nice to have a repeatable process to take any OS instance and have it robustly install all the OS dependencies that ferry needs.

I am interested to run ferry on the following distributions:
Ubuntu
Debian
RHEL/CentOS
Fedora
CoreOS

Hi,

I have installed ferry as per instruction and created a hadoop.yml as per getting started section, but got the following when I run sudo ferry start hadoop.yml. Did I missed anything?

Traceback (most recent call last):
File "/usr/local/bin/ferry", line 9, in
load_entry_point('ferry==0.2.2', 'console_scripts', 'ferry')()
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 809, in main
msg = cli.dispatch_cmd(c, all_cmds[c], options)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 684, in dispatch_cmd
self._check_ssh_key()
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 620, in _check_ssh_key
keydir, _ = self._read_key_dir(root)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 606, in _read_key_dir
return k[1], k[0]
IndexError: list index out of range

Just fired up ferry server after installing a log monitor on the system files, and I get a burst of syslog entries like this:

Nov 2 07:45:46 ### kernel [404983 796677] device-mapper: thin: Creation of a new shapshot 258 of device 48 failes
Nov 2 07:45:46 ### kernel [404983 796628] device-mapper: thin: Creation of a new shapshot 257 of device 48 failes
Nov 2 07:45:46 ### kernel [404983 796578] device-mapper: thin: Creation of a new shapshot 256 of device 48 failes
Nov 2 07:45:46 ### kernel [404983 796528] device-mapper: thin: Creation of a new shapshot 255 of device 48 failes
....

Does anyone know what these messages from device-mapper represent in the ferry server context?

Vagrant version:

~/work/VAGRANT/Proxy/Ferry vagrant --version
Vagrant 1.6.5

Virtual box version:

~/work/VAGRANT/Proxy/Ferry vboxmanage --version
4.3.16r95972

OS version:

vagrant@vagrant-ubuntu-trusty-64:~$ lsb_release  -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:        14.04
Codename:       trusty

docker version:

vagrant@vagrant-ubuntu-trusty-64:~$ docker --version
Docker version 1.0.1, build 990021a

python version:

vagrant@vagrant-ubuntu-trusty-64:~$ python --version
Python 2.7.6

An aside for others installing through a proxy:
If you see pip errors after running $ sudo pip install -U ferry, see pypa/pip#1805. One workaround is to:

$ git clone https://github.com/ctxis/pip.git
$ cd pip
$ sudo python setup.py install

running:

vagrant@vagrant-ubuntu-trusty-64:~$ ferry-dust install
Error: No such image or container: ferry/ferry-server
pulling the ferry image
Pulling repository ferry/ferry-server
4498499b4e65: Pulling dependent layers
4498499b4e65: Download complete
249a44f2f965: Download complete
511136ea3c5a: Download complete
5e66087f3ffe: Download complete
4d26dd3ebc1c: Download complete
d4010efcfd86: Download complete
99ec81b80c55: Download complete
3bfcd6184e6c: Download complete
fcadef5d73c3: Download complete
4ebaec4748e1: Download complete
077fd1caf696: Download complete
c45967030e32: Download complete
888f62d73c9d: Download complete
d63d7f192762: Download complete
1e7811cc4fca: Download complete
22ff4d9f52c1: Download complete
b18d0a2076a1: Download complete
1c7f181e78b9: Download complete
9f676bd305a4: Download complete
67b66f26d423: Download complete
6030677eb5d2: Download complete
25c4824a5268: Download complete
90177ae68092: Download complete
8b1c48305638: Download complete
3f2d795e3309: Download complete
c900195dcbf3: Download complete
6b4e8a7373fe: Download complete
f8a57046c3fc: Download complete
20af62704915: Download complete
db4fe0f2d561: Download complete
c2c0da7cede7: Download complete
4ca1aa06d7c7: Download complete
07740b15b3fd: Download complete
1a1d7ef08dcc: Download complete
6753e3778944: Download complete
cc400379cb68: Download complete
67a2a12aa03f: Download complete
af09a11cf5d4: Download complete
798a83a04a19: Download complete
943a7afc77c7: Download complete
de95eeb92afc: Download complete
ab04cc56c1b2: Download complete
25098d3c6c7a: Download complete
7533a39e7d69: Download complete
24494120dbe9: Download complete
67e689edc7a6: Download complete
9d8ebd110cd4: Download complete
1f7dd8a6fafc: Download complete
e482b718265f: Download complete
5cba33de10c6: Download complete
64e27cc09237: Download complete
starting ferry, using /opt/ferry to save state
2014/10/14 12:21:53 Error response from daemon: No such id: ferry/ferry-base:0.3.3.3
2014/10/14 12:24:01 Error response from daemon: No such id: ferry/cassandra:0.3.3.3
2014/10/14 12:26:08 Error response from daemon: No such id: ferry/titan:0.3.3.3
2014/10/14 12:28:16 Error response from daemon: No such id: ferry/cassandra-client:0.3.3.3
2014/10/14 12:30:23 Error response from daemon: No such id: ferry/gluster:0.3.3.3
2014/10/14 12:32:31 Error response from daemon: No such id: ferry/openmpi:0.3.3.3
2014/10/14 12:34:38 Error response from daemon: No such id: ferry/openmpi-client:0.3.3.3
2014/10/14 12:36:46 Error response from daemon: No such id: ferry/hadoop-base:0.3.3.3
2014/10/14 12:38:53 Error response from daemon: No such id: ferry/hadoop-client:0.3.3.3
2014/10/14 12:41:00 Error response from daemon: No such id: ferry/hive-metastore:0.3.3.3
2014/10/14 12:41:00 Error response from daemon: No such id: ferry/hive-metastore:0.3.3.3
2014/10/14 12:41:01 Error response from daemon: No such id: ferry/hive-metastore:latest
2014/10/14 12:43:08 Error response from daemon: No such id: ferry/mongodb:0.3.3.3
2014/10/14 12:43:08 Error response from daemon: No such id: ferry/mongodb:0.3.3.3
2014/10/14 12:43:08 Error response from daemon: No such id: ferry/mongodb:latest
2014/10/14 12:45:15 Error response from daemon: No such id: ferry/spark:0.3.3.3
2014/10/14 12:45:15 Error response from daemon: No such id: ferry/spark:0.3.3.3
2014/10/14 12:45:15 Error response from daemon: No such id: ferry/spark:latest
2014/10/14 12:47:23 Error response from daemon: No such id: ferry/hadoop:0.3.3.3
2014/10/14 12:47:23 Error response from daemon: No such id: ferry/hadoop:0.3.3.3
2014/10/14 12:47:23 Error response from daemon: No such id: ferry/hadoop:latest
[10/14/2014 12:17:34] [WARNING] all prerequisites met...
[10/14/2014 12:17:34] [WARNING] Creating Ferry bridge 172.18.42.1/16
[10/14/2014 12:17:34] [WARNING] checking for btrfs
[10/14/2014 12:17:34] [WARNING] using devmapper backend
[10/14/2014 12:17:34] [WARNING] using public dns
[10/14/2014 12:17:34] [WARNING] nohup docker -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid -b ferry0 --dns 8.8.8.8 --dns 8.8.4.4 -e lxc -s devicemapper --icc=true 1>/var/lib/ferry/docker.log  2>&1 &
[10/14/2014 12:17:38] [WARNING] performing select rebuild (['cassandra-client', 'openmpi', 'ferry-base', 'openmpi-client', 'hadoop-client', 'hive-metastore', 'hive', 'mongodb', 'mongodb-client', 'titan', 'cassandra', 'spark', 'spark-client', 'hadoop', 'yarn', 'hadoop-base', 'gluster'])
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] transforming dockerfile
[10/14/2014 12:17:38] [WARNING] building image cassandra-client
[10/14/2014 12:17:38] [WARNING] docker -H=unix:////var/run/ferry.sock pull ubuntu:14.04
Pulling repository ubuntu
[10/14/2014 12:19:46] [WARNING] 2014/10/14 12:19:46 Get https://index.docker.io/v1/repositories/ubuntu/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:19:46] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/ferry-base:0.3.3.3
Pulling repository ferry/ferry-base
[10/14/2014 12:21:53] [WARNING] 2014/10/14 12:21:53 Get https://index.docker.io/v1/repositories/ferry/ferry-base/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:21:53] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/ferry-base:0.3.3.3 ferry/ferry-base:latest
[10/14/2014 12:21:53] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/cassandra:0.3.3.3
Pulling repository ferry/cassandra
[10/14/2014 12:24:00] [WARNING] 2014/10/14 12:24:00 Get https://index.docker.io/v1/repositories/ferry/cassandra/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:24:00] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/cassandra:0.3.3.3 ferry/cassandra:latest
[10/14/2014 12:24:00] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/titan:0.3.3.3
Pulling repository ferry/titan
[10/14/2014 12:26:08] [WARNING] 2014/10/14 12:26:08 Get https://index.docker.io/v1/repositories/ferry/titan/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:26:08] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/titan:0.3.3.3 ferry/titan:latest
[10/14/2014 12:26:08] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/cassandra-client:0.3.3.3
Pulling repository ferry/cassandra-client
[10/14/2014 12:28:15] [WARNING] 2014/10/14 12:28:15 Get https://index.docker.io/v1/repositories/ferry/cassandra-client/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:28:15] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/cassandra-client:0.3.3.3 ferry/cassandra-client:latest
[10/14/2014 12:28:15] [WARNING] building image openmpi
[10/14/2014 12:28:15] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/gluster:0.3.3.3
Pulling repository ferry/gluster
[10/14/2014 12:30:23] [WARNING] 2014/10/14 12:30:23 Get https://index.docker.io/v1/repositories/ferry/gluster/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:30:23] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/gluster:0.3.3.3 ferry/gluster:latest
[10/14/2014 12:30:23] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/openmpi:0.3.3.3
Pulling repository ferry/openmpi
[10/14/2014 12:32:30] [WARNING] 2014/10/14 12:32:30 Get https://index.docker.io/v1/repositories/ferry/openmpi/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:32:30] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/openmpi:0.3.3.3 ferry/openmpi:latest
[10/14/2014 12:32:30] [WARNING] building image ferry-base
[10/14/2014 12:32:30] [WARNING] building image openmpi-client
[10/14/2014 12:32:30] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/openmpi-client:0.3.3.3
Pulling repository ferry/openmpi-client
[10/14/2014 12:34:38] [WARNING] 2014/10/14 12:34:38 Get https://index.docker.io/v1/repositories/ferry/openmpi-client/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:34:38] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/openmpi-client:0.3.3.3 ferry/openmpi-client:latest
[10/14/2014 12:34:38] [WARNING] building image hadoop-client
[10/14/2014 12:34:38] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/hadoop-base:0.3.3.3
Pulling repository ferry/hadoop-base
[10/14/2014 12:36:45] [WARNING] 2014/10/14 12:36:45 Get https://index.docker.io/v1/repositories/ferry/hadoop-base/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:36:45] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hadoop-base:0.3.3.3 ferry/hadoop-base:latest
[10/14/2014 12:36:45] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/hadoop-client:0.3.3.3
Pulling repository ferry/hadoop-client
[10/14/2014 12:38:53] [WARNING] 2014/10/14 12:38:53 Get https://index.docker.io/v1/repositories/ferry/hadoop-client/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:38:53] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hadoop-client:0.3.3.3 ferry/hadoop-client:latest
[10/14/2014 12:38:53] [WARNING] building image hive-metastore
[10/14/2014 12:38:53] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/hive-metastore:0.3.3.3
Pulling repository ferry/hive-metastore
[10/14/2014 12:41:00] [WARNING] 2014/10/14 12:41:00 Get https://index.docker.io/v1/repositories/ferry/hive-metastore/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:41:00] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hive-metastore:0.3.3.3 ferry/hive-metastore:latest
[10/14/2014 12:41:00] [WARNING] tagging images hive-metastore
[10/14/2014 12:41:00] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hive-metastore:0.3.3.3 ferry/hive:0.3.3.3
[10/14/2014 12:41:00] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hive-metastore:latest ferry/hive:latest
[10/14/2014 12:41:00] [WARNING] building image mongodb
[10/14/2014 12:41:00] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/mongodb:0.3.3.3
Pulling repository ferry/mongodb
[10/14/2014 12:43:08] [WARNING] 2014/10/14 12:43:08 Get https://index.docker.io/v1/repositories/ferry/mongodb/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:43:08] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/mongodb:0.3.3.3 ferry/mongodb:latest
[10/14/2014 12:43:08] [WARNING] tagging images mongodb
[10/14/2014 12:43:08] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/mongodb:0.3.3.3 ferry/mongodb-client:0.3.3.3
[10/14/2014 12:43:08] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/mongodb:latest ferry/mongodb-client:latest
[10/14/2014 12:43:08] [WARNING] building image titan
[10/14/2014 12:43:08] [WARNING] building image cassandra
[10/14/2014 12:43:08] [WARNING] building image spark
[10/14/2014 12:43:08] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/spark:0.3.3.3
Pulling repository ferry/spark
[10/14/2014 12:45:15] [WARNING] 2014/10/14 12:45:15 Get https://index.docker.io/v1/repositories/ferry/spark/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:45:15] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/spark:0.3.3.3 ferry/spark:latest
[10/14/2014 12:45:15] [WARNING] tagging images spark
[10/14/2014 12:45:15] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/spark:0.3.3.3 ferry/spark-client:0.3.3.3
[10/14/2014 12:45:15] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/spark:latest ferry/spark-client:latest
[10/14/2014 12:45:15] [WARNING] building image hadoop
[10/14/2014 12:45:15] [WARNING] docker -H=unix:////var/run/ferry.sock pull ferry/hadoop:0.3.3.3
Pulling repository ferry/hadoop
[10/14/2014 12:47:23] [WARNING] 2014/10/14 12:47:23 Get https://index.docker.io/v1/repositories/ferry/hadoop/images: dial tcp 162.242.195.84:443: connection timed out

[10/14/2014 12:47:23] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hadoop:0.3.3.3 ferry/hadoop:latest
[10/14/2014 12:47:23] [WARNING] tagging images hadoop
[10/14/2014 12:47:23] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hadoop:0.3.3.3 ferry/yarn:0.3.3.3
[10/14/2014 12:47:23] [WARNING] docker -H=unix:////var/run/ferry.sock tag ferry/hadoop:latest ferry/yarn:latest
[10/14/2014 12:47:23] [WARNING] building image hadoop-base
[10/14/2014 12:47:23] [WARNING] building image gluster
[10/14/2014 12:47:23] [ERROR] images not built: ['mongodb', 'ferry-base', 'hadoop-base', 'hadoop', 'hadoop-client', 'hive-metastore', 'gluster', 'openmpi', 'openmpi-client', 'cassandra', 'cassandra-client', 'titan', 'spark']
[10/14/2014 12:47:23] [WARNING] stopping docker daemon
Some images were not installed. Please type 'ferry install' again.

Would be awesomesauce if Ferry supported launching Apache Spark instances.

There has been work done by the Spark team to support Spark on Docker: https://amplab.cs.berkeley.edu/2013/10/23/got-a-minute-spin-up-a-spark-cluster-on-your-laptop-with-docker/

Found this dependency that doesn't come installed in a vanilla Ubuntu instance: brctrl

root@sw-100:~# ferry server
[10/10/2014 02:45:36] [WARNING] Creating Ferry bridge 172.18.42.1/16
/bin/sh: 1: brctl: not found
Cannot find device "ferry0"
Cannot find device "ferry0"
[10/10/2014 02:45:36] [WARNING] checking for btrfs
[10/10/2014 02:45:36] [WARNING] using devmapper backend
[10/10/2014 02:45:36] [WARNING] using public dns
[10/10/2014 02:45:36] [WARNING] nohup docker -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid -b ferry0 --dns 8.8.8.8 --dns 8.8.4.4 -e lxc -s devicemapper --icc=true 1>/var/lib/ferry/docker.log 2>&1 &
[10/10/2014 02:45:39] [ERROR] Could not start ferry servers.

[10/10/2014 02:45:39] [ERROR] MongoDB images not found. Try executing 'ferry install'.
root@sw-100:#
root@sw-100:# brctl
The program 'brctl' is currently not installed. You can install it by typing:
apt-get install bridge-utils

Once bridge-utils is installed that failure goes away and I am just left with the missing MongoDB instances. Still working on where that comes from.

Hi,

i have installed the ferry vagrant box following the guide. When I execute sudo ferry server i got
the following error:

vagrant@vagrant-ubuntu-trusty-64:~$ sudo ferry server
[10/18/2014 09:36:41] [WARNING] Creating Ferry bridge 172.18.42.1/16
[10/18/2014 09:36:41] [WARNING] checking for btrfs
[10/18/2014 09:36:41] [WARNING] using devmapper backend
[10/18/2014 09:36:41] [WARNING] using public dns
[10/18/2014 09:36:41] [WARNING] nohup docker -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid -b ferry0 --dns 8.8.8.8 --dns 8.8.4.4 -e lxc -s devicemapper --icc=true 1>/var/lib/ferry/docker.log 2>&1 &
/bin/sh: 1: cannot create /var/lib/ferry/docker.log: Directory nonexistent
[10/18/2014 09:36:44] [ERROR] Could not start ferry servers.

[10/18/2014 09:36:44] [ERROR] MongoDB images not found. Try executing 'ferry install'.

I tried to execute ferry clean and then ferry install but ferry install get stucked.

Hi,
can I scale the cluster?
I have created a hadoop cluster using ferry，and can i add a new hadoop data node into the cluster?

Thanks!

During the installation of ferry, I came across the following severe error:

$ sudo ferry install
Either LXC is not installed or you are running an older version of LXC.
Only versions > 0.7.5 are supported.

$ sudo dpkg -s lxc-docker
Package: lxc-docker
Status: install ok installed
Priority: extra
Section: default
Installed-Size: 0
Maintainer: [email protected]
Architecture: amd64
Version: 1.6.2
Depends: lxc-docker-1.6.2
Description: Linux container runtime
Docker complements LXC with a high-level API which operates at the process
level. It runs unix processes with strong guarantees of isolation and
repeatability across servers.
Docker is a great building block for automating distributed systems:
large-scale web deployments, database clusters, continuous deployment systems,
private PaaS, service-oriented architectures, etc.
License: Apache-2.0
Vendor: none
Homepage: http://www.docker.com/

Maybe I got it wrong, but isn't lxc-docker-1.6.2 slightly newer than lxc-docker-0.7.5?

For several reasons I can NOT downgrade to anything less than lxc-docker-1.6.x .

Any ideas?

hi,
i have found that a couple of (light)changes need to be done to make 'ferry install' work on ubuntu12.04

• the only lxc version >0.7.5 available for ubuntu 12.04 is '1.0.0.alpha1' (in backports repository), so this needs to be added to 'install.py' script:

in lines 57-58 remove the non-integer bit of 'lxc-version' response (otherwise the script will fail as 'alpha1' is not an int)
As a quick hack i have just added 'replace('.alpha1','')' to 'output'

• in line 241:
  there no 'explanation' attribute for OSError, just remove it or replace with a valid attribute such as strerror

done!

Commands do not seem to have the assumed affect: how do you trouble shoot what is going on in this sequence:

root@sw-100:# ferry ps
UUID Storage Compute Connectors Status Base Time
---- ------- ------- ---------- ------ ---- ----
sa-8ee65f68 [] [] [] buildin g mongo
sa-b791c821 [] [] [] buildin g spark
sa-ecc761ea [u'se-a5298b6d'] [' '] [u'se-4035a591'] running cassandra
root@sw-100:# ferry ps
UUID Storage Compute Connectors Status Base Time
---- ------- ------- ---------- ------ ---- ----
sa-8ee65f68 [] [] [] building mongo
sa-b791c821 [] [] [] building spark
sa-ecc761ea [u'se-a5298b6d'] [' '] [u'se-4035a591'] running cassandra
root@sw-100:~# ferry stop sa-8ee65f68

root@sw-100:# ferry ps
UUID Storage Compute Connectors Status Base Time
---- ------- ------- ---------- ------ ---- ----
sa-8ee65f68 [] [] [] building mongo
sa-b791c821 [] [] [] building spark
sa-ecc761ea [u'se-a5298b6d'] [' '] [u'se-4035a591'] running cassandra
root@sw-100:# ferry rm sa-8ee65f68

root@sw-100:~# ferry ps
UUID Storage Compute Connectors Status Base Time
---- ------- ------- ---------- ------ ---- ----
sa-8ee65f68 [] [] [] building mongo
sa-b791c821 [] [] [] building spark
sa-ecc761ea [u'se-a5298b6d'] [' '] [u'se-4035a591'] running cassandra

In the error log of a failed "ferry server" command, it showed a failure on lxc-start not being found. That manifested other errors with misdirecting explanations, in particular:
[10/10/2014 04:00:11] [ERROR] MongoDB images not found. Try executing 'ferry install'.

here was the sequence of events:
root@sw-100:~# ferry server
[10/10/2014 04:00:08] [WARNING] checking for btrfs
[10/10/2014 04:00:08] [WARNING] using devmapper backend
[10/10/2014 04:00:08] [WARNING] using public dns
[10/10/2014 04:00:08] [WARNING] nohup docker -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid -b ferry0 --dns 8.8.8.8 --dns 8.8.4.4 -e lxc -s devicemapper --icc=true 1>/var/lib/ferry/docker.log 2>&1 &
[10/10/2014 04:00:11] [ERROR] Could not start ferry servers.

[10/10/2014 04:00:11] [ERROR] MongoDB images not found. Try executing 'ferry install'.
root@sw-100:# ferry install
/bin/sh: 1: lxc-start: not found
Traceback (most recent call last):
File "/usr/local/bin/ferry", line 9, in
load_entry_point('ferry==0.3.3.3', 'console_scripts', 'ferry')()
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 750, in main
msg = cli.dispatch_cmd(c, all_cmds[c], options)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 678, in dispatch_cmd
msg = self.installer.install(args, options)
File "/usr/local/lib/python2.7/dist-packages/ferry/install.py", line 345, in install
if not _supported_lxc():
File "/usr/local/lib/python2.7/dist-packages/ferry/install.py", line 154, in _supported_lxc
supported, _ = _supported_tuple(output.strip(), (0, 7, 5))
File "/usr/local/lib/python2.7/dist-packages/ferry/install.py", line 141, in _supported_tuple
ver = tuple(map(int, cleaned))
TypeError: int() argument must be a string or a number, not 'NoneType'
root@sw-100:# lxc-start
The program 'lxc-start' is currently not installed. You can install it by typing:
apt-get install lxc

After installing lxc and reissuing 'ferry install', and 'ferry server', I have a working ferry system: yay!

Hi all,

I hope this is an error and not a case of RTFM. I have started a ferry.box via vagrant on virtualbox. I have then started ferry server and ferry start spark. I have not changed any config, but rather followed the getting started quickstart guide. When I try to ferry ssh into the spark client/connector I receive:

vagrant@vagrant-ubuntu-trusty-64:~$ ferry ps
       UUID            Storage               Compute              Connectors         Status      Base     Time
       ----            -------               -------              ----------         ------      ----     ----
   sa-b8653a1e     [u'se-45981657']     [[u'se-7c7438d3']]     [u'se-bfbd5492']     running     spark
vagrant@vagrant-ubuntu-trusty-64:~$ ferry ssh sa-b8653a1e
[12/01/2014 02:55:17] [WARNING] ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /usr/local/lib/python2.7/dist-packages/ferry/data/key/insecure_ferry_key.pem [email protected]
Warning: Permanently added '172.18.0.10' (ECDSA) to the list of known hosts.
[email protected]'s password:

I think the operation is meant to be passwordless, hence the -i option to ssh with the insecure key.
I can see the docker containers, jvm's running hadoop etc in the list of processes. When starting or stopping the "cluster", I do receive many warning of ssh:

WARNING:root:LC_ALL=C && ssh -o ConnectTimeout=20 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /usr/local/lib/python2.7/dist-packages/ferry/data/key/insecure_ferry_key.pem -t -t [email protected] '/service/sbin/startnode halt'
WARNING:root:com msg: Warning: Permanently added '172.18.0.6' (ECDSA) to the list of known hosts.

So the ssh seems to work there, as it starts or stops the respective services in the docker container via ssh.

How can I fix it.

Keep up the good work.

Cheers
Carsten

When I follow the directions to start up a ferry VM using Vagrant on OS X, I get the following errors:

(yoshi_env)14109fd57a3f:ferry rqa267$ vagrant --version
Vagrant 1.7.2
(yoshi_env)14109fd57a3f:ferry rqa267$ vagrant plugin install vagrant-vbguest
Installing the 'vagrant-vbguest' plugin. This can take a few minutes...
Installed the plugin 'vagrant-vbguest (0.10.0)'!
(yoshi_env)14109fd57a3f:ferry rqa267$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'opencore/ferry'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: ferry_default_1427391446311_81240
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if its present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
GuestAdditions versions on your host (4.3.18) and guest (4.3.10) do not match.
stdin: is not a tty
 * Stopping VirtualBox Additions
   ...done.
stdin: is not a tty
Reading package lists...
Building dependency tree...
The following packages will be REMOVED:
  virtualbox-guest-dkms* virtualbox-guest-utils* virtualbox-guest-x11*
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 12.1 MB disk space will be freed.
(Reading database ... 64468 files and directories currently installed.)
Removing virtualbox-guest-dkms (4.3.10-dfsg-1) ...

-------- Uninstall Beginning --------
Module:  virtualbox-guest
Version: 4.3.10
Kernel:  3.13.0-36-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

vboxguest.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.13.0-36-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


vboxsf.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.13.0-36-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.


vboxvideo.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.13.0-36-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....

DKMS: uninstall completed.

------------------------------
Deleting module version: 4.3.10
completely from the DKMS tree.
------------------------------
Done.
Removing virtualbox-guest-x11 (4.3.10-dfsg-1) ...
Purging configuration files for virtualbox-guest-x11 (4.3.10-dfsg-1) ...
Removing virtualbox-guest-utils (4.3.10-dfsg-1) ...
Purging configuration files for virtualbox-guest-utils (4.3.10-dfsg-1) ...
Processing triggers for man-db (2.6.7.1-1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
stdin: is not a tty
Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-3.13.0-36-generic is already the newest version.
dkms is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Copy iso file /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
stdin: is not a tty
mount: block device /tmp/VBoxGuestAdditions.iso is write-protected, mounting read-only
Installing Virtualbox Guest Additions 4.3.18 - guest version is 4.3.10
stdin: is not a tty
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.3.18 Guest Additions for Linux............
VirtualBox Guest Additions installer
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox DKMS kernel modules ...done.
Removing existing VirtualBox non-DKMS kernel modules ...done.
Building the VirtualBox Guest Additions kernel modules
The headers for the current running kernel were not found. If the following
module compilation fails then this could be the reason.

Building the main Guest Additions module ...fail!
(Look at /var/log/vboxadd-install.log to find out what went wrong)
Doing non-kernel setup of the Guest Additions ...done.
Installing the Window System drivers
Could not find the X.Org or XFree86 Window System, skipping.
An error occurred during installation of VirtualBox Guest Additions 4.3.18. Some functionality may not work as intended.
In most cases it is OK that the "Window System drivers" installation failed.
stdin: is not a tty
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => /Users/rqa267/ferry
Failed to mount folders in Linux guest. This is usually because
the "vboxsf" file system is not available. Please verify that
the guest additions are properly installed in the guest and
can work properly. The command attempted was:

mount -t vboxsf -o uid=`id -u vagrant`,gid=`getent group vagrant | cut -d: -f3` vagrant /vagrant
mount -t vboxsf -o uid=`id -u vagrant`,gid=`id -g vagrant` vagrant /vagrant

The error output from the last command was:

stdin: is not a tty
/sbin/mount.vboxsf: mounting failed with the error: No such device

I have tried the fixes suggested here:

http://stackoverflow.com/questions/22717428/vagrant-error-failed-to-mount-folders-in-linux-guest

but they did not work.

Let me know if you need any further information beyond these logs.

While running sudo ferry server:

[09/14/2014 06:43:22] [WARNING] checking for btrfs
[09/14/2014 06:43:22] [WARNING] using devmapper backend
[09/14/2014 06:43:22] [WARNING] using public dns
[09/14/2014 06:43:22] [WARNING] nohup docker-ferry -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid --dns 8.8.8.8 --dns 8.8.4.4 -s devicemapper --icc=true 1>/var/lib/ferry/docker.log  2>&1 &
[09/14/2014 06:43:25] [WARNING] cleaning previous mongo resources
[09/14/2014 06:43:25] [WARNING] docker-ferry -H=unix:////var/run/ferry.sock ps | grep ferry/mongodb | awk '{print $1}' | xargs docker-ferry -H=unix:////var/run/ferry.sock stop
[09/14/2014 06:43:25] [WARNING] docker-ferry -H=unix:////var/run/ferry.sock run -privileged -d -h ferrydb  -expose 27017 -v /var/lib/ferry/mongo:/service/data/ -v /var/lib/ferry/mongolog:/service/logs/ -v /usr/local/lib/python2.7/dist-packages/ferry/data/key:/service/keys -e "KEY=insecure_ferry_key" ferry/mongodb /service/sbin/startnode init
[09/14/2014 06:43:27] [WARNING] docker-ferry -H=unix:////var/run/ferry.sock inspect be599d5c7883c1e30a56140d6622653ec2d9e9a95065460de14487feacffeaba
[09/14/2014 06:43:30] [WARNING] ssh -o ConnectTimeout=20 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /usr/local/lib/python2.7/dist-packages/ferry/data/key/insecure_ferry_key.pem -t -t [email protected] 'chown -R ferry:docker /service/data/'
[09/14/2014 06:43:30] [WARNING] com msg: Warning: Permanently added '10.1.0.2' (ECDSA) to the list of known hosts.
Connection to 10.1.0.2 closed.

[09/14/2014 06:43:30] [WARNING] ssh -o ConnectTimeout=20 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /usr/local/lib/python2.7/dist-packages/ferry/data/key/insecure_ferry_key.pem -t -t [email protected] 'chown -R ferry:docker /service/logs/'
[09/14/2014 06:43:30] [WARNING] com msg: Warning: Permanently added '10.1.0.2' (ECDSA) to the list of known hosts.
Connection to 10.1.0.2 closed.

[09/14/2014 06:43:30] [WARNING] ssh -o ConnectTimeout=20 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /usr/local/lib/python2.7/dist-packages/ferry/data/key/insecure_ferry_key.pem -t -t [email protected] '/service/sbin/startnode login'
[09/14/2014 06:43:31] [WARNING] com msg: Warning: Permanently added '10.1.0.2' (ECDSA) to the list of known hosts.
Connection to 10.1.0.2 closed.

Traceback (most recent call last):
  File "/usr/local/bin/ferry", line 9, in <module>
    load_entry_point('ferry==0.3.1.3', 'console_scripts', 'ferry')()
  File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 750, in main
    msg = cli.dispatch_cmd(c, all_cmds[c], options)
  File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 690, in dispatch_cmd
    self.installer.start_web(options)
  File "/usr/local/lib/python2.7/dist-packages/ferry/install.py", line 472, in start_web
    config_dirs, entry_point = self.mongo.apply(mongoconf, [s])
  File "/usr/local/lib/python2.7/dist-packages/ferry/config/mongo/mongoconfig.py", line 166, in apply
    login_info = json.loads(str(output))
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

By adding some prints, I found that the content of output is:

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
{"user":"e4fda1b8-9b9f-42ae-9297-#####","pass":"05ead4b4-dcef-42c6-9b03-####"}

(#s are added by me)

I got the following error when i use snapshots.

vagrant@vagrant-ubuntu-trusty-64:$ ferry snapshot sa-318f8f6b
WARNING:root:docker -H=unix:////var/run/ferry.sock inspect 2c4b2dde6c48c373225d3e38d284d5ba543caa1b4f86ae71ca714ada25718ba5
vagrant@vagrant-ubuntu-trusty-64:$ WARNING:root:docker -H=unix:////var/run/ferry.sock commit -run='{"Cmd": ["/service/sbin/startnode", "init"]}' 2c4b2dde6c48c373225d3e38d284d5ba543caa1b4f86ae71ca714ada25718ba5 ferry/openmpi-client-sa-318f8f6b-control-0:SNAPSHOT-0

vagrant@vagrant-ubuntu-trusty-64:~$ ferry snapshots
ERROR:main:Exception on /snapshots [GET]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/usr/local/lib/python2.7/dist-packages/ferry/http/httpapi.py", line 708, in snapshots
return docker.query_snapshots()
File "/usr/local/lib/python2.7/dist-packages/ferry/docker/manager.py", line 452, in query_snapshots
time = v['snapshot_ts'].strftime("%m/%w/%Y (%I:%M %p)")
AttributeError: 'unicode' object has no attribute 'strftime'
ERROR:tornado.access:500 GET /snapshots (127.0.0.1) 14.07ms
Traceback (most recent call last):
File "/usr/local/bin/ferry", line 9, in
load_entry_point('ferry==0.3.3.4', 'console_scripts', 'ferry')()
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 775, in main
msg = cli.dispatch_cmd(c, all_cmds[c], options)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 699, in dispatch_cmd
return self._list_snapshots()
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 477, in _list_snapshots
json_reply = json.loads(res.text)
File "/usr/lib/python2.7/json/init.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

I'm still getting the same errors after "sudo ferry install:

Traceback (most recent call last):
File "/usr/local/bin/ferry", line 9, in
load_entry_point('ferry==0.2.0', 'console_scripts', 'ferry')()
File "/usr/local/lib/python2.7/dist-packages/ferry-0.2.0-py2.7.egg/ferry/cli/cli.py", line 529, in main
msg = cli.dispatch_cmd(c, all_cmds[c], options)
File "/usr/local/lib/python2.7/dist-packages/ferry-0.2.0-py2.7.egg/ferry/cli/cli.py", line 463, in dispatch_cmd
msg = self.installer.install(args, options)
File "/usr/local/lib/python2.7/dist-packages/ferry-0.2.0-py2.7.egg/ferry/install.py", line 137, in install
if not _supported_lxc():
File "/usr/local/lib/python2.7/dist-packages/ferry-0.2.0-py2.7.egg/ferry/install.py", line 58, in _supported_lxc
ver = tuple(map(int, (output.strip().split(".")[:3])))
ValueError: invalid literal for int() with base 10: '0-rc1'

Here is my output:

$ sudo ferry install
Password:
03/14/2014 07:27:49 PM checking for btrfs
cat: /etc/mtab: No such file or directory
03/14/2014 07:27:50 PM nohup docker-ferry -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid --icc=true 1>/var/lib/ferry/docker.log  2>&1 &
03/14/2014 07:27:52 PM using key directory ~/anaconda/envs/FerryEnv/lib/python2.7/site-packages/ferry/data/key
Traceback (most recent call last):
  File "~/anaconda/envs/FerryEnv/bin/ferry", line 9, in <module>
    load_entry_point('ferry==0.1.20', 'console_scripts', 'ferry')()
  File "~/anaconda/envs/FerryEnv/lib/python2.7/site-packages/ferry/cli/cli.py", line 443, in main
    msg = cli.dispatch_cmd(c, all_cmds[c], options)
  File "~/anaconda/envs/FerryEnv/lib/python2.7/site-packages/ferry/cli/cli.py", line 379, in dispatch_cmd
    msg = self.installer.install(args)
  File "~/anaconda/envs/FerryEnv/lib/python2.7/site-packages/ferry/install.py", line 106, in install
    _touch_file(DEFAULT_DOCKER_KEY, GLOBAL_KEY_DIR, root=True)
  File "~/anaconda/envs/FerryEnv/lib/python2.7/site-packages/ferry/install.py", line 47, in _touch_file
    gid = grp.getgrnam("docker").gr_gid
KeyError: 'getgrnam(): name not found: docker'

I am using OS X 10.9 and installed Docker and Boot2Docker using Homebrew.

I tried to install on Debian following the procedure with no problems until ferry install. When i try ferry install:

sudo ferry install

Traceback (most recent call last):
File "/usr/local/bin/ferry", line 9, in
load_entry_point('ferry==0.1.28', 'console_scripts', 'ferry')()
File "/usr/local/lib/python2.7/dist-packages/ferry-0.1.28-py2.7.egg/ferry/cli/cli.py", line 529, in main
msg = cli.dispatch_cmd(c, all_cmds[c], options)
File "/usr/local/lib/python2.7/dist-packages/ferry-0.1.28-py2.7.egg/ferry/cli/cli.py", line 463, in dispatch_cmd
msg = self.installer.install(args, options)
File "/usr/local/lib/python2.7/dist-packages/ferry-0.1.28-py2.7.egg/ferry/install.py", line 137, in install
if not _supported_lxc():
File "/usr/local/lib/python2.7/dist-packages/ferry-0.1.28-py2.7.egg/ferry/install.py", line 58, in _supported_lxc
ver = tuple(map(int, (output.strip().split("."))))
ValueError: invalid literal for int() with base 10: '0-rc1'

Hi,

I followed the instruction in http://ferry.opencore.io/en/latest/install.html and http://ferry.opencore.io/en/latest/examples/openmpi.html to try ferry with openmpi. But it failed as follow:

root@ip-10-179-155-66:~/Ferry#  ferry --key=/root/.ssh start openmpi
Traceback (most recent call last):
  File "/usr/local/bin/ferry", line 9, in <module>
    load_entry_point('ferry==0.2.2', 'console_scripts', 'ferry')()
  File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 809, in main
    msg = cli.dispatch_cmd(c, all_cmds[c], options)
  File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 684, in dispatch_cmd
    self._check_ssh_key()
  File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 620, in _check_ssh_key
    keydir, _ = self._read_key_dir(root)
  File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 604, in _read_key_dir
    with open(keydir, 'r') as f: 
IOError: [Errno 2] No such file or directory: '/var/lib/ferry/keydir'

It seems that the keydir is incorrect.

You should reproduce the example on AWS from in US-EAST with AMI: ami-8c679ce4

As log analytics are par for the course in distributed computing, it would be productive if this stack is readily available (and integrated into) the ferry application set.

There are several systems and applications that I think Ferry should support. For example:

• PostGres-XL
• Prediction.io
• Kafka

It would be great to standardize the ability to add/configure additional images, and actually get a few of these images in place.

Hi
I followed the instruction in http://ferry.opencore.io/en/latest/install.html ,But it failed as follow:
root@test-VirtualBox:# ferry server
[08/12/2014 08:30:44] [WARNING] checking for btrfs
[08/12/2014 08:30:44] [WARNING] using public dns
[08/12/2014 08:30:44] [WARNING] nohup docker-ferry -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid --dns 8.8.8.8 --dns 8.8.4.4 --icc=true 1>/var/lib/ferry/docker.log 2>&1 &
[08/12/2014 08:30:46] [WARNING] using key directory tmp:///usr/local/lib/python2.7/dist-packages/ferry/data/key
[08/12/2014 08:30:46] [WARNING] cleaning previous mongo resources
[08/12/2014 08:30:46] [WARNING] docker-ferry -H=unix:////var/run/ferry.sock ps | grep ferry/mongodb | awk '{print $1}' | xargs docker-ferry -H=unix:////var/run/ferry.sock stop
[08/12/2014 08:30:46] [WARNING] using key directory tmp:///usr/local/lib/python2.7/dist-packages/ferry/data/key
Traceback (most recent call last):
File "/usr/local/bin/ferry", line 9, in
load_entry_point('ferry==0.2.3.1', 'console_scripts', 'ferry')()
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 837, in main
msg = cli.dispatch_cmd(c, all_cmds[c], options)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 769, in dispatch_cmd
self.installer.start_web(options)
File "/usr/local/lib/python2.7/dist-packages/ferry/install.py", line 417, in start_web
self.cli._check_ssh_key(root=True, server=True)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 667, in _check_ssh_key
keydir, proto = self._read_key_dir(options=options, root=root)
File "/usr/local/lib/python2.7/dist-packages/ferry/cli/cli.py", line 612, in _read_key_dir
with open(keydir, 'r') as f:
IOError: [Errno 2] No such file or directory: '/home/test/.ferry/rootdir'
root@test-VirtualBox:# ls /home/test/.ferry/
keydir
root@test-VirtualBox:~#

I have several applications that are 'hung' and none of the ferry clean up commands seem to have an affect on the state of these applications (stop/kill/rm).

What is the proper way to get the ferry server back into a sane state?

i have a laptop where locale is es_ES and the 'sudo ferry start' fails because in _get_gateway method of file docker/fabric.py has two conflicting lines, one to fetch the drydock0 ip address, and other to fetch the drydock0 gateway

cmd = "ifconfig drydock0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'"
[...]
cmd = "ifconfig drydock0 | grep 'inet addr:' | cut -d: -f4 | awk '{ print $1}'"

Those two fail because in es_ES locale, the 'inet addr:' string becomes 'Direc. inet:'
but this can be solved by prefixing the command with the locale to be used:

cmd = "LC_MESSAGES=C ifconfig drydock0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'"
[...]
cmd = "LC_MESSAGES=C ifconfig drydock0 | grep 'inet addr:' | cut -d: -f4 | awk '{ print $1}'"

$ sudo uname -a
Linux dell2950 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ sudo ferry server
[05/18/2015 07:15:16] [WARNING] checking for btrfs
[05/18/2015 07:15:16] [WARNING] using devmapper backend
[05/18/2015 07:15:16] [WARNING] using public dns
[05/18/2015 07:15:16] [WARNING] nohup docker -d -H=unix:////var/run/ferry.sock -g=/var/lib/ferry -p=/var/run/ferry.pid -b ferry0 --dns 8.8.8.8 --dns 8.8.4.4 -e lxc -s devicemapper --icc=true 1>/var/lib/ferry/docker.log 2>&1 &
[05/18/2015 07:15:19] [WARNING] cleaning previous mongo resources
[05/18/2015 07:15:19] [WARNING] docker -H=unix:////var/run/ferry.sock ps | grep ferry/mongodb | awk '{print $1}' | xargs docker -H=unix:////var/run/ferry.sock stop
[05/18/2015 07:15:19] [WARNING] docker -H=unix:////var/run/ferry.sock run -privileged -d -h ferrydb -v /var/lib/ferry/mongo:/service/data/ -v /var/lib/ferry/mongolog:/service/logs/ -v /usr/local/lib/python2.7/dist-packages/ferry/data/key:/service/keys -e "KEY=insecure_ferry_key" ferry/mongodb /service/sbin/startnode init
[05/18/2015 07:15:20] [WARNING] docker -H=unix:////var/run/ferry.sock inspect 9f01dd5845542b8f94bd70174d93143a305210a3dbd6b5cec33a28f3e0e04408
[05/18/2015 07:15:20] [ERROR] container for ferry/mongodb is not running
[05/18/2015 07:15:20] [ERROR] Could not start MongoDB image

$ sudo ferry info
[05/18/2015 07:15:26] [ERROR] could not connect to ferry server
It appears Ferry servers are not running.
Type sudo ferry server and try again.

/var/lib/ferry$ less docker.log
[...]
time="2015-05-18T19:20:05+02:00" level=warning msg="Your kernel does not support OOM notifications: open /sys/fs/cgroup/memory/user.slice/user-1000.slice/session-12.scope/lxc/6eaa53e56c14fde3d93b0e6a88085ecb4b4386588de73a68e2d46b124af64a93/memory.oom_control: no such file or directory"
[...]
time="2015-05-18T19:37:07+02:00" level=warning msg="Your kernel does not support OOM notifications: open /sys/fs/cgroup/memory/user.slice/user-1000.slice/session-15.scope/lxc/d969e66552543f0158996e57e01d2387a98d891ce0bd0a01e1f75569d4a9f76f/memory.oom_control: no such file or directory"

Can I configure this somewhere? Or is this a "feature" ;-)

Following through the tutorial listed http://ferry.opencore.io/en/latest/examples/hadoop.html#hadoop.
Everything works up until I do ferry ssh sa. It requires a username and password. I cannot find these listed anywhere in the documentation. Thank you for your time.

AWS support is pretty basic at this point. Some enhancements that might be useful include:

• Spot instances
• Multi-region support (for Cassandra, etc.)
• Fine-grained image selection (different images for compute, storage, etc.)

Starting with an up to date Ubuntu 14.04 environment, trying to install ferry through ferry-dust

stillwater@sw-100:$ sudo pip install -U ferry
[sudo] password for stillwater:
Downloading/unpacking ferry
Downloading ferry-0.3.3.3.tar.gz (208kB): 208kB downloaded
Running setup.py (path:/tmp/pip_build_root/ferry/setup.py) egg_info for packag e ferry
...
Installing collected packages: ferry, boto, Flask, pymongo, python-novaclient, python-neutronclient,uests, sh, six, Sphinx, sphinxcontrib-httpdomain, termcolor, threading2, tornado, Werkzeug, Jinja2, n, pbr, python-keystoneclient, cliff, netaddr, Pygments, docutils, certifi, backports.ssl-match-host pip, oslo.config, stevedore, pyparsing, cmd2
Running setup.py install for ferry
....
.... many compilation warnings
....
Successfully installed ferry boto Flask pymongo python-novaclient python-neutronclient hinx sphinxcontrib-httpdomain termcolor threading2 tornado Werkzeug Jinja2 itsdangerou t cliff netaddr Pygments docutils certifi backports.ssl-match-hostname markupsafe pytz cmd2
Cleaning up...
stillwater@sw-100:$ which ferry-dust
/usr/local/bin/ferry-dust
stillwater@sw-100:$ export FERRY_DIR=/var/lib/ferry
stillwater@sw-100:$ sudo ferry-dust install
Error: No such image or container: ferry/ferry-server
pulling the ferry image
Pulling repository ferry/ferry-server
4498499b4e65: Pulling dependent layers
97363f6e9079: Download complete
249a44f2f965: Download complete
511136ea3c5a: Download complete
5e66087f3ffe: Download complete
4d26dd3ebc1c: Download complete
d4010efcfd86: Download complete
99ec81b80c55: Download complete
4498499b4e65: Download complete
fcadef5d73c3: Download complete
4ebaec4748e1: Download complete
077fd1caf696: Download complete
c45967030e32: Download complete
888f62d73c9d: Download complete
d63d7f192762: Download complete
1e7811cc4fca: Download complete
22ff4d9f52c1: Download complete
1c7f181e78b9: Download complete
b18d0a2076a1: Download complete
9f676bd305a4: Download complete
67b66f26d423: Download complete
25c4824a5268: Download complete
8b1c48305638: Download complete
6030677eb5d2: Download complete
c900195dcbf3: Download complete
90177ae68092: Download complete
6b4e8a7373fe: Download complete
3f2d795e3309: Download complete
f8a57046c3fc: Download complete
c2c0da7cede7: Download complete
4ca1aa06d7c7: Download complete
1a1d7ef08dcc: Download complete
cc400379cb68: Download complete
20af62704915: Download complete
db4fe0f2d561: Download complete
07740b15b3fd: Download complete
6753e3778944: Download complete
67a2a12aa03f: Download complete
af09a11cf5d4: Download complete
798a83a04a19: Download complete
943a7afc77c7: Download complete
de95eeb92afc: Download complete
ab04cc56c1b2: Download complete
25098d3c6c7a: Download complete
7533a39e7d69: Download complete
24494120dbe9: Download complete
67e689edc7a6: Download complete
9d8ebd110cd4: Download complete
1f7dd8a6fafc: Download complete
e482b718265f: Download complete
5cba33de10c6: Download complete
64e27cc09237: Download complete
you must set the FERRY_DIR environment variable
stillwater@sw-100:$ echo $FERRY_DIR
/var/lib/ferry
stillwater@sw-100:$ sudo ferry-dust start
found ferry image, proceeding
you must set the FERRY_DIR environment variable
stillwater@sw-100:~

So FERRY_DIR is set, but doesn't propagate?