monetdb / monetdb-r-docker Goto Github PK

Docker container for MonetDB with R

Shell 28.20% Dockerfile 71.80%

monetdb-r-docker's Introduction

The MonetDB Database System

The MonetDB database was originally developed by the CWI database research group (see https://www.monetdb.org/). Development has now shifted to the spin-off company MonetDB Solutions.

Via the MonetDB project we have brought the MonetDB system in open source, where it is accessible at https://www.monetdb.org/easy-setup/. Even though development happens mostly in a company, the MonetDB database system will remain open source. It is available under the Mozilla Public License 2.0.

The MonetDB database system is a high-performance database kernel for query-intensive applications. The MonetDB source can be found at our Mercurial server. There is also a github mirror that is updated once an hour.

Building

MonetDB is built using the cmake program. It is recommended to build in a directory that is not inside the source tree. In order to build, use the following commands when inside your build directory:

cmake [options] /path/to/monetdb/source
cmake --build .
cmake --build . --target install

In order to install into a different directory than the default /usr/local, add the option -DCMAKE_INSTALL_PREFIX:

cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/monetdb /path/to/monetdb/source
cmake --build .
cmake --build . --target install

Build Options

There are many options that can be used to select how MonetDB is to be built. Options can be turned ON or OFF using a -D flag on the first of the cmake command lines. Except when specified otherwise, options are ON when the relevant libraries can be found. Available options are:

Option	Explanation
ASSERT	Enable asserts (default=ON for development sources, OFF for tarball installation)
CINTEGRATION	Enable support for C UDFs (default=ON except on Windows)
CMAKE_SUMMARY	Show a summary of the cmake configuration (for debug purposes, default=OFF)
CMAKE_UNITTESTS	Build and run the unittest for the build system (default=OFF)
FITS	Enable support for FITS
GEOM	Enable support for geom module (using libgeos library)
INT128	Enable support for 128-bit integers (if compiler supports them)
NETCDF	Enable support for netcdf
ODBC	Compile the MonetDB ODBC driver
PY3INTEGRATION	Enable support for Python 3 integration into MonetDB
RINTEGRATION	Enable support for R integration into MonetDB
SANITIZER	Enable support for the GCC address sanitizer (default=OFF)
SHP	Enable support for ESRI Shapefiles
STRICT	Enable strict compiler flags (default=ON for development sources, OFF for tarball installation)
TESTING	Enable support for testing
WITH_BZ2	Include bz2 support
WITH_CMOCKA	Include cmocka support (default=OFF)
WITH_CURL	Include curl support
WITH_LZ4	Include lz4 support
WITH_LZMA	Include lzma support
WITH_OPENSSL	Include TLS support (secure client/server connection)
WITH_PCRE	Include pcre support
WITH_PROJ	Include proj support
WITH_READLINE	Include readline support
WITH_VALGRIND	Include valgrind support
WITH_XML2	Include xml2 support
WITH_ZLIB	Include zlib support

Required and Optional Packages

On Fedora, the following packages are required: bison, cmake, gcc, pkgconf, python3.

The following packages are optional but recommended: bzip2-devel, lz4-devel, openssl-devel, pcre-devel, readline-devel, xz-devel, zlib-devel.

The following packages are optional: cfitsio-devel, gdal-devel, geos-devel, libasan, libcmocka-devel, libcurl-devel, libxml2-devel, netcdf-devel, proj-devel, python3-devel, python3-numpy, R-core-devel, unixODBC-devel, valgrind-devel.

On Ubuntu and Debian the following packages are required: bison, cmake, gcc, pkg-config, python3.

The following packages are optional but recommended: libbz2-dev, liblz4-dev, libpcre3-dev, libreadline-dev, liblzma-dev, libssl-dev, zlib1g-dev.

The following packages are optional: libasan5, libcfitsio-dev, libcmocka-dev, libcurl4-gnutls-dev, libgdal-dev, libgeos-dev, libnetcdf-dev, libproj-dev, libxml2-dev, python3-dev, python3-numpy, r-base-dev, unixodbc-dev, valgrind.

cmake must be at least version 3.12, python must be at least version 3.5.

Bugs

We of course hope there aren't any, but if you do find one, you can report bugs in our github repository.

Please note that we do not accept github Pull Requests. See the developers page for instructions.

Copyright Notice

SPDX-License-Identifier: MPL-2.0

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

monetdb-r-docker's People

Contributors

Stargazers

Watchers

Forkers

arjenderijke kutsurak kleopatra999 davidmoraisferreira inkrement zysam maxplanck-ie saiqi usbrandon madgik mudam

monetdb-r-docker's Issues

database crashes when a user-defined function is executed

Hi, I followed this suggestion to create a sampling function:

CREATE FUNCTION my_sample(p_date date, st char(2)) RETURNS TABLE(value float)
BEGIN
  RETURN TABLE(SELECT value 
  FROM payments 
  WHERE payment_date = p_date 
  AND state = st 
  SAMPLE 400);
END;

When I execute the query below for the first time, no results appear. On the second time, the docker quits.

select * from my_sample('2011-01-01', 'DF');

Payments table was created using this DDL:

CREATE TABLE "payments" (
	"payment_date" date,
	"payee" bigint,
	"state" char(2),
	"value" decimal(7,2)
);

The output of /var/log/monetdb/merovingian.log is:

2017-04-21 16:28:29 MSG db[151]: # MonetDB 5 server v11.25.15 "Dec2016-SP3"
2017-04-21 16:28:29 MSG db[151]: # Serving database 'db', using 8 threads
2017-04-21 16:28:29 MSG db[151]: # Compiled for x86_64-redhat-linux-gnu/64bit with 128bit integers
2017-04-21 16:28:29 MSG db[151]: # Found 15.654 GiB available main-memory.
2017-04-21 16:28:29 MSG db[151]: # Copyright (c) 1993-July 2008 CWI.
2017-04-21 16:28:29 MSG db[151]: # Copyright (c) August 2008-2017 MonetDB B.V., all rights reserved
2017-04-21 16:28:29 MSG db[151]: # Visit http://www.monetdb.org/ for further information
2017-04-21 16:28:29 MSG db[151]: # Listening for UNIX domain connection requests on mapi:monetdb:///var/monetdb5/dbfarm/db/.mapi.sock
2017-04-21 16:28:29 MSG db[151]: # MonetDB/GIS module loaded
2017-04-21 16:28:29 MSG db[151]: # MonetDB/SQL module loaded
2017-04-21 16:28:29 ERR db[151]: Traceback (most recent call last):
2017-04-21 16:28:29 ERR db[151]: File "<string>", line 1, in <module>
2017-04-21 16:28:29 ERR db[151]: ImportError: No module named numpy
2017-04-21 16:28:29 MSG db[151]: !MALException:pyapi.eval:Failed to initialize embedded python
2017-04-21 16:29:38 ERR db[151]: could not find (null).L7
2017-04-21 16:29:38 ERR db[151]: L6.L6
2017-04-21 16:29:39 MSG merovingian[9]: database 'db' (151) has crashed (dumped core)
2017-04-21 16:29:45 MSG merovingian[9]: database 'db' has crashed after start on 2017-04-21 16:28:29, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.40 0.13 (4-0=4)
2017-04-21 16:29:45 MSG merovingian[9]: proxying client (local) for database 'db' to mapi:monetdb:///var/monetdb5/dbfarm/db/.mapi.sock?database=db
2017-04-21 16:29:45 MSG merovingian[9]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying
2017-04-21 16:29:45 MSG db[174]: arguments: /usr/bin/mserver5 --dbpath=/var/monetdb5/dbfarm/db --set merovingian_uri=mapi:monetdb://32015d88ec9d:50000/db --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/monetdb5/dbfarm/db/.mapi.sock --set monet_vault_key=/var/monetdb5/dbfarm/db/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set embedded_r=true --set embedded_py=true --set monet_daemon=yes

I've installed numpy with yum, but the problem persists.

I want to add volume mapping to keep the data in host volumes. But file permission make it tough to do this. Any suggestions?

Alpine

Please will create a base image on Linux Alpine, because now your image very fat >500MB

Network Connectivity: How to start the container correctly?

Hi, I want to connect to the monet-container [172.17.0.7] from another one (lets call it Rcont [172.17.0.2]) . Both are in the same network (bridge) docker0.

If I start monet-container it like this:

docker run -d -P --name $NAME monetdb/monetdb-r-docker

Then I cannot reach the datavase from the host using it's IP or the hosts IP (both commands return something like "connection refused"):

telnet 172.17.0.7 50000 
telnet localhost 50000

And I am also not able to reach it from the other container Rcont (neither on the host' s IP nor the `monet-container's IP).

But when I start it like this:

docker run -d -p 50000:50000 --name $NAME monetdb/monetdb-r-docker

it is not possible to reach the database using the containers IP (telnet 172.17.0.7 50000 from host), but telnet localhost 50000 works on the docker-host (as expected, because I bind the IP).

So I am able to connect from docker-host to the container, but within the container it is a different story. It seems like I am able to reach the database somehow (I get a no route instead of connection refused), but I am not able to connect to it:

root@2ba2030ac77f:/# wget 172.17.0.1:50000
--2017-02-28 16:09:14--  http://172.17.0.1:50000/
Connecting to 172.17.0.1:50000... failed: No route to host.
root@2ba2030ac77f:/# wget 172.17.0.7:50000
--2017-02-28 16:09:19--  http://172.17.0.7:50000/
Connecting to 172.17.0.7:50000... failed: Connection refused.

Although, wget indicates some sort of connection, a minimal example in R (on Rcont) fails:

install.packages("MonetDB.R")
library(MonetDB.R)
library(dplyr)
monetdb_conn <- src_monetdb("demo", host="172.17.0.1", password="MyPassword")

with the error:

Error in socketConnection(host = host, port = port, blocking = TRUE, open = "r+b",  : 
  cannot open the connection
In addition: Warning message:
In socketConnection(host = host, port = port, blocking = TRUE, open = "r+b",  :
  172.17.0.1:50000 cannot be opened

Distributed Query Processing: Can't connect with remote table

Hi:

I created a group of dockers and I am trying to make distributed query processing with these dockers. Unfortunately, after creating and loading all databases and tables, the remote table on master docker can't reach the worker dockers. When I use the command below, the databases are found:

$ sudo docker exec -it monetdb-master monetdb discover
location
mapi:monetdb://2f86a3fb136d:50000/db
mapi:monetdb://4e0a4950e056:50000/db
mapi:monetdb://81b490c59855:50000/db
mapi:monetdb://ac6c88b502c5:50000/db
mapi:monetdb://af59587ec2b4:50000/db
mapi:monetdb://ce801171d525:50000/db
mapi:monetdb://fa1fd9df572c:50000/db

$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ce801171d525 monetdb/monetdb-r-docker "/usr/bin/supervis..." About a minute ago Up About a minute 0.0.0.0:32804->50000/tcp monetdb-master
81b490c59855 monetdb/monetdb-r-docker "/usr/bin/supervis..." About a minute ago Up About a minute 0.0.0.0:32803->50000/tcp monetdb-2016
fa1fd9df572c monetdb/monetdb-r-docker "/usr/bin/supervis..." 2 minutes ago Up 2 minutes 0.0.0.0:32802->50000/tcp monetdb-2015
4e0a4950e056 monetdb/monetdb-r-docker "/usr/bin/supervis..." 3 minutes ago Up 3 minutes 0.0.0.0:32801->50000/tcp monetdb-2014
2f86a3fb136d monetdb/monetdb-r-docker "/usr/bin/supervis..." 4 minutes ago Up 4 minutes 0.0.0.0:32800->50000/tcp monetdb-2013
ac6c88b502c5 monetdb/monetdb-r-docker "/usr/bin/supervis..." 5 minutes ago Up 5 minutes 0.0.0.0:32799->50000/tcp monetdb-2012
af59587ec2b4 monetdb/monetdb-r-docker "/usr/bin/supervis..." 6 minutes ago Up 6 minutes 0.0.0.0:32798->50000/tcp monetdb-2011

First, I tried to create a remote table in monetdb-master connected to monetdb-2011 docker. The table t2011 has exactly the same name and declaration as the remote table:

CREATE TABLE "t2011" (
"state" char(2),
"p_date" date,
"payee" bigint,
"value" decimal(7,2),
"log_value" decimal(18,16)
);

sql>\D
START TRANSACTION;
SET SCHEMA "sys";
CREATE MERGE TABLE "sys"."payments" (
"state" CHAR(2),
"p_date" DATE,
"payee" BIGINT,
"value" DECIMAL(7,2),
"log_value" DECIMAL(18,16)
);
CREATE REMOTE TABLE "sys"."t2011" (
"state" CHAR(2),
"p_date" DATE,
"payee" BIGINT,
"value" DECIMAL(7,2),
"log_value" DECIMAL(18,16)
) ON 'mapi:monetdb://af59587ec2b4:50000/db';
COMMIT;

As documented, a remote table is created without test the connection. The connection will occur only when it is used:

$ sudo docker exec -it monetdb-master mclient -d db
sql>select count(*) from t2011;
unable to connect to 'mapi:monetdb://af59587ec2b4:50000/db': getaddrinfo failed: Name or service not known

The connection seems to be okay, but I don't figure out what is happening:

$ sudo docker exec -it monetdb-2011 monetdb -p50000 status
name state health remarks
db R 10m 50% 0s mapi:monetdb://af59587ec2b4:50000/db

Can you help me? My project can be found on Github.

Messages and warnings during startup of monetdb in container

When running the container, starting monetdbd outputs some errors and warnings. I suspect this is caused by the fact that the application returns before it is finished with the work. Because the container works as expected.

This container was run on the google cloud platform, but i would guess that this problem happens always.

$ sudo docker logs 891ebdb02438
/usr/lib/python2.7/site-packages/supervisor/options.py:295: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2015-03-29 05:34:59,680 CRIT Supervisor running as root (no user in config file)
2015-03-29 05:34:59,680 WARN Included extra file "/etc/supervisord.d/supervisord.ini" during parsing
2015-03-29 05:34:59,719 INFO RPC interface 'supervisor' initialized
2015-03-29 05:34:59,719 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-03-29 05:34:59,719 INFO supervisord started with pid 1
2015-03-29 05:35:00,722 INFO spawned: 'monetdb' with pid 9
2015-03-29 05:35:00,725 INFO spawned: 'monetdbd' with pid 10
2015-03-29 05:35:01,385 INFO exited: monetdb (exit status 1; not expected)
2015-03-29 05:35:01,664 INFO exited: monetdbd (exit status 0; not expected)
2015-03-29 05:35:02,668 INFO spawned: 'monetdb' with pid 15
2015-03-29 05:35:03,173 INFO spawned: 'monetdbd' with pid 18
2015-03-29 05:35:04,185 INFO success: monetdb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-03-29 05:35:04,185 INFO success: monetdbd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-03-29 05:35:04,833 INFO exited: monetdbd (exit status 1; not expected)
2015-03-29 05:35:04,835 INFO spawned: 'monetdbd' with pid 21
2015-03-29 05:35:04,836 CRIT reaped unknown pid 20)
2015-03-29 05:35:05,162 INFO exited: monetdbd (exit status 1; not expected)
2015-03-29 05:35:05,163 CRIT reaped unknown pid 22)
2015-03-29 05:35:06,167 INFO spawned: 'monetdbd' with pid 23
2015-03-29 05:35:06,205 INFO exited: monetdbd (exit status 1; not expected)
2015-03-29 05:35:06,205 CRIT reaped unknown pid 24)
2015-03-29 05:35:08,209 INFO spawned: 'monetdbd' with pid 25
2015-03-29 05:35:08,220 INFO exited: monetdbd (exit status 1; not expected)
2015-03-29 05:35:08,221 CRIT reaped unknown pid 26)
2015-03-29 05:35:09,179 INFO exited: monetdb (exit status 0; expected)
2015-03-29 05:35:10,181 INFO spawned: 'monetdb' with pid 28
2015-03-29 05:35:10,190 INFO exited: monetdb (exit status 1; not expected)
2015-03-29 05:35:11,192 INFO spawned: 'monetdb' with pid 29
2015-03-29 05:35:11,201 INFO exited: monetdb (exit status 1; not expected)
2015-03-29 05:35:12,204 INFO spawned: 'monetdbd' with pid 30
2015-03-29 05:35:12,214 INFO exited: monetdbd (exit status 1; not expected)
2015-03-29 05:35:12,214 INFO gave up: monetdbd entered FATAL state, too many start retries too quickly
2015-03-29 05:35:12,214 CRIT reaped unknown pid 31)
2015-03-29 05:35:13,217 INFO spawned: 'monetdb' with pid 32
2015-03-29 05:35:13,226 INFO exited: monetdb (exit status 1; not expected)
2015-03-29 05:35:16,231 INFO spawned: 'monetdb' with pid 33
2015-03-29 05:35:16,240 INFO exited: monetdb (exit status 1; not expected)
2015-03-29 05:35:17,242 INFO gave up: monetdb entered FATAL state, too many start retries too quickly