druid-io / docker-druid Goto Github PK
View Code? Open in Web Editor NEWDruid Docker
docker-druid's People
Contributors
Stargazers
Watchers
Forkers
gjayaprakash mrijke lukehan worrel albertocsm mindis gabrieltaylor adqio tarekrached allenli1990 wpmedia optimizely alexrovner cjbutenhoff jglanz dzer6 guobingkun redlion99 sharonds rasahner heamon7 taopercolate doubaokun xilon-jul chinpeng jerryjung bugra prodest avisri jon-wei ghoranyi joancardonasa rybex dj-haile devk2 listnplay asrayousuf alonegem wdicharry-rs rem0temeth0d xiaoyao1991 layer3switch mkleinert nagyistge gurugv-zz glycerine fokko laxman-sm briansneddon michalsikora starlkj deepak-rai ken-liul rushabhnagda11 pazone schmichri tianshangjun vuchau niteesh simonellistonball treff7es zeevmoney danklynn vihag sashadt ipvs-dds lastlegion fryderykhuang otherpirate yieldr gonenradai zippyai gpeavy mentholi alanhuang281 blessonmathewsam sumelio muxinc koldbyte greg-v dferstay antimoz-om nikolabravo iondulgheru dinuduke alaamub dbrand666 odysseyprime hanmd82 patrickosborne debdutto giltron mahenn aiyi2099 asloud thuylevn atlaspilotpuppy jaimeyu aditiverma baumgarodocker-druid's Issues
Unable to build docker image
Steps to reproduce:
- git clone https://github.com/druid-io/docker-druid.git
- docker build -t druid/cluster docker-druid
Error message:
Oct 28, 2015 8:44:59 AM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
2015-10-28T08:45:00,153 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='druid_extensions', hadoopDependenciesDir='hadoop_druid_dependencies', loadList=null}]
2015-10-28T08:45:01,402 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@2ca47471]
2015-10-28T08:45:01,408 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[]}]
Exception in thread "main" com.google.inject.ProvisionException: Guice provision errors:
1) Unknown provider[mysql] of Key[type=io.druid.metadata.MetadataStorageConnector, annotation=[none]], known options[[derby]]
at io.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:86)
while locating io.druid.metadata.MetadataStorageConnector
1 error
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1014)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1040)
at io.druid.cli.CreateTables.run(CreateTables.java:108)
at io.druid.cli.Main.main(Main.java:91)
The command '/bin/sh -c /etc/init.d/mysql start && java -cp /usr/local/druid/lib/druid-services-*-selfcontained.jar -Ddruid.extensions.coordinates=[\"io.druid.extensions:mysql-metadata-storage\"] -Ddruid.metadata.storage.type=mysql io.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://localhost:3306/druid" --user=druid --password=diurd && /etc/init.d/mysql stop' returned a non-zero code: 1
error when run task with port 8090
how to run tasks with port 8090 now
SQL queries
https://github.com/ThuyLeVn/druid-docker/importHello,
SQL quries on druid is limited, for example
SELECT LEFT("bbbbb-aaaa", LOCATE("-", bbbbb-aaaa")-1) AS lef, RIGHT("bbbbb-aaaa", LOCATE("-", "bbbbb-aaaa")) AS rig;
SELECT LEFT("bbbbb-aaaa", POSITION("-" IN "bbbbb-aaaa")-1) AS lef, RIGHT("bbbbb-aaaa", POSITION("-" IN "bbbbb-aaaa")) AS rig;
How to use split column to multi columns in druid?
Thanks you so much
ThuyLe
how to run realtime node
can you suggest the way to run realtime node, and ingest data from kaka/rabbitmq with realtime node?
issue with port 8090
how to run tasks with port 8090 now
how to split to multiple containers
How can we split the services to individual containers . We need to scale certain services like historical and broker . Any guidelines will be helpful.
-Rahul
Not enough direct memory.
Hi,
When loading the index example from tutorial, with 5 lines of input, the index process dumps:
Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[954,728,448], memoryNeeded[8,589,934,592] = druid.pr
ocessing.buffer.sizeBytes[1,073,741,824] * ( druid.processing.numThreads[7] + 1 )
I Have issue with with Starting MySQL database server mysqld
The Error thrown at the step is
Step 17/26 : RUN /etc/init.d/mysql start && mysql -u root -e "GRANT ALL ON druid.* TO 'druid'@'localhost' IDENTIFIED BY 'diurd'; CREATE database druid CHARACTER SET utf8;" && java -cp /usr/local/druid/lib/druid-services-*-selfcontained.jar -Ddruid.extensions.directory=/usr/local/druid/extensions -Ddruid.extensions.loadList=["mysql-metadata-storage"] -Ddruid.metadata.storage.type=mysql io.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://localhost:3306/druid" --user=druid --password=diurd && mysql -u root druid < sample-data.sql && /etc/init.d/mysql stop
---> Running in db9cce62a289
- Starting MySQL database server mysqld
...fail!
ERROR: Service 'docker-druid' failed to build: The command '/bin/sh -c /etc/init.d/mysql start && mysql -u root -e "GRANT ALL ON druid.* TO 'druid'@'localhost' IDENTIFIED BY 'diurd'; CREATE database druid CHARACTER SET utf8;" && java -cp /usr/local/druid/lib/druid-services-*-selfcontained.jar -Ddruid.extensions.directory=/usr/local/druid/extensions -Ddruid.extensions.loadList=["mysql-metadata-storage"] -Ddruid.metadata.storage.type=mysql io.druid.cli.Main tools metadata-init --connectURI="jdbc:mysql://localhost:3306/druid" --user=druid --password=diurd && mysql -u root druid < sample-data.sql && /etc/init.d/mysql stop' returned a non-zero code: 1
Ingestion doesn't work with zookeeper 3.4.12
If ENV ZOOKEEPER_VERSION 3.4.12 in DockerFile
runtime properties
Hi,
Where do I place the common runtime properties file and other runtime properties files ? I need to setup the deep storage as hdfs , thus need to configure the common runtime properties.
Regards
Rahul
rabbitmq to druid error with docker
curl -X POST -H 'Content-Type: application/json' -d @Druid_Ingestion/rabbitmq-index.json http://localhost:8081/druid/indexer/v1/task
{
"type": "index_realtime",
"spec": {
"dataSchema" : {
"dataSource" : "rabbitmq_test",
"parser" : {
"type" : "base64",
"parseSpec" : {
"format" : "json",
"timestampSpec" : {
"column" : "submit_date",
"format" : "auto"
},
"dimensionsSpec" : {
"dimensions": [ "request_type","origin","description" ],
"dimensionExclusions" : [],
"spatialDimensions" : []
}
}
},
"metricsSpec" : [ {
"type" : "count",
"name" : "count"
} ],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "hour",
"queryGranularity" : "none"
}
},
"ioConfig" : {
"type" : "realtime",
"firehose": {
"type": "rabbitmq",
"connection" : {
"host": "172.29.242.173",
"port": "5672",
"username": "guest",
"password": "guest",
"virtualHost": "/",
"uri": "amqp://172.29.242.173/"
},
"config" : {
"exchange": "NiFi_Exchanges_test_1",
"queue" : "NiFi_Queues_test_1",
"routingKey": "NiFi_Routing_key_test_1",
"durable": "true",
"exclusive": "false",
"autoDelete": "false",
"maxRetries": "10",
"retryIntervalSeconds": "1",
"maxDurationSeconds": "300"
},
"plumber": {
"type": "realtime"
}
}
},
"tuningConfig": {
"type" : "realtime",
"maxRowsInMemory": 7500000,
"intermediatePersistPeriod": "PT10m",
"windowPeriod": "PT10m",
"basePersistDirectory": "/tmp/realtime/basePersist",
"rejectionPolicy": {
"type": "serverTime"
}
}
}
}
2018-08-17T11:57:21,400 WARN [MonitorScheduler-0] io.druid.segment.realtime.RealtimeMetricsMonitor - [1] events thrown away because they are outside the window period!
2018-08-17T12:10:00,001 INFO [rabbitmq_test-overseer-0] io.druid.segment.realtime.plumber.RealtimePlumber - Starting merge and push.
2018-08-17T12:10:00,001 INFO [rabbitmq_test-overseer-0] io.druid.segment.realtime.plumber.RealtimePlumber - Found [0] segments. Attempting to hand off segments that start before [2018-08-17T12:00:00.000Z].
2018-08-17T12:10:00,001 INFO [rabbitmq_test-overseer-0] io.druid.segment.realtime.plumber.RealtimePlumber - Found [0] sinks to persist and merge
Build fail
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 51.318 s
[INFO] Finished at: 2015-04-30T05:34:55+00:00
[INFO] Final Memory: 18M/44M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:versions-maven-plugin:2.2:set (default-cli) on project druid: Execution default-cli of goal org.codehaus.mojo:versions-maven-plugin:2.2:set failed. NullPointerException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
INFO[0452] The command [/bin/sh -c mvn -U -B versions:set -DnewVersion=$DRUID_VERSION && mvn -U -B clean install -DskipTests=true -Dmaven.javadoc.skip=true && cp services/target/druid-services-$DRUID_VERSION-selfcontained.jar /usr/local/druid/lib] returned a non-zero code: 1 $ docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): darwin/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
host01 * virtualbox Running tcp://192.168.99.100:2376
$ uname -a
Darwin jake.local 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64Kafka-tranquility error with druid-docker
curl -X 'POST' -H 'Content-Type: application/json' -d @Druid_Ingestion/kafka-tranquility-index.json http://localhost:8081/druid/indexer/v1/task
{"error":"Unexpected token (END_OBJECT), expected FIELD_NAME: missing property 'type' that is to contain type id (for class io.druid.indexing.common.task.Task)\n at [Source: HttpInputOverHTTP@2fec7265[c=1475,q=1,[0]=EOF,s=STREAM]; line: 1, column: 1475]"}
{
"dataSources" : {
"metrics-kafka" : {
"spec" : {
"dataSchema" : {
"dataSource" : "metrics-kafka",
"parser" : {
"type" : "string",
"parseSpec" : {
"timestampSpec" : {
"column" : "submit_date",
"format" : "auto"
},
"dimensionsSpec" : {
"dimensions" : [ "request_type", "origin", "description" ],
"dimensionExclusions" : [ ]
},
"format" : "json"
}
},
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "hour",
"queryGranularity" : "none"
},
"metricsSpec" : [
{
"type" : "count",
"name" : "count"
}
]
},
"ioConfig" : {
"type" : "realtime"
},
"tuningConfig" : {
"type" : "realtime",
"maxRowsInMemory" : "100000",
"intermediatePersistPeriod" : "PT10M",
"windowPeriod" : "PT10M"
}
},
"properties" : {
"task.partitions" : "1",
"task.replicants" : "1",
"topicPattern" : "NiFi_Kafka_test_1"
}
}
},
"properties" : {
"zookeeper.connect" : "172.29.242.173:8081",
"commit.periodMillis" : "15000",
"consumer.numThreads" : "2",
"kafka.zookeeper.connect" : "172.29.242.173:8081",
"kafka.group.id" : "tranquility-kafka"
}
}
Issues building using Docker's overlay2
For some reason using overlay2 will result in a failure for building the image. Specifically at the mysql start step. Here's an explanation docker/for-linux#72
The fix is to add find /var/lib/mysql -type f -exec touch {} \; && to line 65 before running mysql.
Edit: You also have to add that to the command, very last line
Unable to send data from Java program using tranquility
Hi,
I am trying to use this Dockerized druid to send data from a Java application using tranquility.
The above sample application works fine when i setup druid locally on a VM following the getting started guide.
However when i try to send data to dockerized druid i get following error:
Failed to send message: %sjava.lang.IllegalStateException: Failed to create merged beam: druid:overlord/pageviews
I was getting similar error when setting up druid on my VM, which was fixed by setting the property druid.host in common.runtime.properties to machine's reachable ip address.
Is something similar required for this dockerized setup? The setup looks different from the druid setup mentioned in the getting started guide. The middle manager process seems to be missing.
Any pointers on this would be helpful.
Expected Replicants[2]
Hi, docker-druid says Expected Replicants[2] when trying to execute index task from the wiki example. where can this be changed? docker is a single node, cannot be replicated
Where is deep storage?
There are couple rows in supervisord.conf:
-Ddruid.s3.accessKey=AKIAIMKECRUYKDQGR6YQ
-Ddruid.s3.secretKey=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
And I suppose that means druid's segments are stored into amazon s3, but why there are no rows:
-Ddruid.storage.type=s3
-Ddruid.storage.bucket=your-bucket
-Ddruid.storage.baseKey=something
denied execution of stop/start mysql
Configuring mysql-server-5.5
While not mandatory, it is highly recommended that you set a password for the
MySQL administrative "root" user.
If this field is left blank, the password will not be changed.
New password for the MySQL "root" user:
Use of uninitialized value $_[1] in join or string at /usr/share/perl5/Debconf/DbDriver/Stack.pm line 111.
invoke-rc.d: policy-rc.d denied execution of stop.
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libhtml-template-perl (2.95-1) ...
Setting up tcpd (7.6.q-25) ...
tu0.14.04.1) ...
Setting up mysql-server-core-5.5 (5.5.44-0ubuntu0.14.04.1) ...
Setting up psmisc (22.20-1ubuntu2) ...
Setting up mysql-server-5.5 (5.5.44-0ubuntu0.14.04.1) ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype
Setting up supervisor (3.0b2-1) ...
invoke-rc.d: policy-rc.d denied execution of start.
Not enough direct memory.
The only modification is to supervisor.conf:
-Ddruid.processing.numThreads=1
-Ddruid.processing.buffer.sizeBytes=23000
This was supposed to eliminate the memory issue, but gave this:
2015-10-05T14:00:16,552 WARN [main] io.druid.guice.LifecycleScope - Caught exception when trying to create a[Key[type=io.druid.indexing.worker.executor.ExecutorLifecycle, annotation=[none]]]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_60]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_60]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_60]
at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_60]
at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:325) ~[druid-services-master-selfcontained.jar:master]
at com.metamx.common.lifecycle.Lifecycle.addMaybeStartHandler(Lifecycle.java:230) ~[druid-services-master-selfcontained.jar:master]
at com.metamx.common.lifecycle.Lifecycle.addMaybeStartManagedInstance(Lifecycle.java:176) ~[druid-services-master-selfcontained.jar:master]
at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:76) [druid-services-master-selfcontained.jar:master]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [druid-services-master-selfcontained.jar:master]
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005) [druid-services-master-selfcontained.jar:master]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051) [druid-services-master-selfcontained.jar:master]
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001) [druid-services-master-selfcontained.jar:master]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1040) [druid-services-master-selfcontained.jar:master]
at io.druid.cli.CliPeon.run(CliPeon.java:225) [druid-services-master-selfcontained.jar:master]
at io.druid.cli.Main.main(Main.java:91) [druid-services-master-selfcontained.jar:master]
Caused by: com.metamx.common.ISE: Failed to run isReady
at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:124) ~[druid-services-master-selfcontained.jar:master]
... 15 more
Caused by: com.metamx.common.ISE: Unable to grant lock to inactive Task [index_wikipedia_2015-10-05T14:00:10.733Z]
at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:224) ~[druid-services-master-selfcontained.jar:master]
at io.druid.indexing.overlord.TaskLockbox.tryLock(TaskLockbox.java:201) ~[druid-services-master-selfcontained.jar:master]
at io.druid.indexing.common.actions.LockTryAcquireAction.perform(LockTryAcquireAction.java:58) ~[druid-services-master-selfcontained.jar:master]
at io.druid.indexing.common.actions.LockTryAcquireAction.perform(LockTryAcquireAction.java:29) ~[druid-services-master-selfcontained.jar:master]
at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:62) ~[druid-services-master-selfcontained.jar:master]
at io.druid.indexing.common.task.AbstractFixedIntervalTask.isReady(AbstractFixedIntervalTask.java:89) ~[druid-services-master-selfcontained.jar:master]
at io.druid.indexing.worker.executor.ExecutorLifecycle.start(ExecutorLifecycle.java:120) ~[druid-services-master-selfcontained.jar:master]
... 15 more
2015-10-05T14:00:16,559 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.NullPointerException
at io.druid.cli.CliPeon.run(CliPeon.java:225) [druid-services-master-selfcontained.jar:master]
at io.druid.cli.Main.main(Main.java:91) [druid-services-master-selfcontained.jar:master]
CentOs Version
CentOs is a fairly popular enterprise OS and as such would be great to include the centos version:
https://github.com/alexrovner/docker-druid
I can create a pull to incorporate the changes, just need to know how to best organize the multiple images.
Communications link failure - Installation
Simply downloaded and ran the docker image to receive the following:
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
How to load data
Thanks for this Docker image! It's really great for giving Druid a quick try. (Also thanks for Druid. 😄)
I can't figure out how to load my data into it though. The docs tell me to send the request to localhost:8090/druid/indexer/v1/task. Port 8090 is open on this container but it immediately closes when I connect to it.
There is no Overlord configured in supervisor.conf. But the Coordinator is configured with druid.coordinator.asOverlord.enabled=true. I guess I'm supposed to use port 8081 then? When I send the request to 8081 (curl -X 'POST' -H 'Content-Type:application/json' -d @ingest.json localhost:8081/druid/indexer/v1/task) I get a 500 error back and the Coordinator logs the following:
2017-10-17T15:28:38,528 ERROR [qtp1933799970-57] io.druid.curator.discovery.ServerDiscoverySelector - No server instance found
2017-10-17T15:28:38,528 WARN [qtp1933799970-57] org.eclipse.jetty.servlet.ServletHandler - /druid/indexer/v1/task
com.metamx.common.ISE: Can't find indexingService, did you configure druid.selectors.indexing.serviceName same as druid.service at overlord?
at io.druid.server.http.OverlordProxyServlet.rewriteURI(OverlordProxyServlet.java:55) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.proxy.ProxyServlet.service(ProxyServlet.java:390) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:71) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.Server.handle(Server.java:497) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [druid-services-0.9.2-selfcontained.jar:0.9.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
I'll keep trying to figure this out. But it would be very helpful to include an example of ingestion in README.md. Thanks!
build stop
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): darwin/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64
stops here
Step 19 : RUN java "-Ddruid.extensions.coordinates=[\"io.druid.extensions:druid-s3-extensions\",\"io.druid.extensions:mysql-metadata-storage\"]" -Ddruid.extensions.localRepository=/usr/local/druid/repository -Ddruid.extensions.remoteRepositories=[\"file:///root/.m2/repository/\",\"https://repo1.maven.org/maven2/\"] -cp /usr/local/druid/lib/* io.druid.cli.Main tools pull-deps
---> Running in 47a389392a5b
May 23, 2015 7:14:46 PM org.hibernate.validator.internal.util.Version <clinit>
INFO: HV000001: Hibernate Validator 5.1.3.Final
2015-05-23T19:14:46,494 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[io.druid.extensions:druid-s3-extensions, io.druid.extensions:mysql-metadata-storage], defaultVersion='master', localRepository='/usr/local/druid/repository', remoteRepositories=[file:///root/.m2/repository/, https://repo1.maven.org/maven2/]}]
2015-05-23T19:14:46,616 INFO [main] io.druid.initialization.Initialization - Loading extension[io.druid.extensions:druid-s3-extensions] for class[io.druid.cli.CliCommandCreator]
Error response from daemon
The docker run command, return error response from daemon, here is the detail:
$ docker run --rm -i -p 8888:8888 druidio/example-cluster
docker: Error response from daemon: no status provided on response: unknown.
Starting only particular process
What are your thoughts on having a docker file that can start with a specific role -Either a coordinator or historical or broker etc.
So this way we can use the same image in production depending on the role type. So basically it will be
docker start coordinator 8081
docker start all (could start everything up)
What do you think @gianm ?
Can't build docker image
I try to build the docker image and it fail evry time with this error :
---> cbf04910b89a
Removing intermediate container 67eb999231f6
Step 20 : RUN /etc/init.d/mysql start && echo "GRANT ALL ON druid.* TO 'druid'@'localhost' IDENTIFIED BY 'diurd'; CREATE database druid CHARACTER SET utf8;" | mysql -u root && /etc/init.d/mysql stop
---> Running in bfabd965d3f9
2015/04/17 09:19:18 /var/lib/docker/aufs/mnt/bfabd965d3f93f4aaaf7b93160a9dfd07c04a875b03e7607da0f6ec03b820080 is not within /var/lib/docker/aufs/mnt/bfabd965d3f93f4aaaf7b93160a9dfd07c04a875b03e7607da0f6ec03b820080
Overlord service not going up
ss -tunlop command inside docker shows that nothing is running on port 8090
How to manually start and debug this overlord service ?
Unable to find common.runtime.properties
In order to add statsd emitter to this docker image, We need update common.runtime.properties.
But I am unable to locate this directory /druid/conf/druid/_common inside the container. Am i missing something here?
Build different versions using tags
Hi Guys,
Right now it always build the master branch. It would be nicer to use tags to build specific versions. So we can use specific version inside of our application and it also makes it easier to test when upgrading to a newer version.
This would require building specific version and tagging them, and also adding them to the Docker Hub: https://hub.docker.com/r/druidio/example-cluster/
Please let me know,
Cheers, Fokko
overlord address is set to unreachable container address
While working with docker-druid image and tranquility library i found that the address stored under druid/discovery/druid:overlord in zookeeper is the container address as shown below:
{"name":"druid:overlord","id":"91bb6dbd-9b8c-4863-9530-e70c9f7f6587","address":" 172.17.0.2","port":8081,"sslPort":-1,"payload":null,"registrationTimeUTC":154416 6430226,"serviceType":"DYNAMIC","uriSpec":null}
This container address would be the one that an application using tranquility library would get from zookeeper through curator service discovery. Since this address is not the machine's reachable IP address data would not be sent to druid using tranquility.
The address is probably coming from supervisor.conf :
-Ddruid.host=%(ENV_HOSTIP)s
Should this be set to host machine's IP rather than container IP?
This issue appears to be related to druid-io/tranquility#186
Discussion on google groups:
https://groups.google.com/forum/#!topic/druid-user/TDBOaBNZXZQ
https://groups.google.com/forum/#!topic/druid-user/DAi5xfSrVhY
https://groups.google.com/forum/#!topic/druid-user/BHWbldJy1ks
[Proposal] Deploying truly distributed Druid clusters with docker
Hey guys,
I see that the image created from this repo is running every node together in one single container. While it's helpful for users to test and try out Druid, it isn't particularly useful when it comes to deploying a production cluster.
I've been working on using docker to deploy a truly distributed Druid cluster lately. I have had something working and I'd like to share it and contribute back. I'm wondering if it will be valuable though. Take a look at this fork to see what I've done so far. While the settings in that fork are set specifically for my team's research purpose, they can be generalized and made extendable. I'm still in an exploration stage on docker deployments, so I may have made stupid mistakes.
Due to its lightweight, Docker philosophy encourages running only one service in a container and having containers talk to each other, rather than running all dependencies and services in one container. Therefore, I defined separate images for each type of Druid node(druid-broker, druid-coordinator, etc.) as well as dependencies(druid-zookeeper, druid-mysql, druid-kafka, etc.) I also packed the jvm and runtime configurations into a separate image(druid-conf).
I had a discussion earlier with @cheddar on deployment stuff. He made it clear that deployment in general should only have 3 steps:
- Download the artifact.
- Download the configurations.
- Run the artifact with the configurations.
I followed this guideline: when running a specific druid node, say broker, all you need to do is:
- Pull
druid-brokerimage - Pull
druid-confimage - Run
druid-confin a container, and then link it as a volume provider(using--volumes-from) for thedruid-brokercontainer.
Containers on different nodes can freely communicate with each others as long as they are within a same overlay network. I leverages docker-machine to manage/provision remote nodes, and docker swarm for container orchestration. Running a broker node for example is just as simple as:
docker run -itd --volumes-from=node-1-conf --name=broker --network=p-my-net --env="constraint:node==p-node-5" druid-broker:latest
403 Forbidden during docker run
Hey,
I am getting the following error when i run the docker image:
2019-01-22 16:56:43,455 DEBG 'druid-historical' stdout output:
2019-01-22T16:56:43,446 ERROR [ZkCoordinator] io.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=io.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class io.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z], segment=DataSegment{size=24664730, shardSpec=NoneShardSpec, metrics=[count, delta, variation, added, deleted], dimensions=[dma_code, continent_code, geo, area_code, robot, country_name, network, city, namespace, anonymous, unpatrolled, page, postal_code, language, newpage, user, region_lookup], version='2013-08-08T21:22:48.989Z', loadSpec={type=>s3_zip, bucket=>static.druid.io, key=>data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip}, interval=2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z, dataSource='wikipedia', binaryVersion='9'}}
io.druid.segment.loading.SegmentLoadingException: Exception loading segment[wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:265) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:47) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.coordination.ZkCoordinator$1.childEvent(ZkCoordinator.java:118) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:520) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:514) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:296) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:512) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:771) [druid-services-0.12.1-selfcontained.jar:0.12.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.RuntimeException: org.jets3t.service.ServiceException: Request Error. -- ResponseCode: 403, ResponseStatus: Forbidden, RequestId: B11F1C837771B9B6, HostId: vVwwPXur/GyoYZ3wmK0pSa9gtYL06t5dtFJIqerGDqQwM7Rek/ABBrnbGJQtLCgiwDzzItlCIh0=
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:327) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
... 18 more
Caused by: org.jets3t.service.ServiceException: Request Error.
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:426) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
... 18 more
Caused by: org.jets3t.service.impl.rest.HttpException: 403 Forbidden
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.storage.s3.S3Utils.isObjectInBucket(S3Utils.java:96) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:318) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller$4.call(S3DataSegmentPuller.java:314) ~[?:?]
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:89) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.isObjectInBucket(S3DataSegmentPuller.java:312) ~[?:?]
at io.druid.storage.s3.S3DataSegmentPuller.getSegmentFiles(S3DataSegmentPuller.java:176) ~[?:?]
at io.druid.storage.s3.S3LoadSpec.loadSegment(S3LoadSpec.java:60) ~[?:?]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocation(SegmentLoaderLocalCacheManager.java:205) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadInLocationWithStartMarker(SegmentLoaderLocalCacheManager.java:193) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:151) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:133) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:108) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.SegmentManager.getAdapter(SegmentManager.java:196) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.SegmentManager.loadSegment(SegmentManager.java:157) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
at io.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:261) ~[druid-services-0.12.1-selfcontained.jar:0.12.1]
... 18 more
2019-01-22 16:56:43,459 DEBG 'druid-coordinator' stdout output:
2019-01-22T16:56:43,458 INFO [main-EventThread] io.druid.server.coordinator.CuratorLoadQueuePeon - Server[/druid/loadQueue/172.17.0.2:8083] done processing [/druid/loadQueue/172.17.0.2:8083/wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
2019-01-22 16:56:43,459 DEBG 'druid-historical' stdout output:
2019-01-22T16:56:43,458 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - Completed request [LOAD: wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z]
2019-01-22T16:56:43,458 INFO [ZkCoordinator] io.druid.server.coordination.ZkCoordinator - zNode[/druid/loadQueue/172.17.0.2:8083/wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z] was removed
How to load extensions?
Hi,
i'm trying to load extensions by placing their names in -Ddruid.extensions.loadList of [program:druid-coordinator] section in supervisord.conf. An excerpt from my supervisord.conf file is the following:
[...]
[program:druid-coordinator]
user=druid
command=java
-server
-Xmx1g
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Ddruid.host=%(ENV_HOSTIP)s
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\", \"druid-avro-extensions\"]
-Ddruid.extensions.directory=/usr/local/druid/extensions
-Ddruid.extensions.hadoopDependenciesDir=/usr/local/druid/hadoop-dependencies
-Ddruid.metadata.storage.type=mysql
-Ddruid.metadata.storage.connector.connectURI=jdbc:mysql://localhost:3306/druid
-Ddruid.metadata.storage.connector.user=druid
-Ddruid.metadata.storage.connector.password=diurd
-Ddruid.coordinator.asOverlord.enabled=true
-Ddruid.coordinator.asOverlord.overlordService=druid/overlord
-Ddruid.indexer.fork.property.druid.processing.numThreads=1
-Ddruid.indexer.storage.type=metadata
-Ddruid.indexer.queue.startDelay=PT0M
-Ddruid.indexer.runner.javaOpts="-server -Xmx1g -XX:MaxDirectMemorySize=2147483648"
-Ddruid.processing.buffer.sizeBytes=536870912
-Ddruid.coordinator.startDelay=PT5S
-cp /usr/local/druid/lib/*
io.druid.cli.Main server coordinator
redirect_stderr=true
priority=100
[...]
Unfortunately, i keep getting the following error in druid-coordinator-stdout---supervisor.log:
Error: Could not find or load main class "druid-avro-extensions"
Folder druid-avro-extensions with .jars (f.eg. druid-avro-extensions-0.10.1.jar) exists in /usr/local/druid/extensions with druid:druid owner.
Unfortunately i can't load any other extension due to the similar error so i must be missing something.
The question is - how to successfully load an extension? Any advice highly appreciated.
DRUID_VERSION 0.10.1.
Regards,
Łukasz
hadoop depency for druid should be renamed
I encountered error there is no such file or directory for hadoop_dependencies while build image.
in Dockerfile
below line should be
RUN cp -r distribution/target/hadoop_dependencies /usr/local/druid/like this.
RUN cp -r distribution/target/hadoop-dependencies /usr/local/druid/Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.