pagerduty / scheduler Goto Github PK

Uncaught error from thread [Scheduler-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[Scheduler]
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
at com.pagerduty.eris.FutureConversions$.googleListenableToScalaFuture(FutureConversions.scala:53)
at com.pagerduty.eris.widerow.WideRowDriverImpl.fetchData(WideRowDriverImpl.scala:99)
at com.pagerduty.eris.dao.WideRowDriverWithMetrics.fetchData(WideRowDriverWithMetrics.scala:23)
at com.pagerduty.widerow.chain.QueryPage.fetchNextPage(QueryPage.scala:68)
at com.pagerduty.widerow.chain.QueryPage.nextPage(QueryPage.scala:54)
at com.pagerduty.widerow.WideRowView.com$pagerduty$widerow$WideRowView$$rec$2(WideRowView.scala:260)
at com.pagerduty.widerow.WideRowView.get(WideRowView.scala:280)
at com.pagerduty.scheduler.dao.TaskScheduleDaoImpl.load(TaskScheduleDao.scala:148)
at com.pagerduty.scheduler.dao.TaskScheduleDaoImpl.load(TaskScheduleDao.scala:129)
at com.pagerduty.scheduler.akka.TaskPersistence$$anonfun$2.applyOrElse(TaskPersistence.scala:106)
at com.pagerduty.scheduler.akka.TaskPersistence$$anonfun$2.applyOrElse(TaskPersistence.scala:103)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162)
at akka.actor.FSM$class.processEvent(FSM.scala:604)
at com.pagerduty.scheduler.akka.TaskPersistence.akka$actor$LoggingFSM$$super$processEvent(TaskPersistence.scala:90)
at akka.actor.LoggingFSM$class.processEvent(FSM.scala:734)
at com.pagerduty.scheduler.akka.TaskPersistence.processEvent(TaskPersistence.scala:90)
at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:598)
at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:592)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at com.pagerduty.scheduler.akka.TaskPersistence.aroundReceive(TaskPersistence.scala:90)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I have tried following methods to migrate schema to local db ,

1. https://stackoverflow.com/questions/11682197/keyspace-schema-import-and-export-in-cassandra
2. https://stackoverflow.com/questions/16440606/import-and-export-schema-in-cassandra

cassandra-cli method is deprecated.
cqlsh method says not valid query,

How to import cassandra-migrations schema to my local cassandra db. thanks.

2018-01-30 20:17:22 ERROR SchedulerClient {com.pagerduty.scheduler.LoggingSupport$$anonfun$reportFutureResults$2$$anonfun$apply$2 apply} - Failed: enqueuing task to Kafka: TaskKey(2018-01-31T04:15:00.882Z,9b37a98b-9543-44e3-abf9-a197e16a52d1,561b891d-1c2b-4e29-aab6-27f6b1a55172)..
org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for scheduler-0 due to 30034 ms has passed since batch creation plus linger time

In Same Server (127.0.0.1) - > Working
I try to connect different server (10.0.0.1) -> TimeoutException
(telnet with (10.0.0.1:9092) connecting)

Is there a way to cancel an already scheduled task that has not run yet?

Hello,

I've followed the steps referenced in the User Guide to add the scheduler as a dependency, but when i go and build the project i get

sbt:dir> compile
←[0m[←[0m←[0minfo←[0m] ←[0m←[0mUpdating ...←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m        module not found: com.pagerduty#scheduler_2.11;9.1.2←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m==== local: tried←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m  .ivy2\local\com.pagerduty\scheduler_2.11\9.1.2\ivys\ivy.xml←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m==== public: tried←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m  https://repo1.maven.org/maven2/com/pagerduty/scheduler_2.11/9.1.2/scheduler_2.11-9.1.2.pom←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m==== bintray-pagerduty-oss-maven: tried←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m  https://dl.bintray.com/pagerduty/oss-maven/com/pagerduty/scheduler_2.11/9.1.2/scheduler_2.11-9.1.2.pom←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m        ::::::::::::::::::::::::::::::::::::::::::::::←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m        ::          UNRESOLVED DEPENDENCIES         ::←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m        ::::::::::::::::::::::::::::::::::::::::::::::←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m        :: com.pagerduty#scheduler_2.11;9.1.2: not found←[0m
←[0m[←[0m←[33mwarn←[0m] ←[0m←[0m        ::::::::::::::::::::::::::::::::::::::::::::::←[0m

Getting forbidden errors when trying to access the PagerDuty scheduler (or any other package) on bintray. May be related to https://www.jfrog.com/confluence/pages/viewpage.action?pageId=122564110

Is there any way to install library?

I don't see a deployment guide for this. Can you point me to one or give some idea how you are deploying this?

I'm facing issue related data loss ,

Scheduler was working initial start time , if scheduler not used long time after some time when I try to schedule task that task not getting added in "TaskSchedule" table, but in log I can able to see schedule task enqueuing scueessfuly, if I restart application it getting added to "TaskSchedule"

Logs=>

2018-11-29 19:16:21 INFO Scheduler$ {com.pagerduty.scheduler.LoggingSupport$$anonfun$reportFutureResults$1 apply} - Attempting: sending tasks to scheduler: List(TaskKey(2018-11-29T14:19:33.325Z,de2283d8-d279-4abf-9454-138729d094d1,79ea6593-06f9-47a4-8dba-de0fcce76610))
2018-11-29 19:16:21 INFO SchedulerClient {com.pagerduty.scheduler.LoggingSupport$$anonfun$reportFutureResults$2$$anonfun$apply$1 apply} - Succeeded: enqueuing task to Kafka: TaskKey(2018-11-29T14:19:33.325Z,de2283d8-d279-4abf-9454-138729d094d1,79ea6593-06f9-47a4-8dba-de0fcce76610).

Any configuration I want to take care about this kind of issues,

thanks

Hi
I am evaluating different alternatives to schedule tasks, and this is a good candidate as we are basing a lot of out work on Scala, Akka, Cassandra and Kafka.
But when I browse through, I do not see any active work on it, quite some pieces of the code is 3 years old, the Cassandra migration directory uses deprecated cassandra calls.
So is this project still maintained ?
Is the solution still in use ?
-Tobias Eriksson

Thanks for sharing this amazing scheduler.

Just curious , it appears that rt-apollo is a private project , however its referred as an example.
Any chance you can update a user guide with a portion of that example?

thanks

Scheduler currently doesn't work well with IntelliJ. See https://youtrack.jetbrains.com/issue/SCL-10768

We should see whether this is due to our sbt setup, maybe simplify it, check whether there are restrictions in multi-module setups, etcetera. Worst case, we may even have to check in project info (which is portable these days, so not as bad as it used to be).

2018-11-30 11:13:55 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-11-30 11:14:55 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-11-30 11:15:55 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-11-30 11:16:55 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.

I'm getting log every minutes but task not scheduled still in "TaskSchedule" table.

We are evaluating this module for our use. I would like to know whether @ PagerDuty you have separate repository. If so how long it takes to sync it with open source one?

Hi @DWvanGeest ,
I think reached final step, Now i want to implement logic under task runner please share some document or examples it helpful us. thanks

2017-08-04 19:05:48 INFO CountingConnectionPoolMonitor {com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor onHostAdded} - AddHost: localhost
2017-08-04 19:05:49 ERROR LocalActorRefProvider(akka://Scheduler) {akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$1 apply$mcV$sp} - guardian failed, shutting down system
java.util.concurrent.ExecutionException: com.netflix.astyanax.connectionpool.exceptions.TransportException: TransportException: [host=localhost(127.0.0.1):9042, latency=124(250), attempts=1]org.apache.thrift.transport.TTransportException: Read a negative frame size (-2062548992)!
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_25]
at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_25]
at com.pagerduty.eris.FutureConversions$$anon$1.run(FutureConversions.scala:49) ~[eris-core_2.10-2.0.4.jar:2.0.4]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91) ~[guava-18.0.jar:na]
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:384) ~[na:1.8.0_25]
at java.util.concurrent.FutureTask.setException(FutureTask.java:251) ~[na:1.8.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:271) ~[na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_25]
Caused by: com.netflix.astyanax.connectionpool.exceptions.TransportException: TransportException: [host=localhost(127.0.0.1):9042, latency=124(250), attempts=1]org.apache.thrift.transport.TTransportException: Read a negative frame size (-2062548992)!
at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:197) ~[astyanax-thrift-3.6.0.jar:3.6.0]
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:139) ~[astyanax-thrift-3.6.0.jar:3.6.0]
at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:119) ~[astyanax-core-3.6.0.jar:3.6.0]
at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:342) ~[astyanax-core-3.6.0.jar:3.6.0]
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186) ~[astyanax-thrift-3.6.0.jar:3.6.0]
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$4.call(ThriftColumnFamilyQueryImpl.java:297) ~[astyanax-thrift-3.6.0.jar:3.6.0]
at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$4.call(ThriftColumnFamilyQueryImpl.java:294) ~[astyanax-thrift-3.6.0.jar:3.6.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_25]
... 3 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: Read a negative frame size (-2062548992)!
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:133) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) ~[libthrift-0.9.1.jar:0.9.1]
at org.apache.cassandra.thrift.Cassandra$Client.recv_set_keyspace(Cassandra.java:599) ~[cassandra-thrift-2.0.12.jar:2.0.12]
at org.apache.cassandra.thrift.Cassandra$Client.set_keyspace(Cassandra.java:586) ~[cassandra-thrift-2.0.12.jar:2.0.12]
at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.execute(ThriftSyncConnectionFactoryImpl.java:127) ~[astyanax-thrift-3.6.0.jar:3.6.0]
... 9 common frames omitted

Is their any plan scheduler release build with 2.12.

Hi Pagerduty Team,

I'm facing this after some days, scheduler system started and working what we expected, after some day it getting stoped, I need help to fix this issue,

2018-12-05 03:06:12 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-12-05 03:07:13 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
### 2018-12-05 03:07:43 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl trackResourceShutdown} - Shutting down SchedulingSystem...
2018-12-05 03:07:44 INFO SchedulerKafkaConsumer {com.pagerduty.kafkaconsumer.SimpleKafkaConsumer initializeConsumerAndEnterPollLoop} - Stopping Kafka consumer.
2018-12-05 03:07:44 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl trackResourceShutdown} - Shutting down SchedulingSystem...
2018-12-05 03:07:45 ERROR SchedulerKafkaConsumer {com.pagerduty.kafkaconsumer.SimpleKafkaConsumer com$pagerduty$kafkaconsumer$SimpleKafkaConsumer$$backoffOnUnhandledExceptionLoop} - Fatal error in polling thread.
2018-12-05 03:07:45 INFO SchedulerKafkaConsumer {com.pagerduty.kafkaconsumer.SimpleKafkaConsumer$$anon$1 run} - Shutting down polling thread.
2018-12-05 03:07:45 INFO AbstractCoordinator {org.apache.kafka.clients.consumer.internals.AbstractCoordinator markCoordinatorUnknown} - [Consumer clientId=consumer-1, groupId=scheduler] Group coordinator 10.200.1.91:9092 (id: 2147483645 rack: null) is unavailable or invalid, will attempt rediscovery
2018-12-05 03:07:45 INFO AbstractCoordinator {org.apache.kafka.clients.consumer.internals.AbstractCoordinator$FindCoordinatorResponseHandler onSuccess} - [Consumer clientId=consumer-1, groupId=scheduler] Discovered group coordinator 10.200.1.91:9092 (id: 2147483645 rack: null)
2018-12-05 03:08:13 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-12-05 03:09:13 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-12-05 03:10:13 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-12-05 03:11:13 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.
2018-12-05 03:12:13 INFO Scheduler$ {com.pagerduty.scheduler.Scheduler$LoggingImpl$$anonfun$staleTasksGaugeSampleConsumer$1 apply$mcVI$sp} - Scheduler stale tasks result: 0 stale tasks.

Thanks for sharing the scheduler.

I like to use in our application. I am new to Scala, so if you give me an the Sample main class to run this scheduler it will be very helpful for us.

Also some of the docs are private, is it possible to make as public. For example https://docs.pd-internal.com/scala/scheduler

Scheduler starting working properly , if any cassandra connection renew with 9160 has problem. where should be a problem, Cassandra or scheduler library.

Following Exception

java.util.concurrent.ExecutionException: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=127.0.0.1(127.0.0.1):9160, latency=60001(60001), attempts=2]Timed out waiting for connection
Caused by: com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:PoolTimeoutException: [host=127.0.0.1(127.0.0.1):9160, latency=60001(60001), attempts=2]Timed out waiting for connection

thanks

Hi,
I saw the talk about this tool on youtube and was really joyful about the opportunities that this tool could bring. Unfortunately the documentation is so bad and does not help alot when there are questions about how to implement this lib and it's dependencies. Especially the client side is a big questionmark for me. How do I register what to do when a task is running? How do I tell the lib when my task should run?
Could you please provide some documentation about a simple sample task that does prints out a "hello scheduler" when the scheduled task is running?
That would be great.