typesafehub / activator-akka-distributed-workers-java Goto Github PK

Hi,

I am working on upgrading the akka distributed workers (in similar lines with the distributed works example), master is ClusterSingletonManager and WorkersSystem is coming up, but getting the strange Association error.

Using Akka 2.4.17 also faced the same issue with akka 2.5.4.

Please suggest, looks like we are stuck here.

2018-03-30 10:31:02,963 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-25) [INFO] [03/30/2018 10:31:02.953] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:4002]
2018-03-30 10:31:02,965 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-30) [INFO] [03/30/2018 10:31:02.960] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:4002]
2018-03-30 10:31:03,046 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-15) [INFO] [03/30/2018 10:31:03.021] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Starting remoting
2018-03-30 10:31:03,083 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-28) [INFO] [03/30/2018 10:31:03.077] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:4003]
2018-03-30 10:31:03,100 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-35) [INFO] [03/30/2018 10:31:03.077] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:4003]
2018-03-30 10:31:03,145 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-14) [INFO] [03/30/2018 10:31:03.144] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Starting remoting
2018-03-30 10:31:03,185 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-27) [INFO] [03/30/2018 10:31:03.183] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:4004]
2018-03-30 10:31:03,186 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-29) [INFO] [03/30/2018 10:31:03.184] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:4004]
2018-03-30 10:31:03,231 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-10) [INFO] [03/30/2018 10:31:03.230] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Starting remoting
2018-03-30 10:31:03,259 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-24) [INFO] [03/30/2018 10:31:03.255] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:4005]
2018-03-30 10:31:03,261 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-28) [INFO] [03/30/2018 10:31:03.256] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:4005]
2018-03-30 10:31:03,325 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-13) [INFO] [03/30/2018 10:31:03.324] [ServerService Thread Pool -- 78] [akka.remote.Remoting] Starting remoting
2018-03-30 10:31:03,331 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-5) [ERROR] [03/30/2018 10:31:03.330] [WorkerSystem-akka.remote.default-remote-dispatcher-22] [akka.tcp://[email protected]:4002/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FClusterSystem%40127.0.0.1%3A2551-0/endpointWriter] AssociationError [akka.tcp://[email protected]:4002] -> [akka.tcp://[email protected]:2551]: Error [[B cannot be cast to [C] [
2018-03-30 10:31:03,331 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-5) java.lang.ClassCastException: [B cannot be cast to [C
2018-03-30 10:31:03,332 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-5) at deployment.jaxrpc.war//akka.remote.artery.FastHash$.ofString(LruBoundedCache.scala:18)
2018-03-30 10:31:03,332 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-5) at deployment.jaxrpc.war//akka.remote.serialization.ActorRefResolveCache.hash(ActorRefResolveCache.scala:61)
2018-03-30 10:31:03,332 INFO [stdout] (WorkerSystem-akka.actor.default-dispatcher-5) at deployment.jaxrpc.war//akka.remote.serialization.ActorRefResolveCache.hash(ActorRefResolveCache.scala:55)

I guess POM dependencies are not upgraded along with sbt configuration. Please update POM with latest dependencies.

Hi,
I am facing some gap when I am testing the blocking call on Work Executor. in the example.

the modified code.
`public class WorkExecutor extends AbstractActor {

private LoggingAdapter log = Logging.getLogger(getContext().system(), this);

@OverRide
public Receive createReceive() {
return receiveBuilder()
.match(Work.class, message -> {

	  Integer n = (Integer) message.job;
	  
	  
	  log.info("inside WorkExecutor processing the job : {}",message);
      
      //introducing the blocking
      if (n == 3){
    	
    	  
    	  try {
			Thread.sleep(10000);
		} catch (InterruptedException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
      }
    
      int n2 = n.intValue() * n.intValue();
      String result = n + " * " + n + " = " + n2;
      log.info("Produced result {}", result);
      getSender().tell(new Worker.WorkComplete(result), getSelf());
  
  
  })
  .build();

}
`

and the corresponding logs:
(see the work executor...)

[INFO] [10/06/2018 22:53:43.065] [main] [akka.remote.Remoting] Starting remoting
[INFO] [10/06/2018 22:53:43.313] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:2551]
[INFO] [10/06/2018 22:53:43.315] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:2551]
[INFO] [10/06/2018 22:53:43.330] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Starting up...
[INFO] [10/06/2018 22:53:43.450] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Registered cluster JMX MBean [akka:type=Cluster]
[INFO] [10/06/2018 22:53:43.450] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Started up successfully
[WARN] [10/06/2018 22:53:43.539] [ClusterSystem-akka.actor.default-dispatcher-14] [akka.tcp://[email protected]:2551/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[INFO] [10/06/2018 22:53:43.560] [ClusterSystem-akka.actor.default-dispatcher-10] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2551] is JOINING, roles [backend, dc-default]
[INFO] [10/06/2018 22:53:43.578] [ClusterSystem-akka.actor.default-dispatcher-10] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2551] to [Up]
[INFO] [10/06/2018 22:53:43.646] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:2551/user/master] Singleton manager starting singleton actor [akka://ClusterSystem/user/master/singleton]
[INFO] [10/06/2018 22:53:43.647] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.tcp://[email protected]:2551/user/master] ClusterSingletonManager state change [Start -> Oldest]
[INFO] [10/06/2018 22:53:48.582] [main] [akka.remote.Remoting] Starting remoting
[INFO] [10/06/2018 22:53:48.651] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:2552]
[INFO] [10/06/2018 22:53:48.652] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:2552]
[INFO] [10/06/2018 22:53:48.657] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2552] - Starting up...
[WARN] [10/06/2018 22:53:48.663] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config
[INFO] [10/06/2018 22:53:48.663] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2552] - Started up successfully
[WARN] [10/06/2018 22:53:48.667] [ClusterSystem-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:2552/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[INFO] [10/06/2018 22:53:48.712] [main] [akka.remote.Remoting] Starting remoting
[INFO] [10/06/2018 22:53:48.786] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:58365]
[INFO] [10/06/2018 22:53:48.787] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:58365]
[INFO] [10/06/2018 22:53:48.846] [ClusterSystem-akka.actor.default-dispatcher-6] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Received InitJoin message from [Actor[akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1357334455]] to [akka.tcp://[email protected]:2551]
[INFO] [10/06/2018 22:53:48.846] [ClusterSystem-akka.actor.default-dispatcher-6] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Sending InitJoinAck message from node [akka.tcp://[email protected]:2551] to [Actor[akka.tcp://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#1357334455]]
[INFO] [10/06/2018 22:53:48.871] [WorkerSystem-akka.actor.default-dispatcher-2] [akka.tcp://[email protected]:58365/user/clusterClient] Connected to [akka.tcp://[email protected]:2551/system/receptionist]
[WARN] [SECURITY][10/06/2018 22:53:48.876] [WorkerSystem-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$RegisterWorker] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:48.881] [ClusterSystem-akka.actor.default-dispatcher-7] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:2552] is JOINING, roles [backend, dc-default]
[INFO] [10/06/2018 22:53:48.896] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Worker registered: ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:53:48.991] [ClusterSystem-akka.actor.default-dispatcher-22] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2552] - Welcome from [akka.tcp://[email protected]:2551]
[INFO] [10/06/2018 22:53:49.351] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:53:49.541] [ClusterSystem-akka.actor.default-dispatcher-5] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:2552] to [Up]
[INFO] [10/06/2018 22:53:49.564] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://[email protected]:2552/user/master] ClusterSingletonManager state change [Start -> Younger]
[INFO] [10/06/2018 22:53:53.826] [main] [akka.remote.Remoting] Starting remoting
[INFO] [10/06/2018 22:53:53.888] [main] [akka.remote.Remoting] Remoting started; listening on addresses :[akka.tcp://[email protected]:58379]
[INFO] [10/06/2018 22:53:53.888] [main] [akka.remote.Remoting] Remoting now listens on addresses: [akka.tcp://[email protected]:58379]
[INFO] [10/06/2018 22:53:53.890] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:58379] - Starting up...
[WARN] [10/06/2018 22:53:53.892] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Could not register Cluster JMX MBean with name=akka:type=Cluster as it is already registered. If you are running multiple clusters in the same JVM, set 'akka.cluster.jmx.multi-mbeans-in-same-jvm = on' in config
[INFO] [10/06/2018 22:53:53.893] [main] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:58379] - Started up successfully
[WARN] [10/06/2018 22:53:53.897] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://[email protected]:58379/system/cluster/core/daemon/downingProvider] Don't use auto-down feature of Akka Cluster in production. See 'Auto-downing (DO NOT USE)' section of Akka Cluster documentation.
[INFO] [10/06/2018 22:53:53.914] [ClusterSystem-akka.actor.default-dispatcher-12] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Received InitJoin message from [Actor[akka.tcp://[email protected]:58379/system/cluster/core/daemon/joinSeedNodeProcess-1#1918361624]] to [akka.tcp://[email protected]:2551]
[INFO] [10/06/2018 22:53:53.914] [ClusterSystem-akka.actor.default-dispatcher-12] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Sending InitJoinAck message from node [akka.tcp://[email protected]:2551] to [Actor[akka.tcp://[email protected]:58379/system/cluster/core/daemon/joinSeedNodeProcess-1#1918361624]]
[INFO] [10/06/2018 22:53:53.918] [ClusterSystem-akka.actor.default-dispatcher-10] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Node [akka.tcp://[email protected]:58379] is JOINING, roles [dc-default]
[INFO] [10/06/2018 22:53:53.922] [ClusterSystem-akka.actor.default-dispatcher-6] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:58379] - Welcome from [akka.tcp://[email protected]:2551]
[INFO] [10/06/2018 22:53:53.939] [ClusterSystem-akka.actor.default-dispatcher-4] [akka.tcp://[email protected]:58379/user/frontend/masterProxy] Singleton identified at [akka.tcp://[email protected]:2551/user/master/singleton]
[INFO] [10/06/2018 22:53:54.338] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:53:54.927] [ClusterSystem-akka.actor.default-dispatcher-22] [akka://ClusterSystem/user/producer] Produced work: 1
[WARN] [SECURITY][10/06/2018 22:53:54.931] [ClusterSystem-akka.remote.default-remote-dispatcher-13] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$Work] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:54.939] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 1e4fdb85-b746-4f1d-ad7d-192b48684e31
[WARN] [SECURITY][10/06/2018 22:53:54.962] [ClusterSystem-akka.actor.default-dispatcher-5] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkAccepted] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:54.974] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:53:54.974] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:53:54.974] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[WARN] [SECURITY][10/06/2018 22:53:54.975] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$Ack] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[WARN] [SECURITY][10/06/2018 22:53:54.977] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkIsReady] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:54.980] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Inside Worker Work is ready : worker idba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:53:55.541] [ClusterSystem-akka.actor.default-dispatcher-12] [akka.cluster.Cluster(akka://ClusterSystem)] Cluster Node [akka.tcp://[email protected]:2551] - Leader is moving node [akka.tcp://[email protected]:58379] to [Up]
[INFO] [10/06/2018 22:53:57.988] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 2
[INFO] [10/06/2018 22:53:57.991] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 80b26a93-b809-43bb-a221-e26a30c8681c
[INFO] [10/06/2018 22:53:58.045] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:53:58.045] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:53:58.045] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:53:58.047] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] Inside Worker idle state : worker idba438def-2074-42b9-a522-9147c828ecdf
[WARN] [SECURITY][10/06/2018 22:53:58.048] [WorkerSystem-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkerRequestsWork] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:58.053] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkerRequestsWork with worker ba438def-2074-42b9-a522-9147c828ecdf
[WARN] [SECURITY][10/06/2018 22:53:58.061] [ClusterSystem-akka.actor.default-dispatcher-8] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkStarted] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:58.065] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] 1111111 Giving worker ba438def-2074-42b9-a522-9147c828ecdf some work 1e4fdb85-b746-4f1d-ad7d-192b48684e31
[WARN] [SECURITY][10/06/2018 22:53:58.066] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$Work] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Inside idle state changing state to working.. Got work: 1
[INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker/exec] inside WorkExecutor processing the job : Work{workId='1e4fdb85-b746-4f1d-ad7d-192b48684e31', job=1}
[INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker/exec] Produced result 1 * 1 = 1
[INFO] [10/06/2018 22:53:58.068] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] inside Working state. Work is complete. Result 1 * 1 = 1.
[WARN] [SECURITY][10/06/2018 22:53:58.069] [WorkerSystem-akka.remote.default-remote-dispatcher-7] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkIsDone] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:58.073] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work 1e4fdb85-b746-4f1d-ad7d-192b48684e31
[INFO] [10/06/2018 22:53:58.073] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Work 1e4fdb85-b746-4f1d-ad7d-192b48684e31 is done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:53:58.074] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Work state of the worker ba438def-2074-42b9-a522-9147c828ecdf is Busy{work=1e4fdb85-b746-4f1d-ad7d-192b48684e31, deadline=Deadline(305859126391953 nanoseconds)}
[INFO] [10/06/2018 22:53:58.074] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] changing worker state to idle
[WARN] [SECURITY][10/06/2018 22:53:58.079] [ClusterSystem-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkCompleted] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[WARN] [SECURITY][10/06/2018 22:53:58.085] [ClusterSystem-akka.remote.default-remote-dispatcher-26] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.Master$WorkResult] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:53:58.087] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Got Ack from Master inside waitForWorkIsDoneAck, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> 1e4fdb85-b746-4f1d-ad7d-192b48684e31)
[INFO] [10/06/2018 22:53:58.090] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/consumer] Consumed result: 1 * 1 = 1
[INFO] [10/06/2018 22:53:58.091] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkerRequestsWork with worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:53:58.095] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] 1111111 Giving worker ba438def-2074-42b9-a522-9147c828ecdf some work 80b26a93-b809-43bb-a221-e26a30c8681c
[INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker] Inside idle state changing state to working.. Got work: 2
[INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker/exec] inside WorkExecutor processing the job : Work{workId='80b26a93-b809-43bb-a221-e26a30c8681c', job=2}
[INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker/exec] Produced result 2 * 2 = 4
[INFO] [10/06/2018 22:53:58.097] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker] inside Working state. Work is complete. Result 2 * 2 = 4.
[INFO] [10/06/2018 22:53:58.099] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work 80b26a93-b809-43bb-a221-e26a30c8681c
[INFO] [10/06/2018 22:53:58.099] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work 80b26a93-b809-43bb-a221-e26a30c8681c is done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:53:58.100] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work state of the worker ba438def-2074-42b9-a522-9147c828ecdf is Busy{work=80b26a93-b809-43bb-a221-e26a30c8681c, deadline=Deadline(305859155830547 nanoseconds)}
[INFO] [10/06/2018 22:53:58.100] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] changing worker state to idle
[INFO] [10/06/2018 22:53:58.104] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] Got Ack from Master inside waitForWorkIsDoneAck, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> 80b26a93-b809-43bb-a221-e26a30c8681c)
[INFO] [10/06/2018 22:53:58.105] [ClusterSystem-akka.actor.default-dispatcher-22] [akka://ClusterSystem/user/consumer] Consumed result: 2 * 2 = 4
[INFO] [10/06/2018 22:53:58.830] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:53:59.349] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:01.064] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/producer] Produced work: 3
[INFO] [10/06/2018 22:54:01.066] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:01.070] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:01.070] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:01.070] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:01.075] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] Inside Worker idle state : worker idba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:01.078] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkerRequestsWork with worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:01.082] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] 1111111 Giving worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:01.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] Inside idle state changing state to working.. Got work: 3
[INFO] [10/06/2018 22:54:01.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker/exec] inside WorkExecutor processing the job : Work{workId='dd37150b-276d-4165-8f95-e1d389d34bda', job=3}
[INFO] [10/06/2018 22:54:04.351] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:05.093] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/producer] Produced work: 4
[INFO] [10/06/2018 22:54:05.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: cfdde0a5-0890-4b29-826b-8fc0a98987ce
[INFO] [10/06/2018 22:54:05.101] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:05.101] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:06.101] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] Time out in the working state for worker id: (ba438def-2074-42b9-a522-9147c828ecdf doing work -> dd37150b-276d-4165-8f95-e1d389d34bda)
[WARN] [SECURITY][10/06/2018 22:54:06.102] [WorkerSystem-akka.remote.default-remote-dispatcher-6] [akka.serialization.Serialization(akka://WorkerSystem)] Using the default Java serializer for class [worker.MasterWorkerProtocol$WorkFailed] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkFailed with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda failed by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Work state of the worker ba438def-2074-42b9-a522-9147c828ecdf is Busy{work=dd37150b-276d-4165-8f95-e1d389d34bda, deadline=Deadline(305862142751089 nanoseconds)}
[INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] changing worker state to idle
[INFO] [10/06/2018 22:54:06.108] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside WorkFailed -- changing work state to idle for worker dd37150b-276d-4165-8f95-e1d389d34bda WARNING arguments left: 1
[WARN] [SECURITY][10/06/2018 22:54:06.110] [ClusterSystem-akka.actor.default-dispatcher-8] [akka.serialization.Serialization(akka://ClusterSystem)] Using the default Java serializer for class [worker.WorkState$WorkerFailed] which is not recommended because of performance implications. Use another serializer or disable this warning using the setting 'akka.actor.warn-about-java-serializer-usage'
[INFO] [10/06/2018 22:54:06.114] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:06.114] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:06.114] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:08.833] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:09.107] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/producer] Produced work: 5
[INFO] [10/06/2018 22:54:09.109] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 8727e25a-12e9-4fa4-b1a3-d391549196a6
[INFO] [10/06/2018 22:54:09.253] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:09.253] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:09.253] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:09.338] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:11.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker/exec] Produced result 3 * 3 = 9
[INFO] [10/06/2018 22:54:12.268] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/producer] Produced work: 6
[INFO] [10/06/2018 22:54:12.273] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: cf0cfcd7-c16d-4bea-a316-3a7b3f69d5b7
[INFO] [10/06/2018 22:54:12.280] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:12.280] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:12.280] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:14.339] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:16.296] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 7
[INFO] [10/06/2018 22:54:16.300] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 900b4ee1-cf61-4c7e-813e-4959604584b8
[INFO] [10/06/2018 22:54:16.306] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:16.306] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:16.306] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:18.830] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:19.348] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:21.318] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:54:21.323] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:21.323] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:21.327] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/producer] Produced work: 8
[INFO] [10/06/2018 22:54:21.329] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 2052afb2-e19f-4fd3-b474-8a8fdd73ede2
[INFO] [10/06/2018 22:54:21.332] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:21.332] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:21.332] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:24.342] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:25.349] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/producer] Produced work: 9
[INFO] [10/06/2018 22:54:25.351] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 98ac1ecf-223b-4f3d-a56b-8d59fa5ce544
[INFO] [10/06/2018 22:54:25.354] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:25.354] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:25.354] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:28.821] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:29.340] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:30.369] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:54:30.371] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:30.371] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:33.367] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/producer] Produced work: 10
[INFO] [10/06/2018 22:54:33.369] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: b3db93cb-521a-4f9f-ab35-5ad0d7895b32
[INFO] [10/06/2018 22:54:33.372] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:33.372] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:33.372] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:34.352] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:38.388] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:54:38.390] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:38.390] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:38.830] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:39.342] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:40.388] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/producer] Produced work: 11
[INFO] [10/06/2018 22:54:40.391] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 89984b02-19d9-404b-945d-0e7a8d3529ed
[INFO] [10/06/2018 22:54:40.402] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:40.402] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:40.402] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:44.340] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:45.420] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:54:45.423] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:45.423] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:48.419] [ClusterSystem-akka.actor.default-dispatcher-22] [akka://ClusterSystem/user/producer] Produced work: 12
[INFO] [10/06/2018 22:54:48.422] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 3bce5c92-7e5c-46bf-b551-dd6905294c9f
[INFO] [10/06/2018 22:54:48.436] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:48.436] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:48.436] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:48.830] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:49.353] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:53.448] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:54:53.450] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:53.450] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:54.339] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:54.455] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 13
[INFO] [10/06/2018 22:54:54.458] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 44558722-46c9-446e-ad9d-65a783e4ff86
[INFO] [10/06/2018 22:54:54.463] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:54.463] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:54.463] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:54:58.824] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:54:59.340] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:54:59.482] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:54:59.484] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:54:59.484] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:01.475] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 14
[INFO] [10/06/2018 22:55:01.477] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 164c35e5-d857-4b95-8c2d-f64deb707280
[INFO] [10/06/2018 22:55:01.481] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:01.481] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:01.481] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:04.353] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:06.499] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:06.501] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:06.501] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:08.509] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/producer] Produced work: 15
[INFO] [10/06/2018 22:55:08.511] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: d87b9664-98bf-4c11-ac24-47e57579552c
[INFO] [10/06/2018 22:55:08.515] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:08.515] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:08.515] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:08.834] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:09.341] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:13.530] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:13.532] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:13.532] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:14.351] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:15.527] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/producer] Produced work: 16
[INFO] [10/06/2018 22:55:15.532] [ClusterSystem-akka.actor.default-dispatcher-8] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 38f5deec-35cf-47fa-b8e4-1fd33884d7d4
[INFO] [10/06/2018 22:55:15.540] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:15.541] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:15.541] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:18.560] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/producer] Produced work: 17
[INFO] [10/06/2018 22:55:18.562] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 1144ec1f-e8e0-4fba-83f3-21ca2f31839f
[INFO] [10/06/2018 22:55:18.567] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:18.567] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:18.567] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:18.831] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:19.350] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:22.580] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/producer] Produced work: 18
[INFO] [10/06/2018 22:55:22.582] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 22810256-d265-4b9f-bd99-edce97e60cd1
[INFO] [10/06/2018 22:55:22.650] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:22.650] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:22.650] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:24.340] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:27.662] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:27.664] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:27.664] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:28.824] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:29.345] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:30.669] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 19
[INFO] [10/06/2018 22:55:30.671] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 86bca828-4ec3-498e-8323-442e79085666
[INFO] [10/06/2018 22:55:30.674] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:30.674] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:30.674] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:34.341] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:35.690] [WorkerSystem-akka.actor.default-dispatcher-2] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:35.691] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:35.691] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:38.833] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:39.353] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:39.690] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/producer] Produced work: 20
[INFO] [10/06/2018 22:55:39.693] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 47ad1a5b-d869-4f8a-a91d-e047288601a9
[INFO] [10/06/2018 22:55:39.697] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:39.697] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:39.697] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:44.343] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:44.710] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:44.712] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:44.712] [ClusterSystem-akka.actor.default-dispatcher-3] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:46.710] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 21
[INFO] [10/06/2018 22:55:46.712] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: df36abc0-004a-4fab-9e98-c13e371e0035
[INFO] [10/06/2018 22:55:46.816] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:46.816] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:46.816] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:48.835] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:49.335] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:51.822] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:51.825] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:51.825] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:54.342] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:54.829] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 22
[INFO] [10/06/2018 22:55:54.831] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 211ab4fd-e57f-41f5-b492-7b9b0627d785
[INFO] [10/06/2018 22:55:54.834] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:54.834] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:54.834] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:55:58.835] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:55:59.356] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:55:59.854] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:55:59.856] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:55:59.856] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:02.847] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 23
[INFO] [10/06/2018 22:56:02.849] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: a63d4fe7-1d75-495b-9a23-47e9e8b45bd4
[INFO] [10/06/2018 22:56:02.913] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:02.913] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:02.913] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:04.342] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:56:07.934] [WorkerSystem-akka.actor.default-dispatcher-16] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:56:07.936] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:56:07.936] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:08.836] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:09.355] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:56:10.938] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/producer] Produced work: 24
[INFO] [10/06/2018 22:56:10.941] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 8a1dbac1-7458-4b8e-9d6e-a8c606b8e7ce
[INFO] [10/06/2018 22:56:11.061] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:11.061] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:11.061] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:14.335] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:56:16.085] [WorkerSystem-akka.actor.default-dispatcher-15] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:56:16.087] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:56:16.087] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:18.078] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 25
[INFO] [10/06/2018 22:56:18.081] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 2ae5a703-f979-40cc-b1b9-34a8ad80b56f
[INFO] [10/06/2018 22:56:18.086] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:18.086] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:18.086] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:18.825] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:19.345] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:56:23.103] [WorkerSystem-akka.actor.default-dispatcher-4] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:56:23.105] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:56:23.105] [ClusterSystem-akka.actor.default-dispatcher-6] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:24.342] [ClusterSystem-akka.actor.default-dispatcher-14] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:56:25.089] [ClusterSystem-akka.actor.default-dispatcher-5] [akka://ClusterSystem/user/producer] Produced work: 26
[INFO] [10/06/2018 22:56:25.091] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 4d6e53f1-d6f8-4478-a86d-4c706c0c819d
[INFO] [10/06/2018 22:56:25.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:25.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:25.096] [ClusterSystem-akka.actor.default-dispatcher-2] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:28.834] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under instance RegisterWorker with key containing worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:29.112] [ClusterSystem-akka.actor.default-dispatcher-12] [akka://ClusterSystem/user/producer] Produced work: 27
[INFO] [10/06/2018 22:56:29.115] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 71e6b052-fd4a-4732-89cc-d4e7b99d7e5e
[INFO] [10/06/2018 22:56:29.118] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:29.118] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:29.118] [ClusterSystem-akka.actor.default-dispatcher-10] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:29.346] [ClusterSystem-akka.actor.default-dispatcher-4] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick
[INFO] [10/06/2018 22:56:34.132] [WorkerSystem-akka.actor.default-dispatcher-5] [akka://WorkerSystem/user/worker] No ack from master, retrying (ba438def-2074-42b9-a522-9147c828ecdf -> dd37150b-276d-4165-8f95-e1d389d34bda)
[INFO] [10/06/2018 22:56:34.133] [ClusterSystem-akka.actor.default-dispatcher-21] [akka://ClusterSystem/user/producer] Produced work: 28
[INFO] [10/06/2018 22:56:34.135] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Inside Master under instance WorkIsDone with worker ba438def-2074-42b9-a522-9147c828ecdf some work dd37150b-276d-4165-8f95-e1d389d34bda
[INFO] [10/06/2018 22:56:34.135] [ClusterSystem-akka.actor.default-dispatcher-7] [akka://ClusterSystem/user/master/singleton] Work dd37150b-276d-4165-8f95-e1d389d34bda not in progress, reported as done by worker ba438def-2074-42b9-a522-9147c828ecdf
[INFO] [10/06/2018 22:56:34.136] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] Inside Master under state Work -- Accepted work: 8a5340d2-f05a-4d9c-8d54-4d0531c3570c
[INFO] [10/06/2018 22:56:34.139] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:34.139] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside if workstate has work notifyWorkers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:34.139] [ClusterSystem-akka.actor.default-dispatcher-9] [akka://ClusterSystem/user/master/singleton] inside notifyWorkers affected workers [ba438def-2074-42b9-a522-9147c828ecdf]
[INFO] [10/06/2018 22:56:34.346] [ClusterSystem-akka.actor.default-dispatcher-11] [akka://ClusterSystem/user/master/singleton] Inside Master checking the CleanupTick

thanks in advance.