swissborg / lithium Goto Github PK

Using "com.typesafe.akka" %% "akka-cluster-sharding-typed" % "2.6.4"

When running lithium with the following config

akka {
  actor {
    provider = "cluster"
    no-serialization-verification-needed-class-prefix = [
      "akka.",
      "com.swissborg.lithium"
    ]
  }
  cluster.downing-provider-class = "com.swissborg.lithium.DowningProviderImpl"
}
com.swissborg.lithium {
  active-strategy = "keep-oldest"
  stable-after = 30s
  keep-oldest {
    down-if-alone = yes
  }
}

I see the following log on startup

2020-04-22 09:44:01,562 INFO  com.swissborg.lithium.DowningProviderImpl - Starting Lithium with the keep-oldest strategy.
2020-04-22 09:44:01,664 INFO  akka.actor.LocalActorRef - Message [akka.cluster.ClusterEvent$CurrentClusterState] from Actor[akka://MySystem/system/cluster/core/publisher#-317651006] to Actor[akka://MySystem/system/cluster/core/daemon/downingProvider/split-brain-reporter/reachability-reporter#-978528538] was unhandled. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

I do not see the unhandled warning when running lithium on akka-classic.

Is this message cause for concern or can it be ignored? @DennisVDB

akka.cluster.down-removal-margin should be set to the same value as com.swissborg.lithium.stable-after. Otherwise, shards and singletons could be migrated too early.

Hi @DennisVDB Thanks for the implementation :) There are only a few available and I'm glad you started this one.

We use akka 2.5.25 in prod and cannot upgrade to 2.6.x yet, so in order to use lithium, we need to use version 0.9.5 of it, which can work with akka 2.5.25.

I'm trying to use the instructions from readme from the same version which states :

1. resolvers += Resolver.bintrayRepo("swissborg", "maven")
2. libraryDependencies += "com.swissborg" %% "lithium" % "0.9.5"

which results in

(ssExtractDependencies) sbt.librarymanagement.ResolveException: Error downloading com.swissborg:lithium_2.13:0.9.5
[error]   Not found
[error]   Not found
[error]   not found: /Users/nikhilpatil/.ivy2/local/com.swissborg/lithium_2.13/0.9.5/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/com/swissborg/lithium_2.13/0.9.5/lithium_2.13-0.9.5.pom
[error]   not found: https://dl.bintray.com/swissborg/maven/com/swissborg/lithium_2.13/0.9.5/lithium_2.13-0.9.5.pom

I also tried adding 2.12 in the artifactid part
libraryDependencies += "com.swissborg" %% "lithium_2.12" % "0.9.5"
which results in similar error, except this time it tries to download from follorwing url
https://dl.bintray.com/swissborg/maven/com/swissborg/lithium_2.12_2.13/0.9.5/lithium_2.12_2.13-0.9.5.pom

I can see that the artifacts exists at https://repo.jfrog.org/artifactory/libs-release-bintray/com/swissborg/. I am not sure how to correctly refer to it.

Members that are gracefully exiting can still become unreachable. Normally this is not a problem since the leader will remove exiting+unreachable members from the cluster on convergence. Thus not hindering it.
In its detection of a network partition these nodes are still counted and thus puts Lithium in a state where it waits for the cluster to stabilize before issuing a downing command. However, for instance during a k8s rolling update, a lot of nodes will gracefully leave the cluster, potentially become exiting+unreachable, and if the rolling update is longer than stable-after + down-all-when-unstable this could lead to the cluster being downed.

Hi,

What's the current status of this project? Do you have a rough roadmap or are the features stable at this point? Do you know of anybody using this in production? If so, do you know of multiple groups using this in production?

I am using Lagom 1.5.4 which uses scala 2.12 and akka 2.5
I tried to add the swizz borg dependency but it fails due to incompatible akka version

        <dependency>
            <groupId>com.swissborg</groupId>
            <artifactId>lithium_2.12</artifactId>
            <version>0.11.2</version>
        </dependency>

During the startup the app fails

Detected java.lang.NoSuchMethodError error, which MAY be caused by incompatible Akka versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use akka-actor [2.5.25 (resolved from current classpath)] all other core Akka modules MUST be of the same version. External projects like Alpakka, Persistence plugins or Akka HTTP etc. have their own version numbers - please make sure you're using a compatible set of libraries.   

which versions should I use with lagom 1.5.4 ?
Should I exclude the transitive dependency of akka 2.6 from lithium? Will the downing provider work with akka 2.5?

One test case that we have observed in prod settings is detecting potential SBR, but then not downing nodes because the nodes reconnect again. Consider adding such a test case, if you don't have one already.

Code has drifted
https://github.com/SwissBorg/lithium/blob/master/src/multi-jvm/scala/com/swissborg/lithium/strategy/staticquorum/StaticQuorumSpec9.scala

A node that has detected itself as indirectly-connected is never seen as reachable again.

Potential problem (TBD):
Akka's Reachable event seems to be never sent to the node being seen as reachable again. Hence, since Lithium listen to that event to come out of the indirectly-connected status a node that detected itself as indirectly-connected cannot become reachable again.

The name of the downing provider is a bit ugly. Another name should be found for it. LithiumProvider?