Code Monkey home page Code Monkey logo

rediscala's Introduction

rediscala Build Status Coverage Status Maven Central

A Redis client for Scala (2.11+) and (AKKA 2.5+) with non-blocking and asynchronous I/O operations.

  • Reactive : Redis requests/replies are wrapped in Futures.

  • Typesafe : Redis types are mapped to Scala types.

  • Fast : Rediscala uses redis pipelining. Blocking redis commands are moved into their own connection. A worker actor handles I/O operations (I/O bounds), another handles decoding of Redis replies (CPU bounds).

State of the project

This is an unofficial fork from etaty/rediscala maintained by @herzrasen, @kardapoltsev and @Ma27 which was started as the original maintainer was too unresponsive to keep the project alive as discussed in etaty/rediscala#191.

Note: please keep in mind that we mainly want to keep the project alive, but lack time to do active development. If you're interested in helping out, just open an issue.

Set up your project dependencies

If you use SBT, you just have to edit build.sbt and add the following:

From version 1.8.0:

  • use Akka 2.5 (Java 1.8)
  • released for Scala 2.12, 2.13
libraryDependencies += "com.github.Ma27" %% "rediscala" % "1.9.1"

Connect to the database

import redis.RedisClient
import scala.concurrent.Await
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

object Main extends App {
  implicit val akkaSystem = akka.actor.ActorSystem()

  val redis = RedisClient()

  val futurePong = redis.ping()
  println("Ping sent!")
  futurePong.map(pong => {
    println(s"Redis replied with a $pong")
  })
  Await.result(futurePong, 5 seconds)

  akkaSystem.terminate()
}

Basic Example

https://github.com/etaty/rediscala-demo

You can fork with : git clone [email protected]:etaty/rediscala-demo.git then run it, with sbt run

Redis Commands

All commands are supported :

Blocking commands

RedisBlockingClient is the instance allowing access to blocking commands :

  • blpop
  • brpop
  • brpopplush
  redisBlocking.blpop(Seq("workList", "otherKeyWithWork"), 5 seconds).map(result => {
    result.map({
      case (key, work) => println(s"list $key has work : ${work.utf8String}")
    })
  })

Full example: ExampleRediscalaBlocking

You can fork with: git clone [email protected]:etaty/rediscala-demo.git then run it, with sbt run

Transactions

The idea behind transactions in Rediscala is to start a transaction outside of a redis connection. We use the TransactionBuilder to store call to redis commands (and for each command we give back a future). When exec is called, TransactionBuilder will build and send all the commands together to the server. Then the futures will be completed. By doing that we can use a normal connection with pipelining, and avoiding to trap a command from outside, in the transaction...

  val redisTransaction = redis.transaction() // new TransactionBuilder
  redisTransaction.watch("key")
  val set = redisTransaction.set("key", "abcValue")
  val decr = redisTransaction.decr("key")
  val get = redisTransaction.get("key")
  redisTransaction.exec()

Full example: ExampleTransaction

You can fork with : git clone [email protected]:etaty/rediscala-demo.git then run it, with sbt run

TransactionsSpec will reveal even more gems of the API.

Pub/Sub

You can use a case class with callbacks RedisPubSub or extend the actor RedisSubscriberActor as shown in the example below

object ExamplePubSub extends App {
  implicit val akkaSystem = akka.actor.ActorSystem()

  val redis = RedisClient()

  // publish after 2 seconds every 2 or 5 seconds
  akkaSystem.scheduler.schedule(2 seconds, 2 seconds)(redis.publish("time", System.currentTimeMillis()))
  akkaSystem.scheduler.schedule(2 seconds, 5 seconds)(redis.publish("pattern.match", "pattern value"))
  // shutdown Akka in 20 seconds
  akkaSystem.scheduler.scheduleOnce(20 seconds)(akkaSystem.terminate())

  val channels = Seq("time")
  val patterns = Seq("pattern.*")
  // create SubscribeActor instance
  akkaSystem.actorOf(Props(classOf[SubscribeActor], channels, patterns).withDispatcher("rediscala.rediscala-client-worker-dispatcher"))

}

class SubscribeActor(channels: Seq[String] = Nil, patterns: Seq[String] = Nil) extends RedisSubscriberActor(channels, patterns) {
  override val address: InetSocketAddress = new InetSocketAddress("localhost", 6379)

  def onMessage(message: Message) {
    println(s"message received: $message")
  }

  def onPMessage(pmessage: PMessage) {
    println(s"pattern message received: $pmessage")
  }
}

Full example: ExamplePubSub

You can fork with : git clone [email protected]:etaty/rediscala-demo.git then run it, with sbt run

RedisPubSubSpec will reveal even more gems of the API.

Scripting

RedisScript is a helper, you can put your LUA script inside and it will compute the hash. You can use it with evalshaOrEval which run your script even if it wasn't already loaded.

  val redis = RedisClient()

  val redisScript = RedisScript("return 'rediscala'")

  val r = redis.evalshaOrEval(redisScript).map({
    case b: Bulk => println(b.toString())
  })
  Await.result(r, 5 seconds)

Full example: ExampleScripting

Redis Sentinel

SentinelClient connect to a redis sentinel server.

SentinelMonitoredRedisClient connect to a sentinel server to find the master addresse then start a connection. In case the master change your RedisClient connection will automatically connect to the new master server. If you are using a blocking client, you can use SentinelMonitoredRedisBlockingClient

Pool

RedisClientPool connect to a pool of redis servers. Redis commands are dispatched to redis connection in a round robin way.

Master Slave

RedisClientMasterSlaves connect to a master and a pool of slaves. The write commands are sent to the master, while the read commands are sent to the slaves in the RedisClientPool

Config Which Dispatcher to Use

By default, the actors in this project will use the dispatcher rediscala.rediscala-client-worker-dispatcher. If you want to use another dispatcher, just config the implicit value of redisDispatcher:

implicit val redisDispatcher = RedisDispatcher("akka.actor.default-dispatcher")

ByteStringSerializer ByteStringDeserializer ByteStringFormatter

ByteStringSerializer

ByteStringDeserializer

ByteStringFormatter

case class DumbClass(s1: String, s2: String)

object DumbClass {
  implicit val byteStringFormatter = new ByteStringFormatter[DumbClass] {
    def serialize(data: DumbClass): ByteString = {
      //...
    }

    def deserialize(bs: ByteString): DumbClass = {
      //...
    }
  }
}
//...

  val dumb = DumbClass("s1", "s2")

  val r = for {
    set <- redis.set("dumbKey", dumb)
    getDumbOpt <- redis.get[DumbClass]("dumbKey")
  } yield {
    getDumbOpt.map(getDumb => {
      assert(getDumb == dumb)
      println(getDumb)
    })
  }

Full example: ExampleByteStringFormatter

Scaladoc

Rediscala scaladoc API (version 1.8)

Rediscala scaladoc API (version 1.7)

Rediscala scaladoc API (version 1.6)

Rediscala scaladoc API (version 1.5)

Rediscala scaladoc API (version 1.4)

Rediscala scaladoc API (version 1.3)

Rediscala scaladoc API (version 1.2)

Rediscala scaladoc API (version 1.1)

Rediscala scaladoc API (version 1.0)

Performance

More than 250 000 requests/second

The hardware used is a macbook retina (Intel Core i7, 2.6 GHz, 4 cores, 8 threads, 8GB) running the sun/oracle jvm 1.6

You can run the bench with :

  1. clone the repo git clone [email protected]:etaty/rediscala.git
  2. run sbt bench:test
  3. open the bench report rediscala/tmp/report/index.html

rediscala's People

Contributors

123avi avatar actsasbuffoon avatar adrien-aubel avatar alexanderscott avatar analytically avatar balagez avatar btd avatar ddworak avatar dontgitit avatar etaty avatar fehmicansaglam avatar gchudnov avatar herzrasen avatar hussachai avatar jtanner avatar kardapoltsev avatar karelcemus avatar ma27 avatar matthewedge avatar mmagn avatar npeters avatar ryanlecompte avatar stanch avatar stsmedia avatar th0br0 avatar tovbinm avatar vshalts avatar wb14123 avatar xiaohai2016 avatar xuwei-k avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

rediscala's Issues

Test are not stable

There are a lot of hardcoded timeouts in tests and they are not stable (tests fail on slow environment like travis).
Also, what do you think about moving to scalatest instead of specs? I have some free time now for this project (:

zrevrangebylex uses max parameter for min

This is the definition for zrevrangebylex:

def zrevrangebylex[R: ByteStringDeserializer](key: String, max: Option[String], min: Option[String], limit: Option[(Long, Long)] = None): Future[Seq[R]] =
send(Zrevrangebylex(key, max.getOrElse("+"), max.getOrElse("-"), limit))

It appears to be using the max parameter for both min and max arguments.

Compile with Scala 2.12.4

When trying to bump the Scala version in our SBT build to 2.12.4 the following happens:

[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /home/ma27/Projects/rediscala/project
[info] Loading settings from build.sbt,version.sbt ...
[info] Set current project to rediscala (in build file:/home/ma27/Projects/rediscala/)
[info] Executing in batch mode. For better performance use sbt's shell
[info] Updating ...
[info] Done updating.
[info] Compiling 51 Scala sources and 1 Java source to /home/ma27/Projects/rediscala/target/scala-2.12/classes ...
[error] /home/ma27/Projects/rediscala/src/main/scala/redis/api/Keys.scala:105:24: case classes should have a non-implicit parameter list; adapting to 'case class Randomkey()(...)'
[error] case class Randomkey[R](implicit deserializerR: ByteStringDeserializer[R]) extends RedisCommandBulkOptionByteString[R] {
[error]                        ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 3 s, completed Mar 17, 2018 7:02:04 PM

We have to investigate if that's a compiler bug or an intentional change and fix our code accordingly.

Fix SBT deprecations

[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:83: `<++=` operator is deprecated. Try `lhs ++= { x.value }`
[warn]   or see http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html.
[warn]       scalacOptions in (Compile, doc) <++= baseDirectory in LocalProject("rediscala") map { bd =>
[warn]                                       ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:90: `<++=` operator is deprecated. Try `lhs ++= { x.value }`
[warn]   or see http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html.
[warn]       scalacOptions in (Compile, doc) <++= version in LocalProject("rediscala") map { version =>
[warn]                                       ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:97: `<++=` operator is deprecated. Try `lhs ++= { x.value }`
[warn]   or see http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html.
[warn]       siteMappings <++= (mappings in packageDoc in Compile, version in LocalProject("rediscala")) { (mm, version) =>
[warn]                    ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:102: `<<=` operator is deprecated. Use `key := { x.value }` or `key ~= (old => { newValue })`.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]       cleanSite <<= (updatedRepository, git.gitRunner, streams, version in LocalProject("rediscala")) map { (dir, git, s, v) =>
[warn]                 ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:111: `<<=` operator is deprecated. Use `key := { x.value }` or `key ~= (old => { newValue })`.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]       synchLocal <<= (cleanSite, privateMappings, updatedRepository, ghpagesNoJekyll, git.gitRunner, streams) map { (clean, mappings, repo, noJekyll, git, s) =>
[warn]                  ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:126: `<<=` operator is deprecated. Use `key := { x.value }` or `key ~= (old => { newValue })`.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]     sourceDirectory in BenchTest <<= baseDirectory / "src/benchmark",
[warn]                                  ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:131: `<<=` operator is deprecated. Use `key := { x.value }` or `key ~= (old => { newValue })`.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]     testGrouping in BenchTest <<= definedTests in BenchTest map partitionTests
[warn]                               ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:46: trait Build in package sbt is deprecated: Use .sbt format instead
[warn] object RediscalaBuild extends Build {
[warn]                               ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:50: value defaultSettings in object Defaults is deprecated: Default settings split into coreDefaultSettings and IvyModule/JvmModule plugins.
[warn]   lazy val standardSettings = Defaults.defaultSettings ++
[warn]                                        ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:97: method t2ToApp2 in object Scoped is deprecated: The sbt 0.10 style DSL is deprecated: '(k1, k2) map { (x, y) => ... }' should now be '{ val x = k1.value; val y = k2.value }'.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]       siteMappings <++= (mappings in packageDoc in Compile, version in LocalProject("rediscala")) { (mm, version) =>
[warn]                         ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:102: method t4ToTable4 in object Scoped is deprecated: The sbt 0.10 style DSL is deprecated: '(k1, k2) map { (x, y) => ... }' should now be '{ val x = k1.value; val y = k2.value }'.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]       cleanSite <<= (updatedRepository, git.gitRunner, streams, version in LocalProject("rediscala")) map { (dir, git, s, v) =>
[warn]                     ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:111: method t6ToTable6 in object Scoped is deprecated: The sbt 0.10 style DSL is deprecated: '(k1, k2) map { (x, y) => ... }' should now be '{ val x = k1.value; val y = k2.value }'.
[warn] See http://www.scala-sbt.org/0.13/docs/Migrating-from-sbt-012x.html
[warn]       synchLocal <<= (cleanSite, privateMappings, updatedRepository, ghpagesNoJekyll, git.gitRunner, streams) map { (clean, mappings, repo, noJekyll, git, s) =>
[warn]                      ^
[warn] /home/travis/build/Ma27/rediscala/project/Rediscala.scala:126: method / in class RichFileSetting is deprecated: Use a standard setting definition.
[warn]     sourceDirectory in BenchTest <<= baseDirectory / "src/benchmark",
[warn]                                                    ^
[warn] 13 warnings found

Requests never completed if decoding fails

Operation completes its promise with the result of redisCommand.decodeReply (in completeSuccess and tryCompleteSuccess).
However, if the deserializer throws an exception, it never completes the promise (and I think the thrown exception might cause other issues too).
This is possible with custom ByteStringDeserializer or RedisReplyDeserializer that throws in certain scenarios. It's even possible if you try to get something holding an unexpected value, for instance, set("key", "value") followed by get[Double]("key"). The ActorSystem ends up dumping something like java.lang.NumberFormatException: For input string: "value", but the Future returned by get never completes.

Maybe it should be changed to something like:

  def completeSuccess(redisReply: RedisReplyT): Promise[T] = {
    promise.complete(Try(redisCommand.decodeReply(redisReply)))
  }
  def tryCompleteSuccess(redisReply: RedisReply) = {
    promise.tryComplete(Try(redisCommand.decodeReply(redisReply.asInstanceOf[RedisReplyT])))
  }

Move to GitLab?

It seems to be a thing to move repositories to GitLab ATM.
Except the Microsoft deal I really like GitLab CI (currently using it at work) which allows advanced features like custom containers or build outputs. Additionally I personally prefer the GitLab UI over GitHub's UI. As many people started moving to GitLab I decided to do this as well with my repos hoping that enough people use gitlab.com now, so it becomes way more relevant.

@kardapoltsev @herzrasen as you have commit access to this repo, what do you think about this? This is just my own opinion, I'll keep my GitHub account for several reasons, so in case you disagree it's absolutely fine to continue working here :-)

Get rid of `/usr/local/bin` reference in `RedisTest`

I'm not a big fan of placing random binaries into /usr/local/bin using hacky shell scripts. Instead it's way easier to use actual package managers that make it less likely to screw up your entire system.
Let's evaluate some alternatives for install-redis.sh and refactor the RedisTest to get rid of the /usr/local references.

Connecting to a sentinel should not take into account s_down slaves

We are using a Redis HA deployment that has 3 sentinels, 1 master, and 2 slaves. When using rediscala client for scala, we use SentinelMonitoredRedisClientMasterSlaves.

There is a bug in the code that, when this client gets the list of slaves from the sentinel, it gets all the slaves that the sentinel has recorded, regardless of the flags. As you may know, a slave recorded in a sentinel has a property called flag, that usually contains the value slave. If a slave goes down, sentinels record it and also adds the flag s_down to the list of flags of that slave.

So basically a sentinel could have a list of 4 slaves, but only 2 of them would be active, and the rest of them would have flags="slave,s_down". The result of this is that we see in the logs that the client successfully connects to the active slaves, but also tries indefinitely to connect to the non-active slaves. This does not block anything, but it pollutes the logs with Trying to connect to XYZ and then fail with Error trying to connect to XYZ because that slave is already not active. I'm guessing that one actor is created per slave to connect, so these actors are left hanging around trying to connect to nothing.

Thanks.

Scala 2.13

Are there any plans for Scala 2.13 cross-release? It would be great.

Question: Thoughts on mocking underlying connection in tests?

Hello!

I am new to Scala, Akka, and this library, so please excuse my naiveté. I'm working on a very simple little Redis-backed job queue library to learn about Akka and Scala, and I'm using rediscala to deal with Redis. I was hoping to write pure in-memory unit tests, but ScalaMock seems to struggle with mocking RedisClient, because it depends on implicit constructors arguments:

My band-aid solution was to use a subclassed fake of the RedisClient, and override the methods necessary for the sake of my tests.

One of the problems I'm having now is that overriding a generic method that takes a type argument in its signature is very difficult to do when the overridden version should return a concrete type.

The other, perhaps more significant issue, is that my fake is still attempting to connect to a real Redis server. When I run my tests in an environment where there is no real Redis server listening for connections at the default localhost:6379, the tests still pass, and because I shut down the Actor system in an "AfterAll" hook, the connection failures only happen once, and then reconnections cease to attempt to be made. However, this limitation raises concerns for me, because I want my unit tests to be exclusively in-memory, deterministic, and focused only on my higher-level domain logic (such as there is any in my simple library), not on the implementation detail of how RedisClient works under the hood.

Do you have any suggestions about how to get these tests to run without actually invoking the connection logic, or somehow allow the tests to only concern themselves with the inputs and return values of the methods RedisClient::llen and RedisClient::lpop? (If anyone is feeling generous, I also wouldn't mind having my terrible Scala code ripped to shreds, learning new things in total isolation is hard! 😄)

Thanks!!!

Examine approaches to allow multiple publishers for this package

it seemed to me as the namespace for a package on Maven Central had to be owned by a single user. However we should check if there are alternatives to allow multiple users to publish new releases.

This would be pretty helpful as I'm rather busy these days and can't give this library the time it actually deserves.

cc @kardapoltsev @herzrasen (and anybody else who's interested in helping out).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.