Writing a Redis Client with a Free Algebra
Does this sound interesting?
We're HiringSharethrough Engineering takes these things seriously: testability, speed, and the freedom to engineer elegant solutions. Many of our apps are written in Scala, and we leverage the benefits of a statically-typed language as much as possible. Type-checking is one benefit, but another is the ability to program with pure functions.
In languages like Haskell, developers can clearly separate the effects of a program (e.g. writing to a file handle) from pure computation. That separation is enforced because functions are pure, which just means that the result of any function call is fully determined by its arguments. Pure languages employ abstractions such as the IO monad to “get things done,” like printing to standard out, writing to databases, and the like.
Scala supports both pure and impure code, which makes this harder, but we can isolate and sequence these effects in Scala with the same abstractions. It’s just up to us to use them.
A Redis algebra
After seeing Rúnar Bjarnason’s talk at ScalaDays 2014 (pdf) on free monads that describe programs with disparate classes of effects, I’ve wondered if I could build anything similar that could test, in a pure way, the side effect of persisting to a key-value store.
I stumbled across two excellent projects since then, one being ethul’s free algebra describing redis actions, the other being the Stoop library for CouchDB interaction (by J. Kerr, D. Wall & R. Bjarnason). Both describe database actions as algebras (think of “algebras” as mini-DSLs), and build interpreters for them. We often need to run multiple Redis commands in sequence, and mocking and testing these side-effects is clunky.
Here follows a few snippets of code describing a miniature boiled-down toy version of an algebra for Redis actions (with a whole two functions - wow!), in the hope that it helps others to have a play with the method. (Check out the edhul/redis-algebra and related interpreter for a full-blown redis algebra.) First lets import some scalaz magic (caveat - this isn’t magic, it’s just pure functions).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// This should work with Scala 2.10.4 & scalaz 7.1,
// using the core, effect, and concurrent packages
import scalaz._, Scalaz._
import Free.{freeMonad => _, _}
// Describe the set of actions - which are functors:
sealed trait RedisF[+A] {
def map[B](fn: A => B): RedisF[B]
}
object RedisF {
implicit val redisFunctor: Functor[RedisF] = new Functor[RedisF] {
def map[A, B](fa: RedisF[A])(f: A => B): RedisF[B] = fa map f
}
}
// and some instances of that trait describing our desired actions:
case class Get[+A](k: String, next: String => A) extends RedisF[A] {
def map[B](fn: A => B): RedisF[B] = copy(next = next andThen fn)
}
case class Set[+A](k: String, value: String, next: String => A) extends RedisF[A] {
def map[B](fn: A => B): RedisF[B] = copy(next = next andThen fn)
}
case class Fail[A](excp: Throwable) extends RedisF[A] {
def map[B](fn: A => B): RedisF[B] = Fail[B](excp)
}
// Now we have a simple dsl for getting and setting strings to Redis.
We’ll need a few definitions, helper functions, and instances:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
object FreeRedis {
type RedisFree[A] = Free[RedisF, A]
def just[A](a: => A): RedisFree[A] =
Monad[RedisFree].pure(a)
def noAction[A]: RedisFree[A] =
liftF(Fail(new Throwable("Action failed"))): RedisFree[A]
// Helper functions describing our redis commands below
def set(k: String, v: String): RedisFree[String] =
liftF(Set(k, v, identity))
def get(k: String): RedisFree[String] =
liftF(Get(k, identity))
implicit val redisFreeMonad: Monad[RedisFree] = new Monad[RedisFree] {
def point[A](a: => A) = Free.point(a)
def bind[A,B](action: RedisFree[A])(f: A => RedisFree[B]) = action flatMap f
}
}
A type variable, an instance of Monad for RedisFree, and some functions for lifting values into the monad. With this we have absolutely everything we need to define some sequences (programs).
Trying it out
How about setting a value and getting it?
1
2
3
4
5
6
7
// Ensure FreeRedis stuff is in scope...
import FreeRedis._
// Make a pure program value
val program1: RedisFree[String] = set("keyA", "valueB") >> get("keyA")
// n.b. ">>" is scalaz for forget the bound value and just sequence the next action
program1 is a pure program, but it doesn’t do anything yet. We have to build an interpreter for the program. We start by stepping through the program (folding) and building a stringified list of actions. We use the resume method to do this from Free.
1
2
3
4
5
6
7
8
9
object Interpreters extends FreeRedis {
// Take a program and turn each action
// into a string, aggregate those strings in a list
def stepList[A](program: RedisFree[A], actions: List[String] = Nil): List[String] = program.resume.fold({
case Set(k, v, next) => stepList(next("success"), s"setting key: ${k} value = ${v}" :: actions)
case Get(k, next) => stepList(next("success"), s"getting key: ${k}" :: actions)
case Fail(excp) => ("ERROR!" :: actions).reverse
}, { a: A => ("End of program" :: actions).reverse })
}
And interpreting it to produce another pure value:
1
2
> Interpreters.stepList(program1)
> res0: List[String] = List(setting key: keyA value = valueB, getting key: keyA, End of program)
We successfully interpreted the actions and collected them up in a list.
How about introducing an error:
1
2
3
4
5
// Ensure FreeRedis stuff is in scope...
val program2: RedisFree[String] = set("keyA", "valueB") >> fail(new Throwable("connection died")) >> get("keyA")
> Interpreters.stepList(program2)
> res1: List[String] = List(setting key: keyA value = valueB, ERROR!)
Adding some side-effects
We’ve converted the program into another data format. Now let’s try using this for side effects. How about just printing to console? We’re going to need to build a different program interpreter.
1
2
3
4
5
6
7
8
9
// import Task...
import scalaz.concurrent.Task
// Interpret a step (action) of our program and produce a Task which we can run later
def step[A](action: RedisF[RedisFree[A]]): Task[RedisFree[A]] = action match {
case Set(k, v, next) => Task { println(s"setting key: ${k} value = ${v}"); "success" } map { next }
case Get(k, next) => Task { println(s"getting key: ${k}"); "I'm the key you got!" } map { next }
case Fail(excp) => Task.fail(excp)
}
Then run the new side-effect generating interpreter for our previous program values (program1 and 2) like this:
1
2
3
4
5
6
7
8
9
10
> program1.runM(step)
> res0: scalaz.concurrent.Task[String] = scalaz.concurrent.Task@7ae246cd // got the task ready to roll
> res0.run
> setting key: keyA value = valueB
> getting key: keyA
> res1: String = I'm the key you got!
> program2.runM(step).attempt.run // intercept exceptions with `attempt` (we know our program will throw an exception)
> setting key: keyA value = valueB
res2: scalaz.\/[Throwable,String] = -\/(java.lang.Throwable: Action failed)
And there we have it, a successfully side-effecting program. Imagine that the side-effecting function you call inside the Task was actually a client library for Redis, and it produced a value instead of “printing to console, semicolon, then a pure value” I typed. You can subsitute any set of actions for the ones I used; this makes testing, mocking, and sanity-checking simple and painless.
When I first got this to work (thanks to gratuitous copying and pasting from the stoop library, it blew my mind. We can now interpret our effects in a completely pure and composable way, enabling us to “mock” effects and test our code completely. And with Task, you have access to very useful combinators for retrying and waiting etc.
I hope this little run through is enough to get you interested in having a go yourself and reading more. You can download this code and have a go by going to this gist snippet I for one will definitely be trying to learn as much as I can! Please comment if you find any mistakes or errors in my admittedly basic understanding.