Code Monkey home page Code Monkey logo

circe-golden's Introduction

circe-golden

Build status Coverage status Gitter Maven Central

Golden testing for Circe encoders and decoders.

Motivation

One common criticism of deriving type class instances in the context of serialization is that it makes it too easy to accidentally break compatibility with other systems, since the magic of derivation can obscure the fact that changes to our data type definitions may also change their encoding.

For example, suppose we're working with some JSON like this:

{ "id": 12345, "page": "/index.html", "ts": "2019-10-22T14:54:13Z" }

And we're decoding it into a Scala case class using Circe:

import io.circe.Codec
import io.circe.generic.semiauto.deriveCodec
import java.time.Instant

case class Visit(id: Long, page: String, ts: Instant)

object Visit {
  implicit val codecForVisit: Codec[Visit] = deriveCodec
}

And because we're responsible people, we're even checking the codec laws:

import cats.kernel.Eq
import io.circe.testing.{ArbitraryInstances, CodecTests}
import org.scalacheck.Arbitrary
import org.scalatest.flatspec.AnyFlatSpec
import org.typelevel.discipline.scalatest.Discipline
import java.time.Instant

trait VisitTestInstances extends ArbitraryInstances {
  implicit val eqVisit: Eq[Visit] = Eq.fromUniversalEquals
  implicit val arbitraryVisit: Arbitrary[Visit] = Arbitrary(
    for {
      id   <- Arbitrary.arbitrary[Long]
      page <- Arbitrary.arbitrary[String]
      ts   <- Arbitrary.arbitrary[Long].map(Instant.ofEpochMilli)
    } yield Visit(id, page, ts)
  )
}

class VisitSuite extends AnyFlatSpec with Discipline with VisitTestInstances {
  checkAll("Codec[Visit]", CodecTests[Visit].codec)
}

This will verify that our JSON codec round-trips values, has consistent error-accumulation and fail-fast modes, etc. Which is great! Except that if we make a small change to our case class…

import java.time.Instant

case class Visit(id: Long, page: String, date: Instant)

…then our tests will continue to pass, but we won't be able to decode any of the JSON we could previously decode, and any JSON we produce will be broken from the perspective of external systems that haven't made equivalent changes.

We can fix this by adding some tests for specific examples:

import java.time.Instant
import io.circe.testing.CodecTests
import org.scalatest.flatspec.AnyFlatSpec
import org.typelevel.discipline.scalatest.FlatSpecDiscipline

class VisitSuite extends AnyFlatSpec with FlatSpecDiscipline with VisitTestInstances {
  checkAll("Codec[Visit]", CodecTests[Visit].codec)

  val good = """{"id":12345,"page":"/index.html","ts":"2019-10-22T14:54:13Z"}"""
  val value = Visit(12345L, "/index.html", Instant.parse("2019-10-22T14:54:13Z"))

  "codecForVisit" should "decode JSON that's known to be good" in {
    assert(io.circe.parser.decode[Visit](good) === Right(value))
  }

  it should "produce the expected results" in {
    import io.circe.syntax._
    assert(value.asJson.noSpaces === good)
  }
}

The only problem is that it's really unpleasant to do this by hand! Also the "problem" we were originally trying to solve is only a problem if it happens accidentally. Often we're changing our data type definition specifically because some schema changed. In that case the fact that we only have to change our code in one place is actually one of the advantages of type class derivation: there are fewer things to worry about keeping in sync as our data types and schemas evolve. These example-based tests make this process a little safer, but at the cost of adding back a lot of the friction we were using derivation to avoid.

Golden testing

This library is an attempt to provide the benefits of example-based tests without all the annoying noise and maintenance. The usage looks like this:

import io.circe.testing.golden.GoldenCodecTests
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.prop.Configuration
import org.typelevel.discipline.scalatest.FlatSpecDiscipline

class VisitSuite extends AnyFlatSpec with FlatSpecDiscipline with VisitTestInstances with Configuration {
  checkAll("GoldenCodec[Visit]", GoldenCodecTests[Visit].goldenCodec)
}

This is almost identical to our first VisitSuite, but the first time we run the test, it will use the Arbitrary[Visit] instance (which we need for the round-trip testing, anyway) to generate an example Visit value, which it will serialize to a JSON string and write to a test resource file. The next time we run the test, it will find that file and will confirm that the current decoder can decode it, as well as that the current encoder will produce the same result.

This approach is called "golden testing", and this library is inspired specifically by hspec-golden-aeson.

Usage

Add the dependency to your sbt build:

libraryDependencies += "io.circe" %% "circe-golden" % "0.4.0" % Test

Change all of your CodecTests laws-checking tests to GoldenCodecTests with goldenCodec, then run your tests as usual. This will check all of the laws you were previously running, plus the new golden tests.

In general you'll want to check the generated golden test files into version control, since otherwise you won't get any of the benefits of golden testing in CI (or any other time you test a fresh check-out).

If you make a change to your codecs that intentionally breaks serialization compatibility, you have to delete the JSON files in your test resources. You can find these directories by running show test:resourceDirectory in sbt.

Warnings and known issues

While it's possible to use GoldenCodecLaws directly, it's inconvenient, and there's a lot of magic involved in the ResourceFileGoldenCodecLaws implementation. In particular the heuristics for determining where to write resource files is likely to be kind of fragile. It probably doesn't work on Windows or many moderately complex cross-builds, for example.

The off-the-shelf golden tests will currently fail if you change your Arbitrary instances in such a way that different seeds produce different values. This generally shouldn't be a problem, since it's generally likely to be a good idea to isolate changes to your Arbitrary instances from unrelated changes that may break serialization compatibility, anyway. You just have to rebuild your golden files after changing your Arbitrary instances (see the previous section for details).

The golden tests will also fail if you change the number of golden examples to generate. This may change in the future. In the meantime you have to rebuild your golden files after changing this configuration.

One extremely inconvenient thing about this library as it exists right now is that every time you rebuild your golden files, you'll get new ScalaCheck seeds, and therefore new file names, which means some unnecessary churn in your version control system, as well as less useful diffs. This is something I'm hoping to address soon.

Other future work

I'm also planning to add some tools for making it easier to rebuild your golden files, so that this can be done with a runMain from inside the sbt console instead of by manually tracking down and deleting the resources.

It would probably be possible to make this work for Scala.js projects with some macro magic. I don't personally care enough, but would be happy to review PRs.

It's possible this functionality will be moved into circe-testing someday, but I kind of doubt it.

Contributors and participation

All Circe projects support the Scala code of conduct and we want all of their channels (Gitter, GitHub, etc.) to be welcoming environments for everyone.

Please see the Circe contributors' guide for details on how to submit a pull request.

License

circe-golden is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

circe-golden's People

Contributors

ceedubs avatar domaspoliakas avatar fredfp avatar fthomas avatar hamnis avatar scala-steward avatar schrepfler avatar sullis avatar tobiasroland avatar travisbrown avatar zarthross avatar zmccoy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

circe-golden's Issues

Golden File Detection

First off, thanks very much for the latest release to support Scala 3.

I'm still using Scala 2, and upgrading to 0.5.0 seems to have changed the golden file detection in at least some cases so that a new file is generated in subsequent runs.

Here's an excerpt of the class in question:

package fossil.domain

object endpoint {

  final case class EndpointDescriptor(
      name: String,
      enabled: Boolean,
      topics: List[String],
      config: Map[String, String]
  )

}

I have other classes that are top level (i.e. not residing inside an object) that don't exhibit this issue.

`NullPointerException` when trying to read resources in bloop

Evaluating Resources.inferRootDir is throwing a NPE when it's run in bloop - I believe this is because (in sbt) the initial new File(getClass.getResource("/").toURI) call returns something like

(project root)/target/scala-2.12/test-classes,

but in Bloop it's

(project root)/src/test/resources - the actual source directory.

So in the latter case, the loop:

while (current.ne(null) && current.getName != "target") {
  current = current.getParentFile
}

basically runs until it finds the root directory (/) and its parent, null (because there's usually no target in the path).

It would be great to support the case of running these tests through bloop, as it's often used by users of Metals to avoid duplicate compilations with sbt.

Maybe instead of the fragile classpath-based logic, we could have something compile-time, getting the path of the current file in a sourcecode-like fashion and moving upwards until we hit test, then just adding /resources?

Use same seed every time you rebuild your golden files

Just making the plan in the README.md into a ticket (so I can watch it, and also ask the following)

Is there a workaround to configure the seed manually? I'm really rusty with scalacheck - as well as this library being sufficiently simple to obscure the "normal" seed-fixing techniques.

Unstable encoding of maps

This is a fantastic tool, it is however troublesome to use as soon as you are encoding maps because the order of keys in generated JSON objects is not stable (also see circe/circe#556).

A way to work around this would be to allow passing an io.circe.Printer to use in io.circe.testing.golden.GoldenCodecLaws.printJson when building the GoldenCodecTests instance. This would allow users to pass Printer.spaces2SortKeys which would guarantee stable encoding and printing of json when maps are involved.

When you encode something as `Option[Option[String]]` it can not properly decode

Expected: Right(VacancyProfileF(1683177f-5edb-4a9c-8f41-ca1b9e9ac631,Some(PartTime),Some(None),Some(O)))
Recieved: Right(VacancyProfileF(1683177f-5edb-4a9c-8f41-ca1b9e9ac631,Some(PartTime),None,Some(O)))
final case class VacancyProfileF[F[_]](
  id: VacancyProfileId,
  vacancyType: F[VacancyType],
  title: F[Option[NonEmptyString]],
  description: F[NonEmptyString]
)

Where F is set op Option

As you encode Some(None) and expect that.. but it's isomorphic to None which you get back

The use case is, when F is set to cats.Id it's an insert when it's set to Option it's an update where you choose to update specific fields by supplying a Some of something. When there is a Option inside this data structure, like title you'll get a Option[Option[NonEmptyString]]. Setting the title to nothing is Some(None)

Release a new version?

Hi Travis,

Is it possible for you to release a new version? The conflicts with the new version of disciple-scalatests are annoying and you already updated it in master.

Thanks.
Jules 🙂

Generic case classes and Resources.inferName don't work

There is no file written when I:

checkAll("GoldenCodec[PlanResult[AppointmentDelivery]]", GoldenCodecTests[PlanResult[AppointmentDelivery]].unserializableGoldenCodec)
final case class PlanResult[A](appointment: A, succeeded: Boolean)

Guess Resources.inferName doesn't work with that type

Backward-compatible changes fail the golden tests

When we evolve a data type in a backward-compatible way (e.g., by adding or removing an optional property), the golden tests fail because the Arbitrary generators produce different values.

Instead, it would be useful to have tests that support backward-compatible evolutions of data types.

We have implemented it internally at Bestmile, I can make a PR if you are interested. In essence, when we create a new version of the data type, we keep the old golden files and add new golden files. The law we test is the following:

      goldenExamples.flatMap {
        _.traverse { case (encodedString, maybeValue) =>
          maybeValue match {
            case Some(value) =>
              // The golden file contains the current version of the resource.
              // We check that:
              // - encoding the `value` produces the same `encodedString`
              // - decoding the `encodedString` produces the `value`
              parser.decode[A](encodedString)(decode).map { decoded =>
                val decoding = isEqToProp(decoded <-> value)
                val encoding = isEqToProp(printJson(encode(value)) <-> encodedString)
                decoding && encoding
              }.toTry
            case None =>
              // The golden file contains an old version of the resource.
              // We check that:
              // - we can still decode it
              // - encoding and then decoding it produces the same value as
              //   when decoding the old version
              (
                for {
                  decodedOldValue <- parser.decode[A](encodedString)(decode)
                  decodedNewValue <- decode.decodeJson(encode(decodedOldValue))
                } yield isEqToProp(decodedOldValue <-> decodedNewValue)
              ).toTry
          }
        }
      }

Doesn't seem to work with Enumeratum

Hi Travis,

Here's an example to reproduce the problem:

package com.example

import cats.Eq
import enumeratum.{CirceEnum, Enum, EnumEntry}
import enumeratum.EnumEntry.Uppercase
import io.circe.Codec
import io.circe.generic.semiauto._
import io.circe.testing.ArbitraryInstances
import io.circe.testing.golden.GoldenCodecTests
import org.scalacheck.{Arbitrary, Gen}
import org.scalacheck.Arbitrary.arbitrary
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.prop.Configuration
import org.typelevel.discipline.scalatest.FlatSpecDiscipline

object CurrencyGoldenTest {
  sealed abstract class Currency extends EnumEntry
  object Currency extends Enum[Currency] with Uppercase with CirceEnum[Currency] {
    override final val values = findValues

    final case object AUD extends Currency
  }

  final case class Price(value: Double, currency: Currency)
  object Price {
    implicit final val codec: Codec[Price] = deriveCodec
  }

  implicit val arbCurrency: Arbitrary[Currency] = Arbitrary(Gen.oneOf(Currency.values))
  implicit val arbPrice: Arbitrary[Price] = Arbitrary {
    for {
      v <- arbitrary[Double]
      c <- arbitrary[Currency]
    } yield Price(value = v, currency = c)
  }
}

class CurrencyGoldenTest extends AnyFlatSpec with FlatSpecDiscipline with Configuration with ArbitraryInstances {

  import CurrencyGoldenTest._

  implicit final val eqCurrency: Eq[Currency] = Eq.fromUniversalEquals
  implicit final val eqPrice: Eq[Price]       = Eq.fromUniversalEquals

  checkAll("GoldenCodec[Currency]", GoldenCodecTests[Currency].goldenCodec)
  checkAll("GoldenCodec[Price]", GoldenCodecTests[Price].goldenCodec)

}

When I run it, I have the following error:

NotSerializableException was thrown during property evaluation.
  Message: com.example.CurrencyGoldenTest$Currency$
  Occurred when passed generated values (

  )
ScalaTestFailureLocation: org.scalatestplus.scalacheck.CheckerAsserting$$anon$2 at (CurrencyGoldenTest.scala:45)
org.scalatest.exceptions.GeneratorDrivenPropertyCheckFailedException: NotSerializableException was thrown during property evaluation.
  Message: com.example.CurrencyGoldenTest$Currency$
  Occurred when passed generated values (

  )
	at org.scalatestplus.scalacheck.CheckerAsserting$$anon$2.indicateFailure(CheckerAsserting.scala:251)
	at org.scalatestplus.scalacheck.CheckerAsserting$$anon$2.indicateFailure(CheckerAsserting.scala:238)
	at org.scalatestplus.scalacheck.UnitCheckerAsserting$CheckerAssertingImpl.check(CheckerAsserting.scala:160)
 ...

I've added a breakpoint on the GeneratorDrivenPropertyCheckFailedException exception, and we can find the following message in the pos parameter of this type:

Please set the environment variable SCALACTIC_FILL_FILE_PATHNAMES to yes at compile time to enable this feature.

Any idea of how to mitigate that problem and/or where it can come from?

Thanks,
Jules

File encoding must be explicitly set when reading a file?

On my local machine all the tests runs fine, whenever I run it on my build server it complains with

java.nio.charset.MalformedInputException: Exception raised on property evaluation.> Exception: java.nio.charset.MalformedInputException: Input length = 1

I guess that's because the file contains chinese/arabic characters and must be read with a specific file encoding which is not the default of that operating system?

Scala 3 support

Has there been any discussion, or planning for supporting scala 3?

nulls in generated Json break backwards compatiblity checks

Opening this to see if anyone has found a better solution to this.

Consider for example:

case class Person(name: String, address: Option[String])

object Person {

  implicit val decoderForPerson: Decoder[Person] = deriveDecodder[Person

  implicit val encoderForPerson: Encoder[Person] = deriveEncoder[Person]

  implicit val eqForPerson: Eq[Person] = Eq.fromUniversalEquals[Person]
}

The address field is optional and if we are unlucky then scalacheck will generate Person("some name", None)
which circe golden will serialize to

{
  "name": "some name",
  "address": null
}

Unfortunately this means that the address subtree is effectively untested for backwards compatiblity. For example,
the following breaking change to the codec for address still passes the golden tests

case class Person(name: String, address: Option[String])


object Person {

  implicit val decoderForPerson: Decoder[Person] = new Decoder[Person] {
    def apply(c: HCursor): Decoder.Result[Person] =
      for {
        name <- c.downField("name").as[String]
        address <- c.downField("address").as[Option[Value]]
      } yield Person(name, address.map(_.value))
  }

  implicit val encoderForPerson: Encoder[Person] = new Encoder[Person] {

    def apply(a: Person): Json = Json.obj(
      "name" -> a.name.asJson,
      "address" -> a.address.fold(Json.Null)(addr => Json.obj("value" -> addr.asJson))
    )

  }

  implicit val eqForPerson: Eq[Person] = Eq.fromUniversalEquals[Person]
}

case class Value(value: String)

object Value{
  implicit val decoderForValue: Decoder[Value] = deriveDecoder[Value]
}

I've hacked around it using circe-droste:

  val containsNulls: Algebra[JsonF, Boolean] = Algebra {
    case JsonF.NullF       => true
    case JsonF.ArrayF(xs)  => xs.exists(identity)
    case JsonF.ObjectF(xs) => xs.map(_._2).exists(identity)
    case _                 => false
  }

generator.filter(person => !cata(containsNulls).apply(person))

but was wondering if anyone had a better solution? Or if things would go terribly wrong if circe-golden did this by default?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.