As in start.spring.io's explore

https://twitter.com/Anh32920995/status/1559206242329001986

A starting one might be high memory usage of the JVM - maybe along with the JVM dashboard that we have, there's a standard set of alerts?

Plus we need to document on confluence how to add new alerts

Introduction in #4

Add to zipped file script ( wrapper similar to gradlew) which allows to build a project without natively installed sbt.

Notes

• proposed solution: https://bgasparotto.com/add-sbt-wrapper-to-project

Introduction in #4

Client can choose if he want to have json endpoints. ( no / circe / jsoniter / zio-json)

Notes

• !! zio-json can be chosen only for ZIO effects
• consider if it is better to provide separate new endpoint or modify existing one
• add support for json.
  • dependencies
  • clases
  • tests
• if docs endpoints are created update them

Looks like Mergify configuration stops merging validated PR's automatically.

The task is to make it work again.

Could be a good fit a small, simple project?

https://scala-cli.virtuslab.org

I think it would be cleaner to create the list of endpoints once. So instead of:

  val docEndpoints: List[ServerEndpoint[Any, IO]] =
    SwaggerInterpreter().fromEndpoints[IO](List(helloEndpoint, metricsEndpoint.endpoint, booksListing), "project3", "1.0.0")

  val all: List[ServerEndpoint[Any, IO]] = List(helloServerEndpoint, booksListingServerEndpoint, metricsEndpoint) ++ docEndpoints

I would do sth like:

  val apiEndpoints: List[ServerEndpoint[Any, IO]] = List(helloServerEndpoint, booksListingServerEndpoint)
  val docEndpoints: List[ServerEndpoint[Any, IO]] = SwaggerInterpreter().fromServerEndpoints[IO](endpoints, "project3", "1.0.0")
  val allEndpoints: List[ServerEndpoint[Any, IO]] = endpoints ++ docEndpoints ++ List(metricsEndpoint)

(do we want to document the metrics endpoints? I'm not sure :) )

There is a lack of test which will run endpoint from zipped project & check if it works.

This need to be added to fully cover project

As the complexity of the form grows, let's add unit tests for most crucial helper functions as well as UI tests to happy path flow.

• test configuration form helpers
• test configuration form validation schema
• test configuration form component useEffects
• test configuration form component submit flow
• test configuration form component reset flow

Introduction in #4

Client can choose if he want to have files generated in Scala 2 (latest) or Scala 3

Notes

• provide templating for both solutions, before doing that consider scalameta in order to create a common source tree that gets printed to scala 2 or 3 depending on user's preference

Add Share configuration button (as highlighted below)

that generates link with all configuration passed as URL parameters. Configuration from screenshot would result in the link below

https://adopt-tapir.softwaremill.com?addDocumentation=true&addMetrics=true&effect=IOEffect&groupId=org.example&implementationHttp4s&json=Jsoniter&projectName=share

Note that clicking on share would add the URL to the clipboard.

Probably not at the beginning and end, though.

See https://twitter.com/ClintCombs/status/1538368359217102848

In the event of #93 PR being under review it will further degrade to >1h so it is high time to take care it 😃

We want to count how many requests there were for downloading starter projects, and in what config (each config value - a label)

When request doesn't contain valid body. Response is properly prepared, but logs are not giving any real value.

How to reproduce

For request which have typo in implementation field: Adkka instead of allowed values: Akka

curl 'https://adopt-tapir.softwaremill.com/api/v1/starter.zip' \
  -H 'authority: adopt-tapir.softwaremill.com' \
  -H 'accept: */*' \
  -H 'content-type: application/json' \
  --data-raw '{"tapirVersion":"1.0.0","addDocumentation":false,"json":"No","projectName":"asdasd","groupId":"com.softwaremill","effect":"FutureEffect","implementation":"Adkka"}' \
  --compressed

It correctly handle errors but sends trash into logs:

2022-06-28 16:47:34.878 CEST io.circe.Errors: null Wrapped by: sttp.tapir.DecodeResult$Error$JsonDecodeException: null

Expected behavior.

• logs are self-explanatory AND/OR logs which comes from external libraries are suppressed.

1. the AtomicReference wrapper for books is unnecessary - it's only a read
2. booksListing.serverLogicSuccess(_ => IO.pure(books.get())) it's not that clear where the books come from - would be better to just have Library.books() without the import Library.*

Rationale

Now on ui module endpoint is hardcoded and pointing to production.

As ui module is static content copied to the backend module. There exists possibility to run both backend and frontend locally,
but it requires dynamic change of address.

Manual way

Port value is under backend/src/main/resources/application.conf

Manual modification:
ui/src/components/ConfigurationForm/ConfigurationForm.component.tsx:89

      // const response = await fetch('https://adopt-tapir.softwaremill.com/api/v1/starter.zip', {
      const response = await fetch('http://localhost:9090/api/v1/starter.zip', {

Run command

 sbt 'copyWebapp; ~backend/reStart'

Acceptance criteria

• Allow to change endpoint in UI. Keep the current default behaviour - (endpoint will point to production ).
• Implementation method (flags/envs or args in copyWebapp command) is up to developer based on best UX.

Description

Curently in CI for branches we are not generating docker artifacts. As we have 2 modules it occurs that sometimes building a docker image can cause problems. And currently it is discovered on the deploy phase.

Acceptance criteria

• add generating docker artifact as next step after test validation
• artifact is NOT published to official repo

Introduction in #4

Client can choose if he want to have documentation endpoints. (Yes/No)

• Add support for documentation endpoints.

It's enough to have a top-level @main def run(): Unit

The imports are the following:

import sttp.tapir.{PublicEndpoint, endpoint, query, stringBody}

import Library.*
import cats.effect.IO
import com.github.plokhotnyuk.jsoniter_scala.core.JsonValueCodec
import com.github.plokhotnyuk.jsoniter_scala.macros.JsonCodecMaker
import java.util.concurrent.atomic.AtomicReference
import sttp.tapir.*

The first one is not needed, as this is already covered by import sttp.tapir.*?

Background

In near future we will have Grafana instance. This task is about adding on dedicated adopt-tapir board :

• charts for JVM condition
• charts with metrics acquired from adopt-tapir

Write also documentation at Kiwi.

Acceptance criteria

• on graphana there is a dedicated board for adopt-tapir project
• configuration of board can be in project repository (talk with devops team if there is a better place)
• documentation created at Kiwi

As groupId is used for also package naming, there is need to forbid starting words from digit OR BETTERLY provide legalizing package names mechanism.

See details

Error
2022-06-27T06:03:00.338050746Zerror: /tmp/sbtProject8668560210853035356/src/main/scala/io/9dg/Endpoints.scala: org.scalafmt.dynamic.exceptions.PositionExceptionImpl: /tmp/sbtProject8668560210853035356/src/main/scala/io/9dg/Endpoints.scala:1: error: Invalid literal number
Error
2022-06-27T06:03:00.355153843Zerror: /tmp/sbtProject8668560210853035356/src/main/scala/io/9dg/Main.scala: org.scalafmt.dynamic.exceptions.PositionExceptionImpl: /tmp/sbtProject8668560210853035356/src/main/scala/io/9dg/Main.scala:1: error: Invalid literal number
Error
2022-06-27T06:03:38.679323542Zerror: /tmp/sbtProject9609624339204670922/src/test/scala/io/9dg/EndpointsSpec.scala: org.scalafmt.dynamic.exceptions.PositionExceptionImpl: /tmp/sbtProject9609624339204670922/src/test/scala/io/9dg/EndpointsSpec.scala:1: error: Invalid literal number
Error
2022-06-27T06:03:38.717718572Zerror: /tmp/sbtProject9609624339204670922/src/main/scala/io/9dg/Endpoints.scala: org.scalafmt.dynamic.exceptions.PositionExceptionImpl: /tmp/sbtProject9609624339204670922/src/main/scala/io/9dg/Endpoints.scala:1: error: Invalid literal number
Error
2022-06-27T06:03:38.765755859Zerror: /tmp/sbtProject9609624339204670922/src/main/scala/io/9dg/Main.scala: org.scalafmt.dynamic.exceptions.PositionExceptionImpl: /tmp/sbtProject9609624339204670922/src/main/scala/io/9dg/Main.scala:1: error: Invalid literal number

Description

Instead of generation zip file for every request. Introduce backend internal cache which will gnerate zip file or reuse previously created zip file sligthly modifying the packages and project name.

It should dramaticallly lower our response from ~3s.

The server implementation is now fixed

The following warning is being logged in CI:

The ubuntu-18.04 environment is deprecated, consider switching to ubuntu-20.04(ubuntu-latest), or ubuntu-22.04 instead. For more details see https://github.com/actions/virtual-environments/issues/6002

Rewrite backend to fully use Scala3.

Probably it will require get rid of DI library.

• add a simple index.html to webapp
• package as docker
• after each push to main:
  • push to our docker repo
  • deploy on our GCP
• map adopt-tapir.softwaremill.com to the deployed app

Problem

In our application we are using scalafmt as standalone library: https://scalameta.org/scalafmt/docs/installation.html#standalone-library

It dynamically downloads all necessary dependencies at runtime. It occurs that sometimes when downloading the required jobs fails, app cannot generate zip file, and throw 500 Internal Error.

2022-08-11T10:40:58.751037523Zerror: /tmp/sbtProject9026876679681506486/.scalafmt.conf: org.scalafmt.interfaces.ScalafmtException: [v3.5.8] failed to download Caused by: coursierapi.error.DownloadingArtifactsError: https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar: download error: Caught java.net.SocketException: Connection reset (Connection reset) while downloading https://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar at coursierapi.error.DownloadingArtifactsError.of(DownloadingArtifactsError.java:23) at coursierapi.shaded.coursier.internal.api.ApiHelper$.doFetch(ApiHelper.scala:346) at coursierapi.shaded.coursier.internal.api.ApiHelper.doFetch(ApiHelper.scala) at coursierapi.Fetch.fetchResult(Fetch.java:244) at coursierapi.Fetch.fetch(Fetch.java:239) at org.scalafmt.dynamic.CoursierDependencyDownloader.$anonfun$download$1(CoursierDependencyDownloader.scala:25) at scala.util.Try$.apply(Try.scala:210) at org.scalafmt.dynamic.CoursierDependencyDownloader.download(CoursierDependencyDownloader.scala:16) at org.scalafmt.dynamic.ScalafmtModuleLoader$WithDownloader.load(ScalafmtModuleLoader.scala:34) at org.scalafmt.dynamic.ScalafmtModuleLoader$CachedProxy.load$1(ScalafmtModuleLoader.scala:56) at org.scalafmt.dynamic.ScalafmtModuleLoader$CachedProxy.$anonfun$load$3(ScalafmtModuleLoader.scala:57) at scala.util.Try$.apply(Try.scala:210) at org.scalafmt.dynamic.utils.ReentrantCache.getOrAddToCache(ReentrantCache.scala:41) at org.scalafmt.dynamic.ScalafmtModuleLoader$CachedProxy.load(ScalafmtModuleLoader.scala:57) at org.scalafmt.dynamic.ScalafmtConfigLoader$.$anonfun$load$1(ScalafmtConfigLoader.scala:34) at scala.util.Either.flatMap(Either.scala:352) at org.scalafmt.dynamic.ScalafmtConfigLoader$.load(ScalafmtConfigLoader.scala:32) at org.scalafmt.dynamic.ScalafmtConfigLoader$CachedProxy.load$1(ScalafmtConfigLoader.scala:74) at org.scalafmt.dynamic.ScalafmtConfigLoader$CachedProxy.$anonfun$load$12(ScalafmtConfigLoader.scala:76) at scala.util.Try$.apply(Try.scala:210) at org.scalafmt.dynamic.utils.ReentrantCache.getOrAddToCache(ReentrantCache.scala:41) at org.scalafmt.dynamic.ScalafmtConfigLoader$CachedProxy.$anonfun$load$7(ScalafmtConfigLoader.scala:76) at scala.util.Success.fold(Try.scala:281) at org.scalafmt.dynamic.ScalafmtConfigLoader$CachedProxy.load(ScalafmtConfigLoader.scala:70) at org.scalafmt.dynamic.ScalafmtDynamic.resolveConfig(ScalafmtDynamic.scala:53) at org.scalafmt.dynamic.ScalafmtDynamic.createSession(ScalafmtDynamic.scala:47) at org.scalafmt.dynamic.ScalafmtDynamic.format(ScalafmtDynamic.scala:44) at com.softwaremill.adopttapir.starter.FormatScalaFiles$.$anonfun$apply$4(FormatScalaFiles.scala:22) at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563) at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561) at scala.collection.AbstractIterator.foreach(Iterator.scala:1293) at com.softwaremill.adopttapir.starter.FormatScalaFiles$.$anonfun$apply$1(FormatScalaFiles.scala:20) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) at cats.effect.unsafe.WorkerThread.blockOn(WorkerThread.scala:591) at scala.concurrent.package$.blocking(package.scala:124) at cats.effect.IOFiber.runLoop(IOFiber.scala:967) at cats.effect.IOFiber.execR(IOFiber.scala:1332) at cats.effect.IOFiber.run(IOFiber.scala:139) at cats.effect.unsafe.WorkerThread.run(WorkerThread.scala:461)

Walkaround

Restart of application helps temporarily.

Proposed solution.

As this loader uses coursier for downloading all necessary jars the main idea is to provide them ahead of time.
For that purpose we could attach volume with already fullfilled coursier cache which will be persisted and independent from deployment of an app.

Proposed tasks:

• create in docker mountable cache for coursier with write access
  • add persistent volume in k8s with read/write access with already attached required files.
    `kubectl cp :/home/demiourgos728/.cache <local_destination>
• setup readines probe to check zip endpoint

From time to time one of the sbt tests fails, combination of (Scala3, Circle) - https://github.com/softwaremill/adopt-tapir/runs/7742550077?check_suite_focus=true, usually rerunning the job helps the situation, but randomly failing tests are non-deterministic and hard to work with. Maybe there is some kind of race condition between jobs 🤔.

similar cases of occurence:

• https://github.com/softwaremill/adopt-tapir/actions/runs/2803866353

The akka & http4s interpreters come with a dependency on slf4j and the server have the logging enabled by default. For zio http and netty, it has to be separately added.

Additionally, ZIO 2 has its own logging infrastructure, which needs to be integrated with slf4j on startup

Background

App for generation template of the sbt project with Tapir.

MVP

From client perspective:

Client can select how the sbt zipped file is prepared:

• select tapirVersion
• provide projectName
• provide groupId
• select scala version (2- latest or 3)
• select effect type (Future / IO / ZIO)
• select specific implementation for effect ( Future -> Akka / Netty , IO -> Http4s / Netty, ZIO -> Http4s / zio-http)
• select documentation endpoints (yes/no)
• select metrics endpoints (yes/no)
• select json input/output (no / circe / jsoniter / zio-json) (zio-json can be picked only for ZIO effect)

Additionally zipped file should contain:

• wrapper similar to gradlew which allows run project without native installed sbt

Tasks

MVP:

• projectName #3
• groupId #3
• effects -> server implementation #3
• tapirVersion in query #3
• #5
• #17
• #10
• #12
• #14
• #7
• #15

Others:

• #9
• #6
• #8
• #13

Background

Currently our app doesn't have any alerting when something will go wrong.
In near future there will be Graphana created. Consider using it (if it is not created prepare all required things ahead. )

Acceptance criteria

• all critical and major issues are added on separate slack channel
• fulfill documentation at KiWi

1. Add a module to build.sbt with these dependencies
2. Setup scala-steward for the repository
3. Add the buildinfo plugin which propagates the build-time dependency versions to the source code
4. Use the versions propagated from the build using buildinfo

Maybe we can do the same for tapir version as well?

Overview

List 5-6 most recent successful configuration generations under the configuration form. Data that is being send as a response to /starter.zip request is in binary format, on the frontend side we do wrap it using the Blob API, this allows us to keep raw data in storages like localStorage or sessionStorage.

Visual representation of such solution might be a simple list with the file name + human readable timestamp of generation. Hovering over each entry would list selected options used to generate configuration. Upon user clicking the entry row or some icon user would be able download .zip file without need of additional request.

Motivation

As a developer I want to have a quick access to last configurations in order not to fill the form again if the setting that I use the most doesn't change.

Pitfalls

As the project emerges some options might change and potentially make the cached version of some configuration variation not usable.

It happens that the EOF error appears from time to time during working of the application.

Your task is to recreate and fix this error.

Details

2022-06-28 04:38:53.482 CEST
02:38:53.478�[1;33m �[0;39m[io-compute-1] ERROR o.h.b.s.Http1ServerStage$$anon$1 - Error writing body
org.http4s.blaze.pipeline.Command$EOF$: EOF
Wrapped by: fs2.CompositeFailure: Multiple exceptions were thrown (2), first org.http4s.blaze.pipeline.Command$EOF$: EOF
 at fs2.CompositeFailure$.apply(CompositeFailure.scala:58)
 at fs2.CompositeFailure$.apply(CompositeFailure.scala:45)
 at fs2.Pull$.addError$1(Pull.scala:1143)
 at fs2.Pull$.viewCont$1(Pull.scala:1150)
 at fs2.Pull$.$anonfun$compile$22(Pull.scala:1187)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at flatMap @ fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:932)
 at delay @ org.http4s.blazecore.util.EntityBodyWriter.$anonfun$writePipe$4(EntityBodyWriter.scala:83)
 at flatMap @ org.http4s.blazecore.util.package$.fromFutureNoShift(package.scala:40)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at handleErrorWith @ fs2.Compiler$Target.handleErrorWith(Compiler.scala:160)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
 at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1060)

Just before errors there are only request which need only static content of the page:

02:38:52.246�[1;33m [IXP-WWJ-GGA] �[0;39m[io-compute-2] DEBUG c.s.a.h.HttpApi - Request: GET /embedded-form, handled by: GET /*, took: 1ms; response: 200
02:38:52.532�[1;33m [SNI-VDL-ZIO] �[0;39m[io-compute-2] DEBUG c.s.a.h.HttpApi - Request: GET /static/js/main.dc05fc37.js, handled by: GET /*, took: 1ms; response: 200
2022-06-28 04:38:52.590 CEST

controller
10.0.64.38 - - [28/Jun/2022:02:38:52 +0000] "GET /static/js/main.dc05fc37.js HTTP/2.0" 200 201812 "https://adopt-tapir.softwaremill.com/embedded-form" "Mozilla/5.0 (Linux; Android 12; SM-G973U1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Mobile Safari/537.36" 125 0.060 [default-adopt-tapir-80] [] 10.0.33.105:8080 201812 0.060 200 8e6bc5ef0f2a4aa74724b4024ade446e
2022-06-28 04:38:52.619 CEST

Currently all generated files are not formatted properly.

The idea is to:

• run formatter just before zipping the files.
• add scalafmt to zipped directory

1. Simplify dependencies
2. Get rid from unnecessary dependencies for Akka implementation.

When downloading in safari, I get the content unpacked to an Unknown directory

When downloading in chrome, I get a .zip file with a random name (3410a2eb-2c70...), which unpacks to a directory with the same name.

Would be nice if the name of the zipfile was sth like $projectname-tapir-starter.zip and the directory maybe simply $projectname?

Although it doesn't make much sense to download a tapir bundle on mobile, people can still want to just see what this site is, so we need a decent mobile view.

Both in frontend & backend

Either @marcin-jozefowicz or @lubarskyy :)

During IT tests execution there is problem with cleaning resources.

java.nio.file.DirectoryNotEmptyException: /tmp/sbtTesting2003744321465379716/target
[info]   at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247)
[info]   at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
[info]   at java.base/java.nio.file.Files.delete(Files.java:1142)
[info]   at better.files.File.delete(File.scala:995)
[info]   at better.files.File.$anonfun$delete$1(File.scala:994)
[info]   at scala.collection.immutable.List.foreach(List.scala:333)
[info]   at better.files.File.delete(File.scala:994)
[info]   at com.softwaremill.adopttapir.starter.ServiceUnderTest.$anonfun$createTempDirectory$3(StarterServiceITTest.scala:131)
[info]   at blocking @ com.softwaremill.adopttapir.starter.StarterService.$anonfun$generateZipFile$2(StarterService.scala:24)
[info]   at blocking @ com.softwaremill.adopttapir.starter.StarterService.$anonfun$generateZipFile$2(StarterService.scala:24)

Find what causes it and fix cleaning up.

Introduction in #4

Client can choose if he want to have metrics endpoints. (Yes/No)

• Add support for documentation endpoints.

I think the form might use some grouping and tidying up as it became a bit chaotic.
Maybe something along these lines:

Group 1, build settings: Scala 2 / 3 toggle, sbt / Scala CLI toggle (by default: 3 & sbt)
Group 2, project name & group ID. Maybe we should have defaults here as well (such as: com.softwaremill / my-tapir - or sth auto-generated like curious-badger ;) ) so that generating a project can be even quicker if somebody just wants to explore?
Group 3: effect & server, if we would go with the something-chosen-by-default route, IO + http4s is a good choice
Group 4: options, swagger / json / metrics