practicalli / clojure-web-services Goto Github PK

A pre-cursor to talking about Duct and other web frameworks that use it

Dependency: #12 Introducting component lifecycle management

Using integrant at REPL startup
https://clojureverse.org/t/where-to-put-main-and-how-to-call-it-with-integrant/6239

https://github.com/yogthos/migratus

https://www.graphqlize.org/blog/announcing-graphqlize-alpha/

Also supports MySQL (and MariaDB ?)

expose your data as resources while automatically complying with all the relevant requirements of the HTTP specification (RFC-2616). Your resources will automatically gain useful HTTP features, such as caching and content negotiation.

http://clojure-liberator.github.io/liberator/

middleware library that handles the preflight checking
https://github.com/r0man/ring-cors
https://clojurians.slack.com/archives/C053PTJE6/p1594585523158700?thread_ts=1594585523.158700&cid=C053PTJE6

Safely is a Clojure's circuit-breaker library for handling retries in an elegant declarative way.

The library offers out of the box:

declarative exception handling
declarative circuit breaker (in pure Clojure)
automatic retry policies
randomized delays retries
attenuation of self-emergent behaviour is distributed systems
sleepless-mode for testing
automatic and customizable logging of errors
automatic tracking of errors rate/count in monitoring tools
automatic tracking of common performance metrics

https://stackoverflow.com/questions/40094135/how-to-mock-test-post-requests-with-body-as-json-using-ring-mock-request

http://www.learndatalogtoday.org/ - useful for learning queries

https://youtu.be/oo-7mN9WXTw - domain modelling with datalog

Lambda Island (not all free)
https://lambdaisland.com/collections/datomic

https://www.tadam-framework.dev/

Excellent project website - is this Clojure / tadam ?

Review the G project for comments management, replacing discuss.

NOTE: uses Leiningen, can it be easily adapted for Clojure deps.edn and use end config instead of yml ?

lein-ring version 0.12.5 fixes isse with Leiningen 2.9.x issue

Using lein-ring 0.12.4 or earlier and running lein ring server generates a Compiler Exception

Syntax error macroexpanding clojure.core/fn at (clojure/core/unify.clj:83:18)
error when using lein-ring 0.12.4 or earlier.

https://lispcast.com/hiccup-tips/
https://purelyfunctional.tv/mini-guide/hiccup-tips/
https://github.com/yokolet/hiccup-samples

https://medium.com/makimo-tech-blog/hiccup-lightning-tutorial-6494e477f3a5

Libraries or tools that support log-line parsers for logs across the variety of languages/platforms?

• instaparse or clj-antlr are feature rich libraries
• regex for a more manual approach

Notes from the community

clj-antlr might be the most robust as you might find compatible grammar definitions around the internet
(that's how we kick started our Lucene-like query interface in our product)

Started with regex, too limiting, have moved on to instaparse. Thought about clj-antlr, but didn't have a lot of hope for finding open-sourced grammars. I guess I should actually look...

If care about speed ANTLR is much faster than instaparse

Definitely. Instaparse is nice for REPL-driven development, but once I get the grammars nailed down I'll probably move to ANTLR.

i noticed that https://github.com/antlr/grammars-v4 had a variety of grammars -- is it possible one of them might already be close to what you might need?

turning out to be pretty easy to define log parsers in instaparse.

https://github.com/bakku/clj-rest-web-app

A session stores the currently logged in user ID, configured the ring-session middleware to use a cookie store. After restarting the server the session is refreshed as if it is not using any cookie store.

• configuring the cookie-store: https://github.com/bakku/clj-rest-web-app/blob/master/src/clj_rest_web_app/core.clj#L24
• setting the session after a successful login: https://github.com/bakku/clj-rest-web-app/blob/master/src/clj_rest_web_app/controllers/sessions.clj#L6
• running the application in docker, here I am setting the cookie secret: https://github.com/bakku/clj-rest-web-app/blob/master/docker-compose.yml#L15

By default Ring uses an in-memory session store. you can change it to store data in a cookie or something persistent like Redis
https://github.com/ring-clojure/ring/wiki/Sessions#session-stores (edited)
Its configured it to store the data in a cookie
using the site-defaults from ring which already wrapped my routes using the session middleware but the site-defaults are using the in-memory session. I had to update the session configuration of the site-defaults according to my needs to get it to work

some simple examples of transit serializers / deserialiizers, https://github.com/noisesmith/poirot/blob/master/src/org/noisesmith/poirot/handlers.cljc and how they get used https://github.com/noisesmith/poirot/blob/master/src/org/noisesmith/poirot.cljc#L83 https://github.com/noisesmith/poirot/blob/master/src/org/noisesmith/poirot.cljc#L107

https://www.youtube.com/watch?v=mgSSVTDZvkI

Using Postgress in an idiomatic clojure way, doing selects, inserts etc would be great! I'm finding it hard to find good sources on this.

Thanks for the suggestion, it's a very useful topic so I'll move it up my (very long) Todo list.

I have some basic content in the Practicalli webapps book, but it's very simplistic and needs updating.

Many people use SQL fairly directly so
jdbc.next.jdbc seems a great project to use and
https://cljdoc.org/d/seancorfield/next.jdbc/1.1.547/doc/getting-started

https://www.hugsql.org/ also seems a nice tool.

I also like to use migrations with SQL too.

Visit the site and a random icebreaker question is displayed. A button is displayed to generate another icebreaker question if the first one is not wanted.

Include a way to avoid repeating icebreaker questions to the same person, e.g. cookie, login

Show history of questions so someone knows what questions they asked recently.

Overview of available tools and libraries for capturing, efficiently searching and visualising data from log events, especially mulog style events.

Approaches

• using kibana / elastic search for logging
• Loki and Grafana

https://github.com/bpringe/auth-template

A template for web apps with user auth using OWASP best practices and pedestal
A project containing the extracted authorization aspect of an application, especially relevant for projects using pedestal. Includes packaging and deployment config as well.

flow chart for proper REST resource for-GET/http-decision-diagram
https://github.com/for-GET/http-decision-diagram
An activity diagram to describe the resolution of HTTP response status codes, given various headers.
http://hyperrest.github.io/2013-06-10-http-hell-no
https://github.com/for-GET/http-decision-diagram|for-GET/http-decision-diagramfor-GET/http-decision-diagram | 17 Feb 2013 | Added by GitHub

Seems the session must be tied to a user - which makes me thinking how would you implement features like "require captcha after logging in failed for 3+ times" ?
The sessions can be anonymous - user_id is null in the db row in that case (to be specific).
See the anonymous-session interceptor here: https://github.com/bpringe/auth-template/blob/954b8a78f7e6d91b4dd12446ec69aba004c53d1e/src/auth_template/service.clj#L208
There may be a better way to handle anon sessions - but this is a simple starting point / works.

Overview of lifecycle management in Clojure,

Discuss why and when livecycle management is required

Briefly covering the different approaches taken by Mount, Component and Integrant.

Link to a section and examples of each

component

• ?

integrant

• duct
• ?

Check to see which luminus and pedestal templates use.

Write section on application lifecycle management tools

Application lifecycle management tools configure the many components that are part of an application, such as HTTP server, persistence store access, cache management, passing configuration to routing (db connections, security, etc)

When running an application, lifecycle tools ensure components are configured and started correctly, to provide a consistent startup and clean shutdown of all components.

Some tools also have specific REPL libraries (e.g. Integrant REPL, Component REPL) to support a REPL driven development workflow. These tools can start, restart and stop components in the system or the whole system itself, without requiring the Clojure REPL process to be restarted.

Commonly used projects

• https://github.com/tolitius/mount - simple, atom based, lightweight

• https://github.com/weavejester/integrant - data focus, more involved, separate Integrant REPL project, most common for new projects with multiple components, works well with JUXT aero

• https://github.com/stuartsierra/component - classic component approach, less used for new projects, but still many projects using this approach

Projects to evaluate

• https://github.com/juxt/clip - combination of code and configuration in one structure
• https://github.com/donut-power/system - combination of code and configuration in one structure
• https://github.com/tolitius/yurt - manage stateful components in isolation (related to Mount)

Semi-related projects

• https://github.com/kepler16/gx.cljc - a data-oriented library for defining and manipulating directed acyclic graph's of state machines

Resources

Related issues

• #13
• practicalli/clojure#48
• practicalli/clojure#47
• practicalli/clojure#46
• practicalli/clojure#45

https://github.com/http-kit/chat-websocket

Update: use Clojure CLI, reitit over Leiningen approach

Instead of lein ring server and having to set up an nrepl connection to the running process (or use wrap-reload), you can start a repl in the project and then start the ring server (jetty, etc) from inside the repl (or evaluating some commented out code in the source file)

Starting the ring server inside the clojure-jack-in repl will block further input at the REPL buffer.

Use the join? option to be able to interact with repl after starting a ring server

Set join? to false to continue using the repl after starting the server and allows request handlers to be changed without restarting the repl or server.

-dev-main function is often created to set different code for development. This is also suggested by lein-ring for including wrap-reload.

-repl-main could be a function that is configured to start the ring server using the join? option

join? option - set to false to prevent the server blocking https://ring-clojure.github.io/ring/ring.adapter.jetty.html

trouble testing routes in an app that uses compojure-api. One of the handlers is asynchronous and returns a channel. The server runs fine, but when running tests the async handler produces the following exception:

java.lang.IllegalArgumentException: No implementation of method: :render of protocol: #'compojure.response/Renderable found for class: clojure.core.async.impl.channels.ManyToManyChannel```
(edited)
Since i’m using lein-ring to run the server and build the uberjar and it works fine under those conditions, I assume I’ve missed some bit of configuration in the tests. If anyone could point me in the direction of some docs (or a solution!) it’d be much appreciated. I can’t find anything that looks relevant on the repos for compojure-api or ring

https://drewverlee.github.io/posts-output/2020-4-13-learn-datomic-part-1
https://drewverlee.github.io/posts-output/2020-4-18-learn-datomic-part-2
https://drewverlee.github.io/posts-output/2020-5-30-learn-datomic-part-3
https://drewverlee.github.io/posts-output/2020-6-8-learn-datomic-part-4
https://drewverlee.github.io/posts-output/2020-6-6-learn-datomic-cloud-part-1

Local Datomic instance
https://docs.datomic.com/cloud/dev-local.html

Example projects

• https://github.com/practicalli-john/datomic-data-transformation-idea

Hi I stumbled across this looking for a guide on doing Clojure development on the backend. This was a great tutorial until you start arriving at blank pages. Do you need assistance completing this guide? It seems like the community at large needs something like this to get started and I don't mind helping if I can.

https://medium.com/@_jba/setting-up-a-clojure-web-api-6bda099e76e8

write a simple Clojure console program that interacts with a database.
extend that to a Ring web app that produces HTML pages on the server.
Then connect a Reagent front end to a REST API on the back end over http/s and then Ajax and websocket calls

Clojure specific migration libraries
https://github.com/yogthos/migratus
https://github.com/weavejester/ragtime

General migrations tools
https://flywaydb.org/ - RedGate

Validating content sent to an API using middleware, returning a http status 422 if an error.

Example of simple validation middleware

https://clojurians.slack.com/archives/C053AK3F9/p1580340789444600?thread_ts=1580340789.444600&cid=C053AK3F9

https://slipset.github.io/posts/the-problem-i-dont-have

Build a Reitit / http-kit API service from scratch

• initially using an atom to hold the server reference

https://www.youtube.com/watch?v=LcpbBth7FaQ

https://medium.com/swlh/building-a-rest-api-in-clojure-3a1e1ae096e
https://www.codementor.io/@tamizhvendan/developing-restful-apis-in-clojure-using-compojure-api-and-toucan-part-1-oc6yzsigc

Rest design resources

• https://developer.atlassian.com/server/framework/atlassian-sdk/atlassian-rest-api-design-guidelines-version-1/

• https://swagger.io/specification/

• https://cloud.google.com/apis/design

• https://thenewstack.io/how-radical-api-design-changed-the-way-we-access-databases/

capture params that are multiple path parts
/path/{rest/of/the/path)

/path/{*path} that'll make it available under path

reitit route-syntax is defined here, for future reference: https://cljdoc.org/d/metosin/reitit/CURRENT/doc/basics/route-syntax

in compojure that would just be /path/:stuff and then you'd have that all in stuff (and you can use regex to specify a pattern

writing a function to call specifically from the REPL, and then calling it from -main with a different argument:

(defn start [& [join?]] (jetty/run-jetty-app {:port 9091 :join? join?}))
;; call from REPL as (start)
(defn -main [& args ]
(start true))

but what do you think of using future like that if ring didn’t had :join? ?

It will mean your -main will just exit when you run it from the command line -- which is not what you want.

It also means you won't be able to stop the server within your REPL either.

run-jetty should return a handle for the server, that you can call .stop on.

See this code https://github.com/seancorfield/usermanager-example/blob/develop/src/usermanager/main.clj#L140-L175 which has logic to start/stop either Jetty or http-kit (so it's a bit more complex that what you're doing). (edited)

It captures the result of calling (jetty/run-jetty app {:port ... :join? false}) so that it can .stop it later.

http-kit works differently -- it starts in the background anyway, and returns a handle to the server that you invoke with no args to stop it.
Another reason not to use future like that is that if the server throws an exception, you won't see it -- the process inside the future will just silently die.
You only see the failure if you deref the future -- when it will throw the exception that the process failed with.

Rewrite the Todo ap using a simple atom restart and reitit-ring for request routing

https://github.com/enricoteterra/clojure-practice-todo-app-api/blob/master/src/api/core.clj

https://github.com/cjohansen/compojure-api-tools-deps

A simple website with bootstrap and forms that lets you select two cities and it tells you how far they are apart (select Km, Miles, flaps of a birds wing, etc)

Resources
https://stackoverflow.com/questions/44042246/clojure-hiccup-dropdown-menu
https://weavejester.github.io/hiccup/hiccup.form.html
https://stackoverflow.com/questions/8140867/clojure-hiccup-form-handler
https://clojurebridge.org/community-docs/docs/web-applications/hiccup/
http://practicalli.github.io/clojure-webapps/compojure/show-request-info.html

ClojureScript
https://yogthos.github.io/reagent-forms-example.html

Dependencies:
#12 Introducing component lifecycle management
#13 Introduce Integrant

A section on Duct, including introduction of integrant, atraxy, reitit, etc

Short introduction (not official)
Duct (Integrant) is a framework for configuring your project. It handles setting up all daemons (http server, for example) and dependencies between components (web handlers need the database to connected, so handlers are dependent on database).Ring is an HTTP server abstraction, so that actually has nothing to do with Duct. Default Duct actually uses Ring, so you don’t need to compare them.Reitit is a routing library, again not comparable with Duct. Default Duct uses Ataraxy for routing, which can be compared to Reitit. Ataraxy vs Reitit is just preference, but Reitit is more mature / popular at the moment.You can compare integrant with Component or Mount. Those two are also meant to manage your applications state. The main difference between Integrant and the other two options I mentioned are that Integrant is Data driven. It tries to configure your application in Data (EDN), and keep the actual code minimal. The benefit of this is transparency, you can look at static data and understand what it is. Function calls on the other hand can do anything. Duct / Integrant might be a bit difficult to get your head wrapped around, but I believe once you understand the concepts you can build really clear / clean applications. Just finished my first “real” production application in Duct at work today 🙂

Video: James Reeves, the creator of Duct discusses the concepts involved
https://www.youtube.com/watch?v=zznwKCifC1A

Writing HTML

Hiccup
https://github.com/yogthos/Selmer
Templating

CSS

Including existing CSS frameworks (Bulma, Tailwind, Bootstrap, etc)
Writing your own CSS in Clojure ?
Compiling custom CSS - SaaS, ect.

SVG

https://clojureverse.org/t/what-is-the-best-way-to-use-custom-svg-icons-in-reagent/5957/2

https://medium.com/swlh/building-a-rest-api-in-clojure-3a1e1ae096e
https://github.com/functionalhuman/rest_demo/blob/master/src/rest_demo/core.clj

Using generative testing to test API's rather than unit tests that tend to include structural tests (is something a map, string, int uuid)

Edit: see if this adapts to clojure.spec and reitit implementations.

https://github.com/plumatic/schema
https://github.com/plumatic/schema-generators
http://leeorengel.com/clojure-property-based-testing-with-plumatics-schema-and-test-check-chuck/

Section on Containerization - Docker for Clojure apps

• images available for Clojure (CLI & Leiningen)
• image optimisation - cache and build images
• build application for the image
• security
  • avoid running as root (images may have non-root user already)
  • check for security vulnerabilities (snyk)

8 docker practices - video
https://youtu.be/8vXoMqWgbQQ

https://deangerber.com/blog/2018/10/07/host-simple-clojure-http-endpoint-using-docker/

Libraries
https://github.com/into-docker/clj-docker-client
An idiomatic, data-driven, REPL friendly Clojure Docker client inspired from Cognitect's AWS client.

https://github.com/cognitect-labs/aws-api
aws-api is a Clojure library which provides programmatic access to AWS services from your Clojure program.

Related Content

• https://practical.li/blog/posts/build-and-run-clojure-with-multistage-dockerfile/
• https://github.com/practicalli/clojure-app-template/blob/main/Dockerfile

Update: is this still relevant to next.jdbc

https://stackoverflow.com/questions/22586804/with-connection-what-happened

from [org.clojure/java.jdbc "0.3.3"] onwards

with-connection was not only considered harmful for reasons leonardoborges mentioned, it also makes working with connection pools harder. Decoupling function database access from specific connections makes for a much easier model. Forcing queries to use the same connection should be the exception, not the rule. So clojure.core/java.jdbc 0.3.0 was designed specifically to deprecate with-connection. To accomodate that, the whole API needed to be changed.

Each database access function now takes the db spec as a parameter. If the db-spec is a connection pool the function will be executed on one of it's connections, otherwise an implicit new connection is made from the db-spec. So all database access functions result in a connection each when connectionpools are not used.

This also means resultsets can no longer be returned lazily. Formerly, processing lazy sequences could be postponed while still inside the with-connection block. Now they need to be realized during function execution, or its connection could be closed or a new connection could be returned from the pool for the next access function.

So processing can now be done within the scope of the functions themselves, through two new named parameters: :row-fn and :result-set-fn. The first transforms each row, the second the collection of rows. If the :result-set-fn returns a lazy sequence, you will get a connection or resultset closed exception when using it later. The default :result-set-fn is doall. When using your own, make sure it is realized.

So much for the general case, where access and connections are decoupled. Now for the exception: Needing functions to use the same connection.

The most common of these is transaction use, which uses scope to indicate the beginning and the end of the transaction. The old transaction only provided this scope. The new with-db-transaction function takes a binding of a new var and the dp-spec.

This var will be bound to one specific connection from a pool, or when no connection pools are used, a newly created connection. All db access functions used inside the block should use the var instead of the db-spec parameter.

(def db {..})

(with-db-transaction
[c db]
(let [from 1111
to 2222
sum 10
saldo-from (query c ["select saldo from account where id=?" from]
:row-fn :saldo
:result-set-fn first)
saldo-to (query c ["select saldo from account where id=?" to]
:row-fn :saldo
:result-set-fn first)]
(update! c :account {:saldo (- saldo-from sum)} ["id=?" from])
(update! c :account {:saldo (+ saldo-to sum)} ["id=?" to])))
A begin transaction command will be issued at the beginning. All access will use the same connection that's now specifically passed to the functions instead of through dynamic scoping magic. When no exceptions are generated, a commit will be given at the end of the scope.

When one specific connection, but no transaction mechanic is needed, there's the with-db-connection function which has the same semantics. So if you want to execute a command to set a session setting, and do some queries on that connection, you can do the following:

(def db {..})

(with-db-connection [c db]
(execute! c ["alter session set NLS_SORT='ITALIAN'"])
(query c ["select * from person where name=?" "Mario"]
:row-fn (comp concat (juxt :name :surname))))
Connection pools usually have specific on-open and on-close commands that are part of their specs. Using those all connections from that pool will have the same session settings set, and with-db-connection is not even needed at all.

Related
https://purelyfunctional.tv/mini-guide/clojure-spec-vs-schema/

https://dev.to/dpsutton/exploring-the-core-cache-api-57al

https://findka.com/blog/migrating-to-biff/

https://medium.com/@rakyll/things-i-wished-more-developers-knew-about-databases-2d0178464f78

https://practical.li/clojure-web-services/projects/leiningen/todo-app/hiccup/updating-handlers-with-hiccup.html

Review options other than using :refer :all

• namespaces - html/ html-page/ etc - more verbose but very clear where functions are from, easier to refactor
• :refer [html5] - advantage of this is it can helps stick to a specific set of functions to help with consistency, e.g. always using html5 rather than html and html5 mixture

A website that tracks specific GitHub / GitLab projects and creates analytics for a project

• number of issues (raised, closed, marked stale) / PR's (raised, merged, closed) / commits / forks / stars / unstars / etc over time - daily, weekly, monthly, yearly

Constraints
Should have minimal impact on GitHub repositories analysed
No files or settings required for public repos
Export data

https://github.com/metosin/compojure-api

  (:require [monitor.handler :refer [handler]]
            [ring.adapter.jetty :refer [run-jetty]])
  (:gen-class))

 (defn -main [& args]
   (let [port (Integer/parseInt "3005")]
     (run-jetty handler {:port port :join? false})))```

where `handler` is a call to `compojure.core/defroutes`, is a fully functional microservice style server

Use environ to pick up port and use or to set a fixed default port

describe what the :join? directive does (when running in the repl join? false prevents the repl from being blocked by the server process)

https://8thlight.com/blog/colin-jones/2010/12/05/clojure-libs-and-namespaces-require-use-import-and-ns.html