Baum

Baum is an extensible EDSL in EDN for building rich configuration files.

It is built on top of clojure.tools.reader and offers the following features.

Basic mechanism for building simple and extensible DSL in EDN
- Reader macros
- Post-parsing reductions
  - A transformation of a map which has a special key to trigger it
Built-in DSL for writing modular and portable configuration files
- Access to environment variables, Java system properties, project.clj, and so on (powered by Environ)
- Importing external files
- Local variables
- Extensible global variables
- Conditional evaluation
  - if, some, match, …
  - This allows you to write environment specific configurations.
- etc…
Selectable reader
- A complete Clojure reader (clojure.tools.reader/read-string)
- An EDN-only reader (clojure.tools.reader.edn/read-string)

Setup

Add the following dependency in your project.clj file:

Reading your config file

To read your config files, use read-file:

(ns your-ns
  (:require [baum.core :as b]))

(def config (b/read-file "path/to/config.edn")) ; a map

(:foo config)
(get-in config [:foo :bar])

It also supports other arguments:

(ns your-ns
  (:require [baum.core :as b]
            [clojure.java.io :as io]))

(def config (b/read-file (io/resource "config.edn")))

(def config2
  (b/read-file (java.io.StringReader. "{:a :b}"))) ; Same as (b/read-string "{:a :b}")

See clojure.java.io/reader for a complete list of supported arguments.

Baum uses clojure.tools.reader/read-string as a reader by default. If you want to use the EDN-only reader, pass an option as follows:

(ns your-ns
  (:require [baum.core :as b]))

(def config (b/read-file "path/to/config.edn"
                         {:edn? true}))

Even if you use the EDN-only reader, some features of Baum may compromise its safety. So you should only load trusted resources.

In addition, to disable #baum/eval, set clojure.tools.reader/*read-eval* to false:

(ns your-ns
  (:require [baum.core :as b]
            [clojure.tools.reader :as r]))

(def config (binding [r/*read-eval* false]
              (b/read-file "path/to/config.edn"
                           {:edn? true})))

Examples

Database settings in one file

{:db {
      ;; Default settings
      :adapter       "mysql"
      :database-name "baum"
      :server-name   "localhost"
      :port-number   3306
      :username      "root"
      :password      nil

      ;; Override settings per ENV
      :baum/override
      #baum/match [#baum/env :env
                   "prod" {:database-name "baum-prod"

                           ;; When DATABASE_HOST is defined, use it,
                           ;; otherwise use "localhost"
                           :server-name #baum/env [:database-host "localhost"]

                           ;; Same as above. DATABASE_USERNAME or "root"
                           :username #baum/env [:database-username "root"]

                           ;; DATABASE_PASSWORD or nil
                           :password #baum/env :database-password}
                   "dev"  {:database-name "baum-dev"}
                   "test" {:adapter "h2"}]}}

Database settings in multiple files (w/ shorthand notation)

For details about the shorthand notation, see Built-in shorthand notation.

your_ns.clj:

(ns your-ns
  (:require [baum.core :as b]
            [clojure.java.io :as io]))

(def config (b/read-file (io/resource "config.edn")))

config.edn:

{$let [env      #env [:env "prod"]      ; "prod" is fallback value
       env-file #str ["config-" #- env ".edn"]]

 ;; If ENV is "prod", `config-default.edn` and `config-prod.edn` will
 ;; be loaded. These files will be merged deeply (left to right).
 $include ["config-default.edn"
           #- env-file]

 ;; If `config-local.edn` exists, load it. You can put private config
 ;; here.
 $override* "config-local.edn"}

config-default.edn:

{:db {:adapter       "mysql"
      :database-name "baum"
      :server-name   "localhost"
      :port-number   3306
      :username      "root"
      :password      nil}}

config-prod.edn:

{:db {:database-name "baum-prod"
      :server-name   #env [:database-host "localhost"]
      :username      #env [:database-username "root"]
      :password      #env :database-password}}

config-dev.edn:

{:db {:database-name "baum-dev"}}

config-local.edn:

{:db {:username "foo"
      :password "mypassword"}}

Aliasing

If the built-in reader macros or special keys are verbose, you can define aliases for them:

(read-file "path/to/config.edn"
           {:aliases {'baum/env 'env
                      :baum/let '$let
                      'baum/ref '-}})

Then you can rewrite your configuration as follows:

Before:

{:baum/let [user #baum/env :user
            loc  "home"]
 :who   #baum/ref user
 :where #baum/ref loc}

After:

{$let [user #env :user
       loc  "home"]
 :who   #- user
 :where #- loc}

Built-in shorthand notation

You can use built-in opinionated aliases if it is not necessary to worry about the conflict for you. The shorthand notation is enabled by default, but you can disable it if necessary:

(b/read-file "path/to/config.edn"
             {:shorthand? false})

And its content is as follows:

{'baum/env       'env
 'baum/str       'str
 'baum/regex     'regex
 'baum/if        'if
 'baum/match     'match
 'baum/resource  'resource
 'baum/file      'file
 'baum/files     'files
 'baum/read      'read
 'baum/read-env  'read-env
 'baum/import    'import
 'baum/import*   'import*
 'baum/some      'some
 'baum/resolve   'resolve
 'baum/eval      'eval
 'baum/ref       '-
 'baum/inspect   'inspect
 :baum/let       '$let
 :baum/include   '$include
 :baum/include*  '$include*
 :baum/override  '$override
 :baum/override* '$override*}

Of course, it is possible to overwrite some of them:

(b/read-file "path/to/config.edn"
             {:aliases {'baum/ref '|}})

Context-aware path resolver

You can refer to external files from your config file by using #baum/import, :baum/include or :baum/override.

Baum resolves specified paths depending on the path of the file being parsed. Paths are resolved as follows:

parent	path	result
foo/bar.edn	baz.edn	PROJECT_ROOT/baz.edn
foo/bar.edn	./baz.edn	PROJECT_ROOT/foo/baz.edn
foo/bar.edn	/tmp/baz.edn	/tmp/baz.edn
jar:file:/foo/bar.jar!/foo/bar.edn	baz.edn	jar:file:/foo/bar.jar!/baz.edn
jar:file:/foo/bar.jar!/foo/bar.edn	./baz.edn	jar:file:/foo/bar.jar!/foo/baz.edn
jar:file:/foo/bar.jar!/foo/bar.edn	/baz.edn	/baz.edn
http://example.com/foo/bar.edn	baz.edn	http://example.com/baz.edn
http://example.com/foo/bar.edn	./baz.edn	http://example.com/foo/baz.edn
http://example.com/foo/bar.edn	/baz.edn	/baz.edn
nil	foo.edn	PROJECT_ROOT/foo.edn
nil	./foo.edn	PROJECT_ROOT/foo.edn
nil	/foo.edn	/foo.edn

If you need to access local files from files in a jar or a remote server, use #baum/file:

{:baum/include #baum/file "foo.edn"}

Merging strategies

There are cases where multiple data are merged, such as reading an external file or overwriting a part of the setting depending on the environment. Baum does not simply call Clojure’s merge, but deeply merges according to its own strategy.

Default merging strategy

The default merging strategy is as follows:

Recursively merge them if both left and right are maps
Otherwise, take right

Controlling merging strategies

A mechanism to control merge strategy by metadata has been added since version 0.4.0. This is inspired by Leiningen, but the fine behavior is different.

Controlling priorities

To control priorities, use :replace, :displace:

{:a {:b :c}
 :baum/override {:a {:d :e}}}
;; => {:a {:b :c, :d :e}}

{:a {:b :c}
 :baum/override {:a ^:replace {:d :e}}}
;; => {:a {:d :e}}

{:a ^:displace {:b :c}
 :baum/override {:a {:d :e}}}
;; => {:a {:d :e}}

If you add :replace as metadata to right, right will always be adopted without merging them.

If you add :displace to left, if left does not exist, left is adopted as it is, but right will always be adopted as it is if right exists.

Combining collections

Unlike Leiningen, Baum only merges maps by default. In the merging of other collections like vectors or sets, right is always adopted. If you want to combine collections, use :append or :prepend:

{:a [1 2 3]
 :baum/override {:a [4 5 6]}}
;; => {:a [4 5 6]}

{:a [1 2 3]
 :baum/override {:a ^:append [4 5 6]}}
;; => {:a [1 2 3 4 5 6]}

{:a [1 2 3]
 :baum/override {:a ^:prepend [4 5 6]}}
;; => {:a [4 5 6 1 2 3]}

{:a #{1 2 3}
 :baum/override {:a ^:append #{4 5 6}}}
;; => {:a #{1 2 3 4 5 6}}

Built-in Reader Macros

#baum/env

Read environment variables:

{:foo #baum/env :user}                  ; => {:foo "rkworks"}

Environ is used internally. So you can also read Java properties, a .lein-env file, or your project.clj (you need lein-env plugin). For more details, see Environ’s README.

You can also set fallback values:

#baum/env [:non-existent-env "not-found"]       ; => "not-found"
#baum/env [:non-existent-env :user "not-found"] ; => "rkworks"
#baum/env ["foo"]                               ; => "foo"
#baum/env []                                    ; => nil

#baum/read-env

Read environment variables and parse it as Baum-formatted data:

#baum/env      :port                    ; "8080"
#baum/read-env :port                    ; 8080

You can also set a fallback value like a #baum/env:

#baum/read-env [:non-existent-env 8080]       ; => 8080
#baum/read-env [:non-existent-env :port 8080] ; => 3000
#baum/read-env ["foo"]                        ; => "foo"
#baum/read-env []                             ; => nil

NB! The Baum reader does NOT parse fallback values. It parses only values from environment variables.

#baum/read

Parse given strings as Baum-formatted data:

#baum/read "100"                        ; => 100
#baum/read "foo"                        ; => 'foo
#baum/read "\"foo\""                    ; => "foo"
#baum/read "{:foo #baum/env :user}"     ; => {:foo "rkworks"}

#baum/if

You can use a conditional sentence:

{:port #baum/if [#baum/env :dev
                 3000                   ; => for dev
                 8080                   ; => for prod
                 ]}

A then clause is optional:

{:port #baum/if [nil
                 3000]}                 ; => {:port nil}

#baum/match

You can use pattern matching with baum/match thanks to core.match.

{:database
 #baum/match [#baum/env :env
              "prod" {:host     "xxxx"
                      :user     "root"
                      :password "aaa"}
              "dev"  {:host     "localhost"
                      :user     "root"
                      :password "bbb"}
              :else  {:host     "localhost"
                      :user     "root"
                      :password nil}]}

baum/case accepts a vector and passes it to clojure.core.match/match. In the above example, if #baum/env :env is “prod”, the result is:

{:database {:host     "xxxx"
            :user     "root"
            :password "aaa"}}

If the value is neither “prod” nor “dev”, the result is:

{:database {:host     "localhost"
            :user     "root"
            :password nil}}

You can use more complex patterns:

#baum/match [[#baum/env :env
              #baum/env :user]
             ["prod" _]        :prod-someone
             ["dev" "rkworks"] :dev-rkworks
             ["dev" _]         :dev-someone
             :else             :unknown]

For more details, see the documentation at core.match.

#baum/file

To embed File objects in your configuration files, you can use baum/file:

{:file #baum/file "project.clj"}      ; => {:file #<File project.clj>}

#baum/resource

Your can also refer to resource files via baum/resource:

{:resource #baum/resource "config.edn"}
;; => {:resource #<URL file:/path/to/project/resources/config.edn>}

#baum/files

You can obtain a list of all the files in a directory by using baum/files:

#baum/files "src"
;; => [#<File src/baum/core.clj> #<File src/baum/util.clj>]

You can also filter the list if required:

#baum/files ["." "\\.clj$"]
;; => [#<File ./project.clj>
;;     #<File ./src/baum/core.clj>
;;     #<File ./src/baum/util.clj>
;;     #<File ./test/baum/core_test.clj>]

#baum/regex

To get an instance of java.util.regex.Pattern, use #baum/regex:

#baum/regex "^foo.*\\.clj$"       ; => #"^foo.*\.clj$"

It is useful only when you use the EDN reader because EDN does not support regex literals.

#baum/import

You can use baum/import to import config from other files.

child.edn:

{:child-key :child-val}

parent.edn:

{:parent-key #baum/import "path/to/child.edn"}
;; => {:parent-key {:child-key :child-val}}

If you want to import a resource file, use baum/resource together:

{:a #baum/import #baum/resource "config.edn"}

The following example shows how to import all the files in a specified directory:

#baum/import #baum/files ["config" "\\.edn$"]

NB: The reader throws an exception if you try to import a non-existent file.

#baum/import*

Same as baum/import, but returns nil when FileNotFound error occurs:

{:a #baum/import* "non-existent-config.edn"} ; => {:a nil}

#baum/some

baum/some returns the first logical true value of a given vector:

#baum/some [nil nil 1 nil]              ; => 1

#baum/some [#baum/env :non-existent-env
            #baum/env :user]            ; => "rkworks"

In the following example, if ~/.private-conf.clj exists, the result is its content, otherwise :not-found

#baum/some [#baum/import* "~/.private-conf.clj"
            :not-found]

#baum/str

Concatenating strings:

#baum/str [#baum/env :user ".edn"]      ; => "rkworks.edn"

#baum/resolve

baum/resolve resolves a given symbol and returns a var:

{:handler #baum/resolve my-ns.routes/main-route} ; => {:handler #'my-ns.routes/main-route}

#baum/eval

To embed Clojure code in your configuration files, use baum/eval:

{:timeout #baum/eval (* 1000 60 60 24 7)} ; => {:timeout 604800000}

When clojure.tools.reader/*read-eval* is false, #baum/eval is disabled.

NB: While you can use #= to eval clojure expressions as far as clojure.tools.reader/*read-eval* is true, you should still use Baum’s implementation, that is #baum/eval, because the official implementation doesn’t take account into using it with other Baum’s reducers/readers. For example, the following code that uses baum/let doesn’t work:

;; NG
{$let [v "foo"]
 :foo #=(str "*" #- v "*")} ; => error!

You can avoid the error using Baum’s implementation instead:

;; OK
{$let [v "foo"]
 :foo #baum/eval (str "*" #- v "*")} ; => {:foo "*foo*"}

#baum/ref

You can refer to bound variables with baum/ref. For more details, see the explanation found at :baum/let.

You can also refer to global variables:

{:hostname #baum/ref HOSTNAME}          ; => {:hostname "foobar.local"}

Built-in global variables are defined as follows:

Symbol	Summary
HOSTNAME	host name
HOSTADDRESS	host address

It is easy to add a new variable. Just implement a new method of multimethod refer-global-variable:

(defmethod c/refer-global-variable 'HOME [_]
  (System/getProperty "user.home"))

#baum/inspect

#baum/inspect is useful for debugging:

;;; config.edn

{:foo #baum/inspect {:baum/include [{:a :b} {:c :d}]
                     :a :foo
                     :b :bar}
 :bar :baz}


;;; your_ns.clj

(b/read-file "config.edn")
;; This returns {:bar :baz, :foo {:a :foo, :b :bar, :c :d}}
;; and prints:
;;
;;  {:baum/include [{:a :b} {:c :d}], :a :foo, :b :bar}
;;
;;  ↓ ↓ ↓
;;
;;  {:b :bar, :c :d, :a :foo}
;;

Built-in Reducers

:baum/include

:baum/include key deeply merges its child with its owner map.

For example:

{:baum/include {:a :child}
 :a :parent}                        ; => {:a :parent}

In the above example, a reducer merges {:a :parent} into {:a :child}.

:baum/include also accepts a vector:

{:baum/include [{:a :child1} {:a :child2}]
 :b :parent}                            ; => {:a :child2 :b :parent}

In this case, the merging strategy is like the following:

(deep-merge {:a :child1} {:a :child2} {:b :parent})

Finally, it accepts all other importable values.

For example:

;; child.edn
{:a :child
 :b :child}

;; config.edn
{:baum/include "path/to/child.edn"
 :b :parent}                            ; => {:a :child :b :parent}

Of course, it is possible to pass a vector of importable values:

{:baum/include ["child.edn"
                #baum/resource "resource.edn"]
 :b :parent}

:baum/include*

Same as :baum/include, but ignores FileNotFound errors:

;; child.edn
{:foo :bar}

;; config.edn
{:baum/include* ["non-existent-file.edn" "child.edn"]
 :parent :qux}                          ; => {:foo :bar :parent :qux}

It is equivalent to the following operation:

(deep-merge nil {:foo :bar} {:parent :qux})

:baum/override

The only difference between :baum/override and :baum/include is the merging strategy. In contrast to :baum/include, :baum/override merges child values into a parent map.

In the next example, a reducer merges {:a :child} into {:a :parent}.

{:baum/override {:a :child}
 :a :parent}                            ; => {:a :child}

:baum/override*

Same as :baum/override, but ignores FileNotFound errors. See also :baum/include*.

:baum/let

You can use :baum/let and baum/ref to make a part of your config reusable:

{:baum/let [a 100]
 :a #baum/ref a
 :b {:c #baum/ref a}}            ; => {:a 100 :b {:c 100}}

Destructuring is available:

{:baum/let [{:keys [a b]}  {:a 100 :b 200}]
          :a #baum/ref a
          :b #baum/ref b}
;; => {:a 100 :b 200}

{:baum/let [[a b] [100 200]]
 :a #baum/ref a
 :b #baum/ref b}
;; => {:a 100 :b 200}

Of course, you can use other reader macros together:

;;; a.edn
{:foo :bar :baz :qux}

;;; config.edn
{:baum/let [{:keys [foo baz]} #baum/import "a.edn"]
 :a #baum/ref foo
 :b #baum/ref baz}
;; => {:a :bar :b :qux}

baum/let’s scope is determined by hierarchical structure of config maps:

{:baum/let [a :a
            b :b]
 :d1 {:baum/let [a :d1-a
                 c :d1-c]
      :a #baum/ref a
      :b #baum/ref b
      :c #baum/ref c}
 :a #baum/ref a
 :b #baum/ref b}
;; => {:d1 {:a :d1-a
;;          :b :b
;;          :c :d1-c}
;;     :a  :a
;;     :b  :b}

You will get an error if you try to access an unavailable variable:

{:a #baum/ref a
 :b {:baum/let [a 100]}}
;; => Error: "Unable to resolve symbol: a in this context"

Writing your own reader macros

It is very easy to write reader macros. To write your own, use defreader.

config.edn:

{:foo #greet "World"}

your_ns.clj:

(ns your-ns
  (:require [baum.core :as b]))

(b/defreader greeting-reader [v opts]
  (str "Hello, " v "!"))

;; Put your reader macro in reader options:
(b/read-file "config.edn"
             {:readers {'greet greeting-reader}}) ; => {:foo "Hello, World!"}

;; Another way to enable your macro:
(binding [*data-readers* (merge *data-readers*
                                {'greet greeting-reader})]
  (b/read-file "config.edn"))

For more complex examples, see implementations of built-in readers.

Differences from Clojure’s reader macro definition

If you have ever written reader macros, you may wonder why you should use defreader to define them even though they are simple unary functions.

This is because it is necessary to synchronize the evaluation timing of reducers and reader macros. To achieve this, defreader expands a definition of a reader macro like the following:

(defreader greeting-reader [v opts]
  (str "Hello, " v "!"))

;;; ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓

(let [f (fn [v opts]
          (str "Hello, " v "!"))]
  (defn greeting-reader [v]
    {:baum.core/invoke [f v]}))

So, the actual evaluation timing of your implementation is the reduction phase and this is performed by an internal built-in reducer.

One more thing, you can access reader options!

Writing your own reducers

In contrast to reader macros, there is no macro to define reducers. All you need to do is define a ternary function. Consider the following reducer:

{:your-ns/narrow [:a :c]
 :a :foo
 :b :bar
 :c :baz
 :d :qux}

;;; ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓

{:a :foo
 :c :baz}

To implement this, you could write the following:

(ns your-ns
  (:require [baum.core :as b]))

(defn narrow [m v opts]
  (select-keys m v))

;; Put your reducer in reader options:
(b/read-file "config.edn"
             {:reducers {:your-ns/narrow narrow}})

In the above example, v is a value under the :your-ns/narrow key and m is a map from which the :your-ns/narrow key has been removed. opts holds reader options. So narrow will be called as follows:

(narrow {:a :foo :b :bar :c :baz :d :qux}
        [:a :c]
        {...})

By the way, the trigger key does not have to be a keyword. Therefore, you can write, for example, the following:

;;; config.edn
{narrow [:a :c]
 :a :foo
 :b :bar
 :c :baz
 :d :qux}

;;; your_ns.clj
(b/read-file "config.edn"
             {:reducers {'narrow narrow}})

License

Distributed under the Eclipse Public License, the same as Clojure.

rfkm / baum Goto Github PK

baum's Introduction