obazl / tools_opam Goto Github PK

View Code? Open in Web Editor NEW

5.0 4.0 1.0 1.16 MB

Bazel rules for OPAM support

Home Page: https://obazl.github.io/docs_obazl/

License: Eclipse Public License 2.0

Starlark 7.98% Shell 0.25% Makefile 0.06% C 87.37% C++ 1.50% Yacc 2.85%

bazel ocaml opam

tools_opam's Introduction

OBazl tools_opam

Tools & rules for using OPAM with OBazl

manpages

You can open them by file path:

$ man ./man/man1/coswitch.1

outdated

summary:

bazel run @opam//here - creates project-local OPAM installation with --root .opam and --switch obazl
bazel run @opam//init -- --import <file> - same as @opam//init but then imports packages using , which must 1) be located in .opam.d and 2) have the form of the files produced by opam switch export
bazel run @opam//install -- <pkg> - installs one OPAM package in the project-local switch (root .opam, switch 'obazl')
bazel run @opam//status - prints info about the effective OPAM switch.
bazel run @opam//ingest - generates BUILD.bazel files from effective OPAM installation (either sys install or project-local), writes them to .opam.d/buildfiles, and .opam.d/opam_repos.bzl containing repo rules.

development/testing

In https://github.com/obazl/dev_obazl/toolchains

Run in debug mode:

-D: pass --debug to opam cmd
-d: enable debug in @tools_opam cmd
-D: pass --verbose to opam cmd
-D: enable verbose in @tools_opam cmd
--@opam//bzl/debug:trace - compile @tools_opam cmd with DEBUG_TRACE. Put this before --

E.g.

$ bazel run @opam//here/opam:install --@opam//bzl/debug:trace -- -DdVv <pkg>

opam commands

Useful commands:

opam var - prints global opam vars, config vars from current switch, and a list of package variables that you can get by running:
opam var PKG:VAR - print value of variable VAR for package PKG, e.g.
- opam var ocaml:lib - prints path to the ocaml std lib, something like ~/ 4.12.0/lib/ocaml. So this will show you where to find the META file for any OPAM package. You can get the list of available variables with:
opam config list PKG - print variables for package PKG; for example:
- opam config list ocaml (NB: opam treats ocaml as a package)
opam info PKG - prints a bunch of metadata, like versions, maintainer, etc.
opam show PKG - same as opam info PKG?
opam install --download-only (opam v. 2.1.?)

These commands have lots of options, use --help to see them.

"core" libraries/archives/packages

WARNING OBSOLETE

WARNING In the literature "standard library" is sometimes used to refer to lib/ocaml. Not to be confused with the module Stdlib, "The OCaml Standard Library", which is a namespaced archive (stdlib.cmxa) installed in lib/ocaml.

The standard compiler distribution contains a set of resources (archives, plugins) that the findlib system describes in "special" META files. The packages desribed in these META files refer to files in lib/ocaml; the directory holding the META file contains no Ocaml files. In contrast, most (all?) other META package specs refer to files in the same directory or one of its subdirectories.

Since these are standard, their BUILD.bazel files are predefined by OBazl. That is, they are not generated from the META files by the bootstrapper.

Furthermore we adapt findlib names to make them more idiomatic in Bazel. Where findlib has compiler-libs.X, we have @rules_ocaml//cfg/compiler-libs:X:

compiler-libs => @rules_ocaml//cfg/compiler-libs
compiler-libs.common => @rules_ocaml//cfg/compiler-libs/common
compiler-libs.bytecomp => @rules_ocaml//cfg/compiler-libs/bytecomp
compiler-libs.optcomp => @rules_ocaml//cfg/compiler-libs/optcomp
compiler-libs.toplevel => @rules_ocaml//cfg/compiler-libs/toplevel
compiler-libs.native-toplevel => @rules_ocaml//cfg/compiler-libs/native-toplevel

The others we also put in the @ocaml namespace:

bigarray => @rules_ocaml//cfg/bigarray
dynlink => @rules_ocaml//cfg/dynlink
str => @rules_ocaml//cfg/str
unix => @rules_ocaml//cfg/unix

The stdlib module is always included (and opened) by the compiler, so there is no build target for it.

Threads require special treatment because threading support has changed over the years. ocamlfind has command-line switchs (-mt, -mt_vm, -mt_posix) for threading, but there are no corresponding options for the compilers; building with thread support just requires depending the the threading lib. Support for virtual threads was removed in version X, but threads/META still exposes threads.vm.

OBazl does not support the ocamlfind threading options, eliminates the distinction between posix and vm threads.

threads.posix => @rules_ocaml//cfg/threads
threads.vm => @rules_ocaml//cfg/threads

The list:

lib/bigarray
lib/compiler-libs
lib/dynlink
[lib/stdlib]?
lib/str
lib/threads
lib/unix

Note that these correspond to archives or subdirs in /lib/ocaml.

Creating a fresh 4.13.0 switch installs the following in ${SWITCHPREFIX}/lib. No findlib META files are installed.

ocaml - the "core" archives and related files (cmx, cmi, etc.):
- bigarray.cmxa
- dynlink.cmxa
- stdlib.cmxa
- str.cmxa
- unix.cmxa
"core" plugins:
- bigarray.cmxs
- str.cmxs
- unix.cmxs
lib/ocaml also has the following subdirectories:
- stublibs - libs used by ocamlrun to deal with C dependencies.: dllcamlstr.so, dllthreads.so, dllunix.so
- caml - C headers
- compiler-libs - contains archives, exported by /lib/compiler-libs:
  - ocamlbytecomp.cmxa
  - ocamlcommon.cmxa
  - ocamlmiddleend.cmxa
  - ocamloptcomp.cmxa
- threads - threads.cmxa, exported by lib/threads
- ocamldoc
stublibs - empty!
toplevel - empty!

special opam packages

compiler libs

pkg "compiler-libs" - provided by OCaml core? Not found as a separate package in opam-repository, so evidently it gets installed when a switch is created(?). The META file in an opam installation says "distributed with Ocaml", and defines subpackages whose files come from `lib/ocaml/compiler-libs" ( directory = "+compiler-libs" ).
pkg ocaml-compiler-libraries - dune-based repackaging of compiler-libraries. Essentially partitions the libraries into namespaces.

tools_opam's People

Contributors

Stargazers

Watchers

Forkers

jgertm

tools_opam's Issues

findlib predicates

findlib META files commonly have both requires(ppx_driver) and requires(-ppx_driver) and similar. For example ppx_sexp_conv.

Currently our default target, e.g. @opam//lib/ppx_sexp_conv:ppx_sexp_conv is based on the ppx_driver predicate. Where we also find -ppx_driver we also have a no_ppx_driver target.

TASK: a second target is not necessary; we can handle this situation with build settings and select on the deps attribute. So instead of:

ocaml_import(
    name = "ppx_sexp_conv",
    version = """v0.14.0""",
    archive = select({
        "@ocaml//mode:bytecode": [
            "//:_lib/ppx_sexp_conv/ppx_sexp_conv.cma",
        ],

        "@ocaml//mode:native": [
            "//:_lib/ppx_sexp_conv/ppx_sexp_conv.cmxa",
            "//:_lib/ppx_sexp_conv/ppx_sexp_conv.a",
        ],

    }),
    deps = [
        "@opam//lib/ppx_sexp_conv/expander",
        "@opam//lib/ppxlib",
    ],
    visibility = ["//visibility:public"]
)

and another `ocaml_import(name = "no_ppx_driver...), we would have something like:

ocaml_import(
    name = "ppx_sexp_conv",
    version = """v0.14.0""",
    archive = select({
        "@ocaml//mode:bytecode": [
            "//:_lib/ppx_sexp_conv/ppx_sexp_conv.cma",
        ],

        "@ocaml//mode:native": [
            "//:_lib/ppx_sexp_conv/ppx_sexp_conv.cmxa",
            "//:_lib/ppx_sexp_conv/ppx_sexp_conv.a",
        ],

    }),
    deps = select({
       "ppx_driver_only": [ ## emulates `-predicates ppx_driver`
            "@opam//lib/ppx_sexp_conv/expander",
            "@opam//lib/ppxlib",
        ],
        "no_ppx_driver_only": [ ## emulates `-predicates -ppx_driver`
            "@opam//lib/ppx_sexp_conv/runtime-lib",
        ],
        "both_disabled": [ ## predicates: -ppx_driver, -custom_ppx,
            "@opam//lib/ppx_deriving",
            "@opam//lib/ppx_sexp_conv/runtime-lib",
        ],
        no_match_error("some error msg...")
    }),
    visibility = ["//visibility:public"]
)
load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
bool_flag( name = "ppx_driver", build_setting_default = True )
bool_flag( name = "custom_ppx", build_setting_default = True )

config_setting(name = "ppx_driver_only",
               flag_values = {":ppx_driver": "True",
                              ":custom_ppx": "True"})
config_setting(name = "no_ppx_driver_only",
               flag_values = {":ppx_driver": "False",
                              ":custom_ppx": "True"})
config_setting(name = "both_disabled",
               flag_values = { ":ppx_driver": "False",
                               ":custom_ppx": "False" })

Note that we do not have "//conditions:default" because we want to exclude any combinations we do not explicitly handle, rather than send them to a default resolution. So the effective default is both flags true, which is case ppx_driver_only.

opam external (system) deps

Some opam deps need locally installed libs, e.g. libffi. This needs to be addressed for hermetic builds.

example, running `opam switch import switch.import --switch 4.10:

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
[ERROR] The compilation of ctypes failed at "/Users/gar/.opam/opam-init/hooks/sandbox.sh build make XEN=disable libffi.config".

#=== ERROR while compiling ctypes.0.18.0 ======================================#
# context     2.0.7 | macos/x86_64 | ocaml-base-compiler.4.10.2 | https://opam.ocaml.org#5c5a526e
# path        ~/.opam/4.10/.opam-switch/build/ctypes.0.18.0
# command     ~/.opam/opam-init/hooks/sandbox.sh build make XEN=disable libffi.config
# exit-code   2
# env-file    ~/.opam/log/ctypes-64700-ad6566.env
# output-file ~/.opam/log/ctypes-64700-ad6566.out
### output ###
# [...]
# ./discover -ocamlc "ocamlfind  ocamlc" > libffi.config || (rm libffi.config && false)
# testing for brew: .............................. available
# testing for MacPorts: .......................... unavailable
# testing for pkg-config: ........................ available
# testing for libffi:Fatal error: exception Failure("You need to 'brew install libffi' to get a suitably up-to-date version")
# ocamlfind  ocamlc -o discover -package str -strict-sequence -linkpkg src/discover/commands.mli src/discover/commands.ml src/discover/discover.ml -I src/discover
# ./discover -ocamlc "ocamlfind  ocamlc" > libffi.config || (rm libffi.config && false)
# testing for brew: .............................. available
# testing for MacPorts: .......................... unavailable
# testing for pkg-config: ........................ available
# testing for libffi:Fatal error: exception Failure("You need to 'brew install libffi' to get a suitably up-to-date version")
# make: *** No rule to make target `libffi.config'.  Stop.

cmt/cmti files

How should these be handled by ocaml_import? For now we ignore them. We have one special case, bls12-381.

OpamSwitch: enable selection of 'default' switch

Allow the developer to select the 'default' switch, no matter what compiler version it uses. Maybe: if no listed switch is marked with default = True, then use the 'default' switch? A side effect of this is that using the 'default' switch disables all version checking and pinning.

Related: select a switch based on name only, accepting any compiler version.

handle 'exists_if' property

Sometimes subpackages must be installed separately. Example, from ctypes/META:

package "foreign" (
 version = "0.19.1"
 description = "Dynamic linking of C functions"
 requires = "threads ctypes"
 archive(byte) = "ctypes-foreign.cma"
 archive(byte, plugin) = "ctypes-foreign.cma"
 archive(native) = "ctypes-foreign.cmxa"
 archive(native, plugin) = "ctypes-foreign.cmxs"
 exists_if = "ctypes-foreign.cma"
)

Currently the bootstrapper does not check 'exists_if', so it installs lib/ctypes/foreign/BUILD.bazel, and if ctypes-foreign is not installed we get "no such target".

handle ppx.exe in opam META files

META files for pkgs containing a ppx.exe file have something like:

# This line makes things transparent for people mixing preprocessors
# and normal dependencies
requires(-ppx_driver) = ""
ppx(-ppx_driver,-custom_ppx) = "./ppx.exe --as-ppx"
library_kind = "ppx_rewriter"

no such package '@opam//init'

I'm trying to see if I can use bazel to build OCaml code since I'm using bazel in same repo anyway.

I checked out latest rules_ocaml repo and since I know it relies on tools_opam I ran the command that's in readme and it doesn't exist:

√ rules_ocaml % bazel run @opam//init -- --import .obazl.d/opam/local.manifest
ERROR: Skipping '@opam//init': no such package '@opam//init': BUILD file not found in directory 'init' of external repository @opam. Add a BUILD file to a directory to mark it as a package.
WARNING: Target pattern parsing failed.
ERROR: no such package '@opam//init': BUILD file not found in directory 'init' of external repository @opam. Add a BUILD file to a directory to mark it as a package.
INFO: Elapsed time: 0.154s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

The rules_ocaml demos readme says to use opam update, opam switch. Does this mean I need an opam installation on my system?

macos: libffi needed by ctypes

needed by ctypes. brew install:

...
libffi is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

For compilers to find libffi you may need to set:
  export LDFLAGS="-L/usr/local/opt/libffi/lib"
  export CPPFLAGS="-I/usr/local/opt/libffi/include"

For pkg-config to find libffi you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/libffi/lib/pkgconfig"

In spite of this warning, after this opam install ctypes succeeds.

update tool: special-case opam pkg 'num'

Add special-case logic for this pkg to 'update' cmd.

handle ocamldoc pkg

lib/ocamldoc/META:

# Specification for the "ocamldoc" library
requires = "compiler-libs"
version = "[distributed with Ocaml]"
description = "ocamldoc plugin interface"
directory= "^ocamldoc"

tools (e.g. @opam//coswitch refresh) currently do not handle ^ocamldoc correctly:

pkg_symlinks: Unable to opendir /Users/gar/.opam/4.14.0/lib/ocamldoc/^ocamldoc

fix opam verification and pinning

Verification should not entail pinning, but pinning entails verification.

Empty spec should fail verification. Currently an empty version string fails (correctly), but the empty list does not:

"ppx_jane": [""],  ## fails verification
"ppxlib": []   ## passes verification

Problem: the empty list is currently used for packages that are distributed with the compiler and have no version string, e.g. bytes, unix, etc. Use [None] for these?

Should failed verification throw an exception or just print a warning?

handle `requires(-ppx_driver)` correctly

Using findlib, -ppx_driver means "NOT a ppx_driver", meaning use this as a "normal" dep, not a ppx dep. I think.

So in such cases we need two build targets, for for ppx and one not. Currently we only have one, for ppx.

Examples: ppx_bin_prot, ppx_expect

ppx_expect META:

# This is what dune uses to find out the runtime dependencies of
# a preprocessor
ppx_runtime_deps = "ppx_expect.collector ppx_expect.config"
# This line makes things transparent for people mixing preprocessors
# and normal dependencies
requires(-ppx_driver) = "ppx_expect.collector
                         ppx_expect.config
                         ppx_here.runtime-lib
                         ppx_inline_test.config
                         ppx_inline_test.runtime-lib"
ppx(-ppx_driver,-custom_ppx) = "./ppx.exe --as-ppx"
library_kind = "ppx_rewriter"

From this we emit ppx_codeps = ["@ppx_expect//collector", "@ppx_expect//config"] because of the ppx_runtime_deps entry, but we don't handle the requires(-ppx_driver) correctly, nor the ppx(...) entry.

In the meantime: this does not prevent the user from using the lib, it just means the deps may need to be hand-tuned.