Code Monkey home page Code Monkey logo

bazel-deps's Introduction

bazel-deps

Generate bazel dependencies transitively for maven artifacts, with scala support.

Quickstart

This repo can be cloned and built locally, or you can download pre-build binaries for MacOS and Linux in the releases page. Automatic releases are generated for every commit against master. We also include a bash script in the releases which will let you easily download/run on mac/linux a default configuration for running bazel-deps.

A flow like:

  1. Download the bash script paired with the release, it has the expected per platform sha256's embedded in it
  2. Place in your repo and chmod +x update_dependencies.sh, maybe in a scripts folder if you wish.
  3. Copy the dependencies.yaml from this repo, or write your own
  4. Run the script, it should produce some files in 3rdparty
  5. Add to your workspace:
load("//3rdparty:workspace.bzl", "maven_dependencies")
maven_dependencies()
load("//3rdparty:target_file.bzl", "build_external_workspace")
build_external_workspace(name = "third_party")
  1. You can now add dependencies to your BUILD files as described below. eg deps = ["@third_party//3rdparty/jvm/org/scalaj:scala_http"].

Usage

First, list all of your maven artifact dependencies in a Dependencies file.

Next, run parseproject on your project yaml file. This will create a tree of BUILD files that match the maven group id, and the artifact id will be a label in a BUILD file. You should not edit these by hand, and instead have a separate directory for any exceptions that you manage along with Replacements. For instance, this project is setup with:

./bazel run //:parse -- generate -r `pwd` -s 3rdparty/workspace.bzl -d dependencies.yaml

We give three arguments: the path to the file we will include in our workspace. The path to the root of our bazel repo. The path to the dependencies file. You can also run with --help.

Then you should add

load("//3rdparty:workspace.bzl", "maven_dependencies")

maven_dependencies()

to your workspace to load the maven dependencies.

For example, if your project is located at MY_PROJ_DIR, your dependencies file is at MY_PROJ_DIR/dependencies.yaml, and your checkout of bazel-deps is at BAZEL_DEPS, to generate the dependencies you need to do the following:

cd $BAZEL_DEPS
./bazel run //:parse generate -- --repo-root "$MY_PROJ_DIR" --sha-file 3rdparty/workspace.bzl --deps dependencies.yaml

The final result in MY_PROJ_DIR will look like this

MY_PROJ_DIR
├── 3rdparty             <-- everything under here is generated by running gen_maven_deps.sh
│   ├── workspace.bzl        <-- load() this from main WORKSPACE
│   └── jvm/                 <-- generated BUILD files in this directory.
├── BUILD
├── WORKSPACE
└── dependencies.yaml    <-- your project's dependencies are declared in here.

Whenever you update the dependencies declared in dependencies.yaml you will need to regenerate the contents of the 3rdparty directory by re-running $BAZEL_DEPS/gen_maven_deps.sh generate.

CI integration

In a CI, you will often want to make sure there is alignment between the configuration file for bazel-deps and the resulting generated files or directories, you can run generate with --check-only and it will check that each file matches bit-for-bit, but does not generate. If something does not match what would have been generated, you get a non-zero return value and a list of the mismatches logged to error.

Alternate outputs, external repo

Bazel-deps can also prepare the outputs, not as a file tree but an external repo. With this one would refer to targets as @third_party//foo:bar rather than //3rdparty/jvm/foo/bar. This is useful if you do not want to check in generated code to your repo. Also if multiple repos are depending upon one another and using bazel deps this can avoid broken transitive dependencies. That is if there are two repos A and B where B depends on A: where A has Foo 1.0 dependson Jackson27 and B has Foo 2.0 depends on circe with the checked in version both will compile from source against the local copy of Foo, but transitively on the classpath in the repo B Jackson27 will be on the classpath rather than circe.

To use this option you would execute bazel-deps like:

cd $BAZEL_DEPS
./bazel run //:parse generate -- --repo-root "$MY_PROJ_DIR" --sha-file 3rdparty/workspace.bzl --deps dependencies.yaml --target-file 3rdparty/target_file.bzl --disable-3rdparty-in-repo

In your dependencies.yaml file you will likely want: thirdPartyDirectory: "" to avoid prefixing the remote repo path with 3rdparty/jvm.

And finally to load it from your WORKSPACE you would use:

load("//3rdparty:target_file.bzl", "build_external_workspace")

build_external_workspace(name = "third_party")

Customized integration

If you want to fully control how you create your third party dependencies, you can use bazel deps simply to normalize all the jars into a single canonical version for each artifact and present a json lock file which has the hashes and dependencies of each artifact. To do this you would do:

./bazel run //:parse -- generate -r `pwd` -d dependencies.yaml --resolved-output lock.json

The schema of the lock file should be rather obvious and it has all the information you would need.

Assumptions and usage

This tool will generate one canonical version for every jar in the transitive dependencies of the root dependencies declared. You have three conflict resolution modes currently (which currently apply globally):

  • fail: if more than one version is found transitively, fail.
  • fixed: for all artifacts explicitly added, use that version, otherwise fail if any other artifact has multiple versions.
  • highest: for all artifacts explicitly added, use that version, otherwise take the highest version.

In any case, we add a comment for any duplicates found in the workspace loading file.

To declare dependencies, add items to the dependencies key in your yaml file. The format should be yaml or json. It should have dependencies and it may have replacements and options. Important: only dependencies explicitly named have public visibility, transitive dependencies not listed in the dependencies file have visibility limited to the third party directory.

Dependencies are a map from maven group id to artifact id, with some metadata, such as:

dependencies:
  com.google.guava:
    guava:
      version: "18.0"
      lang: java

Language is always required and may be one of java, scala, scala/unmangled. To control the scala version, see the Options section. A common case are projects with many modules. For instance in the scalding project there are many modules: -core, -date, -args, -db, -avro to name a few. To reduce duplication you can do:

dependencies:
  com.twitter:
    scalding:
      version: 0.16.0
      lang: scala
      modules: [core, date, args, db, arvo]

The version field is optional. If it is absent, it means this jar is expected to be found by transitive dependencies, and it is available to be used outside of the thirdparty directory, but the exact version used can be selected according to the version resolution rules. It is an error to have an unversioned dependency that is not a transitive dependency of another versioned dependency.

A target may optionally add exports and exclude lists to a dependency. exports should be just the group and artifact (such as: com.twitter:scalding-core in the above), and they should be listed in the dependencies. exclude list should also be only the group and artifact.

It's possible to add generateNeverlink option to a dependency, which will make the generator to generate this dependency twice:

  1. With the normalized name as usual.
  2. With the name ${normalized}_neverlink and neverlink is set as true. This option should be used only for java dependencies, it will be ignored in any other lang.

Each group id can only appear once, so you should collocate dependencies by group. WARNING the parsing library we are using does not fail on duplicate keys, it just takes the last one, so watch out. It would be good to fix that, but writing a new yaml parser is out of scope.

Depending on artifacts with classifiers is straightforward: just add the packaging and classifier as part of the artifact id:

dependencies:
  net.sf.json-lib:
    json-lib:jar:jdk15: # artifact:packaging:classifier
      lang: java
      version: "2.4"

Note: Currently, only jar packaging is supported for dependencies. More work is needed on the bazel-deps backend to ensure that non-jar dependencies are written as data attributes, instead of regular jar dependencies.

Excluding artifacts with packaging or classifiers is similar to including dependencies. Non-jar packaging is supported for exclude.

  com.amazonaws:
    DynamoDBLocal:
      lang: java
      version: "1.11.86"
      exclude:
        - "com.almworks.sqlite4java:sqlite4java-win32-x86:dll"
        - "com.almworks.sqlite4java:sqlite4java-win32-x64:dll"
        - "com.almworks.sqlite4java:libsqlite4java-osx:dylib"
        - "com.almworks.sqlite4java:libsqlite4java-linux-i386:so"
        - "com.almworks.sqlite4java:libsqlite4java-linux-amd64:so"

A target may also optionally add processorClasses to a dependency. This is for annotation processors. bazel-deps will generate a java_library and a java_plugin for each annotation processor defined. For example, we can define Google's auto-value annotation processor via:

dependencies:
  com.google.auto.value:
    auto-value:
      version: "1.5"
      lang: java
      processorClasses: ["com.google.auto.value.processor.AutoValueProcessor"]

This will yield the following:

java_library(
    name = "auto_value",
    exported_plugins = [
        ":auto_value_plugin",
    ],
    visibility = [
        "//visibility:public",
    ],
    exports = [
        "//external:jar/com/google/auto/value/auto_value",
    ],
)

java_plugin(
    name = "auto_value_plugin",
    processor_class = "com.google.auto.value.processor.AutoValueProcessor",
    deps = [
        "//external:jar/com/google/auto/value/auto_value",
    ],
)

If there is only a single processorClasses defined, the java_plugin rule is named <java_library_name>_plugin. If there are multiple processorClasses defined, each one is named <java_library_name>_plugin_<processor_class_to_snake_case>.

There are a number of ways to customize the generated build files. These are controlled by the options dictionary at the root-level of the dependencies file. This is a list of all of the supported options.

  • buildHeader: usually you will want to configure your scala support here:
  buildHeader:
    - load("@io_bazel_rules_scala//scala:scala_import.bzl", "scala_import")
  • languages: an array of languages to be supported either Java or a specific version of Scala, e.g. [ "java", "scala:2.12.8" ].
  • thirdPartyDirectory: path to where we write the BUILD files for thirdparty. The default is 3rdparty/jvm. If you choose the Google default of third_party you will need to configure the licenses option as well.
  • versionConflictPolicy: fixed, fail or highest
  • transitivity: runtime_deps or exports
  • resolvers: the maven servers to use. Each resolver is defined by three keys, an "id", a "type", and a "url".
  resolvers:
    - id: "mavencentral"
      type: "default"
      url: https://repo.maven.apache.org/maven2/
    - id: "myserver"
      type: "default"
      url: https://my.private.maven.server.com/mvn/
  • resolverCache: (with resolverType: aether) where bazel-deps should cache resolved packages. local (target/local-repo in the repository root) or bazel_output_base (bazel-deps/local-repo inside the repository's Bazel output base -- from bazel info output_base). Coursier ignores this option and uses ~/.cache/coursier.
  • namePrefix: a string added to the generated workspace names, to avoid conflicts. The external repository names and binding targets of each dependency are prefixed.
  • strictVisibility: this is enabled by default, when enabled a target must be explicitly declared in the dependencies.yaml file or it will not be visible to the rest of the workspace. If it is set to false all targets will be generated with public visibility.
  • licenses: a set of strings added a licenses rule to each generated bazel target. Required by bazel if your build targets are under third_party/. See the licenses function in Bazel.
  • resolverType: aether or coursier. Note that aether is slower and seems to silently miss some dependencies for reasons we don't yet understand.
  • buildFileName: filename of the generated build files

In the default case, with no options given, we use:

  • allow java and scala 2.11
  • 3rdparty/jvm as the thirdPartyDirectory.
  • highest versionConflictPolicy
  • exports transitivity
  • use maven central as the resolver
  • local resolverCache
  • empty namePrefix ("")
  • coursier resolverType
  • BUILD as build file name

Some maven jars should not be used and instead are replaced by internal targets. Here are some examples of this:

  1. A subproject in the repo is published as a maven artifact (A). Others (B) depend on this artifact (B -> A) and in turn we depend on those (we have added B to our dependencies file). We don't want to pull A from a maven repo, since we build it internally, so we replace that artifact with an internal target.
  2. We get some scala artifacts directly from the sdk. So, if a jar says it needs org.scala-lang:scala-library we already have that (and a few other jars) declared, and we don't want to risk having two potentially incompatible versions.
  3. A small external project has both a bazel build and a maven publishing. We prefer to use the bazel build so we can easily pull more recent versions by bumping up a gitsha rather than waiting for jar to be published.

The replacements work on the level of artifacts. An artifact is replaced one-for-one with a local bazel target. For instance:

replacements:
  org.scala-lang:
    scala-library:
      lang: scala/unmangled # scala-library is not mangled like sbt does with other jars
      target: "@io_bazel_rules_scala_scala_library"
    scala-reflect:
      lang: scala/unmangled
      target: "@io_bazel_rules_scala_scala_reflect"

In this way, we redirect maven deps to those providers.

Note, we stop walking the graph when we see a replaced node, so the replacement target is now responsible for building correctly, and correctly exporting any dependencies that need to be on the compile classpath.

Code

This code was originally forked from pgr0ss/bazel-deps

This code was inspired by the aether examples for walking maven dependencies.

bazel-deps's People

Contributors

bjchambers avatar dependabot[bot] avatar drigz avatar eed3si9n avatar greggdonovan avatar hexaglow avatar hsyed avatar ianoc avatar ianoc-stripe avatar jin avatar johnynek avatar kevingessner avatar kojustin avatar kuliniew avatar lgirault avatar lionelbarrow avatar mcamenzind avatar mjduijn avatar non avatar oscar-stripe avatar pgr0ss avatar roblg avatar rynbrd avatar snoble avatar tekumara avatar travisbrown avatar travisbrown-stripe avatar uri-canva avatar vmax avatar zacharya19 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bazel-deps's Issues

Macro-support workaround breaks IntelliJ plugin

Apparently, the pattern bazel-deps uses to get around ijar+scala macros issue confuses the IntelliJ plugin. The problem is with using "//jar:file".

See the corresponding issue in the intellij plugin: bazelbuild/intellij#142
I'm cross-posting between repos because it's not clear whether this issue should be fixed by bazel-deps, the intellij plugin or maybe even bazel core.

The discussion which points at bazel-deps as a source of problem is here: bazelbuild/intellij#97 (comment)

The transitive dependency which has a version variable won't be generated

For example, the properties and dependencies in the pom.xml of artifact com.google.inject:guice:3.0 are:

<properties>
    <cglib.version>2.2.1-v20090111</cglib.version>
</properties>
<dependencies>
    <dependency>
      <groupId>javax.inject</groupId>
      <artifactId>javax.inject</artifactId>
      <version>1</version>
    </dependency>
    <dependency>
      <groupId>aopalliance</groupId>
      <artifactId>aopalliance</artifactId>
      <version>1.0</version>
    </dependency>
    <!--
     | Replace with official CGLIB artifact when it's released
    -->
    <dependency>
      <groupId>org.sonatype.sisu.inject</groupId>
      <artifactId>cglib</artifactId>
      <version>${cglib.version}</version>
    </dependency>
    <dependency>
      <groupId>javax.inject</groupId>
      <artifactId>javax.inject-tck</artifactId>
      <version>1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-beans</artifactId>
      <version>3.0.5.RELEASE</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>biz.aQute</groupId>
      <artifactId>bnd</artifactId>
      <version>0.0.384</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.felix</groupId>
      <artifactId>org.apache.felix.framework</artifactId>
      <version>3.0.5</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

Then only add the com.google.inject:guice:3.0 to the dependencies.yaml and run bazel-deps. However, it will only generate three dependencies without artifact org.sonatype.sisu.inject:cglib:2.2.1-v20090111.

def maven_dependencies(callback = declare_maven):
    callback({"artifact": "aopalliance:aopalliance:1.0", "lang": "java", "sha1": "0235ba8b489512805ac13a8f9ea77a1ca5ebe3e8", "repository": "http://szxy1.artifactory.cd-cloud-artifact.tools.huawei.com/artifactory/third-party-maven/", "name": "aopalliance_aopalliance", "actual": "@aopalliance_aopalliance//jar", "bind": "jar/aopalliance/aopalliance"})
    callback({"artifact": "com.google.inject:guice:3.0", "lang": "java", "sha1": "9d84f15fe35e2c716a02979fb62f50a29f38aefa", "repository": "http://szxy1.artifactory.cd-cloud-artifact.tools.huawei.com/artifactory/third-party-maven/", "name": "com_google_inject_guice", "actual": "@com_google_inject_guice//jar", "bind": "jar/com/google/inject/guice"})
    callback({"artifact": "javax.inject:javax.inject:1", "lang": "java", "sha1": "6975da39a7040257bd51d21a231b76c915872d38", "repository": "http://szxy1.artifactory.cd-cloud-artifact.tools.huawei.com/artifactory/third-party-maven/", "name": "javax_inject_javax_inject", "actual": "@javax_inject_javax_inject//jar", "bind": "jar/javax/inject/javax_inject"})

improve inference of scala-ness

A heuristic that is probably very good if that if your artifact id ends with _a.b for some integers a and b and you depend on a version of scala-library version a.b.x, we can assume you are scala.

If we get it wrong (before java-sandwich in bazel ships) you can just explicitly add the dependency and label the language as java.

This should help with macros, since we assume any scala dependency could have macros.

Parser Failure and Inability to Import Package

Context

I was trying to build a simple maven project with the following external dependencies:

dependencies:
  org.powermock:
    powermock-module-junit4:
      version: "1.5.1"
      lang: java

    powermock-api-mockito:
      version: "1.5.1"
      lang: java

  com.github.stefanbirkner:
    system-rules:
      version: "1.14.0"
      lang: java
      exclude: "junit:junit-dep"  # <--- THIS IS WHERE WEIRD BEHAVIOR DERIVES FROM

  junit:
    junit:
      version: "4.11"
      lang: java

Problem

Parser Issues

I found that the inclusion of the "excludes" statement led to a number of issues with the parser. In particular it led to the following error message.

[ERROR]: Failed to parse dependencies.yaml.
DecodingFailure([A]Set[A], List(DownField(exclude), DownField(system-rules), DownField(com.github.stefanbirkner), DownField(dependencies)))

It's clear that the tool is unable to generate transitive dependencies for com.github.stefanbirkner group. However it is unclear why this is the case.

Importing Package Issues

When I comment out the exclude line, the BUILD files are successfully generated. However, when I try to build the target that depends on com.github.stefanbirkner, I receive the following errors

src/test/java/com/example/javamavenjunithelloworld/HelloWithTestsIT.java:5: error: package org.junit.contrib.java.lang.system does not exist

This is a package within com.github.stefanbirkner, indicating that the package was unsuccessfully included.

Potential Cause

Versioning Issue?
I do know that the missing package org.junit.contrib.java.lang.system is a deprecated junit package that is now included within stefanbirkner's system rules. So perhaps it is some sort of versioning issue with junit. According to it's pom.xml, system-rules depends on junit-dep [4.9, ). However, junit-dep has been deprecated after junit-4.11

Optionally include the correct load statement for Scala files

For build files that include a scala rule, include the correct rules_scala load statement. E.g. if the BUILD file includes a scala_library rule, insert load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library") before the rules in that file.

This provides an option when you might not want to include scala_library in the prelude. See discussion.

Add support for classifier

For example, the artifact "net.sf.json-lib:json-lib:jar:jdk15:2.4" has a classifier jdk15,but the dependencies in the yaml do not have this option.

Resolve the circular dependency instead of throwing a stackoverflow exception?

For example, currently, when dependency A and dependency B depend on each other, this tool will throw a stackoverflow exception. As far as I know, the gradle will resolve it automatically by ignore the dependencies of one of A and B. Should this tool resolve the circular dependency like gradle or any other ways? Any suggestions?

Take Maven coords instead of YAML

... well, maybe not instead, both support something like

dependencies:
  'javax.servlet:javax.servlet-api:jar:4.0.0'

Instead of

dependencies:
  javax.servlet:
    javax.servlet-api:
      version: "4.0.0"
      lang: java

Add a scope option to dependencies

For example, the scopes of some dependencies in pom.xml are set to "provided", and we should add "neverlink = 1" to the corresponding java_library.

Generated BUILD files should use `java_import` instead of `java_library`

That is, instead of

java_library(
    name = "args4j",
    exports = [
        "//external:jar/args4j/args4j"
    ],
    runtime_deps =[
              <dependencies of args4j, if any>
    ],
    visibility = [
        "//visibility:public"
    ]
)

we should have

java_import(
    name = "args4j",
    jars = [
        "@args4j_args4j//jar:file"  # <--- this is a filegroup generated by maven_jar
    ],
    deps = [
        <dependencies of args4j, if any>
    ],
    visibility = [
        "//visibility:public"
    ]
)

The reason is that javac sometimes needs to access symbols from the dependencies of a jar during a compilation. I can come up with an exact explanation, but waving my hands, one example is using an interface defined in the superclass of your superclass.

Using runtime_deps will result in a broken build in this case.

(you can ask bazel-deps to put everything in exports, but this hurts readability and maintainability, and I recommend against it)

  1. What says you?
  2. How does this interact with Scala?

minimization in format has bugs.

Examples:

-    bijection:
+    bijection-core:
       lang: scala
-      modules: [ "core", "macros" ]
       version: "0.9.2"
     bijection-json:
       exports:
@@ -167,15 +164,22 @@ dependencies:
         - "org.codehaus.jackson:jackson-mapper-asl"
       lang: scala
       version: "0.9.2"
+    bijection-macros:
+      lang: scala
+      version: "0.9.2"

Since we go through in order, the first one that fails to match makes a cut in the sorted order.

     chill:
       lang: scala
-      modules: [ "", "algebird", "bijection", "scrooge" ]
+      modules: [ "algebird", "bijection" ]
       version: "0.7.3"
-    chill-hadoop:
-      lang: java
+    chill-:
+      lang: scala
       version: "0.7.3"
-    chill-java:
+    chill:
       lang: java
+      modules: [ "hadoop", "java" ]
+      version: "0.7.3"
+    chill-scrooge:
+      lang: scala
       version: "0.7.3"

There are two bugs here: 1 the cut bug, 2 somehow chill- is showing up, but it should be just chill.

Unable to build

#It seems that the specified version of git://github.com/bazelbuild/rules_scala doesn't work for me. I can can get it to build by reverting to an older version:

10:00:41 /tmp$ bazel version
Build label: 0.4.3-homebrew
Build target: bazel-out/local-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Dec 22 15:20:22 2016 (1482420022)
Build timestamp: 1482420022
Build timestamp as int: 1482420022
10:00:49 /tmp$ git clone [email protected]:johnynek/bazel-deps.git
Cloning into 'bazel-deps'...
remote: Counting objects: 1373, done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 1373 (delta 72), reused 0 (delta 0), pack-reused 1143
Receiving objects: 100% (1373/1373), 232.15 KiB | 70.00 KiB/s, done.
Resolving deltas: 100% (504/504), done.
10:01:44 /tmp$ cd bazel-deps/
10:01:51 (master) /tmp/bazel-deps$ bazel build src/scala/com/github/johnynek/bazel_deps:parseproject_deploy.jar
..........
ERROR: /private/var/tmp/_bazel_timd/a7de4f7e235bf03b78f61c24d46128fc/external/io_bazel_rules_scala/scala/scala.bzl:26:16: function 'depset' does not exist.
ERROR: com.google.devtools.build.lib.packages.BuildFileContainsErrorsException: error loading package '': Extension 'scala/scala.bzl' has errors.
INFO: Elapsed time: 5.442s
10:02:23 (master) /tmp/bazel-deps$ sed -i 's/b51e54cf0a77f66c269c8c8fa24d62cac388337d/690cf39eba8eccf3c6e21ca77d8a4e1710aa3629/' WORKSPACE
10:02:57 {master} /tmp/bazel-deps$ bazel build src/scala/com/github/johnynek/bazel_deps:parseproject_deploy.jar
.... works ok

`add-dep` won't create or handle an empty dependencies.yaml

  1. Build bazel-deps (tested this @ 1ec2d84)
  2. Run this in an empty directory: ~/development/bazel-deps/gen_maven_deps.sh add-dep --deps dependencies.yaml --lang java junit:junit:4.12

Expected: a new dependencies.yaml with the junit library added

Actual:

[ERROR]: Failed to parse dependencies.yaml.
io.circe.ParsingFailure: No content to map due to end-of-input
 at [Source: java.io.StringReader@1b66c0fb; line: 1, column: 1]
  1. touch dependencies.yaml and run add-dep again. Different error:
[ERROR]: Failed to parse dependencies.yaml.
io.circe.ParsingFailure: No content to map due to end-of-input
 at [Source: java.io.StringReader@1b66c0fb; line: 1, column: 1]

It'd be great if bazel-deps supported both of these cases. It will make the tool more friendly to new users and repos.

Support authentication to private maven repositories (Artifactory)

I tried to use a private Artifactory repository that requires authentication.

I have the appropriate credentials in my ~/.m2/settings.xml, but it looks like the resolver does not reference these credentials. The http request for the artifact gets a 401 unauthorized response.

The nonexistent optional dependency causes an exception

For example, I add the following to the dependencies.yaml.

org.springframework:
    spring-orm:
      lang: java
      version: "3.2.12.RELEASE"

And then run the tool to generate the files, I get the error:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at org.springframework:spring-orm:jar:3.2.12.RELEASE -> org.eclipse.persistence:org.eclipse.persistence.core:jar:1.0.1
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:291)
	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:316)
	at com.github.johnynek.bazel_deps.Resolver.request(Resolver.scala:64)
	at com.github.johnynek.bazel_deps.Resolver.addToGraph(Resolver.scala:155)
	at com.github.johnynek.bazel_deps.Resolver$$anonfun$addAll$1.apply(Resolver.scala:151)
	at com.github.johnynek.bazel_deps.Resolver$$anonfun$addAll$1.apply(Resolver.scala:151)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
	at com.github.johnynek.bazel_deps.Resolver.addAll(Resolver.scala:151)
	at com.github.johnynek.bazel_deps.MakeDeps$.apply(MakeDeps.scala:45)
	at com.github.johnynek.bazel_deps.ParseProject$.main(ParseProject.scala:11)
	at com.github.johnynek.bazel_deps.ParseProject.main(ParseProject.scala)
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.eclipse.persistence:org.eclipse.persistence.core:jar:1.0.1
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:329)
	at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.resolveCachedArtifactDescriptor(DefaultDependencyCollector.java:535)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.getArtifactDescriptorResult(DefaultDependencyCollector.java:519)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:409)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351)
	at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:254)
	... 16 more

The artifact org.eclipse.persistence:org.eclipse.persistence.core:jar:1.0.1 is an optional dependency of the artifact org.springframework:spring-orm:jar:3.2.12.RELEASE, but org.eclipse.persistence:org.eclipse.persistence.core:jar:1.0.1 is not in our private repositories or even in "http://repo.maven.apache.org/maven2". Any suggestion or workaround?

document why we transitively build all the targets

oscar [15:33] 
the reason I did it was simple: in a big repo all kinds of dependencies will be needed


[15:33] 
so, just pruning the graph back to a minimal set of targets is a pain


[15:33] 
if you just let bazel see all the granularity you don't have to worry


[15:33] 
so, we use cats-core and circe, which uses cats-core, no problem

collapsing modules sometimes fails

example from @ianoc

  org.apache.poi:
    poi:
      lang: java
    poi-ooxml:
      lang: java
    poi-ooxml-schemas:
      lang: java

as

    poi:
      lang: java
      modules: [ "ooxml", "ooxml-schemas" ]

But then next time i run the tool it fails because the non-suffixed target isn’t included. Doing

    poi:
      lang: java
      modules: [ "", "ooxml", "ooxml-schemas" ]

add support for exports

sometimes we want to add exported jars which the maven pom may have been too narrow with.

org.slf4j.impl.StaticLoggerBinder error from `generate`

I consistently see this warning when I run generate (tested @ 1ec2d84):

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

I've seen this on both linux and OS X. Can that missing dependency be added so the correct logs are written and the error is suppressed?

add an exclude option

some dependencies are not used by all targets, and we don't want to transitively pull them in.

gen_maven_deps.sh no longer working on OSX

I just updated this repo after a few months. It no longer works as expected on OSX:

It seems to generate a full path from the cwd to the workspace directory now, e.g. ROOT/Users/seanmcl/foo/.... See below.


$ cd ~/tmp/foo 
$ git clone https://github.com/johnynek/bazel-deps.git
Cloning into 'bazel-deps'...
...
$ ls
bazel-deps  dependencies.yaml
$ cd bazel-deps
$ git log -1
d886f0566f867329b0f5c559edf41781c67eebca
...
$ bazel build src/scala/com/github/johnynek/bazel_deps:parseproject_deploy.jar
...........
INFO: Found 1 target...
...
$ cd ..
$ ls
bazel-deps  dependencies.yaml
$ ./bazel-deps/gen_maven_deps.sh generate -r `pwd` -s 3rdparty/workspace.bzl -d dependencies.yaml
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
wrote 90 targets in 39 BUILD files
$ ls
3rdparty  Users  bazel-deps  dependencies.yaml  target
$ ls 3rdparty
jvm
$ find Users
Users
Users/seanmcl
Users/seanmcl/tmp
Users/seanmcl/tmp/foo
Users/seanmcl/tmp/foo/3rdparty
Users/seanmcl/tmp/foo/3rdparty/workspace.bzl

Allow user to avoid creating `declare_maven`

I'm guessing it allows Stripe to customize the callback?
I'm not sure it's useful for most users. Perhaps allow users to tell bazel-deps "create a callback inside 3rdparty/workspace.bzl while you write it?

Support annotation processors

For example, auto_value needs to run an annotation processor as part of a build.

The BUILD file that bazel-deps produces for auto-value is something like:

java_library(
    name = "auto_value",
    visibility = [
        "//visibility:public",
    ],
    exports = [
        "//external:jar/com/google/auto/value/auto_value",
    ],
)

In order to support annotation processing, one needs to change it to

java_library(
    name = "auto_value",
    exported_plugins = [
        ":auto_value_plugin",
    ],
    visibility = [
        "//visibility:public",
    ],
    exports = [
        "//external:jar/com/google/auto/value/auto_value",
    ],
)

java_plugin(
    name = "auto_value_plugin",
    processor_class = "com.google.auto.value.processor.AutoValueProcessor",
    deps = [
        "//external:jar/com/google/auto/value/auto_value",
    ],
)

Note the exported_plugins and processor_class attributes.

Proposal:
A dependencies entry in the yaml file will contain a field named processor_class. When it's present, bazel-deps will create a BUILD file similar to the example above.

empty module name generates invalid dependencies.yaml after add-dep

Start with this dependencies.yaml:

options:
  languages: [ "scala:2.11" ]
  resolvers:
    - id: "mavencentral"
      type: "default"
      url: https://repo.maven.apache.org/maven2/

dependencies:
  com.twitter:
    finagle:
      lang: scala
      modules: [ "", "core" ]
      version: "6.43.0"

Run gen_maven_deps.sh add-dep --deps dependencies.yaml --lang scala com.twitter:finagle-stats:6.43.0

I'd expect that "stats" would be added to finagle's modules, but instead this invalid dependencies.yaml is produced:

options:
  languages: [ "scala:2.11" ]
  resolvers:
    - id: "mavencentral"
      type: "default"
      url: https://repo.maven.apache.org/maven2/

dependencies:
  com.twitter:
    finagle-:
      lang: scala
      version: "6.43.0"
    finagle:
      lang: scala
      modules: [ "core", "stats" ]
      version: "6.43.0"

The finagle- dependency is invalid, and breaks generate. format-deps does not repair it.

(This is "bug 2" of number issue #44)

"could not normalize versions"

We're moving from many SBT projects to a single monorepo with Bazel. I'm in the midst of trying to figure out our dependencies.

We have versionConflictPolicy: fixed.

I'm getting the following message:

[ERROR] could not normalize versions:
io.netty:netty: Version(3.6.6.Final), Version(3.7.0.Final), Version(3.10.1.Final)
(snip)

I've tried fixing the netty version with

dependencies:
  (snip)
  "io.netty":
    "netty":
      version: "3.10.1.Final"
      lang: "java"
  (snip)

However, I keep getting the version conflict.

I've made sure that there are no other io.netty dependencies declared anywhere else in dependencies.yaml.

Is this a bug or am I simply missing something?

make it easy to depend on remote repos using bazel-deps

If we had the maven_repositories method accepts a {} of already registered maven jars, so we don't duplicate, we could skip those, then return the list of actually registered items.

This would make it fairly easy to compositionally depend on many remote repos using this tool. Seems like it would be a nice win.

Highest version policy might not always take the highest version

Hi,
Looking at the implementation of highest https://github.com/johnynek/bazel-deps/blob/b2c4eb5335afee268397a937fd3738e5b29226ff/src/scala/com/github/johnynek/bazel_deps/DepsModel.scala#L957 it seems like we take highest version according to simple string comparison but maven has a bit of different rules for latest/highest version

For instance take spring versioning system:

  • 3.5.0-RELEASE (last released version)
  • 3.5.0-SNAPSHOT (last snapshot version before the release)

But according to simple string comparison the max is the SNAPSHOT version.

Hint:
maven-artifact has an object called "ComparableVersion" that can help in comparing versions according to maven rules.

duplicates in dependencies does not error

if you have duplicate groups in the dependencies, the last one is silently taken, which is quite confusing.

I think we should just fail the parse and explain which one is duplicated.

Support for version ranges.

This piggy backs off of issue #45.

When passed a maven coordinate with a range of dependencies, it seems that aether stops walking the graph at that node.

duplicate entry generated in workspace.bzl

Not really sure how to reproduce, but we just saw an example of a duplicate artifact being added to workspace.bzl (same maven artifact and group id, but different versions).

I think we need to finally build the scalacheck property checks on normalizer. We have put it off because generating random sensible fake dependency graphs is not trivial.

We should do it and verify that for all input graphs, each group and artifact id comes out once.

Make indirect dependencies have restricted visibility

Right now, all the generated targets have visibility = ["//visibility:public"]. This allows a kind of strict dependency violation where a build target could depend on a maven artifact not mentioned in the yaml file.

The targets for the indrect dependencies should have visibility = ["//${thirdPartyDirectory}:__subpackages__"] so that they can only be referenced by targets under the third party directory.

Originally #98 (comment)

Cannot redefine repository after any load statement in the WORKSPACE file

Hi,
a crosslink to a bazel bazelbuild/bazel#1984 issue that happens when two same maven_jar dependencies are specified. That may happen, when two independent workspaces are included with "local_repository"

See also

https://github.com/ramtej/bazel-multiworkspace-sampler.git

I;m not really sure if the problem can/should be solved within bazel-deps or if that is something that has to be done in bazel "core".

Best Regards,
jj

Documentation request: How do I specify a resolver for a dependency?

I'm trying to use maven central for all dependences that live there, but jcenter for those that don't. In options I have

options:
  ...
  resolvers:
    - id: "mavencentral"
      type: "default"
      url: https://repo.maven.apache.org/maven2/
    - id: "jcenter"
      type: "secondary"  # What are the legal types?
      url: https://jcenter.bintray.com/
  ...

but I don't know how to tell bazel-deps that a particular dep should use jcenter. Currently, workspace.bzl is generated with all dependencies having resolver maven central, even though I get errors when running bazel. One example is

  com.monsanto.arch:
    cloud-formation-template-generator:
      version: "3.5.2"
      lang: scala

This generates

    callback({"artifact": "com.monsanto.arch:cloud-formation-template-generator_2.12:3.5.2", "lang": "scala", "sha1": "096d8df8dbcf702413c1f8fea3a24314b5888979", "repository": "https://repo.maven.apache.org/maven2/", "name": "com_monsanto_arch_cloud_formation_template_generator_2_12", "actual": "@com_monsanto_arch_cloud_formation_template_generator_2_12//jar:file", "bind": "jar/com/monsanto/arch/cloud_formation_template_generator_2_12"})

I tried

  com.monsanto.arch:
    cloud-formation-template-generator:
      version: "3.5.2"
      lang: scala
      resolver: jcenter

but that had no effect. I grepped through the source code looking for where you look up the resolver, but couldn't figure it out.

Thanks!

load.bzl

The README refers to load.bzl but doesn't mention that you have to write it yourself, or link to any default implementation.

If you copy the one from this repo's 3rdparty, you will get errors about the bind attribute being missing.

clean up stale builds

if a build file should be totally removed, this tool does not do it, causing confusing errors on bazel build //...

add runtime_deps

sometimes we may want to forcibly set a dependency as only a runtime dep, or add an unmentioned dependency.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.