The julia version check for packages is currently disabled:

https://github.com/JuliaLang/Pkg3.jl/blob/0263099e2dac1b75f2988893ba9bce8911c096cb/src/Operations.jl#L137

since the current registry upper bounds everything to 0.7. Disabling this checks allows testing this package on 0.7. We should probably upgrade the registry or figure out another solution to re-enable this check.

_find_package is used by Base.edit to search the source file. This function is also overridden by Pkg3, and when JULIA_ENV is set and not initialized, Operations.init_if_interactive interrupts.

julia> import Pkg3
julia> Base.edit(isfile, [String])
Could not find local environment in /home/osboxes/temp/Project.toml, do you want to create it?
 > yes
   no

When Pkg.build fails it prints a message, but does not propagate an exception (or result in a non-0 exit code when run from a script). I know this was probably done intentionally to allow trying to build multiple packages with minimal blockage, but I don't think it's a good default. It's very surprising to have, say, a VM setup script appear to succeed but have package build failures.

Right now it only uses the default of 6 weeks.

Due to this, I am unable to load any added packages

This is an addendum to #90. I think an important use-case for thinking about the precompilation when using containers for reproducible research and CI. Maybe the design of the package and precompilation system already support this use-case.

In particular,

• I would love to be able to create a docker image for my department which has precompiled versions of all of the main libraries required by economists.
• Then, we could write easy scripts for users to run custom scripts for individual projects which use that parent image, get the sourcecode for the computation, etc.
• The key would be that in the process, no precompilation outside of the users own code would be required.

After using Pkg3 for a while, this is quite annoying. I think one should explicitly have to instantiate these using some init command. If it is a git repo, we go to the base, otherwise just in the cwd.

I recently spent a some time with Julia v0.6 and Pkg2 and thinking like a new user to get a sense of usability issue. Forgetting about the actual speed of packages, my general feeling is that the perceived slowness around packages comes down to 3 main things: (1) there is insufficient visual feedback during add/using/update operations; (2) expectations are not properly set that time is required for both the download/installation and the recompilation; (3) user steps are required between installation and pre-compilation.

For problem (2) and (3) of perceived slowness, the users can become furious that after they wait out an update or add (with insufficient visual feedback that it is working, see #89 ) they try to use the package and suddenly they need to wait again (again without enough visual feedback on what is happening in the background).

This can be easily solved by two things: (a) pre-compiling at the end of an add or update so all waiting takes place at the same time, and (b) setting expectations in the output that pre-compilation will take some time. To do this:

• By default, precompile all packages upon add or update. You could give an option that turns the automatic pre-compilation off (e.g. Pkg.update(:precompile=false) , but most starting users would want to leave it on, so the default should be precompile.
• With the using intended to automatically add packages now, I suspect that the separation between the activities would already help things out.
• As in #89, much more output during the precompilation process. The extra output has the added benefit of grounding expectations that this might be a slow process.

A while ago tkelman helped me make RequirementVersions.jl, which via repeated testing automatically set version bounds on the packages in your REQUIRE file. It did this by progressively lowering the version numbers of the each required packages until it broke, then resetting versions to their initial values. Pkg2 made it pretty difficult to do this kind of thing. But since version bounds are essential to the stability of the package ecosystem, it might be worth considering whether this kind of thing would be possible to do in Pkg3.

Not sure this is on purpose, but customizations in ~/.juliarc.jl are erased:

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.6.1 (2017-10-24 22:15 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> LOAD_PATH
4-element Array{Any,1}:
 "/home/tamas/src/julia-v0.6.1/local/share/julia/site/v0.6"
 "/home/tamas/src/julia-v0.6.1/share/julia/site/v0.6"      
 "/home/tamas/src/julia-local-packages/"                   
 "/home/tamas/research/julia-projects/"                    

julia> using Pkg3

julia> LOAD_PATH
1-element Array{Any,1}:
 "/home/tamas/.julia/v0.6"

I recently spent a some time with Julia v0.6 and Pkg2 and thinking like a new user to get a sense of usability issue. Forgetting about the actual speed of packages, my general feeling is that the perceived slowness around packages comes down to 3 main things: (1) there is insufficient visual feedback during add/using/update operations; (2) expectations are not properly set that time is required for both the download/installation and the recompilation; (3) user steps are required between installation and pre-compilation.

For problem (1) of perceived slowness, I found myself angrily hitting <Enter> throughout add/using/update operations, because it was unclear if the REPL was stuck, whether I needed to have some input to flush the screen buffer, whether some user input was required for it to go to the next step, etc. I realized this was part of the reason it felt slow, even though there is no reason a user should otherwise expect downloading, installing, and compiling packages to be fast.

So, my suggestion is simple: much more verbose output for add/using/update operations, especially for new users. A few notes on this suggestion:

• It is especially important at the beginning of the operation, to make sure users understand that the process has started. As with a lot of installation processes, it will make the process feel faster because the user sees the computer working (even if the output itself is meaningless). From the pkg> add output in Pkg3 at this point, it already looks like an improvement on the quantity of output, but I would take it even further.
• After it figures out the dependency graph of packages, I would love to see output along the lines of downloading package XX of XX, installing package XX of XX, etc. This gives the users a sense of the amount of progress, even if there is no reason to believe thee operation is linear in the number of packages.
• It may be irritating for advanced users to get too much output text, so it probably makes sense to have a low vs. high verbose option for the add and update operations. Having it as a global setting isn't the worst idea for power users.
• I strongly suggest that the default is the high verbosity since the newest users are the least likely to play around with options.

Ref #53, #62. I've figured out a design that integrates the old LOAD_PATH loading style and the new Pkg3 project environment concept. This stems from two observations:

• A directory in LOAD_PATH is a kind of environment, if we consider the mapping from names to paths to be implicitly given by the file layout.

• We could put Pkg3-style environments in LOAD_PATH as well.

Instead of the JULIA_ENV environment variable that I had introduced to specify one and only one environment to load packages from, we can keep LOAD_PATH but with some changes in behavior:

1. If an entry in LOAD_PATH is a path to a directory not containing a Project.toml or JuliaProject.toml file, then it is considered an old-style implicit environment where package names are mapped to entry points based on the layout of the directory.

2. If an entry in LOAD_PATH is a path to a directory containing a Project.toml or JuliaProject.toml file, then it is considered a new-style explicit environment where package names are mapped to entry points based on the contents of the project and manifest files.

3. If an entry in LOAD_PATH is a path to a TOML file, then it is interpreted as the project file of a new-style explicit environment where package names are mapped to entry points based on the contents of the project file and the corresponding manifest files.

4. You can continue to control the default contents of LOAD_PATH via the JULIA_LOAD_PATH environment variable.

5. The julia --env=<env>... command line flag replaces the contents of LOAD_PATH with the given environment specifications, <env>....

6. The julia --env+<env>... command line flag appends the contents of LOAD_PATH with the given environment specifications, <env>....

Environment specification

Pkg3 has somewhat more complex needs for LOAD_PATH entries than just paths of directories. For example, one wants to be able to find the current project by looking for a git repo in the current working directory's parent directories. It's also common in Pkg3-style loading to want to consider only a sinlge environment for the sake of having one consistent package mapping, rather than an overlay of possibly maps which may not be consistent with each other. To that end, one may want to specify and exclusive alternation of possible environments and only use the first one that exists, rather than loading packages piecemeal from different LOAD_PATH entries. To that end, I propose introducing the following "rich" interpretations of --env=<env>, --env+<env> and JULIA_LOAD_PATH entries:

1. If an entry starts and ends with [ and ] it is interpreted as a comma-separated list of sub-entries, which are considered as exclusive environment alternatives, resolving to the first one which exists. This is represented in Julia as an array of environment specifiers.

2. If an entry starts with a valid Julia identifier followed by a ( and ends with a ) then it is interpreted as a custom environment specifier, and is represented in Julia as a corresponding type. There will be a whitelist of allowed identifiers and fixed corresponding types that they construct. If the syntax is used with a non-whitelisted identifier, the entry is invalid and will be ignored.

3. In custom environment specifiers, in the contents of any string literal, string interpolation syntax will replace certain whitelisted identifier names with corresponding values. This will include major, minor and patch which will be replaced with VERSION.major, VERSION.minor and VERSION.patch, respectively. Any custom environment specifier which uses an identifier name that is not whitelisted, is invalid and will be ignored.

Some custom environment specifiers that we'll want to initially support include:

• CurrentProject() to look for a project directory in the parents of the current working directory;
• NamedEnv("name") to look for a named environment in joinpath(DEPOTS[1], "environments");
• NamedEnv("name", create=true) to look for a named environment and create it if it doesn't exist.

These syntaxes are designed to mimic standard Julia function call syntax, but they are a not general: only a very specific, limited subset of Julia syntax is allowed; you cannot put arbitrary Julia code in the JULIA_LOAD_PATH environment variable and have it be executed. The contents are parsed and mapped to specific expected behaviors, not evaled. Arbitrary code can be evaluated to construct the contents of LOAD_PATH in the ~/.juliarc.jl file, however.

Default LOAD_PATH

The following is a possible good default value of LOAD_PATH should be in Julia syntax:

LOAD_PATH = Any[
    [ CurrentProject(),
      NamedEnv(“v$(VERSION.major).$(VERSION.minor).$(VERSION.patch)”),
      NamedEnv(“v$(VERSION.major).$(VERSION.minor)”),
      NamedEnv(“v$(VERSION.major)”),
      NamedEnv(“default”),
      NamedEnv(“v$(VERSION.major).$(VERSION.minor)”, create=true),
]

This would be specified in JULIA_LOAD_PATH of --env=<env> with the following string:

[CurrentProject(), NamedEnv(“v$major.$minor.$patch”), NamedEnv(“v$major.$minor”), NamedEnv(“v$major”), NamedEnv(“default”), NamedEnv(“v$major.$minor”, create=true)]

The meaning of this LOAD_PATH is that only a single environment is used to load packages, and it is the first of the following, which exists, assuming Julia version 1.2.3:

• the current project, found by searching the parent directories of the current working directory for a directory containing a Project.toml or a JuliaProject.toml file;
• the directory ~/.julia/environments/v1.2.3 if it exists
• the directory ~/.julia/environments/v1.2 if it exists
• the directory ~/.julia/environments/v1 if it exists
• the directory ~/.julia/environments/default if it exists
• the directory ~/.julia/environments/v1.2, creating it if it does not exist

Name collisions

When loading a package from a given environment, all dependencies recursively loaded in the process are resolved within the same environment. This is a significant semantic change from the previous LOAD_PATH behavior, wherein if you had LOAD_PATH = [ "dir1", "dir2" ] you could, in the process of loading Foo from dir1 have Foo require Bar and load it from dir2. This behavior would be incompatible with explicit Pkg3 environments, and since the whole premise of this scheme is that directories are an implicit environment, they should work in the same way. Fortunately, it seems unlikely that this would be a common problem in practice since if Foo is installed in dir1 one would expect all of Foo's dependencies to also be installed there.

In the presence of multiple environments in the LOAD_PATH there is a possibility of load order becoming significant in the following way: if one loads A which loads dependency D, if one subsequently loads B which depends on D as well, then B will get the version of D provided by the environment that A was loaded from; if A and B come from different environments, this could be a different version of D than the environment B comes from would provide, and if B had been loaded first a different version of D would have been loaded. It seems like this is just an inherent problem of having multiple LOAD_PATH entries overlaying different, potentially incompatible environments. In production, the LOAD_PATH should only ever contain a single environment entry and it should probably be spelled out explicitly as an absolute or relative path. The utility of multiple environments in the LOAD_PATH is primarily so that one can work on a project and easily load tools that don't belong in that project, but which one has installed in some directory or in a named environment. This may lead to version incompatibilities, but that's a acceptable for interactive debugging usage.

pkg> up JSOERROR (in the keymap): type Void has no field options
Stacktrace:
 [1] edit_backspace(::Base.LineEdit.PromptState) at .\repl\LineEdit.jl:609
 [2] edit_backspace(::Base.LineEdit.MIState) at .\repl\LineEdit.jl:178
 [3] (::getfield(Base.REPL, Symbol("##18#21")){Base.LineEdit.Prompt})(::Base.LineEdit.MIState, ::Void, ::Vararg{Any,N} where N) at .\repl\REPL.jl:701
 [4] (::getfield(Base.LineEdit, Symbol("##19#20")){getfield(Base.REPL, Symbol("##18#21")){Base.LineEdit.Prompt},String})(::Base.LineEdit.MIState, ::Void) at .\repl\LineEdit.jl:1059
 [5] prompt!(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface, ::Base.LineEdit.MIState) at .\repl\LineEdit.jl:2061
 [6] run_interface(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface) at .\repl\LineEdit.jl:1962
 [7] run_frontend(::Base.REPL.LineEditREPL, ::Base.REPL.REPLBackendRef) at .\repl\REPL.jl:986
 [8] run_repl(::Base.REPL.LineEditREPL, ::getfield(Base, Symbol("##542#543"))) at .\repl\REPL.jl:182
 [9] _start() at .\client.jl:439

This is on 0.7

I am trying to install PyCall. I was able to successfully add the package, but could not use it.

julia> using PyCall
splits out 
ERROR: LoadError: LoadError: Conda is not properly configured.  Run Pkg.build("Conda") before importing the Conda module.
Stacktrace:
 [1] error(::String) at ./error.jl:21
 [2] include_from_node1(::String) at ./loading.jl:569
 [3] eval(::Module, ::Any) at ./boot.jl:235
 [4] _require(::Symbol) at ./loading.jl:483
 [5] require(::Symbol) at ./loading.jl:398
 [6] include_from_node1(::String) at ./loading.jl:569
 [7] eval(::Module, ::Any) at ./boot.jl:235
 [8] _require(::Symbol) at ./loading.jl:483
 [9] require(::Symbol) at ./loading.jl:398
while loading /home/osboxes/.julia/packages/8f4d0f93-b110-5947-807f-2305c1781a2d/b26546424d5f3ac33b50f13b8442236deac42546/src/Conda.jl, in expression starting on line 43
while loading /home/osboxes/.julia/packages/438e738f-606a-5dbb-bf0a-cddfbfd45ab0/2ba54612238a85ee6c187a863faf6443eb645dcb/src/PyCall.jl, in expression starting on line 28

I checked the Conda.jl package directory. There was no deps/deps.jl file

Maybe a better error message or different behaviour?

julia> using Pkg3
INFO: Precompiling module Pkg3.

pkg> up
INFO: Cloning default registries into /Users/ken/.julia/registries
INFO:  [+] Uncurated = "https://github.com/JuliaRegistries/Uncurated.git"
INFO: Updating registry at /Users/ken/.julia/registries/Uncurated
INFO: Resolving package versions
ERROR: ArgumentError: reducing over an empty collection is not allowed
Stacktrace:
 [1] mr_empty_iter(::Function, ::Function, ::Base.Generator{Array{Pkg3.Types.PackageSpec,1},Pkg3.Operations.##30#33{Dict{Base.Random.UUID,String}}}, ::Base.EltypeUnknown) at ./reduce.jl:257
 [2] mapfoldl(::Base.#identity, ::Function, ::Base.Generator{Array{Pkg3.Types.PackageSpec,1},Pkg3.Operations.##30#33{Dict{Base.Random.UUID,String}}}) at ./reduce.jl:69
 [3] apply_versions(::Pkg3.Types.EnvCache, ::Array{Pkg3.Types.PackageSpec,1}) at /Users/ken/.julia/v0.6/Pkg3/src/Operations.jl:402
 [4] up(::Pkg3.Types.EnvCache, ::Array{Pkg3.Types.PackageSpec,1}) at /Users/ken/.julia/v0.6/Pkg3/src/Operations.jl:604
 [5] #up#13(::Pkg3.Types.UpgradeLevel, ::Symbol, ::Bool, ::Function, ::Pkg3.Types.EnvCache, ::Array{Pkg3.Types.PackageSpec,1}) at /Users/ken/.julia/v0.6/Pkg3/src/API.jl:112
 [6] (::Pkg3.API.#kw##up)(::Array{Any,1}, ::Pkg3.API.#up, ::Pkg3.Types.EnvCache, ::Array{Pkg3.Types.PackageSpec,1}) at ./<missing>:0
 [7] do_up!(::Pkg3.Types.EnvCache, ::Array{Tuple{Symbol,Vararg{Any,N} where N},1}) at /Users/ken/.julia/v0.6/Pkg3/src/REPLMode.jl:392
 [8] do_cmd!(::Pkg3.Types.EnvCache, ::Array{Tuple{Symbol,Vararg{Any,N} where N},1}, ::Base.REPL.LineEditREPL) at /Users/ken/.julia/v0.6/Pkg3/src/REPLMode.jl:141
 [9] do_cmd(::Base.REPL.LineEditREPL, ::String) at /Users/ken/.julia/v0.6/Pkg3/src/REPLMode.jl:112
 [10] (::Pkg3.REPLMode.##3#4{Base.REPL.LineEditREPL,Base.LineEdit.Prompt})(::Base.LineEdit.MIState, ::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/ken/.julia/v0.6/Pkg3/src/REPLMode.jl:470

julia> versioninfo()
Julia Version 0.6.1
Commit 0d7248e (2017-10-24 22:15 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, haswell)

StaticArrays 0.3.1 is upper bounded to julia 0.6, so it's uninstallable. Could use a better error message if you try to anyway:

pkg> add StaticArrays @0.3.1
INFO: Resolving package versions
ERROR: Unsatisfiable requirements detected for package 90137ffa-7385-5640-81b9-e52037218182:
└─version range [0.3.1,0.3.2) set by an explicit requirement
The intersection of the requirements is [0.3.1,0.3.2).
None of the available versions can satisfy this requirement.

pkg> add StaticArrays @0.3
INFO: Resolving package versions
ERROR: Unsatisfiable requirements detected for package 90137ffa-7385-5640-81b9-e52037218182:
└─version range [0.3.0,0.4.0) set by an explicit requirement
The intersection of the requirements is [0.3.0,0.4.0).
None of the available versions can satisfy this requirement.
Stacktrace:
 [1] filter_versions(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{AbstractString,Base.Pkg.Types.ResolveBacktraceItem}) at ./pkg/query.jl:260
 [2] prune_versions(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{AbstractString,Base.Pkg.Types.ResolveBacktraceItem}) at ./pkg/query.jl:328
 [3] prune_dependencies(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}, ::Dict{AbstractString,Base.Pkg.Types.ResolveBacktraceItem}) at ./pkg/query.jl:546
 [4] prune_dependencies(::Dict{String,Base.Pkg.Types.VersionSet}, ::Dict{String,Dict{VersionNumber,Base.Pkg.Types.Available}}) at ./pkg/query.jl:541
 [5] resolve_versions!(::Pkg3.Types.EnvCache, ::Array{Pkg3.Types.PackageSpec,1}) at /global/cscratch1/sd/kfischer/.julia/v0.6/Pkg3/src/Operations.jl:166
 [6] add(::Pkg3.Types.EnvCache, ::Array{Pkg3.Types.PackageSpec,1}) at /global/cscratch1/sd/kfischer/.julia/v0.6/Pkg3/src/Operations.jl:480
 [7] do_add!(::Pkg3.Types.EnvCache, ::Array{Tuple{Symbol,Vararg{Any,N} where N},1}) at /global/cscratch1/sd/kfischer/.julia/v0.6/Pkg3/src/REPLMode.jl:191
 [8] do_cmd(::Base.REPL.LineEditREPL, ::String) at /global/cscratch1/sd/kfischer/.julia/v0.6/Pkg3/src/REPLMode.jl:127
 [9] (::Pkg3.REPLMode.##3#4{Base.REPL.LineEditREPL,Base.LineEdit.Prompt})(::Base.LineEdit.MIState, ::Base.AbstractIOBuffer{Array{UInt8,1}}, ::Bool) at /global/cscratch1/sd/kfischer/.julia/v0.6/Pkg3/src/REPLMode.jl:273

Also maybe hide the backtrace?

Startup time is really slow even with precompilation. Presumably having a precompiles file would help that out, so it's worth trying. I guess the way to do this these days is with SnoopCompile?

Problem: We've had a recent situation where JuliaPro ships with a fixed set of package versions which were compatible at the time it was released, but a subsequent update to METADATA (JuliaLang/METADATA.jl#12106, see discussion in JuliaLang/METADATA.jl#12264) caused that set of versions to become incompatible. Unfortunately, this means that doing almost anything with that set of packages causes the resolver to say that the operation is not allowed because it leads to an incompatible ("infeasible") set of package versions.

Potential solution: In general, it would be good to allow operations that don't make the current situation any worse, and in general to afford users the option to pick their poison in situations where no fully compatible resolution exists. This seems to lead to a notion of "levels of incompatibility" for sets of versions rather than the current black-and-white notion of compatible or not. If you're already at a level of incompatibility, then an operation which doesn't make that level worse should be allowed without warning. If there are no solutions which don't make the situation worse, then it would be ideal to prompt the user with the best options for resolving the situation as well as possible.

I'd like to have a command that shows all the versions of a new package I could add to my Project without conflicting with existing requirements.

We could hook into the REPL auto completions system to provide completions when writin e.g. free CST<TAB> to complete it to free CSTParser.

How often should the registry be updated? Every time we run something that needs to resolve, should there be a "grace period" between registry updates etc?

Thoughts?

This currently has no tests. It needs some.

If a user accidentally modifies a package (for example by using @edit), the package could easily break and I don't think there is a way to detect it.

Perhaps there should be some option of doing an integrity check that checks that the tree hashes in the saved packages actually are what they are supposed to be?

Reference JuliaLang/julia#17418 (comment). Someone who knows more about the package machinery than me should comment on whether it's safe to assume there's a module with the same name as the package.

https://github.com/JuliaLang/Pkg3.jl/blob/3b8d98be0d43fce884d544ee0d4c6b6ee833d900/src/Operations.jl#L50

Is there such a functionality right now? It is likely needed, right?

Hi, I'm very optimistic about the new package manager. However, I can't get it to install anything. Following the readme:

using Pkg3

]

pkg> add JSON

ERROR: The following package names could not be resolved:                                                                                                                                        
 * JSON (not found in project, manifest or registry)
Please specify by known `name=uuid`.

versioninfo()

Julia Version 0.6.0                                                                                                                                                                              
Commit 903644385b* (2017-06-19 13:05 UTC)
Platform Info:
  OS: Linux (x86_64-unknown-linux-gnu)                                                                                                                                                           
  CPU: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz                                                                                                                                                  
  WORD_SIZE: 64                                                                                                                                                                                  
  BLAS: libopenblas (DYNAMIC_ARCH Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake-avx512)

The following conversion can (silently) overflow when dealing with typemax(VersionNumber), for example when a package has no upper bound.

https://github.com/JuliaLang/Pkg3.jl/blob/d335252ae8ea39b383daeb96217d6d8d3405e822/src/Types.jl#L137-L138

Could you explain the reason for the +1 @StefanKarpinski

While I appreciate how Julia's standard library handles package management as just another feature rather than a separate command line tool, Pkg is missing a documented search() capability. Unlike every package manager I've seen or used for programming language/os/userspace ecosystems, the only options here are web repository searches or pulling up a graphically enabled browser pointing to pkg.julialang.org. And if the latter is unavailable, such as working (on a machine) with no/restricted internet access, one must use a tool to snoop inside/search METADATA.jl.

Following the installation instructions in the README, I encounter the following error:

Pkg.clone("https://github.com/JuliaLang/Pkg3.jl.git")
# ERROR: Package metadata directory /tmp/pcarbo/julia/v0.6/METADATA doesn't exist; run Pkg.init() to initialize it.

The solution is given; however, I do not want to run Pkg.init() because it create the large number of small metadata files, the problem I'm seeking to avoid in the first place (because I'm running Julia on a networked filesystem on a compute cluster).

Has this issue already been addressed previously? Is there a workaround you can suggest?

Right now, the way that tarballs are downloaded is by pattern matching on the repo url and redirect to an archive.

In addition, perhaps the registry should allow an archive URL for a version of a package.

Alternatively, we could allow hooking into the pattern matching which is a bit "uglier" but might work ok.

cc @aviks

Perhaps it would be a good idea to be able to run the commands in a "dry mode" where the command is run but no modifications are made to anything.

This could be a good way to allow people to explore different commands without being scared of messing up something.

It would be nice to have a separate build.jl file just for running tests (e.g., pull down larger datasets just during testing).

Packages that are "discovered" during creation of the dependency graph are not added to the UUID -> name dictionary that is sent to the resolver, and are thereby not printed by name when resolution fails.

Ref #10

Using 32-bit and 64-bit versions of Julia on one computer is clunky with Julia v0.6 / Pkg2. When alternating, between versions, all packages have to be recompiled. In addition, if packages load binary libraries, you generally have to manually do a Pkg.build("MyPackage").

Does Pkg3 support this better?

This use case is mainly for testing different versions of Julia.

Eg in a context of

julia> using JLD2
Could not find package JLD2, do you want to install it?
 > yes
   yes (remember)
   no
   no (remember)

At the same time, the same terminal handles up/down keys for the REPL fine.

I did a custom open command for Pkg3 and recorded every time a file was opened and prints the count after each command. Here is some output:

pkg> add DataFrames
INFO: Resolving package versions
INFO: Updating "Project.toml"
 [no changes]
INFO: Updating "Manifest.toml"
 [no changes]
"/Users/kristoffer/.julia/registries/Uncurated/registry.toml"=>10
"/Users/kristoffer/.julia/registries/Uncurated/D/DataFrames/package.toml"=>3
"/Users/kristoffer/.julia/v0.6/Pkg3/Project.toml"=>3
"/Users/kristoffer/.julia/v0.6/Pkg3/Manifest.toml"=>3
"/Users/kristoffer/.julia/registries/Uncurated/J/JSON/dependencies.toml"=>2
"/Users/kristoffer/.julia/registries/Uncurated/C/Compat/versions.toml"=>2
"/Users/kristoffer/.julia/registries/Uncurated/F/FileIO/dependencies.toml"=>2
"/Users/kristoffer/.julia/registries/Uncurated/F/FileIO/versions.toml"=>2
....

It seems suspect that we are opening registry.toml 10 times for one command. Should be able to do better.

PkgDev needs to be updated / internalized into Pkg3.

Perhaps we should consider pulling in https://github.com/invenia/PkgTemplates.jl as a replacement to PkgDev.generate.

This surprised me a bit:

pkg> add JSON
INFO: Resolving package versions
INFO: Updating project file /Users/kfischer/.julia/environments/v0.6/Project.toml
 [682c06a0] + JSON
INFO: Updating manifest file /Users/kfischer/.julia/environments/v0.6/Manifest.toml
 [34da2185] + Compat v0.31.0
 [682c06a0] + JSON v0.13.0

pkg> add Gadfly
INFO: Resolving package versions
INFO: Updating project file /Users/kfischer/.julia/environments/v0.6/Project.toml
 [c91e804a] + Gadfly
INFO: Updating manifest file /Users/kfischer/.julia/environments/v0.6/Manifest.toml
 [13072b0f] + AxisAlgorithms v0.2.0
 [9e28174c] + BinDeps v0.6.0
 [49dc2e85] + Calculus v0.2.2
 [5ae59095] + Colors v0.8.1
 [3da002f7] + ColorTypes v0.6.1
 [bbf7d656] + CommonSubexpressions v0.0.1
 [a81c6b42] + Compose v0.5.4
 [d38c429a] + Contour v0.4.0
 [7ad07ef1] + CoupledFields v0.0.1
 [0fe7c1db] + DataArrays v0.6.2
 [a93c6f00] + DataFrames v0.10.1
 [864edb3b] + DataStructures v0.7.0
 [c5cfe0b6] + DiffBase v0.2.0
 [b4f34e82] + Distances v0.5.0
 [31c24e10] + Distributions v0.14.2
 [fa6b7ba4] + DualNumbers v0.3.0
 [5789e2e9] + FileIO v0.5.1
 [53c48c17] + FixedPointNumbers v0.4.0
 [f6369f11] + ForwardDiff v0.5.0
 [c91e804a] + Gadfly v0.6.4
 [92fee26a] + GZip v0.3.0
 [a1b4810d] + Hexagons v0.1.0
 [9fb69e20] + Hiccup v0.1.1
 [a98d9a8b] + Interpolations v0.6.3
 [c8e1da08] + IterTools v0.1.0
 [e5e0dc1b] + Juno v0.3.0
 [5ab0869b] + KernelDensity v0.3.2
 [d3d80556] + LineSearches v2.2.1
 [4345ca2d] + Loess v0.3.0
 [1914dd2f] + MacroTools v0.3.7
 [442fdcdd] + Measures v0.1.0
 [e89f7d12] + Media v0.3.0
 [77ba4419] + NaNMath v0.2.6
 [d41bc354] + NLSolversBase v2.1.3
 [429524aa] + Optim v0.9.3
 [d96e819e] + Parameters v0.7.3
 [90014a1f] + PDMats v0.7.0
 [85a6dd25] + PositiveFactorizations v0.0.4
 [1fd47b50] + QuadGK v0.1.3
 [c84ed2f1] + Ratios v0.1.0
 [a9d5a852] + RealInterface v0.0.3
 [189a3867] + Reexport v0.0.3
 [79098fc4] + Rmath v0.3.0
 [ea8e919c] + SHA v0.5.1
 [9966252f] + ShowItLikeYouBuildIt v0.1.0
 [992d4aef] + Showoff v0.1.1
 [a2af1166] + SortingAlgorithms v0.1.1
 [276daf66] + SpecialFunctions v0.3.1
 [90137ffa] + StaticArrays v0.6.2
 [2913bbd2] + StatsBase v0.18.0
 [4c63d2b9] + StatsFuns v0.5.0
 [30578b45] + URIParser v0.2.0
 [efce3f68] + WoodburyMatrices v0.2.2

pkg> rm JSON
INFO: Updating project file /Users/kfischer/.julia/environments/v0.6/Project.toml
 [c91e804a] - Gadfly
 [682c06a0] - JSON
INFO: Updating manifest file /Users/kfischer/.julia/environments/v0.6/Manifest.toml
 [13072b0f] - AxisAlgorithms v0.2.0
 [9e28174c] - BinDeps v0.6.0
 [49dc2e85] - Calculus v0.2.2
 [5ae59095] - Colors v0.8.1
 [3da002f7] - ColorTypes v0.6.1
 [bbf7d656] - CommonSubexpressions v0.0.1
 [34da2185] - Compat v0.31.0
 [a81c6b42] - Compose v0.5.4
 [d38c429a] - Contour v0.4.0
 [7ad07ef1] - CoupledFields v0.0.1
 [0fe7c1db] - DataArrays v0.6.2
 [a93c6f00] - DataFrames v0.10.1
 [864edb3b] - DataStructures v0.7.0
 [c5cfe0b6] - DiffBase v0.2.0
 [b4f34e82] - Distances v0.5.0
 [31c24e10] - Distributions v0.14.2
 [fa6b7ba4] - DualNumbers v0.3.0
 [5789e2e9] - FileIO v0.5.1
 [53c48c17] - FixedPointNumbers v0.4.0
 [f6369f11] - ForwardDiff v0.5.0
 [c91e804a] - Gadfly v0.6.4
 [92fee26a] - GZip v0.3.0
 [a1b4810d] - Hexagons v0.1.0
 [9fb69e20] - Hiccup v0.1.1
 [a98d9a8b] - Interpolations v0.6.3
 [c8e1da08] - IterTools v0.1.0
 [682c06a0] - JSON v0.13.0
 [e5e0dc1b] - Juno v0.3.0
 [5ab0869b] - KernelDensity v0.3.2
 [d3d80556] - LineSearches v2.2.1
 [4345ca2d] - Loess v0.3.0
 [1914dd2f] - MacroTools v0.3.7
 [442fdcdd] - Measures v0.1.0
 [e89f7d12] - Media v0.3.0
 [77ba4419] - NaNMath v0.2.6
 [d41bc354] - NLSolversBase v2.1.3
 [429524aa] - Optim v0.9.3
 [d96e819e] - Parameters v0.7.3
 [90014a1f] - PDMats v0.7.0
 [85a6dd25] - PositiveFactorizations v0.0.4
 [1fd47b50] - QuadGK v0.1.3
 [c84ed2f1] - Ratios v0.1.0
 [a9d5a852] - RealInterface v0.0.3
 [189a3867] - Reexport v0.0.3
 [79098fc4] - Rmath v0.3.0
 [ea8e919c] - SHA v0.5.1
 [9966252f] - ShowItLikeYouBuildIt v0.1.0
 [992d4aef] - Showoff v0.1.1
 [a2af1166] - SortingAlgorithms v0.1.1
 [276daf66] - SpecialFunctions v0.3.1
 [90137ffa] - StaticArrays v0.6.2
 [2913bbd2] - StatsBase v0.18.0
 [4c63d2b9] - StatsFuns v0.5.0
 [30578b45] - URIParser v0.2.0
 [efce3f68] - WoodburyMatrices v0.2.2

I guess I'd have expected Gadfly to stick around by default. I could see there being some syntax (rm -r?) to remove the reverse dependencies.

See discussion here: #5. Want some way to provide the ability to download and use artifacts in projects without needing to check them into the project git repo. The concept is adding the concept of content-addressed artifacts to Pkg3, registering them, and then allowing projects to declare that they need them.

As discussed in https://discourse.julialang.org/t/package-naming-guidelines/7140/48

Having a keyword tagging system for packages would really help with discoverability of packages and being able to sort them, to be able to find all the similar packages that belong into the same category.

Another idea is to have keyword ID numbers, that package developers could include in their REQUIRE file.

Then to each number you can associate a keyword category, such as “metaprogramming” or “numerical methods”. The advantage of having a numbering system is that the names of the categories can easily be modified, updated, and changed, without requiring the package developers to change their REQUIRE files.

So basically, the ID numbers will be invariant, but you can still easily change the names of the categories then, so that as the community evolves it is easier to also make the discovery and categorization evolve in time.

With thousands of packages avaialbale, it doesn’t make sense to have a single huge list of packages, it needs to be filterable and browsable using kewyord categories, and using ID numbers to specify them gives the extra flexibility of allowing the category names to be updated as things evolve.

Otherwise, instead of having the ID numbers in the REQUIRE file, they could be required as part of a Pull-Request to METADATA, so that they keyword category ID’s can be approved and agreed by the community.

you could use tags and keywords to group and organize packages, instead of forcing related packages to be grouped into a single package. Then you can immediately tell which domain a package belongs to and that helps more than anything with understanding its relevance to a user browsing for packages.

Having a tagging keyword system is more flexible than grouping packages into a single package, because the package you want to re-group might actually be relevant to multiple keyword domains, and organizing it as part of a single meta package will be less clear than having it organized into multiple keyword categories.

So I’d advocate for having keyword tags, then any package related to that can add a tag ID to it’s listing on the Julia package listing. This is more flexible and allows evolving changes in future

This is a feature I'd like to create a pull-request for in the future; however, I will post it as an issue for now since during the next few weeks I am going to be very busy studying for examinations in pure math and writing some papers. If somebody else has time for implementing this for 0.7 or 1.0, please go ahead, otherwise I will eventually look into trying this out on my own branch when I actually have time for something like this, but it's not my main priority right now.

Please let me know your comments if this is a good idea, or why it's not a good idea.

0.6.2 no Pkg3

julia> @time using JuliaDB
INFO: Precompiling module JuliaDB.
 58.453762 seconds (1.10 M allocations: 60.739 MiB, 0.46% gc time)

0.6.2 Pkg3

julia> @time using JuliaDB
INFO: Recompiling stale cache file /Users/kristoffer/.julia/lib/v0.6/JuliaDB.ji for module JuliaDB.
239.210437 seconds (7.44 M allocations: 376.640 MiB, 0.11% gc time)

This unfortunately makes using Pkg3 not very pleasant.

Since Pkg is going away, something need to be done with Pkg.test.

There are a few options:

• Just move it to Pkg3. Easiest, and I haven't seen that many complaints about the testing. Of course people want to be able to run it in parallel and do more fine grained stuff but it is not a major source of complaints.
• Replace it with a new, better interface that can run stuff in parallel, have options etc.

Perhaps the simplest way forward is just to move Pkg.test here and then let external testing packages develop, and if/when one of them mature to the point that it is clearly better, consider it for inclusion in Base.