commercialhaskell / stack Goto Github PK

To be taken from stackage-build which already has tried and tested code for this. We already have the basic Stackage.Package module which can read a package description:

-- | Anything to do with a package associated with some directory.
module Stackage.Package where
data PackageException
data Package = Package
  {packageName :: !PackageName
  ,packageVersion :: !PackageVersion
  ,packageDir :: !FilePath
  ,packageFiles :: !(Set FilePath)
  ,packageDependencies :: !(Map PackageName VersionRange)
  ,packageTools :: !(Set Tool)
  ,packageFlags :: !(Map Text Bool)}
data BuildResult
data PackageConfig = PackageConfig
  { packageConfigEnableTests :: !Bool
  , packageConfigEnableBenchmarks :: !Bool
  , packageConfigFlags :: !Map Text Bool
  }

In particular, we need these functions garnered from the Shake-based code in stackage-build:

readPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
            => PackageConfig -> Loc Absolute File -> m Package
buildPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
             => Package -> m BuildResult
testPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
            => Package -> m TestResult
copyPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
            => Package -> m ()
registerPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
                => Package -> m ()

Currently, if a dependency fails to build, there is no way to read the log file for it.

If GPG verification is turned on and updating the index fails, we should check the GPG keychain to see if the relevant key is present and notify the user. Either way, we should print a link to some documentation on how to securely set up GPG verification.

Putting down thoughts here before I take a break. In Slack, I wrote that I think we need to do something like:

1. Let Stack.BuildPlan figure out which package versions it thinks should be used
2. Compare that to the version ranges
3. Ensure that all of those packages are present

We need to be careful about getting this right vs using whatever's already installed, which may not work (see #38). Letting the resolver be responsible for first determining the set of package/versions we'll use, and then modifying the package database to get to that state, is likely the right thing to do.

-- | Package database handling. This includes reading the package
-- database via ghc-pkg, registering and unregistering, etc.
module Stackage.PackageDatabase where
data PackageDatabaseException
data PackageDatabaseId -- (as in foo-0.0.0-9c293923c0685761dcff6f8c3ad8f8ec)
getInstalledPackages :: (MonadLogger m,MonadIO m,MonadThrow m)
                     => Config
                     -> m (Set PackageIdentifier)
-- or, alternatively, this style seems to have more mileage in our
-- codebases.
getInstalledPackages :: (MonadLogger m,MonadIO m,MonadThrow m)
                     => Config -> m (Map PackageName PackageVersion)
unregisterPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
                  => Config -> PackageIdentifer -> m ()
getBrokenPackages :: (MonadLogger m,MonadIO m,MonadThrow m)
                  => Config -> m (Map PackageName PackageVersion)
findPackageId :: (MonadLogger m,MonadIO m,MonadThrow m)
              => Config -> PackageIdentifier -> m (Maybe PackageDatabaseId)

There is already a bit of code for this in Stackage.GhcPkg.

One thing cabal did right was support for multiple package indices (remote-repos). I think we should probably do the same thing. One thing cabal did badly was how multiple repos get coalesced from multiple config files. We can do better. Here's a proposal:

• Each repo/index gets a unique identifier (e.g., hackage.haskell.org)
• Each repo can have Git and HTTP URLs configured for it
• We follow normal overriding rules for Stack.Config
• We also support the notion that if neither a Git nor HTTP URL is specified, the repo is disabled (we could also be more explicit with disabled: true). That would allow you, for example, to disable usage of hackage on a per-project basis

We need to ensure that tarballs are downloaded to repo-specific directories (like cabal does), since hackage's foo-1.2.3 may very well be different than some other repo's foo-1.2.3.

I just ended up in a situation where I got an inconsistent package database. We need to ensure that can't happen and get a test case in place for triggering the problem (pinging @DanBurton). Here's the output I just saw:

src/Network/HTTP/Download.hs:133:40:                                                                                                                          [4/1829]
    Couldn't match type ‘Data.Attoparsec.Internal.Types.Parser
                           S.ByteString aeson-0.8.1.0:Data.Aeson.Types.Internal.Value’
                   with ‘attoparsec-0.12.1.6:Data.Attoparsec.Internal.Types.Parser
                           S.ByteString aeson-0.8.1.0:Data.Aeson.Types.Internal.Value’
    NB: ‘Data.Attoparsec.Internal.Types.Parser’
          is defined in ‘Data.Attoparsec.Internal.Types’
              in package ‘attoparsec-0.13.0.0’
        ‘attoparsec-0.12.1.6:Data.Attoparsec.Internal.Types.Parser’
          is defined in ‘Data.Attoparsec.Internal.Types’
              in package ‘attoparsec-0.12.1.6’
    Expected type: attoparsec-0.12.1.6:Data.Attoparsec.Internal.Types.Parser
                     S.ByteString aeson-0.8.1.0:Data.Aeson.Types.Internal.Value
      Actual type: attoparsec-0.13.0.0:Data.Attoparsec.ByteString.Internal.Parser
                     aeson-0.8.1.0:Data.Aeson.Types.Internal.Value
    In the first argument of ‘sinkParser’, namely ‘json'’
    In the second argument of ‘($$)’, namely ‘sinkParser json'’

I haven't tracked down exactly what the problem is yet, but somehow I'm seeing that vector-algorithms's dependency on mwc-random - which is in an executable which is off by default - is causing build failures.

-- | Fetching packages from whatever sources are specified in the
-- index.
module Stackage.FetchPackage where
data FetchException
data FetchConfig
fetchPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
             => FetchConfig
             -> Config
             -> PackageIdentifier
             -> m ()
unpackPackage :: Config -> PackageIdentifier -> m ()

There is some existing reference implementation code under Stackage.Fetch taken from stackage-build which does the necessary unpack and updates the .cabal file from the index, which is necessary because Hackage does not update .tar.gz downloads with subsequent cabal file revisions. But in this version, it will use the index you're defining in #4.

Mostly applies to Stack.Docker, so assigning to @manny-fp

Would have caught this bug @chrisdone just introduced ;)

/home/vagrant/haskell/stackage-common/dist/build/libHSstack-0.0.0.a(Config.o):(.text+0x1cb7f): undefined reference to `stackzm0zi0zi0_StackziTypesziDocker_zdwzdcshowsPrec_info'
/home/vagrant/haskell/stackage-common/dist/build/libHSstack-0.0.0.a(Config.o):(.data+0x3250): undefined reference to `stackzm0zi0zi0_StackziTypesziDocker_zdfFromJSONZLzmzgZRzuzdcparseJSON_closure'
/home/vagrant/haskell/stackage-common/dist/build/libHSstack-0.0.0.a(Config.o):(.data+0x3490): undefined reference to `stackzm0zi0zi0_StackziTypesziDocker_zdfFromJSONZLzmzgZRzuzdcparseJSON_closure'
/home/vagrant/haskell/stackage-common/dist/build/libHSstack-0.0.0.a(Config.o):(.data+0x3658): undefined reference to `stackzm0zi0zi0_StackziTypesziDocker_zdwzdcshowsPrec_closure'
/home/vagrant/haskell/stackage-common/dist/build/libHSstack-0.0.0.a(Config.o):(.text+0x2797): undefined reference to `stackzm0zi0zi0_StackziTypesziDocker_zdwzdcshowsPrec_info'
/home/vagrant/haskell/stackage-common/dist/build/libHSstack-0.0.0.a(Config.o):(.data+0x9c0): undefined reference to `stackzm0zi0zi0_StackziTypesziDocker_zdwzdcshowsPrec_closure'

FYI: I'm fixing that problem in the codebase right now

These are some stories that we should address before considering the tool ready. Assigning to myself, @chrisdone and I will need to break out smaller work items most likely.

1. We offer the stack executable for all relevant platforms
2. User downloads the executable and puts it somewhere on his/her PATH
3. User goes into a project directory and types stack build
4. stack notices: oh, you don't have a stack.config file, I'll create one with sensible defaults (e.g., newest LTS version, with a shared package database for it)
   • Whenever "stack notices" something, stack will notify the user in a friendly way.
   • When determining the default build plan for a new project, stack will check the .cabal file. If the .cabal file is incompatible with LTS, it will fall back to the dependency solver instead.
5. stack notices: oh, you don't have the right GHC version available, I'll download the bindist and install it for you
6. stack figures out the necessary dependencies, downloads/installs them, and then builds your project
7. Similar functionality happens for stack test, though it runs the test suite afterward.
8. You also have a stack ghc or stack runghc command to compile individual files. We could theoretically have it look at module headers, figure out which packages have those modules, and install them automatically too

This would just be default behavior. By modifying the config file, you could change from LTS to Nightly, change the sandboxing behavior, tell it to not use a snapshot at all (e.g., use a dependency solver), switch on some Docker image, use a systemwide GHC.

There are also some fine points we should address around modifying a package that's in a snapshot, specifically not using a shared package database in that case since it will "contaminate" the database for others. These details are likely too fine-grained for this current thread, just wanted to let everyone know I'm giving it thought.

See: https://github.com/commercialhaskell/all-cabal-metadata-tool/blob/3a2c8d94ff9ef74e3b7bda9b1ba460b5d82afabe/all-cabal-metadata-tool.hs#L18

Once stackage-common is out the door, we should switch all-cabal-metadata-tool to use it

The Config now tracks the expected GHC version. Ideally we'll be getting the correct GHC version set on the PATH automatically. However, if that doesn't happen for some reason we should give the user a meaningful error message.

As an example:

ResolveFileFailed "/tmp/stack-unpack26267/js-flot-0.8.3/" "jquery.flot.min.js" "/tmp/stack-unpack26267/js-flot-0.8.3/jquery.flot.min.js"

I think the code in Stack.Package just needs to take the data-dir into account.

As mentioned in Slack:

I'm going to make a recommendation that's pretty counter to what I'd normally say: let's try and reduce our dependency footprint, especially by getting rid of the classy-prelude dep
Extra deps will (1) be a barrier to entry for contributors, (2) hurt bootstrapping, and (3) inflate executable size

I think I described this requirement elsewhere, and it may already be implemented (in which case, we can close it). Motivation: add an additional package not found in LTS/Nightly, and/or tweak a single package. Functionality would be:

• Unpack that to somewhere like $DIR/.stack/extra-packages/foo-1.2.3
• Treat it like any other local dependency

cabal-install uses this to avoid the need to traverse the entire 00-index.tar file in some cases. We could use this for a sourcePackageIndexIdents function or something like that, and use it in findNewestVersions.

Should be replaced with usage of Stackage.BuildPlan instead

Here is the list of things that our Stackage library needs to be able to do:

• Download the package index. Options: use all-cabal-files (optionally with GPG verification), download 00-index from Hackage, use the new TUF stuff, etc. Currently there is a Hackage downloader reference implementation in Stackage.PackageIndex.
• Download an individual package source. Options: download from Hackage, download from S3. This should be careful to reject downloads that exceed the size specified in the all-cabal-files metadata about that package. There is currently a little reference implementation of this in Stackage.Fetch.
  • We should support verification of integrity via the all-cabal-files repo.
  • Optionally support verification of authenticity via the sig-archive archive.
• Determine the dependencies to install for a given set of package targets. Options: refuse to install dependencies at all, ask stackage.org (or the stackage-build-plan package) which packages to install based on an LTS or NIghtly release, use a dependency solver (Nathan's solution), ask cabal install --dry-run. There is currently an implementation of asking stackage.org in Stackage.Resolve.
• Determine the package databases to reference for packages, and the database to register new packages into. Options: global/user like normal, cabal sandboxes, something completely custom. We currently have some work done in this in stackage-cli, i.e. having snapshot-specific sandboxes. I think we can have that be the default, and when inside a Docker container, the snapshot location will come with the image, initally setup.
• Modifications to PATH/where to install binaries. Options: no mods and install in ~/.cabal/bin, use a cabal sandbox /bin/, something completely custom. In stackage-cli, packages are installed into the snapshot location under ~/.stackage/sandboxes/ghc-7.8.4/lts-2.1/bin, ../lib, etc.
• How to build packages. I don't think there's any reason right now to tweak this, as using runghc Setup ... seems like the right call. stackage-build has an implementation of this that has yet to be extracted.
• Where does the build occur. Options: the exact same place stackage-build is being run, inside a Docker container, theoretically in the future: a completely different machine that we rsync files to. In any case I think that the build process should be agnostic and whatever containment is happening should setup the environment and then run the build inside it.
• Reading and exposing package information after resolving flags properly (used by fpbuild, fpview, stackage-curator, etc.).
• Unpacking packages (and properly updating the cabal file). There's a basic reference implementation in Stackage.Fetch.
• Make sure that everything is logged, in a MonadLogger m => m, which in the usual case will probably be a runNoLoggingT but as a library can be configured to write somewhere, and when used as a program can, with -v, log to stdout.
• Ability to query the ghc-pkg database. Related to the other issue above. There are some limited wrapper functions for this in Stackage.GhcPkg.

These tasks can all be in stackage-common, but optionally can be in other packages. In any case they will come under the same module hierarchy. The fact that this library is used for commandline programs is a matter of incidence rather than design -- for example, fpbuild, fpview, stackage-curator will use these as well as stackage-sandbox and stackage-setup.

Necessary for implementing good defaulting for #18. Some tricks we can employ would be:

• Only check newest minor release in an LTS series. Theoretically check for a relevant upper bound in a config file, though it's rare to have such upper bounds mid-major release
• Look at the bounds on base to immediately eliminate build plans with incompatible GHC versions

@chrisdone I'm thinking this code would go in some kind of module like Stack.CreateConfigFile. Make sense?

After we've addressed our basic use cases, we should run through all of the commands that cabal supports and see which ones we want to port. For example, I can imagine wanting to run stack unpack.

Will be used to install GHC and other necessary tools. Will handle all OS-specific issues. Will have an API along the lines of:

setup :: (MonadIO m, MonadReader env m, HasConfig env) => Version -> m ()
isSetup :: (MonadIO m, MonadReader env m, HasConfig env) => Version -> m Bool

data Paths = Paths
    { binPath :: Path Abs Dir
    -- maybe others
    }

getPaths :: (MonadIO m, MonadReader env m, HasConfig env) => Version -> m Paths

The build plan files are massive, containing a lot of information we don't need in general in a format that's difficult to parse (YAML). We basically need to know (1) the GHC major version and (2) a list of package identifiers. Let's have a simple datatype with that information, and create a cache with that.

We can consider overcoming limitations in package registration. Two things I've found so far:

• --enable-multi-instance would be the key to having sandbox-like functionality without having to create sandboxes
• We could replace the description field with the contents of the README file

We'll need some functionality for determining the right environment variables based on the configuration. Making this issue to track any discussion needed.

-- | Handling of environment variables, such as the PATH,
-- GHC_PACKAGE_SANDBOX, etc.
module Stackage.Environment where
getEnvironmentVariables :: (MonadLogger m,MonadIO m,MonadThrow m)
                        => Config -> m (Map Text Text)
getPATH :: (MonadLogger m,MonadIO m,MonadThrow m)
        => Config -> m Text
withConfiguredEnvironment :: (MonadLogger m,MonadIO m,MonadThrow m)
                          => Config -> (m a -> m a) -> m a

This will avoid the (incredibly annoying) cabal bugs of letting the user package database leak into the build

• Running a command from a subdirectory should find the main config file
• Referring to packages via ../directory should work
• Possible to set the project's stack.yaml via an environment variable (STACK_YAML?)

Then we need to have stackage-curator and stackage-server depend on stack, which makes perfect sense.

Consider a project with a stack.yaml at $DIR/stack.yaml. I propose the following rules:

• If using an LTS or Nightly, and none of the packages in the stack.yaml file shadow a package in the LTS or Nightly:
  • install dependencies into ~/.stack/shared-database/lts-X.Y or something like that
  • install local packages to $DIR/.stack-database
• Otherwise, put everything in $DIR/.stack-database

We can short term just use $DIR/.stack-database

In particular: git output and the output from Stack.Setup is pretty noisy. Perhaps we should by default stream to a log file, with an option/config value to send to stdout.

Possibly: config setting to say "use whatever GHC is on the path." Also possible: config setting giving path to various GHC installations that can be used.

One annoyance of dealing with cabal is having to remember to update the package index. Should we be more intelligent? Since we're defaulting to LTS releases, we may have an easy option here:

1. If package index doesn't exist, update it (obviously, it's a cabal stupidity that it doesn't do that)
2. If the index exists, but is missing some package/version combinations that exist in the snapshot we're using, update
3. Otherwise: no update

We should still provide a stack update command to explicitly update, which will be useful for cases like stack unpack and the dependency solver.

Thoughts?

We need based on Stackage.Config to download a package index. That means:

• Downloading from our all-cabal-hashes.
• Putting it somewhere sensible e.g. ~/.stackage/index.
• Verifying the git commit via gpg?

-- | Downloading, maintaining and querying the package index from
-- various sources.
module Stackage.PackageIndex where
data IndexException
updateIndex :: (MonadLogger m,MonadIO m,MonadThrow m)
            => Config -> m ()
getPkgVersions :: MonadIO m
               => PackageName -> m (Maybe (Set Version))
isIndexOutdated :: (MonadLogger m,MonadIO m,MonadThrow m) => m Bool

This is a rough sketch of the types. Getting versions available for a package is useful for the dependency resolution process, updating the index needs to be done once in a while and isIndexOutdated is a way of telling the user that and also automatically updating on the first time.

Would be nice. Based on whatever code we're using to detect dependencies for the setup code, we could use that to detect what's not being used to aid in cleaning up dependencies.

We need to make sure that GPG signatures are checked on both

• package index
• individual package tarballs

It looks like this is not currently happening in Stack.Build

• Rename the repo to stack
• Rename the cabal file and module hierarchy (just a flat Stack.*?)
• Put the repo in commercialhaskell instead of fpco
• Setup a Travis job and add a Travis badge
• Release to Hackage
• Publish blog post

The codebase currently uses about 3 different approaches:

• [FlagName]
• Set FlagName
• Map FlagName Bool

The first two give a list of True flags. This is incorrect: we should set the flag by default to its default value, and then overwrite based on the parameter here.

A common user error with cabal is when alex or happy is not available. We should do better: why not install them automatically when they aren't present?

Make all naming consistent, like we have with PackageName and FlagName.

Highly related to #37. I'm going to propose an entire philosophy here of how the system works: building of dependencies is completely separate of building local packages. They get installed to different package databases, they run as two separate internal steps, etc. One important outcome here (which we need a test case for) is that dependencies can't depend on local packages. Let me give an example:

• Using an LTS snapshot that has wai-5.0.0 and warp-5.0.0
• Working on a project that depends on wai and warp
• Realize you need to use wai-5.0.1, so you add it to your list of packages (as per #41)

When you try to build this, it needs to fail. The reason being: the warp specified by LTS cannot depend on the local one we're going to build. The mechanism should be:

• Take the LTS
• Remove any packages from it that are shadowed by local packages
• Remove any packages from it that depend on packages which have been removed

The end result should be an error message along the lines of "Dependency warp depends on wai, but you have a local copy of wai. Recommended actions: either remove wai from your local packages, or add warp in as well."

We need a test suite for this.

Should help avoid confusion around typos in a config file leading to stack doing something the user doesn't expect

So we already have an implementation of this in stackage-common but under slightly different superficial arguments. We'd like it updated to match the proposal (see end of ticket). Currently,

getPackage :: FinalAction
         -> Map Text Bool
         -> Map PackageName (Map Text Bool)
         -> Loc Absolute File
         -> IO Package

the two flag structure arguments are redundant, it can be just one. And the final action is remnant from fpbuild, it's really only used to determine whether the test suite or benchmarks are enabled. So these things can be put in a PackageConfig record data type (as below). It should just be a case of updating the places that need to be updated to enforce this API.

You can this along with your stackage-curator stuff and then pass to me and I'll port stackage-build to the latest stackage-common API and update anything as necessary.

-- | Anything to do with a package associated with some directory.
module Stackage.Package where
data PackageException
data Package = Package
  {packageName :: !PackageName
  ,packageVersion :: !PackageVersion
  ,packageDir :: !FilePath
  ,packageFiles :: !(Set FilePath)
  ,packageDependencies :: !(Map PackageName VersionRange)
  ,packageTools :: !(Set Tool)
  ,packageFlags :: !(Map Text Bool)}
data BuildResult
data PackageConfig = PackageConfig
  { packageConfigEnableTests :: !Bool
  , packageConfigEnableBenchmarks :: !Bool
  , packageConfigFlags :: !Map Text Bool
  }
readPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
            => PackageConfig -> Loc Absolute File -> m Package

Assigned to @DanBurton, who will do initial design of how the test suite will work.

Basic API is fleshed out below. @DanBurton will implement enough needed for stackage-cli and then we'll review for updates later.

-- | The general Stackage configuration that starts everything off. This should
-- be smart to falback if there is no stackage.config, instead relying on 
-- whatever files are available.
--
-- If there is no stackage.config, and there is a cabal.config, we
-- read in those constraints, and if there's a cabal.sandbox.config,
-- we read any constraints from there and also find the package
-- database from there, etc. And if there's nothing, we should
-- probably default to behaving like cabal, possibly with spitting out
-- a warning that "you should run `stk init` to make things better".
module Stackage.Config where
data Config =
  Config {configPkgDbLocation :: FilePath
         ,configSandboxLocation :: FilePath
         ,configInDocker :: Bool}
data ConfigException
data Settings
getConfig :: (MonadLogger m,MonadIO m,MonadThrow m)
          => Settings -> m Config

Assigning to myself as I wrote this code and it's nontrivial.

buildPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
             => Package -> m BuildResult
testPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
            => Package -> m TestResult
copyPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
            => Package -> m ()
registerPackage :: (MonadLogger m,MonadIO m,MonadThrow m)
                => Package -> m ()

It looks like the current code base won't allow for Setup.lhs, which could be important for the use case of custom Cabal files.

Discussed in Slack: https://fpcomplete.slack.com/archives/stackage/p1431363938000114

Will subsume much of: https://github.com/fpco/stackage-build-plan/blob/master/Stackage/BuildPlan.hs