denibertovic / docker-hs Goto Github PK

I haven't investigated this thoroughly, but it appears that the memorySwap configuration documented here:

https://hackage.haskell.org/package/docker-0.6.0.6/docs/Docker-Client-Types.html#t:ContainerResources

seems to be ignored by docker, and when inspecting the running container it always has:

"MemorySwap": -1,

I will try to investigate further, but I am posting this here in the meantime in case anyone else has any more information.

Thank you

@jprider63 Regarding: commercialhaskell/stackage#3204

I'm fine with breaking compatibility to be honest. What do you think?

We should look into adding streaming support for certain endpoints. Currently only the logs endpoint and the build endpoint come to mind.

The way reading logs is implemented now is not that great as we consume the entire response at once
rather than chunk by chunk. For large log outputs this can be a problem.

This even more important for the build endpoint as we're sending a tar.gz of the build context which
might end up being very large.

This will likely need a revamp of the HttpHandler type and making use of http-conduit.
I'm not yet sure how to best accomplish this especially since we only need streaming
for a few endpoints (it probably wouldn't work for all endpoints actually).

Hi,

docker will not presently build as a dependency in projects using LTS-15.0, but there is an easy fix.

Line 1567 of src/Docker/Client/Types.hs, function parseIntegerText, just add MonadFail m to signature:

parseIntegerText :: (MonadFail m, Monad m) => Text -> m Integer
parseIntegerText t = case readMaybe $ T.unpack t of
    Nothing ->
        fail "Could not parse Integer"
    Just i ->
        return i

thanks for the great work,
Stephen.

Hi, thank you for this module!
I'd like to ask, would it be difficult to make it work with the latest stack resolver (lts-9)?
Could I do that and make a contribution?

I think the use of Either DockerError x as the return type of all the Docker.Client.API.* functions makes the API somewhat cumbersome to use, since it requires pattern-matching on each return value before using it.

Instead I would suggest throwing an exception whenever a DockerError is returned in the current version. I can volunteer to implement this change, but wanted to check if there was another reason for the current design decision that I'm missing.

2 PRs should not have been closed. See here for explanation: #27

Cherry pick #22 and #25 back into master (with the requested changes fixed).

By luck (or not), we have seen this:

DockerClientDecodeError "Error in $[1].State: Unknown Status: String \"removing\""

This was thrown while the client requested the state of a container that was being removed.

The non-streaming log API doesn't return a normal bytestring. It returns the string encoded in some way that I couldn't manage to google.

This is the only place I could find how to decode it: https://github.com/mafintosh/docker-raw-stream

Would be nice to implement archive endpoint.

https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/get-an-archive-of-a-filesystem-resource-in-a-container
https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/extract-an-archive-of-files-or-folders-to-a-directory-in-a-container

PS I'd like to try implement it myself, but possible need little guiding/hinting

hello
I get 2 errors inspecting containers

• [WARN] DockerClientDecodeError "Error in $.Config.Cmd: parsing [] failed, expected Array, but encountered Null"
• [WARN] DockerClientDecodeError "Error in $.Config: key \"StopSignal\" not found"

[a.pv@ansible 18:07:21 ~] docker version
Client: Docker Engine - Community
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.9
 Git commit:        dea9396
 Built:             Thu Nov 18 00:38:53 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:37:17 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

where should I look to fix them? any hint very appreciated

I would like to add a guide how to setup the development environment for contributing to the project in CONTRIBUTING.md (which is a stub currently). Is a pull request for that welcome? Any pointers to already existing documentation?

So that we can run them on PR's.

Currently, only the HTTP status code and the standard HTTP description for this code is returned on API errors. But the Docker API also always returns some JSON with a message attribute holding a more verbose description of the error which occurred. This description would be helpful for debugging and should also be returned in the DockerError structure.

getting this when trying to list containers with listContainers defaultListOpts

it lists images however just fine with same setup but listContainers substituted with listImages

This is on OSX HighSierra 10.13.6
Docker for OSX: Version 18.06.0-ce-mac69 (26398)

this is what curling the endpoint looks like if this helps:

$ curl -v 127.0.0.1:2375/v1.24/containers/json
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 2375 (#0)
> GET /v1.24/containers/json HTTP/1.1
> Host: 127.0.0.1:2375
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Api-Version: 1.38
< Content-Type: application/json
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/18.06.0-ce (linux)
< Date: Wed, 08 Aug 2018 15:21:50 GMT
< Transfer-Encoding: chunked
<
[{"Id":"af124ec80359ea194f4dbf471af8b9fb9fa24eeb6383ef6a051a294c166a048b","Names":["component-name"],"Image":"...","ImageID":"sha256:829299b6ededf742f12557fb5b6a7d8c0f8728c9b7f99df277538a8db2e33806","Command": ...

so the Id is clearly in the json object returned

any other way to debug this?

inspectContainer is failing with

Left (DockerClientDecodeError "Error in $.NetworkSettings.Ports: parsing [] failed, expected Array, but encountered Null")

Another call createContainer seems to work fine under the same DockerT context.

• Documentation (we need a little more haddock and some basic usage/tutorial in the README)
• Review dependencies in the cabal file. I've been adding all sorts of things there while playing around and I'm not sure all of it is needed.
• Set version bounds in the cabal file.
• Add more tests (most importantly tests that check that we properly serialize/de-serialize json)
• Move to Docker.Client namespace (to follow more closely how the other libraries work).
• Add caching of stack builds for Travis CI (to speed up tests)
• Bump License to current year.

We need an integration test for each endpoint we're targeting (to be sure it actually works and doesn't break on different docker/api versions).

Hi, I have pushed this error to the repository here at branch lc/invalid_reference_format. Clone recursively.

The code is in the Shakefile, use stack exec shake dockers to observe the error. The getDockerVersion call works, whilst the buildImageFromDockerfile returns Right () but does not execute a build and in the log I see

ERRO[2017-10-24T14:40:40.606285141+01:00] Handler for POST /v1.24/build returned error: invalid reference format

Since we're targeting docker 1.12 and Travis has not yet upgraded to that version we're installing it manually (see travis.yml). The issue is that the package name (and apt repo for that matter) has to reference the correct Ubuntu version but Travis is apparently switching between Precise and Trusty randomly on each job.

We needz docs.

We should add this package to stackage. @denibertovic, should you and I be co-maintainers?

I was getting the following error while attempting to connect with ssl:

TlsExceptionHostPort (HandshakeFailed (Error_Misc "unimplemented RSA signature hash type: HashSHA384"))

It looks like support for HashSHA384 was added in tls-1.3.7. I'm currently updating to stackage lts-5.18, so hopefully that takes care of it.

This issue is mostly for future reference in case someone else runs into this problem.

Should I submit a PR to add an API method for renaming docker container.

Hey @jprider63 @kseo just wanted to let you know I've deleted the dev branch. I don't think we need the "git-flow" dev branch in this repo since it complicates thing and imo is overkill. From here on we're just going to treat master as the main branch for everything that's coming in (All PR's should be based on master and committed back to it.)

We will continue to use tags for new releases but HEAD of master should always be stable and working.

Since I've deleted the dev branch 2 of the open PR's have been closed automatically :(. @kseo these were yours. I'm going to cherry pick those into master once I make time (they had some changes requested so I first have to make those changes... Feel free to re-open a PR to master if you beat me to it.)

Environment

• Docker CE 17.09.0-ce-mac35 (19611)

Example code

module Main where

import           Control.Monad.IO.Class
import           Docker.Client

main :: IO ()
main = do
  http <- unixHttpHandler "/var/run/docker.sock"
  runDockerT (defaultClientOpts, http) $ do
    containers <- listContainers defaultListOpts
    liftIO $ print containers

Results in:

Left (DockerClientDecodeError "Error in $[0].Id: ContainerID is not an object.")

From what I can tell, the JSON decoder expects the Container JSON to look something like:

{ "Id": {"Id": "…"}, }

but it's just:

{"Id": "…"}

Relevant lines:

• Types.hs:532
• Types.hs:377

@jprider63 Since we're actually parsing State from the API and not Status (which carries more information) should we rename our data type accordingly?

Decoding the json response for inspecting a container fails. The following:

{"Id":"baf92b44405fcc5dae3f4ac1abf85125167ec73f74d7cf1e357e266518609434","Created":"2017-06-09T15:17:59.126028918Z","Path":"/usr/sbin/sshd","Args":["-D"],"State":{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":6589,"ExitCode":0,"Error":"","StartedAt":"2017-06-09T15:17:59.911539388Z","FinishedAt":"0001-01-01T00:00:00Z"},"Image":"sha256:7fb322e32bf5daf2679d3e7ce06dc790c3876eef99365303b4bcdae636e53063","ResolvConfPath":"/home/bibifi/docker/containers/baf92b44405fcc5dae3f4ac1abf85125167ec73f74d7cf1e357e266518609434/resolv.conf","HostnamePath":"/home/bibifi/docker/containers/baf92b44405fcc5dae3f4ac1abf85125167ec73f74d7cf1e357e266518609434/hostname","HostsPath":"/home/bibifi/docker/containers/baf92b44405fcc5dae3f4ac1abf85125167ec73f74d7cf1e357e266518609434/hosts","LogPath":"/home/bibifi/docker/containers/baf92b44405fcc5dae3f4ac1abf85125167ec73f74d7cf1e357e266518609434/baf92b44405fcc5dae3f4ac1abf85125167ec73f74d7cf1e357e266518609434-json.log","Name":"/agitated_engelbart","RestartCount":0,"Driver":"aufs","MountLabel":"","ProcessLabel":"","AppArmorProfile":"","ExecIDs":null,"HostConfig":{"Binds":[],"ContainerIDFile":"","LogConfig":{"Type":"json-file","Config":{}},"NetworkMode":"bridge","PortBindings":{},"RestartPolicy":{"Name":"no","MaximumRetryCount":0},"AutoRemove":false,"VolumeDriver":"","VolumesFrom":[],"CapAdd":[],"CapDrop":[],"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":[],"GroupAdd":null,"IpcMode":"","Cgroup":"","Links":null,"OomScoreAdj":0,"PidMode":"","Privileged":false,"PublishAllPorts":true,"ReadonlyRootfs":false,"SecurityOpt":[],"UTSMode":"","UsernsMode":"","ShmSize":67108864,"Runtime":"runc","ConsoleSize":[0,0],"Isolation":"","CpuShares":2,"Memory":0,"NanoCpus":0,"CgroupParent":"","BlkioWeight":0,"BlkioWeightDevice":null,"BlkioDeviceReadBps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteIOps":null,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DiskQuota":0,"KernelMemory":0,"MemoryReservation":0,"MemorySwap":-1,"MemorySwappiness":-1,"OomKillDisable":null,"PidsLimit":0,"Ulimits":[],"CpuCount":0,"CpuPercent":0,"IOMaximumIOps":0,"IOMaximumBandwidth":0},"GraphDriver":{"Name":"aufs","Data":null},"Mounts":[],"Config":{"Hostname":"baf92b44405f","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"22/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","DEBIAN_FRONTEND=noninteractive","NOTVISIBLE=in users profile"],"Cmd":["/usr/sbin/sshd","-D"],"ArgsEscaped":true,"Image":"7fb322e32bf5","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{},"StopSignal":"SIGTERM"},"NetworkSettings":{"Bridge":"","SandboxID":"42945cbaa4be999091747764404492fef845ae2c2ea41b54622d488327e3abb4","HairpinMode":false,"LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"Ports":{"22/tcp":[{"HostIp":"0.0.0.0","HostPort":"32838"}]},"SandboxKey":"/var/run/docker/netns/42945cbaa4be","SecondaryIPAddresses":null,"SecondaryIPv6Addresses":null,"EndpointID":"2b6fe96c93413ce0a690966f5dd4eaa88b5d8d402d467418bd0156480a099d1d","Gateway":"172.17.0.1","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"172.17.0.5","IPPrefixLen":16,"IPv6Gateway":"","MacAddress":"02:42:ac:11:00:05","Networks":{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"NetworkID":"adb7ae95705b9fc9e2661586146e4a79cef31ba0dcbcdf684d105ec09619da77","EndpointID":"2b6fe96c93413ce0a690966f5dd4eaa88b5d8d402d467418bd0156480a099d1d","Gateway":"172.17.0.1","IPAddress":"172.17.0.5","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:ac:11:00:05"}}}}

leads to the following error: Error in $.HostConfig: failed to parse field HostConfig: expected Bool, encountered Null. I think this is due to OomKillDisable being null.

Both defaultHttpHandler and unixHttpHandler use defaultManagerSettings for which responseTimeout = ResponseTimeoutDefault. This just came to bite me when I used the waitContainer API with a non-trivial container task. It seems that the /wait API does not send a response until the container has stopped, so using ResponseTimeoutDefault does not make much sense here. I am wondering whether:

• It would make sense to document this and add a suitable warning to the documentation for waitContainer?
• Modifying the request made in waitContainer to override the responseTimeout to set it to ResponseTimeoutNone?

I try to start or stop a container with empty name, it says ok:

> runDockerT (defaultClientOpts, h) $ stopContainer DefaultTimeout  $ fromJust $ toContainerID $ T.pack ""
Right ()

I also get Right () when I supply container id that does not exist on my host.

Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.9.1
 Git commit:   092cba3
 Built:        Sat Oct 14 16:54:40 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.9.1
 Git commit:   092cba3
 Built:        Sat Oct 14 16:54:40 2017
 OS/Arch:      linux/amd64
 Experimental: false

First of all, just wanted to say thanks for putting this library together. I've been using it for a few weeks and it works great. I'm primarily using it to programmatically build, tag and push docker images on an Amazon EC2 instance to a private ECR registry.

I've added a few pieces of functionality in my fork that I'd be happy to merge back in but I wanted to run the implementations by you first.

Easy stuff:

1. Tagging images.

tagImage :: forall m. MonadUnliftIO m => T.Text -> T.Text -> Maybe Tag -> DockerT m (Either DockerError ())
tagImage name repo maybeTag = requestUnit POST (TagImageEndpoint name repo maybeTag)

2. Pushing images.

pushImage :: forall m. MonadUnliftIO m => T.Text ->  Maybe Tag -> DockerT m (Either DockerError ())
pushImage name maybeTag = requestUnit POST (PushImageEndpoint name maybeTag)

Medium Stuff

1. MonadUnliftIO. How would you feel about switching the constraints on the Api.hs functions from (MonadIO m, MonadMask m) to just MonadUnliftIO m? I noticed you have a #if MIN_VERSION_http_conduit(2,3,0) check in Http.hs for backwards compatibility with older snapshots.

Bigger Stuff

1. Authentication. I added X-REGISTRY-CONFIG for building images and X-REGISTRY-AUTH for tagging and pushing images.
2. Credentials helpers. AWS and other platforms have different methods for doing docker authentication that are a little more complex than just the docker username and password. It seems like the preferred method for accomplishing this is described here. The gist of it is that you have a section in your ~/.docker/config.json which specifies a binary to run to get registry creds. The API that I'm thinking of for accessing these would let the user specify how to do auth in DockerClientOpts:

data DockerAuthStrategy = NoAuthStrategy
                        | ExplicitStrategy RegistryConfig
                        | DiscoverStrategy

NoAuthStrategy obviously uses no authentication. ExplicitStrategy let's you explicitly pass in a RegistryConfig mapping. DiscoverStrategy will look for a credsStore field in the ~/.docker/config.json file and use the binary it points to to automatically retrieve the auth parameters for each request. The DiscoverStrategy is inspired by amazonka

Let me know what you think!

CI is broken.

@denibertovic What's the status of docker socket support? I see unixHttpHandler but it doesn't look like it's being used.

My use case is that Docker for Mac exposes the docker socket, which I would like to use instead of running an additional container to get "on the network".

Building docker-hs emits -Wdeprecations warnings.

/Users/kseo/HaskellProjects/docker-hs/src/Docker/Client/Http.hs:88:26: warning: [-Wdeprecations]
    In the use of ‘parseUrl’
    (imported from Network.HTTP.Client, but defined in http-client-0.4.31.2:Network.HTTP.Client.Request):
    Deprecated: "Please use parseUrlThrow, parseRequest, or parseRequest_ instead"

/Users/kseo/HaskellProjects/docker-hs/src/Docker/Client/Http.hs:130:23: warning: [-Wdeprecations]
    In the use of ‘sClose’ (imported from Network.Socket):
    Deprecated: "use 'close'"

If there is no reason to use these deprecated APIs, I will create a PR to replace them.

This is a huge undertaking and a major breaking change but it would cover the entire api in one single swoop.

There's a TODO in the Haddocks on pullImage: https://hackage.haskell.org/package/docker-0.6.0.5/docs/Docker-Client-Api.html#v:pullImage I'd like to take a swing at the second of those, and then the first. Any thoughts/suggestions/recommendations/requirements on the implementation?