sepulworld / aptly_cli Goto Github PK
View Code? Open in Web Editor NEWAptly client
License: MIT License
Aptly client
License: MIT License
My aptly api is behind a reverse proxy that exposes it at http://example.com:8888/aptly/api. After a lot of fiddling, I was able to get aptly_cli to eventually connect to it. Unfortunately, the behavior is non intuitive because of the way the base uri is constructed:
https://github.com/sepulworld/aptly_cli/blob/master/lib/aptly_command.rb#L55
Instead of having a config file with:
:servername: example.com/aptly
:port: 8888
it was necessary to specify the config as:
:servername: example.com
:port: 8888/aptly
It seems like an extra optional configuration argument that specifies the root prefix (/aptly) or constructing the base url in a different way that recognizes example.com/aptly and inserts the port after .com would be helpful.
$ aptly-cli publish_update --forceoverwrite --gpg_batch --gpg_passphrase 'TEST' --distribution test-platform
{"Batch"=>"true", "Passphrase"=>"TEST"}
Batch should not have a string as a value. It needs to be a Boolean
{"error"=>"json: cannot unmarshal string into Go value of type bool", "meta"=>"Operation aborted"}
I'm having trouble getting file_upload
to work, and I can't find any obvious errors anywhere. Suspecting some kind of bug. Uploading a file by accessing the aptly API using curl works. Other aptly-cli commands work, such as file_list
, repo_upload
and publish_update
.
My aptly-cli server is Debian 9.6, Ruby 2.3.3, aptly_cli 0.4.1.
My aptly api server is Debian 9.6, aptly 1.3.0.
File to upload:
# ls -l aptly-test-file.txt
-rw-r--r-- 1 root root 22 Jan 8 09:33 aptly-test-file.txt
# cat aptly-test-file.txt
1234567890
abcdefghij
Uploading the file using aptly-cli causes the directory to be created on the server, but the file is not uploaded. The response also does not contain the filename as expected:
# aptly-cli file_upload --trace --upload aptly-test-file.txt --directory /testfile
opening connection to my.aptly.server:443...
opened
starting SSL for my.aptly.server:443...
SSL established
<- "POST /api/files/testfile HTTP/1.1\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nAccept: */*\r\nUser-Agent: Ruby\r\nContent-Type: multipart/form-data; boundary=-----------RubyMultipartPost\r\nContent-Length: 447\r\nAuthorization: Basic <redacted>\r\nConnection: close\r\nHost: my.aptly.server\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "Server: nginx/1.14.1\r\n"
-> "Date: Tue, 08 Jan 2019 08:42:46 GMT\r\n"
-> "Content-Type: application/json; charset=utf-8\r\n"
-> "Content-Length: 2\r\n"
-> "Connection: close\r\n"
-> "Strict-Transport-Security: max-age=15768000\r\n"
-> "\r\n"
reading 2 bytes...
-> "[]"
read 2 bytes
Conn close
My /etc/aptly-cli.conf
looks like this:
# cat /etc/aptly-cli.conf
---
:proto: https
:server: my.aptly.server
:port: 443
:username: aptlyuser
:password: <redacted>
:debug: true
Using curl to upload the same file works, and shows a significant difference in the reported Content-Length of the upload. I'm not sure exactly how this is calculated:
# curl -v -F [email protected] https://aptlyuser:<redacted>@my.aptly.server/api/files/testfile
* Trying ip.of.my.aptly.server...
* TCP_NODELAY set
* Connected to my.aptly.server (ip.of.my.aptly.server) port 443 (#0)
[..handshake lines removed..]
> POST /api/files/testfile HTTP/1.1
> Host: my.aptly.server
> Authorization: Basic <redacted>
> User-Agent: curl/7.52.1
> Accept: */*
> Content-Length: 219
> Content-Type: multipart/form-data; boundary=------------------------61a6064b8ba6fb9f
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: nginx/1.14.1
< date: Tue, 08 Jan 2019 08:46:08 GMT
< content-type: application/json; charset=utf-8
< content-length: 32
< strict-transport-security: max-age=15768000
<
* Curl_http_done: called premature == 0
* Connection #0 to host my.aptly.server left intact
["testfile/aptly-test-file.txt"]
These requests are, as you can see, all done through Nginx, but I have tried to also talk directly to the aptly API on port 8080, which gives me the exact same results.
For completeness, this is the relevant part in the Nginx config that handles requests to /api/
:
location /api/ {
client_max_body_size 100M;
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd.aptly;
proxy_redirect off;
proxy_pass http://localhost:8080/api/;
proxy_redirect http://localhost:8080/api/ /api;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header Origin "";
}
And this is the supervisor config, used to start the aptly api serve
:
[program:aptly]
command=/usr/bin/aptly api serve -no-lock
directory=/home/aptly
user=aptly
stdout_logfile=/var/log/supervisor/aptly-stdout.log
stderr_logfile=/var/log/supervisor/aptly-stderr.log
Seeing this on Ruby 1.9.1.
# aptly-cli
/usr/local/bin/aptly-cli:23:in `load': /var/lib/gems/1.9.1/gems/aptly_cli-0.1.5/bin/aptly-cli:28: invalid multibyte char (US-ASCII) (SyntaxError)
/var/lib/gems/1.9.1/gems/aptly_cli-0.1.5/bin/aptly-cli:28: invalid multibyte char (US-ASCII)
/var/lib/gems/1.9.1/gems/aptly_cli-0.1.5/bin/aptly-cli:28: syntax error, unexpected $end, expecting keyword_end
... would be created if it doesn’t exist.'
... ^
from /usr/local/bin/aptly-cli:23:in `<main>'
Solution is to add the following to the aptly-cli script just below #!/usr/bin/env ruby
#!/usr/bin/env ruby
# encoding: utf-8
I'm seeing:
/opt/ruby/2.1.2/lib/ruby/gems/2.1.0/gems/aptly_cli-0.1.6/lib/aptly_repo.rb:123:in 'repo_upload': undefined method 'empty?' for nil:NilClass (NoMethodError)
in aptly_repo.rb; code references json_response["failedFiles"], but I see FailedFiles if I run the API manually.
$ docker build -t sepulworld/aptly_api_test ./test/
Sending build context to Docker daemon 75.26 kB
Step 1 : FROM debian:jessie
---> 1b088884749b
Step 2 : EXPOSE 8080
---> Using cache
---> 059055839ed0
Step 3 : RUN echo "deb http://repo.aptly.info/ squeeze main" > /etc/apt/sources.list.d/aptly.list; apt-key adv --keyserver keys.gnupg.net --recv-keys 2A194991; apt-get update; apt-get install aptly curl xz-utils bzip2 gnupg wget graphviz -y --force-yes; wget --quiet http://mirror.as24220.net/pub/ubuntu-archive/pool/main/z/zeitgeist/zeitgeist_0.9.0-1_all.deb -O /tmp/zeitgeist_0.9.0-1_all.deb; wget --quiet http://mirror.as24220.net/pub/ubuntu-archive/pool/main/z/zsh/zsh_5.1.1-1ubuntu1_i386.deb -O /tmp/zsh_5.1.1-1ubuntu1_i386.deb
---> Running in 000ec37cb330
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.OpKMIiSwnv --no-auto-check-trustdb --trust-model always --primary-keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-squeeze-stable.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-wheezy-stable.gpg --keyserver keys.gnupg.net --recv-keys 2A194991
gpg: requesting key 2A194991 from hkp server keys.gnupg.net
?: keys.gnupg.net: Host not found
gpgkeys: HTTP fetch error 7: couldn't connect: Connection timed out
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
Err http://httpredir.debian.org jessie InRelease
Err http://repo.aptly.info squeeze InRelease
Err http://httpredir.debian.org jessie-updates InRelease
Err http://security.debian.org jessie/updates InRelease
Err http://httpredir.debian.org jessie Release.gpg
Could not resolve 'httpredir.debian.org'
Err http://security.debian.org jessie/updates Release.gpg
Could not resolve 'security.debian.org'
Err http://repo.aptly.info squeeze Release.gpg
Could not resolve 'repo.aptly.info'
Err http://httpredir.debian.org jessie-updates Release.gpg
Could not resolve 'httpredir.debian.org'
Reading package lists...
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/InRelease
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/InRelease
W: Failed to fetch http://security.debian.org/dists/jessie/updates/InRelease
W: Failed to fetch http://repo.aptly.info/dists/squeeze/InRelease
�
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/Release.gpg Could not resolve 'httpredir.debian.org'
W: Failed to fetch http://repo.aptly.info/dists/squeeze/Release.gpg Could not resolve 'repo.aptly.info'
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/Release.gpg Could not resolve 'httpredir.debian.org'
W: Failed to fetch http://security.debian.org/dists/jessie/updates/Release.gpg Could not resolve 'security.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Package bzip2 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
Package xz-utils is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Unable to locate package aptly
E: Unable to locate package curl
E: Package 'xz-utils' has no installation candidate
E: Package 'bzip2' has no installation candidate
E: Unable to locate package wget
E: Unable to locate package graphviz
/bin/sh: 1: wget: not found
/bin/sh: 1: wget: not found
The command '/bin/sh -c echo "deb http://repo.aptly.info/ squeeze main" > /etc/apt/sources.list.d/aptly.list; apt-key adv --keyserver keys.gnupg.net --recv-keys 2A194991; apt-get update; apt-get install aptly curl xz-utils bzip2 gnupg wget graphviz -y --force-yes; wget --quiet http://mirror.as24220.net/pub/ubuntu-archive/pool/main/z/zeitgeist/zeitgeist_0.9.0-1_all.deb -O /tmp/zeitgeist_0.9.0-1_all.deb; wget --quiet http://mirror.as24220.net/pub/ubuntu-archive/pool/main/z/zsh/zsh_5.1.1-1ubuntu1_i386.deb -O /tmp/zsh_5.1.1-1ubuntu1_i386.deb' returned a non-zero code: 127
Presently, CLI returns JSON received from Aptly API. It might be good to consider retuning something a bit more user friendly to read on command line.
$ aptly-cli repo_upload --dir test1 --name ssr-precise -trace
{"FailedFiles"=>[], "Report"=>{"Warnings"=>[], "Added"=>["mariadb-common_5.5.32+maria-1~precise_all added"], "Removed"=>[]}}
would look like...
$ aptly-cli repo_upload --dir test1 --name ssr-precise -trace
Added: mariadb-common_5.5.32+maria-1~precise_all
or if it failed to add...
$ aptly-cli repo_upload --dir test1 --name ssr-precise -trace
FailedFiles: mariadb-common_5.5.32+maria-1~precise_all
Basically,
puts ["FailedFiles"] if not empty.
puts ["Report"]["Warnings"] if not empty
puts ["Report"]["Added"] if not empty
puts ["Report"]["Removed"] if not empty
From aptly.info api docs...
CREATE SNAPSHOT FROM PACKAGE REFS
POST /api/snapshots
Create snapshot from list of package references.
This API creates snapshot out of any list of package references. Package references could be obtained from other snapshots, local repos or mirrors.
Hi,
Is it possible to get an exit code different than 0 when aplty_cli command failing?
Cheers,
Vasilis
Bug with aptly-cli snapshot_list
$ aptly-cli snapshot_list --sort all --trace
/Users/zanew/.rvm/gems/ruby-2.2.1/gems/aptly_cli-0.1.8/lib/aptly_snapshot.rb:25:in `snapshot_list': wrong number of arguments (2 for 0..1) (ArgumentError)
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/aptly_cli-0.1.8/bin/aptly-cli:314:in `block (2 levels) in <top (required)>'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/command.rb:178:in `call'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/command.rb:153:in `run'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/runner.rb:428:in `run_active_command'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/runner.rb:68:in `run!'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/delegates.rb:15:in `run!'
from /Users/zanew/.rvm/gems/ruby-2.2.1/gems/commander-4.3.5/lib/commander/import.rb:5:in `block in <top (required)>'
Looks like global options in aptly_command.rb are not working presently.
debug_output $stdout
L66 in aptly_command.rb doesn't appear to respond and even if you set debug_output $stdout
directly in class AptlyCommand it isn't used when running http queries in let's say aptly_repo
$ aptly-cli repo_list --debug --no-config
{"Name"=>"testrepo", "Comment"=>"", "DefaultDistribution"=>"", "DefaultComponent"=>"main"}
{"Name"=>"testrepocreate", "Comment"=>"testing repo creation", "DefaultDistribution"=>"precisecreatetest", "DefaultComponent"=>""}
{"Name"=>"testrepotoshow", "Comment"=>"testing repo show", "DefaultDistribution"=>"preciseshowtest", "DefaultComponent"=>""}
{"Name"=>"testrepo20", "Comment"=>"", "DefaultDistribution"=>"", "DefaultComponent"=>"main"}
{"Name"=>"testrepotoquery", "Comment"=>"testing repo query with name", "DefaultDistribution"=>"precisequerytest", "DefaultComponent"=>""}
{"Name"=>"testrepoedit", "Comment"=>"testing repo edit distro name", "DefaultDistribution"=>"preciseeditdistnew", "DefaultComponent"=>""}
The above command should include HTTP debug output from HTTParty.
Move from httmultiparty to httparty
httparty now supports multipart file uploads.
It would leak password.
I suggest that aptly-cli tries to load:
$ aptly-cli publish_update --distribution precise --forceoverwrite true
{"error"=>"json: cannot unmarshal string into Go value of type bool", "meta"=>"Operation aborted"}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.