krobertson / deb-s3 Goto Github PK
View Code? Open in Web Editor NEWEasily create and manage an APT repository on S3 -- NO LONGER MAINTAINED
License: MIT License
Easily create and manage an APT repository on S3 -- NO LONGER MAINTAINED
License: MIT License
>> Retrieving existing manifests
>> Examining package file checkout-universal-checkout-web-3861-2014-07-02-22-10-10.deb_3861_amd64.deb
/usr/bin/dpkg
>> Uploading packages and new manifests to S3
-- Transferring pool/c/ch/checkout-universal-checkout-web-3861-2014-07-02-22-10-10.deb_3861_amd64.deb
/usr/lib/ruby/1.9.1/openssl/buffering.rb:317:in `syswrite': Connection reset by peer (Errno::ECONNRESET)
from /usr/lib/ruby/1.9.1/openssl/buffering.rb:317:in `do_write'
from /usr/lib/ruby/1.9.1/openssl/buffering.rb:335:in `write'
from /usr/lib/ruby/1.9.1/net/protocol.rb:199:in `write0'
from /usr/lib/ruby/1.9.1/net/protocol.rb:173:in `block in write'
from /usr/lib/ruby/1.9.1/net/protocol.rb:190:in `writing'
from /usr/lib/ruby/1.9.1/net/protocol.rb:172:in `write'
from /usr/lib/ruby/1.9.1/net/http.rb:1955:in `send_request_with_body_stream'
from /usr/lib/ruby/1.9.1/net/http.rb:1921:in `exec'
from /usr/lib/ruby/1.9.1/net/http.rb:1317:in `block in transport_request'
from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
from /usr/lib/ruby/1.9.1/net/http.rb:1293:in `request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/connection_pool.rb:340:in `request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/net_http_handler.rb:61:in `block in handle'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/connection_pool.rb:129:in `session_for'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/http/net_http_handler.rb:55:in `handle'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:252:in `block in make_sync_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:288:in `retry_server_errors'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:248:in `make_sync_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:510:in `block (2 levels) in client_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:390:in `log_client_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:476:in `block in client_request'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:372:in `return_or_raise'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/core/client.rb:475:in `client_request'
from (eval):3:in `upload_part'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/multipart_upload.rb:204:in `add_part'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:1722:in `block in write_with_multipart'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:724:in `multipart_upload'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:1721:in `write_with_multipart'
from /usr/lib/ruby/gems/1.9.1/gems/aws-sdk-1.38.0/lib/aws/s3/s3_object.rb:605:in `write'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/utils.rb:75:in `s3_store'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/manifest.rb:85:in `block in write_to_s3'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/manifest.rb:82:in `each'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/manifest.rb:82:in `write_to_s3'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:141:in `block in upload'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:140:in `each_value'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:140:in `upload'
from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /usr/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /usr/lib/ruby/gems/1.9.1/gems/deb-s3-0.6.2/bin/deb-s3:8:in `<top (required)>'
from /usr/bin/deb-s3:23:in `load'
from /usr/bin/deb-s3:23:in `<main>'
More of a question than an actual issue, but when I upload a package apt-get does not seem to find it. The repository is http://repository.homeki.com/, and I use --prefix packages
so all files are located at http://repository.homeki.com/packages. The GPG public key can be found at http://repository.homeki.com/homeki-public.key.
http://repository.homeki.com/packages/dists/unstable/Release contains the following:
Codename: unstable
Date: Wed, 10 Sep 2014 21:57:16 UTC
Architectures: all amd64 i386 armhf
Components: main
MD5Sum:
4f3cf5178b0dd03424da2fd14b71e1ae 744 main/binary-all/Packages
96e0f20dfecc7a5632874b86d9399305 519 main/binary-all/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/binary-amd64/Packages
8400fa1de14b9d32e31b1cea5c4ff83e 20 main/binary-amd64/Packages.gz
309336f2c4b3735122f113733cae3d09 593 main/binary-armhf/Packages
80e16a7549bc9a101050e40a6bae9149 439 main/binary-armhf/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/binary-i386/Packages
e9cc6202fcb07bf0c3b566b58a992687 20 main/binary-i386/Packages.gz
SHA1:
d430bd1513c071e60c0c5fe2eec74701c88b43b0 744 main/binary-all/Packages
c1fcc0929684ec545217c35631a3a044fff6db52 519 main/binary-all/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/binary-amd64/Packages
57d33b0adb18bc672fa3f1830ebba6564ee59cbd 20 main/binary-amd64/Packages.gz
67178b66271827e6bd290aaf302b6aca90016200 593 main/binary-armhf/Packages
4d8ad1038830f5ec3449d4e8d91407b918f318ed 439 main/binary-armhf/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/binary-i386/Packages
22b71702958a741fbaf314ee05c92e70640f0b08 20 main/binary-i386/Packages.gz
SHA256:
942c526090e1bfa2b85d13cbcc97c5edb45441fb1902b7afd85b8dfc7829a7b0 744 main/binary-all/Packages
3aaebeef47f6d20db4fce2121d420f41b11ed9f8b51fdf1dcb37ec71aa5a227b 519 main/binary-all/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/binary-amd64/Packages
056d24401881a17aff07d43a6ea546039a65c56ead73ce85f9074199134c2898 20 main/binary-amd64/Packages.gz
8f4c5ed1ad2d2a7c4c1c45e005120ece7e5b0e49fcfd9acfa0b87a187908ea13 593 main/binary-armhf/Packages
d5cf6598b1017217da8c9499a4a95284ff6be66ddae7ee4344f3e8781af29bcc 439 main/binary-armhf/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/binary-i386/Packages
894f125c15a2c1bd529dad459981745141e3318f3c313c92c5cba78e1672f343 20 main/binary-i386/Packages.gz
And if we follow the path into Packages
by going to http://repository.homeki.com/packages/dists/unstable/main/binary-all/Packages, we find:
Package: homeki
Version: 3.1410383081.8a4d4ac
License: unknown
Vendor: none
Architecture: all
Maintainer: Homeki Development Team <[email protected]>
Installed-Size: 24240
Depends: openjdk-7-jre-headless, postgresql, telldus, owfs, libjna-java
Section: misc
Priority: extra
Homepage: http://nourlgiven.example.com/
Filename: pool/h/ho/homeki.deb
Size: 23156866
SHA1: 4f9ab1b0fa19998422a447da6b9491da452f880f
SHA256: 996124535d21d1cc99ae3cd7c07bf13a325d2580bbda690f8592be99aa3e5f6d
MD5sum: 2de33dcdcd24b2578a0552117fabf795
Description: Unified interface for all your sensors and devices connected to your computer.
Provides a REST HTTP API for accessing, logging, modifying and managing various devices and sensors connected to your computer.
So it really seems the package is there and available. I then go ahead and add the below to /etc/apt/sources.list
, as well as add the GPG public key.
# Homeki
deb http://repository.homeki.com/packages unstable main
If I now run
sudo apt-get update
sudo apt-get install homeki
I get
E: Unable to locate package homeki
even though the sudo apt-get update
returns no error and it specified it downloaded the Release
from unstable
.
Does anyone have any idea why my package is not picked up by apt-get?
Thanks in advance!
Be nice is deb-s3 had a config file with the default bucket name, aws keys, etc.
Steps to reproduce:
sleep(60)
here: https://github.com/krobertson/deb-s3/blob/master/lib/deb/s3/cli.rb#L233deb-s3 upload -l -p -b your-bucket -v private your-package.deb
your-bucket/dists/stable/main
you'll discover a binary-
folder holding the lockfileThe fix should be as simple as getting a list of all arch types up front, but this is complicated. Reasons:
We are developing software that runs on amdhf as well as i386/amd64. It would be good if this was supported.
CodeDeploy triggers more actions when deploying a build artifact. It uploads them to S3 which means it is within the scope of the project.
If you could provide an option to use it to upload to debian packages to S3 I'd be happy to donate a 100$ to you or to a charity of your choosing.
Is there some locking when multiple deb-s3 upload debs to the same repo?
I recently ended with a corrupt repo due hash-sum errors from accidently runnning multiple deb-s3 instances.
Maybe a temporary .lock file on the s3 would be enought to defer other deb-s3.
With s3, if you want to setup an apt repo that people access via apt.example.com, you need the bucket in s3 to be called apt.example.com
however, if you do this, and then you try to do:
deb-s3 upload ---bucket="apt.example.com" examplefile.deb
it will throw an error
Retrieving existing manifests
'
/var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/core/client.rb:375:inreturn_or_raise': AWS::Errors::Base (AWS::Errors::Base) from /var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/core/client.rb:476:in
client_request'
from (eval):3:inhead_object' from /var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/s3/s3_object.rb:293:in
head'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.42.0/lib/aws/s3/s3_object.rb:270:inexists?' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/utils.rb:55:in
s3_exists?'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/utils.rb:59:ins3_read' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/release.rb:23:in
retrieve'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/cli.rb:107:inupload' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in
run'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:ininvoke_command' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in
dispatch'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:instart' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.1/bin/deb-s3:8:in
<top (required)>'
from /usr/local/bin/deb-s3:23:inload' from /usr/local/bin/deb-s3:23:in
The current way of checking if a file is already present in the repository works by getting the value from the etag, and comparing it to the md5 of the file.
For files uploaded in multipart, the etag doesn't contain the md5 but should contain the md5sum of the concatenated md5sums of all parts, followed by a dash and the number of parts. [1]
The AWS SDK seems to uses a default multipart size of 5MB, and seems to use multipart uploads for files larger than 5MB.
[1] http://permalink.gmane.org/gmane.comp.file-systems.s3.s3tools/583
Hi,
I get this when I try to upload a package to my empty bucket.
On the blogpost it says that if it's the first package, all the necessary files will be created. Is this accurate? Or does a repository structure need be created before hand (e.g. via reprepro)?
deb-s3 puts arch-independent (*_all.deb) files in a binary-all
when they should, AFAICT, be placed in the manifests for both binary-amd64 and binary-i386.
I'm working around this for now by setting the arch flag (-a amd64
).
deb-s3 upload currently bottlenecks our CI builds while uploading our 2.1 GB of packages to S3. (Yes, we're working on reducing that.)
It can't be run concurrently (uploading manifests clobber each other), so we've throttled the uploads to go one at a time. However, at 10 minutes per upload, that caps us to 6 builds per hour. Add in 40 developers, and Jenkins routinely gets backed up like whoa.
It sure would be nice if we could parallelize the 2.1 GB uploads to /pool (which should be safe for unique package versions), but single-track the mere MB of manifest uploads to /dists!
Here's a proof-of-concept that seems to be working well for us: https://github.com/htmldoug/deb-s3/commit/24338da032b77c63e5617d49c882f4b8bdb68631. Running deb-s3 upload --manifest_path /workspace
is safe to be done concurrently and uploads /pool
, but dumps the manifest additions to /workspace/dists/...
instead of S3. A second deb-s3 append_manifest /workspace
call can be run when under mutual exclusion to pull the fresh manifest and add the updates. That drops our bottleneck to 3 minutes which is a nice throughput increase from 6/hr to 20/hr.
No pull request yet because my ruby's rusty, I'm time crunched, and my changes have probably introduced bugs in the features we don't use (particularly --arch all
).
I may find some time later to spruce this up to your standards if you're interested in adopting our approach. Or if you can come up with something even better, we'll switch to that!
I created a PPA with deb-s3, and it worked very well. However, when users try to install the package hosted in the PPA, they are asked if they want to install without verification. How can I get rid of this message?
$ sudo add-apt-repository 'deb http://mslinn-ppa.s3.amazonaws.com stable main' $ sudo apt-get update $ sudo apt-get install pound Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: pound 0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded. Need to get 105 kB of archives. After this operation, 239 kB of additional disk space will be used. WARNING: The following packages cannot be authenticated! pound Install these packages without verification? [y/N] y E: Some packages could not be authenticated
Is it a goal of this project to be able to initialise a new repository up on S3, or is that still the job of reprepro (or something else?)
Thank you
First, this is a super-handy gem, thank you 👍
I'm interested in the preserve-versions option offered but have not been able to make the command run when including this flag. I tried all kinds of yes/true/no-arg formatting until looking at the code and seeing that it seemed to indeed function as a flag (at least, since changing how it worked in 73a1e45, after Pull Request #1 got merged)
deb-s3 upload my-package_0.8_amd64.deb --bucket=my-packages.skyebook.net --visibility=authenticated --arch=amd64 --access-key=foo --secret-key=bar --preserve-versions=yes
deb-s3 upload requires at least 1 argument: "deb-s3 upload FILE -b, --bucket=BUCKET".
This is the consistent result of putting the --preserve-versions (or -p) flag anywhere in the order of commands. Removing it lets uploads proceed as normally.
I apologize for not being able to bring a Pull Request with me, but my level of Ruby knowledge is NaN when measured from 1-10
I use deb-s3 got the message in below. What happen about this?
Retrieving existing manifests
/var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/core/client.rb:375:in return_or_raise': AWS::S3::Errors::Forbidden (AWS::S3::Errors::Forbidden) from /var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/core/client.rb:478:in
client_request'
from (eval):3:in head_object' from /var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/s3/s3_object.rb:293:in
head'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.40.2/lib/aws/s3/s3_object.rb:270:in exists?' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/utils.rb:55:in
s3_exists?'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/utils.rb:59:in s3_read' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/release.rb:23:in
retrieve'
from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/lib/deb/s3/cli.rb:116:in upload' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in
run'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in invoke_command' from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in
dispatch'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in start' from /var/lib/gems/1.9.1/gems/deb-s3-0.6.2/bin/deb-s3:8:in
<top (required)>'
from /usr/local/bin/deb-s3:19:in load' from /usr/local/bin/deb-s3:19:in
Doesn't look like you can delete a package from the repo. Be nice to add.
It would be useful if you needed to pass --force-overwrite (or similar) to overwrite existing versions of packages.
(Alternatively, have a --no-overwrite if you disagree about the default.)
I created a bucket on eu-west-1
and tried using deb-s3
with it. Everything seemed to be doing just fine, but the bucket always remained empty.
After digging around and trying different things, it seems I must provide a different endpoint for the EU S3 endpoint.
deb-s3 upload --endpoint=s3-eu-west-1.amazonaws.com -b my_bucket my_package.deb
The weird thing is that no errors are printed out or anything.
I'm trying to get a new repo setup with a package built via FPM and when pushing into a fresh s3 bucket I'm seeing the below error. This error is from the tip of master, and I see the same issue from the most recent gem.
Ubuntu 12.04
Ruby 1.9.3
Is there something I need to do to get an initial repo setup? Are there any debug flags I can set to see what it is doing further?
root@f3ed34bc5efa:/deb-s3# bundle exec deb-s3 upload --bucket=charliek-apt --access-key-id=XXXXX --secret-access-key=XXXX jre1.7.0-45_1.0_amd64.deb
>> Retrieving existing manifests
>> Examining package file jre1.7.0-45_1.0_amd64.deb
/usr/bin/dpkg
/usr/lib/ruby/1.9.1/openssl/buffering.rb:174:in `sysread_nonblock': end of file reached (EOFError)
from /usr/lib/ruby/1.9.1/openssl/buffering.rb:174:in `read_nonblock'
from /usr/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
from /usr/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
from /usr/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
from /usr/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
from /usr/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
from /usr/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
from /usr/lib/ruby/1.9.1/net/http.rb:1293:in `request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/connection_pool.rb:330:in `request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/net_http_handler.rb:61:in `block in handle'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/connection_pool.rb:129:in `session_for'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/http/net_http_handler.rb:55:in `handle'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:244:in `block in make_sync_request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:280:in `retry_server_errors'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:240:in `make_sync_request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:502:in `block (2 levels) in client_request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:382:in `log_client_request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:468:in `block in client_request'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:364:in `return_or_raise'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/core/client.rb:467:in `client_request'
from (eval):3:in `head_object'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/s3/s3_object.rb:293:in `head'
from /var/lib/gems/1.9.1/gems/aws-sdk-1.23.0/lib/aws/s3/s3_object.rb:270:in `exists?'
from /deb-s3/lib/deb/s3/utils.rb:55:in `s3_exists?'
from /deb-s3/lib/deb/s3/utils.rb:59:in `s3_read'
from /deb-s3/lib/deb/s3/manifest.rb:22:in `retrieve'
from /deb-s3/lib/deb/s3/cli.rb:123:in `block in upload'
from /deb-s3/lib/deb/s3/cli.rb:111:in `each'
from /deb-s3/lib/deb/s3/cli.rb:111:in `upload'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /deb-s3/bin/deb-s3:8:in `<top (required)>'
from /var/lib/gems/1.9.1/bin/deb-s3:19:in `load'
from /var/lib/gems/1.9.1/bin/deb-s3:19:in `<main>'
I've got a 60MB Packages file that prevents deb-s3 from completing. We upload upload upload and never delete packages. I'm currently investigating more and will update this ticket with more details as I dig.
Hi, I'm running into an issue with deb-s3's argument parsing for the delete operation. I'm not quite sure what's going on, haven't inserted a pry binding to try to dig around yet, but I wanted to at least bring this up. Here, I'm using delete with the syntax in your readme.
Just for reference, I check the version of golang-amqp from the Packages file in my s3 bucket.
┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> curl -o- https://s3.amazonaws.com/famc-debian-packages/dists/stable/main/binary-amd64/Packages
Package: golang-amqp
Version: 1.0
License: unknown
Vendor: none
Architecture: amd64
Maintainer: Bucky Wolfe <[email protected]>
Installed-Size: 9237
Depends: libc6 (>= 2.4), dpkg-dev
Section: net
Priority: extra
Homepage: https://github.com/streadway/amqp
Filename: pool/g/go/golang-amqp_1.0_amd64.deb
Size: 1175260
SHA1: e8cbaaa12524afac390db8f50de9715413f27904
SHA256: f44ea5f945d7b505975dfb807cbf384b6c210c1999f73e8b00eb86ead3bb9ffb
MD5sum: b48ff7a9620adcba0b824c1d41789e3e
Description: streadway's amqp library for go,
Package: orc-agent
Version: 0.1
License: unknown
Vendor: none
Architecture: amd64
Maintainer: Bucky Wolfe <[email protected]>
Installed-Size: 4922
Depends: libc6 (>= 2.4), dpkg-dev
Section: net
Priority: extra
Homepage: https://github.com/Igneous/orc-agent
Filename: pool/o/or/orc-agent_0.1_amd64.deb
Size: 919738
SHA1: d71b45f3ce3836fc67e74b27ffc524467fde3419
SHA256: c5b39b6ee072067861a53f718e9398a3bf647456069140d96b53278075569979
MD5sum: 36b43f477572bd6582382f54e1b7bd0a
Description: amqp consumer, generic message handler for orchestration tasks
Trying to delete the golang-amqp package with deb-s3 delete
.
┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 delete --arch amd64 --bucket famc-debian-packages --versions 1.0 golang-amqp
ERROR: deb-s3 delete was called with no arguments
Usage: "deb-s3 delete PACKAGE".
Trying without the rbenv shim here, just in case it's doing something goofy.
┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> ~/.rbenv/versions/2.0.0-p195/bin/deb-s3 delete --arch amd64 --bucket famc-debian-packages --versions 1.0 golang-amqp
ERROR: deb-s3 delete was called with no arguments
Usage: "deb-s3 delete PACKAGE".
Trying some other deb-s3 operations.
┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 verify -b famc-debian-packages
>> Retrieving existing manifests
>> Checking for missing packages in: stable/main amd64
>> Checking for missing packages in: stable/main armel
>> Checking for missing packages in: stable/main i386
>> Checking for missing packages in: stable/main all
Successfully uploading a package.
┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 upload -b famc-debian-packages orc-agent_0.1_amd64.deb
>> Retrieving existing manifests
>> Examining package file orc-agent_0.1_amd64.deb
>> Uploading packages and new manifests to S3
-- Transferring pool/o/or/orc-agent_0.1_amd64.deb
-- Transferring dists/stable/main/binary-amd64/Packages
-- Transferring dists/stable/main/binary-amd64/Packages.gz
-- Transferring dists/stable/Release
>> Update complete.
Trying to remove the same package.
┌[igneous☮localhost]-(~/chef/orc-agent-build)
└> deb-s3 delete --arch amd64 --bucket famc-debian-packages --versions 0.1 orc-agent
ERROR: deb-s3 delete was called with no arguments
Usage: "deb-s3 delete PACKAGE".
I don't know if this is the right place for this (I'm new at managing repo's), but my machine complains when I use the repo this generated, because it can't find a Sources.gz. How do you make a Sources.gz and would this benefit from having it generate one?
Allow users to provide a profile name to use when authenticating against AWS
/var/lib/gems/1.9.1/gems/deb-s3-0.6.1/lib/deb/s3/package.rb:237:in `[]': invalid byte sequence in US-ASCII (ArgumentError)
I'm trying to upload the latest nodejs deb in my s3 repo.
In the Maintainer field there are some utf8 chars.
Less output:
nodejs_0.10.28-1chl1~quantal1_i386.deb:
new debian package, version 2.0.
size 5619768 bytes: control archive=34984 bytes.
25 bytes, 1 lines conffiles
861 bytes, 22 lines control
122444 bytes, 1283 lines md5sums
282 bytes, 17 lines * postinst #!/bin/sh
240 bytes, 13 lines * prerm #!/bin/sh
Package: nodejs
Version: 0.10.28-1chl1~quantal1
Architecture: i386
Maintainer: J<C3><A9>r<C3><A9>my Lal <[email protected]>
Installed-Size: 16654
Depends: libc6 (>= 2.15), libgcc1 (>= 1:4.1.1), libstdc++6 (>= 4.1.1), rlwrap
Conflicts: nodejs-dev, npm
Replaces: nodejs-dev, npm
Provides: nodejs-dev, npm
Section: web
Priority: extra
Homepage: http://nodejs.org/
deb-s3 generates Packages.gz while doing deb-s3 upload
.
Suppose I am using s3 as my custom apt-repository with the package: alpha-pkg
Now, when I do sudo apt-get update
, then only the latest version of alpha-pkg comes. All the previous versions are gone. Is there any way to keep the previous versions too?
This way, I can quickly switch back to any previous version of alpha-pkg
Trying to use --gpg-options produces an error like this:
$ deb-s3 upload --bucket redacted --arch amd64 --component redacted --codename production --access-key redacted --secret-key redacted --sign=redacted --gpg-options="--no-tty" redacted_162_all.deb
!! File '--no-tty' doesn't exist
Inserting a little debug output shows that function upload is being passed an incorrect set of values for *files; is Thor incorrectly parsing the command-line?
["--no-tty", "redacted_all.deb"]
The following upload command with the --fail-if-exists and the same filename will upload some of the Packages file to s3, then exit because of the file exists. Since some of the Packages file got uploaded but not the Release file. It causes the repository index corruption:
deb-s3 upload
--fail-if-exists
--preserve-versions
--bucket ABC
--codename precise
--component main
--arch all
--sign ABC
graphicsmagick_1.3.26~ABC_all.deb
Output:
00:09:11 >> Retrieving existing manifests
00:09:11 >> Examining package file graphicsmagick_1.3.26ABC_all.debABC_all.deb
00:09:18 !! Uploading manifest failed because: file https://s3.amazonaws.com/ABC/apt/pool/g/gr/graphicsmagick_1.3.26~ABC_all.deb already exists with different contents
00:09:18 >> Uploading packages and new manifests to S3
00:09:18 -- Transferring dists/precise/main/binary-amd64/Packages
00:09:18 -- Transferring dists/precise/main/binary-amd64/Packages.gz
00:09:18 -- Transferring dists/precise/main/binary-i386/Packages
00:09:18 -- Transferring dists/precise/main/binary-i386/Packages.gz
00:09:18 -- Transferring pool/g/gr/graphicsmagick_1.3.26
Seems like one solution is to upload the graphicsmagick_1.3.26~ABC_all.deb first and if it exists and exit without uploading other Packages files.
Something that we've been needing for a while is package promotion. Being able to move it from one distribution/component to another.
This way we can build against a commit, and promote that package up to different repositories vs having to publish it to multiple repositories. Since the GPG, file management has already been done in deb-s3 it makes sense to me to add this feature here.
It's also something that would save on upload time, since you can do an internal S3 copy that doesn't leave the AWS network.
Happy to start plugging away at it, just thought I'd put it up for discussion here, in terms of how the command should look.
deb-s3 cp "package_name/path" "package_name/path" where path = distribution/component elements.
e.g.
deb-s3 cp redis2.8.1 stable/main/staging stable/main/production
I’m trying to delete a package from my S3 APT repo.
delete my_package -v authenticated -b my-apt-repo -c stage --arch amd64 --versions 0.90.1
>> Versions to delete: 0.90.1
>> Retrieving existing manifests
/var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:72:in `+': can't convert nil into String (TypeError)
from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:72:in `block in delete_package'
from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:67:in `select'
from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/manifest.rb:67:in `delete_package'
from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/lib/deb/s3/cli.rb:390:in `delete'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /var/lib/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /var/lib/gems/1.9.1/gems/deb-s3-0.7.0/bin/deb-s3:8:in `<top (required)>'
from /usr/local/bin/deb-s3:19:in `load'
from /usr/local/bin/deb-s3:19:in `<main>'
Here is what I tried. my.oregon.bucket is hosted in Oregon. I tried the upload command with and without the endpoint "--endpoint=s3-website-us-west-2.amazonaws.com". Doesn't work for me.
Then I setup a bucket in US standard (us-east-1) with the exactly setting as the one in Oregon except the region. It works.
Here's the error:
deb-s3 upload my.deb --codename=stable --component=main --bucket=my.oregon.bucket
>> Retrieving existing manifests
/Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:375:in `return_or_raise': AWS::Errors::Base (AWS::Errors::Base)
from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'
from (eval):3:in `head_object'
from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:296:in `head'
from /Library/Ruby/Gems/2.0.0/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:273:in `exists?'
from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/utils.rb:58:in `s3_exists?'
from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/utils.rb:62:in `s3_read'
from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/release.rb:30:in `retrieve'
from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/lib/deb/s3/cli.rb:136:in `upload'
from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /Library/Ruby/Gems/2.0.0/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /Library/Ruby/Gems/2.0.0/gems/deb-s3-0.7.1/bin/deb-s3:8:in `<top (required)>'
from /usr/bin/deb-s3:23:in `load'
from /usr/bin/deb-s3:23:in `<main>'
I'd like to enable unattended-upgrades, per this:
http://plusbryan.com/my-first-5-minutes-on-a-server-or-essential-security-for-linux-servers
However, to follow those instructions, the release.rb file would need to be changed to do "Origin" properly, per these instructions:
http://askubuntu.com/questions/87849/how-to-enable-silent-automatic-updates-for-any-repository
I opened it and started looking into how to do it, but I'm not a Ruby wizard and so it didn't seem obvious.
It works spectacularly, otherwise! :-)
I am trying to create private repo in S3. So starting with just basic stuff and trying to add two debs in an empty S3 bucket. Both of the debs have the same package name but different versions.
I first issue this command ...
deb-s3 upload sample-deb-app_20131113-165611-864050148_amd64.deb --preserve-versions --arch=amd64 --access-key-id=MYKEY --prefix=smalik --secret-access-key=MYSECRET --codename=precise --component=main --visibility=private --bucket=my-s3-repo
This creates the appropriate structure in my bucket ... and using apt-get and https://github.com/krobertson/apt-s3, I can even access that repo and install that package. Then I run this next command ...
deb-s3 upload sample-deb-app_20131113-165709-087212215_amd64.deb --preserve-versions --arch=amd64 --access-key-id=MYKEY --prefix=smalik --secret-access-key=MYSECRET --codename=precise --component=main --visibility=private --bucket=my-s3-repo
This ends up over writing the single package entry in Packages file that it generates. The old deb file is still present in S3 but just no entry for it in Packages file. I was expecting that the preserve-versions flag would allow me to keep the old version.
The listing in S3 look like this after the 2nd call ...
Object 1,227 Wed Nov 13 13:29:54 EST 2013 smalik/dists/precise/Release
Object 540 Wed Nov 13 13:29:54 EST 2013 smalik/dists/precise/main/binary-amd64/Packages
Object 387 Wed Nov 13 13:29:54 EST 2013 smalik/dists/precise/main/binary-amd64/Packages.gz
Object 0 Wed Nov 13 12:48:49 EST 2013 smalik/dists/precise/main/binary-i386/Packages
Object 20 Wed Nov 13 12:48:49 EST 2013 smalik/dists/precise/main/binary-i386/Packages.gz
Object 854 Wed Nov 13 12:48:49 EST 2013 smalik/pool/115/sa/sample-deb-app_20131113-165611-864050148_amd64.deb
Object 854 Wed Nov 13 13:29:54 EST 2013 smalik/pool/115/sa/sample-deb-app_20131113-165709-087212215_amd64.deb
The DEBIAN/control file in the first package was ...
Package: sample-deb-app
Version: 20131113-165611-864050148
Maintainer: smalik
Architecture: amd64
Section: main
Priority: optional
Depends: varnish
Description: Sample Debian Service Package
In the second package it was ...
Package: sample-deb-app
Version: 20131113-165709-087212215
Maintainer: smalik
Architecture: amd64
Section: main
Priority: optional
Depends: varnish
Description: Sample Debian Service Package
Any insight/help would be appreciated
In our case we are building package for 3 distributions: wheezy, precise, trusty.
Packages have same version.
When we are uploading them, everything is ok in every "package" file.. but bad things happen in pool
directory: only last file uploaded take place and every distribution other distributions point to this one...
$ ./bin/deb-s3 upload -a amd64 -p -b joel-test-xyz -v private joel_2-1_amd64.deb
>> Retrieving existing manifests
>> Examining package file joel_2-1_amd64.deb
/usr/bin/dpkg
>> Uploading packages and new manifests to S3
-- Transferring pool/j/jo/joel_2-1_amd64.deb
-- Transferring dists/stable/main/binary-amd64/Packages
-- Transferring dists/stable/main/binary-amd64/Packages.gz
-- Transferring dists/stable/main/binary-i386/Packages
-- Transferring dists/stable/main/binary-i386/Packages.gz
-- Transferring dists/stable/Release
>> Update complete.
Hi Ken,
Here is the error messge when I'm trying use deb-s3 to upload my deb.
My apt repo on s3 was initialized using reprepro, and it is functional correctly.
BTW, my desktop is running ubuntu 12.04
seems like errors when parsing the aws response, would you pls take some time to look into it, thanks in advance:-)
Examining package file mydeb.deb
'
Retrieving existing manifests
/usr/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:410:inblock in pull': Undefined prefix Net: found (REXML::UndefinedNamespaceException) from /usr/lib/ruby/1.9.1/set.rb:221:in
block in each'
from /usr/lib/ruby/1.9.1/set.rb:221:ineach_key' from /usr/lib/ruby/1.9.1/set.rb:221:in
each'
from /usr/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:408:inpull' from /usr/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:22:in
parse'
from /usr/lib/ruby/1.9.1/rexml/document.rb:243:inbuild' from /usr/lib/ruby/1.9.1/rexml/document.rb:43:in
initialize'
from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:innew' from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:in
parse'
from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:164:inxml_in' from /var/lib/gems/1.9.1/gems/xml-simple-1.1.2/lib/xmlsimple.rb:203:in
xml_in'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:64:inparse' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:55:in
initialize'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:innew' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:in
parsed'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:183:inblock in parsed' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:152:in
expirable_memoize'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:182:inparsed' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:45:in
error?'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:72:inrequest' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:88:in
get'
from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:134:invalue' from /var/lib/gems/1.9.1/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:138:in
stream'
from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/lib/deb/s3/utils.rb:47:ins3_read' from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/lib/deb/s3/release.rb:23:in
retrieve'
from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/lib/deb/s3/cli.rb:114:inupload' from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor/task.rb:27:in
run'
from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor/invocation.rb:120:ininvoke_task' from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor.rb:344:in
dispatch'
from /var/lib/gems/1.9.1/gems/thor-0.17.0/lib/thor/base.rb:434:instart' from /var/lib/gems/1.9.1/gems/deb-s3-0.2.0/bin/deb-s3:8:in
<top (required)>'
from /usr/local/bin/deb-s3:19:inload' from /usr/local/bin/deb-s3:19:in
When doing
deb-s3 upload --bucket cbox-pkg-repo-test.s3.amazonaws.com cbox-webapp_1.0.5_all.deb --access-key=mykey --secret-key=mysecret
on OSX 10.8 I get:
>> Retrieving existing manifests
>> Examining package file cbox-webapp_1.0.5_all.deb
ar: control.tar.gz: not found in archive
/usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:65:in `read': No such file or directory - /var/folders/97/1jdjqycx60xf5vcmc3l2lh740000gn/T/d20130815-37872-fxbt4o/control (Errno::ENOENT)
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:65:in `block in extract_control'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/1.9.1/tmpdir.rb:83:in `mktmpdir'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:63:in `extract_control'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/package.rb:47:in `parse_file'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:108:in `block in upload'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:106:in `each'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:106:in `upload'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /usr/local/Cellar/ruby/1.9.3-p362/lib/ruby/gems/1.9.1/gems/deb-s3-0.5.1/bin/deb-s3:8:in `<top (required)>'
from /usr/local/Cellar/ruby/1.9.3-p362/bin/deb-s3:23:in `load'
from /usr/local/Cellar/ruby/1.9.3-p362/bin/deb-s3:23:in `<main>'
Also, not sure how and if it's related, but the package does contain control.tar.gz (it's built with fpm).
This doesn't seem to happen on Ubuntu 12.04LTS.
It would be awesome if deb-s3 could use IAM roles for credentials.
Is there any reason for locking Thor version on ~> 0.18.0
?
I have conflicts with other deb in my Gemfile and I was thinking about allowing using version 0.19.
Could we change it to ~> 0.18
?
Hi,
Currently the gem depends on "aws-sdk", "~> 1.18"
is there a chance to bump it up to recent version of aws-sdk
? I could see in the commit history you did change the dependency once already but reverted it back the next day. Where there any problems with it? Does it involve a lot of code changes of deb-s3
itself? Is this planned anytime soon?
Thanks!
It used to be you could go to the url for a package, say
dists/master/debian/binary-armhf/Packages
and the browser would load the XML for you to view in the browser.
Instead, as of version 7.1, the system now causes the browser to download the file content. Is this change by design? If not, I assume the mime type got changes somewhere, if you can give a hint on where to look I can try fixing it.
Hi.
When trying to upload a package to an empty bucket, deb-s3 seems to hang.
Here's the command:
deb-s3 upload --bucket cbox-pkg-repo-test.s3.amazonaws.com cbox-webapp_1.0.5_all.deb --access-key=mykey --secret-key=mysecret
here's what is shown:
>> Retrieving existing manifests
>> Examining package file cbox-webapp_1.0.5_all.deb
>> Uploading packages and new manifests to S3
-- Transferring pool/c/cb/cbox-webapp_1.0.5_all.deb
Nothing happens after this. It seems to hang. I monitored traffic, and while initially it seems to send data to AWS, it stops after a few seconds, and hangs indefinitely.
Is there a way to further diagnose this? Or is something wrong with my command?
When I interrupt it, here's the stack trace:
^C/home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:199:in `write': Interrupt
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:199:in `write0'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:173:in `block in write'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:190:in `writing'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/protocol.rb:172:in `write'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:330:in `send_request_with_body_stream'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1922:in `exec'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1318:in `block in transport_request'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1317:in `catch'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1317:in `transport_request'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:1294:in `request'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/connection.rb:45:in `block in request'
from /home/vagrant/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/net/http.rb:746:in `start'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/connection.rb:52:in `request'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:69:in `request'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:88:in `put'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:241:in `store'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:58:in `block in s3_store'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:57:in `open'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:57:in `s3_store'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/manifest.rb:69:in `block in write_to_s3'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/manifest.rb:66:in `each'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/manifest.rb:66:in `write_to_s3'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:127:in `block in upload'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:126:in `each_value'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:126:in `upload'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/bin/deb-s3:8:in `<top (required)>'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `load'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `<main>'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `eval'
from /home/vagrant/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `<main>'
Sometimes I want to promote a package to a different component to support staging environments. So I use the components unstable, releasecandidate and stable.
Currently I have to re-upload the package in order to install them in releasecandidate or stable but it is actually already in the pool so it would be faster if we could skip this step.
I'm trying to debug a problem even though the Releases file looks ok. Any binary-all package in my repo generated by deb-s3 isn't picked up
[root@mysql1 /etc/apt]# apt-get update
Ign http://company-apt-trusty.s3.amazonaws.com stable InRelease
Hit http://company-apt-trusty.s3.amazonaws.com stable Release.gpg
Hit http://company-apt-trusty.s3.amazonaws.com stable Release
Hit http://company-apt-trusty.s3.amazonaws.com stable/main amd64 Packages
Hit http://company-apt-trusty.s3.amazonaws.com stable/main i386 Packages
Ign http://company-apt-trusty.s3.amazonaws.com stable/main Translation-en_US
Ign http://company-apt-trusty.s3.amazonaws.com stable/main Translation-en
My Releases file looks like
Mikes-MacBook-Pro:~ mike$ cat Release
Codename: stable
Date: Thu, 17 Jul 2014 02:29:05 UTC
Architectures: all amd64 i386
Components: main
MD5Sum:
96dc9836059769df3751601db3106016 22376 main/binary-all/Packages
70d286c2c73dee5efefc79806f59dc7b 6029 main/binary-all/Packages.gz
315f477e0d604b618a6aa66f69c9701a 17520 main/binary-amd64/Packages
7a21a6b4a08311743f3c9e39758c19be 4870 main/binary-amd64/Packages.gz
d41d8cd98f00b204e9800998ecf8427e 0 main/binary-i386/Packages
5216f264cc0ec4d7f87846910a9e79dc 20 main/binary-i386/Packages.gz
SHA1:
65062c1f86eb76adc0c70ced023faa94700b7dce 22376 main/binary-all/Packages
48a4e212fc2d0e9b23227b4762acc078a5eaee4c 6029 main/binary-all/Packages.gz
59d93bc40d61fbb425f45f267812a7cde78c1e4c 17520 main/binary-amd64/Packages
6c73cf9843aafe9a21e20619849c294c85b2061d 4870 main/binary-amd64/Packages.gz
da39a3ee5e6b4b0d3255bfef95601890afd80709 0 main/binary-i386/Packages
6359d0d0a7927fabc2c7407a5084e82c3e514863 20 main/binary-i386/Packages.gz
SHA256:
c49cec29c78e104a8b765d552edb4d5e5ed172f3b62b1a6144d1fcd80d27351d 22376 main/binary-all/Packages
7b7880a8edb631bfa87ad237d2f6814263486fb038d61063b7a07e585c4991bf 6029 main/binary-all/Packages.gz
1923a7b095259caff3e8c76c17c5d129cb788d10233ff74741907bd9216f3b4b 17520 main/binary-amd64/Packages
4a0839c6a3b1839c03694e0490b1cfab692b043cf52cb3009eaf69b421c79453 4870 main/binary-amd64/Packages.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 0 main/binary-i386/Packages
6e10437fdf39929b91cff0097600c7e3e76e0545a4320f518ad15c125bf7f7b6 20 main/binary-i386/Packages.gz
Also might want to add this to the help as it is a hacky way to fix my problem. I just told deb-s3 that the packages were amd64
# validate we have them
error("No architcture given and unable to determine one for #{file}. " +
"Please specify one with --arch [i386,amd64].") unless arch
That isn't in the --help at all
Feel free to close this if you think this isn't a deb-s3 issue
If the filename has a + in it which some packages use in their versions deb-s3 writes the packages wrong since it look like it switches the + with the urlencoded version
see the following
Package: salt-cloud
Version: 2014.1.7+ds-1trusty1
License: unknown
Vendor: none
Architecture: all
Maintainer: Joe Healy <[email protected]>
Installed-Size: 99
Depends: python:any, salt-common (= 2014.1.7+ds-1trusty1), sshpass
Section: admin
Priority: extra
Homepage: http://saltstack.org/
Filename: pool/s/sa/salt-cloud_2014.1.7%2Bds-1trusty1_all.deb
It was causing apt-get to 403 trying to get the file
Err http://company-apt-trusty.s3.amazonaws.com/ stable/main salt-common all 2014.1.7+ds-1trusty1
403 Forbidden
Fetched 6,041 kB in 4s (1,311 kB/s)
E: Failed to fetch http://company-apt-trusty.s3.amazonaws.com/pool/s/sa/salt-minion_2014.1.7%2Bds-1trusty1_all.deb 403 Forbidden
E: Failed to fetch http://company-apt-trusty.s3.amazonaws.com/pool/s/sa/salt-common_2014.1.7%2Bds-1trusty1_all.deb 403 Forbidden
The file on the s3 bucket had the + in there
If I renamed the file to remove the + things worked
First off...LOVE the idea of this gem.
I run an OpenStack Swift cluster that has the s3-proxy plugin which is supposed to emulate the S3 api...the only problem is, the aws-sdk gem (dependent upon here) has a hard time with non-AWS URLs.
What about native OpenStack Swift support? Of course, Swift is just another flavor of the week...
Actually, what I'm proposing is a pluggable design where any class satisfying a ObjectStore interface will work here.
I'm happy to do the abstraction, just wanting some upstream "approval" that this might be accepted.
This ticket is mostly a discussion.
At the least I think adjusting https://github.com/krobertson/deb-s3/blob/master/lib/deb/s3/cli.rb#L177 to be arch = pkg.architecture || options[:arch]
would be more correct, as long as it throws a warning.
Thoughts on this @krobertson?
APT has deprecated SHA1 signatures on repositories since version 1.2.7. Right now it's only a warning when downloading packages from a SHA1-signed repository, but starting in January 2017 it will become an error.
GPG signs using SHA1 by default, but can be configured to sign using SHA256. This should be the default on deb-s3 to avoid issues in the future. SHA256 has been supported in APT since version 0.7.7 (October 2007) so it should be safe to set as default.
To sign with SHA256 right now, the following option can be used: --gpg-options='--digest-algo SHA256'
Some of our packages have Pre-Depends
dependencies in the control file, these dependencies are unfortunately not picked up by deb-s3.
For an explanation about Pre-Depends vs Depends:
http://askubuntu.com/questions/83553/what-is-the-difference-between-dependencies-and-pre-depends
I'm trying to upload my first package to brand new bucket and I get the following error:
>> Retrieving existing manifests
/home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:406:in `block in pull_event': Undefined prefix Net: found (REXML::UndefinedNamespaceException)
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb:222:in `block in each'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb:222:in `each_key'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/set.rb:222:in `each'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:404:in `pull_event'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:183:in `pull'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/parsers/treeparser.rb:22:in `parse'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/document.rb:245:in `build'
from /home/timfall/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/rexml/document.rb:43:in `initialize'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:in `new'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:965:in `parse'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:164:in `xml_in'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/xml-simple-1.1.2/lib/xmlsimple.rb:203:in `xml_in'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:64:in `parse'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/parsing.rb:55:in `initialize'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:in `new'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:55:in `parsed'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:183:in `block in parsed'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:152:in `expirable_memoize'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/extensions.rb:182:in `parsed'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/response.rb:45:in `error?'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:72:in `request'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/base.rb:88:in `get'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:134:in `value'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/aws-s3-0.6.3/lib/aws/s3/object.rb:138:in `stream'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/utils.rb:49:in `s3_read'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/release.rb:23:in `retrieve'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/lib/deb/s3/cli.rb:102:in `upload'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/gems/deb-s3-0.5.1/bin/deb-s3:8:in `<top (required)>'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `load'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/deb-s3:23:in `<main>'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/ruby_executable_hooks:15:in `eval'
from /home/timfall/.rvm/gems/ruby-1.9.3-p448/bin/ruby_executable_hooks:15:in `<main>'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.