dnsmasq package not available in pinned apt repo about ansible-lxc-rpc HOT 24 CLOSED

Comment by claco
Wednesday Aug 20, 2014 at 17:33 GMT

Actually, it is in the repo: http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com/pool/main/d/dnsmasq/, and it's dated the same as other packages.

Some things to check. When this has happened to me in the past, it's because the OS was deployed from a source/image that is newer than what we've created the frozen repo from. For example, a net-install that fetches the latest packages from the internet, rather than a specific set in an iso.

This means, if you're using a public cloud image, or a vagrant box, they could be newer, or are the very least, not the same as our kick images will be for real installs.

The safest thing atm is to start with an image that is 14.04.0, and dist-upgrade should do the right thing. That's not to say this isn't an actual issue. Just want to make sure we're starting from a correct base b4 applying the repo.

from ansible-lxc-rpc.

Comment by johnmarkschofield
Wednesday Aug 20, 2014 at 17:36 GMT

This is using the image "14.04 LTS (Trusty Tahr) (PVHVM)" from mycloud.rackspace.com. I'm not sure if that's actually 14.04.1, but in any case, there's no other option on mycloud for a 14.04 release. (Other than the non-PVHVM image, which I'm assuming is the same base image.)

from ansible-lxc-rpc.

Comment by johnmarkschofield
Wednesday Aug 20, 2014 at 17:36 GMT

I'm also going to build a box right now and verify this, just to make sure my scripts aren't doing something to upgrade prior to changing the sources.list.

from ansible-lxc-rpc.

Comment by johnmarkschofield
Wednesday Aug 20, 2014 at 17:41 GMT

On a freshly-imaged "14.04 LTS (Trusty Tahr) (PVHVM)" server, on mycloud.rackspace.com, with no scripts or anything other than what you see here:

root@cloud-server-31:~# echo deb [arch=amd64] http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA main > /etc/apt/sources.list
root@cloud-server-31:~# cat /etc/apt/sources.list
deb [arch=amd64] http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA main
root@cloud-server-31:~# apt-get update
Get:1 http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA InRelease [1,704 B]
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA InRelease
Get:2 http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main amd64 Packages [2,116 kB]
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main Translation-en_US
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main Translation-en
Fetched 2,118 kB in 18s (114 kB/s)
Reading package lists... Done
W: GPG error: http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 443A026F1A3DF17E
root@cloud-server-31:~# curl http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com/repo.gpg | apt-key add -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3307  100  3307    0     0   5977      0 --:--:-- --:--:-- --:--:--  5969
OK
root@cloud-server-31:~# apt-get update
Get:1 http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA InRelease [1,704 B]
Hit http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main amd64 Packages
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main Translation-en_US
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main Translation-en
Fetched 1,704 B in 12s (131 B/s)
Reading package lists... Done
root@cloud-server-31:~# apt-get install  dnsmasq
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package dnsmasq is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  dnsmasq-base

E: Package 'dnsmasq' has no installation candidate
root@cloud-server-31:~#

from ansible-lxc-rpc.

Comment by johnmarkschofield
Wednesday Aug 20, 2014 at 17:49 GMT

Reproduced on vagrant Ubuntu 14.04.0:

schof@bigbertha(5458):~/code/rackspace/dnsmasq_test:
$ vagrant init ubuntu/trusty64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

  schof@bigbertha(5459):~/code/rackspace/dnsmasq_test:
$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/trusty64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Setting the name of the VM: dnsmasq_test_default_1408556760173_92871
==> default: Clearing any previously set forwarded ports...
==> default: Fixed port collision for 22 => 2222. Now on port 2201.
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2201 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2201
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => /Users/schof/code/rackspace/dnsmasq_test

  schof@bigbertha(5460):~/code/rackspace/dnsmasq_test:
$ vagrant ssh
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-29-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Wed Aug 20 17:46:17 UTC 2014

  System load:  0.15              Processes:           88
  Usage of /:   2.7% of 39.34GB   Users logged in:     0
  Memory usage: 17%               IP address for eth0: 10.0.2.15
  Swap usage:   0%

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.


vagrant@vagrant-ubuntu-trusty-64:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04 LTS
Release:    14.04
Codename:   trusty
vagrant@vagrant-ubuntu-trusty-64:~$ sudo su
root@vagrant-ubuntu-trusty-64:/home/vagrant# echo deb [arch=amd64] http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA main > /etc/apt/sources.list
root@vagrant-ubuntu-trusty-64:/home/vagrant# curl http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com/repo.gpg | apt-key add -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3307  100  3307    0     0    591      0  0:00:05  0:00:05 --:--:--   811
OK
root@vagrant-ubuntu-trusty-64:/home/vagrant# apt-get update
Get:1 http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA InRelease [1,704 B]
Get:2 http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main amd64 Packages [2,116 kB]
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main Translation-en_US
Ign http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com LA/main Translation-en
Fetched 2,118 kB in 14s (147 kB/s)
Reading package lists... Done
root@vagrant-ubuntu-trusty-64:/home/vagrant# apt-get install dnsmasq
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package dnsmasq is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  dnsmasq-base

E: Package 'dnsmasq' has no installation candidate
root@vagrant-ubuntu-trusty-64:/home/vagrant#

from ansible-lxc-rpc.

Comment by johnmarkschofield
Thursday Aug 21, 2014 at 18:52 GMT

@jacobwagner @hughsaunders Can either of you take a look at this? Or is there someone else I should ping?

from ansible-lxc-rpc.

Comment by jcannava
Friday Aug 22, 2014 at 13:28 GMT

adding @aedan @valw to take a look.

from ansible-lxc-rpc.

Comment by aedan
Friday Aug 22, 2014 at 13:32 GMT

I am looking into this issue now.

from ansible-lxc-rpc.

Comment by aedan
Friday Aug 22, 2014 at 13:47 GMT

I checked the offical trusty mirror and did not find dnsmasq either. It is apparently now called dnsmasq-base. The package dnsmasq-base is in the frozen repo. I attempted to install dnsmasq-base. After install I do have the dnsmasq daemon installed.

root@trusty-frozen:~# dnsmasq --help
Usage: dnsmasq [options]

Valid options are:
-a, --listen-address= Specify local address(es) to listen on.
-A, --address=// Return ipaddr for all hosts in specified domains.
-b, --bogus-priv Fake reverse lookups for RFC1918 private address ranges.
-B, --bogus-nxdomain= Treat ipaddr as NXDOMAIN (defeats Verisign wildcard).
-c, --cache-size= Specify the size of the cache in entries (defaults to 150).
-C, --conf-file= Specify configuration file (defaults to /etc/dnsmasq.conf).
-d, --no-daemon Do NOT fork into the background: run in debug mode.
-D, --domain-needed Do NOT forward queries with no domain part.
-e, --selfmx Return self-pointing MX records for local hosts.
-E, --expand-hosts Expand simple names in /etc/hosts with domain-suffix.
-f, --filterwin2k Don't forward spurious DNS requests from Windows hosts.
-F, --dhcp-range=,... Enable DHCP in the range given with lease duration.
-g, --group= Change to this group after startup (defaults to dip).
-G, --dhcp-host= Set address or hostname for a specified machine.
--dhcp-hostsfile= Read DHCP host specs from file.
--dhcp-optsfile= Read DHCP option specs from file.
--tag-if=tag-expression Evaluate conditional tag expression.
-h, --no-hosts Do NOT load /etc/hosts file.
-H, --addn-hosts= Specify a hosts file to be read in addition to /etc/hosts.
-i, --interface= Specify interface(s) to listen on.
-I, --except-interface= Specify interface(s) NOT to listen on.
-j, --dhcp-userclass=set:, Map DHCP user class to tag.
--dhcp-circuitid=set:,Map RFC3046 circuit-id to tag.

I believe we just need to update the playbook to reference dnsmasq-base instead of dnsmasq

from ansible-lxc-rpc.

Comment by johnmarkschofield
Friday Aug 22, 2014 at 17:49 GMT

@aedan Not sure about that. I can tell you with 100% certainty that you can install the dnsmasq package when using the Ubuntu standard repos, or our playbooks would never have worked.

I'll investigate why, and I'll test your dnsmasq-base fix.

from ansible-lxc-rpc.

Comment by aedan
Friday Aug 22, 2014 at 17:56 GMT

@johnmarkschofield I checked the package list for the mirror.rackspace.com/ubuntu trusty repo and found:

Package: dnsmasq-base
Priority: optional
Section: net
Installed-Size: 648
Maintainer: Ubuntu Developers [email protected]
Original-Maintainer: Simon Kelley [email protected]
Architecture: amd64
Source: dnsmasq
Version: 2.68-1
Replaces: dnsmasq (<< 2.63-1~)
Depends: adduser, libc6 (>= 2.15), libdbus-1-3 (>= 1.1.1), libidn11 (>= 1.13), libnetfilter-conntrack3
Breaks: dnsmasq (<< 2.63-1~)
Filename: pool/main/d/dnsmasq/dnsmasq-base_2.68-1_amd64.deb
Size: 257512
MD5sum: 60bafb9b863671bb02595505a447270c
SHA1: ab4cba860fc099fd1badb07a68afddfd4b10bb5d
SHA256: 8064b65d22724a4dca2a0cdbab8305b6a6054fac8e2b593dc1c1c2a11ff3ca24
Description: Small caching DNS proxy and DHCP/TFTP server
Description-md5: 1f9c3f0c557ca377bcc6c659e4694437
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Origin: Ubuntu
Supported: 5y
Task: virt-host, ubuntu-desktop, ubuntu-usb, kubuntu-desktop, kubuntu-full, kubuntu-active, kubuntu-active-desktop, kubuntu-active-full, kubuntu-active, edubuntu-desktop, edubuntu-usb, xubuntu-desktop, mythbuntu-frontend, mythbuntu-desktop, mythbuntu-backend-slave, mythbuntu-backend-master, lubuntu-desktop, ubuntustudio-desktop, ubuntu-gnome-desktop

So not that I don't believe you or anything. It just looks like it is being replaced by dnsmasq-base

from ansible-lxc-rpc.

Comment by johnmarkschofield
Friday Aug 22, 2014 at 20:05 GMT

mirror.rackspace.com and archive.ubuntu.com have the same dnsmasq files:

$ curl -s http://mirror.rackspace.com/ubuntu/pool/main/d/dnsmasq/ | grep ".deb" | grep "href" | sed 's/^.*a href="\([-_.a-z0-9]*\).*$/\1/g' | sort
dnsmasq-base_2.52-1_amd64.deb
dnsmasq-base_2.52-1_i386.deb
dnsmasq-base_2.52-1ubuntu0.1_amd64.deb
dnsmasq-base_2.52-1ubuntu0.1_i386.deb
dnsmasq-base_2.59-4_amd64.deb
dnsmasq-base_2.59-4_i386.deb
dnsmasq-base_2.59-4ubuntu0.1_amd64.deb
dnsmasq-base_2.59-4ubuntu0.1_i386.deb
dnsmasq-base_2.66-4ubuntu1_amd64.deb
dnsmasq-base_2.66-4ubuntu1_i386.deb
dnsmasq-base_2.68-1_amd64.deb
dnsmasq-base_2.68-1_i386.deb
dnsmasq-base_2.71-1_amd64.deb
dnsmasq-base_2.71-1_i386.deb
dnsmasq-utils_2.59-4_amd64.deb
dnsmasq-utils_2.59-4_i386.deb
dnsmasq-utils_2.59-4ubuntu0.1_amd64.deb
dnsmasq-utils_2.59-4ubuntu0.1_i386.deb
dnsmasq-utils_2.66-4ubuntu1_amd64.deb
dnsmasq-utils_2.66-4ubuntu1_i386.deb
dnsmasq-utils_2.68-1_amd64.deb
dnsmasq-utils_2.68-1_i386.deb
dnsmasq-utils_2.71-1_amd64.deb
dnsmasq-utils_2.71-1_i386.deb

$ curl -s http://archive.ubuntu.com/ubuntu/pool/main/d/dnsmasq/ | grep ".deb" | grep "href" | sed 's/^.*a href="\([-_.a-z0-9]*\).*$/\1/g' | sort
dnsmasq-base_2.52-1_amd64.deb
dnsmasq-base_2.52-1_i386.deb
dnsmasq-base_2.52-1ubuntu0.1_amd64.deb
dnsmasq-base_2.52-1ubuntu0.1_i386.deb
dnsmasq-base_2.59-4_amd64.deb
dnsmasq-base_2.59-4_i386.deb
dnsmasq-base_2.59-4ubuntu0.1_amd64.deb
dnsmasq-base_2.59-4ubuntu0.1_i386.deb
dnsmasq-base_2.66-4ubuntu1_amd64.deb
dnsmasq-base_2.66-4ubuntu1_i386.deb
dnsmasq-base_2.68-1_amd64.deb
dnsmasq-base_2.68-1_i386.deb
dnsmasq-base_2.71-1_amd64.deb
dnsmasq-base_2.71-1_i386.deb
dnsmasq-utils_2.59-4_amd64.deb
dnsmasq-utils_2.59-4_i386.deb
dnsmasq-utils_2.59-4ubuntu0.1_amd64.deb
dnsmasq-utils_2.59-4ubuntu0.1_i386.deb
dnsmasq-utils_2.66-4ubuntu1_amd64.deb
dnsmasq-utils_2.66-4ubuntu1_i386.deb
dnsmasq-utils_2.68-1_amd64.deb
dnsmasq-utils_2.68-1_i386.deb
dnsmasq-utils_2.71-1_amd64.deb
dnsmasq-utils_2.71-1_i386.deb

Verification that they have identical versions:

$ curl -s http://mirror.rackspace.com/ubuntu/pool/main/d/dnsmasq/ | grep ".deb" | grep "href" | sed 's/^.*a href="\([-_.a-z0-9]*\).*$/\1/g' | sort | md5sum
5a26ff62fd0158fd4623650046cb3746  -
root@vagrant-ubuntu-trusty-64:/home/vagrant# curl -s http://archive.ubuntu.com/ubuntu/pool/main/d/dnsmasq/ | grep ".deb" | grep "href" | sed 's/^.*a href="\([-_.a-z0-9]*\).*$/\1/g' | sort | md5sum
5a26ff62fd0158fd4623650046cb3746  -

The pinned apt repo is missing most of these files:

$ curl -s http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com/pool/main/d/dnsmasq/ | grep ".deb" | grep "href" | sed 's/^.*a href="\([-_.a-z0-9]*\).*$/\1/g' | sort
dnsmasq-base_2.68-1_amd64.deb
dnsmasq-utils_2.68-1_amd64.deb

I looked at the main/d directory to count the number of packages:

$ curl -s http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com/pool/main/d/ | grep href | wc -l
108
$ curl -s http://archive.ubuntu.com/ubuntu/pool/main/d/ | grep href | wc -l
184

We're short 76 packages in our pinned repo. (Many or all of these may be unnecessary for our install; this isn't a conclusive test.)

I'm happy to do the updating on our pinned repo myself, but I don't know how to do that.

from ansible-lxc-rpc.

Comment by aedan
Friday Aug 22, 2014 at 20:20 GMT

I would assume that it is a lot more than 76 packages missing. We only took the 64 bit packages and only the latest one at creation time. So it looks like in the case of dnsmasq there is now a 2.71 version. The goal was not to have every package, but to have the ones needed to fully deploy. If you run across any packages that are missing. Like this morning. I will add them or figure out what needs to be done to correct. It is not that we cannot have a full repo. I was just trying to reduce the size as much as possible. I believe the repo is currently 11GB in size where is could have been 68GB

Jake

from ansible-lxc-rpc.

Comment by johnmarkschofield
Friday Aug 22, 2014 at 20:28 GMT

@aedan Harumph. I missed the obvious.

The reason we can install dnsmasq using the standard repos and not using our pinned repo is because the dnsmasq package is present in universe in the standard repos, and is not present in our merged & pinned repo. Here's the missing files:

http://archive.ubuntu.com/ubuntu/pool/universe/d/dnsmasq/

It's the 2.68-1 package we need.

from ansible-lxc-rpc.

Comment by jcannava
Friday Aug 22, 2014 at 20:32 GMT

Out of curiosity, space aside, wouldn't it be easier to have everything from a maintenance perspective? That way we at least start from an everything point and only move forward what makes sense.

I really don't see the point of doing it this way, especially when we are asking people to lock with this.

from ansible-lxc-rpc.

Comment by aedan
Friday Aug 22, 2014 at 20:34 GMT

No. We can do whatever. So I am thinking you would prefer then that I pull all packages? From universe and whatnot as well?

from ansible-lxc-rpc.

Comment by jcannava
Friday Aug 22, 2014 at 20:39 GMT

I think it will save us a headache later, but that's my opinion curious what everyone else thinks.

from ansible-lxc-rpc.

Comment by johnmarkschofield
Friday Aug 22, 2014 at 20:44 GMT

I don't think we need all packages -- we can omit the 32-bit ones, for instance.

In my experience if we pick and choose we're going to have a lot of cases just like this, so grabbing more-or-less wholesale is better.

In the past I've mirrored the repos and then used tags to manage them -- so we have a stable, alpha, RC, etc. repo, with each repo matching our git release tags. Not sure if the current infrastructure supports that.

@aedan Where is the code that generates the repo?

from ansible-lxc-rpc.

Comment by aedan
Friday Aug 22, 2014 at 20:47 GMT

Currently the repo is maintained by hand using aptly on a cloud server in DFW called trusty-frozen. The plan is to automate it with the jenkins-rpc repo.

from ansible-lxc-rpc.

Comment by hughsaunders
Tuesday Aug 26, 2014 at 12:13 GMT

@jcannava when you say 'everything' do you mean all of main, or main + universe? I'm guessing we can leave multiverse to prevent legal issues.

from ansible-lxc-rpc.

Comment by jcannava
Tuesday Aug 26, 2014 at 14:36 GMT

main + universe

from ansible-lxc-rpc.

Comment by aedan
Tuesday Aug 26, 2014 at 15:23 GMT

Ok. I will work on getting universe imported

from ansible-lxc-rpc.

Comment by aedan
Tuesday Aug 26, 2014 at 17:03 GMT

Ok its done.

deb [arch=amd64] http://dc0e2a2ef0676c3453b1-31bb9324d3aeab0d08fa434012c1e64d.r5.cf1.rackcdn.com universe-LA universe

from ansible-lxc-rpc.

This issue should be fixed. The dnsmasq package is available in LA-universe

from ansible-lxc-rpc.