Code Monkey home page Code Monkey logo

docker-postgres-upgrade's Introduction

pg_upgrade, Docker style

This is a PoC for using pg_upgrade inside Docker -- learn from it, adapt it for your needs; don't expect it to work as-is!

(Source for this image is available at https://github.com/tianon/docker-postgres-upgrade.)

Tags of this image are of the format OLD-to-NEW, where OLD represents the version of PostgreSQL you are currently running, and NEW represents the version of PostgreSQL you would like to upgrade to.

In order to get good performance, it is recommended to run this image with docker run image --link (see pg_upgrade documentation for more details).

For this to be feasible, your directory structure should look something like this: (if yours does not, either adjust it, or scroll down to see the alternate pattern for running this image)

$ find DIR -mindepth 2 -maxdepth 2
DIR/OLD/data
DIR/NEW/data

$ docker run --rm \
	-v DIR:/var/lib/postgresql \
	tianon/postgres-upgrade:OLD-to-NEW \
	--link

...

More concretely, assuming OLD of 9.4, NEW of 9.5, and DIR of /mnt/bigdrive/postgresql:

$ find /mnt/bigdrive/postgresql -mindepth 2 -maxdepth 2
/mnt/bigdrive/postgresql/9.4/data
/mnt/bigdrive/postgresql/9.5/data

$ docker run --rm \
	-v /mnt/bigdrive/postgresql:/var/lib/postgresql \
	tianon/postgres-upgrade:9.4-to-9.5 \
	--link

...

(which assumes that your previous postgres:9.4 instance was running with -v /mnt/bigdrive/postgresql/9.4/data:/var/lib/postgresql/data, and your new postgres:9.5 instance will run with -v /mnt/bigdrive/postgresql/9.5/data:/var/lib/postgresql/data)


If your two directories (denoted below as PGDATAOLD and PGDATANEW) do not follow this structure, then the following may also be used (but will be slower):

$ docker run --rm \
	-v PGDATAOLD:/var/lib/postgresql/OLD/data \
	-v PGDATANEW:/var/lib/postgresql/NEW/data \
	tianon/postgres-upgrade:OLD-to-NEW

...

More concretely, assuming OLD of 9.4, NEW of 9.5, PGDATAOLD of /mnt/bigdrive/postgresql-9.4, and PGDATANEW of /mnt/bigdrive/postgresql-9.5:

$ docker run --rm \
	-v /mnt/bigdrive/postgresql-9.4:/var/lib/postgresql/9.4/data \
	-v /mnt/bigdrive/postgresql-9.5:/var/lib/postgresql/9.5/data \
	tianon/postgres-upgrade:9.4-to-9.5

...

(which assumes that your previous postgres:9.4 instance was running with -v /mnt/bigdrive/postgresql-9.4:/var/lib/postgresql/data, and your new postgres:9.5 instance will run with -v /mnt/bigdrive/postgresql-9.5:/var/lib/postgresql/data)


Putting it all together:

$ mkdir -p postgres-upgrade-testing
$ cd postgres-upgrade-testing
$ OLD='9.4'
$ NEW='9.5'

$ docker pull "postgres:$OLD"
$ docker run -dit \
	--name postgres-upgrade-testing \
	-e POSTGRES_PASSWORD=password \
	-v "$PWD/$OLD/data":/var/lib/postgresql/data \
	"postgres:$OLD"
$ sleep 5
$ docker logs --tail 100 postgres-upgrade-testing

$ # let's get some testing data in there
$ docker exec -it \
	-u postgres \
	postgres-upgrade-testing \
	pgbench -i -s 10

$ docker stop postgres-upgrade-testing
$ docker rm postgres-upgrade-testing

$ docker run --rm \
	-v "$PWD":/var/lib/postgresql \
	"tianon/postgres-upgrade:$OLD-to-$NEW" \
	--link

$ docker pull "postgres:$NEW"
$ docker run -dit \
	--name postgres-upgrade-testing \
	-e POSTGRES_PASSWORD=password \
	-v "$PWD/$NEW/data":/var/lib/postgresql/data \
	"postgres:$NEW"
$ sleep 5
$ docker logs --tail 100 postgres-upgrade-testing

$ # can now safely remove "$OLD"
$ sudo rm -rf "$OLD"

docker-postgres-upgrade's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-postgres-upgrade's Issues

10-to12: connection to database failed: could not connect to server: No such file or directory

Directory structure under /mnt/ocfs2/timescaledb:
find /mnt/ocfs2/timescaledb/ -mindepth 2 -maxdepth 2
/mnt/ocfs2/timescaledb/10/data
/mnt/ocfs2/timescaledb/12/data

Command run:
docker run --rm -v /mnt/ocfs2/timescaledb:/var/lib/postgresql tianon/postgres-upgrade:10-to-12 --check -O "-c timescaledb.restoring='on'"

Results in following output:

Performing Consistency Checks on Old Live Server

Checking cluster versions ok
connection to database failed: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Failure, exiting

It is just impossible to get to a Unix socket in another running container!
Moreover, the 10-to-12 image downloads and installs the 10 version package, so in theory could run it locally; this would lead to:

  • no need to keep the OLD running
  • ability to reach the OLD db via the Unix socket.

10 to 11 role "postgres" does not exist

When trying to upgrade from 10 to 11 - the following error pops up:

...

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/11/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

connection to database failed: FATAL:  role "postgres" does not exist

could not connect to source postmaster started with the command:
"/usr/lib/postgresql/10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/10/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

I tried setting the POSTGRES_* environment variables, but still getting the same error. Any idea why this is happening?

pg_hba.conf does not match postgres image

This means that anything not connecting via the local socket will fail, so most applications linking the postgres container will not work.

At least for me, pg_hba.conf was missing this:

host all all 0.0.0.0/0 md5

Normally this would be set up in https://github.com/docker-library/postgres/blob/master/9.5/docker-entrypoint.sh, but because we use initdb during the upgrade, that never happens.

check for "/var/lib/postgresql/9.4/data/base" failed: No such file or directory Failure, exiting

Hello and merry Christmas,

I was just looking to upgrade my sonarqube database from postgresql 9.4 to 9.5.

I took the snippet at the very bottom and used it for myself, when I started I had:

  • $PWD = /srv/tmp containing a data directory.
root@master-sbg-01:/srv/tmp# docker logs -f 070158ff887a

PostgreSQL Database directory appears to contain a database; Skipping initialization

FATAL:  database files are incompatible with server
DETAIL:  The data directory was initialized by PostgreSQL version 9.4, which is not compatible with this version 9.5.20.

I've never succeeded to run verify script, how do you run it (it's not in the snippet), do you run it from host? do you remount the volume and create a new container and how ?

Do you guys know what I have done wrong here? Can this be reverted?

Thanks and best!

Thanks and best!

Unsupported locale

While trying to migrate a 9.6.11-alpine db to 13, I receive the following error:

connection to database failed: FATAL:  database locale is incompatible with operating system
DETAIL:  The database was initialized with LC_COLLATE "utf-8",  which is not recognized by setlocale().
HINT:  Recreate the database with another locale or install the missing locale.
Failure, exiting

This is the locale / collation of the old db:
CREATE DATABASE xx WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'utf-8' LC_CTYPE = 'UTF-8';

After upgrading from 9.5 to 10, new db has no tables

I ran the upgrade from 9.5 to 10 and it appeared that everything went smoothly. However, when I inspected the database using psql, the databases do not seem to have any tables defined. Any idea what could be wrong?

Here's a transcript:

docker run --rm \
> -v /data:/var/lib/postgresql \
> tianon/postgres-upgrade:9.5-to-10 \
> --link
Unable to find image 'tianon/postgres-upgrade:9.5-to-10' locally
9.5-to-10: Pulling from tianon/postgres-upgrade
723254a2c089: Already exists
39ec0e6c372c: Already exists
ba1542fb91f3: Already exists
c7195e642388: Already exists
95424deca6a2: Already exists
2d7d4b3a4ce2: Already exists
fbde41d4a8cc: Already exists
880120b92add: Already exists
9a217c784089: Already exists
d581543fe8e7: Already exists
e5eff8940bb0: Already exists
462d60a56b09: Already exists
135fa6b9c139: Already exists
e913dd1518e8: Pull complete
8cf5f006816f: Pull complete
628f58081730: Pull complete
0de97691839d: Pull complete
Digest: sha256:087e932c42d377af249604cc34d205a7b7c7dfa7e8e8be623603ec172ee9d10f
Status: Downloaded newer image for tianon/postgres-upgrade:9.5-to-10
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/10/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/10/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for invalid "unknown" user columns                 ok
Checking for roles starting with "pg_"                      ok
Creating dump of global objects                             ok
Creating dump of database schemas
  csdb
  postgres
  template1
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_clog to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
  csdb
  postgres
  template1
                                                            ok
Adding ".old" suffix to old global/pg_control               ok

If you want to start the old cluster, you will need to remove
the ".old" suffix from /var/lib/postgresql/9.5/data/global/pg_control.old.
Because "link" mode was used, the old cluster cannot be safely
started once the new cluster has been started.

Linking user relation files
  /var/lib/postgresql/9.5/data/base/32889/2613
  /var/lib/postgresql/9.5/data/base/32889/2683
  /var/lib/postgresql/9.5/data/base/32889/2995
  /var/lib/postgresql/9.5/data/base/32889/2996
  /var/lib/postgresql/9.5/data/base/32889/32890
  /var/lib/postgresql/9.5/data/base/32889/32893
  /var/lib/postgresql/9.5/data/base/32889/32897
  /var/lib/postgresql/9.5/data/base/32889/32901
  /var/lib/postgresql/9.5/data/base/32889/32903
  /var/lib/postgresql/9.5/data/base/32889/32904
  /var/lib/postgresql/9.5/data/base/32889/32906
  /var/lib/postgresql/9.5/data/base/32889/32910
  /var/lib/postgresql/9.5/data/base/32889/32914
  /var/lib/postgresql/9.5/data/base/32889/32916
  /var/lib/postgresql/9.5/data/base/32889/32917
  /var/lib/postgresql/9.5/data/base/32889/32919
  /var/lib/postgresql/9.5/data/base/32889/32928
  /var/lib/postgresql/9.5/data/base/32889/32932
  /var/lib/postgresql/9.5/data/base/32889/32934
  /var/lib/postgresql/9.5/data/base/32889/32935
  /var/lib/postgresql/9.5/data/base/32889/32944
  /var/lib/postgresql/9.5/data/base/32889/32948
  /var/lib/postgresql/9.5/data/base/32889/32950
  /var/lib/postgresql/9.5/data/base/32889/32957
  /var/lib/postgresql/9.5/data/base/32889/32960
  /var/lib/postgresql/9.5/data/base/32889/32962
  /var/lib/postgresql/9.5/data/base/32889/32963
  /var/lib/postgresql/9.5/data/base/32889/32982
  /var/lib/postgresql/9.5/data/base/32889/32986
  /var/lib/postgresql/9.5/data/base/32889/32988
  /var/lib/postgresql/9.5/data/base/32889/32989
  /var/lib/postgresql/9.5/data/base/32889/33013
  /var/lib/postgresql/9.5/data/base/32889/33017
  /var/lib/postgresql/9.5/data/base/32889/33031
  /var/lib/postgresql/9.5/data/base/32889/33035
  /var/lib/postgresql/9.5/data/base/32889/33037
  /var/lib/postgresql/9.5/data/base/32889/33038
  /var/lib/postgresql/9.5/data/base/32889/33050
  /var/lib/postgresql/9.5/data/base/32889/33063
  /var/lib/postgresql/9.5/data/base/12379/2613
  /var/lib/postgresql/9.5/data/base/12379/2683
  /var/lib/postgresql/9.5/data/base/12379/2995
  /var/lib/postgresql/9.5/data/base/12379/2996
  /var/lib/postgresql/9.5/data/base/1/2613
  /var/lib/postgresql/9.5/data/base/1/2683
  /var/lib/postgresql/9.5/data/base/1/2995
  /var/lib/postgresql/9.5/data/base/1/2996
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok
Checking for hash indexes                                   ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh
mark@:xxxx/data$ ls
10  9.5  analyze_new_cluster.sh  delete_old_cluster.sh
mark@:xxxx/data$ sudo ls 10/data
[sudo] password for mark:
base	      pg_hba.conf    pg_notify	   pg_stat	pg_twophase  postgresql.auto.conf
global	      pg_ident.conf  pg_replslot   pg_stat_tmp	PG_VERSION   postgresql.conf
pg_commit_ts  pg_logical     pg_serial	   pg_subtrans	pg_wal	     postmaster.opts
pg_dynshmem   pg_multixact   pg_snapshots  pg_tblspc	pg_xact
mark:/data$ docker exec -it cs_postgres_1 psql --username=csroot --dbname=csdb
psql (10.1)
Type "help" for help.

csdb=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 csdb      | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(4 rows)

csdb=# \d
Did not find any relations.

could not open version file

Don't know how to use script
Assuming I have database _data files of pgSQL binded to host /v11
How to perform upgrade to v12 ?
No matter which path combination I will use according to your instruction getting error:

could not open version file: /var/lib/postgresql/11/data/PG_VERSION
Failure, exiting

Upgrading from 9.6 to 10 : invalid PG_VERSION path

Hello,

I'm a beginner to docker and to postgres as well. I've been asked to upgrade postgres from 9.6 to 10. So thank you very much for this image !

In my case, all the data is in 9.6/main, instead of 9.6/data. When i try to upgrade postgres, i get the following message :
`$ sudo docker run --rm -v /etc/postgresql/9.6/main:/var/lib/postgresql/9.6/main -v /etc/postgresql/10/main:/var/lib/postgresql/10/main tianon/postgres-upgrade:9.6-to-10 --link

...
Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/10/data -l logfile start

could not open version file: /var/lib/postgresql/9.6/data/PG_VERSION
Failure, exiting`

Can you help me ?

Best regards,
Ludovic

using volumes as data path for 11 & 12

Hi Team,
I tested this https://github.com/tianon/docker-postgres-upgrade in my local & tried upgrade 11 to 12 postgres version. I need some details on this.

i have docker volumes as data path for postgres & mounted to /var/lib/postgres docker postgres container.

In this case, how can use the 11 to 12 upgrade -v path..??
below will it be work..? or have to change..?

docker run --rm
-v /var/lib/docker/volumes/volumes-name:/var/lib/postgresql
"tianon/postgres-upgrade:$OLD-to-$NEW"
--link

could not open file "./analyze_new_cluster.sh": Permission denied

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/13/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/13/data -l logfile start

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Checking for invalid "unknown" user columns                 ok
Checking for roles starting with "pg_"                      ok
Creating dump of global objects                             ok
Creating dump of database schemas
  mattermost
  postgres
  template1
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_clog to new server                           ok
Setting oldest XID for new cluster                          ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
  template1
  mattermost
  postgres
                                                            ok
Adding ".old" suffix to old global/pg_control               ok

If you want to start the old cluster, you will need to remove
the ".old" suffix from /var/lib/postgresql/9.4/data/global/pg_control.old.
Because "link" mode was used, the old cluster cannot be safely
started once the new cluster has been started.

Linking user relation files
  /var/lib/postgresql/9.4/data/base/16384/12095
  /var/lib/postgresql/9.4/data/base/16384/12097

..
...
..

  /var/lib/postgresql/9.4/data/base/1/12095
  /var/lib/postgresql/9.4/data/base/1/12097
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster
could not open file "./analyze_new_cluster.sh": Permission denied
Failure, exiting

linux version : Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

CentOS Linux release 7.8.2003 (Core)
Derived from Red Hat Enterprise Linux 7.8 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.8.2003 (Core)
CentOS Linux release 7.8.2003 (Core)
cpe:/o:centos:centos:7

docker version : Docker version 19.03.13, build 4484c46d9d

could not open file "./analyze_new_cluster.sh": Permission denied
Failure, exiting

It doesn't go any further with that error.
help me.

Missing DBs

Hi,
I'm trying to upgrade my PG container from 11 to 13. I ran docker-postgres-upgrade without any trouble, all the output traces were "ok".

I can login to my new PG13 instance with the postgresuser. But through Adminer, I don't see the databases which were supposed to be migrated.

And of course, the containers which rely on those DBs don't work, no surprise here.

Any idea?

Upgrade from postgres:11.5-alpine to postgres:12.0-alpine

Hi Team,
I am using docker & currently running 11.5-alpine postgres
I would like to upgrade to 12.0-alpine postgres. But If i use this "tianon/docker-postgres-upgrade" then i got error because 11.5 / 12.0 version folder is not there.

Please help me how to upgrade in this case.
Thank you in advance

pg_upgrade 12 to 14 with postgis - several hurdles - solved

Just wanted to drop this in here as I had several issues during my upgrade, but managed to get through after a couple of days searching and failing.

  1. There seems to be a postmaster servicing the old cluster. Please shutdown that postmaster and try again.
    see #15
    I increased the shutdown period to 10min, but it was not enough. I had to run the 4 commands described in issue 15 above.
  2. Next failure was with error ""no password supplied"
    This was fixed with by editing pg_hba.conf on original (v12) postgresql. Changed settings for the two "local" lines from my "md5" to "peer". (Note that changing the other to peer will genereate another error).
    See postgres official upgrade step7 (Adjust authentication). https://www.postgresql.org/docs/14/pgupgrade.html
  3. Finally something started to happen. But as I had postgis extention installed I had to run the upgrade with this extentions.
    see #33
    I build a new docker container for upgrade with the postgis extentions as explained above.
  4. Next failure was during upgrade of the postgis extention:
    -- For binary upgrade, handle extension membership the hard way
    ALTER EXTENSION "postgis_sfcgal" ADD FUNCTION "public"."postgis_sfcgal_version"();
    I could not find much information about this. But I ended up removing this extension on the databases that had it. and then create them again when the pg14 was up and running. I deleted and created these with pgadmin4 web ui.
    I suspect this failure might be due to incorrect versions of postgis in my upgrade docker vs the original PG12 docker.
  5. when upgrade finally completed I had to change the PG14 pg_hba.conf file to get access again. basically I copied the file from v12, and changed back the "peer" changes made at failure 2 above.
  6. finally getting the connections up with pgadmin4 to the new PG14 I created the postgis_sfcgal extension from failure 4 above, and also updated the extensions as suggested in the "update_exstensions.sql" generated by this docker_upgrade.

Hope this can help others that also has issues upgrading

Docker image should run pg_upgrade's generated update_extension.sql file when present

Issue:
When update_extension.sql is generated, docker-postures-upgrade's script does not run the file automatically, while I was expecting to since built-in Postgres extensions are easy to upgrade with the generated sql file.

If this is beyond the intended scope of the project, let me know.

Command:

docker run --rm \
	-v ....
        -v ....
	tianon/postgres-upgrade:12-to-15

Output

....
Performing Upgrade
------------------
...
Creating script to delete old cluster                       ok
Checking for extension updates                              notice

Your installation contains extensions that should be updated
with the ALTER EXTENSION command.  The file
    update_extensions.sql.                             <<<<<<<<<<< HERE
when executed by psql by the database superuser will update
these extensions.
...

"exec format error" when upgrading to Postgres 11 on ARM

Hello,
I'm currently trying to update to postgres from 10 to 11 (necessary to update my funkwhale instance, for background) on an Raspberry Pi 4 (so with an arm architecture) as stated here : https://dev.funkwhale.audio/funkwhale/funkwhale/-/releases (docker all-in-one situation)

More precisely, I tried this :

docker run --rm
-v $(pwd)/data:/var/lib/postgresql/10/data
-v $(pwd)/upgraded-postgresql:/var/lib/postgresql/11/data
-e PGUSER=funkwhale
-e POSTGRES_INITDB_ARGS="-U funkwhale --locale C --encoding UTF8"
tianon/postgres-upgrade:10-to-1

to download and install postgres 11. What I obtained, after the download was complete :

Status: Downloaded newer image for tianon/postgres-upgrade:10-to-11
standard_init_linux.go:211: exec user process caused "exec format error"

I presume that the problem is linked to the fact that it's an arm architecture, and that the image used was not build for arm. Is there a way to build myself the image for arm ? Or do I have to upgrade another way ?
I precise that I'm not really used to the utilisation of postgres and docker, so maybe I miss an obvious solution.

Thank you for your help

Upgrading from 11 to 12 failing with `"/var/lib/postgresql/11/data/PG_VERSION": No such file or directory`

This is related/similar to #29.

The issue

I'm running a timescale postgres on pg11, launching with docker-compose. When running the upgrade I get the following output and failure message

tianon/postgres-upgrade:11-to-12
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/12/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb:
Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/12/data -l logfile start

 warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

 "/var/lib/postgresql/11/data/PG_VERSION": No such file or directory
Failure, exiting

My setup

I am starting with a docker-compose setup where I recently upgrade the timescale from 1.3.1 to 17.5 (I give this for context but I don't think this is in anyway the source of the issue).

The compose takes the pattern as following:

version: '3.5'

services:
  timescale:
    image: timescale/timescaledb:1.7.5-pg11
    container_name: postgres
    restart: always
    command: postgres -c shared_preload_libraries=timescaledb
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=mytimescaledatabase
    volumes:
      - ./data/postgres:/var/lib/postgresql/data

As you'll see, My config is already a bit outside the pattern(s) described in the README.

Steps to run upgrade

Since my PGDATAOLD is effectively mounted, quite awkwardly, as ~/DIR/data/postgres:/var/lib/postgresql/data, may plan was to mount PGDATANEW as ~/DIR/data/12/postgres:/var/lib/postgresql/data.

I didn't do any 'upgrade testing' outlined in the step outline in the README, so be kind. ;)

I ran (from within working directory):

$ docker run --rm \
-v $PWD/data/postgres:/var/lib/postgresql/data \ 
-v $PWD/data/12/postgres:/var/lib/postgresql/12/data \
tianon/postgres-upgrade:11-to-12

Result

As outlined above.

Also...

  • The database runs, and is recognised as non-empty by postgres, when attached to a pg12 image.
  • My database/tables are missing.
  • Trying to connect to the database from PGAdmin I am greeted with FATAL: no pg_hba.conf entry for host "w.x.y.z", user "postgres", database "postgres", SSL off. This led to further investigation, but probably not worthwhile to explain in detail here (I can supply more details if requested).

Does not work with official postgres:10 ( or earlier ) due to pg_hba.conf

When using your image to migrate the databases, pg_hba.conf will be replaced using a version, which will not allow other hosts to connect using a password

this line is not included in the pg_hba.conf of the newly migrate DATA folder, lets assume pg11

host all all all md5

One can easily copy this file from pg10/data/pg_hba.conf to pg10/data/pg_hba.conf or just add the line i mentioned - but we might also consider mention this in the readme.

Without that, PG boots up just fine, the migratoin did work just fine - just no other service in the docker stack can use PG since they cannot connect to it :)

Failure, exiting. Can't find log.

Hello. I am trying to use this image with the following arguments:

docker run --rm 
-e POSTGRES_INITDB_ARGS="--wal-segsize=1024"
-v /home/user/db/12/main:/var/lib/postgresql/12/data 
-v /home/user/db/13/main:/var/lib/postgresql/13/data 
tianon/postgres-upgrade:12-to-13

but it fails with this output:

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
  postgres
  my_database

*failure*

Consult the last few lines of "pg_upgrade_dump_116017.log" for
the probable cause of the failure.
Failure, exiting

Where can I found that logfile pg_upgrade_dump_116017.log please?

Error: Invalid cross-device link when using named volumes for data directory

I am trying to upgrade my postgres 10 to 14 (on my windows machine)

When I do this, I get the error:

could not create hard link between old and new data directories: Invalid cross-device link
In link mode the old and new data directories must be on the same file system.
Failure, exiting

This is the command I used:

docker run --rm \
  -v postgresql-data-volume:/var/lib/postgresql/10/data \
  -v postgresql-data-volume-14:/var/lib/postgresql/14/data \
  tianon/postgres-upgrade:10-to-14 --link

It works, when I do not use the link option. I would need the link option to do it (as I need the same thing to be also done on production)

I would think it is due to the named volumes I use (just like how it is configured on production)

postgresql-data-volume
postgresql-data-volume-14:

Is it possible to do the in place upgrade using different named volumes, or should I be using a local directory for the same (it seems I cannot do this on windows - using named directory)

(running the test setup as per the readme file worked without any issues)

In short, does it also support a --link upgrade for named volumes ?

9.6 to 1.0 returning error

postgres_1 | check for "/usr/lib/postgresql/9.6/bin" failed: No such file or directory
postgres_1 |
postgres_1 | Failure, exiting

More automated upgrade

Is there any consideration for a more in-place upgrade?

  1. Accept the input data directory mount DATA (no NEW or OLD)
  2. Install postgres version based on DATA/PG_VERSION
  3. Create a new data directory, run initdb
  4. Migrate run pg_upgrade
  5. If successful
    • Optionally move the old data from DATA to a backup directory
    • Delete contents of DATA
    • Move everything from the new data directory to DATA

Simply does not work

Bunch of strange errors despte of simple instrucion, looks like complicated usage, nothing precised.
Has it worked to anyone at all?

Could not access file "timescaledb": No such file or directory

I'm trying to run the 11-to-12 image in order to upgrade a db being run from docker-compose like so:

timescaledb:
  image: timescale/timescaledb:1.7.4-pg11
  volumes:
      - timescale_data:/var/lib/postgresql/data
  ports:
      - 5431:5432
  environment:
      POSTGRES_PASSWORD: REDACTED
      POSTGRES_DB: timescale
  networks:
      - default

Unfortunately, I'm getting the following error:

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: connection to server on socket "/var/lib/postgresql/.s.PGSQL.50432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?

could not connect to source postmaster started with the command:
"/usr/lib/postgresql/11/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/11/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

Looking at pg_upgrade_server.log, I see these lines:

waiting for server to start....2022-09-09 14:37:25.959 UTC [82] FATAL:  could not access file "timescaledb": No such file or directory
2022-09-09 14:37:25.960 UTC [82] LOG:  database system is shut down
 stopped waiting
pg_ctl: could not start server

Any idea what's the cause and how to fix it?

Upgrade postgresql with postgis extension

I'm trying to upgrade a database that has postgis installed in it. Since the container doing the upgrade doesn't have postgis installed in it it gives this error:

pg_dump: error: query failed: ERROR:  could not access file "$libdir/postgis-3": No such file or directory

chown: cannot read directory: Permission denied

Hi,

Upgrading from pg 9.6 - 10 results in this output:

["chown: cannot read directory '/var/lib/postgresql/9.6/data': Permission denied", "chown: cannot read directory '/var/lib/postgresql/10/data': Permission denied"]

Directory permissions for pg 9.6 and 10:

[root@awx2 pgdocker]# ll
total 4
drwxr-xr-x.  3 root    root   18 Nov 22 10:27 10
drwx------. 19 polkitd root 4096 Nov 22 12:27 pgdata

Running upgrade as root.

Any ideas on how fix this?

where are the scripts to analyze?

I got these messages after upgraded from 10 to 11

Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

but how can I find these scripts?

Failure, exiting: The source cluster was not shut down cleanly

First I stop my postgres 9.4 service instance, next:

$ tree -L 2
.
├── 12
│   └── data
├── 9.4
│   └── data
├── data
├── pg_upgrade_internal.log
├── pg_upgrade_server.log
└── pg_upgrade_utility.log
$ docker run --rm -v `pwd`/:/var/lib/postgresql tianon/postgres-upgrade:9.4-to-12
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

The source cluster was not shut down cleanly.
Failure, exiting

Can you help me? What is my mistake?

Best regards,
Stéphane

11 to 12 role "postgres" does not exist

When trying to upgrade from 11 to 12 - the following error pops up:

docker run --rm \
        -e PGUSER=$POSTGRES_USER \
        -e POSTGRES_INITDB_ARGS="-U $POSTGRES_USER" \
        -v aip-pgs-data:/var/lib/postgresql/$OLD/data \
        -v aip-pg12-data:/var/lib/postgresql/$NEW/data \
        "tianon/postgres-upgrade:$OLD-to-$NEW"


LOGS:---- 
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

connection to database failed: FATAL:  role "postgres" does not exist

could not connect to source postmaster started with the command:
"/usr/lib/postgresql/11/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/11/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

I tried this workaround #10 (comment) , but still getting the same error. Any idea why this is happening?
OR is that open bug.?

check for "/var/lib/postgresql/9.4/data/base" failed: No such file or directory

Hi there.

First of all many thanks to all the people involved in this project for their time.

I'm trying to follow the provided example in the README, and I get an error.

Steps to reproduce

mkdir -p postgres-upgrade-testing

cd postgres-upgrade-testing

OLD='9.4'
NEW='9.5'

docker pull "postgres:$OLD"

docker run -dit \
  --name postgres-upgrade-testing \
  -v "$PWD/$OLD/data":/var/lib/postgresql/data \
  "postgres:$OLD"

sleep 5 

docker logs --tail 100 postgres-upgrade-testing

docker exec -it \
  -u postgres \
  postgres-upgrade-testing \
  pgbench -i -s 10

docker stop postgres-upgrade-testing

docker rm postgres-upgrade-testing

docker run --rm \
 -v "$PWD":/var/lib/postgresql \
 "tianon/postgres-upgrade:$OLD-to-$NEW" \
 --link

Expected behaviour

See a success message in the standard out

Found behaviour

The process stopped with an error message:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.5/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/9.5/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/9.5/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

check for "/var/lib/postgresql/9.4/data/base" failed: No such file or directory

Failure, exiting

Error message

check for "/var/lib/postgresql/9.4/data/base" failed: No such file or directory

Failure, exiting

Additional information

  • Operating system: GNU/Linux debian Debian GNU/Linux bullseye/sid \n \l
  • Docker version: 19.03.8, build afacb8b7f0

I'm almost sure that the root cause of the issue is the owner of the docker host directories.

The start of the workaround will require re-building a custom image with something like below:

ARG GNAME='groupname'
ARG GID=123
ARG USERID=113

# fix permissions so it can persist data on the host nfs file system
RUN groupadd -g $GID $GNAME \
 && usermod -g $GNAME postgres \
 && usermod -u $USERID postgres

Break database made with Alpine-based container

I tried to update a postgres database running on postgres:12.13-alpine3.17 to postgres:14.6-alpine3.17 (or postgres:14.6-bulleyes).

Everything worked, however I noticed that the software using it couldn't commit anything to it anymore.
This error appears every time it tries to write to the database:
ERROR: duplicate key value violates unique constraint "[REDACTED]" SQL error updateCachedValue: ERROR: current transaction is aborted, commands ignored until end of transaction block (a ROLLBACK; didn't solve it)

My guesses:

After looking at your Dockerfile it looks like you're running the "updater" on a Debian container.

Database made with an Alpine-based container are known to not be compatible with Debian-based container (and vice versa), it looks like it's the problem here.

A solution that might work is to have new containers where the "updater" run on Alpine. For now I think you should add a warning that says that this project isn't compatible with database made with alpine-based container.

Missing support for Postgres 14

Postgres 14 has been released a few days ago and I would like to try out this tool to upgrade my Docker databases this time! 😄 Will keep an eye on this repository.

Thanks!

Cannot create directory: File exists.

I've prepared the required directory structure. The target dir is empty.

What am I doing wrong?

image

The second option works well:

docker run --rm \
 -v /data/pgdata/pg15-dev:/var/lib/postgresql/15/data \
 -v /data/pgdata/pg16-dev:/var/lib/postgresql/16/data \
 tianon/postgres-upgrade:15-to-16 

But it doesn't support the --link argument because, from the container's point of view, they are two different file systems, and cross-dev links are impossible. I don't have enough space for an entire copy of PGDATA.

We are having some problems when trying to upgrade PostgreSQL 9.4. to 9.6

Hello tianon,

Thank you for the image. We hope you can help us with this question: We are trying to upgrade from postgres 9.4 to 9.6, but the docker hub default images do not have the postgres version in there path $PGDATA. So it's /var/lib/postgresql/data and not /var/lib/postgresql/9.4/data.

We are getting these errors when trying the upgrade:

TEST1:

[tiabd@dockerbuild tmp]$ docker run --rm
-v /tmp/9.4:/var/lib/postgresql/data
-v /tmp/9.6:/var/lib/postgresql/data
tianon/postgres-upgrade:9.4-to-9.6
docker: Error response from daemon: Duplicate mount point '/var/lib/postgresql/data'.
See 'docker run --help'.
[tiabd@dockerbuild tmp]$

TEST2:

[tiabd@dockerbuild tmp]$ docker run --rm \

-v /tmp/9.4/data:/var/lib/postgresql/9.4/data \
-v /tmp/9.6/data:/var/lib/postgresql/9.6/data \
tianon/postgres-upgrade:9.4-to-9.6

...
...
initdb: directory "/var/lib/postgresql/9.6/data" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/9.6/data" or run initdb
with an argument other than "/var/lib/postgresql/9.6/data".
Failure, exiting

TEST3:

[tiabd@dockerbuild tmp]$ docker run --rm -v /tmp/9.4:/var/lib/postgresql/9.4/data -v /tmp/9.6:/var/lib/postgresql/9.6/data tianon/postgres-upgrade:9.4-to-9.6
...
check for "/var/lib/postgresql/9.4/data/base" failed: No such file or directory

Failure, exiting

Hope you can help us further.

Kind regards,
Tim

Specify a user & password

Hi, I'm trying to migrate from 12 to 13, but I have the error role "postgres" does not exist.
It's normal because this database doesn't contain this role, is there any way to specifiy it?

I've tried with docker run --rm -v /docker/postgres:/var/lib/postgresql -e POSTGRES_PASSWORD=pas**rd -e POSTGRES_USER=us@r tianon/postgres-upgrade:12-to-13 --link but it seems to me that the environment variables ar not set. :(

support custom username for example "-U my_username" when connecting

The database I am trying to upgrade has a custom username and therefore I would get the following error message when running the upgrade

psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?

could not connect to source postmaster started with the command:
"/usr/lib/postgresql/12/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/12/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

would it be possible to support "-U my_username" ? Or how should I adapt the script ? thank you!

[9.4 to 9.6] Upgrade failure due to different lc_collate values

First, I want to thank you very much for creating this image. This is really added value!
Today, I wanted to upgrade from 9.4 to a 9.6 version and got the following error:

docker run --rm -v /mnt/bigdrive/postgresql:/var/lib/postgresql tianon/postgres-upgrade:9.4-to-9.6 --link
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for roles starting with 'pg_'                      ok
Creating dump of global objects                             ok
Creating dump of database schemas
  mydb
  postgres
  template1
                                                            ok

lc_collate values for database "postgres" do not match:  old "C", new "en_US.utf8"
Failure, exiting

To verify the error message I started the 2 containers again and bashed into the servers and got the following.
9.4 Container:

postgres=# \list
                               List of databases
    Name     |    Owner    | Encoding | Collate | Ctype |   Access privileges   
-------------+-------------+----------+---------+-------+-----------------------
 mydb | mydb | UTF8     | C       | C     | 
 postgres    | postgres    | UTF8     | C       | C     | 
 template0   | postgres    | UTF8     | C       | C     | =c/postgres          +
             |             |          |         |       | postgres=CTc/postgres
 template1   | postgres    | UTF8     | C       | C     | =c/postgres          +
             |             |          |         |       | postgres=CTc/postgres
(4 rows)

9.6 Container:

postgres=# \list
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(3 rows)

So the error message is correct, the Collations and the Ctypes are different. How can I achieve that my data contained in the old verison is migrated to the new version by using the new Collate and Ctype values?

Does not work with official postgres:10 due to shutdown issues

First off, thank you SO much for your work here - incredible!

Today i tried to upgrade from postgres:10 to postgres:11 - so using the official images. The problem is actually, that there seem to be no way to "cleanly" shutdown the "old" docker container so that this "counts" for pg_upgrade to be a clean shutdown and thus allow the migration.

What i had to do is (pg10 stands for your current docker container running pg old):

docker exec -it pg10 bash
su - postgres
export PGDATA=/var/lib/postgresql/data
/usr/lib/postgresql/10/bin/pg_ctl stop

This will now properly shutdown the container. Everything else goes just as you described in the README.md

docker run --rm  --name upgrade -u postgres -v /host/db/10:/var/lib/postgresql/10/data -v /host/db/11:/var/lib/postgresql/11/data tianon/postgres-upgrade:10-to-11

I am not sure we can fix the former (stop issue) here in the docker image automatically, but we can at least add it to the readme. What do you think?

Again, thank you so much!

After latest version upgrade, The 13-14 image no longer works

This was working fine yesterday, did a pull from docker hub today and it's now broken:
Here is the error message:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are enabled.

fixing permissions on existing directory /var/lib/postgresql/14/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/14/data -l logfile start


check for "/usr/lib/postgresql/14/bin/pg_dump" failed: incorrect version: found "pg_dump (PostgreSQL) 14.2 (Debian 14.2-1.pgdg110+1)", expected "pg_dump (PostgreSQL) 14.3 (Debian 14.3-1.pgdg110+1)"
Failure, exiting

Role Postgres does not exist

I've tried to run an update from PG 12 to PG 14, I currently simply have the db data in a docker volume and there's no postgres user on the host machine.

Am I doing anything wrong?

Full command and output attacked below:

$ docker run --rm -v db-data:/var/lib/postgresql/12/data -v db-data-14:/var/lib/postgresql/14/data tianon/postgres-upgrade:12-to-14

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/14/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/14/data -l logfile start

initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

connection to server on socket "/var/lib/postgresql/.s.PGSQL.50432" failed: FATAL:  role "postgres" does not exist

could not connect to source postmaster started with the command:
"/usr/lib/postgresql/12/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/12/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

wont start - fails to find socket of source postgres

docker run --rm \
    -v /srv/appname/data/postgres:/var/lib/postgresql/10 \
    -v /srv/appname/data/postgres12:/var/lib/postgresql/12 \
    tianon/postgres-upgrade:10-to-12 

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

connection to database failed: connection to server on socket "/var/lib/postgresql/.s.PGSQL.50432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?

could not connect to source postmaster started with the command:
"/usr/lib/postgresql/10/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/10/data" -o "-p 50432 -b  -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

Any ideas of next troubleshooting step?
Ive confirmed all the files in /srv/appname/data/postgres are owned by uid 1001. ive tried the 10-14 image and it was the same error
Ive also tried adding "data" to the volume mounts but the same issue occurs, if i put in an incorrect path it warns that it cant find the PG_VERSION file, so its definitely seeing the data:

docker run --rm \
    -v /srv/appname/data/postgres/data:/var/lib/postgresql/10/data \
    -v /srv/appname/data/postgres12/data:/var/lib/postgresql/12/data \
    tianon/postgres-upgrade:10-to-12 

Docker image for ARM64 (aarch64 / Arm Ampere) architecture

Hi! First, thanks for such an incredible tool.

Would you please consider also publishing versions of Docker images for aarch64 architecture? A lot of cloud providers are already suggesting attractive options (like https://www.oracle.com/cloud/compute/arm/ ), so this architecture is becoming more common. Current images are built for arm64 architecture and can not be used

  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  84s                default-scheduler  Successfully assigned default/postgres-upgrade-job-p8b4x to omen
  Normal   Pulling    41s (x3 over 84s)  kubelet            Pulling image "tianon/postgres-upgrade:15-to-16"
  Warning  Failed     39s (x3 over 82s)  kubelet            Failed to pull image "tianon/postgres-upgrade:15-to-16": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/tianon/postgres-upgrade:15-to-16": no match for platform in manifest: not found
  Warning  Failed     39s (x3 over 82s)  kubelet            Error: ErrImagePull
  Normal   BackOff    2s (x5 over 81s)   kubelet            Back-off pulling image "tianon/postgres-upgrade:15-to-16"
  Warning  Failed     2s (x5 over 81s)   kubelet            Error: ImagePullBackOff

If I understand correctly, that is just a matter of adding an arm64 architecture tag to the script that renders the template and using Postgres containers/repos for this architecture

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.