Code Monkey home page Code Monkey logo

Comments (10)

rkoberg avatar rkoberg commented on May 18, 2024

We will also have another container that is kind of similar to the way Openresty is used, but it is a golang container for massive concurrency. The client app will post to the golang endpoint which sends to a redismq to queue things up to send to redis. Then the redis data goes back to the golang container which puts a CSV batch together to send to PostgREST. So there will be three extra containers there with one custon one (the golang endpoint). We wanted to use the subzer-cli to handle these. Are there any plpans to make the cli a more generic tool that can deploy.

from subzero-cli.

rkoberg avatar rkoberg commented on May 18, 2024

Next I tried to dump everything so I could manually setup the DB, but that did not work:

template1=# GRANT ALL PRIVILEGES ON DATABASE app to rkoberg;
GRANT

Got an error:

$ pg_dump -c --if-exists  -f app.ddl -h 0.0.0.0 -p 5432 -U rkoberg -d app
Password: 
pg_dump: [archiver (db)] query failed: ERROR:  permission denied for schema settings
pg_dump: [archiver (db)] query was: LOCK TABLE settings.secrets IN ACCESS SHARE MODE

So,

template1=# \c app
You are now connected to database "app" as user "postgres".
app=# GRANT ALL PRIVILEGES ON SCHEMA auth to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA settings to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA pgjwt to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA request to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA util to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA data to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA api to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA auth  TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA settings TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pgjwt TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA request TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA util TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA data TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA api TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON DATABASE app TO rkoberg;
GRANT

Then I tried a dump again, but:

$ pg_dump -c --if-exists  -f app.ddl -h 0.0.0.0 -p 5432 -U rkoberg -d app
Password: 
pg_dump: [archiver (db)] query failed: ERROR:  permission denied for relation user_id_seq
pg_dump: [archiver (db)] query was: SELECT last_value, is_called FROM user_id_seq

So,

app=# GRANT ALL PRIVILEGES ON SEQUENCE user_id_seq TO rkoberg;
ERROR:  relation "user_id_seq" does not exist

How do you get a full dump? Or any other ideas?

from subzero-cli.

rkoberg avatar rkoberg commented on May 18, 2024

I was able to get a dump as the postgres user from inside the local container. I then used that to setup the RDS DB.

$ dropdb app -h my.rds.amazonaws.com -U superuser
Password: 
$ createdb app -h my.rds.amazonaws.com -U superuser
Password: 
$ psql -f db/app.ddl -h my.rds.amazonaws.com -U superuser -d app
Password: 

It completed but had the following errors sprinkled throughout:

psql:db/app.ddl:293: ERROR:  role "postgres" does not exist

Which is:

ALTER SCHEMA public OWNER TO postgres;
psql:db/app.ddl:340: ERROR:  must be owner of extension plpgsql

which is:

COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
psql:db/app.ddl:354: ERROR:  must be owner of extension pgcrypto

which is:

COMMENT ON EXTENSION pgcrypto IS 'cryptographic functions';

Now the same error that caused the deploy to fail:

psql:db/app.ddl:1363: ERROR:  syntax error at or near "AS"
LINE 2:     AS integer
            ^
psql:db/app.ddl:1366: ERROR:  relation "user_id_seq" does not exist
psql:db/app.ddl:1372: ERROR:  relation "user_id_seq" does not exist

Which is in (line 1363 is indicated below):

--
-- Name: user; Type: TABLE; Schema: data; Owner: superuser
--

CREATE TABLE "user" (
    id integer NOT NULL,
    name text NOT NULL,
    email text NOT NULL,
    password text NOT NULL,
    role user_role DEFAULT (settings.get('auth.default-role'::text))::user_role NOT NULL,
    CONSTRAINT user_email_check CHECK ((email ~* '^[A-Za-z0-9._%-]+@[A-Za-z0-9.-]+[.][A-Za-z]+$'::text)),
    CONSTRAINT user_name_check CHECK ((length(name) > 2))
);


ALTER TABLE "user" OWNER TO superuser;

--
-- Name: user_id_seq; Type: SEQUENCE; Schema: data; Owner: superuser
--

CREATE SEQUENCE user_id_seq
    AS integer          -- THIS IS LINE 1363 !!!!!*******
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;


ALTER TABLE user_id_seq OWNER TO superuser;

--
-- Name: user_id_seq; Type: SEQUENCE OWNED BY; Schema: data; Owner: superuser
--

ALTER SEQUENCE user_id_seq OWNED BY "user".id;
psql:db/app.ddl:1395: ERROR:  relation "user_id_seq" does not exist

Which is:

ALTER TABLE ONLY "user" ALTER COLUMN id SET DEFAULT nextval('user_id_seq'::regclass);
psql:db/app.ddl:1626: ERROR:  relation "user_id_seq" does not exist
LINE 1: SELECT pg_catalog.setval('user_id_seq', 1, false);

Which is:

SELECT pg_catalog.setval('user_id_seq', 1, false);

OK, next to see what happens with seed data (not sample_data, but I needed to put the seeds in the sample_data folder for the deploy. )........

....OK, all of the seeds succeeded! But what about those errors above?

from subzero-cli.

ruslantalpa avatar ruslantalpa commented on May 18, 2024

most probably you are using pg 10 on your dev box but your production pg is 9.6.
Make sure the image of pg is pinned to 9.6 like here https://github.com/subzerocloud/postgrest-starter-kit/blob/master/docker-compose.yml#L11
after the change, delete the migrations folder and start again.

the creation of migration file for the seed data is not created (it can't be), you need to create it manually like explained here https://docs.subzero.cloud/managing-migrations/#adding-data-to-migrations

other containers, no plans for now to make it a more general tool, not really a need for that, if you have other containers, you can setup a build pipeline that does all the steps at a lower level because in the end, all the deploy command does is build a openresty image and runs the migrations using sqitch, you can run those commands yourself.

since you have a fork, you can add a console.log around here https://github.com/subzerocloud/subzero-cli/blob/master/src/common.js#L46 to see how the dump files are being created.

in addition to subzero dashboard, there is also the watch command which is the reload functionality without the log interface.

from subzero-cli.

rkoberg avatar rkoberg commented on May 18, 2024

Yes. Local is 10.2. Ok, easy enough (but just did a db delete and started another DB create.... 30 min later...)

Thanks for the suggestions on the other stuff and thanks for a wonderful tool!

from subzero-cli.

ruslantalpa avatar ruslantalpa commented on May 18, 2024

👍
tell your friends :)))))

from subzero-cli.

rkoberg avatar rkoberg commented on May 18, 2024

I will. The deploy almost worked the structure got in, but failed at:

  + 0000000002-data .. psql:deploy/0000000002-data.sql:19: ERROR:  missing data for column "value"
CONTEXT:  COPY secrets, line 1: "jwt_lifetime '3600'"
not ok
"psql" unexpectedly returned exit value 3

Deploy failed

The line is jwt_lifetime 3600:

COPY secrets (key, value) FROM stdin;
jwt_lifetime 3600

The data type for the column is TEXT, so I tried to change it to jwt_lifetime '3600' but still no luck.

from subzero-cli.

rkoberg avatar rkoberg commented on May 18, 2024

This did not give errors:

$ psql -U superuser -f db/migrations/revert/0000000002-data.sql -d app -h my.rds.amazonaws.com
Password for user superuser: 
BEGIN
SET
TRUNCATE TABLE
COMMIT
$ 

from subzero-cli.

ruslantalpa avatar ruslantalpa commented on May 18, 2024

make sure the format there is the way COPY command expects it, i think the separator is TAB
look at this file for a sample https://github.com/subzerocloud/postgrest-starter-kit/blob/master/db/src/sample_data/data.sql

from subzero-cli.

ruslantalpa avatar ruslantalpa commented on May 18, 2024

you've probably copy/pasted the contents of data from http://docs.subzero.cloud/managing-migrations/
while i have it with tabs in my md files, when it gets transformed to html, (or at least at copy paste) they are transformed to spaces.

that is the problem you are seeing, copy needs tabs as separators

from subzero-cli.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.