Comments (10)
We will also have another container that is kind of similar to the way Openresty is used, but it is a golang container for massive concurrency. The client app will post to the golang endpoint which sends to a redismq to queue things up to send to redis. Then the redis data goes back to the golang container which puts a CSV batch together to send to PostgREST. So there will be three extra containers there with one custon one (the golang endpoint). We wanted to use the subzer-cli to handle these. Are there any plpans to make the cli a more generic tool that can deploy.
from subzero-cli.
Next I tried to dump everything so I could manually setup the DB, but that did not work:
template1=# GRANT ALL PRIVILEGES ON DATABASE app to rkoberg;
GRANT
Got an error:
$ pg_dump -c --if-exists -f app.ddl -h 0.0.0.0 -p 5432 -U rkoberg -d app
Password:
pg_dump: [archiver (db)] query failed: ERROR: permission denied for schema settings
pg_dump: [archiver (db)] query was: LOCK TABLE settings.secrets IN ACCESS SHARE MODE
So,
template1=# \c app
You are now connected to database "app" as user "postgres".
app=# GRANT ALL PRIVILEGES ON SCHEMA auth to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA settings to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA pgjwt to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA request to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA util to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA data to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON SCHEMA api to rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA auth TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA settings TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA pgjwt TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA request TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA util TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA data TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA api TO rkoberg;
GRANT
app=# GRANT ALL PRIVILEGES ON DATABASE app TO rkoberg;
GRANT
Then I tried a dump again, but:
$ pg_dump -c --if-exists -f app.ddl -h 0.0.0.0 -p 5432 -U rkoberg -d app
Password:
pg_dump: [archiver (db)] query failed: ERROR: permission denied for relation user_id_seq
pg_dump: [archiver (db)] query was: SELECT last_value, is_called FROM user_id_seq
So,
app=# GRANT ALL PRIVILEGES ON SEQUENCE user_id_seq TO rkoberg;
ERROR: relation "user_id_seq" does not exist
How do you get a full dump? Or any other ideas?
from subzero-cli.
I was able to get a dump as the postgres user from inside the local container. I then used that to setup the RDS DB.
$ dropdb app -h my.rds.amazonaws.com -U superuser
Password:
$ createdb app -h my.rds.amazonaws.com -U superuser
Password:
$ psql -f db/app.ddl -h my.rds.amazonaws.com -U superuser -d app
Password:
It completed but had the following errors sprinkled throughout:
psql:db/app.ddl:293: ERROR: role "postgres" does not exist
Which is:
ALTER SCHEMA public OWNER TO postgres;
psql:db/app.ddl:340: ERROR: must be owner of extension plpgsql
which is:
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
psql:db/app.ddl:354: ERROR: must be owner of extension pgcrypto
which is:
COMMENT ON EXTENSION pgcrypto IS 'cryptographic functions';
Now the same error that caused the deploy to fail:
psql:db/app.ddl:1363: ERROR: syntax error at or near "AS"
LINE 2: AS integer
^
psql:db/app.ddl:1366: ERROR: relation "user_id_seq" does not exist
psql:db/app.ddl:1372: ERROR: relation "user_id_seq" does not exist
Which is in (line 1363 is indicated below):
--
-- Name: user; Type: TABLE; Schema: data; Owner: superuser
--
CREATE TABLE "user" (
id integer NOT NULL,
name text NOT NULL,
email text NOT NULL,
password text NOT NULL,
role user_role DEFAULT (settings.get('auth.default-role'::text))::user_role NOT NULL,
CONSTRAINT user_email_check CHECK ((email ~* '^[A-Za-z0-9._%-]+@[A-Za-z0-9.-]+[.][A-Za-z]+$'::text)),
CONSTRAINT user_name_check CHECK ((length(name) > 2))
);
ALTER TABLE "user" OWNER TO superuser;
--
-- Name: user_id_seq; Type: SEQUENCE; Schema: data; Owner: superuser
--
CREATE SEQUENCE user_id_seq
AS integer -- THIS IS LINE 1363 !!!!!*******
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE user_id_seq OWNER TO superuser;
--
-- Name: user_id_seq; Type: SEQUENCE OWNED BY; Schema: data; Owner: superuser
--
ALTER SEQUENCE user_id_seq OWNED BY "user".id;
psql:db/app.ddl:1395: ERROR: relation "user_id_seq" does not exist
Which is:
ALTER TABLE ONLY "user" ALTER COLUMN id SET DEFAULT nextval('user_id_seq'::regclass);
psql:db/app.ddl:1626: ERROR: relation "user_id_seq" does not exist
LINE 1: SELECT pg_catalog.setval('user_id_seq', 1, false);
Which is:
SELECT pg_catalog.setval('user_id_seq', 1, false);
OK, next to see what happens with seed data (not sample_data, but I needed to put the seeds in the sample_data folder for the deploy. )........
....OK, all of the seeds succeeded! But what about those errors above?
from subzero-cli.
most probably you are using pg 10 on your dev box but your production pg is 9.6.
Make sure the image of pg is pinned to 9.6 like here https://github.com/subzerocloud/postgrest-starter-kit/blob/master/docker-compose.yml#L11
after the change, delete the migrations folder and start again.
the creation of migration file for the seed data is not created (it can't be), you need to create it manually like explained here https://docs.subzero.cloud/managing-migrations/#adding-data-to-migrations
other containers, no plans for now to make it a more general tool, not really a need for that, if you have other containers, you can setup a build pipeline that does all the steps at a lower level because in the end, all the deploy command does is build a openresty image and runs the migrations using sqitch, you can run those commands yourself.
since you have a fork, you can add a console.log around here https://github.com/subzerocloud/subzero-cli/blob/master/src/common.js#L46 to see how the dump files are being created.
in addition to subzero dashboard, there is also the watch command which is the reload functionality without the log interface.
from subzero-cli.
Yes. Local is 10.2. Ok, easy enough (but just did a db delete and started another DB create.... 30 min later...)
Thanks for the suggestions on the other stuff and thanks for a wonderful tool!
from subzero-cli.
👍
tell your friends :)))))
from subzero-cli.
I will. The deploy almost worked the structure got in, but failed at:
+ 0000000002-data .. psql:deploy/0000000002-data.sql:19: ERROR: missing data for column "value"
CONTEXT: COPY secrets, line 1: "jwt_lifetime '3600'"
not ok
"psql" unexpectedly returned exit value 3
Deploy failed
The line is jwt_lifetime 3600
:
COPY secrets (key, value) FROM stdin;
jwt_lifetime 3600
The data type for the column is TEXT, so I tried to change it to jwt_lifetime '3600'
but still no luck.
from subzero-cli.
This did not give errors:
$ psql -U superuser -f db/migrations/revert/0000000002-data.sql -d app -h my.rds.amazonaws.com
Password for user superuser:
BEGIN
SET
TRUNCATE TABLE
COMMIT
$
from subzero-cli.
make sure the format there is the way COPY command expects it, i think the separator is TAB
look at this file for a sample https://github.com/subzerocloud/postgrest-starter-kit/blob/master/db/src/sample_data/data.sql
from subzero-cli.
you've probably copy/pasted the contents of data from http://docs.subzero.cloud/managing-migrations/
while i have it with tabs in my md files, when it gets transformed to html, (or at least at copy paste) they are transformed to spaces.
that is the problem you are seeing, copy needs tabs as separators
from subzero-cli.
Related Issues (20)
- clear log
- Error with base-project & manage your database structure/migrations here set to NO HOT 1
- feature request: reload schema button HOT 4
- Error when resetting db HOT 1
- init script not working on windows HOT 11
- Integrate with other graphql tools HOT 1
- Error when installing via npm on OSX HOT 5
- subzero-base-project doing nothing on windows HOT 4
- error in migrations init
- add lock file HOT 1
- Migrations fail on Windows + Node 13.x HOT 1
- Migrations are missing roles HOT 2
- resetDb only drop the database but the created role will not HOT 2
- docker containers do not run on raspberry pi HOT 2
- Migrations missing grants after update from 0.1.40 to 0.2.1 HOT 2
- Missing authenticator role when generating init migration script using subzero HOT 4
- "subzero-migrations add" command is not working for a table with partition HOT 5
- subzero-migrations add HOT 5
- Prefer externally given value for APP_DIR instead of .env file path HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from subzero-cli.