As per the Slack discussion:

• Our use case is this: we want Gnorm to generate some files once; those files will then be manually edited.
• Some files Gnorm should always generate. Some, it should only generate if they don't already exist.
• it’s easy enough to say that if an output file matches a glob pattern (like "*.perm.go" or something), then don’t overwrite it.
• probably a list of paths to run through a glob parser is fine. NoOverwrite = ["*.perm.go", "*.xml"]

Because ::regclass returns the name of the table with the schema prepended except for tables in the public schema, parsing a Postgres table name fails.

I could not find a way to get config vars from gnorm.toml . For example, in one of my templates I needed the value of IncludeTables from gnorm.toml. Maybe something like {{.Config.IncludeTables}}. or {{config "IncludeTables"}}.

Case insensitive text type currently gets treated like an enum

Hey @natefinch . I'm keen to add support for Postgres comments on tables and columns etc. I see the readme has

It's useful to be able to blow away the whole generated directory, but then if there are any non-generated files in there, it can become problematic. If we had the ability to specify a "static" folder, like Hugo's static folder, then you'd be able to put non-generated files in there and feel comfortable just blowing away and regenerating with gnorm. This is especially useful if you change the filenames/layout that gets generated.

I had both hugo and statik at older versions in GOPATH. mage build would fail deep in the bowels of the hugo-build. go get -u did the trick in my case.

The .Indexes field on the Table struct in the templates has an inconsistent order. This is due, most likely, to using a map instead of a slice in database/drivers/*/parse.go. The database returns the rows in a consistent order.

It will also prevent regeneration of files from triggering diffs if there's no actual change in the data.

the gnorm preview command needs a bunch of tests to verify that it's doing the right thing

I'm not 100% sure what the equivalent is for Postgres, but the problem that prompted this issue doesn't exist there.

In MySQL, numeric columns can be unsigned. Unfortunately, the DATA_TYPE column does not report that (but the COLUMN_TYPE column does). It's impossible to have a uint or its variants in Go with the currently exposed data. It's therefore possible to send invalid values to MySQL (such as negatives).

Need to be able to define custom types for array columns

gnorm test foo.templ -> parse the template to make sure it's valid (no template typos).

gnorm test foo.templ --table foo -> parse the template and feed it table data from table foo (using the current gnorm.toml) to test the output (write to stdout). (also --schema and --enum)

When I tried to install via go get I get an import error.

$ go get -u -v github.com/gnormal/gnorm 
github.com/gnormal/gnorm (download)
package github.com/gnormal/gnorm: code in directory /Users/markbates/Dropbox/development/gocode/src/github.com/gnormal/gnorm expects import "gnorm.org/gnorm"

Since functions are a key part of making templates simpler to write, we want to add the ability for users of gnorm to implement their own functions to add to the template funcMap. The plugins are simple executables that are run by gnorm, which thus may be written in any language.

Template Function Plugins

Template function plugins are simple executables which implement the following conventions:

Listing available functions

• When run with the --list flag, plugins should write a newline-delimited list of supported function names to stdout.
• When run with a single argument, the argument is the function name being called. The arguments to the function are passed to the executable's stdin as a JSON encoded array.
• On success, a plugin should print the json-encoded results to stdout and exit with an exit code of zero.
• On failure (for example, if the arguments passed into the function are not valid for the function), the plugin should print an error to stderr and exit with an exit code of 1.
• If more than one argument is passed to the executable (i.e. more than just the function name), or if it is passed a function name that is not supported, the plugin should print an error to stderr and exit with an exit code of 2.
• If no arguments are passed to the executable, the executable should print some useful help message to stderr.

I would like to add generic template based filter functions to Columns, Tables, Indexes, and ForeignKeys. I find myself often wanting to filter these slices on one or more properties but I feel directly coding it each time I need a new filter will be cumbersome.

It would look something like:

func (c Columns) FilterT(t string) Columns {
  //for column
  //execute template t
  //if result is truthy
  //add to result
  //return result
}

At this point, every user can define templates that render a truthy or falsey string. Then, they can easily filter in templates like this:

{{$noDefaults := .Table.Columns.FilterT "noDefaultValue.gotmpl" -}}
{{$noAutoIncrement := .Table.Columns.FilterT "noAutoIncrement.gotmpl" -}}

Feedback or alternative solutions are welcome. I don't love this implementation but I feel it is better than adding a bunch of custom filters to the code base every time we want something different.

Right now, database/drivers/postgres/pg and database/drivers/mysql/pg is generated by github.com/xoxo-go/xoxo ... which is a fork of github.com/knq/xo ... this should be replaced by using gnorm itself to generate the code, which should be doable at this point.

Also currently, the mysql/pg code is just a copy of the postgres/pg code, because I couldn't get xoxo to generate it itself. This should be replaced by actual generated code for mysql (and renamed to not be in a package called "pg").

This will require updating the templates in drivers/postgres/template to conform to gnorm's data format and then changing the go generate command in database/drivers/postgres/parse.go (and adding one to mysql), and adding the appropriate gnorm.toml file in those directories as well.

Would be nice to have some default templates.

This would allow using db credentials without putting them in a config file.

^

The gnorm/database code needs live database tests (run mysql, use gnorm/database/mysql to query it, verify the results are as expected).

Tests that run against a live database should be marked with a build tag thusly:

// +build <database_name> 

So that they are only run when requested, since they require outside setup.

Ideally these tests would have idempotent functions to set up docker etc. so that first time devs have the easiest time to onboard.

https://github.com/google/skylark

The strings.Index func added to the func map in environ/funcs.go overwrites the text/template index func, making some operations impossible.

When setting Params in gnorm.toml, gnorm gen prints a warning. This warning comes from BurntSushi/toml's MetaData.Undecoded(): https://github.com/gnormal/gnorm/blob/master/cli/parse.go#L38-L41

The warning should be suppressed because Params is designed to be a bag of whatever.

Default config needs some tinkering:

• Set StaticDir = ""
• Enable typemaps by default*
• Drop schema out of path directives (public)
• Put all generated files in the same folder

Maybe also:

• Provide a demo.sql migration file

But certainly:

• Add CRUD templates that "just work"

*Also add the following types:

• "bigint" = "int64"
• "int8" = "int64"
• "double precision" = "float64"
• "jsonb" = "[]byte"
• "citext" = "string"
• ???

This one needs a little design work before implementation. Thoughts on the best way to represent foreign keys between tables? Probably metadata on a column... but maybe also metadata on a table? How to represent it in the most usable way?

It would be handy to be able sort things like indexes and columns (if not already sorted). This could be achieved by adding the funcs sort package (sort .Table.Indexes.Names; range and index .Table.IndexesByName) or by adding special purpose funcs on top of slice types (Indexes, Columns, ...).

Add IsAutoGenerated to Column

Would be nice for tables to hold information about primary keys.

Tests to ensure the CLI flag parsing does all the right stuff

I was trying to build the project and when I ran $ go run make.go I get no output from the command. I didn't know if it was successful or not.

Some output to let people know the status of the make script would be useful.

Hi,

it seems that custom "index" function overrides golang template "index" function for accessing slice elements. Maybe a rename would be OK solution?

Looks like a copy & paste error. Don't have enough time for a PR right now.

When generating code for enums, knowing the table can be useful.
For example, we could have 2 tables: contract and project, with each an enum named type.
In this case, if we generate a file for each enum in the same package, without the table name we get a collision.
Using the table name, we can generate 2 files / classes: ContractType and ProjectType.

In various places,
https://github.com/gnormal/gnorm/blob/master/CONTRIBUTING.md
https://github.com/gnormal/gnorm/blob/master/site/content/getting-started/_index.md
it calls to go run $GOPATH/src/gnorm.org/gnorm/make.go but there is no make.go anywhere in the project.

My guess is I should be using mage to build? Or running go run $GOPATH/src/gnorm.org/gnorm/main.go in the cases it tells me to check the binary builds.

Happy to submit a PR to update the documentation with either update.

indices should be represented as metadata on tables... though it might also be useful to have a boolean on columns to note if they're unique...

https://github.com/gnormal/gnorm/blob/master/site/content/gnorm-vs/_index.md

1. SqlBoiler only outputs Go code.
Gnorm outputs any langauge, even non-code things, like protobufs or docs.
SqlBoiler produces a fluent API style of code that is non-idiomatic for Go.
2. Gnorm can generate idiomatic go code.
SqlBoiler requires magic strings for queries (e.g. Where("age > ?", 30), which offer no type safety or compile-time checks. If you change the name or type of a column, that code won't fail until it's triggered by a test or in production.
3. Gnorm can generate type-safe queries that ensure that if a column's type or name change, your code will fail to compile.

Of these 1 and 3 are not true, as of version 3.0 you can generate whatever file types you want using sqlboiler using custom templates, and column names, relationship names (3.0 only), and table names are all generated as well which users can use as constants which will change when the database changes to ensure compiler errors. I'd appreciate if this text was rectified to be factually accurate as of the latest released version. Thanks!

would be nice to give the user the option to filter what tables they want code generated for. Let's say we have two configuration values, ExcludeTables and IncludeTables... you can define one or the other but not both. They're both just a list of strings. If you define IncludeTables, then only the listed tables are used to generate code. If you define ExcludeTables, then all tables in the DB except those listed will be used to generate code. Implementation can just be a filter after we query the DB... no need to filter the query itself, we're not returning so much data that a post-facto filter is going to hurt anything.

When running gnorm against a large schema, it can be daunting to figure out what types are being used and thus what values you need in the TypeMap. We should be able to make gnorm preview take a --types flag which then just outputs a consolidated list of types used in the schema.

It would be great to look at, for example, Hugo, and figure out how to add support for template languages other than just text/template. html/template would be trivial to add, but non-go templating languages would be good, too.

We want to make sure that we don't allow PRs to land without tests passing and all the generated code up to date. Once we have tests that need a database, this will start to involve running databases (most likely in docker)... but for now, just running go test is good enough.

With the latest master running gnorm gen I get

$ gnorm gen
Error: generating file for schema Public: error writing generated file "Public.go
": cannot replace "Public.go" with tempfile "/var/folders/ft/j4yq82wd4bd4s74gvmv4n_000000gn/T/Public.go436637883": rename /var/folders/ft/j4yq82wd4bd4s74gvmv4n_000000gn/T/Public.go436637883 Public.go: cross-device link
Usage:
  gnorm gen [flags]

Flags:
  -c, --config string   relative path to gnorm config file (default "gnorm.toml")
  -h, --help            help for gen
  -v, --verbose         show debugging output

Somehow this part

maybe i am doing soemthing wrong ?

go get -u gnorm.org/gnorm

go get github.com/magefile/mage

cd /Users/apple/workspace/go/src/gnorm.org/gnorm && mage build
mage.go:11:2: could not import github.com/magefile/mage/mg (can't find import: "gnorm.org/gnorm/vendor/github.com/magefile/mage/mg")
make: *** [build] Error 1

the gnorm gen command needs tests to ensure it's doing the right thing.

Error: generating file for schema public: error running postrun command ["echo" "..\models\public.go"]: exec: "echo":
executable file not found in %PATH%

gnorm.toml

PostRun = ["echo", "$GNORMFILE"]

windows 10
install

go get gnorm.org/gnorm

cd  gopath/src/gnorm.org/gnorm
go get github.com/magefile/mage
mage build

Give users the ability to define templates that are parsed as embedded templates in the main schema enum and table templates.
This can be a great way to swap out content in templates so as to support multiple databases more easily. A good example is how postgres vs other databases specify value place holders. postgres wants $1 $2 etc and others want ? ? . if you have a "values" template that defines how to render that, then instead of copying the whole table template for each database type, you just need to swap out the values template.

This is an important bit of information about an index. Fortunately, it's easy to find in both MySQL and Postgres.

When I run gnorm init in my directory, it panics about creating the gnorm.toml file.

Can't create gnorm.toml file: %!v(PANIC=runtime error: invalid memory address or nil pointer dereference)

Though the file is successfully created. It does however prevent the templates from being generated so I can't continue to use the tool.

Full trace:

peter@pete-thinkpad ~/projects/gnorm-sandbox % gnorm init
Can't create gnorm.toml file: %!v(PANIC=runtime error: invalid memory address or nil pointer dereference)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7103db]

goroutine 1 [running]:
gnorm.org/gnorm/cli.(*codeErr).Error(0xc4200dcf00, 0xc4200d07c0, 0xc4200d8060)
	<autogenerated>:1 +0x3b
gnorm.org/gnorm/cli.(*codeErr).Error(0xc4200dcf20, 0x9815f8, 0x0)
	<autogenerated>:1 +0x49
gnorm.org/gnorm/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200e8240, 0xc4200cfe60, 0xc4200d28d0, 0x0)
	/home/peter/projects/go/src/gnorm.org/gnorm/vendor/github.com/spf13/cobra/command.go:741 +0x4b7
gnorm.org/gnorm/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200e8240, 0xc4200cfe58, 0x1)
	/home/peter/projects/go/src/gnorm.org/gnorm/vendor/github.com/spf13/cobra/command.go:688 +0x2b
gnorm.org/gnorm/cli.ParseAndRun(0xc4200d0660, 0x1, 0x1, 0x9434a0, 0xc42000e020, 0x9434a0, 0xc42000e018, 0x943460, 0xc42000e010, 0xc4200d28d0, ...)
	/home/peter/projects/go/src/gnorm.org/gnorm/cli/run.go:56 +0x37c
gnorm.org/gnorm/cli.Run(0xc4200001a0)
	/home/peter/projects/go/src/gnorm.org/gnorm/cli/run.go:23 +0x1f4
main.main()
	/home/peter/projects/go/src/gnorm.org/gnorm/main.go:24 +0x22

Thanks