lukasmartinelli / pgclimb Goto Github PK

View Code? Open in Web Editor NEW

388.0 388.0 39.0 109 KB

Export data from PostgreSQL into different data formats

License: MIT License

Go 77.58% Shell 22.42%

pgclimb's People

Contributors

Stargazers

Watchers

pgclimb's Issues

Export JSON without column name as root

@lukasmartinelli Import new line delimited JSON with pgfutter --host=localhost --port=5432 --schema=public json jsonfile and then export same JSON (no transformations) with pgclimb via $ pgclimb -c "select data from jsonfile;" jsonlines > jsonfile yields {"data":{"foo":"bar"}}.

How can I avoid the wrapping of the JSON in the column name? Have to shuffle large datasets in and out for transformations and re-query when an algorithm changes.

pgclimb should respect the usual PostgreSQL variables and .pgpass file

http://www.postgresql.org/docs/9.5/static/libpq-envars.html

I find it rather unsafe to either specify the password from the command-line, or use the environment variable.

Add "quote" option to csv

"quote" option is required to make this program perfect.

[feature request] data hub

This isn't an issue report, but a feature request.

Could you create something like a "data hub" based on your current tool to convert between the output formats of pgclimb? CSV to JSON, JSON to XML etc.? This would save time while re-running big queries stored, which results have been stored to JSON or XML and are needed in CSV or XLSX.

Anyways, good tool! 👍

Graphos format

Please export to RDF https://en.wikipedia.org/wiki/Resource_Description_Framework

export to xml and use of template

hi thanks for your pgclimb.
I'm trying to export data from my postgresql db to xml based on a specific template.
I've tried your pgclimb based on your exemple and its working fine however I'm trying to adapt the template format the way I need and I don't really understand how it shall be written ?
How shall I use the {{range}}, {{end}}, {{column_name}} parameters ?
Thanks

[bug] problem with export of numbers in text column

Hi, I have found one problem with export into jsonlines format. When I have column of type TEXT and this column in some cases contains only number then this number is exported as number and not quoted. Bigquery refuses to import such a record. Example "1111 blabla" is exported with quotes but value "1111" is exported without quotes. Thanks Josef

[question] order of columns in export

Hi, looks like you always export columns in alphabetical order (jsonline format). Would it be possible to preserve order in table or to add some switch into command line? Thanks. Josef

[feature request] json schema of table structure

Hi, this is something "nice to have". Would it be possible to add feature to export json schema of table structure? I created my own select of course but this feature will be move of pgclimb to higher level from my point of view. Thanks Josef

Pglclimb crashes when running sql file

When I try to run a particular sql script and output the results to csv, I get the following error:

panic: runtime error: slice bounds out of range [recovered]
panic: runtime error: slice bounds out of range

goroutine 1 [running]:
github.com/lib/pq.(*conn).errRecover(0xc082068280, 0xc082058f90)
		/gopath/src/github.com/lib/pq/error.go:482 +0x198
github.com/lib/pq.expect(0xc0820028a0, 0xd, 0x7cadf0, 0x1, 0xd)
		/gopath/src/github.com/lib/pq/encode.go:199 +0x263
github.com/lib/pq.parseTs(0x0, 0xc0820028a0, 0xd, 0x0, 0x0)
		/gopath/src/github.com/lib/pq/encode.go:323 +0x4e8
github.com/lib/pq.textDecode(0xc0820684a8, 0xc0820682da, 0xd, 0x1c6, 0xc00000043a, 0x0, 0x0)
		/gopath/src/github.com/lib/pq/encode.go:91 +0x559
github.com/lib/pq.decode(0xc0820684a8, 0xc0820682da, 0xd, 0x1c6, 0x43a, 0x0, 0x0, 0x0)
		/gopath/src/github.com/lib/pq/encode.go:62 +0xa7
github.com/lib/pq.(*rows).Next(0xc082010070, 0xc082060160, 0x15, 0x15, 0x0, 0x0)

		/gopath/src/github.com/lib/pq/conn.go:1457 +0x4d9
database/sql.(*Rows).Next(0xc08202a360, 0xc082060000)
		/goroot/src/database/sql/sql.go:1592 +0x135
github.com/lukasmartinelli/pgclimb/formats.Export(0xc08209c000, 0x9bc, 0xc082096080, 0x78, 0x1c2090, 0xc0820624e0, 0x0, 0x0)
		/gopath/src/github.com/lukasmartinelli/pgclimb/formats/export.go:35 +0x258
main.exportFormat(0xc082064280, 0x1c2090, 0xc0820624e0)
		/usr/src/pgclimb/pgclimb.go:45 +0xcd
main.func┬╖004(0xc082064280)
		/usr/src/pgclimb/pgclimb.go:190 +0x11a
github.com/codegangsta/cli.Command.Run(0x7d5130, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7e6350, 0xa, 0x0, ...)
		/gopath/src/github.com/codegangsta/cli/command.go:174 +0x140a
github.com/codegangsta/cli.(*App).Run(0xc082064000, 0xc08202c000, 0xe, 0xe, 0x0, 0x0)
		/gopath/src/github.com/codegangsta/cli/app.go:187 +0x1228
main.main()
		/usr/src/pgclimb/pgclimb.go:240 +0xe0c

goroutine 5 [runnable]:
database/sql.(*DB).connectionOpener(0xc0820940a0)
		/goroot/src/database/sql/sql.go:589 +0x53
created by database/sql.Open
		/goroot/src/database/sql/sql.go:452 +0x323

I'm not sure what particular information you require in order to make sense of this error, but for starters I was running pgclimb_windows_amd64 on a Postgres 9.4 database.

This is the query I was trying to run:
query what causes the crash.txt

More doc on templates

Are these default go templates? Is there a way to get the index and the length in order to generate some datastructure where there is no comma after the last entry?

basic csv export command

Is this the correct syntax for pgclimb? I have tried several different iterations of this command, but continue to get the "Incorrect Usage" error.

./pgclimb csv --header --host "xxx.xxx.xxx.xxx" --dbname "database" --username "user" --pass "pass" -o test.csv -c "select * from public.test"

Ive also tried a local version of postgres and still no success. Am I missing something?

Thanks!

fatal error on install

Running on macOS High Sierra, Version 10.13.5

Downloaded the binary from https://github.com/lukasmartinelli/pgclimb/releases/download/v0.2/pgclimb_darwin_amd64

...on July 3, 2018. chmod +x and then running help generates the following...

Wills-iMac:pgclimb wiverson$ ./pgclimb --help
failed MSpanList_Insert 0x62c000 0x5317f0d6c0e57 0x0
fatal error: MSpanList_Insert

runtime stack:
runtime.throw(0x59560b)
	/goroot/src/runtime/panic.go:491 +0xad fp=0x7ffeefbff700 sp=0x7ffeefbff6d0
runtime.MSpanList_Insert(0x5bcca8, 0x62c000)
	/goroot/src/runtime/mheap.c:692 +0x8f fp=0x7ffeefbff728 sp=0x7ffeefbff700
MHeap_FreeSpanLocked(0x5b98a0, 0x62c000, 0x100)
	/goroot/src/runtime/mheap.c:583 +0x163 fp=0x7ffeefbff768 sp=0x7ffeefbff728
MHeap_Grow(0x5b98a0, 0x8, 0x0)
	/goroot/src/runtime/mheap.c:420 +0x1a8 fp=0x7ffeefbff7a8 sp=0x7ffeefbff768
MHeap_AllocSpanLocked(0x5b98a0, 0x1, 0x0)
	/goroot/src/runtime/mheap.c:298 +0x365 fp=0x7ffeefbff7e8 sp=0x7ffeefbff7a8
mheap_alloc(0x5b98a0, 0x1, 0x12, 0x0)
	/goroot/src/runtime/mheap.c:190 +0x121 fp=0x7ffeefbff810 sp=0x7ffeefbff7e8
runtime.MHeap_Alloc(0x5b98a0, 0x1, 0x10000000012, 0xee59)
	/goroot/src/runtime/mheap.c:240 +0x66 fp=0x7ffeefbff848 sp=0x7ffeefbff810
MCentral_Grow(0x5c1618, 0x0)
	/goroot/src/runtime/mcentral.c:197 +0x8b fp=0x7ffeefbff8b0 sp=0x7ffeefbff848
runtime.MCentral_CacheSpan(0x5c1618, 0x0)
	/goroot/src/runtime/mcentral.c:85 +0x167 fp=0x7ffeefbff8e8 sp=0x7ffeefbff8b0
runtime.MCache_Refill(0x628000, 0x12, 0x0)
	/goroot/src/runtime/mcache.c:90 +0xa0 fp=0x7ffeefbff910 sp=0x7ffeefbff8e8
runtime.mcacheRefill_m()
	/goroot/src/runtime/malloc.c:368 +0x57 fp=0x7ffeefbff930 sp=0x7ffeefbff910
runtime.onM(0x4885e0)
	/goroot/src/runtime/asm_amd64.s:273 +0x9a fp=0x7ffeefbff938 sp=0x7ffeefbff930
runtime.mallocgc(0x120, 0x3b3680, 0x0, 0x0)
	/goroot/src/runtime/malloc.go:178 +0x849 fp=0x7ffeefbff9e8 sp=0x7ffeefbff938
runtime.newobject(0x3b3680, 0x628000)
	/goroot/src/runtime/malloc.go:353 +0x49 fp=0x7ffeefbffa10 sp=0x7ffeefbff9e8
runtime.newG(0x27dfa)
	/goroot/src/runtime/proc.go:233 +0x2a fp=0x7ffeefbffa28 sp=0x7ffeefbffa10
allocg(0x5aa000)
	/goroot/src/runtime/proc.c:925 +0x1f fp=0x7ffeefbffa38 sp=0x7ffeefbffa28
runtime.malg(0x8000, 0x5aa1c0)
	/goroot/src/runtime/proc.c:2106 +0x1f fp=0x7ffeefbffa68 sp=0x7ffeefbffa38
runtime.mpreinit(0x5aa580)
	/goroot/src/runtime/os_darwin.c:137 +0x27 fp=0x7ffeefbffa80 sp=0x7ffeefbffa68
mcommoninit(0x5aa580)
	/goroot/src/runtime/proc.c:201 +0xc9 fp=0x7ffeefbffaa8 sp=0x7ffeefbffa80
runtime.schedinit()
	/goroot/src/runtime/proc.c:138 +0x55 fp=0x7ffeefbffad0 sp=0x7ffeefbffaa8
runtime.rt0_go(0x7ffeefbffb08, 0x2, 0x7ffeefbffb08, 0x0, 0x0, 0x2, 0x7ffeefbffc20, 0x7ffeefbffc2a, 0x0, 0x7ffeefbffc31, ...)
	/goroot/src/runtime/asm_amd64.s:95 +0x116 fp=0x7ffeefbffad8 sp=0x7ffeefbffad0
Wills-iMac:pgclimb wiverson$

csv: null values not exported

We exported a csv with some null values and found, for example:

Table with 3 columns (int, varchar, varchar).

Normal:

1,a,dog

Bad:

2,cat

The Normal is expected, in the Bad I would instead expect 2,,cat.

Googling around I found at least one other person who has hit this issue: OSMNames/OSMNames#44 (comment)

lukasmartinelli / pgclimb Goto Github PK

pgclimb's People

Contributors

Stargazers

Watchers

Forkers

pgclimb's Issues

Recommend Projects

Recommend Topics

Recommend Org