Passing arguments more than 100 about pg_bulkload HOT 7 CLOSED

100 arguments not being supported is a PostgreSQL server limitation. pg_bulkload developers cannot do anything about it, unfortunately. But it seems you understand that.

You would need to find a way that doesn't require to use a function. Why do you need to use the function anyway?

from pg_bulkload.

I'm validating and transforming the data using SQL filter.For this Postgres function is required.Having more than 100 fields in source file.Doing transformation for nearly 20-30 fields.

Any other way to do it?

from pg_bulkload.

Well, you already said you hit a limitation of PostgreSQL functions (namely it cannot handle more than 100 arguments). So, you will have to perform any transformations on the input file before you use pg_bulkload, perhaps using some UNIX utilities.

The bottom line here is, to use pg_bulkload's FILTER capability, you have to rely on PostgreSQL functions.

from pg_bulkload.

Did the conversion using UNIX SED command from raja|0|100 to (raja,0,100). It's taking 5 minutes for 16 million data. But it's not ideal solution.

I think it is better to use some distributed processing, like hadoop or spark, to transform data for input format...

from pg_bulkload.

Thank you.Going ahead with UNIX approach.

from pg_bulkload.

Alright, OK to close the issue?

from pg_bulkload.

Sure. Thanks

from pg_bulkload.