[Track] odoo_csv_import XOE Labs' interpretation about odoo_csv_import HOT 6 CLOSED

Hello,
Your link is not accessible, I guess this is a private repo.

I just check the click library which is basically odoo shell embedded in your python script, this is not at all the philosophy of my library.
The goal is to use the rpc call to parallelize calls and take advandage of the many worker available on the server.

I'm open to a patch that would allow to take the file to import from a pipe.
You can call https://github.com/tfrancoi/odoo_csv_import/blob/master/odoo_csv_tools/import_threaded.py#L174 import_data with a file descriptor for the parameter file_csv, and fail_file

from odoo_csv_import.

@tfrancoi Just fixed the link to the gist, sorry.

I'm still thinking about a basic design principle for most reusable components.

My idea was that click-odoo script, as an Odoo Server API extensions, could conduct DAG analysis for relations between multiple files and also analyse hierarchy tables to autosort them for internal dependencies (make group-by config obsolete)

The only requirement:

• click-odoo needs a copy of the target codebase in it's environment (which is rather realistic to provide)
• Note I depend on an ugly Odoo patch to reach that goal in my interpretation: https://gist.github.com/blaggacao/ba83a7e500c202cf842010cfddb25e15#file-utils-py-L186

Then, loading can be done using local computing resources (eg. a more powerful machine - I've loaded several million records in just over 4 days with my odoo_csv_import derivate *) and result can be sent directly to the (remote) database.

Especially the idea to provide individually scalable resources to the ETL workflow seems appealing to me.

* It would require temporary production scaling in most cases. Why production? because we usually load the accounting history with some delay after going live because the ETL process takes some while.

from odoo_csv_import.

I also think since ...

• odoo/odoo@b1e83fd
• odoo/odoo@d5b687a

... the default batch size can be significantly increased.

@tfrancoi In this context is groupby config still necessary??? -> I'm short on understanding on this one.

from odoo_csv_import.

@tfrancoi A possible CLI UX... acsone/click-odoo-contrib#15
There, we can join for a happy refactoring, should you be interested...

from odoo_csv_import.

@tfrancoi Would it be reasonable to assume that hierarchy tables should not be scheduled on multiple threads, but be collocated on a single thread to avoid complicated cross-thread-same-dataframe dependency locks? After all, hierarchy tables tend to be rather short.

from odoo_csv_import.

@tfrancoi (FYI)
This is what I've come up so far: https://pypi.org/project/click-odoo-loader/
And here is the source: https://github.com/xoe-labs/click-odoo-loader

from odoo_csv_import.