Comments (6)
Hello,
Your link is not accessible, I guess this is a private repo.
I just check the click library which is basically odoo shell embedded in your python script, this is not at all the philosophy of my library.
The goal is to use the rpc call to parallelize calls and take advandage of the many worker available on the server.
I'm open to a patch that would allow to take the file to import from a pipe.
You can call https://github.com/tfrancoi/odoo_csv_import/blob/master/odoo_csv_tools/import_threaded.py#L174 import_data with a file descriptor for the parameter file_csv, and fail_file
from odoo_csv_import.
@tfrancoi Just fixed the link to the gist, sorry.
I'm still thinking about a basic design principle for most reusable components.
My idea was that click-odoo script, as an Odoo Server API extensions, could conduct DAG analysis for relations between multiple files and also analyse hierarchy tables to autosort them for internal dependencies (make group-by config obsolete)
The only requirement:
- click-odoo needs a copy of the target codebase in it's environment (which is rather realistic to provide)
- Note I depend on an ugly Odoo patch to reach that goal in my interpretation: https://gist.github.com/blaggacao/ba83a7e500c202cf842010cfddb25e15#file-utils-py-L186
Then, loading can be done using local computing resources (eg. a more powerful machine - I've loaded several million records in just over 4 days with my odoo_csv_import derivate *) and result can be sent directly to the (remote) database.
Especially the idea to provide individually scalable resources to the ETL workflow seems appealing to me.
* It would require temporary production scaling in most cases. Why production? because we usually load the accounting history with some delay after going live because the ETL process takes some while.
from odoo_csv_import.
I also think since ...
... the default batch size can be significantly increased.
@tfrancoi In this context is groupby
config still necessary??? -> I'm short on understanding on this one.
from odoo_csv_import.
@tfrancoi A possible CLI UX... acsone/click-odoo-contrib#15
There, we can join for a happy refactoring, should you be interested...
from odoo_csv_import.
@tfrancoi Would it be reasonable to assume that hierarchy tables should not be scheduled on multiple threads, but be collocated on a single thread to avoid complicated cross-thread-same-dataframe dependency locks? After all, hierarchy tables tend to be rather short.
from odoo_csv_import.
@tfrancoi (FYI)
This is what I've come up so far: https://pypi.org/project/click-odoo-loader/
And here is the source: https://github.com/xoe-labs/click-odoo-loader
from odoo_csv_import.
Related Issues (20)
- ModuleNotFoundError: No module named 'lib' on import HOT 2
- Multilanguage HOT 1
- More record from one row HOT 3
- CSV id can not be recognized. ValueError: 'id' is not in list HOT 2
- [bug] Issue with https HOT 1
- [feature] Import image by url HOT 1
- Importing journal items. HOT 2
- [Exports] How to get ids ? HOT 2
- Found multiple matches for field 'State' (2 matches) HOT 2
- Importing large inventory adjustment results in timeout HOT 1
- Question: increased load when importing many lines HOT 2
- Found multiple matches for field 'Invoice Lines' (2 matches) - odoo 11
- I have a question. This module works for exporting data from Odoo.sh? HOT 1
- Can this tool be used to update initial inventory?
- Setting XML_ID for materialized relations HOT 2
- "Command not found" how can I fix? HOT 1
- [Export] res_currency is exported qith the field Active = True HOT 2
- How to get the existing xml_id? HOT 1
- Topostort Implementation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from odoo_csv_import.