techdivision / import Goto Github PK

This is a library that provides generic functionalities for the implementation of imports. In addition to maximum performance and optimized memory consumption, Pacemaker can also be used to implement imports in distributed scenarios that place the highest demands on speed and stability.

Home Page: https://pacemaker.techdivision.com

License: MIT License

PHP 100.00%

magento import products categories csv attributes pacemaker

import's Introduction

Pacemaker - Import Framework

Please visit the Pacemaker website or our documentation for additional information

import's People

Contributors

Stargazers

Watchers

import's Issues

Use the import in Magento 2 module

Hello,
it will be nice if there is a way to use the import module in custom Magento 2 module.

For example let's have this scenario.
There is 3rd party ERP system that provides products, images, categories with REST API.
I've build Magento 2 module that reads the data and transform it in an array.

When we have Array with product data I want to use some Import Adapter from this module and run something like that $this->m2ifImport->import($array)

Thanks :)

Logging Handler to log Serial

A Monolog Handler will be useful to add the import serial to always have the actual serial in the log message context.

Better exception logging

While importing products through a errorneous CSV file missing the SKU of a product, the pipeline step product_import threw an exception and the pipeline errored.
the error:

[2019-10-04 08:45:33] executor.INFO: 
In AbstractSubject.php line 1205:

  Found not supported data type "NULL" when preparing cache key in file /home/weigla/projects/magento2/project/src/var/pacemaker/pipelines/193/product-import_20191002-095901_01.csv on line 3901


In CacheKeyUtil.php line 88:

  Found not supported data type "NULL" when preparing cache key

import:products [--serial SERIAL] [--installation-dir INSTALLATION-DIR] [--system-name SYSTEM-NAME] [--pid-filename PID-FILENAME] [--cache-enabled CACHE-ENABLED] [--magento-edition MAGENTO-EDITION] [--magento-version MAGENTO-VERSION] [--configuration CONFIGURATION] [--source-dir SOURCE-DIR] [--target-dir TARGET-DIR] [--archive-dir ARCHIVE-DIR] [--archive-artefacts ARCHIVE-ARTEFACTS] [--source-date-format SOURCE-DATE-FORMAT] [--use-db-id USE-DB-ID] [--db-pdo-dsn DB-PDO-DSN] [--db-username DB-USERNAME] [--db-password DB-PASSWORD] [--log-level LOG-LEVEL] [--debug-mode DEBUG-MODE] [--single-transaction SINGLE-TRANSACTION] [--params PARAMS] [--params-file PARAMS-FILE] [--] [<operation-name>]

[EXCEPTION] /home/weigla/projects/magento2/project/src/vendor/techdivision/process-pipelines/Model/Executor/AbstractShellExecutor.php: Application run failed with error. See log for more information.%

the log file

Exception: Found not supported data type "NULL" when preparing cache key in /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Utils/CacheKeyUtil.php:88
Stack trace:
#0 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Cache/LocalCacheAdapter.php(98): TechDivision\Import\Utils\CacheKeyUtil->cacheKey(NULL)
#1 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Cache/LocalCacheAdapter.php(110): TechDivision\Import\Cache\LocalCacheAdapter->cacheKey(NULL)
#2 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Cache/ConfigurableCacheAdapter.php(222): TechDivision\Import\Cache\LocalCacheAdapter->isCached(NULL)
#3 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-product/src/Repositories/ProductRepository.php(147): TechDivision\Import\Cache\ConfigurableCacheAdapter->isCached(NULL)
#4 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-product/src/Services/ProductBunchProcessor.php(942): TechDivision\Import\Product\Repositories\ProductRepository->findOneBySku(NULL)
#5 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-product/src/Observers/UrlKeyObserver.php(150): TechDivision\Import\Product\Services\ProductBunchProcessor->loadProduct(NULL)
#6 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-product/src/Observers/UrlKeyObserver.php(93): TechDivision\Import\Product\Observers\UrlKeyObserver->loadProduct(NULL)
#7 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-product/src/Observers/AbstractProductImportObserver.php(56): TechDivision\Import\Product\Observers\UrlKeyObserver->process()
#8 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Observers/GenericCompositeObserver.php(95): TechDivision\Import\Product\Observers\AbstractProductImportObserver->handle(Object(TechDivision\Import\Product\Ee\Subjects\EeBunchSubject))
#9 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Subjects/AbstractSubject.php(813): TechDivision\Import\Observers\GenericCompositeObserver->handle(Object(TechDivision\Import\Product\Ee\Subjects\EeBunchSubject))
#10 [internal function]: TechDivision\Import\Subjects\AbstractSubject->importRow(Array)
#11 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(149): call_user_func(Array, Array)
#12 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(135): TechDivision\Import\Adapter\Goodby\Interpreter->delegate(Array, Array)
#13 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(74): TechDivision\Import\Adapter\Goodby\Interpreter->notify(Array)
#14 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Adapter/Goodby/Lexer.php(120): TechDivision\Import\Adapter\Goodby\Interpreter->interpret(Array)
#15 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Adapter/CsvImportAdapter.php(142): TechDivision\Import\Adapter\Goodby\Lexer->parse('/home/weigla/pr...', Object(TechDivision\Import\Adapter\Goodby\Interpreter))
#16 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Subjects/AbstractSubject.php(722): TechDivision\Import\Adapter\CsvImportAdapter->import(Array, '/home/weigla/pr...')
#17 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Plugins/SubjectExecutor.php(127): TechDivision\Import\Subjects\AbstractSubject->import('193', '/home/weigla/pr...')
#18 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(186): TechDivision\Import\Plugins\SubjectExecutor->execute(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject), Array, '193', '/home/weigla/pr...')
#19 /home/weigla/projects/magento2/project/src/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(121): TechDivision\Import\Plugins\SubjectPlugin->processSubject(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject))
#20 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-app-simple/src/Simple.php(647): TechDivision\Import\Plugins\SubjectPlugin->process()
#21 /home/weigla/projects/magento2/project/src/vendor/techdivision/import-cli/src/Command/AbstractImportCommand.php(108): TechDivision\Import\App\Simple->process('193')
#22 /home/weigla/projects/magento2/project/src/vendor/symfony/console/Command/Command.php(255): TechDivision\Import\Cli\Command\AbstractImportCommand->execute(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /home/weigla/projects/magento2/project/src/vendor/symfony/console/Application.php(893): Symfony\Component\Console\Command\Command->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /home/weigla/projects/magento2/project/src/vendor/symfony/console/Application.php(262): Symfony\Component\Console\Application->doRunCommand(Object(TechDivision\Import\Cli\Command\ImportProductsCommand), Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /home/weigla/projects/magento2/project/src/vendor/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /home/weigla/projects/magento2/project/src/vendor/techdivision/pacemaker-import-m2if/bootstrap.php(56): Symfony\Component\Console\Application->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /home/weigla/projects/magento2/project/src/vendor/techdivision/pacemaker-import-m2if/bin/import-m2if.php(54): require('/home/weigla/pr...')
#28 /home/weigla/projects/magento2/project/src/vendor/techdivision/pacemaker-import-m2if/bin/import-m2if(2): require('/home/weigla/pr...')
#29 {main}

The specified line in the log, doesn't match with the CSV line where the product is missing the SKU.

I would suggest implementing a feature which would check the mandatory fields specified in a configuration file and if the field is empty in a row throw an exception, so that it's easier to debug faulty CSV import files.

url_rewrite key constraint violation when the database has old rewrites

Describe the bug

If there are old rewrites, the import might fail.

To Reproduce
Steps to reproduce the behavior:

Have a url_rewrite "example" in the database, of a product which no longer exists, rewrite type = 0
Import product with the same name
You might get an error

Next TechDivision\Import\Exceptions\WrappedColumnException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'produkte/example.html-11' for key 'URL_REWRITE_REQUEST_PATH_STORE_ID' when executing SQL "INSERT
	       INTO url_rewrite
	            (entity_type,
	             entity_id,
	             request_path,
	             target_path,
	             redirect_type,
	             store_id,
	             description,
	             is_autogenerated,
	             metadata)
	     VALUES (category,
	             967,
	             produkte/example.html,
	             catalog/category/view/id/967,
	             0,
	             11,
	             ,
	             1,
	             )" in file category-create_20210825_1.csv on line 3117 in /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Subjects/AbstractSubject.php:1461
Stack trace:
#0 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Subjects/AbstractSubject.php(973): TechDivision\Import\Subjects\AbstractSubject->wrapException(Array, Object(PDOException))
#1 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Subjects/SubjectExecutor.php(131): TechDivision\Import\Subjects\AbstractSubject->import('bcc42b1f-cccc-4...', 'var/importexpor...')
#2 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(168): TechDivision\Import\Subjects\SubjectExecutor->execute(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject), Array, 'bcc42b1f-cccc-4...', 'var/importexpor...')
#3 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(120): TechDivision\Import\Plugins\SubjectPlugin->processSubject(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject))
#4 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Plugins/PluginExecutor.php(117): TechDivision\Import\Plugins\SubjectPlugin->process()
#5 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import/src/Modules/PluginModule.php(144): TechDivision\Import\Plugins\PluginExecutor->execute(Object(TechDivision\Import\Configuration\Jms\Configuration\Plugin))
#6 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import-app-simple/src/Simple.php(622): TechDivision\Import\Modules\PluginModule->process()
#7 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import-cli/src/Command/AbstractImportCommand.php(116): TechDivision\Import\App\Simple->process('bcc42b1f-cccc-4...')
#8 /var/www/html/shop.example.com.staging.imi.de/web/vendor/symfony/console/Command/Command.php(255): TechDivision\Import\Cli\Command\AbstractImportCommand->execute(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /var/www/html/shop.example.com.staging.imi.de/web/vendor/symfony/console/Application.php(893): Symfony\Component\Console\Command\Command->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/html/shop.example.com.staging.imi.de/web/vendor/symfony/console/Application.php(262): Symfony\Component\Console\Application->doRunCommand(Object(TechDivision\Import\Cli\Command\ImportCategoriesCommand), Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /var/www/html/shop.example.com.staging.imi.de/web/vendor/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import-cli-simple/bootstrap.php(53): Symfony\Component\Console\Application->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import-cli-simple/bin/pacemaker.php(54): require('/var/www/html/s...')
#14 /var/www/html/shop.example.com.staging.imi.de/web/vendor/techdivision/import-cli-simple/bin/import-simple(2): require('/var/www/html/s...')
#15 {main}

Expected behavior
No error should appear. Maybe a new URL key should be generated.
Or this is a won't fix and the workarkound (below) is to be applied.
I am not sure if the URL keys are leftovers of a delete-import, so actually the issue (if any) might be in the delete job.

** Version Info **

$ composer show techdivision/*
techdivision/import                             dev-master 3002727
techdivision/import-app-simple                  dev-master ff102a0
techdivision/import-attribute                   dev-master 48d1f13
techdivision/import-attribute-set               dev-master 7fc309a
techdivision/import-cache                       dev-master 2ad65f8
techdivision/import-cache-collection            dev-master 0e9f9c5
techdivision/import-category                    dev-master 9207358
techdivision/import-category-ee                 dev-master 90f3efb
techdivision/import-cli                         dev-master 3262e99
techdivision/import-cli-simple                  4.0.0-alpha9      
techdivision/import-configuration               dev-master b28d114
techdivision/import-configuration-jms           dev-master 6bd1de4
techdivision/import-converter                   dev-master 0e78fd7
techdivision/import-converter-ee                dev-master 947dfe9
techdivision/import-converter-product-attribute dev-master 22c46b7
techdivision/import-converter-product-category  dev-master 23c616f
techdivision/import-customer                    dev-master ff9aab4
techdivision/import-customer-address            dev-master 38c8f42
techdivision/import-dbal                        dev-master eaf7a3c
techdivision/import-dbal-collection             dev-master 0165c12
techdivision/import-ee                          dev-master 5402a22
techdivision/import-product                     dev-master fb9441f
techdivision/import-product-bundle              dev-master fb81e91
techdivision/import-product-bundle-ee           dev-master 1becedf
techdivision/import-product-ee                  dev-master 65971a7
techdivision/import-product-grouped             dev-master 0d986da
techdivision/import-product-grouped-ee          dev-master 0dc2c86
techdivision/import-product-link                dev-master aa7b39d
techdivision/import-product-link-ee             dev-master 18f5da3
techdivision/import-product-media               dev-master 1788e34
techdivision/import-product-media-ee            dev-master a3600c7
techdivision/import-product-msi                 dev-master 4120443
techdivision/import-product-tier-price          dev-master 2f990dc
techdivision/import-product-url-rewrite         dev-master 23ad32c
techdivision/import-product-variant             dev-master 4bb06c0
techdivision/import-product-variant-ee          dev-master b1a7354
techdivision/import-serializer                  dev-master 995e519
techdivision/import-serializer-csv              dev-master 9f0a428

Magento 2.3.4-p2
Additional context
Add any other context about the problem here.

** Possible Workarkound **

Something like this might help:

> delete from url_rewrite where entity_type = 'category' and redirect_type = 0 AND entity_id not in (SELECT entity_id FROM catalog_category_entity)
    -> ;
Query OK, 806 rows affected, 1 warning (0.12 sec)

But could lead to new errors, for example on product import.

Change default import directory from var/importexport to var/pacemaker/import

Wrong Folder permission after failure

Preconditions (*)

Magento 2.2
techdivision/import: 6.0.1

Steps to reproduce (*)

Run an failed Import wron csv format

Expected result (*)

drwxrwxrwx+ 2 www-data www-data 4096 May 21 23:11 df79607b-9be8-4f7b-9e20-b68768b50002

Actual result (*)

drwx------+ 2 www-data www-data 4096 May 21 13:53 df79607b-9be8-4f7b-9e20-b68768b50002

Add functionality to pass an exit code when stopping the application

Currently the exit code is always 1 when calling \TechDivision\Import\ApplicationInterface::stop. It would be nice to have an option to stop the application with exit code 0.

Refactor PAC-96 image override option

PAC-96 introduced a new option which allows overriding already existing images. In order to stay backwards compabile, new methods and attributes were made private in
src/Subjects/FileUploadTrait.php. These should be either public or protected.

Additionally, a new constant OVERRIDE_IMAGES should be introduced in the ConfigurationKeys utility class for components using the new option:

techdivision/import-attribute
- src/Subjects/OptionSubject.php
- src/Utils/ConfigurationKeys.php
techdivision/import-category
- src/Subjects/BunchSubject.php
- src/Utils/ConfigurationKeys.php
techdivision/import-product
- src/Subjects/BunchSubject.php
- src/Utils/ConfigurationKeys.php

The necessary changes are highligted as "TODOs" in the corresponding files.

Stricter validation of column 'categories' in product import

The validation of categories in the product import should be more specific and provide meaningful error messages.
For example importing a row containting a single forward slash / as value for column 'categories' results in a cryptic exception.

More information: #194

Inject CSV file lexer into AbstractSubject

The CSV file lexer is actually directly instanciated in the AbstractSubject class, which makes testing complicated. The lexer has to be injected.

Call to undefined method TechDivision\Import\Product\UrlRewrite\Services\ProductUrlRewriteProcessor::removeUrlRewrite()

Issue
When running the product import I am receiving the following error message. I am importing one product that already exists in Magento.

PHP Fatal error: Uncaught Error: Call to undefined method TechDivision\Import\Product\UrlRewrite\Services\ProductUrlRewriteProcessor::removeUrlRewrite() in vendor/techdivision/import-product-url-rewrite/src/Observers/UrlRewriteUpdateObserver.php:373 Stack trace: #0 vendor/techdivision/import-product-url-rewrite/src/Observers/UrlRewriteUpdateObserver.php(148): TechDivision\Import\Product\UrlRewrite\Observers\UrlRewriteUpdateObserver->deleteUrlRewrite(Array) #1 vendor/techdivision/import-product-url-rewrite/src/Observers/UrlRewriteObserver.php(253): TechDivision\Import\Product\UrlRewrite\Observers\UrlRewriteUpdateObserver->process() #2 vendor/techdivision/import/src/Subjects/AbstractSubject.php(1005): TechDivision\Import\Product\UrlRewrite\Observers\UrlRewriteObserver->handle(Object(TechDivision\ in vendor/techdivision/import-product-url-rewrite/src/Observers/UrlRewriteUpdateObserver.php on line 373
m2if_3.8.20_error.zip

Environment

Magento CE 2.3.3
PHP 7.3
techdivision/import-cli-simple 3.8.20

Steps To Reproduce

Unzip attached files in Magento root (configurations and product data).
Run vendor/bin/import-simple import:create:ok-file && vendor/bin/import-simple import:products.
Possibly have to import a second time if the issue only happens when the product already exists in Magento.

Interrupt rendering of validation errors up from a configurable number

The validation operation has two results.

Render the result on the CLI
Render the result in a JSON file called validation.json in the working directory

When the validation found too many issues, the rendering on the console may not make sense anymore, because no one will be able to scroll through all the issues on the one hand and additionally it can take a long time to render the output to the CLI if it is huge on the other.

It would be handsome if there will be a configurable value for the number of issues that'll be rendered on the CLI, e. g. 100. Additionally, it'll make sense to allow overriding the number as a console option like --render-validation-issues=1000.

The content or the length of the JSON file should NOT be affected in any way.

Is it possible to use import without csv files?

Hi, there!
i have one question, is it possible to use m2if programmatically? without CSV filels?
as an example magmi:

$dp = Magmi_DataPumpFactory::getDataPumpInstance("productimport");
$dp->beginImportSession("default", "create", new TestLogger());

$product = array(
 "name"=>"value",
....
 );
$dp->ingest($product);
$dp->endImportSession();

Tim, i know u can help me :)

Wbr, Sergej

import category image has wrong path

import-cli-simple 3.4.1
magento 2.3.2

I try to import category images in magento path: pub/media/catalog/category
from another folder which have nested subfolders.

a small piece of my json configuration:

            {
              "id": "import_category.subject.bunch",
              "identifier": "files",
              "file-resolver": {
                "prefix": "category-import"
              },
              "filesystem-adapter" : {
                "id" : "import.adapter.filesystem.factory.league",
                "adapter" : {
                  "type" : "League\\Flysystem\\Adapter\\Local"
                }
              },
              "params" : [
                {
                  "copy-images" : true,
                  "media-directory" : "pub/media/catalog/category",
                  "images-file-directory" : "process_csv/repository/immagini"
                }
              ],
              "observers": [
                {
                  "import": [
                    "import_category.observer.composite.add_update"
                  ]
                }
              ]
            },

example of value in image_path column in CSV:
stili/classic/classic_bombay.jpg

Images and folders are successfully copied from
process_csv/repository/immagini
to
pub/media/catalog/category

but in DB every image path record start with a leading /
ex. /stili/classic/classic_bombay.jpg
so for Magento the path is absolute and not relative .

import/src/Subjects/FileUploadTrait.php

Lines 253 to 260 in f481b60

    
           // trim the leading /, if available 
        
           $trimmedFilename = ltrim($filename, '/'); 
        
           $mediaDir = ltrim($this->getMediaDir(), '/'); 
        
           $imagesFileDir = ltrim($this->getImagesFileDir(), '/'); 
        
           // prepare source/target filename 
        
           $sourceFilename = sprintf('%s/%s', $imagesFileDir, $trimmedFilename); 
        
           $targetFilename = sprintf('%s/%s', $mediaDir, $trimmedFilename);

Suggested fix:

trim all / for a path... just in case there is a trailing slash in configuration and then re-append it.

        // trim the leading /, if available
        $trimmedFilename = ltrim($filename, '/');
        $mediaDir = trim($this->getMediaDir(), '/') . '/';
        $imagesFileDir = trim($this->getImagesFileDir(), '/') . '/';

        // prepare source/target filename
        $sourceFilename = sprintf('%s%s', $imagesFileDir, $trimmedFilename);
        $targetFilename = sprintf('%s%s', $mediaDir, $trimmedFilename);

Service "import_product.action.processor.product.website.update" doesnt exist

You have requested a non-existent service "import_product.action.processor.product.website.update".

Instantiation of Listeners does not use ListenerConfiguration

Hello again!

So, I needed a listener as some kind of post-observer hook, i.e. a method that is called after the last import row has been processed by an observer. Unfortunately the M2IF doesn't provide hooks like that without using listeners.

The problem with listeners is that for instantiation from a configuration JSON, the ListenerConfiguration interface is not used.

The first part of the problem lies in the methods createEmitter() and prepareListeners() of TechDivision\Import\Events\EmitterFactory. The createEmitter() method loops over the array of listeners received from the configuration object and hands each array item to the prepareListeners() method. That one then does weird stuff (I didn't take the time to try and understand what it does, sorry). In any case, prepareListeners() expects the listeners to be given as arrays. Instead the method should simply take an instance of ListenerConfigurationInterface and use it to add a listener to the emitter.

The other part of the problem then lies in TechDivision\Import\Configuration\Jms\Configuration, where the $listeners instance variable does not have the correct @Type set in the doc comment.

I created patches for these two classes.
fix_listener_instantiation.zip

Global service ID conflicts when m2if is used inside another Symfony app

Some service IDs like configuration, loggers, connection are used in global context:
ref: https://github.com/techdivision/import/blob/15.x/symfony/Resources/config/services.xml

it would be best to have them namespaced like so:
import.configuration, import.loggers, import.connection

to not break BC, we can alias them to their original IDs.

(PS: there can be more such service IDs than mentioned above)

Configuration\Jms\Configuration\ParamsTrait ignores JSON array structure

Currently the ParamsTrait trait ignores the array structure of the params JSON node and treats the first object within it as an associative array representing the params.

Example:

{
  "params": [
    {
      "param1": "test",
      "param2": "also test"
    }
  ]
}

This results in an associative array returned by getParams() like the one below:

array (
        'param1' => 'test',
        'param2' => 'also test'
    );

Adding another param in a syntax one would expect does not result in any more params to be returned by getParams method, as internally PHPs reset() function is used which leads to ignoring all other objects in the JSON array.

Example of JSON native second param (second param gets ignored by ParamsTrait):

{
  "params" : [
    {
      "param1": "test",
      "param2": "also test"
    },
    {
      "param1": "test",
      "param2": "also test"
    }
  ]
}

I understand this design decision was made so the getParam() method can be used to retrieve actual params (like param1) by key, but from a JSON point of view the configuration syntax is misleading and not easily understood.

I would propose to change the JSON structure within params nodes to be objects so they can be used with getParams() (as associative array) and getParam() by the actual keys within the object.

Example based on the param extension above would be:

{
  "params" : {
    "param1":
    {
      "param1": "test",
      "param2": "also test"
    },
    "param2":  {
      "param1": "test",
      "param2": "also test"
    }
  }
}

This follows the native JSON structure and is easily extendable.

Category and Attribute creation on the fly

During the product import process, Magento's default importer creates categories if they don't exist, that is very handy. Have this functionality in a later release of M2IF would be cool.

The same on-the-fly creation of custom product attributes would be an extra, but also really helpful solution.

Disable the stock observer on the fly if there are no columns defined that are related to stock.

The stock quantities get overridden with 0 if there is no stock quantity column defined.

Importing files with CRLF line endings leads to unexpected results

Importing a file with DOS-style CRLF line endings leads to unexpected behaviour. For example when importing attribute sets this leads to only the first row being imported and the attribute_set_code being set to null.

Clean-up-empty-columns not work on EAV attributes not serialized in addtional_attributes

When someone tries to import option values (select or multi-select), e. g. for color, which are NOT serialized in the additional_attributes column, instead of coming in a separate column AND added them to the clean-up-empty-columns array in the configuration the import fails with the message:

Can't find select option value "" for attribute "color" in file product-import_20161021-161909_01.csv on line 2

Add support for NativeMailerHandler

When emails for log events are sent, the \Monolog\Handler\NativeMailerHandler allows a more flexible use of the sendmail command as \Monolog\Handler\SwiftMailerHandler does.

Because the NativeMailerHandler evaluates the php ini setting sendmail_path, the way emails are sent can be configured per host. This allows for example the use of mailhog in testing enviroments.

UAC

It is possible to configure \Monolog\Handler\NativeMailerHandler as mailer handler in loggers section

Slash in category name

Hi Tim,

i tried to import with this awesome framework and figured out that importing categories with a forward slash is not working.

The category processor always tries to create additional categories.

I'v written 2 patches that solve the problem. Could you please test them and integrate in your framework.

techdivision.zip

The third patch is for msi if the msi tables do not exist.

GLOB_BRACE is not available on Alpine Linux

We are running some system on the official PHP docker container which is based on Alpine linux

Running M2IF results in an error

PHP Warning:  Use of undefined constant GLOB_BRACE - assumed 'GLOB_BRACE' (this will throw an Error in a future version of PHP) in [/var/www/vendor/techdivision/import/src/Adapter/PhpFilesystemAdapter.php](https://github.com/techdivision/import/blob/16.x/src/Adapter/PhpFilesystemAdapter.php#L161) on line 161

Magento's Alure Version had similar problems and can be upgraded / patched

The code looks like this:

        $dirs = $this->glob(dirname($pattern). DIRECTORY_SEPARATOR . '*', GLOB_ONLYDIR|GLOB_NOSORT|GLOB_BRACE);

I am wondering if GLOB_BRACE is even actively used? Just removing it kind of fixed the issues for us (sideffects not yet tested).

Attribute set group sortorder is not taken into consideration

While importing the attribute set, it can be seen that after the import the attribute group sort order is not taken into consideration, kindly investigate

Sample data can be used

Magento Version: 2.4.1 / 2.4
name : techdivision/import-cli-simple
descrip. : CLI providing single-threaded Magento 2 importing functionality based on Pacemaker
keywords :
versions : * 3.8.33
type : library
license : Open Software License 3.0 (OSL-3.0) (OSI approved) https://spdx.org/licenses/OSL-3.0.html#licenseText
names : techdivision/import-cli-simple

Category Import Rewrite Error

Preconditions (*)

Magento 2.2
techdivision/import: 6.0.1

Steps to reproduce (*)

Run an import-simple import:categories add-update comand with the fowlling file

store_view_code,attribute_set_code,path,name,is_active,is_anchor,include_in_menu,display_mode,url_key,description,image_path,meta_title,meta_keywords,meta_description,landing_page,position,path_in_store,custom_design,custom_design_from,custom_design_to,page_layout,custom_layout_update,available_sort_by,default_sort_by,custom_use_parent_settings,custom_apply_to_products,filter_price_range,created_at,updated_at,additional_attributes
,Default,Default Category/Innenbeleuchtung Klein,Innenbeleuchtung Klein,1,1,1,,,,,Innenbeleuchtung,,,,1,,,,,,,,,,,,,,
de,Default,Default Category/Innenbeleuchtung Klein,Innenbeleuchtung Klein,1,1,1,,,,,Innenbeleuchtung,,,,,,,,,,,,,,,,,,
es,Default,Default Category/Innenbeleuchtung Klein,Innenbeleuchtun Klein,1,1,1,,,,,Innenbeleuchtung,,,,,,,,,,,,,,,,,,

Run an import-simple import:categories add-update comand with the fowlling file

store_view_code,attribute_set_code,path,name,is_active,is_anchor,include_in_menu,display_mode,url_key,description,image_path,meta_title,meta_keywords,meta_description,landing_page,position,path_in_store,custom_design,custom_design_from,custom_design_to,page_layout,custom_layout_update,available_sort_by,default_sort_by,custom_use_parent_settings,custom_apply_to_products,filter_price_range,created_at,updated_at,additional_attributes
,Default,Default Category/Innenbeleuchtung klein,Innenbeleuchtung klein,1,1,1,,,,,Innenbeleuchtung,,,,1,,,,,,,,,,,,,,
de,Default,Default Category/Innenbeleuchtung klein,Innenbeleuchtung klein,1,1,1,,,,,Innenbeleuchtung,,,,,,,,,,,,,,,,,,
es,Default,Default Category/Innenbeleuchtung klein,Innenbeleuchtun klein,1,1,1,,,,,Innenbeleuchtung,,,,,,,,,,,,,,,,,,

Expected result (*)

Renaming from Innenbeleuchtung Klein to Innenbeleuchtung klein should be work fine.

Actual result (*)

Import break with the following error

PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'innenbeleuchtung-klein.html-2' for key 'URL_REWRITE_REQUEST_PATH_STORE_ID' in /project/src/vendor/techdivision/import/src/Actions/Processors/AbstractBaseProcessor.php:146
Stack trace:
#0 /project/src/vendor/techdivision/import/src/Actions/Processors/AbstractBaseProcessor.php(146): PDOStatement->execute(Array)
#1 /project/src/vendor/techdivision/import/src/Actions/Processors/UrlRewriteCreateProcessor.php(58): TechDivision\Import\Actions\Processors\AbstractBaseProcessor->execute(Array, NULL)
#2 /project/src/vendor/techdivision/import/src/Actions/UrlRewriteAction.php(65): TechDivision\Import\Actions\Processors\UrlRewriteCreateProcessor->execute(Array, NULL)
#3 /project/src/vendor/techdivision/import/src/Actions/UrlRewriteAction.php(52): TechDivision\Import\Actions\UrlRewriteAction->create(Array)
#4 /project/src/vendor/techdivision/import-category/src/Services/CategoryBunchProcessor.php(1062): TechDivision\Import\Actions\UrlRewriteAction->persist(Array, NULL)
#5 /project/src/vendor/techdivision/import-category/src/Observers/UrlRewriteObserver.php(223): TechDivision\Import\Category\Services\CategoryBunchProcessor->persistUrlRewrite(Array)
#6 /project/src/vendor/techdivision/import-category/src/Observers/UrlRewriteObserver.php(105): TechDivision\Import\Category\Observers\UrlRewriteObserver->persistUrlRewrite(Array)
#7 /project/src/vendor/techdivision/import-category/src/Observers/UrlRewriteUpdateObserver.php(89): TechDivision\Import\Category\Observers\UrlRewriteObserver->process()
#8 /project/src/vendor/techdivision/import-category/src/Observers/AbstractCategoryImportObserver.php(54): TechDivision\Import\Category\Observers\UrlRewriteUpdateObserver->process()
#9 /project/src/vendor/techdivision/import/src/Subjects/AbstractSubject.php(823): TechDivision\Import\Category\Observers\AbstractCategoryImportObserver->handle(Object(TechDivision\Import\Category\Ee\Subjects\EeBunchSubject))
#10 [internal function]: TechDivision\Import\Subjects\AbstractSubject->importRow(Array)
#11 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(149): call_user_func(Array, Array)
#12 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(135): TechDivision\Import\Adapter\Goodby\Interpreter->delegate(Array, Array)
#13 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(74): TechDivision\Import\Adapter\Goodby\Interpreter->notify(Array)
#14 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Lexer.php(120): TechDivision\Import\Adapter\Goodby\Interpreter->interpret(Array)
#15 /project/src/vendor/techdivision/import/src/Adapter/CsvImportAdapter.php(142): TechDivision\Import\Adapter\Goodby\Lexer->parse('/Volumes/Worksp...', Object(TechDivision\Import\Adapter\Goodby\Interpreter))
#16 /project/src/vendor/techdivision/import/src/Subjects/AbstractSubject.php(736): TechDivision\Import\Adapter\CsvImportAdapter->import(Array, '/Volumes/Worksp...')
#17 /project/src/vendor/techdivision/import/src/Plugins/SubjectExecutor.php(107): TechDivision\Import\Subjects\AbstractSubject->import('c000aa1a-9c61-4...', '/Volumes/Worksp...')
#18 /project/src/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(186): TechDivision\Import\Plugins\SubjectExecutor->execute(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject), Array, 'c000aa1a-9c61-4...', '/Volumes/Worksp...')
#19 /project/src/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(121): TechDivision\Import\Plugins\SubjectPlugin->processSubject(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject))
#20 /project/src/vendor/techdivision/import-app-simple/src/Simple.php(622): TechDivision\Import\Plugins\SubjectPlugin->process()
#21 /project/src/vendor/techdivision/import-cli-simple/src/Command/AbstractImportCommand.php(105): TechDivision\Import\App\Simple->process()
#22 /project/src/vendor/symfony/console/Command/Command.php(245): TechDivision\Import\Cli\Command\AbstractImportCommand->execute(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /project/src/vendor/symfony/console/Application.php(835): Symfony\Component\Console\Command\Command->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /project/src/vendor/symfony/console/Application.php(185): Symfony\Component\Console\Application->doRunCommand(Object(TechDivision\Import\Cli\Command\ImportCategoriesCommand), Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /project/src/vendor/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /project/src/vendor/techdivision/import-cli-simple/bootstrap.php(39): Symfony\Component\Console\Application->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /project/src/vendor/techdivision/import-cli-simple/bin/import-simple.php(54): require('/Volumes/Worksp...')
#28 /project/src/vendor/techdivision/import-cli-simple/bin/import-simple(2): require('/Volumes/Worksp...')
#29 {main}

Next PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'innenbeleuchtung-klein.html-2' for key 'URL_REWRITE_REQUEST_PATH_STORE_ID' when executing SQL "INSERT
               INTO url_rewrite
                    (entity_type,
                     entity_id,
                     request_path,
                     target_path,
                     redirect_type,
                     store_id,
                     description,
                     is_autogenerated,
                     metadata)
             VALUES (category,
                     703,
                     innenbeleuchtung-klein.html,
                     catalog/category/view/id/703,
                     0,
                     2,
                     ,
                     1,
                     )" in /project/src/vendor/techdivision/import/src/Actions/Processors/AbstractBaseProcessor.php:160
Stack trace:
#0 /project/src/vendor/techdivision/import/src/Actions/Processors/UrlRewriteCreateProcessor.php(58): TechDivision\Import\Actions\Processors\AbstractBaseProcessor->execute(Array, NULL)
#1 /project/src/vendor/techdivision/import/src/Actions/UrlRewriteAction.php(65): TechDivision\Import\Actions\Processors\UrlRewriteCreateProcessor->execute(Array, NULL)
#2 /project/src/vendor/techdivision/import/src/Actions/UrlRewriteAction.php(52): TechDivision\Import\Actions\UrlRewriteAction->create(Array)
#3 /project/src/vendor/techdivision/import-category/src/Services/CategoryBunchProcessor.php(1062): TechDivision\Import\Actions\UrlRewriteAction->persist(Array, NULL)
#4 /project/src/vendor/techdivision/import-category/src/Observers/UrlRewriteObserver.php(223): TechDivision\Import\Category\Services\CategoryBunchProcessor->persistUrlRewrite(Array)
#5 /project/src/vendor/techdivision/import-category/src/Observers/UrlRewriteObserver.php(105): TechDivision\Import\Category\Observers\UrlRewriteObserver->persistUrlRewrite(Array)
#6 /project/src/vendor/techdivision/import-category/src/Observers/UrlRewriteUpdateObserver.php(89): TechDivision\Import\Category\Observers\UrlRewriteObserver->process()
#7 /project/src/vendor/techdivision/import-category/src/Observers/AbstractCategoryImportObserver.php(54): TechDivision\Import\Category\Observers\UrlRewriteUpdateObserver->process()
#8 /project/src/vendor/techdivision/import/src/Subjects/AbstractSubject.php(823): TechDivision\Import\Category\Observers\AbstractCategoryImportObserver->handle(Object(TechDivision\Import\Category\Ee\Subjects\EeBunchSubject))
#9 [internal function]: TechDivision\Import\Subjects\AbstractSubject->importRow(Array)
#10 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(149): call_user_func(Array, Array)
#11 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(135): TechDivision\Import\Adapter\Goodby\Interpreter->delegate(Array, Array)
#12 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Interpreter.php(74): TechDivision\Import\Adapter\Goodby\Interpreter->notify(Array)
#13 /project/src/vendor/techdivision/import/src/Adapter/Goodby/Lexer.php(120): TechDivision\Import\Adapter\Goodby\Interpreter->interpret(Array)
#14 /project/src/vendor/techdivision/import/src/Adapter/CsvImportAdapter.php(142): TechDivision\Import\Adapter\Goodby\Lexer->parse('/Volumes/Worksp...', Object(TechDivision\Import\Adapter\Goodby\Interpreter))
#15 /project/src/vendor/techdivision/import/src/Subjects/AbstractSubject.php(736): TechDivision\Import\Adapter\CsvImportAdapter->import(Array, '/Volumes/Worksp...')
#16 /project/src/vendor/techdivision/import/src/Plugins/SubjectExecutor.php(107): TechDivision\Import\Subjects\AbstractSubject->import('c000aa1a-9c61-4...', '/Volumes/Worksp...')
#17 /project/src/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(186): TechDivision\Import\Plugins\SubjectExecutor->execute(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject), Array, 'c000aa1a-9c61-4...', '/Volumes/Worksp...')
#18 /project/src/vendor/techdivision/import/src/Plugins/SubjectPlugin.php(121): TechDivision\Import\Plugins\SubjectPlugin->processSubject(Object(TechDivision\Import\Configuration\Jms\Configuration\Subject))
#19 /project/src/vendor/techdivision/import-app-simple/src/Simple.php(622): TechDivision\Import\Plugins\SubjectPlugin->process()
#20 /project/src/vendor/techdivision/import-cli-simple/src/Command/AbstractImportCommand.php(105): TechDivision\Import\App\Simple->process()
#21 /project/src/vendor/symfony/console/Command/Command.php(245): TechDivision\Import\Cli\Command\AbstractImportCommand->execute(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#22 /project/src/vendor/symfony/console/Application.php(835): Symfony\Component\Console\Command\Command->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /project/src/vendor/symfony/console/Application.php(185): Symfony\Component\Console\Application->doRunCommand(Object(TechDivision\Import\Cli\Command\ImportCategoriesCommand), Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /project/src/vendor/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /project/src/vendor/techdivision/import-cli-simple/bootstrap.php(39): Symfony\Component\Console\Application->run(Object(TechDivision\Import\Cli\Console\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /project/src/vendor/techdivision/import-cli-simple/bin/import-simple.php(54): require('/Volumes/Worksp...')
#27 /project/src/vendor/techdivision/import-cli-simple/bin/import-simple(2): require('/Volumes/Worksp...')
#28 {main}

Empty .OK file results in error message

In the latest 3.8 version, it is no longer possible to use an empty .OK file with the same filename as the single import file. Running the import results in

 Can't read the content of file "/Users/wagnert/Workspace_Idea/techdivision/import-cli-simple/instances/ee/2.3.4/var/importexport/product-import_20161021-161909_01.ok"

Additional Attributes Column

Hi If the additional attributes column has html with inline css in it it has a tendancy to fail due to the ="" it explodes here and fails, but even with the \ escape before the = it doesnt actually do anything different.

Any solution ?

Also Found invalid entity type code "catalog_product" in file media.csv when trying to use a standard magento 2 csv base_image, small_image and thumbnail therefore never actually import.

"Integrity constraint violation: 1048 Column 'entity_id' cannot be null" while importing products

I get the following error messages while importing simple products:

PHP Warning:  implode(): Invalid arguments passed in phar:///var/www/magento2/bin/import-cli-simple.phar/vendor/techdivision/import-category/src/Listeners/SortCategoryListener.php on line 159                                                                                                                                   
PHP Warning:  sizeof(): Parameter must be an array or an object that implements Countable in phar:///var/www/magento2/bin/import-cli-simple.phar/vendor/techdivision/import-category/src/Listeners/SortCategoryListener.php on line 248
[...]
PHP Warning:  sizeof(): Parameter must be an array or an object that implements Countable in phar:///var/www/magento2/bin/import-cli-simple.phar/vendor/techdivis
ion/import-category/src/Listeners/SortCategoryListener.php on line 267
 SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'entity_id' cannot be null when executing SQL "INSERT                                              
               INTO catalog_category_entity_int                                                                   
                    (entity_id,                                                                                                                                  
                     attribute_id,                                                                                
                     store_id,                                                                                                                                   
                     value)                                                                                       
             VALUES (,                                                                                                                                           
                     46,                                                                                          
                     0,                                                                                                                                          
                     1)" in file category-create_20200828-110655_01.csv on line 2

Any hints on how to debug this one? I'm a bit lost here, since the categories where also imported correctly, but I don't know why the entity_id is missing in the query.

A lot of repeated sql select queries are fired

probably in these steps:

general/general/global-data > general/catalog_product/collect-data

SHOW GLOBAL VARIABLES LIKE 'innodb%';
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (3);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (3);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (4);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (3);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (4);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (5);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (3);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (4);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (6);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (3);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (4);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (7);
SELECT t1.* FROM catalog_category_entity AS t0 INNER JOIN catalog_category_entity_varchar AS t1 ON t1.row_id = t0.row_id INNER JOIN eav_attribute AS t2 ON t2.entity_type_id = 3 AND t2.attribute_code = 'name' AND t1.attribute_id = t2.attribute_id AND t1.store_id = 0 AND t0.entity_id IN (2);

and for a 2-3 rows simple product import, there are easily 3000+ sqls queries making everything slower than it should have been.

PS: using redis as a cache doesnt seem to help either.

My guess is: it could be N+1 problem compounded. For each new category it sends query for older category.

For context:
Magento = 2.3.3 EE
entityTypeCode = catalog_product
shortcut = add-update

Attribute visual swatch unable to render properly

Following the guidelines from here:

https://github.com/techdivision/import-sample-data/blob/4.x/generic/data/attributes/add-update-examples/attribute-import_20190103-180312_01.csv

There's an issue with rendering swatches, in the sense that if in the frontend_input column the value is "select" and the attribute_option_swatch is properly populated along with additional_data as a swatch type 2 ( which is visual ), the attribute is still being rendered in the frontend as a dropdown instead of a visual swatch. This is happening because the value of the frontend_input should be allowed to be swatch_visual instead of dropdown.

Properly support slashes in Category names

See #194 #198

We have slashes in category names (something like Valves 1/2")

Describe the solution you'd like

In https://github.com/techdivision/import/blame/master/src/Assembler/CategoryAssembler.php#L152 we should use a CSV implode instead of a simple implode.

Describe alternatives you've considered

Removing slashes? but that is not an option :)

Additional context

If we have such a category tree

This would lead twice to the key Foo/Bar, for ID = 2 as well as ID = 3, so the array indexes collide and the ID = 2 will be overwritten.

Mandatory ok file on import:products:price

Import for import:products:price always throws "Stop processing, because can't find the mandatory OK file to process at least one of 1 files" message even on import:create:ok-file witch produces "product-import-price_-.ok" file.

Refactor AttributeObserverTrait

The trait has to be refactored and the functionality to initialize the cleanUpEmptyColumnKeys array has to be extracted from the clearRow() method to the implementing observer's factory method. This results in the fact, that each observer that handles attributes MUST implement the ObserverFactoryInterface.

Extract functionality to clean-up empty columns from AbstractOberserver::mergeEntity() method

Actual Situation
Up from version 16.1.0, the method AbstractOberserver::mergeEntity() contains a check if the invoking subject implements the CleanUpColumnsSubjectInterface to clean-up columns that has been configured in the clean-up-empty-columns array in the configuration, as described in the documentation.

Additionally, the actual implementation assumes that the column equals the member names

// query whether or not the subject has columns that has to be cleaned-up
if (($subject = $this->getSubject()) instanceof CleanUpColumnsSubjectInterface) {
    // load the columns that has to be cleaned-up
    $cleanUpColumns =  $subject->getCleanUpColumns();
    // load the column/member names from the attributes
    $columnNames = array_keys($attr);

    // iterate over the column names
    foreach ($columnNames as $columnName) {
        // we do NOT clean-up members that HAS a value or ARE in
        // the array with column names that has to be cleaned-up
        if ($this->hasValue($columnName) || in_array($columnName, $cleanUpColumns)) {
            continue;
        }
        // unset the column, because it has to be cleaned-up
        unset($attr[$columnName]);
    }
}

This should also be refactored by add the possibility to map the column to the attribute names.

Expected Situation
The functionality to clean-up the empty columns should be extracted to a utility class, injected, and integrated like the functionality for the state detection. This refactoring should result in something like

protected function mergeEntity(array $entity, array $attr, $changeSetName = null)
{
    return array_merge(
        $entity, 
        $attr, 
        array(
            EntityStatus::MEMBER_NAME => $this->detectState(
                $entity, 
                $this->cleanUpEntity($attr), 
                $changeSetName
         )
    )
);

Circular reference detected for service "import_app_simple.simple"

** Exception that I got **
[Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException] Circular reference detected for service "import_app_simple.simple", path: "import_app_simple.simple -> import.modules -> import.module.plugin".

How to reproduce the issue:
After generating the ok file, was trying to import the stock information.

./vendor/bin/import-simple import:products:inventory:msi --magento-edition CE --magento-version 2.3.3 --db-pdo-dsn mysql:host=172.17.0.1\;port=3306\;dbname=biv_m2 --db-username root --db-password xxxxx --source-dir /xx/xxxx

** Dirty fix solution:**

Comment <argument type="service" id="import_app_simple.simple"/> in the file /vendor/techdivision/import/symfony/Resources/config/services.xml
Remove all the application related code in the TechDivision\Import\Modules\PluginModule class

I did not use Symfony before, not really sure what the error is related to. Looks like it is a looping injecting issue. eg. parent -> child -> child -> parent

	// trim the leading /, if available
	$trimmedFilename = ltrim($filename, '/');
	$mediaDir = ltrim($this->getMediaDir(), '/');
	$imagesFileDir = ltrim($this->getImagesFileDir(), '/');

	// prepare source/target filename
	$sourceFilename = sprintf('%s/%s', $imagesFileDir, $trimmedFilename);
	$targetFilename = sprintf('%s/%s', $mediaDir, $trimmedFilename);

techdivision / import Goto Github PK

import's Introduction

Pacemaker - Import Framework

import's People

Contributors

Stargazers

Watchers

Forkers

import's Issues

Preconditions (*)

Steps to reproduce (*)

Expected result (*)

Actual result (*)

Preconditions (*)

Steps to reproduce (*)

Expected result (*)

Actual result (*)

Recommend Projects

Recommend Topics

Recommend Org