magento / architecture Goto Github PK

Prototype Checkout service.

To do:
[ ] Sync with mainline
[ ] Merge with @antonkril 's lightweight framework

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• Vote on version specific rules
• Research on count in loop
• Need feedback on setTemplate
• Review and comment on commits verification tool

Recording: no recording due to tech difficulties

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• #85 - looks good, give us a couple days to check with other requested reviewers
• #91 - in general, looks good. Needs some clarification of details and to be checked with couple more requested reviewers
• #92 (see #90 for original discussion) - looks good. Let's move on and validate a real solution (with specific rules) with internal and community developers before merging. Prepare developers to such change

🎥 Recording

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• note one...

Recording: TBD

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• Upgrading jQuery to 3.3.1 magento/magento2#15531
  • @buskamuza, @DrewML : move to 2.4
• Upgrade to Promise-based requirejs magento/magento2#19517
  • need updates on debugging and performance improvements from @kirmorozov
• Allow class property for referenceBlock node of layout configuration magento/magento2#18933
  • To prepare a proposal with pros and cons
    • Cons: conflicts between extensions if classes don't follow same contract
• #69
• @kirmorozov: PHP 7.4 preload for Magento 2.4

Recording: TBD

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @paliarush : CI for DevBox. Most likely will work on Travis CI, so it's public
  • The direction for DevBox should be adjusted based on the backlog priority: ability to deploy Magento instances for Admin, Storefront, WebAPI, Cron, etc. (instead of a separate service like Checkout)
• @paliarush : devs are ok with DevBox if it takes < 4GB
• @joni-jones :
  • investigated which service makes sense to separate first from technical perspective
  • will work on extensions impact
• @melnikovi : proposal for exceptions handling
• @shiftedreality : Magento CLI Management
  • @kandy : add ability to run commands in multiple contexts
  • @antonkril : add grouping of contexts
  • @antonkril : add support for IP and login/pw
  • @paliarush : make sure the key is hidden from CLI output
  • @kandy : why not move responsibility of forwarding CLI to correct service to API gateway? @antonkril : API gateway is publicly accessible, this tool should be in the private network
• @rgoncharuk : will present project structure and processes in a week

Repository: https://github.com/magento/magento2-phpstorm-plugin

• process existing PRs
• publish updated package to JetBrains repo
• document process of publication
• update documentation to reflect correct licensing

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• Remote Code Invoker Design Doc, #37 (@AlexMaxHorkun )
  • @buskamuza : we need some framework/tool for handling services routing, etc. But not necessarily to invoke PHP methods remotely
  • @melnikovi : such logic should be responsibility of Service Mesh. @antonkril : we will use Service Mesh in the Services implementation
  • @antonkril : split the document to
    • 1. service invoker
    • 2. services communication
    • approve the document, move to Services Isolation folder
• Frontend Technical Vision (20min, @jedmao)
  • move to public
  • @kandy : add performance testing section
• Standard Promises, #42 (15min, @AlexMaxHorkun )
  • Add more use cases for sync support (discuss with @melnikovi )
  • Add reasoning to why wait() doesn't return the result

🎥 Recording

PR #52

Meeting Notes

• @akaplya concern about the durability of the message queue in case of using it as transport for full cache invalidation purpose.
• @melnikovi suggest using Amazon SQS as a message queue
• @kandy proposed don't setup reverse proxy in front of service in private network until such necessity appears. Also, proposed to make service application as fast as possible at first and only then use reverse proxy to decrease the load on service.
• @MaxLesechko propose to use application data cache versioning instead of full cache cleaning
• @viktym proposed to postpone using a reverse proxy in service-to-service communication on the first iteration, application data cache versioning should be described in a separate document, investigate the possibility and pitfalls of using a message queue for managing cache invalidation process

As per this existing modules will be split into multiple packages.

This has a follow-on impact on existing extensions (ie nearly 100% of extension will require not only a new release but also a separately maintained branch). One idea to minimise the impact would be to keep the package name for CatalogAPI for existing extensions that only use the existing service contracts (ie current best practice extension get less impacted)

As an example an extension today has the following composer.json

    "require": {
        "magento/module-catalog": "^103.0.0",

and only uses the API interfaces. After splitting the modules ideally this should only be

    "require": {
        "magento/module-catalog": "^103.0.0 | ^104.0.0",

This would allow the extension to be useful across both versions instead of requiring two separate versions to support both branches:

    "require": {
        "magento/module-catalog": "^103.0.0",

and

    "require": {
        "magento/module-catalog-api": "^100.0.0",

Further to allow transitioning into the end state of service isolation it would be great if this could be phased in. Once we know the split and what package names the new modules will have we can add them to the existing Catalog module.

So the existing package magento/module-catalog adds the following extra packages:

    "provide": {
        "magento/module-catalog-implementation": "self.version",
        "magento/module-catalog-admin": "self.version",
        "magento/module-catalog-storefront": "self.version",
    }

while the code is not yet moved out. That way it will be possible to release an extension that prepares for the split and is able to work across major release branches.

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• magento/magento-coding-standard#45 - ✅ agreed with the solution, any name is fine
• Add PHPStan - ✅ proceed
• magento/magento-coding-standard#57 (Lars) - run the test on Magento code base and see what are the results

🎥 Recording

The following documents should be updated to reflect that changing argument type for ActionGroup is a breaking change:

• https://github.com/magento/architecture/blob/master/design-documents/testing/functional/versioning-and-backward-compatibility-policy.md

Action Group entity:

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• magento/magento2#19517 Promise-based implementation of requirejs
  • @DrewML: need more input from @kirmorozov with proof that the change is necessary
• Sniffs Consolidation
• URL Rewrites #61
  • @akaplya: both delayed URL rewrites generation and skipping approaches should exist in the system. Skipping approach should be used as a fallback when the generation is not done yet.
    • Collisions handling: 1) fallback to the skipped generation approach; 2) provide UI for the customer to fix the collision.
  • Action items: 1) address review feedback for the implementation (performance and load testing); 2) create a story to implement background URL rewrites generation (including performance requirements/tests).
• magento/devdocs#3316 - the changes can be merged. Docs for older Magento versions to be reverted and processed separately.
• Unify https://devdocs.magento.com/guides/v2.3/coding-standards/technical-guidelines.html (not version it)
  • @lenaorobei: current approach is better
  • @paliarush , @buskamuza : use unified approach
  • Collect more feedback. No reason to change anything unless confirmed by more feedback
• Move Tech Guidelines to arch repo - not necessary

🎥 Recording

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @melnikovi Checkout prototype: synced with mainline, to do: merge with @antonkril 's work on the framework
• @valdislav continue workin on DevBox
• @buskamuza to create Wiki page with info for the project
• @joni-jones Authentication & authorization - #48
• @viktym Caching layer for Magento services
  • Adding reverse proxy between services
  • Cache cleaning
    • @akaplya replace cache cleaning with versioning
  • Take into consideration content staging
• @buskamuza discussed with @ravmenon impact on Marketplace (extensions)
  • Framework changes will impact structure of the extensions (see Magento Service Isolation - Split Modules
  • API changes

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @rgoncharuk :
  • building dependencies graph after breaking down modules
  • proposal: have a desired state for dependencies between components, in addition to desired dependencies on module level
  • we have to use magento GitHub organization for the project, can't use a separate org as it's inconvenient for managing permissions
  • issues with DevBox? Should we try use Magento Cloud? We may have resources issues for many services.
    • We don't have am issue now, with one-two services. We can combine resources (e.g., have single container for MySQL with multiple DBs to reduce resources for containers) in the future if necessary
    • @paliarush : Cloud is not an option for community devs
    • devs usually should not need many services in dev environment

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• magento/devdocs#3316 - discussed, looks ok. There are couple notes

🎥 Recording

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @melnikovi : continue syncing the prototype with the mainline. Continue working on it, fixing issues in the scenarios
• @paliarush : DevBox (Minikube-based)
  • Redis cache
  • xDebug based on Cookies
    • Kubernetes has two containers (w/ and w/o xDebug)
  • Monolith app for now. @valdislav is working on the checkout service
  • @fahdsiddiqui : look at Jenkins X
  • Helm
• Oleg Posyniak: Magento Tool
  • To be integrated in the DevBox
  • Future backlog (see the link for full backlog): security, context refresh, etc.
• @melnikovi : Services decomposition guidelines

Add common areas/topics to be covered by a Technical Vision to the definition (https://devdocs.magento.com/guides/v2.3/coding-standards/technical-vision/index.html).

Based on magento/devdocs#3639 (comment)

Current approach to tech stack upgrade is to review everything in scope of a release, as a specific task. We should move towards "notified" strategy, where we upgrade a specific library or software as it gets released.

Describe the process and tools that can be used or implemented for upgrading tech stack.
Cover:

• tools that can be used to find latest stable versions of used infrastructure and libraries
• tool that can be used to track latest versions of used infrastructure and libraries
  • ideally, know dates of a future release to be prepared for upgrade
• criteria by which an upgrade is included in minor or patch version of Magento
• support policy rules (how many versions of a software we should support)
  Take into account that tech stack upgrades need coordination with other organizations (e.g., MFTF team, DevOps, Cloud, community).

Resources:

• currently supported tech stack
• https://wiki.corp.magento.com/display/MAGE2/Upgrade+of+3rd-party+Dependencies+for+Magento+2.3+Release (internal)

The following documents should be updated to reflect that adding (throwing a new) exception is a breaking change:

• https://devdocs.magento.com/guides/v2.3/extension-dev-guide/versioning/dependencies.html
• https://devdocs.magento.com/guides/v2.3/extension-dev-guide/versioning/codebase-changes.html
• https://devdocs.magento.com/guides/v2.3/contributor-guide/backward-compatible-development/index.html
  • this one covers modification of an exception type, but it's not a precise explanation. This should be improved to reflect a case when there was no exception and then it appears

DevBox with Minikube.

AC:

• Target deployment: Kubernetes for Services Isolation project
• As close to production environment as possible
• Open Source
• Clear instructions (docs) on the usage

Create high level design for Adobe Target Recommendations integration.

• Igor
  • Exception handling (return PHP exceptions or error codes; how to handle network communication exception)
• Eugene
  • Authentication and authorization prototype
• Viktor
  • Need to re review proposal about caching
  • Research on improving application caching
• Ravi
  • Need to figure out how extensions will be distributed on marketplace
• Anton Kaplya
  • Prototype/research on decoupling communication between services; moving knowledge on how services communicate to configuration; service communication will happen through orchestration system.

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @paliarush : DevBox
  • Trying to run on Travis CI to run tests and simplify contribution process. The DevBox works fine on OS X, but doesn't run on Ubuntu which is an option on Travis CI

Provide additional information to be updated in https://docs.magento.com/m2/ce/user_guide/stores/cookie-reference.html :

• modules where the cookie is used/set
• part of Magento configuration (Admin or other) which impacts usage of the cookie. For example, disabling a feature leads to the cookie to be not used
• cookie lifetime

In response to the new FE Tech Vision and the conversations that followed, developers clearly need an easy way to hit the ground running on a new FE project. CRA is a great start, but it's also very simple and unopinionated (as it should be). To take it a step further, developers need a scaffolding tool for new FE projects. Enter Yeoman.

Yeoman is the web's scaffolding tool for modern web apps.

For this task, build a Yeoman generator that specifically implements the paradigms in the FE Tech Vision. This generator shall extend the CRA with the following features:

• State management via React Redux.
• I18n via react-intl.
• React Testing Library for good testing practices.
• Ability to easily create new API clients, actions and reducers.
• TypeScript source and models with clearly defined types throughout the project.

To be clear about what this is, it's more than just a boilerplate. A boilerplate would be a good starting point for a project, but then it stops there. Yeoman generators, on the other hand, build the initial project, but also give you a mechanism to scaffold new files as the project grows.

The result of this ticket will be money saved/earned and a massive amount of technical debt prevented during the initial phases of a project.

Meeting Notes

• @melnikovi
  • improving performance of prepare for cart logic
  • added breaking changes section to the Checkout prototype
  • to make the prototype repo public
• @joni-jones
  • Authentication & authorization prototype to answer questions in the proposal
• @paliarush
  • DevBox prototype - https://github.com/magento/magento2-kubernetes-devbox/tree/prototype
    • Can be installed w/ or w/o Checkout service deployed separately
    • Will need to investigate how to run tests in CI (need starting VM)
    • Not compatible with Win yet
• @AlexMaxHorkun
  • inter-service-communication doc
    • @akaplya : better to use gRPC ❗️ blocker
    • @kandy : we should use GraphQL
      • it is better for extensibility
    • @AlexMaxHorkun : the proposed invoker is written so that it's possible to switch
• #52 - there are still some open questions

From @larsroettig on February 7, 2019 15:21

We should add as grumphp for checking the quality of commits.

Sample Config:

parameters:
    bin_dir: "./vendor/bin"
    git_dir: "."
    hooks_dir: ~
    hooks_preset: local
    stop_on_failure: false
    ignore_unstaged_changes: false
    ascii:
        succeeded: ~
        failed: ~
    tasks:
        phpversion:
            project: '7.1'
        xmllint:
            ignore_patterns: ["src/dev"]
        yamllint:
            ignore_patterns: ["src/dev"]
        securitychecker:
            lockfile: ./src/composer.lock
            format: ~
            end_point: ~
            timeout: ~
            run_always: false
        git_blacklist:
            keywords:
            - "die("
            - "var_dump("
            - "print_r("
            - "var_export("
            - "exit;"
            whitelist_patterns:
            - /^src\/app\/(.*)/
            triggered_by: ['php']
        phpmnd:
            directory: ./src/app/code
            whitelist_patterns: []
            exclude: []
            exclude_name: []
            exclude_path: []
            extensions: []
            hint: false
            ignore_numbers: []
            ignore_strings: []
            strings: false
            triggered_by: ['php']
        phpcs:
            standard: "./src/dev/tests/static/framework/Magento/ruleset.xml"
            severity: ~
            error_severity: ~
            warning_severity: 6
            tab_width: ~
            report: summary
            report_width: ~
            whitelist_patterns:
            - /^src\/app\/code\/(.*)/
            encoding: ~
            ignore_patterns: []
            sniffs: []
            triggered_by: [php]
        phpunit:
            config_file: ./dev/tests/phpunit-no-coverage.xml
            testsuite: ~
            group: []
            always_execute: false

We using this tool for more than one year in our projects. We getting better commits and lower rejection rate of Pull Requests.

Info:
https://www.integer-net.com/magento-2-automatic-code-quality-check-with-grumphp/

To discuss:

• what should checked before commit
• if can use this tool for only this repo

Copied from original issue: magento/magento-coding-standard#35

Magento 2 supports database storage mode, where the master storage location of media files is the database, and each frontend node stores a local "cached" copy of images on disk to use/serve as and when required.

The current solution is HORRENDOUS.

At the moment, every piece of code that needs to access a media file, must be MediaStorage aware, must check if database storage mode is enable, and must copy the file from the database to the local storage if it doesn't exist and then do what it needs to do with the file.

A quick code inspection through Magento shows there are MANY MANY places where content in media storage is accessed either directly, or through Magento's framework filesystem routines, neither of which are database storage aware, without the necessary database checks and storage of local copies.

The whole media storage subsystem should to be extracted to a single subsystem. Every module wishing to write/read/query an item from media storage does it through a single set of Framework functions.

The media storage subsystem should be easily pluggin-able so that additional storage methods can be introduced. This would allow for combinations such as

• Filesystem only (as per default Magento)
• Database backend, copied to filesystem to serve (as per default Magento)
• Amazon S3 storage with media url's set direct to Amazon storage frontend
• Google Cloud Storage with media url's set direct to Google storage frontend
• Other plugin modules people may wish to contribute

Note, the Amazon and Google options are hugely powerful. They allow you to serve the stored content directly from their servers through their own https servers, taking the load off your own machines. This can be easily set in Magento with the media url settings. There would be NO need for ANY local media storage whatsoever which gives a big thumbs up for clustered environments.

I realise there are addon modules available for Amazon and Google storage, but they are hacks into the current system and require a MASSIVE amount of fudging (because the code to access the media storage location is distributed far and wide in M2)

Something to think about as at the moment, both 2.2 and 2.3 are riddled with bugs and nasties related to the database media storage functionality that only rear their head when operating in a clustered setup. (Thats the whole purpose of the DB media storage option IMO)

I'm working my way through the bugs in the current system, but the more I delve into the code, the more I realise, hang on, none of this should be visible to the end user modules.....Where the media is ultimately stored is none of my concern!..... I shouldnt have to be writing code to cache files from the database to local disk, just to check the size of the favicon file, for example.

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

See below.

🎥 Recording

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• Service contracts technical guidelines (@paliarush )
  • Alex to process feedback
• Updates to service isolation proposal (@melnikovi )
• Checkout service - early draft (@melnikovi )
  • Igor to look at new Inventory API
• Workarounds for blockers described in MDE Project Structure
  • Move on with the current workaround

🎥 Recording

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• note one...

Recording: no recording due to tech difficulties

Develop internal guide on how to deal with two inventory versions.
How to fix issues.
How to build customizations.
How to resolve dependencies.

Concerning the Front-End Technical Vision Document, @DrewML requested I do some research behind react-testing-library as an alternative suggestion to enzyme.

@jimbo also requested I take a look at React Test Renderer.

Review magento/devdocs#3524

It's not clear from the readme which of these is the appropriate choice when creating a new PR to this repo.

Investigate a possibility to integrate Depenabot with Magento infrastructure.

• Check PR creation to update composer dependencies
• POC Dependabot notifications into the slack channel

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @antonkril : reiterate that main goal of the current project is evolvability of separate components (isolation, replaceability)
• @shiftedreality : implemented a Magento module for Magento CLI Tool to authenticate it. Will be published and shared later

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

See #126

SVC (Semantic Versioning Checker) is a tool that validates presence of breaking changes. Used for PRs as team merge code to mainline.

AC:

• move SVC to a public repo, so community could contribute (both code and suggestions)
• proposal for added coverage for:
  • GraphQL schema
  • XML changes
  • improve DB changes detection:
    • make whitelists more granular
    • more granular reports for changes in db_schema_whitelist.json files
  • investigate other areas that contribute to breaking changes

Review magento/devdocs#3003

For 2.3, Zend_Mail has been removed, as a result Magento interfaces were changed. Currently there are two interfaces: \Magento\Framework\Mail\MessageInterface (deprecated) and \Magento\Framework\Mail\MailMessageInterface - both used in Magento code.
But the biggest issue is that new interfaces don't cover some of necessary methods that existed in Zend_Mail. Current request from the community is to have getFrom(), getTo(), getCC() and getBCC() methods, as well as methods for managing attachments, supported and they need it in 2.3.

AC:

• update usages of \Magento\Framework\Mail\MessageInterface with the new interface
• review \Magento\Framework\Mail\MailMessageInterface in comparison with the old Zend_Mail interface and add missing methods to the interface
  • there is a request for getFrom() method which is necessary for an extension to work (Wouter Samaey from community)
  • propose a solution for 2.3 patch release

Please add your topic as a comment to the issue. Use following format:
Topic description and link to PR, if any (duration in min)

Meeting Notes

• @melnikovi : splitting modules into new structure
  • need to split base package to separate endpoints
  • cron as a separate service in the future? - yes, we need it, but let's describe all the thoughts and discuss with the community
• @rgoncharuk
  • project board
  • we have teams for the work, will start in a few months, will invite them to the meeting
• @shiftedreality : can help @paliarush with DevBox

The ece-tools package is a thing of beauty. Command line build and deployment tools. Docker compose file creation. An elegant core patch system. Having this tool publicly available would help so many who currently have to roll their own solutions

Meeting Notes

• Igor
  • Pushed branch with checkout service prototype to public https://github.com/magento/magento2ce/tree/checkout-service
  • Look into exception handling in communication between services
• Eugene
  • Prototype for authorization and authentication
  • Looking at proposal for inter service communication
• Victor
  • Updated proposal for caching
• Vladyslav
  • Checkout service works on dev box
  • Created issues under the dev box project
  • Next step is to add tests

Note: there are open questions about communication between services, but other people needed to discuss.

Should API return exceptions that then will be thrown or it should return error codes and then proxy implementation would handle and throw appropriate exception?
@kandy API should return exceptions. Whether API return exceptions or code are details of implementations.

Review current extensibility strategy for Page Builder. Propose a long-term and short-term improvements, if necessary.

Prepare a proposal in scope of Services Isolation work that covers impact on Magento extensions.

The document should include:

• decomposition of Magento core packages with lowest impact on extensions
  • expected impact is clearly described
• expected work for extension developers to support new Magento structure
• gradual plan as we move through different stages of Magento evolution: modularity, services decomposition, distributed deployment
  • will it be necessary to distinguish extensions that support distributed deployment or any other changes in Magento framework?
• changes to Marketplace extension submission workflow

Include Ravi Menon, Lars Roettig, Navarr Barnier, Kristof Fooman to discussions on this topic.