While it would be amazing if people would just use modular from the start of time, in reality there's more value in helping people migrate their existing projects to modular, ideally in an incremental fashion. We should write a document that goes through the things one has to do to achieve that. Further, if there are tools we can provide that make it easier, that would be dope too.

Right now the app fails to compile if type check fails. Seems very strict, and would be annoying especially when prototyping, etc. I'd like to discuss whether we should keep it this strict.

ref https://github.com/jpmorganchase/modular/pull/45/files#r472932015

I was thinking of 'multiple apps' with modular. We already have a couple of usecases popping up, and I can imagine it's not really an edge case for most people. I'd think there are 3 'kinds' of scenarios/solutions:

• The current solution is to use dynamic imports + React.lazy for every entry, and use a router to load them on different paths.

  • pros
    • There's no extra work here, already supported.
    • Conceptually simple
    • Single build, shared assets, one unit.
    • Always available as an option, and can be used alongside any other solution.
  • cons
    • Initial waterfall could be slow.
    • Solving waterfall with prefetch requires server infra
    • Doesn't solve a possible scenario, where multiple apps might need to load different things on the same routes (unlikely, but still)
    • Might need to explicitly disallow routes for different ACLs.

• Separate apps, basically separate CRA instances. So you could have packages/app1 and packages/app2 be full fledged CRA generated repositories.

  • pros
    • Easy-ish to do
    • No extra config per se
    • modular build should accept a entry input; e.g. modular build packages/app2
  • cons
    • The /public boilerplate is annoying, could do better?
    • Super slow, since it effectively doubles build time (or how many ever apps are there)
    • Doesn't reuse work across builds
    • Must be served from different (sub)domains / requires more server infra/config

• We could have multiple entry points in the generated bundle. CRA currently assumes src/index.js; we could theoretically add more. So the command to build would be modular build packages/app/src/index.js packages/app/src/docs.js ....

  • pros
    • This is all in one build, and it would reuse work for all the entry points by default
    • Conceptually simple to understand.
  • cons
    • This is ine one build, so if it's a small change in an entry point that represents a small app, the build would be slow since it has to rebuild the whole app (unless we can figure out incremental build)
    • Unsure whether it's as simple as passing multiple entries to webpack (https://webpack.js.org/concepts/entry-points/), might break other assumptions in CRA's infra.
    • Will require custom server infra that loads the right entry in the html for a given route or whatever.

It feels to me like a combination of options 1 and 2 is the right way to go about this, and we solve the cons (figure out a way to inline prefetches for a given entry, figure out incremental builds, etc.) Open to input.

stub issue, will fill in details later

Other than a small para on motivation, the main README doesn't need to document all the thinking behind this project. The tool itself does very little (It is, after all, just a shim around create-react-app), but leverages fundamentals and education to achieve product goals. We should move all the stuff about concerns and usecases into /docs (or perhaps even the wiki)

If this is about owning the toolchain, we have the opportunity of setting up some default extensions and other dx-y stuff for freshly created repositories. (e.g., the eslint extension isn't installed by default with vscode, but seems fairly critical.) What would we add to this?

The assumption is that one repo = one project. This isn't a bad default to have, and is one level of abstraction above CRA's one repo = one app. And we support having multiple apps in a project too, so that gets us even further mileage.

But in a truly monorepo world, one should be able to have 2 (or a 1000) projects in one repo. I wouldn't recommend it, but software is as software does.

We can support this by simply not initialising a git repo at the root of the project (This assumes a user does the git management by themselves. Heck, maybe they use mercurial.)

This isn't just an option for multiple projects in one repo; maybe a user just wants their project inside a deeper folder in their repo. I dunno! I think the option might be useful tho.

Opening this issue to discuss whether it makes sense, and what the ramifications of the decision are; what features/optimisations does and does it not enable/disable?

context here: #73 (comment)

would be nice instead to have the modular logo, and maybe a little custom styling.

When e2e tests fail, it's left hanging for a period of time; this will eat up our available minutes. We should disable it until we fix that.

We should do this before the 8th. I'll send a PR later today.

Features that need to be implemented for a working Modular Core

• A sealed JavaScript Distribution

• Core Vendor packages like React, React-Dom etc.

• A Modular set of packages

• CSP etc.

• Modular Design

Core App

• What does an entry point for an "App" look like?
• How does tree shaking look like (Multi-Version or Multi-Package resolution)
• Discovery of Module Map
• Loading of Module Map
• Loading of a Module
• Error Handling
• Provision CDN
• SPA and Routing

https://github.com/jpmorganchase/modular/wiki/0001-modular-core

We support writing tests with jest, this covers unit/integration testing (the line between the 2 is blurry for UI tests anyway). We want to favour integration tests for 90%+ of tests.

It still would be nice to support "e2e" tests though, since there's a preference for teams to have them. Some benefits -

• people can test end to end flows that include stuff like logging in, data requests, etc (it's arguable whether that's actually a good thing)
• might be a better way of hitting coverage statistics than leaning on stuff like snapshot tests
• I dunno :)

Broadly, (based off discussion with the mtk folks), some points -

• cypress seems like the best-in-class for writing e2e tests
• storybook provides a good experience for expressing the multitude of prop combinations and behaviour for a component, and has its fans
• nx has a feature where they combine cypress and storybook https://blog.nrwl.io/ui-testing-with-storybook-and-nx-4b86975224c we should investigate this in detail

Open to comments here. In my next comment, I'll propose a possible structure/solution/integration, unless someone beats me to it.

...when it should be using the local version on verdaccio. context: #73 (comment)

Configure tsconfig.json to only load @types/node for Node programs, and not to pollute code which runs on the browser with the following types.

Issue brought up in #8 (comment) and #1 (comment) (although the latter is more from ESLint's perspective).

I suspect 'solution-style tsconfig.json' are the right approach but these are very new so might lack support.

• CONTRIBUTING.md
• CODE_OF_CONDUCT.md #11
• SECURITY.md
• README.md
• CODEOWNERS.md

When running yarn create modular-react-app my-app-name I am getting the following not permitted error:

This looks to be related to:

fs.copySync(
    path.join(newModularRoot, '.gitignore'),
    path.join(newModularRoot, '.eslintignore'),
);

https://github.com/jpmorganchase/modular/blob/master/packages/create-modular-react-app/src/cli.ts#L145

Should the proposed fix be:

fs.copySync(
    path.join(templatePath, '.gitignore'),
    path.join(newModularRoot, '.gitignore'),
);
fs.copySync(
    path.join(templatePath, '.eslintignore'),
    path.join(newModularRoot, '.eslintignore'),
);

Modular design allowing for groups of UI components to be independently developed, tested, and deployed.

• Independent Module Generation

Module Generator

Assuming npx is installed

npx -p yo -p @jpmorganchase/generate-modular-module --yo @jpmorganchase/modular-module

The generator will start and prompt for the following questions:

• What is your lob-namespace(optional, if none provided your module will be scoped to jpmc-modules)?
• What is your module name(required)?
• Will your module have an entry point (y/N)?

Optional Flags

• Type of Module?
  setupTypeOfModule
  • Data Viz
  • Static Content
  • Other
• More to come!

• Independent Module Testing & Assurance Framework

• Independent Deployment Process

(Placeholder Issue) Do we need a RFC for each of the above or even split each task into it's own issue?

We should debug and resolve why the E2E tests need --forceExit when run in a Docker container (or on CI).

See the following comment for further information:

I don't think people would drop source code in here that expects to be typechecked/linted/prettified. We should exclude public from those tools.

Stub issue for how to (or even whether) to allow/enable multiple applications in a modular repo. Will fill in details later.

I can't think of a scenario where it gets published to a registry (or even when it's version would matter); ergo, it should probably be private.

Should we match https://github.com/facebook/create-react-app/blob/2da5517689b7510ff8d8b0148ce372782cb285d7/packages/eslint-config-react-app/package.json#L17-L27 and specify them as peerDependencies instead? You wouldn't want dependencies like typescript to be brought in via an ESLint config package.

Originally posted by @NMinhNguyen in #27

• yarn create modular-react-app xyz should create a repo with package.json's name as '@xyz/workspace' (or '@xyz/root')
• packages, including the app should be prefixed with '@xyz' (except for the default 'shared' package, maybe?)

Things to do before we open it up internally at JPM:

• basic documentation; what the different commands do
• a full fledged example
• workflow documentation; how code reviews would be assigned in github/bitbucket, sparse checkouts, etc
• --template <template> option for create-modular-react-app, and an internal template that includes our own infra stuff (at least) We won't support templates for a while, see #37
• widgets should probably not have a license field, and we should use a new field (instead of private for deciding whether it's included in the main build)
• verify IDE integrations
• flesh out missing/stub README.md files

I mentioned to @NMinhNguyen that I had a few misgiving when adding a templating feature.

1. Partly, the implementation is a little confusing right now due to the way we are piggy-backing on the installation of create-react-app. There are changes that create-react-app makes which are not part of its template and we then need to undo or amend. It'd be nicer if everything was just copied across from a single folder.

2. More substantively, a widget is dependent on an app, so if the configuration of an app and verson of modular installed doesn't include support for TypeScript you should not be able to install a widget that depends on this. I'm not suggesting that we add validation -- probably it means that the template for a widget is stored within the modular package (for now called modular-scripts). This was brought up by @threepointone a week or so ago, and I'm writing it down now for posterity.

3. Finally, there are now three templates (repo, app and widget). Currently, only app and widget are configurable, and we will likely want repo to be configurable, too. (If/when this is required, it might need to support something like wizards.)

We've cruised along with no tests for a while, which is fine, but we're getting to place where we might break stuff while iterating. We have types, which is good, but we should probably have tests too. Some factors:

• it's annoying to test command line programs
• it's annoying to test something that generates a repo with workspaces, from a repo that uses workspaces, because yarn will complain
• it's not clear what we should test, without simply testing CRA's functionality
• Of note, CRA doesn't really have tests either.
• the widget map looks like something that should have tests? Unsure.

Open to ideas.

probably need to tweak a jest config somewhere.

Consider adding https://github.com/atlassian/changesets (see #1 (comment))

Noticed that the e2e tests don't revert all their changes when they fail; I noticed build folders in the source directories, and modified package.json versions that I could have accidentally committed. These should be cleaned on failure.

Bonus: e2e tests fail when there are uncommitted files, which makes dev a bit annoying, having to commit before every run. Could we make this a better experience?

Currently it's an object with one field, 'type', which can have 3 values: 'root', 'widget', 'app' (and a 4th secret one 'package' which is the default). We (I?) need to update the docs with what they mean. Will also affect #67, #65, #39.

First, the bad news. @modular is not an available scope. So this exercise is mostly theoretical, but useful if we can figure out another scope.

(I'm also still trying to get the modular package name, but got busy actually building this thing. Will follow up again.)

Anyway...

I maaay be changing my mind about scoping our packages. Maybe a namespace/scope is the way to go. We can still keep the name of the binary as modular, but I don't like the multiple syllable corner we've backed ourselves into by following CRA's lead (also honestly, create-react-app isn't that great a name. It's very utilitarian, which is good, but isn't meant to be an inspiration.) Here's a proposal for cleaner package names :

@modular/create - replaces create-modular-react-app
@modular/template - replaces cra-template-modular-typescript Users never see this, but still.
@modular/cli - replaces modular-scripts, tho it can still be called with modular like it is now.
@modular/eslint-config - replaces eslint-config-modular-app

(If we were ultra ambitious, we could also consider merging @modular/create and @modular/cli. But one thing at a time.)

Besides, I'm thinking we should lean into scopes for generated repositories as well (#76) and it would be weird if we didn't follow our own recommendation.

(On that note, wish we could use modular to manage modular. Really loving using it otherwise.)

Open to discussion, hit me with your best shot.

This is an umbrella task for all things related to reducing task times in modular projects.

As modular projects grow (as they should), because we do centralised tasks for build/start/test/etc, we will hit bottlenecks in being able to develop and deploy quickly. While this shouldn't affect daily development per se, it'll start affecting productivity as a whole. Some examples:

• a one line change in one package, could take a very long time to get to production because it's triggered a whole test/build cycle in CI/whatever build infrastructure you're using.
• a simple bugfix will have a very long turnaround time to get to production.
• local development will run only some tests, but in CI it might take a very long time to run all tests, meaning a developer will have to wait for a long time to verify whether their code has broken anything.
• modular start might take a long time to warm up, which isn't nice.
• and so on.

(note: We should make a comprehensive list of pain points; the solutions won't be super general, so we should make sure we've looked at every possible pain point.)

(note: this issue is not about runtime performance of react applications, though we should probably make an umbrella task for that too)

Possible solutions and strategies:

• caching third party dependencies, so it doesn't have to be pulled down on every build (existing work: https://circleci.com/docs/2.0/caching/, https://github.com/actions/cache)

• webpack 5's module federation for builds https://webpack.js.org/concepts/module-federation/ We can split the build into pieces (and stitch them up manually, if at all) with this feature; it'll be key that we do this automatically, and without exposing the internals to consumers, or else it'll be hell to unwind later. Things to consider: deduplicating dependencies, verifying module graphs, etc.

• webpack 5's persistent caching https://github.com/webpack/changelog-v5/blob/master/guides/persistent-caching.md Again, it'll be key to do this without exposing any internals to the user.

• configure jest to only run affected tests on builds (https://jestjs.io/docs/en/cli#--changedsince, https://jestjs.io/docs/en/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) A thing to note here is that we should be able to generate complete coverage reports even if we only run a few of them during. (another precedent in java land - https://github.com/jpmorganchase/sandboni-core)

• Use feature flags for rapid release cycles: we should be able to build and ship features rapidly, and be able to turn them on/off as we desire. (https://martinfowler.com/articles/feature-toggles.html, and my own writeup https://gist.github.com/threepointone/2c2fae0622681284410ec9edcc6acf9e)

• lint/prettier only on changed files (copy react's yarn linc, basically): In the modular repo, we've setup linting only for changed files when we do commits, but not as a standalone command. We can copy react's linc command. We should also ship the commit behaviour and commands in generated repositories.

• Incremental builds: This is fairly nascent in production world for the javascript ecosystem; usually because solutions are tightly bound to serving infrastructure and so on. 'Big' companies like fb/google have in-house solutions for the same. There's an opportunity here to start work on a project that's designed for things like from the start. (I hear whispers of parcel 2 also working on similar goals)

Please feel free to add more to this list in the replies, and/or feedback. I'll keep updating this list based on so. If you'd like to start work on any of these, please file a separate issue and link it back here.

Oh, I'd rather we remove support for .jsx entirely for now.

Originally posted by @sebinsua in #33

parent issue for using modular to build libraries.

We have some internal naming conventions we should probably abide by. Also, this thread #35 (comment)

The e2e tests take too long to run, and end up testing cra/yarn’s implementation, which isn’t particularly useful. The commit+test flow is cumbersome and it’s hampering iteration speed. We should mock yarnpkg/cra and verify final folder/file structure (which we already do), that should be enough. This will also let us remove the Verdaccio usage.

I'm wondering if we can just use https://github.com/facebook/create-react-app/tree/master/packages/eslint-config-react-app at the root of the ESLint config and then tweak rules for Node packages where CRA's config doesn't make sense (or perhaps add additional Node-specific rules)

We'll likely need something like Puppeteer for e2e tests and perhaps Jest for unit tests

This issue is simply to discuss the choice of yarn as a critical part of our infra. We don't have to take an immediate call on it, but this issue can serve as a place to collate points that might matter.

We currently use yarn for a couple of things -

• Workspaces are pretty critical for modular. We don't use nested workspaces (only available in yarn v2) yet, and even though we might later, it's not critical.
• When adding a new application (via yarn create modular-react-app)
• At a stretch, yarn.lock for reproducible installs and builds

yarn v2's not been as successful in terms of adoption as it could have been; for reasons of managing community expectations, ease of transition, and generally unclear value proposition. This is a situation that's slowly been getting better, but it's not super encouraging at the moment. Which is a damn shame, because some of the features look nice. This leaves us in a weird place where the preferred version will stagnate.

In my opinion, npx is superior to yarn create. It cleans up after itself after being used, and doesn't require the create suffix, and is pretty fast. Even if it's wasn't superior by feature-set, the fact that it's included with npm makes it easier to adopt.

The upcoming npm v7 (already in beta as of today) supports workspaces, although we haven't evaluated it critically yet. Further, adopting a new version of npm inside the org (and in open source) will take a while.

npm's lockfiles aren't bad, iiuc. There was a discussion a while ago about the comparisons in deterministic installs between the 2 (https://classic.yarnpkg.com/blog/2017/05/31/determinism/), and about the merging experience when on a team, but I'm unsure how that's changed in the last 3.5 years (if at all). Must evaluate.

So broadly, I think we should look at moving away from yarn in the future; This will mean one less 'third party' dependency, which is good. And as mentioned, we can take our time with this decision, as long as we keep an eye on it.

As a corollary, we shouldn't be using any features that lock us in to yarn. As a start, we can move away from using yarn create and instead use npx for adding a new application).

In no particular order, here's a list of 'things' we should address with modular. This doesn't mean that they should be solutions with code, or if code should be included ootb; they could even be recipes/documentation/snippets. Further, orgs may have their own answers for any of these; it may be tempting to offer abstractions when what they really need is guidance. Regardless, we should build a knowledge graph (with foam?) that answers all these questions. Feel free to edit/rename/add/remove.

• make a new application @sebinsua
• make a new widget @sebinsua
• checkout a part of the codebase @threepointone - https://github.com/jpmorganchase/modular/wiki/Sparse-checkouts-with-git
• testing @threepointone / @sebinsua
  • run tests
    • for a subset of files
    • for changed files
    • mocking
      • components
      • data
  • e2e-testing
• layout manager (@susanafcosta)
• automate common code review chores (e.g. via Danger JS)
• add styles
  • design system?
  • product/design/dev handoff?
• send a PR / add reviewers / address feedback / land it
  • bitbucket setup?
  • github setup?
• work on a widget in a separate repository @threepointone
  • release process / CI / CD
  • versioning
  • sanity tests
• add a feature
• analytics
  • error tracking
  • logging
  • product analytics
• routes
• state management
  • global
  • local
  • remote
• data fetching story
• static types: typescript / flow / reason / etc
• auth/auth
• a/b tests
• documentation
• IDE integration
• dev/qa/staging/prod considerations
• concurrent mode @threepointone
• multi platform
• build
  • infra
  • configuration
  • deploy previews
• support channels

• PULL_REQUEST_TEMPLATE.md
• ISSUE_TEMPLATE\BUG_REPORT.md
• ISSUE_TEMPLATE\FEATURE_REQUEST.md

We leverage CRA’s commands to run jest across the entire generated repo, which is fine. Now with support for multiple repos, we just need to make sure it still works as expected. Also need to verify if coverage reports are generated correctly.

yarn lint and yarn prettier work on all files right now, which makes running these a bit slow for no good reason. Instead it should run only on changed files. This should also be a feature that we ship for modular generated projects.

#34 (comment)

It's probably important that we have guidelines for when a module should go in here (for example, it's not good enough if it's shared across 2 workspaces, one could just import from one of them directly; It has to be a utility that's useful AND used across many workspaces.)

The current implementation exposes the internals of modular, and requires installing codegen.macro. This should probably be inside a module. Further, following #49, it'll probably have some 'proprietary' logic, and will be hard to change across releases if we don't encapsulate it.

Opening this issue as a stub, will decide an implementation after #49 lands.

It catches a number of subtle issues that we otherwise barely notice. Useful addition in general.

Also, maybe we should have modular check?

Why tho? lol. Does craco do this behind the scenes or something?

Can confirm by running "yarn global list". Remove it if it's there, run npx create-modular-react-app, and see it again.

Also it might be installing create-modular-react-app globally too, but I've not been able to reproduce it after seeing it once.

Lost some time today wondering why I was generating older version of the app booo.

We plop a tsconfig.json into the generated repository right now, but inline all the settings. This will make it harder to change if and when we make changes in the future (I can already think of a couple of changes I'd like to make). Instead, we should make a tsconfig.json (maybe as a package? dunno) and then in the created file, use extends: "modular.tsconfig".