feat: run create-t3-app in existing projects to add missing parts about create-t3-app HOT 27 CLOSED

Finally some update on this. I'm making some progress on moving this project towards using git patches instead of copying and pasting files. I have a partially working project (has tailwind and trpc). If anyone wants to test this, you could run:

git clone https://github.com/minsk-dev/create-t3-app-template
cd create-t3-app-template

git checkout pkg/tailwind
git format-patch main -o patches -U1
git checkout pkg/tailwind+trpc
git format-patch main -o patches -U1
git checkout pkg/trpc
git format-patch main -o patches -U1

git checkout main
git checkout -b dev

git apply patches/0001-feat-tailwind-adds-boilerplate.patch
git apply patches/0002-feat-tailwind-adds-dependencies.patch
git apply patches/0003-feat-tailwind-adds-imports.patch
git apply patches/0004-feat-tailwind-updates-index.patch

git apply patches/0001-feat-trpc-adds-boilerplate.patch
git apply patches/0002-feat-trpc-adds-dependencies.patch
git apply patches/0003-feat-trpc-adds-api.patch
git apply patches/0004-feat-trpc-finishes-setup.patch

git apply patches/0005-feat-tailwind-trpc-adds-query.patch

echo "patches" >> .gitignore
git add .
git commit -m "chore: inits project"

pnpm install
pnpm dev

Going to create some quality of life for developing with this template, but this is very doable.

from create-t3-app.

To expand a bit the idea with patches, take the template/addons/trpc/base-context.ts file (I added the async word to avoid some conflicts)

// src/server/router/context.ts
import * as trpc from "@trpc/server";
import * as trpcNext from "@trpc/server/adapters/next";

export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
  const req = opts?.req;
  const res = opts?.res;

  return {
    req,
    res,
  };
};

type Context = trpc.inferAsyncReturnType<typeof createContext>;

export const createRouter = () => trpc.router<Context>();

Then we could have 2 patches:

• template/addons/trpc/context-prisma.diff:

diff --git a/template/addons/trpc/base-context.ts b/template/addons/trpc/base-context.ts
index f66d8d9..b6cbb39 100644
--- a/template/addons/trpc/base-context.ts
+++ b/template/addons/trpc/base-context.ts
@@ -2,6 +2,8 @@
 import * as trpc from "@trpc/server";
 import * as trpcNext from "@trpc/server/adapters/next";
 
+import { prisma } from "../db/client";
+
 export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
   const req = opts?.req;
   const res = opts?.res;
@@ -9,6 +11,7 @@ export const createContext = async (opts?: trpcNext.CreateNextContextOptions) =>
   return {
     req,
     res,
+    prisma,
   };
 };
 

• template/addons/trpc/context-auth.diff:

diff --git a/template/addons/trpc/base-context.ts b/template/addons/trpc/base-context.ts
index f66d8d9..36fe3ab 100644
--- a/template/addons/trpc/base-context.ts
+++ b/template/addons/trpc/base-context.ts
@@ -1,14 +1,21 @@
 // src/server/router/context.ts
 import * as trpc from "@trpc/server";
 import * as trpcNext from "@trpc/server/adapters/next";
+import { unstable_getServerSession as getServerSession } from "next-auth";
+
+import { authOptions as nextAuthOptions } from "../../pages/api/auth/[...nextauth]";
 
 export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
   const req = opts?.req;
   const res = opts?.res;
 
+  const session =
+    req && res && (await getServerSession(req, res, nextAuthOptions));
+
   return {
     req,
     res,
+    session,
   };
 };
 

Applying them one after the other:

➜ git apply --3way template/addons/trpc/context-prisma.diff
Applied patch to 'template/addons/trpc/base-context.ts' cleanly.

➜ git apply --3way template/addons/trpc/context-auth.diff  
Applied patch to 'template/addons/trpc/base-context.ts' with conflicts.
U template/addons/trpc/base-context.ts

And would yield the following file with some conflicts:

// src/server/router/context.ts
import * as trpc from "@trpc/server";
import * as trpcNext from "@trpc/server/adapters/next";
import { unstable_getServerSession as getServerSession } from "next-auth";

import { authOptions as nextAuthOptions } from "../../pages/api/auth/[...nextauth]";

import { prisma } from "../db/client";

export const createContext = async (opts?: trpcNext.CreateNextContextOptions) => {
  const req = opts?.req;
  const res = opts?.res;

  const session =
    req && res && (await getServerSession(req, res, nextAuthOptions));

  return {
    req,
    res,
<<<<<<< ours
    prisma,
=======
    session,
>>>>>>> theirs
  };
};

type Context = trpc.inferAsyncReturnType<typeof createContext>;

export const createRouter = () => trpc.router<Context>();

This possible conflicts would be a problem, not sure how they could be handled during installation.

from create-t3-app.

Thanks for your kind words!

I gave up on codemods quite early because it seemed really overcomplicated

Yes the codemods literally blew my mind initially 🤯 it feels such a powerful tool though.

but this could definitely be used for the scaffold process!

My thoughts exactly. IMHO at least it's worth just for conditionally apply changes in the initial installation.

I think this is beyond a scaffolding CLI

To be honest with you, I'm really interested into looking for a potential solution for this because the starters/CLIs I've used are consumed in a way that your created project becomes a snapshot of that starter/CLIs in that point in time and then you are on your own (it's difficult to add updates/new features added in starter). And personally I've been into situations where I'm maintaining several projects created from the same starter that they require a lot of time to keep updated.

Jumping back to our issue at hand, I'll try to break down the next decisions we need to take:

• (a) how to trigger new installations
• (b) how to detect if a project was created using create-t3-app
• (c) what the process of installation of a new package should do

(a) how to trigger new installations

I was thinking that running the create-t3-app from a project created with it should detect it and prompt different options than when creating a new project. Initially we would allow to install new packages, but new features could be potentially added (example: update installed packages).

This feels like a solution relatively simple to implement and easy to consume, as we would remove the need to have different CLIs or commands, but let me know your thoughts on this.

(b) how to detect if a project was created using create-t3-app

AKA: how to detect if it's a t3-app

I see 2 detection approaches:

1. store t3-app information somewhere we can read afterwards. We want to store information such as the version of create-t3-app it was installed from, packages installed, etc. We could store it in package.json under a t3app property or in a new file .t3apprc.
2. write code to detect everytime the create-t3-app is run. For the project I don't know how to detect it, but for each package would be possible (check if certain files added by the package exist)

I think option (1) is much easier and clearer. The main drawback is that users will be free to change the package.json (or whatever file we store the state of the installation) and it would cause to yield different results or even malfunction. Note that future modifications of this data should be commited by the user so missing that could cause issues aswell.

(c) what the process of installation of a new package should do

When selection a new package we must:

1. run the installer for that package (duh)
2. run the installer for all other installed packages, as other installers can apply codemods to their own files based on the newly installed package

Step 1 is pretty straightforward: can behave as if it was the initial installation

For step 2, I see some complications:

• [a] we should avoid copying files from templates to avoid replacing userland code
• [b] codemods should add only features for the package being installed.

In order to properly comply with the above, we need to track which packages are being installed. This is actually what the PkgInstallerMap[keyof AvailablePackages].inUse property does. IMHO we should rename this property to installing and add installed for the packages already installed. For example, imagine that we install trpc on initial project creation. Later we want to install prisma, so the packages object passed to the installers would be:

{
  trpc: { installed: true, installing: false },
  prisma: { installed: false, installing: true },
}

This would allow the trpc installer to run conditional logic:

// in /addons/installers/trpc.ts
if (packages.trpc.installing) {
  // case [a]: initial installation of the package, copy templates files
}
if (packages.prisma.installing || (packages.trpc.installing && packages.prisma.installed)) {
  // case [b]: prisma is being installed OR trpc is being installed and prisma was already installed
  // apply codemods to trpc files with prisma modifications
}

Note that some codemods would still be applied once (ex: adding the same import or adding a variable declaration should be skipped if already there).

Sorry for the wall of text. I'm probably missing ton of things but with the above tackled we could have a good starting point.

from create-t3-app.

As cool as this sounds, it would be nye impossible to do. IMO this would be a different tool (and should probably be built by the maintainers of the specific deps)

from create-t3-app.

So you really got me thinking about how this could be possible. My curiosity has gotten the better of me and I am experimenting with using jscodeshift to build an AST we can parse, and update, to add in new packages. Will report back if I can get anything even remotely working.

from create-t3-app.

Hey! Just wanted to give my input on this because I actually built a similar project a while ago called next-gen which has this exact feature. It's currently unmaintained because I don't have the capacity, oh well..

Imo using a library like jscodeshift will be absolutely indispensable for this, the other approaches I tried were way too brittle, It's a bit complicated to get started, but once you get the hang of it it will make the whole process a lot easier.

Blitz does a lot of things with jscodeshift, they've written a really nice abstraction for creating your own new recipes as well, you should take a look: https://github.com/blitz-js/blitz/tree/canary/recipes.

The nice thing about the jscodeshift approach over templates for scaffolding is that you immensely cut down on combinations of templates. I.e. you won't need a _document.tsx version for withPackageA, withPackageB, withPackageA+B, ... You can just write the codemods and sequentially apply them. This was actually the reason why I initially used them for next-gen, because I wanted to support as many different combinations as possible.

I'm also in the discord, hmu if you have any questions @ochicf .

from create-t3-app.

Trying to think through the steps required here:

• User runs ct3a and requests to incrementally add XYZ package
• Parse the users current codebase to see which packages they do and don't currently have
• Add in the new files in a way that doesn't overwrite their current files
• Make sure the new files added work with the users already setup application

That's going to be a little difficult. And by a little, I mean damn near impossible. If anyone can find a way to make this work, I will be absurdly impressed.

from create-t3-app.

I've pushed more changes. Some comments:

• moved all the codemods to a different folder in root and configured it as a project reference. I'm not an expert with TS but I had to do this for everything to work smooth. See readme for more info: template/README.md#why-a-sub-project
• I've implemented a way to prevent codemods from being applied multiple times if the same installation is run (really naive and not ideal, but works ATM)
• I've added more helpers to work with codemods
• I've added some tag templates to work with typescript template strings. Also by installing this VSCode extension we get syntax highlighting:
  

Now that I have worked out the codemods stuff (and I'm quite happy with it) I'll do an attempt of allowing installation of new packages. Will post here if I make any progress.

I'm also not sure If I have to keep spamming this issue or maybe create a PR to avoid notifying everyone?

from create-t3-app.

BTW @juliusmarminge I don't have right to modify the GitHub project (cannot add items/tasks to it)

Now you do

from create-t3-app.

we can use my fork's branch as starting point? how could multiple people push there?

FWIW, I think we should target the @next branch so we dont do something that will have to be changed in the near future.

from create-t3-app.

I love this idea, but how would the implementation work? There is the possibility to break existing code etc.

from create-t3-app.

Yeah with the current architecture this is not possible since we copy entire files over. If you have clashing files all your current progress would be overwritten.

from create-t3-app.

First of all I want to say thank you all for the great work and effort you guys put into this. I'm loving this stack so far and I see great potential, I'm definitely using it when I need to do some new project.

The topic of this issue is quite complex stuff and as you said, nearly impossible because once the files go to user land this tool will lose control and won't be able to easily modify those files. With this clarified, I have some ideas to throw into the mix. Sorry if I don't explain something properly as English is not my native language.

Imagine that each addon can define 2 different types of template files:

• base/core files: this would be files to be copied entirely when the addon is installed (this is what it's currently being done)
• modification files: these files would describe modifications to specific files (implementation TBD). The target files could be the base files in the project templates, base files of the addon itself or base files belonging to other addons

The addon would perform operations upon installing, which could be copying base files (currently done) and applying patches to certain files (those could be conditionally applied based on installed packages aswell). The installation process could have some sort of flags in order to know whether it's a fresh/initial installation or a later installation of a single addon (though this installation process should always be aware of the already installed addons).

Difficulties:

• order of the operations is critical, and it also can be affected by the order of the addons (as they could patch files belonging to other addons that may need to be created first)
• how the modification files are implemented: my brain thinks of diff files and running git patch. This would easily allow to add/remove lines for existing files (though it's quite strict and maybe difficult to mantain aswell)
• how to handle changing code within ct3app itself to avoid clashes when installing addons for a created project some time ago? Versioning?
• how to know wich addons have been already installed?
• should addons have versioning aswell to better tolerate future changes?

Extra benefits:

• would prevent the addons from having the same file multiple time depending on the other addons installed (example: the trpc addon currently has base-context.js, auth-context.js, auth-prisma-context), and just have the base file and then the modifications applied depending on the other installed addons.

Another semi-related idea:

• define an addon configuration in a more declarative manner to pass it to runPkgManagerInstall so it could run the operations, instead than of the current imperative way

I'll keep thinking on this.

from create-t3-app.

Sorry to spam, but I have dreamed about this.

Let's asume for a moment that the files are generated using a template engine such as handlebars, ejs, eta... Then the same template would be used to generate the same file with different configs. Asume aswell that in the template variables we can find which addons are being installed so we can add code conditionally.

With this in mind, take that the user does a fresh with just one addon (say tRPC). The templates would yield the files without any code for any other addon. If then the user wants to install a new addon (say Prisma), the installer code would do the following for each installed addon (just tTRPC in this case):

1. create the templates without the addon being installed: this would be the "origin" file (same as the one generated in the original install, would only have the tRPC part of code)
2. create the templates with the addon being installed: this would be the "destination" file (would have tRPC + Prisma code)
3. for each file, calculate a diff between "origin" and "destination"
4. apply patch of the generated diffs to the existing files in the filesystem

Considerations:

• in step 1 we re-generate the original because we cannot use the file already saved in the created project (as we don't know if the user has modified it). We only want the changes between the file that ct3app saved in the original install and the file that ct3app would have saved if the new package would have been originally installed
• in step 4 there could arise conflicts if the user has modified the code. This wouldn't be much of an issue as if we apply the patch using git apply --3way then those conflicts could be handled by the user with their conflict resolution tool of choice (ex: easily done in most IDEs)

For the examples I'm going to use EJS and some simpler code as it was a file in the tRPC addon. Take that we have the following template file in template/addons/trpc/context.ejs:

import * as trpc from "@trpc/server";
<% if (addons.nextAuth.inUse) { %>
import { authOptions as nextAuthOptions } from "../../pages/api/auth/[...nextauth]";
<% } %>
<% if (addons.prisma.inUse) { %>
import { prisma } from "../db/client";
<% } %>

If the user initially installs just tRPC, this would yield:

import * as trpc from "@trpc/server";

If the user then installs the Prisma addon, following the above described flow we would get:

1. "origin":

import * as trpc from "@trpc/server";

2. "destination":

import * as trpc from "@trpc/server";
+import { prisma } from "../db/client";

3. diff between "origin" and "destination":

diff --git a/template/addons/trpc/context.ts b/template/addons/trpc/context.ts
index 8e7351d..81efa94 100644
--- a/template/addons/trpc/context.ts
+++ b/template/addons/trpc/context.ts
@@ -1 +1,2 @@
-import * as trpc from "@trpc/server";
\ No newline at end of file
+import * as trpc from "@trpc/server";
+import { prisma } from "../db/client";

Then for step 4 we could encounter 2 cases:

(a). If the user has not modified the file, applying the patch yields no conflicts and we would get the proper merged file:

import * as trpc from "@trpc/server";
import { prisma } from "../db/client";

(b). If the user has modified the file, let's say to this:

import * as trpc from "@trpc/server";
import * as userLandPackage from "@userland/package";

Then applying the patch would result in conflicts, that would need to be managed by the user:

import * as trpc from "@trpc/server";
<<<<<<< ours
import * as userLandPackage from "@userland/package";
=======
import { prisma } from "../db/client";
>>>>>>> theirs

I hope the above makes sense 🤔

from create-t3-app.

@ochicf I think if we choose to let the user to handle any of the merge conflict by using the patch, it will break the purpose of this project itself. Since the purpose of it is to simplify the process of creating the boilerplate so people can use it and add what they need after that. and by letting the user resolve the merge conflict manually will make it much painful to the user itself since probably they will have a ton of merge conflict.

from create-t3-app.

@krsbx you have a really good point. Maybe I wasn't clear enough, but with my last comment I was disregarding my own idea of patching files when creating the project the first time. Hence, creating the initial boilerplate would result in a conflict free project, as we would be just using a template engine to handle conditionals and other injected stuff in the generated files.

But to solve the problem of incrementally add and modify the files once the project is created, I cannot think of anything that could ensure that we don't potentially get conflicts because of the freedom by the user to modify the files freely. My proposal only would yield conflicts when installing addons afterwards, and if the user has modified some of files affected by the new install... which IMHO it's still better than have to do all modifications manually.

Of course It's far from ideal and would add other problems and complexities, I was just trying to throw ideas into the mix to see if we find a way to handle this and it's worth implementing it (which I have some doubts ATM).

from create-t3-app.

Could also look at astro's way of doing their astro add <integration> command. it lets you incrementally add integrations and modifies the necessary files:

from create-t3-app.

I've been playing a bit with the jscodeshift and managed to modify the files during initial installation. For now I've focused on testing the codemods to see if it's the right approach, so then we could implement the script to install addons after the initial installation.

I've pushed some code in my fork with a test just for the trpc/base-context.ts file. See diffs with main.

Some highlights:

• with current changes we could remove the different template files and just keep the trpc/base-context.ts
• I've added a transform suffixed file matching the name of the template, in this case trpc/base-context-transform.ts
• regarding the codemods in general:
  • they can be a bit tricky to write (I feel as a complete noob working with this)
  • if encapsulated in helpers, they can be easily read (check trpc/base-context-transform.ts and let me know your thoughts on this)
  • current codemods are relativelly fragile depending on the changes performed on the file. I guess more resilient codemods will increase in complexity
• regarding codemods as installation tool:
  • we can pass options to transform functions (exemplified with usingPrisma and usingAuth). Other options could be passed depending on the codemod. Note that this is a great tool also for handling relative imports, as instead of being "harcoded" in the template we can dynamically resolve them during installation
  • currently some codemods will re-add the same code every time the transformer is run, we should prevent this in final code

Would love to hear your thoughts.

EDIT: added a commit fixing an import, for some reason when running the transform file in jscodeshift thread does not find module if imported with .js extension

from create-t3-app.

Very interesting stuff you have made. I gave up on codemods quite early because it seemed really overcomplicated for what I initially wanted it to be (maybe there is a market for a better tool here @MWhite-22 😃)

Will post here if I make any progress.

Sounds good. If you get it to a working point without it being too complex for contributors to use I would gladly incorporate this into the existing installers, don't know about you @nexxeln?

I'm also not sure If I have to keep spamming this issue or maybe create a PR to avoid notifying everyone?

I don't think anyone minds the "spam", the findings are very interested (I'll admit the initial "git --diff" approach was beyond my knowledge 😉). With that said, when you get to a somewhat working point, perhaps a draft PR is more appropriate?

Super appreciate the work you have put into this. I don't know if this issue's original point (adding dependencies to an existing project) is something we want to incorporate (I think this is beyond a scaffolding CLI which this is, please confirm @theobr & @nexxeln), but this could definitely be used for the scaffold process!

from create-t3-app.

Yeah we can definitely include it in the scaffolding, as for the installing deps thing, that can be made into another package?

Also this is really exciting stuff!

from create-t3-app.

Quick update

I've created a new branch in my fork (created from the codemods branch) where I'm handling all of these reinstalls/incremental feature addition. IMHO it would be best to first migrate all existing code to apply codemods, so I wanted to keep my original branch branch with just that work so we can work on that independently. @biowaffeln this is where your knowledge of codemods and the references you gave will be super helpful ❤️

Now I'll quote some parts of my previous comment that I could work on:

(b) how to detect if a project was created using create-t3-app

Did some basic implementation to store the installation info in package.json.

{
  "name": "my-t3-app",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "postinstall": "prisma generate"
  },
  "dependencies": {
     // ...
  },
  "devDependencies": {
     // ...
  },
  "createT3App": {
    "version": "5.0.0-beta.7",
    "createdAt": "2022-07-18T11:43:13.512Z",
    "updatedAt": null,
    "packages": {
      "nextAuth": {
        "installed": true
      },
      "prisma": {
        "installed": true
      },
      "tailwind": {
        "installed": false
      },
      "trpc": {
        "installed": true
      }
    }
  }
}

Some notes:

• added createdAt and updatedAt because I can, not because they really matter 😆
• I've stored an object for each package in case we want to add more info (such as configuration used?). Not sure if it's needed though. I'm also thinking that we could skip the non-installed packages instead of having them with installed: false
• what extra data could be useful to store here?

(c) what the process of installation of a new package should do
[...] This is actually what the PkgInstallerMap[keyof AvailablePackages].inUse property does. IMHO we should rename this property to installing and add installed for the packages already installed. [...]

I've done this, but I've kept the inUse for now, being:

const pkg: AvailablePackages = 'tailwind';
const installed = projectPkgJson?.createT3App?.packages?.[pkg]?.installed ?? false;
const installing = packages.includes(pkg);
const inUse = installed || installing;

Next steps

I'll continue working on the reinstalls/incremental feature addition. My goal is to "close the circle" and have it implemented for at least a combination of 2 packages (ex: install 1 first, then install the other and expect all the changes to apply).

from create-t3-app.

moved all the codemods to a different folder in root and configured it as a project reference. I'm not an expert with TS but I had to do this for everything to work smooth. See readme for more info: template/README.md#why-a-sub-project

I see potential here for a separate package that's usable in other projects as well. We can create a new repo under t3-oss if you're down and have it as a dependency for create-t3-app. That way other projects can use it as well. What do you think?

from create-t3-app.

I see potential here for a separate package that's usable in other projects as well. We can create a new repo under t3-oss if you're down and have it as a dependency for create-t3-app. That way other projects can use it as well. What do you think?

That would be so cool. But to be on the safe side maybe I would keep it as a referenced project during the the refactor and make it a separate package afterwards?

Not sure if it's the best approach but I'm feeling such a noob with codemods that I feel it's too ambitious to open that package from minute zero 😆

An important thing though: there will be codemods super specific with the templates of this project so maybe those should stay in the project itself (which is closer to the actual files being modified), and leave the external package for helpers/common codemods?

from create-t3-app.

@juliusmarminge answering #208 (comment) in this issue.

I am more than happy to jump in and work on some parts of this. Is the basic transforms done? In that case we can merge them in here somewhere and I (and perhaps others) can start working on some scaffolding stuff maybe?

I only applied transform for a single file as a proof of concept trpc/base-context.ts.

Should we maybe create a new issue regarding codemods and regroup there? This way we separate from this, which needs the codemods done.

Also in order to properly split the work:

• we can use my fork's branch as starting point? how could multiple people push there?
• would be worth having a check list of all the files that need transform so we can keep track of what is being done, finished and by whom? Or maybe better than a checklist: is there a possibility to use the GitHub projects feature in this repo?

from create-t3-app.

List of file changes

Here you have a (semi)detailed list of the affected files. I tried to keep simple with symbols and such but... I don't know if it's clear as I'm doing this with my eyes half closed 😴

TLDR: Summary of codemods

(you can search the codemod key to see affected files in the wall of text above)

[env-1]

• transform file: /transform/base/src/server/env-schema.mjs
• target file: /template/base/src/server/env-schema.mjs

[auth-1]

• transform file: /transform/addons/next-auth/api-handler.ts
• target file: /template/addons/next-auth/api-handler.ts

[trpc-1]

• transform file: /transform/addons/trpc/context.ts
• target file: /template/addons/trpc/context.ts

[trpc-2]

• transform file: /transform/addons/trpc/index-router.ts
• target file: /template/addons/trpc/index-router.ts

[pages-1]

• transform file: /transform/base/src/pages/_app.ts
• target file: /template/base/src/pages/_app.ts

[pages-2]

• transform file: /transform/base/src/pages/index.ts
• target file: /template/base/src/pages/index.ts

BTW @juliusmarminge I don't have right to modify the GitHub project (cannot add items/tasks to it)

from create-t3-app.

Sorry for the radio silence, I'm currently swamped with work as we are trying to deliver a lot of things before the holidays.

On monday we did a call with @biowaffeln to put things in common and he has done a great job and managed to advance a lot with the codemods. He has a lot more experience with those so he will submit a MR from his fork and we can review and test it. @biowaffeln let me know if you need anything from me ;)

Once that's up we will have the base we need to keep working on incrementally adding features.

from create-t3-app.

moving this over to marlalain/t3-cli#2. if no one opposes

from create-t3-app.