samchou19815 / website Goto Github PK

Footer hides the button to load more posts.

It will allow them to be mocked and make the terminal less coupled with commands it can handle.

Background

Right now, I have a very messy knowledge management system. A lot of knowledges are scattered all over the place:

• Google Drive: mostly my course material content.
• private-monorepo: a monorepo of some docs and code not appropriate for open source
• samlang: public documentation for samlang project
• Random READMEs in this repo: descriptions about each project.
• [NO LONGER EXISTED]: docs.developersam.com: A place I used to host autogenerated docs.

Some of them are github repos, some are not. Some have build systems, some do not. For projects that have build systems, they are all very different from one another. Until this point, I mostly depend on my brain to figure out where to look for things.

Recently I introduced algolia search to samlang docs. It has been working great so far. To introduce search to all other knowledge management systems require setup self-managed crawler for every single one, which is very time consuming and very unlikely to materialize and keep up-to-date given my time constraint. Therefore, it's time to unity everything into a single system: at least, keep the infra code in the same place.

Implementation

Codegen and build system support

Right now this repo has some basic codegen for github action yml and ignore files. It needs be more powerful than that to support the new vault project. It should be able to automatically pull in another repository in CI as if it is a usual dependency. It also needs to be able to setup different environments in CI for different projects. (e.g. only vault needs google cloud CLI tools)

• Cross repo management
• Customization for individual jobs

Identity Aware Proxy

• MVP: using google cloud IAP to only allow dev-sam to visit vault.developersam.com
• Deploy two services: (1) publicly accessible one (2) IAP protected one

Docusaurus Project Builder

• Create different builds based on local environment variables.
• Create private docs generator
• Provide support for private-monorepo to create deployment triggers.
• Improve sidebar generation

Background

As noted in my open source roadmap, the current Wiki setup is far from ideal. For documentation purpose, let me list the architecture and the build system as the time the issue was created:

• Wiki is a standard Docusaurus project. It follows the standard convention of Docusaurus project layout.
• Wiki has a public build and private build.
  • The public build is self-contained in the repo. Everything need for creating a public repo is already checked in.
  • The private build is NOT self-contained. I hacked together some tools that automatically checkout my private-monorepo in CI, copying a bunch of files, doing a bunch of codegen, and then invoke docusaurus build. The build time is also significantly slowed down, because a lot of markdown files are pulled in.
• Wiki has a public deployment and private deployment.
  • Each deployment correspond to each build described above.
  • Public deployment works the same as the rest of the projects: calls the build script and upload files to Firebase hosting.
  • Private deployment deploys the code to an Google App Engine instance, which is guarded by Google Cloud IAP. The IAP puts a blanket login guard before all possible routes, and I configured in a way that only my main email can pass the check.
  • Deployment to GAE is at least two times slower than deployment to firebase hosting, even though in both cases only static files are deployed.

Pain Points

• Slow build time
• Slow deployment time
• Complex setup
• Mandates cross-repo tooling

Thoughts

When initially designing the system, I seem to limit myself to Docusaurus. To be fair, Docusaurus is quite nice, but we have to accept the fact that it's primary use cases is for static sites. To some extent, a static site does make sense for the documents currently used for a private build. However, limiting ourselves to static site solution unnecessary constrained the architecture choices.

Instead, we could accept the outcome that we lose some of Docusaurus benefits, including

• A lot of nice markdown syntax extension
• Working out-of-box docs plugin
• Painless routing
• etc

, we can design a much more scalable solution. Initial exploration in #790 shows that we can use remarkable to dynamically render markdown with the same feature support that will likely to be used by myself.

Therefore, the wiki project will go dynamic. However, it doesn't mean that we have to completely drop Docusaurus for this. The static public docs can still be served by Docusaurus, and the dynamic content could just be another Docusaurus page. What I will happily give up is the ability to throw a markdown and let Docusaurus handle the rest.

New Architecture

Static public docs will remain fully managed by Docusaurus.

Private docs will no longer be pulled from a repo and built by Docusaurus. Instead, they will be stored in a firebase database, and fetched to frontend on demand. I will use remarkable to render markdown on the frontend.

On the backend, the private docs is protected by firebase security rules. Each document should have a permission tag attached to it, so that permission control can become more flexible in the future. (e.g. Currently it's all or nothing, but in the future I can grant a subset of document to some people, without exposing the rest)

Right now we are building and deploying everything even if some projects have no changes.

TODO:

• Finer-grained CI workflow
• Finer-grained CD workflow
• Finer-grained workflow change detection

Example: it missed #811, whose generated changes are only added in #812.

Background

The repo tools subproject starts a year ago after I realized some limitations of GitHub actions: namely, the inability to provide native dependency analysis. Therefore, the tools start with GitHub Actions workflows codegen support, which does dependency analysis on its own and output the workflows with computed workspace dependencies sticked into trigger condition path.

When the repo is converted into Yarn 2, I have to check in the minified Yarn binary. This change causes the .eslintignore and .prettierignore to be out of sync with .gitignore, which implies that previous symlink approach would no longer work. Therefore, .eslintignore and .prettierignore codegen support is added to the repo tools.

Some time in the past, the repo tools also handled some dependency building, but it's no longer the case. Recently, more features are added to the tool to support automating the private wiki project. At this point, it would be clear that I need to rethink the scope of the repo tool project, to avoid letting it be the heap of everything marginally related to automation.

Project Name

Name: monorail

Rationale

The tool only works for monorepo, so its name should start with mono.

Project Goal

After this project, several things should be achieved:

• The tool can be released as a general purpose monorepo management tool on npm. It can be opinionated in setup, but nothing specific with this website repo can be hardcoded into the source code anymore.
• The tool has a clearly defined public facing API. More concretely, a stable CLI API and configuration format. All exported functions can still be considered as implementation detail.
• The tool can support common cross-repo syncing operations for dev-sam both locally and on CI.

Reinstate lib-firebase!

Monitor https://caniuse.com/usage-table, make the switch when safari 14 usage surpasses that of safari 13.

• dot-dev
• www

Describe the bug
In 1600x900 and 1920x1080 window sizes, scrolling down the homepage results in the example SAMLANG code clipping over the card information.

To Reproduce
👀 View the site in 1600x900 and up. Strange behavior also happens at small screen widths where the About card clips the example SAMLANG code on the LHS.

Screenshots

Environment

• Browser: Chrome 81.0.4044.138
• OS: Windows 10

Notes
The redesign looks really clean, great job :devsam: