uponthesky / techlog-article Goto Github PK

For providing static data such as .txt or .md, we need to read them and send them right away to the user of request.

Consult the following references:

• https://www.geeksforgeeks.org/javascript-program-to-read-text-file/
• https://www.sohamkamani.com/nodejs/file-system/

• make a separate private repo
• AWS SAM config file
• other secret info => AWS parameter store
• neon.tech

Mid-term check: read through the following tutorials and check whether we're missing anything

• fullstackopen

• user sign up

  • the user inputs form data: username, email, and password(POST)
  • data validation check(format) at the web adapter
  • the application checks whether there is a user with the same email or name(including deleted)
  • if there is, raise 400 error, otherwise, respond with 201 created
  • the server stores the values in the DB(password => hashed)

• user sign out

  • the user provides his/her token
  • if the token is valid, the server sets the value of user.deleted_at and auth.deleted_at as datetime.now(timezone.utc)

• user account update

  • email, username, or password only(PATCH)
  • data validation check at the web adapter
  • the application checks whether there is a user with the same email or name(including deleted)
  • if there is, raise 400 error, otherwise respond with 200
  • the server updates the values in the DB(password => hashed)

We have the following error when we run all the tests for confirmation:

  → 19 await prismaClient.article.createMany(
    Unique constraint failed on the constraint: `PRIMARY`

A solution might be this: https://stackoverflow.com/a/67935087

• Microservice & DB: https://www.prisma.io/dataguide/managing-databases/microservices-vs-monoliths#how-do-microservices-affect-database-architecture
• Alembic tutorial
• postgresql install
• DB auth:
  • https://www.prisma.io/dataguide/mysql/authentication-and-authorization
  • https://www.prisma.io/dataguide/mysql/short-guides/creating-deleting-users
• Transactional decorator
  • dead simple python -> class finish
  • implementation
• async logic implementation(sqlalchemy)

• Finish implementing the service layer:
  • article service
  • user service
  • unit testings

Implement the backend api for the login page, solely dedicated to the owner user(the author)

We should fetch a given user's data from the database and authenticate a user.

We will follow the security guide provided by FastAPI docs.

• security - 1
• security - 2
• security - 3
• security - 4
• security - 5

Replace the current postgres DB part with Firestore due to the high cost of either AWS RDS or GCP Cloud SQL.

• #76
• implementing the code
• make the transactional annotation abstract
• #77
• #79
• #80
• Setup the infra using terraform

Roadmap: Read and implement the FastAPI docs basic tutorial part, before Error handling.

Article

• create
• read
• update
• delete

User

• create
• read
• update
• delete
• anything involving security issues related to the router part

User Story

1. a user(already signed up) logs in by sending username and password to the server

• checks whether the given data is valid
• if valid, issues a jwt token to the user(just token, not expiry data)
• if not, raises an 400/404 error

2. a user logged in sends its token to the server

• checks whether the token is valid => dependency
• either invalid or expired: 401/400 error respectively
• if valid: allows the user to do what is allowed for her/him

3. a user tries to log out

• deactivate the token

• implement public interface of 1

• implement public interface of 2

• implement public interface of 3

• revisit: after DB adapter

• revisit: after common utils

• (optional) implement a cache server for the user tokens(Redis) => currently, we only focus on the overall implementation of the application.

Implement the backend api for the main page

API implementation cycle

1. write tests, and check that the test fails
2. implement functionality designed in the test code
3. pass the tests

remark: do DB settings simultaneously

• install: Prisma MySQL tutorial
• .env setting: https://www.prisma.io/docs/guides/development-environment/environment-variables/using-multiple-env-files
• unit testing(for services): https://www.prisma.io/docs/guides/testing/unit-testing#singleton
• integrationo testing(for controllers): https://www.prisma.io/docs/guides/testing/integration-testing
• Database Design(MDN tutorial): https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/mongoose#designing_the_locallibrary_models
• Primsa Advanced Tutorial:
  • https://www.prisma.io/blog/backend-prisma-typescript-orm-with-postgresql-data-modeling-tsjs1ps7kip1
  • https://www.prisma.io/blog/backend-prisma-typescript-orm-with-postgresql-auth-mngp1ps7kip4

On branch develop, finish the basic development environment setup.

In detail:

• initialize node.js project via yarn --init
• initialize TypeScript project via tsc --init
• configure the project in package.json and tsconfig.json
• install basic packages:
  • TS developement: typescript, ts-node-dev(instead of nodemon)
  • linter: eslint(with pre-commit hook)
  • testing: ts-jest(with pre-commit hook)
  • formatting: prettier(with pre-commit hook)

Need to consult:

• fullstackopen
• MDN Express tutorial
• Express MVC example
• Nest.js's basic structure
• Urisign Assignment (in Korean)
• Prisma Example

Basic CRUD jobs

• create

  • input: title(str), content(Optional[str]), and user(author) id(dependency injected)
  • validate at the web adapter => service stage: title should be not null
  • store the data into the database(update both article and user using relationship)
  • return None(201 created)

• read

  • input: article id
  • db query: both article and the author
  • output: title, content, created_at, updated_at, user.name, user.email(200 ok)

• read all

  • input: query string -> offset & limit(pagination)
  • db query: both article and the author(join)
  • output: a list of articles with the same dto as "read"(200 ok)

• update

  • input: article_id, title(optional), content(optional)
  • update the article(update)
  • output: None(200 ok)

• delete

  • input: article_id
  • set deleted_at = now
  • output: None(204 no content)

Testing with FE locally => testing on AWS

Originally intending to work with GCP. However due to the time limit, we'll deploy it to vercel

Separate the logic, such that this service only handles CRUD APIs for the admin user only.

Furthermore, this service will be migrated into FastAPI.

• poetry & pyenv
• install pytest, ruffpylint, blackyapf with precommit
• install fastapi

This is kind of intermediate check - so that we don't have to deal with more complex collisions in the future.

Need to spend some time to write about this issue in general

• Read through this article: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
• Could be useful to watch these Youtube videos:
  • https://www.youtube.com/watch?v=4KHiSt0oLJ0
  • https://www.youtube.com/watch?v=bW31xiNB8Nc

Including:

• MeResponse
• Url type
• distinguishing between image and markdown data

• only domain logics are our targets of unit testing

• article

  • create: pass all the data to db
  • read single => db 404 response / list => list + count (check call)
  • update: pass only unset values to db
  • delete: call db

• user

  • sign up: hash password functions are invoked
  • update account: db 404 / hash password if password is reset / exclude unset
  • sign out: call db

• auth

  • token check: invalid, expired
  • login: 404 / password not valid / issue access token & update the auth
  • logout: call db

• API metadata & middleware(db_session)
• applying @transactional & testing

Do the web controller testing with some mocks

• references

  • https://fastapi.tiangolo.com/tutorial/testing/
  • https://fastapi.tiangolo.com/advanced/async-tests/
  • https://fastapi.tiangolo.com/advanced/testing-dependencies/

• user

• auth : not necessary

• article

References:

• https://github.com/redis/node-redis

• install redis locally

• install redis client

Implement the backend api for the individual article pages

Use testcontainers

• relying on this tutorial: https://mariogarcia.github.io/blog/2019/10/pytest_fixtures.html

• user

• auth

• article

• create a Dockerfile

This divides into two parts:

• #47

• #48

• set a precommit hook to generate requirements.txt
• setup a basic cicd pipeline
  • checkout
  • basic setup: python setup => install dependency
  • test
• #61
  • db migration
  • backend deploy

Related to #29

Currently the backend service has very limited performance & devops features.

We will move to GCP for better features.

• frontend: Cloud Run
• #73
• postgres: Neon
• api gateway
• connect all the components together

TBA:

• cloud storage
• redis

make some senarios and do e2e testings

Currently, we would like to do manually. But in the future we want to convert them into code.

• #58

• user signs up

  • unhappy case: already existing user / form validation error

• user logs in

  • unhappy: password, username wrong

• user reads an article

  • unhappy: article doesn't exist

• user reads a list of articles

• user creates an article

• user updates an article

  • unhappy: not an owner of this article

• user deletes an article

  • unhappy: not an owner of this article

• user logs out

• user signs out

Check:

• naming & responsibilities

• duplicates in code

• article

  • in port
  • out port
  • service
  • web adapter
  • persistence adapter

• user

  • in port
  • out port
  • service
  • web adapter
  • persistence adapter

• auth

  • in port
  • out port
  • service
  • web adapter
  • persistence adapter

Implement the backend api for the self introduction(about me) page

This page consists mainly of static data, so not much works to be done: the root directory is public/me

• main picture: /profile
• a short introduction about myself: /shortIntro
• education: /education
• work experience: /workExperience
• comp.sci: /compSci
• hobbies: /hobbies
• (possibly in a near future) related pictures

• #56

• choose one of the followings and deploy:

  • render: https://render.com/pricing
  • fly.io: https://fly.io/
  • railway: https://railway.app/pricing
  • supabase: https://aunicorndev.hashnode.dev/supa-fast-api-supabase-fastapi-building-the-api

• Questions: does this service...

  • support serverless functions in docker image format?
  • support Postgres?
  • support securities? - api gateway, secret manager
  • support loadbalancer?
  • run at reasonable cost?

Edit(May 18th, 2023)
We're going to choose the following path:

• #63

• #64

• #65

• #66