- Contributing
- Features
- Local Development (Docker)
- Testing
- Architecture
- MongoDB Scripts
- Support
- Database Alternatives
- Acknowledgements
We welcome all contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas to code open sourced by the Government Technology Agency of Singapore. Contributors will also be asked to sign a Contributor License Agreement (CLA) to ensure that everybody is free to use their contributions.
Before contributing, please read CONTRIBUTING.md. In particular, we strongly encourage contributors to please first discuss the change you wish to make via GitHub issue, email, or any other method with the repository owners beforehand. Otherwise, we may not be able to review or accept your PR.
FormSG is a form builder application built, open sourced and maintained by the Open Government Products team of the Singapore Government Technology Agency to digitise paper processes.
Notable features include:
- 19 different form field types, including attachments, tables, email and mobile
- Verified email and mobile phone fields via integrations with Twilio and AWS SES
- Automatic emailing of submissions for forms built with Email Mode
- Encryption for data collected on forms built with Storage Mode
- (Singapore government agencies only) Citizen authentication with SingPass
- (Singapore government agencies only) Citizen authentication with sgID
- (Singapore government agencies only) Corporate authentication with CorpPass
- (Singapore government agencies only) Automatic prefill of verified data with MyInfo
- Webhooks functionality via the official FormSG JavaScript SDK and contributor-supported FormSG Ruby SDK
- Variable amount and Itemised payments on forms with stripe integration
Install docker and docker-compose and the node version manager.
First, make sure to install and use the node version used by the project:
nvm install
nvm useTo install the relevant npm packages (frontend, backend and virus-scanner), run the following in the root direcory:
npm install && npm --prefix serverless/virus-scanner installTo prevent breaking changes to webpack4 introduced in node 17 and above, enable the --openssl-legacy-provider flag:
export NODE_OPTIONS=--openssl-legacy-providerIf you are on Mac OS X, you may want to allow Docker to use more RAM (minimum of 4GB) by clicking on the Docker icon on the toolbar, clicking on the "Preferences" menu item, then clicking on the "Resources" link on the left.
First, build the frontend for local development:
npm run build:frontendRun the following shell commands to build the Docker image. The first time will usually take 10 or so minutes. These commands runs the backend services specified under docker-compose.yml and the React frontend on the native host.
npm run devAfter the Docker image has finished building, the following local applications can be accessed:
- React application can be accessed at localhost:3000
- The backend API server can be accessed at localhost:5001
- The development mail server can be accessed at localhost:1080
Run npm install as per usual.
For backend, run
docker-compose up --build --renew-anon-volumes
which will rebuild the backend Docker image and not reuse the existing node_modules volume.
As frontend project is currently not using Docker, no other steps are required.
We use MailDev to access emails in the development environment. The MailDev UI can be accessed at localhost:1080 when the Docker container runs.
- Click on the
Login with Singpassbutton on the login page - In the dropdown menu, select
S9812379B [MyInfo] - Choose the profile with the email
[email protected] - You should now be successfully logged in
Note: Remember to renew your formsg_mongodb_data volume
Docker-compose looks at various places for environment variables to inject into the containers. The following is the order of priority:
- Compose file
- Shell environment variables
- Environment file
- Dockerfile
FormSG requires some environment variables to function. More information about the required environment variables are in DEPLOYMENT_SETUP.md.
We provide a .template-env file with the secrets blanked out. You can copy and
paste the variables described into a self-created .env file, replacing the
required values with your own.
You can consult TROUBLESHOOTING.md for common issues that developers face and how to resolve them.
The docker environment has not been configured to run tests. Thus, you will need to follow the following local build guide to get tests running locally.
The team uses macOS for development.
Make you sure have the following node version & package manager on your machine:
"node": ">=18.12.1""npm": ">=8.19.2""mongo": ">=4.0.0"
Run
nvm install 18
npm install
pip install "localstack[full]"to install node modules and Localstack locally to be able to run tests. Note that
localstack[full] is only compatible with Python 3.7 and above.
npm run testwill build the backend and run our backend unit tests. The tests are located at __tests__/unit/backend.
If the backend is already built, you can run
npm run test-ciFrontend tests are located at frontend/__tests__. They can be run with
npm run test:frontendnpm run test:e2e-v2will build both the frontend and backend then run our end-to-end tests. The tests are located at __tests__/e2e. You will need to stop the Docker dev container to be able to run the end-to-end tests.
If you do not need to rebuild the frontend and backend, you can run
npx playwright testThis project is tested with BrowserStack.
The architecture overview is here.
Scripts for common tasks in MongoDB can be found here.
Please contact FormSG ([email protected]) for any details.
FerretDB is an open source MongoDB alternative built on PostgreSQL. MongoDB can be swapped out of FormSG for FerretDB. In order for this to be done, certain changes to the code should be made as described below:
-
Add postgres to the list of services in the
docker.composefile e.g.image: postgres:15.3-alpine3.18 environment: - POSTGRES_USER=<pguser> - POSTGRES_PASSWORD=<pgpassword> - POSTGRES_DB=<pgdbname> volumes: - pgdata:/var/lib/postgresql/data ports: - '5432:5432' -
In the same file, change the "database" image from MongoDB to FerretDB and update the database section to include the lines below:
image: ghcr.io/ferretdb/ferretdb:1.17.0 environment: - FERRETDB_TELEMETRY=disable - FERRETDB_POSTGRESQL_URL=postgres://pg:5432/formsg?user=<pguser>&password=<pgpassword> ports: - '8080:8080' depends_on: - pg -
Lastly, add the pgdata volume
volumes: mongodb_data: driver: local pgdata: -
FerretDB currently has some limitations and certain database features are not supported, these include TTL, database transactions and some aggregration pipelines which are all features used by FormSG.
The following changes can be made to mitigate the limitations of FerretDB:
- Add the autoRemove: 'interval' property to the initializing of the session object in the
session.tsfile. - Remove the unsupported aggregration pipeline stages e.g. lookup and project, in the
submission.server.model.tsfile. - Replace the findOneAndUpdate code block in the
user.server.model.tsfile with code similar to the one below:const user = await this.exists({ email: upsertParams.email }) if (!user) { await this.create(upsertParams) } return this.findOne({ email: upsertParams.email, }).populate({...
- Add the autoRemove: 'interval' property to the initializing of the session object in the
FormSG uses Mongoose as the Object-Document Mapping (ODM) to MongoDB. This means that our code is strongly coupled with MongoDB as Mongoose solely supports it.
In order to use a different database with FormSG you will have to first migrate from Mongoose to other object modelling libraries. One of which is Prisma.
Prisma is an Object-Relational Mapping (ORM) library that can also be used as the object model for MongoDB. Prisma is compatible with various other relational databases like Cockroach DB.
Follow this guide by Prisma to migrate from Mongoose.
The guide has 4 primary steps:
- Install Prisma CLI
- Introspect the current MongoDB for the data model
- For this section, Prisma’s introspection should be able to create prisma models that will replace your
server.model.tsfor each collection - Additionally, as Prisma is relational, you could add relations between the various documents. One good relation to add will be
formmany to oneuseron the[form.email](http://form.email)field.
- For this section, Prisma’s introspection should be able to create prisma models that will replace your
- Install Prisma Client
- Replace Mongoose Queries with Prisma Client
- This step will likely take the most refactoring efforts
- This will include most files in
formsg/srcending withservice.ts - Including test files ending with
service.spec.ts
Thereafter, you could set up CockroachDB which is a distributed SQL DB. Follow the quick start guide by CockroachDB to create a CockroachDB Serverless cluster.
To replace the local development instance, you can follow this guide. As FormSG uses Docker for local development, you will have to replace the mongoDB container from docker-compose.yml to the cockroachDB version.
Then connect to CockroachDB by changing the DB url in .env to the one from your CockroachDB DATABASE_URL="YOUR_COCKROACH_DB_URL".
For local development, if the DB is replaced as above, you should not need to modify the ports as it will still be hosted on localhost:27017.
MongoDB can be replaced with other various relational databases supported by Prisma in this list.
It is also possible to migrate from Mongoose to Ottoman, which is another ODM.
The process will be simpler than migrating to Prisma, but Ottoman is more restrictive and can only be used together with Couchbase, which is also a noSQL DB like MongoDB.
Refer to this guide to migrate from Mongoose to Ottoman and then replace MongoDB with Couchbase.
FormSG acknowledges the work done by Arielle Baldwynn to build and maintain TellForm, on which FormSG is based.
Contributions have also been made by:
@RyanAngJY
@jeantanzy
@pregnantboy
@namnguyen08
@zioul123
@JoelWee
@limli
@tankevan
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent