foyzulkarim / mern-video-streaming Goto Github PK

MERN Video Streaming is a cutting-edge, open-source platform for video streaming, offering a comprehensive, full-stack solution utilizing the latest MERN stack technologies.

License: MIT License

JavaScript 99.17% HTML 0.61% Dockerfile 0.22%

bullmq ffmpeg hls mern messaging react redis stream streaming video

mern-video-streaming's Introduction

MERN Video Streaming Project

I am writing a full in-depth explanation in my medium.com page. Please visit Build a MERN based video hosting and sharing system to understand the whole architecture and step by step instructions.

Overview

MERN Video Streaming is a cutting-edge, open-source platform for video streaming, offering a comprehensive, full-stack solution utilizing the latest MERN stack technologies. With MERN Video Streaming, users can easily upload, manage, and stream videos on demand, providing a seamless experience. This project features a video processing capability that provides real-time notifications upon completion.

The backend is comprised of three services, utilizing Redis messaging for communication. These include an API server, a video conversion service, and an HTTP server serving HLS video files. On the client side, the project uses create-react-app and MUI library, along with socket.io-client and React Context.

Architecture

Backend Services

API Server

The API server is an Expressjs app that uses Joi, Multer, BullMQ, Socket.io, and MongoDB driver. The server communicates with Redis to process and store data, and uses MongoDB as the primary database. The database and Redis are spun up via the docker-compose command.

Video Conversion Service

The video conversion service is a node process that is responsible for converting videos and communicating with Redis via the BullMQ library. The service is not exposed to any HTTP port.

HTTP Server

The HTTP server is a plain and simple server that serves HLS video files.

Frontend app

The client app is based on create-react-app and MUI library. It uses socket.io-client and React Context to communicate with the API server and display the video content.

Prerequisites

To get started with MERN Video Streaming, you will need to have the following software installed on your local machine:

Built with Open-Source Technologies

MERN Video Streaming has been built with the following open-source technologies:

MongoDB - A document-based database used to store user and video data
Express - A Node.js web application framework used for the server-side of the application
React - A JavaScript library for building user interfaces used for the client-side of the application
Node.js - A JavaScript runtime environment used to run the server-side code

Installation

To run the project locally, follow these steps:

1. Clone the repository

git clone [email protected]:foyzulkarim/mern-video-streaming.git

2. Run project locally using docker-compose

2.1 Run the `server` app

From project directory run bellow commands

cd server
npm run start

2.2 Run the `client` app

From project directory run bellow commands

cd client
yarn install
yarn start

3. Usage

You can now access the application at http://localhost:3000.

Special note: Video folder structures

To upload the videos and serve them from server's folder we need to have below folder structure.

Contributing to MERN Video Streaming

MERN Video Streaming is an open-source project and we welcome contributions from the community. If you would like to contribute to the project, please refer to the CONTRIBUTING.md file for guidelines.

License

MERN Video Streaming is licensed under the MIT License. Please see the LICENSE file for more information.

Note

Before running the above commands, make sure that you have Docker and Node.js installed on your machine.

This readme file is not finished yet. I will update it as I go along with the project. Thanks.

mern-video-streaming's People

Contributors

Stargazers

Watchers

Forkers

rahathosen mahfujtex mh-shuvo imranhasan871 mubasshir00 shohedul350 tamimhasan19702 alamriku ezaz-ahmed mahedihsharif rabbithewebdeveloper binarykitchen patel4452 basher7 fahimforhad496 kimyhgit kamareee aninda052 razuahmedjoy hasiburdev hattima-tim suvrajitkarmaker rakibul-islam-raju shakib448 jubair70 mzshovon droidjahangir divyej ysnarafat mtauhidul qridwan monibbormon reshadsadik muhammadjahidhasan sunilgitb talatmorshedtanbir sajidahmed66 safayetjamil647 kzamanbd abid1174 abirabuasim iftakharalamrizve git-sujon aliakkas006 shahriaras gauravrawat369 cjohnguipo monir-007 kamrul-hasan-1971 aryan3212 nikhiloo7 sureshm-98 amanullhazoha gajraj-m milonahamad asadsanto10 nattakan-phim meetprgiri thenazim07 roshangautam07

mern-video-streaming's Issues

Dockerize the whole `mern-video-streaming` application

Dockerize the mern-video-streaming application so that we can run the whole application using only a single command
npm start

Remove `console.log` and use `logger.info` everywhere in the server side project

We have implemented logger in the server and now we need to replace all of the console.log statements with logger.info

Package Vulnerabilities related to pm2

The package @pm2/agent depends on vulnerable versions of proxy-agent and semver.
The package @pm2/io depends on a vulnerable version of semver.
The package pm2 depends on vulnerable versions of @pm2/agent and @pm2/io.
The package degenerator depends on vulnerable versions of vm2.
The package pac-resolver depends on vulnerable versions of degenerator.
The package pac-proxy-agent depends on vulnerable versions of pac-resolver.
The package proxy-agent depends on vulnerable versions of pac-proxy-agent.
In total, there are 9 vulnerabilities, with 3 of them being moderate and 6 being critical.

upload progress and error handling

Refactor and improvement scopes

In this issue, we will list down all of the issues we are having in the current repository which we should refactor or remove to improvement our current project structure and code quality.
Please keep adding your suggestions in the comments section.

কোডিং এ ভালো করার অন্যতম একটা উপায় হল কোডের ভালোমন্দ বোঝা। যেহেতু কোডিং এর পুরোটাই হল চিন্তার ফসল, ফলে অন্যের কোড দেখলে নিজের জানা অনেক কিছুর সাথেই অমিল খুঁজে পাওয়া যায়। এর দ্বারা অন্যের চিন্তার প্যাটার্ন বুঝে ফেলে নিজের কোডিং এর প্যাটার্ন আরো বেটার করা যায়। অন্যদিকে, কোডিং এর যে বিষয়গুলো বর্জনীয় সেগুলো সম্পর্কেও আরো ভালো ধারণা পাওয়া যায়।

আমার চলমান মার্ন ভিডিও স্ট্রিমিং প্রজেক্টে (https://lnkd.in/g8JXBqwT) আমি ইচ্ছা করেই এতদিন পর্যন্ত প্রচুর পরিমান গার্বেজ কোড লিখেছি। প্ল্যান ছিল আপনাদের সাথে এগুলো আলোচনা করে আর্কিটেকচারটিকে আরো বেটার করবো। সেজন্যই আমি বেটার / রিফ্যাক্টর করার মতো সমস্ত বিষয়গুলো লিস্ট করতেছি যেগুলো আমাদেরকে ফিক্স করতে হবে, এরপর এগুলো নিয়ে ভিডিও বানাবো।

আপনাদের জন্য একটা টেস্ট আছে। সেটা হল, এই রিপোজিটরিতে গিয়ে কোড গুলো ঘাটাঘাটি করে দেখেন কি কি বিষয়ে আপনি ইম্প্রুভ করার মতো খুঁজে পান। সেগুলো এই ইস্যু এর কমেন্টে গিয়ে যোগ করুন। এটা একটা ফান একটিভিটি হিসেবে নিতে পারেন।

sort by view count, then newer first

show published date using date library

show durations on the video thumbnails

Implement Video Edit page

When we click on the Edit link/button, we will navigated to a new page and load all of the video information in the page.
No need to UPDATE the property inthis ticket.

Let's not over-use our upload page and keep it as it is.
Create a new VideoEditPage to edit the text values (NOT the uploaded video itself)

Also, take the latest code from main branch. Update the List page and add a Edit button in the actions (keep the Delete action but remove the other actions).

(Server) Implement seed data mechanism for Roles collection

Currently we have seed data mechanism for Video collection. Follow this process and implement seeding for Role.

`Video List` page not working

"npm run start" shows repeatedly incremental terminal outputs

[1] Error: connect ECONNREFUSED 127.0.0.1:6379
[1] at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) {
[1] errno: -111,
[1] code: 'ECONNREFUSED',
[1] syscall: 'connect',
[1] address: '127.0.0.1',
[1] port: 6379
[1] }

Screenshot:
https://pasteboard.co/4J698lmrrO16.png

Logging with EAK (Elasticsearch, AMP, Kibana)

create folders script

In server folder, we need to create some subfolders where the videos will be uploaded and processed.

Video thumbnail is not loading in video list page

In the videos list page video thumbnail loads from http://localhost whether the SERVER_URL=http://localhost:4000
when a new video is uploaded it saves in monboDB the thumbnail URL starting with http://localhost/thumbnail which is not correct.

Implement logging

In our current application, we haven't implement any type of logging solutions.
As for proposal, we can use winston or pino as the logger library.
We will be logging to Loggly

Create a docker-compose file to run a MongoDB and Redis container

Broken Video Thumbnail

After uploading a video we used to store a thumbnail in db. Now we are getting broken thumbnail on /videos page.

view count increment and manage

(Server) Implement generic `Update` mechanism

Currently we already have insert and some other generic functions. Follow those to create this generic update function.

video manage - search, edit, update, delete

create a video list page, edit page and delete confirmation
list page will have the sorting, searching feature

script doesn't create uploads folders on macOS (apple silicon)

As described in the Readme.

npm i -g zx
npm run init-directories

should create an uploads folder. But when the script was run using zsh on macOS Ventura 13.4.1 (c), nothing happened. Upload folders needed to be created manually. This should be mentioned in the Readme

run using pm2 or other prod methods

Task : If a video is clicked, increase the view count in the database and show the view in the UI

update video status to `published` when processing is finished

when we upload the video, we will set status pending
when we finish the full processing, then we set the status to published

show only status=='published'

`api` container not running

Fix the client build

When we run yarn build we are getting some errors which is failing the github action.

ffprobe is creating problem

error

npm ERR! code EINVALIDPACKAGENAME
npm ERR! Invalid package name "node_modules/@ffprobe-installer/darwin-arm64": name can only contain URL-friendly characters

Error log

0 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   '/Users/foyzul/.nvm/versions/node/v14.21.3/bin/node',
1 verbose cli   '/Users/foyzul/.nvm/versions/node/v14.21.3/bin/npm',
1 verbose cli   'install'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session da2df17eba8b9348
5 silly install runPreinstallTopLevelLifecycles
6 silly preinstall [email protected]
7 info lifecycle [email protected]~preinstall: [email protected]
8 silly install loadCurrentTree
9 silly install readLocalPackageData
10 timing stage:loadCurrentTree Completed in 4ms
11 silly install loadIdealTree
12 silly install cloneCurrentTreeToIdealTree
13 timing stage:loadIdealTree:cloneCurrentTree Completed in 0ms
14 silly install loadShrinkwrap
15 timing stage:rollbackFailedOptional Completed in 0ms
16 timing stage:runTopLevelLifecycles Completed in 25ms
17 silly saveTree [email protected]
17 silly saveTree ├── @faker-js/[email protected]
17 silly saveTree └─┬ @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   ├── @ffmpeg-installer/[email protected]
17 silly saveTree   └── @ffmpeg-installer/[email protected]
18 verbose stack Error: Invalid package name "node_modules/@ffprobe-installer/darwin-arm64": name can only contain URL-friendly characters
18 verbose stack     at invalidPackageName (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/npm-package-arg/npa.js:86:15)
18 verbose stack     at Result.setName (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/npm-package-arg/npa.js:121:11)
18 verbose stack     at new Result (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/npm-package-arg/npa.js:111:23)
18 verbose stack     at Function.resolve (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/npm-package-arg/npa.js:58:15)
18 verbose stack     at module.exports (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/lib/install/realize-shrinkwrap-specifier.js:21:14)
18 verbose stack     at /Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/lib/install/inflate-shrinkwrap.js:54:23
18 verbose stack     at tryCatcher (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
18 verbose stack     at Object.gotValue (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/reduce.js:166:18)
18 verbose stack     at Object.gotAccum (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/reduce.js:155:25)
18 verbose stack     at Object.tryCatcher (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
18 verbose stack     at Promise._settlePromiseFromHandler (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:547:31)
18 verbose stack     at Promise._settlePromise (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:604:18)
18 verbose stack     at Promise._settlePromise0 (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:649:10)
18 verbose stack     at Promise._settlePromises (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:729:18)
18 verbose stack     at _drainQueueStep (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:93:12)
18 verbose stack     at _drainQueue (/Users/foyzul/.nvm/versions/node/v14.21.3/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:86:9)
19 verbose cwd /Users/foyzul/personal/github/mern-video-streaming/server
20 verbose Darwin 22.6.0
21 verbose argv "/Users/foyzul/.nvm/versions/node/v14.21.3/bin/node" "/Users/foyzul/.nvm/versions/node/v14.21.3/bin/npm" "install"
22 verbose node v14.21.3
23 verbose npm  v6.14.18
24 error code EINVALIDPACKAGENAME
25 error Invalid package name "node_modules/@ffprobe-installer/darwin-arm64": name can only contain URL-friendly characters
26 verbose exit [ 1, true ]

app.get(`${BASE_URL}/detail/:id`, async (req, res) => {
    console.log(`GET`, req.params);
    const video = await updateViewCount(req.params.id);
    if (video instanceof Error) {
      return res.status(400).json(JSON.parse(video.message));
    }
    res.send(video);
  });

Instead it should be

app.get(`${BASE_URL}/detail/:id`, async (req, res) => {
    console.log(`GET`, req.params);
    const video = await getById(req.params.id);
    if (video instanceof Error) {
      return res.status(400).json(JSON.parse(video.message));
    }
    res.send(video);
  });