colonelparrot / jscanify Goto Github PK

Open-source Javascript mobile document scanner.

Home Page: https://colonelparrot.github.io/jscanify/

License: MIT License

JavaScript 100.00%

js scanner javascript document-scanner nodejs

jscanify's Introduction

Open-source pure Javascript implemented mobile document scanner. Powered with opencv.js
Supports the web, NodeJS, React, and others.

Available on npm or via cdn

Features:

paper detection & highlighting
paper scanning with distortion correction

Image Highlighting	Scanned Result

Quickstart

Developers Note: you can now use the jscanify debugging tool to observe the result (highlighting, extraction) on test images.

Import

npm:

$ npm i jscanify
import jscanify from 'jscanify'

cdn:

<script src="https://docs.opencv.org/4.7.0/opencv.js" async></script>
<!-- warning: loading OpenCV can take some time. Load asynchronously -->
<script src="https://cdn.jsdelivr.net/gh/ColonelParrot/jscanify@master/src/jscanify.min.js"></script>

Note: jscanify on NodeJS is slightly different. See wiki: use on NodeJS.

Highlight Paper in Image

<img src="/path/to/your/image.png" id="image" />

const scanner = new jscanify();
image.onload = function () {
  const highlightedCanvas = scanner.highlightPaper(image);
  document.body.appendChild(highlightedCanvas);
};

Extract Paper

const scanner = new jscanify();
const paperWidth = 500;
const paperHeight = 1000;
image.onload = function () {
  const resultCanvas = scanner.extractPaper(image, paperWidth, paperHeight);
  document.body.appendChild(resultCanvas);
};

Highlighting Paper in User Camera

The following code continuously reads from the user's camera and highlights the paper:

<video id="video"></video> <canvas id="canvas"></canvas>
<!-- original video -->
<canvas id="result"></canvas>
<!-- highlighted video -->

const scanner = new jscanify();
const canvasCtx = canvas.getContext("2d");
const resultCtx = result.getContext("2d");
navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => {
  video.srcObject = stream;
  video.onloadedmetadata = () => {
    video.play();

    setInterval(() => {
      canvasCtx.drawImage(video, 0, 0);
      const resultCanvas = scanner.highlightPaper(canvas);
      resultCtx.drawImage(resultCanvas, 0, 0);
    }, 10);
  };
});

To export the paper to a PDF, see here

Notes

for optimal paper detection, the paper should be placed on a flat surface with a solid background color
we recommend wrapping your code using jscanify in a window load event listener to ensure OpenCV is loaded

jscanify's People

Contributors

Stargazers

Watchers

Forkers

lguzzon-scratchbook qqq-tech luminous8 soumyadeepbi kelenam ideaowl helingai nitishymtpl jade2290 callmemagnus smali-kazmi abhijit47 amirsohail1998 richardsonjf jrcribb berkode magnum gracemorganmaxwell nanderoo jenreuting paulosabayomi oxmoss aksharahegde spl1nes amalkarunakaran raffareis gatoclasssub matijaparavac clayandcarbon createmycookbook dirksen sidalinadji bchr02 immi5556 haegt tortious 5l1v3r1 kaush26 lcsouzamenezes chesterchen915 kyawthura-gg exonos yoavilan kknechtel thenewguy hideosnes cloudnepal vivekrai1999

jscanify's Issues

NodeJS support

On the website it says:

It can run in the browser or on a server with NodeJS.

When importing the library in Node however, I get the following error:

Uncaught ReferenceError: document is not defined

Documentation implies findPaperContour takes HTML image input

The note in the wiki implies that I can pass HTMLImageElement, HTMLCanvasElement and Files for img or image parameters.
However findPaperContour(img) requires a cv.Mat.

Expectation:

const contour = scanner.findPaperContour(canvas); // Results in cv bind error
const corner_points = scanner.getCornerPoints(contour);

Reality:

const cvmat = cv.imread(canvas);
const contour = scanner.findPaperContour(cvmat);
const corner_points = scanner.getCornerPoints(contour);

Hi @ColonelParrot, thank you for this awesome library. I'm currently using it in a personal project, but I've noticed that the image quality is poor. I've attached the scanned document images below for reference.

I made a small modification to the extractPaper() method, but even before that, the quality was not satisfactory. Any suggestions or tips would be highly appreciated.

// Calculate the aspect ratio of the original document
const aspectRatio = img.cols / img.rows;

// Calculate the corresponding height to maintain the aspect ratio
const calculatedHeight = Math.round(resultWidth / aspectRatio);

I applied filter here

Error importing jscanify library in React application

I'm experiencing an issue when trying to import and use the jscanify library in a React application. While the CDN approach works fine, importing the library as a module using the import statement causes an error.

import React, { useEffect, useRef, useState } from 'react';
import jscanify from 'jscanify';

export default function PaperScanner() {
const canvasRef = useRef(null);
const [photoData, setPhotoData] = useState(null);

useEffect(() => {
const scanner = new jscanify(); // <-- Error occurs here

}, []);

}

Error Message:
TypeError: Cannot set property document of # which has only a getter

Passing a number "-1" from JS side to C/C++ side to an argument of type "unsigned long"

I just noticed this error while testing this library. The full error reads:

Passing a number "-1" from JS side to C/C++ side to an argument of type "unsigned long", which is outside the valid range [0, 4294967295]!

This is on the Brave browser

I do actually see it working though, so it doesn't seem very important.

My code:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Scanner</title>

	<script src="https://docs.opencv.org/4.7.0/opencv.js" async></script>
	<script src="https://cdn.jsdelivr.net/gh/ColonelParrot/jscanify@master/src/jscanify.min.js"></script>
  </head>
  <body>
	  
	<video id="video"></video> <canvas id="canvas"></canvas>
	<!-- original video -->
	<canvas id="result"></canvas>
	<!-- highlighted video -->
	  
	  
	  
	  
	<script type="module">
		const scanner = new jscanify();
		const canvasCtx = canvas.getContext("2d");
		const resultCtx = result.getContext("2d");
		navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => {
		  video.srcObject = stream;
		  video.onloadedmetadata = () => {
		    video.play();

		    setInterval(() => {
		      canvasCtx.drawImage(video, 0, 0);
		      const resultCanvas = scanner.highlightPaper(canvas);
		      resultCtx.drawImage(resultCanvas, 0, 0);
		    }, 10);
		  };
		});
	</script>
	
  </body>
</html>

Bug: constructor does not work in browser environment

When attempting to create a scanner object in browser environment following error is thrown:
Uncaught (in promise) TypeError: Cannot set property document of #<Window> which has only a getter at installDOM (jscanify-node.js:9:10) at new jscanify (jscanify-node.js:30:5) at scanner.svelte:11:9 at run (index.mjs:20:12) at Array.map (<anonymous>) at index.mjs:2101:58 at flush (index.mjs:1329:17) at init (index.mjs:2197:9) at new Root (root.svelte:20:25) at createProxiedComponent (svelte-hooks.js?v=6abbeb80:341:9)

Chrome Dev version: 115.0.5762.4 (Official Build) dev (arm64)
Used stack: Vite + React

Full example:
https://codesandbox.io/p/sandbox/dreamy-breeze-wjwo7h

Issue: Round Corners

It does not work well with pages with round corners. Otherwise works well.

React js import support?

Expected Usage

import jscanify from 'jscanify'

Highlighted image does not show

I am using the the starter js code but the highlighted image does not load.

Any way to use this library in Angular?

Does not work inside vue.js or nuxt project

Every time I try, I get this message when loading and camera does not turn on:

Uncaught TypeError: Class extends value undefined is not a constructor or null
    at node_modules/jsdom/lib/jsdom/virtual-console.js

node version:
v18.16.0

code for Scanner.vue component:

<script setup lang="ts">
import {onMounted, ref} from "vue";
import useScript from '../composables/useScript';
//const jscanify = require('jscanify');
//@ts-ignore
import jscanify from 'jscanify'
const openCVURL = 'https://docs.opencv.org/4.7.0/opencv.js';
let scanner:any = new jscanify();

const canvas: HTMLCanvasElement | null = <HTMLCanvasElement> document.getElementById("canvas");
const result: HTMLCanvasElement | null = <HTMLCanvasElement> document.getElementById("result");
const video: HTMLVideoElement | null = <HTMLVideoElement> document.getElementById("video");


onMounted(async () => {
  await useScript(openCVURL);
  const canvasCtx = canvas.getContext("2d");
  const resultCtx = result.getContext("2d");
  scanner = new jscanify();
  navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => {
    video.srcObject = stream;
    video.onloadedmetadata = () => {
      video.play();

      setInterval(() => {
        canvasCtx!.drawImage(video, 0, 0);
        const resultCanvas = scanner.highlightPaper(canvas);
        resultCtx!.drawImage(resultCanvas, 0, 0);
      }, 10);
    };
  });
})

</script>

<template>
  <video id="video"></video>
  <canvas id="canvas"></canvas>
  <!-- original video -->
  <canvas id="result"></canvas>
  <!-- highlighted video -->
<h2>Document scanner component TODO</h2>
</template>

IOS Compatibility

Camer freeze in IOS browsers
any solution please?

Do you have the document detection in real time while using the camera to get the photo through MediaCapture API ?

Do you have the document detection in real time while using the camera to get the photo through MediaCapture API ?
thank you !

Fails to install with Python 3.12

Distutils were removed in python 3.12 meaning the canvas dependancy fails to install.

Fix is to roll back python to 3.11, install then works without.

Just an FYI

extractPaper not working with cluttered background

I'm very impressed by jscanify and I really appreciate that you open sourced your work.

I have two examples where a cluttered background breaks the extractPaper logic to find the paper in the image. I think this is an edge case (no pun intended) but perhaps these can help you future improve the algorithm.

Size Ratio for exported canvas.

how to set auto height or width to maintain the ratio of the image size instead of fixed width and height?
scanner.extractPaper(image, paperWidth, paperHeight)

Doesn't work in web (React)

Hey!

I was trying to use the package on the FE side (React) but faced several issues. If I install it via npm (in my case pnpm, but I guess it doesn't matter) it fails with the error. It requires Node.js variable process.
If I use it via CDN, its dependency - opencv.js requires Node.js variable process.

Error: Uncaught ReferenceError: process is not defined

Could you please clarify if it's possible to use this lib in React? If you do - please clarify the steps to use it.
The React example didn't help...

WebRTC is very complicated for photos

As you know, capturing a photo is basically taking a still frame from the video stream.

When doing video, usually any project is compatible with the idea of wanting a 360, 720 or 1080 video... but when you're working with photos, you usually want the best possible photo resolution.

There is a "WebRTC resolution scanner" but it works pretty poorly. For example, on my iPhone 14, it gives me "fail" as a result in all resolutions, making it difficult for me to set up a stream with "the best resolution the mobile can offer".

And this may not be a problem when you use WebRTC to take a profile picture or a picture where good resolution is not required, but in the case of a document scanner, it is critical to have the best possible resolution to be able to OCR it.

How do you deal with this problem and how do you configure WebRTC to take the best possible mobile resolution?

I have tried with the "ideal" constraints but without good results.

Thanks in advance

[ Demo Vison ] Update to work with User Input image

possible that user can input img and test in the demo version ?

cv.approxPolyDP maxContour.

During testing, I encountered a case where the top-right of my page was incorrectly evaluated as being near the top-left, resulting in a near triangular poly from getCornerPoints(contour). What seems to have happened is that there was a point in the contour that was just right of the centre, yet had a bigger distance than the right-hand corner of the page (although the page was at an almost 45 degree angle in the frame, which would be unusual).
Recommend applying a cv.approxPolyDP to the maxContour prior to getCornerPoints(contour) to reduce occurrences.

      return this.getApproxPoly(maxContour);
    }

    /** refines a contour to get approximate poly
     * @param {*} contour contour to refine
     */
    getApproxPoly(contour) {
      const peri = cv.arcLength(contour, true);
      const approx = new cv.Mat();
      cv.arcLength(contour, true) * 0.02
      cv.approxPolyDP(contour, approx, 0.1 * peri, true);
      return approx;
    }

Calling getCornerPoints throws an Uncaught (in promise) 23302432

I am trying to get the cornerPoints from a scanned image to see how wide the pdf atcually is.
However, when scanner.getCornerPoints(paperContour, image) the console will throw an ambigous error in the opencv.js file as follows uncaught (in promise) 23302432.

Below is a minimal example which shows this error (image with pdf not included)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Test</title>
    <script src="https://docs.opencv.org/4.7.0/opencv.js" async></script>
    <!-- warning: loading OpenCV can take some time. Load asynchronously -->
    <script src="https://cdn.jsdelivr.net/gh/ColonelParrot/jscanify@master/src/jscanify.min.js"></script>
</head>
<body>
    <img src="pic.jpeg" id="image" />
    <script>
        window.addEventListener("load", () => {
            cv['onRuntimeInitialized'] = () => {
                const scanner = new jscanify();
                const image = document.getElementById("image");
                const paperContour = cv.imread(image);
                const resultCanvas = scanner.findPaperContour(paperContour);
                const {
                    topLeftCorner,
                    topRightCorner,
                    bottomLeftCorner,
                    bottomRightCorner,
                } = scanner.getCornerPoints(paperContour, image);
                console.log("topLeftCorner",topLeftCorner);
                console.log("topRightCorner",topRightCorner);
                console.log("bottomLeftCorner",bottomLeftCorner);
                console.log("bottomRightCorner",bottomRightCorner);
            }
        })
    </script>
</body>
</html>

How do I get the width and height for result?

I think this is the biggest problem right now. If you use a fixed value then it will only work for that particular aspect ratio. Any plans for implementing something similar to the methods outlined in the Whiteboard Scanning and Image Enhancement by Microsoft?

Appreciate it.

There is no support for Typescript :(

Improvements to the recognition algorithm

Hi,

thank you for this amazing project. I used it as a base to further improve the recognition algorithm as follows:

Custom compiled opencv.js that only contains the functions we need => reduced file size from ~9 MB to about 1.8 MB.
For paper extraction: Use OpenCV's grabCut algorithm to remove the background for higher precision when using the extractPaper function (=> not used in realtime highlighting as the algorithm needs a few seconds to complete).
Use morphologyEx to remove noise and text inside the documents.
Use Canny edge detection to detect the boundaries of objects.
Add cv.dilate to account for small inaccuracies in the Canny edge detection.
Add a white border around the image to be able to detect documents not completely contained inside the canvas.
Removed thresholding since its replaced by Canny edge and morphologyEx

I would just like to share it, maybe it is useful for other people as well. Maybe you can test it as well and consider adding some changes to your project.

jscanify_modified.zip