[Enhancement]: Review Storage mechanics PLUS! about nvr-js HOT 75 OPEN

Ok! I found it. Some error catching and logs went a long way. I am sorry I had to dirty up your clean code :D

But the problem, I think, is more basic than I initially thought. Since the FIFO dequeue wasn't checking for an error on the run, it just drops those segments. I added an enqueue and reset the timeout. I tested this by simply mving the .db file and moving it back.

I'll submit a pull request once I do more live testing on our setup. I've pushed my changes here and re-deployed on two test and live systems. Will report back with any findings :)

This is the deployed codebase:
https://github.com/city-of-shreveport/nvr-js/tree/retry-for-segment-writes

from nvr-js.

Absolutely! And I'm excited you're excited :)

Agreed about the PR! Sounds like my changes won't be worth much since this will now be flat-file based.

Patching in and deploying the MKV now - will let you know after a couple hours of footage :)

from nvr-js.

I do think making that configurable would be a great update :) But I also don't mind proudly using open-source and the branding that comes with :)

So! I just patched in MKV files, deployed them to another system, and watched two segments hit the timeline.

1. Live Streaming Works (Chrome/Firefox)

2. I can view recorded segments on the timeline (Chrome/Firefox) but can only view the video from the timeline in Chrome

3. All works in Chrome (linux), but playback of a recorded segment DOES NOT work in Firefox with the following error:
   Uncaught (in promise) DOMException: The media resource indicated by the src attribute or assigned media provider object was not suitable.

A quick google seems to indicate that MKV support is just not in Firefox and from what I can see, not coming :/

Thoughts? This isn't a deal breaker for us here since we use Chrome anyways.

from nvr-js.

Live Streaming Works (Chrome/Firefox)

That make sense, since live streaming uses the MP4 container (well fMP4 - with trickery using websockets)
One of the benefits with NVRJS - is that one FFMPEG instance caters for both the segment and live streaming.

See, I split the one input (camera) to 2 outputs [Live, File] - using one FFMPEG instance per camera,

IMO - The benefit to be had with MKV outweighs the use of Firefox, so at this point I can look past that 😅
Firefox prob doesn't work as it doesnt like MKV - but it works for Live as that is using the MP4 container, where as the segments use MKV.

I could test Fragmented MKV - but will likely be pushing my luck 😄

The live stream is passed via web sockets to a custom media file handler - the browser thinks its a single file (but really its live) it uses the MediaSource Bowser API

from nvr-js.

Mmmm - I'm not sure how PM2 identifies something as errored, it might be picking up some FFMPEG debug output, and freaking out maybe.

I don't use PM2 myself these days (well... I did when I created this, hence the example start up 😅)
I mainly use systemd now.

One thing to note about the upcoming version (3.0.0)
Since it now uses a different storage mechanism, it won't pick up on previous recordings, as the DB has gone.

Unless you manually create the JSON files for each segment 😬
I should be able to push a test version out a bit later.

but I will also push it to the branch I created (2.1.0 - which I'll rename to 3.0.0 once uploaded)

3.0.0 also allows you to add events to the live view.
i.e a live stream is being viewed, you can tag anytime an event to it - which will then appear on the timeline view.

from nvr-js.

@KeithHanson

Overhaul complete.
https://github.com/marcus-j-davies/nvr-js/tree/v3.0.0
https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

Its not published yet to NPM - but you should be able to pull down the branch and use it.

from nvr-js.

Ok,

I do believe this is the FS Watcher not recovering from an IO error in that case.
I'll attach it directly to FFMPEG.

I'll patch, and push the code - this will also include the new Ability to disable login (I added it today 🤫 )

There is one thing I need to fix on it - but will do when I patch the FS Watcher

from nvr-js.

Ok, Have done some research....

Moving to FFMPEG to monitor the start of segments isn't straight forward currently, for a few reasons.

• There is no event to tell us (only on completetion of a segment)
• Having it based on the start up of an FFMPEG process, does not guarantee the start timestamp, is the true timestamp recording started (it can be different over several seconds - Hey I'm a perfectionist 😅 )
• Events occurring during a segment cant be written to disc (no segment meta file will exist, to write to and there is quite a lot of logic to rewrite to work around it)

With that said, the watcher does have an error event.

With this, I can restart the watcher, if it had errored due to an IO error (like a power drop say),
what I will do is, wait 10s before restarting it, if I encounter this event.

I think that will be my best approach avoiding the potential to introduce problems with the logic rewrite.
and should, in theory allow the watcher to recover.

from nvr-js.

Roger that! Thanks for the update!

from nvr-js.

@marcus-j-davies Any updates? :) :) :)

Current metadata bug is affecting us - I'm solving it with a cron-based restart of the service for now.

If you think it will be more than a few more days (no judgement! promise! just planning next steps), I'll go ahead and knock out the backfill task for us and have that on a periodic run & reboot service along with that every hour and I don't think we'll see any further metadata writing issues.

If I can help in any way further than that, please let me know! I don't want to step on your toes, though as I'm certain you'll produce a better outcome than I could :P

Again, many thanks for such a great tool and all the collaboration with us!

from nvr-js.

@KeithHanson

The patch is ready to test.

To Disable Login:
add this to your config file.

module.exports = {
	/* System Settings */
	system: {
		/* Disable Security - Know what your doing before changing this! */
		disableUISecurity: true,
		.....
        }
}

Changes

{timestamp}_placeholder.json
A file used to store a temp metadata payload (created on FFMPEG process start), this also allows to continue creating events, before the main metafile is written

{timestamp}.json
The main metafile, created at the end of each segment (as advised by FFMPEG).
any events created in {timestamp}_placeholder.json are moved into this file, then finally a new
{timestamp}_placeholder.json is created.

metafile creation is now tied to FMPEG activity, and is no longer 'watching' for new files, which I still believe seem to have problems recovering after IO errors.

from nvr-js.

Cool - well not, but you know 😅

I'll add a try catch to any reading of a file (and writing for good measure).
I cant do anything special here, other than backing out of the current operation - in favour of not bringing down the instance in its entirety.

EDIT:

If you're already off to the races on the patch I can hang back ofc :)

Already on it

from nvr-js.

Just did a count of all the systems. Deployed most recent commit to 41 raspberry pi's with 2TB and 4TB drives attached :)

I tested it on the previously failing system, and everything went smoothly.

Thank you! :) I'll report in if I find anything.

from nvr-js.

Nice!

This is with the recent patch to stop reading Corrupted JSON files?

The changes mode here.

5639de6

from nvr-js.

Correct! I deployed it after I saw the update.

I see some gaps here and there, but that's from whatever failures happened.

But the only problem I've had is HDD space filling up at this point :D Just means I need to tune the retention.

I've spot checked about 10 of the 41 systems and those that didn't fill up their drive have a full 2 days of history!

Brilliant! :)

EDIT: they all have little gaps here and there (3-6 minutes), but it's obvious our code is recovering from the issue, and your code is handling the recovery gracefully :)

from nvr-js.

Hi @KeithHanson,

It's way passed my bed time where I am (2AM) but wanted to touch base before I hit sleep.

Thanks for the kind words btw :)

I must admit, this was a DIY project, but would welcome any help in making it more accessible.

On to the issue.

My initial thoughts are the way it stores the clip metadata.

It uses SQLITE which I'm not overly a fan of - I wonder if a different storage mechanism can be used?

To stop multiple writes happening, all writes are queued (FIFO), and an interval is happening to check for anything in the queue with the Commit() function (every 10s).

it's based on setTimeout, and if a write happens to error out, it may not reach calling setTimeout again for the next 10s, meaning there is lots in queue, but the Commit function is not being executed, as it crashed before setting the next 10s - the crash may have been due to some SQLITE lock issue, or file IO error, hence why i am not overly happy with my choice in using SQLite.

Another potential is the IO buffer with the pipes between the FFMPEG processs and it's self, if that gets full, it's self will stop receiving FFMPEG output, but I'm sure I ignore stdout...

from nvr-js.

Thank you for responding so quickly! DIY is my approach to handling a fleet of pole cameras for our municipality. We have a decentralized, edge-based, DIY approach. So far, our Real Time Crime Center analysts like the interface :) And it fits my approach (if it can run decentralized straight from the pole, that's the best outcome).

Your response above all makes sense and what I was suspecting - likely some weird failure on the HD that gets taken care of by our other services on the Pi (decrypts and re-mounts the volume).

I will make a branch littered with debug statements to try to catch this (though, as you know, we will need to wait some time for whatever failure to occur).

Have you considered ditching the SQLite DB entirely? It's almost perfect as a use case to just look at the file system and draw the timeline since I only care about showing the user what's on disk. But you may have explored this already, which might lead me down the wrong track if you've already tried it.

If not, I may take a crack at a lightweight solution. It would be great if there were one less moving part to worry about (and no SQLite DB means not having to import gaps in the data, easy recovery, and other icing-on-the-cake so to speak :)

from nvr-js.

Also, one last thought for now :P Converting from SQLite to something like PostgreSQL or MySQL might be pretty straightforward. I'm trying to keep as few moving parts on my setups as possible, though (since there's already so many), and keep processor utilization down as much as possible, so I liked the SQLite DB at first sight.

I think I will spend some time on the error not re-queueing the timeout. I'm not the best nodejs dev but I'm sure we're not the only ones who have had this issue with critical functions not being called / queued back up.

from nvr-js.

To draw clips on the timeline, one could look to use ffinfo, along with the timestamp of the file being created/last modified, to accurately place them in time, the missing piece will be to associate any event data, and associate them and store the event data somewhere (other than SQLite :))

from nvr-js.

I think I will spend some time on the error

Makes perfect sense at this stage, might be a simple overlook on my part 👍

from nvr-js.

To draw clips on the timeline, one could look to use ffinfo, along with the timestamp of the file being created/last modified, to accurately place them in time, the missing piece will be to associate any event data, and associate them and store the event data somewhere (other than SQLite :))

Yes - that is pretty much the implementation I was thinking.

Ah - true true. I forgot about the events timeline (which was one of the reasons I chose this over others - that is very interesting down the road to me, and a simple API for it makes a lot of sense).

Perhaps it might be worth exploring an option for that. Since the timeline of segments is critical for our uses, it would make sense to store event data in an optional database. If you're not using that feature (like us), then you can disable it.

If you DO want that feature, event data is likely considered as non-critical compared to the video on disk itself.

So using a more "fail-proof" mechanism ("this is what is on disk and that's all I know") makes sense to me if it does to you.

from nvr-js.

I think I will spend some time on the error

Makes perfect sense at this stage, might be a simple overlook on my part +1

Quick research suggests the "retry" module might be the ticket to patch it quickly :) Will give it a go in a branch.

I also think I will hit this issue again pretty quickly - I'm testing this on several Pi's on actual poles right now so... one of them is bound to trip up (something is causing problems regularly on our side - which is good for you and I :D )

from nvr-js.

Progress: I've got debug statements running on a camera system right now via this fork/branch:
https://github.com/city-of-shreveport/nvr-js.git/(debug-statements)

Working on setting up a way to cause failure for the SQLite DB and test the retry module in another branch.

from nvr-js.

Wow! You don't waste time 😅

Would you like to test MKV?

I did take a small look at the other PR - but your environment might be a more substantial test.

• can you view live footage
• can you view recorded segments
• all in the browser.

All whilst using mkv.

swap out the mp4 extension.

Here

And here

from nvr-js.

Certainly! That one is very selfishly interesting to me, ha!

One adjustment I'll need make to this issue's branch prior to that is to either revert to console.log or dig into making debug output to stdout instead of stderr - PM2 thought it was simply erroring out due to this but the logs themselves seem to indicate everything is fine.

Happy to say using 3 minute chunks on 3 connected cameras to the pi, it didn't hiccup once with the patch.

Oddly, at about 6:30am my time though (CST here), the process died in an errored state and didn't restart because of it, so I'll dig into that today.

Once I verify we are good to go with logging, I'll push the mkv patch too.

The MKV format is interesting to me because our analysts typically want to roll back the footage within a very short window (they pull footage and monitor on every 911 call).

And we've seen all kinds of things go wrong on a pole (power sag/power loss/cabling coming loose inside our box, disk space, mounted drive unmounted, etc). So, lots of opportunity for improperly closing the mp4. And MKV will allow us to view partially saved footage.

I've gotta go wear the suit and tie for a little while but once I can get some time to code I'll knock those two adjustments out (stdout and MKV testing).

from nvr-js.

Ok - you have twisted my arm 😅
I need a break from other OSS projects

I have created a branch (2.1.0) - I will use this to merge any enhancements being worked on - so any PR's, address them to this branch for the time being.

I think we can use the video file(s) as a way to provide the timeline content, and do away from any DB based index of segments.
this way even MKV files still being written to will be in the timeline.

I'll wait to hear back from the MKV tests, before I venture down that road, but if the MKV files prove to work in the browser, this should be a smooth change - even event metadata could be made extremely simple also - a flat JSON file per event (that contains the linked segment file name) 😎

Quick Example (bf9d2a78-7425-4249-a559-39bc9ff85f6c-1660052517.json)

{
  "event": "Building Car Park Gates Opened",
  "cameraId": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b",
  "linkedSegment": "yyyy-mm-dd-hh:mm.mkv",
  "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
  "timestamp": 1660052517
}

from nvr-js.

Awesome! I love the idea of a flat file metadata setup.

One thing I will need to is to checksum the video upon creation, for example, so as to provide proof that the file was not tampered with (if used in court in the U.S., chain of custody is a big deal when you can't prove authenticity of a file; with a checksum on creation, none of that matters much).

So that change could be interesting. Perhaps a video file + a metadata/events file along with each file on disk? IE: for any events coming in, they'd get appended to a file named the same as the video, but with a json extension.

This way, handing all details related to a clip is just two file downloads.

Re: 2.1.0 branch - will do!

from nvr-js.

So that change could be interesting. Perhaps a video file + a metadata/events file along with each file on disk? IE: for any events coming in, they'd get appended to a file named the same as the video, but with a json extension.

That's the ticket!

A metadata file per segment.
The metadata file will contain the checksum +basic info + any events that occurred in that segment

52e5b562-1a1c-4ae8-88a9-c92dc94b497b.1660054102.json

{
  "segment": {
    "cameraId": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b",
    "fileName": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b.1660054102.mkv",
    "startTime": 1660054102,
    "endTime": 1660098736,
    "checksum": "sha256:xxxxxxxxxxxxxx"
  },
  "events": [
    {
      "event": "Building Car Park Gates Opened",
      "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
      "timestamp": 1660052517
    },
    {
      "event": "Building Car Park Gates Opened",
      "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
      "timestamp": 1660052517
    }
  ]
}

These files then drive the timeline UI.
let me know how the MKV use holds up. 👍

from nvr-js.

PERFECT (for my use cases at least)!

I am happy to let you know that everything has been working perfectly with my branch's patch to re-enqueuing the writes to the SQLDB for over 12 hours now. I'll let it keep running for another 12 before I update that box to using MKV's.

I'm going to deploy this to another system now after making the MKV adjustment.

Question for my current branch, though - I do not submit pull requests often (ever), so I'm not all that versed on how best to handle this.

I have a pending branch that re-queues the SQLDB write. I actually need this in production atm, so am curious if you would:

1. Like me to submit a pull request for just that patch now that it's been fairly well tested in the wild?
2. Would you like another branch modifying the MP4 output to MKV from current main? Or should I spin up another branch based off of my current branch?

Thanks for the guidance!

from nvr-js.

Hey @KeithHanson,

Believe it or not. - I have made massive progress, in removing the need for SQLite altogether (I also don't waste time 😄)

This is what will be coming (have written the logic for).....

• When a new video file is about to be created
  • A new metafile for it will be created (as above - give or take)
    • events created during this segment will be added to that file also.
      • at the end of the segment, its checksum is added to that metafile
        • a new video file is created + a new metafile
          • Rinse and repeat

File name examples:

 {CamerID}.{StartUnixTimstamp}.mkv
 {CamerID}.{StartUnixTimstamp}.json

These files will be used to drive the UI.

This should remove a truck load of surface area for problems to occur.
I think at this stage

• Stick to your "remix of the logic" - for lack of a better term
• Once you patch the files to use mkv (in place of mp4)
  • let me know of the experience
• A PR is not likely worth it at this stage

Once I am happy, you can migrate to the latest version.
does that sounds like a plan?

I have not touched this project in a few weeks - so thanks for getting me excited about it again 😄

Importantly, no difference in the UI, its all backend changes. - but the UI should benefit by being able to view recorded footage that is still being written to (well on the basis MKV works 😅 )

from nvr-js.

By all means, once the new (better) enhancements are in place, feel free to rebrand it - Its OSS after all 😉

Renamed to [Enhancement]: Review Storage mechanics PLUS! 😅

from nvr-js.

My side of the world will be sleeping soon.
update here (for my Morning), to ensure MKV is still a go ahead, and I'll continue to work on the enhancements 👍

from nvr-js.

Ok. I can say that: it works reliably on both test machines. BUT...

For whatever reason, loading MP4 segments are significantly faster than loading MKV segments.

I have both in my timeline and recorded a test for you to see the dramatic difference.
https://cityofshreveport.app.box.com/s/pjjd2y3chp8iepxloy3qn4rh8zw8hyh4 (1 minute .webm file)

Basically, MKV took about 4-5 seconds to load PER SEGMENT (with 3-minute segments - very small compared to regular operation of 15-minute segments), and MP4 took about half a second, reliably.

I have tested it during multiple periods today, and the behavior has been the same. MKV is significantly slower to load than MP4. I have no idea why, though :/

from nvr-js.

Ok,

I have identified why MKV is slower.

As part of the recording chain, I move each files internal metadata to the start of the file (its usually at the end by default)
MP4 allows you to move this metadata to the start (and I do so)

This allows quicker loading of the file into the Browser (by only loading a small portion to get to the metadata and start playing whilst it still downloading), and guess what, MKV does not allow this, so even though I add the flag to the ffmpeg chain, it has no affect to MKV files.

Meaning the browser has to grab the entier file (to get to the meta info at the end of the file, before it can start playing)

Turns out also, that MKV is really only supported by Chrome, I'm going to opt out of using MKV, but if one wanted to, there is a value you can change in code.

I'm almost done with using flat files - so far its working great!
and the UI feels snappier with it

from nvr-js.

Ahhh that makes sense!

I think I am with you. Snappier is better for realtime/in-the-moment lookups. I set the segments small enough that I don't think we will need to worry about that.

That is looking really good! I'm excited to begin testing it 😁

One thing I've noticed on our machines, but PM2 registers the NVRJS.js as errored after some time. When I check PM2 logs NVRJS, I don't see anything that would indicate it errored.

I'll try to dig in when I can today, but if you have some insight there as well, we'd appreciate it 😁

I can also open another issue on it to track if you'd like. But that is really the only problem I'm bumping into with my requeue patch in place while waiting for the flat file updates 😁

from nvr-js.

Going to test this immediately :) THANK YOU! :)

I will keep digging into the PM2 issue. I love it due to PM2.io's interface, custom actions, metrics, etc. It's pretty nice when managing our 20+ camera systems that will eventually be 100+.

One question:

Since it now uses a different storage mechanism, it won't pick up on previous recordings, as the DB has gone.
Can you tell me more here? Is it because we have to generate the metadata file to know it exists? Perhaps we could create an "import" or a "backlog" sort of script that does that work with what's on disk?

Our storage is pretty much temp storage anyways - the important parts are the live monitoring and clip retrieval in the moment. So not having it show up in the timeline for previous isn't a huge deal breaker, though I may just blow it away and start fresh.

Not a huge deal, but could become important in the future to have some sort of "Catch up" script. Most important thing is the video is on disk and files get deleted over time when space or retention dictates.

from nvr-js.

Yup.

Recordings were catalogued in the DB, but since its now scrapped in favour of JSON, it won't use or load the DB.
in theory, one could query the SQL file to write a bunch of JSON files - so whilst possible, just needs someone to do it 😅

Im hoping the prospect of losing previous recordings is extremely rare when updating.
providing this new way is successful, I cant see it changing anytime soon. 👍

PS: Try the event system, events can now be created from live view and has the API of course 🤓

from nvr-js.

One thing I should change.

The timestamp of an event being created from live view is captured, ONLY after the Prompt.
so if a user takes 5 minutes to title the event - the timestamp is 5 minutes out.

easy change - Capture timestamp, then ask for event title.

from nvr-js.

in theory, one could query the SQL file to write a bunch of JSON files - so whilst possible, just needs someone to do it

I was thinking that the UI would simply pull from the filesystem - are you saying that if there's a file with no JSON next to it, it won't load in the UI?

This is a very edge edge case, so no worries IMO. Just want to make sure I understand the intended behavior.

from nvr-js.

Correct....

NVRJS now has an index of JSON files (lives in memory during runtime) - it scans the NVRJS_SYSTEM folder for the json files at start up, then updates it during the process lifetime.

such index looks like this.

Contract (timestamp = start time of segment)

{
  "cameraID": {
    "timestamp": "timestamp.json"
  }
}

Example

{
  "055d5242-b403-42fb-acfc-8b478ae5cebd": {
    "454354353534": "454354353534.json",
    "545656464566": "545656464566.json"
  },
  "055d5242-b403-42fb-acfc-8b478ae5cebd": {
    "454354353534": "454354353534.json",
    "545656464566": "545656464566.json"
  }
}

When the timeline in the UI is scrolled, it queries this index, pulling all JSON files between timeline start and timeline end that is currently in view providing the keys for the files falls within that period of time.

These JSON files, contain the meta - that is used to load clips and events in the UI.

{
  "segment": {
    "metaFileName": "1660154880.json",
    "cameraId": "66e39d21-72c4-405c-a838-05a8e8fe0742",
    "fileName": "1660154880.mp4",
    "startTime": 1660154880,
    "endTime": 1660157880,
    "checksum": "sha256:xxxxxxxxxxxxxxxxxxxxxx",
    "segmentId": "ef633a07-b720-4fd0-a15a-33ef0b8a346f"
  },
  "events": [
    {
      "eventId": "8d720aad-f7e6-49a4-824a-4b0289731a58",
      "event": "Agent 463 - Car theft Witnessed",
      "sensorId": "LIVE-VIEW-EVENT",
      "timestamp": 1660154975
    }
  ]
}

10 minutes off video, where you set a 5 minute segment time = 2 json files (each having there own mp4)
each JSON file will have the unix/epoch start time and end time, and contain any events that occurred during that segment.

During segment record, endTime is set to 0, the JSON file is created the moment a new segment has been started.
once the segment has ended, the endTime is set and checksum is calculated, and the file being saved again of course due to it being modified

from nvr-js.

Brilliant :) If you don't mind, I'd like to keep this open for the next 24 hours while I have my systems testing it?

Edit: I am deploying it to a dozen or so systems now for a bigger test.

But this is fantastic. Tested the event capture too. FANTASTIC.

One tip I'd like to share - I am telling my team to generally only use the live view sparingly because the live view is the full quality stream - which will chew up bandwidth (not a huge worry, but once we enable "buddy system backups" where a pole backs up to another pole in the city, we'll be using the bandwidth at full tilt all the time).

It would be amazing if I could config the low quality stream for live view, and limit the HQ viewing to the timeline.

I also have a few other additions I'd like to make - a "maximize" button on the floating dialog boxes, a config option to always show full controls on the video embed tag (we really need this - being able to go full screen and back quickly, hopping around in the video, etc), and a config option to disable login (we have device based access over an SDWAN called ZeroTier - if someone is on that, it's because I personally clicked the button to allow them to be) - we are not concerned with logins so it would be nice to skip it.

Would you like me to make separate enhancement issues like this one for each of those "wants"?

from nvr-js.

One more question - apologies.

When cleaning out old files, is it ONLY date based? Or does low disk space also trigger the clean as well?

from nvr-js.

It would be amazing if I could config the low quality stream for live view, and limit the HQ viewing to the timeline.

Already Possible, by updating the stream config section 😎

By default it uses copy as the encoder = no transcoding (next to no CPU usage),
will use what ever bitrate is offered by the incoming stream.

if you wanted to reduce it down, it needs to be transcoded (where you have control of encoding)
but that will use CPU.

Example of lowering the quality for live stream (using libx264)

streamConfig: {
  an: '',
  vcodec: 'libx264',
  'b:v': '256k',
  f: 'mp4',
  movflags: '+frag_keyframe+empty_moov+default_base_moof',
  reset_timestamps: '1'
}

There are various encoders available (some hardware ones also - I think it depends on the system that it is running on)
above is using a software renderer (libx264) at a bitrate of 256k

I haven't played with the options much (they are all ffmpeg options) - so tread carefully 😬
these settings are per camera

When cleaning out old files, is it ONLY date based? Or does low disk space also trigger the clean as well?

Its purely based on the number of days old the footage is continuousDays (14 days by default)

from nvr-js.

Ah - so, to clarify, I have 1 RTSP stream coming from the camera that is low quality, and another separate RTSP stream coming in for HQ.

I'd like the live view to use one RTSP stream, and the storage/timeline view to depend on the HQ RTSP Stream? Hope that makes sense.

Its purely based on the number of days old the footage is continuousDays (14 days by default)
Eek. Ok. I'm not 100% on the number of days so I'll play with it. It's another edge case, but bad things will likely happen if we hit 100% :P I'd feel much better if it took that into account.

If I stub my toe on this enough, I'll work on a small patch to keep a buffer of space available on top of days of retention.

Thank you! For everything :)

PS - how would you like me to handle my wishlist above? Enhancement issue + pull request?

from nvr-js.

Just want to report in - so far so good. I've deployed this to several systems (~10 atm) and have recorded roughly 6 hours of footage.

Analysts are kicking the tires on it right now as well :) Several excited reactions when I demo'd creating an event from the live view :D

from nvr-js.

Ah - so, to clarify, I have 1 RTSP stream coming from the camera that is low quality, and another separate RTSP stream coming in for HQ.
I'd like the live view to use one RTSP stream, and the storage/timeline view to depend on the HQ RTSP Stream? Hope that makes sense.

NVRJS uses 1 FFMPEG Process per Camera, it then conjures 2 output streams from the one input.
I sketched the below, this is per camera.

To pull in 2 streams from each camera, will require 2 FFMPEG instances.
so having just 3 cameras, will require 6 FFMPEG processes, which could start to see performance issues.

As below, the live stream can be tailored to suite, i.e to reduce bandwidth, but it will require transcoding (i.e using libx264, instead of copy for vcodec) - nothing stopping you using a HW encoder i.e h264_v4l2m2m (if your system has one, I know most recent RPIs do) but the HW encoder h264_v4l2m2m has a bug, in that it produces a green tint, so you need to build FFMPEG from source, to get the latest build (unless its been fixed already)

moving away from copy for the live stream, means you can set a bitrate for the video (b:v), but at the cost of having to transcode (using a HW encoder will save on CPU tax)

If I stub my toe on this enough, I'll work on a small patch to keep a buffer of space available on top of days of retention.

The way it works in NVRJS is a combination of continuousPurgeIntervalHours and continuousDays
Given continuousDays set to 7 and continuousPurgeIntervalHours set to 12.

it will purge any footage that is 7 days old every 12hours.

Maybe a version of this can live along side the purge schedule, with the ability to pass in the number of days, and triggered, based on a drive space poll. but the drive space poll, needs to be OS independent.

Just want to report in - so far so good. I've deployed this to several systems (~10 atm) and have recorded roughly 6 hours of footage.
Analysts are kicking the tires on it right now as well :) Several excited reactions when I demo'd creating an event from the live view :D

That's encouraging to hear, the project was a hobbyist project, so nice to see it used in this way 😄

PS - how would you like me to handle my wishlist above? Enhancement issue + pull request?

I would say a PR, raising issues is fine, but may take me longer to get time to work on them 😅

from nvr-js.

Hm. I just want to utilize the already encoded low-quality stream. But I will handle it another way (like spin up a second nvr-js to a backup drive) as I don't want to handle transcoding.

I do have an issue I've identified on two of the ten systems.

I don't know what causes it (logs look good), but I suspect it may be similar to before.

As you can see, we're doing great up to 1660218300.mp4, then the json files stop being created.

Any ideas? Assume all things can happen on the pole, because we've seen it plenty - power sag drops USB disk and things have to recover, for instance.

from nvr-js.

As a quick fix, I'm going to hack together an "import" job that will generate the missing json files.

Other than these two systems' and this one issue, the rest have been rock solid.

from nvr-js.

Mmm, nothing obvious is coming to mind - was it one camera or a whole unit?
I wonder if a small IO error caused the NodeJS FileSystem Watcher to halt.

See, I use Nodes FS Watcher API to monitor the directory, for new MP4s - as that is the trigger to create a new JSON file.

from nvr-js.

I could move that logic to listen for FFMPEG instead. - as I receive the segment entry name at the end of each segment.
but it currently does nothing

The above will receive the file name after each segment - so In theory, I could use it instead of monitoring for files (if IO errors can stop the watcher)

The only slight draw back - no JSON file will be available, only after the segment has finished - but during record, the JSON does not really serve a purpose, plus I need to see how I can detect the start of a new segment, if doing that 🤔

from nvr-js.

I would much prefer attaching to ffmpeg since that is working so well. We def have IO errors for about 10 seconds sometimes. Everything recovers within about that timeframe but the json unfortunately.

Waiting until the segment is finished is pretty typical of most NVRs so.... Not a critical need (to me).

from nvr-js.

Fantastic! I'll tune in here and redeploy everywhere once ready!

(Analysts will love disabling user/pass 😁)

from nvr-js.

Ah, that makes a lot of sense why that's not as straightforward as I was hoping.

Yes - that would be fine - as long as we're able to recover automatically we are happy :)

from nvr-js.

Just an update - I am moving the Meta Creation Logic over to listen for FFMPEG directly after all.
its WIP - so don't pull down any changes just yet.

from nvr-js.

Hi @KeithHanson ,

Only a couple of lines left really.
I'll see if I can knock it out this evening.

Will ping you shortly

from nvr-js.

THANK YOOOOOU!!!! Will deploy tonight!

from nvr-js.

Got excited. Deployed to one of our systems for testing and so far things are working :P

I've got about 30+ systems I'll deploy to tonight to verify.

from nvr-js.

Ok - I've got this running on 10 of our live deployments as of about 20 minutes ago :)

Going to let this run for 24 hours or so and report back :)

from nvr-js.

Oooh, as a bonus - I moved the files from one folder to the renamed camera folders and bam - everything was picked up. I did have to restart the service for it to show in the timeline, but suuuuper convenient!

from nvr-js.

Ok, I have deployed to all systems we have under control. ~30.

Something else I thought of that is a win because of this is the way we want to backup our data.

I'm planning on having a second NVRJS running on 81, but it will pull the low quality streams to disk. We're designing an algorithm that will pick the most appropriate "buddy" to back up to and from and kick off some kind of copy process.

Because of these changes, we can easily obtain at least low quality footage by simply using the interface and update the config to display the backed up folder <3

This lets us focus on the hard/critical part for us (choosing the right pole to back up to with a variety of factors) while getting a UI for free! :) :) :)

from nvr-js.

You are excited! 😆

There are APIs built in to fetch segments (metadata), current system utilisation and camera details, if you wanted to create a dashboard to monitor all instances (not footage, more health)- i have not spent a great deal documenting them but they do exist.

Also if I am reading your comments correctly - NVRJS has not been tested to run 80+ cameras (i.e pole 81) - just one to bare in mind.

Let me know if the recent patch fixes the missing meta files (I'm hoping they have)

from nvr-js.

30 total systems - We'll get to 80 before end of year :)

Excellent - we will definitely tap into that - we have a heartbeat service that checks all kinds of things relevant to us (disks, encryption status, temps, software versions, etc).

from nvr-js.

We did hit an issue on two systems.

If the metadata file is corrupted for any reason, parsing it fails and NVRJS goes into a boot loop.

2|NVRJS  | SyntaxError: Unexpected end of JSON input
2|NVRJS  |     at JSON.parse (<anonymous>)
2|NVRJS  |     at /home/pi/nvr-js/NVRJS.js:494:19
2|NVRJS  |     at Array.forEach (<anonymous>)
2|NVRJS  |     at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
2|NVRJS  |     at /home/pi/nvr-js/NVRJS.js:393:2
2|NVRJS  |     at Array.forEach (<anonymous>)
2|NVRJS  |     at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
2|NVRJS  |     at Module._compile (internal/modules/cjs/loader.js:1068:30)
2|NVRJS  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
2|NVRJS  |     at Module.load (internal/modules/cjs/loader.js:933:32)
PM2      | App [NVRJS:2] exited with code [1] via signal [SIGINT]
PM2      | App [NVRJS:2] starting in -fork mode-
PM2      | App [NVRJS:2] online

I am fairly sure we just need a try-catch here: https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/NVRJS.js#L494

I patched in a try/catch and things seemed to work:

$ node NVRJS.js 
 - Checking config.
 - Config loaded: /home/pi/nvrjs.config.js
 - Checking volumes and ffmpeg.
 - Creating express application.
 - Compiling pages.
 - Configuring camera: Camera 1
Unable to parse: 1660650121.json
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/pi/nvr-js/NVRJS.js:495:20
    at Array.forEach (<anonymous>)
    at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
    at /home/pi/nvr-js/NVRJS.js:393:2
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
    at Module._compile (internal/modules/cjs/loader.js:1068:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
 - Configuring camera: Camera 2
Unable to parse: 1660650120.json
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/pi/nvr-js/NVRJS.js:495:20
    at Array.forEach (<anonymous>)
    at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
    at /home/pi/nvr-js/NVRJS.js:393:2
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
    at Module._compile (internal/modules/cjs/loader.js:1068:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
 - Configuring camera: Camera 3
Unable to parse: 1660650120.json
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /home/pi/nvr-js/NVRJS.js:495:20
    at Array.forEach (<anonymous>)
    at InitCamera (/home/pi/nvr-js/NVRJS.js:492:15)
    at /home/pi/nvr-js/NVRJS.js:393:2
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/home/pi/nvr-js/NVRJS.js:391:9)
    at Module._compile (internal/modules/cjs/loader.js:1068:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
 - Strting purge timer.
 - NVR JS is Ready!
 - Purging data.

from nvr-js.

Yeah - a try catch should be added in a couple of places really.
I am more interested in the root problem.

What did 1660650120.json & 1660650121.json look like?

Just want to make sure I am doing nothing silly, and its more IO Disk errors, when writing files.

from nvr-js.

Almost undoubtedly disk error - we have to deal with it on our crappy power grid and recover automatically (sad, I know - drink one for me in commiseration :P).

Both files were empty.

from nvr-js.

If you want I can add in this try catch for now and submit a pull request. I've patched the two systems that failed manually for now.

If you're already off to the races on the patch I can hang back ofc :)

from nvr-js.

Awesome :) And that is fine - I'm not sure if there is any magical code that could solve that lol. But that's also a reason I reduce things to 3 minute chunks (faster streaming, faster downloading, less risk of missing important things because of gremlins/dragons like this).

from nvr-js.

Rock solid :D

from nvr-js.

Assuming all is well?
I am aiming to publish 3.0 soon.

from nvr-js.

i think this is a great project, and it gives me a good idea to make a NVR system. thank you very much.
i tested v2.0 and v3.0 version.
every .mp4 file record video for one minute. the record video part works well.
after i record video for 16 hours, if i scroll the mouse wheel many times to zoom in/out timeline quickly,
and then scroll the timeline from right to left a few seconds. it will crash, although i press the refresh button, it also doesn't work.
two versions all happened the same issue.

1. i tested the vis-timeline using the link: https://visjs.github.io/vis-timeline/examples/graph2d/08_performance.html, the issue didn't happen.
2. record video for 5 minutes(not many hours), the issue didn't happen.

i think when i scroll the mouse wheel, the timeline.on('rangechanged') will be called many times,
query SQLite or ReadMetaFile maybe have some delay.

from nvr-js.

Hi @trc-turing,

Version 3 has removed SQLIte entirely, and its now based on a JSON file per segment, this seems to have removed a lot of problems, v3 is currently being used in a very large installation, that seems to be quite stable.

See v3 Change Log
https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

As for rangechanged - yes! this is what causes the query (previously SQLite), but now uses the file mech.
rangechanged will only continue if rangechange is not triggered within 500ms.

from nvr-js.

thanks for your reply.
yes, you are right. if i change the settimout duration from 500 to 1000 or 3000.
it doesn't crash. but after i did it, when i click the timeline,
because the duration it will have a delay, or click event doesn't work.

from nvr-js.

Yup, the timeline needs to load data based on the time span in view +- 2 hours.

As the NVR can be running for weeks/months/years at a time, I don't load everything - as the browser could be overloaded with data, can you imagine 1 min segments over a week? That's 10,080 segments in the timeline!

But then zooming out will do the same 😅

I therefore need to cap it based on the current view.

I load what timespan is in view (plus 2 hours)

You can override the 2 hour buffer by changing SearchTimeBufferHours in the scripts file.

At the moment, I don't have a method (or time) to stop the unnecessary loading.

I can probably improve it, but I need to find the time todo so, and it's currently not a priority of mine.

I welcome PRs if you want to contribute 😇

from nvr-js.

你好@trc-turing,

版本 3 完全删除了 SQLIte，现在它基于每个段的 JSON 文件，这似乎消除了很多问题，v3 目前正在一个非常大的安装中使用，这似乎相当稳定。

请参阅 v3 更改日志 https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

至于rangechanged——是的！这就是导致查询的原因（以前是 SQLite），但现在使用文件 mech. 只有在 500 毫秒内未触发 rangechanged时才会继续。rangechange

nvr-js/web/static/js/scripts.js

Line 78 in 5639de6

timeline.on('rangechange', (event) => {

换句话说，滚动/缩放必须是静态的 500 毫秒 - 尝试弄乱这个值。

published yet to NPM

wait published to npm

from nvr-js.

@baozi510

The 3.0 Pull Request is ready -> #13

I'll publish in a day or so.

from nvr-js.