bluecherrydvr / bluecherry-apps Goto Github PK
View Code? Open in Web Editor NEWBluecherry surveillance system (server application)
Home Page: http://www.bluecherrydvr.com
License: GNU General Public License v2.0
Bluecherry surveillance system (server application)
Home Page: http://www.bluecherrydvr.com
License: GNU General Public License v2.0
There have been several reports of recording threads for IP cameras (particularly the AXIS m3011?) freezing, which makes the camera silently stop streaming and recording until the server is manually restarted.
We need to detect frozen recording threads by tracking their activity from the main thread, and should warn loudly if any are unresponsive. I'm not sure what we can do to respond to unresponsive threads, though - we can't terminate the thread, and starting a new one for the same device seems like a bad idea. The best solution I can think of is to kill the process, so it will be restarted and can resume recording - but that is a bit of a nuclear option.
When cleanup is needed for a recording path, we should (after #1062) delete files that aren't tracked in the database. Aside from avoiding non-empty directories during configuration, we should consider only deleting files that look like they were created by this software (e.g. following the media folder/filename pattern). The goal here is to avoid losing track of media files - for example, during a clean reinstallation - and filling up recording paths with unnecessary and invisible files.
One of the options should allow to close the feed.
We need to work out a system for real-time communications from the server and web interface to the client. This is necessary/useful for:
Real-time event notification (especially motion)
Notification of recording status
Notification of device changes (added/removed/reconfigured)
..?
The actual protocol of this can be nearly anything, even as simple as a never-ending XML stream over HTTP. However, it must be possible for the client to maintain a connection and receive immediate notification of the situations above and anything else we find useful.
Details of implementation can be discussed at a later date.
Axis supports on-camera motion detection with notification via meta data, see the docs repo for more information.
By using one of the javascript frameworks below we can create a touch-enabled version of the web interface for live viewing, events and eventually event playback:
http://jqtouch.com/
http://www.iwebkit.net
http://www.sencha.com/products/touch/ (Commercial)
http://www.sproutcore.com/
Client reports adding a generic RTSP url in IP Cam Viewer (http://www.bluecherrydvr.com/2013/09/accessing-your-cameras-from-third-party-applications-or-mobile-devices/) causes the feed to crash and the server feed locks up
Client should be ablt to determine which version of API server supports (for example: if it supports afterId or ids parameters to events call).
It should be a simple integer updated each time a new API call/parameter is available.
Currently the license warning says this:
bc-server994: E: [licensing] The 30 day trial of Bluecherry has expired. You can buy a license from http://www.bluecherrydvr.com
It should say purchase from:
ids parameter should be added to events request in following format:
ids=1,2,3,4
This will allow client to download only given subset of events.
For example:
client already displays 10 000 events, 20 of them are 'in progress'
client asks for newsets events using "afterId" parameter
client adds these events to display list
client asks for 20 'in progress' events using ids paremter
client updates these events in display list
Then we will have nicely working events lists that updated only neccessary items and does not loose selection.
I have a customer with the following setup:
I was able to connect internet to the machine and reset the license key on the server and deleted and reinstalled the same key on the v2 installation and the key was accepted. As soon as I restarted the bluecherry service, it popped up this error:
Mar 15 09:48:43 dvr bc-server31106: F(): Authorization failed for license WRT2-WYDJ-66UO-SYCJ. The machine ID may have changed, which requires manual reauthorization.
I then deleted the key, reset it on our side, then did a manual authorization, installed the machine ID and the license key was accepted. The bluecherry service worked and I then restarted the bluecherry service and the same error populated again.
Between customers not following the dbconfig-common instructions and random issues with dbconfig-common and SQL upgrades we need to come up with a better solution for database upgrades and migrations.
After several hours the RTSP live streaming dies, recording to disk still works (which is odd), MJPEG live feed also works.
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(3/Duke's): Error writing to live stream: Invalid argument
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(2/Gym): Error writing to live stream: Invalid argument
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(1/Kitchen): Error writing to live stream: Invalid argument
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(3/Duke's): Error writing to live stream: Invalid argument
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(4/Grand Hall 1): Error writing to live stream: Invalid argument
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(3/Duke's): Error writing to live stream: Invalid argument
Mar 20 19:13:30 mrbdvr1 bc-server[1060]: E(2/Gym): Error writing to live stream: Invalid argument
We will need the server to 'mark' motion events in the database during continuous recording so they uniquely show up during the playback part of the UI.
Needed for MJPEG only support
Events remain in the 'event list' even if the video is long deleted.
Currently there is a fixed one second pause that occurs between each camera when the server starts up. This means the cameras are staggered by one second, which is fine for a couple cameras, but if you have 40 cameras on a server the last one start recording for forty seconds.
A mysql 'crashed' table can make for a very bad day, and the end user won't know bad things are happening unless they look in the logs or notice nothing is being recorded:
Jan 16 09:59:17 BigBrother bc-server[924]: (SQL ERROR): [INSERT INTO EventsCam (time,level_id,device_id,type_id,length,media_id) VALUES(1326733147,'warn',13,'motion',-1,10837154)] => Table './bluecherry/EventsCam' is marked as crashed and last (automatic?) repair failed
Jan 16 09:59:17 BigBrother bc-server[924]: (SQL ERROR): [INSERT INTO EventsCam (time,level_id,device_id,type_id,length,media_id) VALUES(1326733157,'alrm',13,'not found',-1,NULL)] => Table './bluecherry/EventsCam' is marked as crashed and last (automatic?) repair failed
Aug 2 13:57:42 myhost apache2[14764]: (SQL ERROR): [DELETE FROM Devices WHERE id='5'] => Cannot delete or update a parent row: a foreign key constraint fails (bluecherry
.EventsCam
, CONSTRAINT EventsCam_ibfk_3
FOREIGN KEY (device_id
) REFERENCES Devices
(id
))
Unknown if this crash is specifically related to ACTi's MJPEG feeds over RTP.
Jul 27 15:13:33 ladon bc-server[5990]: RTSP session for rtsp://Admin:[email protected]:7070/ has multiple video streams. Only the first stream will be recorded.
Jul 27 15:13:33 ladon bc-server[5990]: I(1/Lobby): RTP stream started: Video: mjpeg
Jul 27 15:13:33 ladon bc-server[5990]: I(1/Lobby): avlib error: colorspace not supported in jpeg
Jul 27 15:13:33 ladon kernel: [ 6697.021638] bc-server[18853]: segfault at 4 ip 013e057a sp b6dfef70 error 4 in libavcodec.so.53[f62000+6cc000]
Jul 27 15:13:33 ladon kernel: [ 6697.034710] init: bluecherry main process (5990) killed by SEGV signal
Jul 27 15:13:33 ladon kernel: [ 6697.034924] init: bluecherry main process ended, respawning
Jul 27 15:13:33 ladon bc-server[18956]: I: Started Bluecherry daemon
Jul 27 15:13:33 ladon bc-server[18956]: I: SQL database connection opened
Jul 27 15:13:33 ladon bc-server[18956]: Event 5 left in-progress at shutdown; setting duration
Jul 27 15:13:33 ladon bc-server[18956]: I: Not licensed; running in trial mode
Jul 27 15:13:33 ladon bc-server[18956]: I(1/Lobby): Camera configured
Jul 27 15:13:33 ladon bc-server[18956]: I(1/Lobby): Switching to new recording schedule 'continuous'
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib fatal: No JPEG data found in image
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: invalid id 30
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib fatal: No JPEG data found in image
Jul 27 15:13:34 bc-server[18956]: last message repeated 3 times
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: mjpeg: unsupported coding type (c6)
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib fatal: No JPEG data found in image
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: mjpeg: unsupported coding type (c5)
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: only 8 bits/component accepted
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib fatal: No JPEG data found in image
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: mjpeg: unsupported coding type (cd)
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: mjpeg: unsupported coding type (ca)
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: only 8 bits/component accepted
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib fatal: No JPEG data found in image
Jul 27 15:13:34 ladon bc-server[18956]: I(1/Lobby): avlib error: invalid id 231
Analog audio recordings (from solo devices) are currently done with the ALSA API, which seems to conflict with PulseAudio's desire to have control of all devices (snd_pcm_open returns EBUSY). I suspect that the problems with the current code run deeper than that, and from a casual glance, Pulse may match our recording pattern better anyway.
Look at using Pulse rather than ALSA for audio capture; if that fails, we must find a way to get ALSA recording working regardless of Pulse.
Currently the video files are cut and restarted every 15 minutes, customers want the ability to change that
Using a flash player a customer should be able to stream videos from a web browser without losing much quality.
We really need this to be HTML5 based, however HTML5 doesn't support the Matroska container in it's video tag. We can't tie up the CPU either transcoding videos on the fly, so we need to look into some way to support this.
Currently we support banning web client connections but we do not support disconnecting RTP streams.
ONVIF is a standard that allows cameras to be detected and configured in a universal way. Links are below:
http://www.onvif.org/
http://www.openipcam.com/forum/index.php/topic,155.0.html
http://en.wikipedia.org/wiki/Open_Network_Video_Interface_Forum
We're going to need a CGI script to transcode a source (or pass-thru if the source is MJPEG already) from bc-server to an HTTP MJPEG feed.
It is going to be a replacement for www/media/mjpeg.php.
If it's for me, it can be written in any language; PHP, Python, whatever, even C. It just has to do the job.
It doesn't need to be identical in usage to mjpeg.php, it could even require a full source URL to be passed via POST; or it could support only bc-server from localhost, whatever you feel is simpler.
Currently we log any error as 'Not Found'. We need to support at least these two:
'Offline' (for connection timeouts)
'Authentication failed'
We also need verbose handling of errors. The actual error received from the network camera should be attached to the XML so the client can read and display it.
From @vogel
Fix for bug #1015 disallows sending more than 15000 event records from server to clients.
This can be overriden by adding limit=25000 parameter to URL request.
But I think that sending even more events should not be a problem as long as we dont save all events in memory before sending them.
This line:
$events = data::query($query);
executes this:
$db->fetchAll()
which loads all results into memory as an array.
Using http://php.net/manual/en/mysqli-result.fetch-row.php and sending records one by one will allow us to send as many results as we want.
Currently this column is not used in the web, since ids are changed each time bc-detect runs.
If IDs are not used in the server we need to remove the column or truncate the table instead of deleting old records.
Currently, any frame flagged as having motion by the driver will begin a motion event. The threshold required to trigger a flag is configurable, but this still results in a huge number of false positive motion events, often generated by very brief movement like noise or shadows. Most actual motion will be much more consistent, and last over a period longer than a single frame.
Implement a server-level threshold, where a motion event is started only after a number of frames have been flagged within a certain period. This number should be fairly low.
One suggested approach for the threshold is to trigger only when at least N% of frames in the past X seconds contain motion, where N is a low tunable value, and X is a short period of time (generally between .25 and 2 seconds?). A fraction of frames within period X should be used over a static count to avoid false negatives on low framerate devices.
bluecherry.Devices.deleted
Currently users have to manually change permissions when storage paths are added. We need a wrapper that checks the path, and runs a command (sudo without a password for the wrapper?) that changes permissions to bluecherry.bluecherry and 755 for the storage path.
@ruminsam Is this added in your disk management work already?
In some cases if the machine id is changed the confirmation should be done automatically.
Tested with 2.0.3 and 2.0.4:
root@BC-DVR:~# tail -f /var/log/bluecherry.log
Apr 29 14:53:23 BC-DVR bc-server[8129]: I: SQL database connection opened
Apr 29 14:53:23 BC-DVR bc-server[8129]: Media /var/lib/bluecherry/recordings/2013/04/29/000001/14-53-14.mkv left in-progress so updating length to 8
Apr 29 14:53:23 BC-DVR bc-server[8129]: I: Found 0000:02:05.0[solo6110] id 0, 8 ports
Apr 29 14:53:24 BC-DVR bc-server[8129]: I: Licensed for 8 devices
Apr 29 14:53:24 BC-DVR bc-server[8129]: I(1/Port 1 on Card 1): Camera configured
Apr 29 14:53:24 BC-DVR bc-server[8129]: I(1/Port 1 on Card 1): Switching to new recording schedule 'continuous'
Apr 29 14:53:24 BC-DVR bc-server[8129]: W(1/Port 1 on Card 1): Invalid Video Format, assuming MP4V-ES
Apr 29 14:53:26 BC-DVR bc-server[8129]: W(1/Port 1 on Card 1): Invalid Video Format, assuming MP4V-ES
Apr 29 14:53:26 BC-DVR bc-server[8129]: E(1/Port 1 on Card 1): Cannot decode video frame: Operation not permitted
Apr 29 14:53:26 BC-DVR bc-server[8129]: E(1/Port 1 on Card 1): No video frame for snapshot
Adding afterId parameter will allow us to only download new events on the client and add them to existing event view (very usefull in main window).
That will help users with lots of events:
http://community.bluecherrydvr.com/responses/problem-with-client-events-201
To help better keep files in sync, and to help cleanup the time-line, event durations should be cut off four times an hour, every 15 minutes (:00, :15, :30, :45).
This will have no effect on motion events that exceed 15 minutes (they will remain cut off after 15 minutes)
At the same time as this change, we should implement proper cutting just before a keyframe, so that we never lose any video frames between continuous recordings.
Add support for a utility to run and detect if the drive(s) used for storage are SMART capable (most are) or software raid and check the status of the drive every X amount of minutes. If an error or problem is detected report that to the event log and mark it as critical. It might also be useful to send an email to the system administrator notifying them of the status.
name field collapses, so there is no way to rename the camera if the name was removed once
A option in live view should enable low bandwidth mode where only 1 or 2FPS is pulled from the server.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.