dogoncouch / logesp Goto Github PK

Feature Idea

List rule - compare a field in events to a list

Rule needs:

• Field
• File name & type (text file/JSON)
• Time interval
• Whitelist/blacklist option
• Severity, etc.

Feature Idea

Customize forms for creating objects (LimitRule, HardwareAsset, etc)

• Split into sections
• Add form documentation

(myenv) sai@sai-VirtualBox:/opt/LogESP$ make newdb
--- Loading initial migrations...
python manage.py migrate
File "manage.py", line 14
) from exc
^
SyntaxError: invalid syntax
Makefile:90: recipe for target 'newdb-setup' failed
make: *** [newdb-setup] Error 1

Feature Idea

Dead process rule - trigger if event doesn't happen (like limit rules, but with minimum limit instead of maximum).

Rule needs

• Basically the same logic as limit rules
• Fire if under minimum event limit

New App Idea

was installing in ubuntu 16.04LTS had everything configured with virtual environment python 3
(env) root@syslogserver-VirtualBox:/opt/LogESP# python --version Python 3.5.2

Feature Idea

the error

(env) root@syslogserver-VirtualBox:/opt/LogESP# make newdb
--- Loading initial migrations...
python manage.py migrate
Traceback (most recent call last):
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
return Database.connect(**conn_params)
File "/opt/LogESP/env/lib/python3.5/site-packages/MySQLdb/init.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/opt/LogESP/env/lib/python3.5/site-packages/MySQLdb/connections.py", line 164, in init
super(Connection, self).init(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (1698, "Access denied for user 'logespd'@'localhost'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "manage.py", line 15, in
execute_from_command_line(sys.argv)
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/management/base.py", line 361, in execute
self.check()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/management/base.py", line 390, in check
include_deployment_checks=include_deployment_checks,
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 64, in _run_checks
issues = run_checks(tags=[Tags.database])
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/checks/registry.py", line 72, in run_checks
new_errors = check(app_configs=app_configs)
File "/opt/LogESP/env/lib/python3.5/site-packages/django/core/checks/database.py", line 10, in check_database_backends
issues.extend(conn.validation.check(**kwargs))
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/mysql/validation.py", line 9, in check
issues.extend(self._check_sql_mode(**kwargs))
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/mysql/validation.py", line 13, in _check_sql_mode
with self.connection.cursor() as cursor:
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 256, in cursor
return self._cursor()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 233, in _cursor
self.ensure_connection()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/utils.py", line 89, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
self.connect()
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/base/base.py", line 195, in connect
self.connection = self.get_new_connection(conn_params)
File "/opt/LogESP/env/lib/python3.5/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
return Database.connect(**conn_params)
File "/opt/LogESP/env/lib/python3.5/site-packages/MySQLdb/init.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/opt/LogESP/env/lib/python3.5/site-packages/MySQLdb/connections.py", line 164, in init
super(Connection, self).init(*args, **kwargs2)
django.db.utils.OperationalError: (1698, "Access denied for user 'logespd'@'localhost'")
Makefile:90: recipe for target 'newdb-setup' failed
make: *** [newdb-setup] Error 1

Bug

Overview

many thanks would love to hear your response : )

Expected Behavior

Actual Behavior

Steps To Reproduce Behavior

Specifications

• LogESP version (in LogESP/__init__.py):
• Python version:
• Django version:
• Operating system:
• Operating system version:
• Deployment info
  • SQL server name/version:
  • SQL client library name/version:
  • WSGI name/version:
  • Web server name/version:
• Log file type:
• Other info that might be useful:

Feature Idea

Add install instructions for CentOS.

Feature Idea

Add Makefile target for updating.

• Stop LDSI
• Pull from git
• Migrate database
• Update static files
• Update fixtures
• Restart LDSI

Feature Idea

Add more security to Ubuntu install documentation to make it more production-ready.

• Review Django deployment
• Set up virtual environment updates
• Test MariaDB/Nginx configuration

Bug

Overview

Default Ubuntu install hangs on shutdown/reboot unless you kill uwsgi beforehand (use -11; SIGTERM seems to have no effect).

Specifications

• LogESP version (in LogESP/__init__.py): 0.2-dev10
• Operating system: Ubuntu
• Operating system version: 16.04
• Deployment info
  • WSGI name/version: uwsgi 2.0.17
  • Web server name/version: nginx 1.10.3-0ubuntu0.16.04.2

Bug

Overview

Some parser threads lose connection to the database periodically, even if the database is on localhost.

Expected Behavior

No DB error events (or at least not many) logged by LogESP.

Actual Behavior

Daemon log events logged by logesp (manage.py) when a thread gets repeated DB errors. The error portion of the events:

Error: (2006, 'MySQL server has gone away')

The threads never re-establish a connection.

Steps To Reproduce Behavior

Seems to happen more as the number of parser threads goes up.

Specifications

• LogESP version (in LogESP/__init__.py):
• Python version: 3.5.2
• Django version: 2.0.4
• Operating system: Ubuntu server
• Operating system version: 16.04
• Deployment info
  • SQL server name/version: MariaDB 10.0.34-0ubuntu0.16.04.1
  • SQL client library name/version: mysqlclient (1.3.12)

New App Idea

Feature Idea

Bug

Overview

Expected Behavior

Actual Behavior

Steps To Reproduce Behavior

Specifications

• LogESP version (in LogESP/__init__.py):
• Python version:
• Django version:
• Operating system:
• Operating system version:
• Deployment info
  • SQL server name/version:
  • SQL client library name/version:
  • WSGI name/version:
  • Web server name/version:
• Log file type:
• Other info that might be useful:

Feature Idea

ESP rule - triggered by chain of events in a specific order.

Rule Needs

• Event chain specifics - use standard ESP language:

RuleEvent.source_rule_name=RULENAME FOLLOWED BY
LogEvent.source_host=X.X.X.X AND LogEvent.log_source=LOGSOURCE
FOLLOWED BY RuleEvent.source_rule_name=RULENAME

• Time range to check
• Check interval

Logic

1. Convert ESP language to list of dictionaries
2. Get events in time interval (work on making this more efficient later)
3. Check events in reverse, comparing to reversed list of dictionaries
4. Create rule event if sequence is matched