etsy / 411 Goto Github PK

Hi,

I've made a new install with the latest version, configure mysql instead of a previous Sqlite default install, and now i don't have an SQL syntax error with the LIMIT clause ! Multiple-indicies is OK too !
I've deleted the 411 indices in elasticsearch to start from a clean situation !

A search is configured to send notification once a day with the cron notation and is enabled (high priority) !
I don't know what i'm doing wrong, but i can't receive any email notification ???
If i execute the search everything is ok, and i receive the email notification !

My local smtp is up and running (Postfix) ...

Can someone help me please ?

Thanks for your help.

PS : The enable button is not so clear with the default colors (enable vs disable) !

Is this type of query supported by 411?

metrics.server_lag:[0 TO 86400]

It gives me an error. I assume it is either not supported or I need to use different syntax?

Thanks!

I mentioned that graph Actions never shows anything:

Even though I acknowledged and resolved a lot of alerts.

Could that be a bug related to SQLite/MySQL differences?

I think it is better to rename config.php into config_example.php. Otherwise every time I pull from GitHub my config.php is getting overwritten. Just a suggestion.

Hi,

In the initial setup guide, the cronjob configuration seems outdated.
I've followed the guide today, and got the system up and running, but cronjob didn't work

# Add the following line into your crontab.
$ crontab -u USER -e
 /var/www/411/bin/cron.php > /dev/null 2>&1 && /var/www/411/bin/worker.php > /dev/null 2>&1

Running the CRON itself produces the following error

[+] Scheduler: 1475078702
except [NONE] Errno 2: "pcntl_exec(): Error has occurred: (errno 13) Permission denied" at [/var/www/411/phplib/Scheduler.php:44] 0:[pcntl_exec() called at [/var/www/411/phplib/Scheduler.php:44]] 1:[FOO\Scheduler->process() called at [/var/www/411/bin/cron.php:47]]
err [411_Scheduler] Scheduler error site:[1] ret:[256]

I had to change to this, which seems to make the cron functionality work

* * * * * php /var/www/411/bin/cron.php --backfill --date=/$(date +\%Y-\%m-\%d) --site=1
* * * * * php /var/www/411/bin/worker.php --site=1**

am i completely off, or is this correct?

Could you provide more info on these data soirees?

Thanks!

Hey,

I cannot generate a successful login.

I installed 411 on a host and then:

:/var/www/411$ sudo sqlite3 data.db < db.sql
:/var/www/411$ sudo bin/create_site.php
Creating new site
Site name: test
Hostname: test
From email: [email protected]
From Error email: [email protected]
Default To email: [email protected]

Site created! ID: 1
marius@rkv-c3s-selks2:/var/www/411$ sudo bin/create_user.php
Creating new user
Username: user
Real name: User User
Password: 
Email: [email protected]
Admin (y/n): n

User created! ID: 1

So far, so good. Now I go to the site and voila:

I have the distinct feeling that the hostname needs to be correct. Is that so?

I checked out the DB:

 sudo sqlite3 data.db
SQLite version 3.8.7.1 2014-10-29 13:59:56

sqlite> .tables
alert_logs      groups          report_targets  search_targets  users
alerts          jobs            reports         searches
config          lists           search_filters  sites
group_targets   meta            search_logs     slogs
sqlite> select * from users;
1|1|user|User User|$2y$10$EvfB6.2XkRcuobAEPnNLKuu5IqLx0BH/aQKXkrkRS9uzmaNcGHy9.|[email protected]|0|{}|0|1471875162|1471875162

I keep the hash here since it's obviously just a test in the lab ;). Anyways, I don't get a successful login with this.

What time does 411 sends daily roll up alert?

We do not use HTTPS but all links in the email alert point to HTTPS:

It would be great if alerts page would auto refresh every x seconds/minutes/etc.

We're seeing cron output every minute as a result of running cron.php & worker.php.

It might be an idea to silence the scripts (or have a switch to do this) and/or modify the setup instructions to include > /dev/null 2>&1 for the cron entry.

Cheers.

Running "requirejs:js" (requirejs) task
Error: ENOENT: no such file or directory, open '/411/htdocs/dist/js/views/searches/search/push.js'
In module tree:
main
app
views/searches/search/load

{ [Error: Error: ENOENT: no such file or directory, open '/411/htdocs/dist/js/views/searches/search/push.js'
In module tree:
main
app
views/searches/search/load

at Error (native)

]
originalError:
{ [Error: ENOENT: no such file or directory, open '/411/htdocs/dist/js/views/searches/search/push.js']
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/411/htdocs/dist/js/views/searches/search/push.js',
fileName: '/411/htdocs/dist/js/views/searches/search/push.js',
moduleTree: [ 'views/searches/search/load', 'app', 'main' ] } }
The command '/bin/sh -c grunt prod' returned a non-zero code: 1

I mentioned that alert and alert resolution have different subjects:

[411] xxxx xxxxx marked Resolved (Not an issue) --> Resolution
[411] Zoom --> Alert

Ideally we would have them match as far as search name. So in this example I would see:

[411] Zoom xxxx xxxxx marked Resolved (Not an issue) --> Resolution
[411] Zoom --> Alert

In both cases we have name of the search. This is very important for systems like PagerDuty - I could match [411] Zoom to be able to open/close PagerDuty tickets. Right now it is problematic as any resolution would close any ticket in PagerDuty since we do not have search name in the subject.

Within config.php, is it possible to create another section to query a logstash index of a different name?
I tried adding one but it doesn't come up as an option when creating a new Search.

Example:

# Configuration for the logstash index that 411 queries.
    'logstash' => [
        'hosts' => ['http://192.168.0.11:9200'],
        'index_hosts' => [],
        'index' => 'logstash-apache',
        'date_based' => true,
        'date_field' => '@timestamp',
        'src_url' => null,
    ],
     # Syslog configuration
    'syslog' => [
        'hosts' => ['http://192.168.0.11:9200'],
        'index_hosts' => [],
        'index' => 'syslog',
        'date_based' => true,
        'date_field' => '@timestamp',
        'src_url' => null,
    ],
];

When in the search view - try to switch from Detailed view to Compact view. I always get an error.

Hello,
I really like the project so far.
I was wondering if It's possible to create an alert using the API.
I already have custom app that can call a webhook when an alert is triggered. I'd like to add them into 411.

Edit : TLDR : Can I push an alert to 411 rather than pulling it from ES or Graphite ?

The queries used to generate the Dashboard data are pretty slow. Rewrite them so they're faster or experiment with querying ES instead.

I receive the following error when trying to query field names that have a hyphen in them. Have tried a few ways around it i.e. escaping and such and only to be given other errors. Just curious if this is expected behaviour, if there is a way around it, or if we could get a bug fix. Almost all my fields have hyphens, works fine on any fields that don't. sad face.

Query given:

os-cpu-one:>10

Error Thrown:

Error: Catch all: Expected ":" or [a-zA-Z0-9._] but "-" found.

What is the difference between All and Rotation types? I can't find any info on this.

Thanks!

Hey, have created a few users with "bin/create_user.php"
unfortunaly keep getting "Invalid credentials" errors on the login screen.

Could not find a way to check the logs or troubleshoot, any tips?

Regards,
AB

I am using MySQL and getting this error when I login to website:

except [NONE] PDOException: "SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ' "unixepoch") as date, action, COUNT(*) as count FROM alert_logs INNER J' at line 1" at [/var/www/411/phplib/DB.php:97] 0:[PDOStatement->execute() called at [/var/www/411/phplib/DB.php:97]] 1:[FOO\DB::query() called at [/var/www/411/phplib/REST/Dashboard.php:59]] 2:[FOO\Dashboard_REST->GET() called at [/var/www/411/phplib/REST.php:107]] 3:[FOO\REST->route() called at [/var/www/411/htdocs/api/dashboard.php:6]], referer: http://test.test.com/

Using the demo site logging in as user / user I was able to just edit my user and grant admin access.

Firstly - Great project!
Re. config.php
I have a time based index 'logstash-apache-*' in ES. I have tried logstash, logstash-apache-, logstash-2016.08.12 in config.php but still get a Failure in the health screen. The alerts section works fine.
Any debug logging available?

'logstash' => [
'hosts' => ['http://192.168.1.196:9200'],
'index_hosts' => [],
'index' => 'logstash-apache-',
'date_based' => true,
'date_field' => '@timestamp',
'src_url' => null,

Curl request:
curl http://192.168.1.196:9200/logstash-apache-2016.08.14?pretty
{
"logstash-apache-2016.08.14" : {
"aliases" : { },
"mappings" : {
"default" : {
"_all" : {
"enabled" : true,
"omit_norms" : true
.....

Adding a new data source for an existing Search is tedious atm, see #3 for details. We should add support for registering multiple subtypes under a Search. Ideally, this would let you just modify a config file and have the new type up and running.

Here's the curl to reproduce the exception

curl 'https://demo.fouroneone.io/api/alert/bootstrap?query=state%3A(0+1)' --1.0 -H 'Host: demo.fouroneone.io' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://demo.fouroneone.io/alerts?query=state:(0+1)' -H 'X-Requested-With: XMLHttpRequest' -H 'Cookie: 411sess=9a060399e2e44a4332b18f55098d0b8cf5d907fd5fbe7a2fe6705c95cb235db1%3A%7B%22nonce%22%3A%2263IlrPHsjCnCw3YEZ5p9%5C%2F2xvNViSmDMA%22%2C%22_%22%3A1474008875%2C%22id%22%3A2%7D' -H 'Connection: keep-alive'

I follow in install instruction ... but I get Invalid Credentials
I install 411 in Ubuntu 16.04.1 LTS

The user exist in the database everything looks fine ... any idea ?

411 is using epoch_second pattern which isnt standard available in ES 1.5

How does email notification actually work? I would like to use smtp on my CentOS 6.6 to relay messages to our primary email server. Is this possible or do I need to use Exim?

Thanks!

Andrew

Some of the queries that 411 uses are expensive and the results never change. The Dashboard queries are a good example of this. Rather than recomputing this information every time someone loads the page, just generate it at the end of the day.

There might be a bug on the dashboard API.
On my installation, the Crons are running, but i continue to get the Processor has not run in more than 20 mins! Error on the dashboard.

no_recent_cron comes from the /api/dashboard endpoint, which backs of
master/phplib/REST/Dashboard.php[line 81]

Looking in the database directly, the meta table, "last_cron_date" "2016"
$data['no_recent_cron'] = (int) $meta['last_cron_date'] < ($_SERVER['REQUEST_TIME'] - 20 * 60);

Since ($_SERVER['REQUEST_TIME'] - 20 * 60) returns a very big number, this statement will always be "TRUE" triggering the alert message.

To fix?

composer.lock defines ecl url as './ecl' however that path doesn't exist, this causes composer install to fail.

"dist": { "type": "path", "url": "./ecl", "reference": "565c5929ab503b6241c9fd6ddc65ea478a817f1e", "shasum": null }

Does Autoclose feature work? I selected to close Alert in 1 minute but they sit there forewver until I manually close them.

Please provide docker image for testing 411 locally.

Also, docker-compose would be nice.

Is it possible to have time in CEST instead of GMT Please ?

Thanks for your help.

Is it planned to manage users/groups from Active Directory please ?

Thanks for your help.

It would be great if email formatting was a little bit better:

Same with the view on Alerts page:

If I delete Search before I resolve all alerts that it produced those alerts are staying forever on Alerts page - no matter what I do (Resolve, Acknowledge etc):

I keep getting the Apache2 Default Page.

All my configurations has the hostname and even setup the SetEnv as well in the
vi /var/www/411/411.conf
&
bin/create_site.php
please help

Hi,

I'm trying to use the following query:

type:authlog AND failure:* NOT ssh_interface:public

Results in the error Error: Catch all: Expected ":" or [a-zA-Z0-9._] but " " found.

Is this supposed to work with plain Lucene query syntax, or am I making horrible assumptions about ESquery?

Thanks :)

What does this message mean?

I saw on demo site and on my own. What could be the cause of that?

Hi,

It looks like jobs are not running on my system:

[411]$ /var/www/411/bin/cron.php && /var/www/411/bin/worker.php
[+] Scheduler: 1474926082
[+] Maintenance
[+] Search Health
[+] Rollups
[+] Searches
[+] Reports
[+] Summary
[+] Autoclose
[+] Cleanup
[+] Worker
[+] Worker: 1474926093
[+] Job count: 64
PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 1 near "LIMIT": syntax error in /var/www/411/phplib/DB.php:78
Stack trace:
#0 /var/www/411/phplib/DB.php(78): PDO->prepare('UPDATE jobs S...')
#1 /var/www/411/phplib/Job.php(167): FOO\DB::query('UPDATE jobs S...', Array, 0)
#2 /var/www/411/phplib/Worker.php(77): FOO\JobFinder::getAndLock('1', 1474926093)
#3 /var/www/411/bin/worker.php(34): FOO\Worker->processSite(Object(FOO\Site), 1474926093)
#4 {main}

thrown in /var/www/411/phplib/DB.php on line 78
err [411_Worker] Worker error site:[1] ret:[65280]

And I see this in the web UI:

Any ideas on why this is happening?

Thanks!

Andrew

Is it possible possible to alert on cardinality having some range. ex:
*| agg:terms field:event_data.TargetUserName| agg:card field:event_data.IpAddress

This returns the unique count of ip adresses per user. I want to trigger an alarm when the count is in range [x TO y].

411's ESQuery syntax isn't 100% compatible with Lucene's query syntax - mostly because we don't currently make use of those features. Opening up this issue to keep track of all these inconsistencies so we can fix them.

Query reference

List:

• NOT
• "Quoted" clauses are not equivalent to shorthand

Hi,

I am trying to use the webhook capability but keep getting Target webhook: Remote server returned 0 on a slack webhook

Hey,

i can't get this issue solved while trying to install 411.

core@ubuntu:/var/www$ 411/bin/create_site.php 
PHP Fatal error:  Uncaught exception 'PDOException' with message 'could not find driver' in /var/www/411/phplib/DB.php:49
Stack trace:
#0 /var/www/411/phplib/DB.php(49): PDO->__construct('sqlite:/var/www...', 'root', NULL, Array)
#1 /var/www/411/phplib/DB.php(35): FOO\DB::connect('sqlite:/var/www...', 'root', NULL)
#2 /var/www/411/phplib/411bootstrap.php(62): FOO\DB::init()
#3 /var/www/411/bin/create_site.php(7): require_once('/var/www/411/ph...')
#4 {main}
  thrown in /var/www/411/phplib/DB.php on line 49

Right now 411 only allows for, without changes to code outside config, searching a single index index prefix inside a single cluster.

It would be nice be able to configure multiple indexes or clusters then for each search specify against which indexes or clusters it should be executed against.

Having a problem with the SSL connection, Elasticsearch is reporting an 'unknown CA' error and 411 has the error listed below.
If I create a simple ES PHP connector as follows, the connection works fine:

<?php

require 'vendor/autoload.php';

$hosts = ['https://localhost:9200'];
$myCert = '/home/mycert.pem';
use Elasticsearch\ClientBuilder;
$client = ClientBuilder::create()
                    ->setHosts($hosts)
                    ->setSSLVerification($myCert)
                    ->build();

$params = [
    'index' => 'logstash-2016.08.30',
    'type' => 'logs',
    'id' => 'AVbdvHEwAG-7iPeMOVRs'
];

$response = $client->get($params);

411 Configuration

# Elasticsearch configuration
$config['elasticsearch'] = [
    # Configuration for the 411 Alerts index.
    'alerts' => [
        'hosts' => ['https://localhost:9200'],
        'index_hosts' => [],
        'ssl_cert' => '/home/mycert.pem',
        'index' => null,
        'date_based' => false,
        'date_field' => 'alert_date',
        'src_url' => null,
    ],
    # Configuration for the logstash index that 411 queries.
    'logstash' => [
        'hosts' => ['https://localhost:9200'],
        'index_hosts' => [],
        'ssl_cert' => '/home/mycert.pem',
        'index' => 'logstash',
        'date_based' => true,
        'date_field' => '@timestamp',
        'src_url' => null,
    ],
];

411 Error

[Wed Aug 31 07:50:46.005023 2016] [:error] [pid 1598] [client 1.1.1.1:35031] except [NONE] Elasticsearch\\Common\\Exceptions\\NoNodesAvailableException: "No alive nodes found in your cluster" at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ConnectionPool/StaticNoPingConnectionPool.php:51] 0:[Elasticsearch\\ConnectionPool\\StaticNoPingConnectionPool->nextConnection() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php:71]] 1:[Elasticsearch\\Transport->getConnection() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php:89]] 2:[Elasticsearch\\Transport->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:228]] 3:[Elasticsearch\\Connections\\Connection->Elasticsearch\\Connections\\{closure}() called at [/var/www/411/vendor/react/promise/src/FulfilledPromise.php:25]] 4:[React\\Promise\\FulfilledPromise->then() called at [/var/www/411/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php:55]] 5:[GuzzleHttp\\Ring\\Future\\CompletedFutureValue->then() called at [/var/www/411/vendor/guzzlehttp/ringphp/src/Core.php:341]] 6:[GuzzleHttp\\Ring\\Core::proxy() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:283]] 7:[Elasticsearch\\Connections\\Connection->Elasticsearch\\Connections\\{closure}() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:159]] 8:[Elasticsearch\\Connections\\Connection->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php:99]] 9:[Elasticsearch\\Transport->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/AbstractEndpoint.php:79]] 10:[Elasticsearch\\Endpoints\\AbstractEndpoint->performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Namespaces/BooleanRequestWrapper.php:34]] 11:[Elasticsearch\\Namespaces\\BooleanRequestWrapper::performRequest() called at [/var/www/411/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Namespaces/IndicesNamespace.php:919]] 12:[Elasticsearch\\Namespaces\\IndicesNamespace->existsTemplate() called at [/var/www/411/phplib/ESClient.php:68]] 13:[FOO\\ESClient->initializeIndex() called at [/var/www/411/phplib/ESClient.php:31]] 14:[FOO\\ESClient->__construct() called at [/var/www/411/phplib/REST/Dashboard.php:16]] 15:[FOO\\Dashboard_REST->GET() called at [/var/www/411/phplib/REST.php:107]] 16:[FOO\\REST->route() called at [/var/www/411/htdocs/api/dashboard.php:6]], referer: http://myserver.com/

in Slack.php:62 the CURL 'username' => Util::getSiteName(), attribute seems to cause problems.
I created a standard webhook to a slack-channel, and got a 400 Bad request back.
I Changed it to "webhookbot" and it startet to work.

After a fresh install with the latest update, cannot login too !

except [NONE] FOO\UnauthorizedException: "Invalid nonce specified" at [/var/www/411/phplib/REST.php:101] 0:[FOO\REST->route()

/api/data give the right hostname:8443

Can someone help please ?

Thanks