Track Email Conversation Threads with owasync.py about iris HOT 6 OPEN

Hey,

Thanks for looking into this. While we could store some metadata in the incident context and then iterate backward parsing the ID out of the json looking for it, this is likely to be inefficient and slow when you have lots of incidents.

We were hoping to find headers that only exist on email replies to a thread. Do you see any other headers aside from Thread-Index that we could merely check the presence of?

from iris.

I agree that looking backwards in all incident context carries the risk of being very slow with a lot of incidents. For this reason, I'd restrict it to incidents within the last 24 hours by default (with a configurable lookback time). As long as the incident.created is indexed (and it is), I don't foresee this being too much of a drag on performance.

Can I show you what I have in mind? Here's a diff of the proposed changes:
master...p6knewman:issue_274_track_thread_index

from iris.

Hey,

This might cause problems for us and other very high volume iris instances as we have around 2000 incidents per day.

It would be cleanest for this process to not have state. Would it be possible for you to instead of CC'ing, BCC iris when you need to loop it into a thread and that way subsequent replies won't create incidents?

Regards

from iris.

Not sure I understand, or maybe we're not on the same page. The query of recent incidents is this line:

+                    if config.get('owa'):
+                      lookback_hours = config['owa'].get('incident_lookback_time', 24)
+                    query = 'SELECT context FROM `incident` WHERE `created` >= (NOW() - INTERVAL ' + str(lookback_hours)  + ' hour)'

If the worst-case scenario for incident volume is on the order of 2000 per/day, the WHERE clause on the default configuration setting would only produce 2000 rows returned. That doesn't seem too onerous to work with, IMO. If you have a high volume DB, would you be willing to pull down my branch and test performance with a couple test incidents created via email? I could also work on creating a ton of junk data on my dev instance to see what performance is like.

As for BCC vs. CC, that unfortunately wouldn't solve my use case. The users that may/will be sending incidents in via email may are unlikely to remember to go through such specific steps when logging an incident.

from iris.

I went ahead and generated some junk data on my dev instance. I generated 2000 incidents per day going back for the last 365 days (730k total). I timed the performance of both the MySQL query and the combined time it takes to query the DB + ujson.loads the context and iterate over it looking for the same thread_id.

The performance numbers were quite acceptable even up to a retrieval of ~28,000 incidents (~336hrs at 2000 incidents/day), which really seems like a silly long period to be looking back for duplicate incidents. Long before that far back, I'd be considering this a new incident and it's time to page someone out again. At a more reasonable period of 24 hours, the performance impact is roughly 90-100ms.

with 730k incidents of random context data, 2000 incidents per a day, incident_lookback_time 48 hours:
web.1    | [2017-10-05 09:05:38 +0000] [98023] [WARNING] iris.api Fetched 4003 incidents from DB in 0.09 seconds
web.1    | [2017-10-05 09:05:38 +0000] [98023] [WARNING] iris.api Total time to fetch from DB + parse 4003 recent incidents: 0.1 seconds
web.1    | 127.0.0.1 - - [05/Oct/2017:09:05:38 -0500] "POST /v0/response/email HTTP/1.1" 204 - "-" "python-requests/2.9.1"


incident_lookback_time 72 hours:
web.1    | [2017-10-05 09:06:38 +0000] [98742] [WARNING] iris.api Fetched 6003 incidents from DB in 0.12 seconds
web.1    | [2017-10-05 09:06:38 +0000] [98742] [WARNING] iris.api Total time to fetch from DB + parse 6003 recent incidents: 0.13 seconds
web.1    | 127.0.0.1 - - [05/Oct/2017:09:06:38 -0500] "POST /v0/response/email HTTP/1.1" 204 - "-" "python-requests/2.9.1"


incident_lookback_time 144 hours:
web.1    | [2017-10-05 09:07:45 +0000] [99102] [WARNING] iris.api Fetched 12001 incidents from DB in 0.3 seconds
web.1    | [2017-10-05 09:07:45 +0000] [99102] [WARNING] iris.api Total time to fetch from DB + parse 12001 recent incidents: 0.35 seconds
web.1    | 127.0.0.1 - - [05/Oct/2017:09:07:45 -0500] "POST /v0/response/email HTTP/1.1" 204 - "-" "python-requests/2.9.1"


incident_lookback_time 336 hours:
web.1    | [2017-10-05 09:09:28 +0000] [99599] [WARNING] iris.api Fetched 27994 incidents from DB in 0.49 seconds
web.1    | [2017-10-05 09:09:28 +0000] [99599] [WARNING] iris.api Total time to fetch from DB + parse 27994 recent incidents: 0.54 seconds
web.1    | 127.0.0.1 - - [05/Oct/2017:09:09:28 -0500] "POST /v0/response/email HTTP/1.1" 204 - "-" "python-requests/2.9.1"

from iris.

@jrgp, any thoughts on this? If preferred, I could make this whole feature subject to an on/off config option and default it to off.

from iris.