sjacks26 / flockwatch Goto Github PK

This should definitely be an option, not a necessary function.
Might want to allow the user to specify the number of simultaneous runs.
Might want to do it smart, based on the number of CPUs.

Figure out the best way to report context. Olga suggests columns for example messages in the trending report and the co-occurrence report.

Add around https://github.com/sjacks26/FlockWatch/blob/deploy/FlockWatch.py#L86

Building the co-occurrence matrix is the most computationally expensive part of this code. Would it be faster/cheaper to use a nested dict instead?

Rather than:

Something like:
{collection term 1: { co-occurrence term 1: #, co-occurrence term 2: #, ..., co-occurrence term n: #}, collection term 2: { co-occurrence term 1: #, co-occurrence term 2: #, ..., co-occurrence term n: #}, {...}, collection term n: { co-occurrence term 1: #, co-occurrence term 2: #, ..., co-occurrence term n: #}}

Is that computationally easier??

Hi,
I'm trying to use FlockWatch, I've followed all the set up instructions and when trying to run Flockwatch (python3 FlockWatch.py -n file) I get this error:
AttributeError: 'PurePosixPath' object has no attribute 'startswith'
Please, could you help me? I don't know how to deal with this error?
Thanks in advance
Berta

Not a lot of error handling going on yet. Need to build that out.

Need to allow for multiple config files in case reminder emails for different projects should be sent to different email addresses.

One option is in #9. Another option would be to have a default, but allow users to specify an alternative via CLI argument.

Might only want to ignore these things for bigrams. Might also want to ignore them for co-occurring terms.

Allow users to specify more than one file for input. This is particularly helpful for json input, but probably csv as well.

Two approaches:

1. Build this into the main process
2. Create a stand-alone script that combines multiple files into the single file expected by FlockWatch.py

Add more output for time analysis.

I might want to see the number of times a term has co-occurred regardless of which collection term it has co-occurred with.

I might want to see the number of times a term has appeared in a co-occurrence report, regardless of the number of times it appears or what its rate it. But this output should still take the rate into account somehow or maybe just report an average or something.

To allow folks who don't know python to use this more easily, let users choose whether to:

1. Specify a complete config file
2. Be interactively guided through generating a config file (use input to have FlockWatch ask questions to populate a config file)

If users specify a complete config file through a CLI argument, this has the added benefit of letting them use multiple config files if they are running multiple simultaneous FlockWatches.

Add bigrams to trending report. For some collections, few unigrams are sufficiently precise to be useful for collection terms. Especially for more focused collections, bigrams (or even longer) will be more useful.

Won't run on Windows right now.

Right now, FlockWatch uses raw frequency counts from two time windows to identify trending terms. If there are many more messages in t2 than in t1, FlockWatch will find a lot of trending terms (simply because more messages means more opportunities for a term to appear).
Maybe FlockWatch should use frequency rates (normalized by the number of messages in a time window) rather than raw frequency counts?

Allow users to use data in json format in place of csv.

The only potential problem (I think) is with tokenizing. I don't know whether that would be a problem or not.