Flags for Rewrite about clustr HOT 7 CLOSED

Great! That looks like what I was originally envisioning. Basically the approach we're taking is to "trust" that the user is passing a valid column name, but to have it fail loudly if it isn't so that they are aware they did something wrong. This will make it much easier for future people down the line to make cuts based off of new columns we haven't thought of yet, while also keeping a history of config files that show exactly how each run was done.

from clustr.

(UPDATE)
First flag test was successful. I used the merger flag to test out the function and 61 out of 1092 rows were accepted for analysis.

I'll need to cross verify If I'm actually removing the correct rows using pandas in a separate session. Once that is finished I'll continue working to include other flag types and update when that works.

from clustr.

Check out 0d935a3 as a possible final draft for clustr.py.

This is the output of the above version.

from clustr.

Removed 89 clusters due to Serendipitous flag of type True
Removed 62 clusters due to merger flag of type True
NOTE: Removed counts may be redundant, as some data fail multiple flags.
Accepted 245 data out of 1092
mean x error: 5.60469675e+42
mean y error: 3.874964
Using Kelly Algorithm...

Making Plots...
Done!

We just need to find out if cutoff refers to detected or not in the catalog I think.

from clustr.

(Update)

What I've done is completely reworked how the boolean flag was done in the version referenced by me above. Boolean flags are explicitly in the config.yml file and not hard coded in clustr.py. I will upload both files as soon as I'm done reworking the cutoff flags in clustr.py (later tonight).

from clustr.

(Update 2)

Okay so I've uploaded clustr.py with its newly and improved create_cuts() method. This was done to avoid hard coding the flags in the source code as @sweverett advised. To replace the previous strategy of having to referencing a list to know which flags to apply, I've created nested dictionaries for both the boolean flags and range flags in the new config.yml file. The content in the config.yml file is exactly the same as param.config file with the following dictionary structure.

For example,

Boolean: {
    key1: value1,
    key2: value2,
    key3: value3,
    key4: value4
}

Range: {
    keyA: {
        keyB: {
            keyC: valueC,
            keyD: valueD,
            keyE: valueE}},
    keyF: {
        keyG: {
            keyH: valueH,
            keyI: valueI,
            keyJ: valueJ}}
}

Doing this gives the user the freedom of adding in a new flag type into the config file without having to mess with the source code.

Below is the result I get when running the code:

The cutoff section in create_cuts() is not finished yet. I'm hoping to have it done by tomorrow afternoon.

from clustr.

Went ahead and did some further changes to config.yml. The notation I used in the example above was not the correct for a YAML file so I went ahead and simplified it 😅 along with some instructions on how to add new flags.

from clustr.