tudelft-cda-lab / groot Goto Github PK

Hello! I am trying to reproduce results from the article, and I can't figure out certain problem. First I am trying to run train_kfold_models, but the code always ouputs an error: "ImportError: cannot import name 'GrootTree' from 'groot.model'". Is there something wrong with the .py file I am trying to run, or is this problem something that doesn't occur to you and everyone else (-->something wrong on computer or files or environment)?

Onni Mansikkamäki

Hello! I am trying to reproduce the results from the paper. I am struggling to find, where these files: generate_k_fold_results.py, train_kfold_models.py, fit_chen_xgboost.py, image_experiments.py, summarize_datasets.py and visualize_threat_models.py are provided?

Onni Mansikkamäki

Running the example code on the make moons data in the README I get:

Traceback (most recent call last):
  File "/home/.../groot_test.py", line 11, in <module>
    tree = GrootTreeClassifier(attack_model=attack_model, is_numerical=is_numerical, random_state=0)
TypeError: __init__() got an unexpected keyword argument 'is_numerical'

Leaving out the argument and having this line instead:
tree = GrootTreeClassifier(attack_model=attack_model, random_state=0)
results in this error:

Traceback (most recent call last):
  File "/home/.../groot_test.py", line 15, in <module>
    adversarial_accuracy = DecisionTreeAdversary(tree, "groot").adversarial_accuracy(X_test, y_test)
  File "/home/.../venv/lib/python3.9/site-packages/groot/adversary.py", line 259, in __init__
    self.is_numeric = self.decision_tree.is_numerical
AttributeError: 'GrootTreeClassifier' object has no attribute 'is_numerical'

I'm guessing the code got an update, but the readme didn't. Or I made a stupid mistake, also very possible.

Hello,

When predicting the label of a sample with prediction_value = 0.0, the predicted class is 1 with the "greater or equal" inequality in groot/toolbox.py:

def predict(self, X):
        prediction_values = self.decision_function(X)
        if self.n_classes == 2:
            return (prediction_values >= 0).astype(int)
        else:
            return np.argmax(prediction_values, axis=1)

So it is for adversarial accuracy with tree ensembles, as the inequality is also "greater or equal" in groot.verification/kantchelian_attack.py:

def optimal_adversarial_example(self, sample, label):
        if self.binary:
            pred = (
                1 if self.check(sample, self.json_model) >= self.pred_threshold else 0
            )

with self.pred_threshold = 0.0

However, when computing the adversarial accuracy for a single decision tree, in groot/verification/decision_tree_attack.py, the inequality is strict.

def adversarial_examples(self, X, y, order, options={}):
        # Turn 'leaves' into bounding boxes and leaf prediction values
        bound_dicts, leaf_values = zip(*self.leaves)
        predictions = [value > 0 for value in leaf_values]

For consistency, I think the inequality should be large too