EASE Material

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== NEEMS Lecture: 5. Train the Next Action Classifier =====

In [[https://ease-crc.org/material/ease/machinelearning/machine_learning_theory|the previous section]] we talked about some machine learning theory. This section will cover the training of our decision tree model.

First, the prepared_narratives that were created earlier in this lecture, are split between train_set and test_set. Then the train_set is split into the features and labels, where the features contain the previous and parent actions for each action, and the labels contain the data about which action comes after another (with the //next_// prefix).

Since all the functionality is already implemented, the test_set can be prepared as the train_set has already been.
<code python>
#TODO Split the test set into features and labels. Their variables should have the prefix test_set
test_set_features, test_set_features_cols, test_set_labels, test_set_labels_cols \
    = split_data_in_features_and_labels(test_set)
test_set_labels.head()
</code>

After preparing the test- and dataset, two suitable parameters need to be decided for, determining the depth and breadth of the decision tree. The parameters that we speak of are within the two //range// constructions, namely the 9 for the max_depth and 21 for max_leaf_notes. Finding parameters that train the model to an F1 value above 0.9 can be done programmatically by increasing both values, since more depth and nodes result in a more precise model. Taking a maximum depth of 14 and 26 leaf nodes is the lowest setting to reach a score of just above 0.9.
<code python>
parameters = {'max_depth':range(1,14,1), 'max_leaf_nodes': range(2,26,1)}
</code>
Now if the tree model is exported, a .dot file should be generated in the //data// directory, where this exercise is downloaded to. By executing the shell command below the export, a PNG image is generated.
<code bash>
# cd to the data directory, then execute this 
dot -Tpng tree.dot -o tree.png
</code>
Open the PNG file to see the decision tree of the model we just trained. With the parameters used above, the generated decision tree looks like this.

{{ :ease:tree.png |}}

Zooming in to the root node we can see the fist decisions made. The root contains the decision, that gains the most information. First the //type// of the current action is determined, then the higher-order //parent// action. Both information finally lead to a concluded //next// action in one of the leaf nodes.

{{ :ease:tree_root.png |}}

Depending on the highest entry in the //value// list in a node, the most likely //next// action is determined, which is //NoAction// in the root node, since this is the best prediction so far, without having made any decisions. Also, the Gini score shows very well how certain a decision have been made at any point in the tree, where we begin with a pretty high score of 0.813 at the root.

1.) Say, the type of action we currently try to predict something for is //MovingToLocation// (value higher than 0.5). From the root node we would go down the right branch (false), since //type_MovingToLocation %%<=%% 0.5// is false.\\
2a.) The next decision to be done is based on whether the //parent// is //MovingToOperate// and let's say it is **not** (value **below** 0.5). Also recognize, that the //class// entry on the node changed to //LookingForSomething//, since this is the most likely //next// action so far. When //parent_MovingToOperate %%<=%% 0.5// evaluates to true the branching goes on to the left, finally hitting a leaf node.\\
3a.) The model predicts, when the type of action is //MovingToLocation// and its parent is **not** //MovingToOperate//, the most likely //next// action to follow will be //LookingForSomething// since this is the only entry in the //value// list. With a Gini score of 0.0 this decision is unlikely to be wrong, based the data we have.

2b.) In a different scenario the //parent_MovingToOperate// is **above** 0.5, meaning the decision would evaluate false. We now hit a different leaf node to the right.\\
3b.) Even though the //value// list has multiple positive entries, the highest value is //ClosingADrawer//, which would be the prediction in a scenario where //type// is //MovingToLocation// and //parent// is //MovingToOperate//. You can see that with a Gini Score of 0.579 the prediction is not very certain, since there is at least one other high entry in the //value// list, representing the //OpeningADrawer// action. But with the features given, this is still our best prediction.

Finding the right parameters is one of the hardest things when it comes to training a model. Achieving a Gini score of 0.0 in every single leaf might indicate that the model is heavily overfit. On the other hand, having a model too vague may result in generalized prediction with small variance. By reducing the feature space just right, the predictions will be certain enough for your purposes.

If you are interested in specific entries in the //value// list, there will be an illustration in the next chapter, where the values are represented in a confusion matrix. The labels of said confusion matrix are in the same order as the //value// list, such that you can investigate what other classes could have been predicted aside from the highest value.

In [[https://ease-crc.org/material/ease/machinelearning/classifier_evaluation|the last section]] we will evaluate the trained decision tree model.