Week 45: Decisions Trees, Random Forests, Bagging and Boosting

 

 

 

Entropy and the ID3 algorithm

The ID3 algorithm learns decision trees by constructing them in a top down way, beginning with the question which attribute should be tested at the root of the tree?

  1. Each instance attribute is evaluated using a statistical test to determine how well it alone classifies the training examples.
  2. The best attribute is selected and used as the test at the root node of the tree.
  3. A descendant of the root node is then created for each possible value of this attribute.
  4. Training examples are sorted to the appropriate descendant node.
  5. The entire process is then repeated using the training examples associated with each descendant node to select the best attribute to test at that point in the tree.
  6. This forms a greedy search for an acceptable decision tree, in which the algorithm never backtracks to reconsider earlier choices.

The ID3 algorithm selects which attribute to test at each node in the tree.

We would like to select the attribute that is most useful for classifying examples.

What is a good quantitative measure of the worth of an attribute?

Information gain measures how well a given attribute separates the training examples according to their target classification.

The ID3 algorithm uses this information gain measure to select among the candidate attributes at each step while growing the tree.