What's the Problem?
Before discussing ways to prevent or cure problems with adaptive AI, we must first identify them. Picking out problems involves a combination of external observation and analysis of the internal workings of the system.
Symptom: Learning does not occur, or happens inconsistently.
Example: The decision tree for weapon selection usually selects the weakest weapon, despite better alternatives being available.
Diagnostic: The model or the implementation is (partly) broken.
Debug the code and design.
Verify the source code, comparing it to the theory.
Validate the model by step-through analysis.
Symptom: The learning does not match specific results, or degenerates over time.
Example: The reinforcement learning animat does not retreat when it has low health, but instead attempts heroic attacks.
Diagnostic: The system is not equipped to reliably provide the desired control.
Use explicit ways to control the learning with supervision.
Design an architecture to deal with the control problem without learning.
Limit learning to other subsets of behaviors or actions.
Decrease the learning over time as performance reaches satisfactory levels.
Symptom: Learning does not reach the perfect result.
Example: The average error of a neural network used for target selection is high.
Diagnostic: The design does not assist the adaptation; the system relies on optimality.
Design the system such that suboptimality is not a problem.
Provide hints to the learning by example (supervision) or guidance (feedback).
Model the problem better so it's easier to find the best solution (for instance, expert features).
Symptom: The behaviors are not realistic enough during the adaptation or at the end of the learning.
Example: Learning to aim causes the animat to spin around in circles for a few seconds.
Diagnostic: There is too much to learn; the policy is not designed for realism; the actions are inappropriate.
Learn as much as possible offline.
Select a better policy that rewards safe exploration and exploitation.
Design the actions at a higher level to reduce the unrealistic combinations.