Simulated Annealing

• Reducing the learning rate is closely related to the long used search technique of simulated annealing.
• The idea is that you heat the system up, making big jumps, then cool it down, making small jumps.
• You repeat this process but heating it up less each time.
• This comes from the way smiths fashion steel, annealing.
• In both cases, it's more likely to find a good stable state.