when do you use tôt rather than en advance?

tôt = early (in an absolute sense)
en avance = earlier than a certain time.

Le train arrive à 5 heures du matin. Il arrive tôt. -- The train arrives at 5 am. It arrives early.
Le train arrive avec 5 heures d'avance. -- The train arrives 5 hours early (than it was supposed to).

Le train arrive toujours 1 heure en avance -- The train always arrive 1 hour early

Chris, why did you use "avec 5 heures" in one example, and "1 heure" in the other example?

I think this is because one says ‘avec 5 heures d’avance’, while the other says ‘1 heure en avance’. Slightly different ways of saying the same thing. 

The verbs arriver, venir, finir, commencer all use either (duration) «en avance» or «en retard»,  or (avec duration) «d'avance or de retard». Chris has given an example of each use with «arriver».

Note the difference from the construction with verbs avoir or être. Expressing timeliness in French - late/early - precise