J'ai frappé à la porte mais tu dormais. -- I knocked at the door but you were sleeping.
Tu t'étais endormi. -- You had fallen asleep.
Tu étais en train de dormir. - That sounds a bit strange to my ears, because it sounds as if the act of sleeping as something one is actively doing. At least that's how it sounds to me.

Agree, which was why I questioned it. Have checked Linguee and DuoLingo and they both seem to like Etre endormis as being asleep. I suspect the question was related to the lesson on En train de.. fair enough.. but it came up in a general kwiz where it did not look right.

S'endormir is, strictly speaking, to fall asleep. Être endormi = to have fallen asleep, i.e., to be asleep. 

Elle s'est endormie il y a 10 minutes. -- She fell asleep 10 minutes ago.

yes.. endormi.. adjective.. asleep. My Robert has sleeping.. à moitié endormi. So, you could say Elle était endormie (or ils étaient endormis!)

My original endormis is wrong.. but I cannot edit it