I just wanted to offer that it helps me to remember the correct use of this verb if I think about it in this way.
"Je lui manque" = "I am missed by her/him." -> (ie. S/he misses me.)
"Nous manquons à Pat." = " We are missed by Pat." (ie. Pat misses us.)
This method helps me to remember 1) the order of the sentence, 2) the subject to use to conjugate the verb, and 3) that there is an indirect object (by, à) so you have to use an indirect pronoun (or à if using a noun).