This is the way I attempted to come to grips with this difficult concept. If it’s wrong, more to learn ! 

In the French sentence you want to translate the English ‘I remind your nanny of you’ 

We have :

The person reminding/“stimulating a memory” - the subject - Je (I) 

The person being reminded/in whom the memory is stimulated - the indirect object in French - à votre nounou (your nanny) 

The person being remembered/“the memory” - the direct object in French - vous (you) 

I find it easier constructed like this : 

Verb  - Rappeler (conjugated according to the subject)

Subject           Memory             Person Reminded 

I                        You                    Your nanny 

Je                     Vous                   á votre nounou

Object pronouns go before the verb, and the sentence structure here becomes 

Subject/Direct object pronoun/verb/indirect object : 

“Je vous rappelle à votre nounou.” (As you note ‘je rappelle vous à votre nounou’ is not correct because the object pronoun is following the verb).

The sentences ‘je vous rappelle votre nounou’ and ‘je rappelle votre nounou à vous’ (this one doesn’t sound right to me) both would translate as ‘I remind you of your nanny’, as votre nounou is ‘direct’ in both. (Vous is the indirect object pronoun here - still placed before the verb normally).

Je vous rappelle votre nounou. -- I remind you of your nanny.
Je rappelle votre nounou à vous. -- I remind your nanny of you