rappeler de and rappeler du

Clive, this illustrates the 2 different forms of the verb 'se rappeler' that are in everyday use.

"Se rappeler de" in the first case, with 'vous vous rappelez (de le) petit restaurant' contracting to 'du petit restaurant'. This could have been said/written 'vous vous rappelez le petit restaurant' 

Se rappeler (sans de) in the 2nd example - Tu te rappelles ton professeur de maths.  The 2nd sentence would read "Tu te rappelles de ton professeur de maths ' with the use of the alternative form 'se rappeler de'. 

The prepositional phrase in '(professeur) de maths' is a separate component of the sentence and not a specific part of the lesson. The 'de' here is part of a descriptive expression. 

With 'se souvenir de..' there is no option to drop the preposition 'de'. 

With both forms of se rappeler, and with se souvenir de, article-noun follows (proper names aside), and when the preposition 'de' is used, contraction occurs with the article, or name, as usual.  

 https://www.lawlessfrench.com/grammar/de-preposition/