Hi Ann,

You could use this expression but the tense is wrong it should be-

Ils m'avaient permis de les accompagner = They had allowed me to accompany them

and it is more formal than -

They had let me come with them = Ils m'avaient laissé venir avec eux

Hope this helps!