“Je ne devrais pas avoir à passer derrière toi !”

Brian, devoir is a ‘semi-auxiliary’ verb, and except in one context, will always be followed by an infinitive verb. You cannot use devoir on its own here - it makes no sense in French. 

In the construct, it fills the role that the modal ‘should’ does in English, and the overall structure is not that different - 

‘ne devrais pas’  - shouldn’t/ should not

 ‘avoir à’ - have to

‘passer derrière toi’ ‘ - go/walk/pass behind you.

The exception is devoir-noun - in which devoir has a different meaning, “owe something”. 

 https://www.lawlessfrench.com/grammar/devoir-lesson/

 https://www.wordreference.com/fren/avoir%20à%20faire

 https://progress.lawlessfrench.com/my-languages/french/glossary/111

Hi Maarten, thanks for breaking that down. And great links too!