yes, you could say that. It would mean that there was one occasion when you were small that he scared you. If you wanted to express that he did it habitually when you were small you would use the imperfect in both clauses:
Il me faisait peur quand j'étais petit.
-- Chris (not a native speaker).
Sign in to submit your answer
Don't have an account yet? Join today
Test your French to the CEFR standard