Don't brush him off right away if you aren't interested; just have a short and pleasant conversation.
The other guys nearby won't think you are so intimidating. Now, before you get upset, let me just say that there is absolutely nothing wrong with an opinionated woman. I think it's time we stop thinking of the label "intimidating" as a bad thing.
You may have a large social network or be an accomplished dancer or activist. If a man actually says "you're too intimidating" as an excuse to break up with you, let him walk away without any fuss.
In any case, a man you might date and form a relationship with should admire and appreciate the accomplishments that make you, you. What's he's really saying is he's sure you're too cool, too beautiful and too all-around amazing to stick with a schlub like him if something better comes along. You deserve to be with a confident man has more faith in your relationship than that.
If your facial expression and your walk suggest you have absolutely no interest in talking to others, he might decide it's is better to steer clear.