That is a trickier idea than what it sounds like.
If I don't say anything, answers can cover anything.
If I say I have a performance concern, answers can still cover anything, and I can accept the one that best addresses performance.
Now, if I say I have a performance concern and I don't want to hear about anything other than performance... then I'm not asking for a code review, I'm asking for a performance tune-up.
A recent answer was downvoted for not addressing a performance issue, while still actively reviewing the OP's code. The (assumed-to-be) downvoter left a comment:
The OP didn't ask for a critique of syntax, but instead was looking for a way to make the code run with better performance. Yes, this is a code review site, but focus on the question at hand.
I feel that asking reviewers to not look at X or Y would go against the spirit of this site ("do I want feedback on any facets of the code?"), and could reduce the number of answers given. We don't need that.