Well, if you focus on the more common Pokemon types, you may be able to escape detection. It all depends on the person who spots you - some are more observant than most, after all.

Honestly, if you aren't stupid about it, you shouldn't have to worry.