I don't feel that way but I know people who are "pigeon holed" and it bugs me. A little off topic of what you have described but I know a guy who has a learning disability, I forget what the term is but his brain processes certain things backwards. People said he'd never be able to do this or that and (I blame his parents) just let it be his excuse all through school and growing up. I know he's capable of doing more or better if he was just encouraged. Or if his parents worked with him or tired harder. Myself, when he's spelled things wrong, I've told him to think about it, how the word sounds and is used and he is usually able to figure it out. I've listened to his dad complain about how he's been pigeon holed since birth, how he gets stuck in certain programs or in groups because of it. I bite my tongue because I don't want to say anything, but to me if they rejected it and worked with him then he wouldn't have been as much.
To answer your question though, I say no. For those of us who get to live a so called normal life. In my experience it's the person who lets themselves be pigeon holed. Be it from a poor neighbourhood, having a disability, or a strict family. Like the guy I described above, so capable of doing so much more than what is expected of him, and people I know who have disabilities who refuse to let it dictate their life. They still do what they want to do or try their best because they don't want it to be an excuse to hold them back, to be pigeon holed.