Just wondering what your colleagues are like? Who are making the comments? Are they females or males?
It could be some female coworkers are trying to make you feel bad, it's not unlikely, especially if it's a workplace with many women? I don't know your workplace, but I did experience some...