Ignoring stuff like this just makes you look like a soft target, I went through 9 years of hell in my first job, all because I sat and ignored the problem, so the key is in how you react.
Here's a good trick to employ.......
Go to your boss (as high up as possible), tell him/her that your concerned about certain people (don't mention names) and how they seem to spend a lot of time socialising and gossiping about others at work.
Explain that your a professional and you come to work to do your job and whilst you want to work as part of the team, you don't intend to socialise with other staff during work hours.
Explain that your concerned that their misguided views are spreading and that your worried about these views spreading upwards within the company.
Explain that you don't care what people think, but you do care if their idle gossip starts to affect your job security.
Any manager worth their salt, will take this as an indication of you being a valuable employee who dedicates all their work time to actually doing their job. He/She will take a dim view of those who are trying to disrupt your working day.
I know this works, I've done it myself and the managing director (the man at the very top) of the company told me "I don't care what anyone else says, I think you do a **** good job and you've got a lot of untapped potential, if they give you any more trouble just tell them to go f**k themselves or send them to me"