If the person actually knocks instead of ringing the doorbell then it's immediately suspicious to me. It doesn't even have to be someone dangerous or annoying, I think the worst would be someone from my past like an old classmate I haven't spoken to in a decade and perhaps barely knew - or worse, actually did know well enough. Not that any of them know my address, but I just wouldn't know how to deal with the situation.