I seem to always be accused of staring at others, even when I'm absolutely not

It seems that I am accused of staring at other people...often...when I'm not. Most of the time I am totally oblivious to whether or not that person was even in the room. If I'm looking at a restaurant menu and someone just happens to be in the direction of the menu, that person would accuse me of staring at them. If I'm at work and am looking at my computer monitor and someone is standing behind the monitor a distance away (try to picture that), I'll be accused of looking at them. My thing is, if you notice someone looking at you, don't you have to look at them. I feel eyes on me, all the time. But I know that no one is that into me that they have to stare. What do you think the problem is? I'm not overly concerned with what other people do and I don't "zone out." I'm in my 30s and am very conscious of what I do and how I behave and I'm not "cross-eyed" and I don't have a lazy eye. I just need help understanding this foolishness.