On the upside, if someone were gossiping about you, at least that means they care enough about your existence to comment on it. As long as that's true, you can count on someone saying something somewhere.

But, if they aren't plotting active harm, then whatever they're saying doesn't generally matter. Gossip happens, sure, but that doesn't make it relevant enough to be paranoid over.