Maybe, it is not entirely true. We need recognition. What other people think and say about us, effect the way we see ourselves. They are the voices, the louder version of our thought. We can always stay positive and keep that positive thought in our head. But a person can only take so much negative words from others. When everybody is saying that you're fat and ugly, you cant really be smiling and feel pretty. It hurts and these words can bring you down.
So, be careful for every words that come out from your mouth. Stupid joke -as you said- can make someone/somebody feel bad about themselves. It can silently bring them down, with or without you realizing it. Like wise man said; think before saying anything.