I feel like I can’t speak up for myself when people talk crap about me or even mild criticism

This mostly happens at work(work construction) Im 25 male, consider myself a pretty sensitive person . I take things very literally at times. The trash talk I’ll normally hear is, how Im a female, a little man, suck dick etc. which I normally just laugh off because I don’t know how to respond to it or give a comeback to it. I figure they just gotta get it out of their system or something for whatever reason. Or they’re just come up to me and be like “I hate you” the tone I can tell he’s just trying to fuck with me but in the back of my mind Im still like, damn does he really hate me? I don’t know why I let small shit like that effect me so much 😅 anyone else have any advice? I sometimes want to talk smack back but I hesitate to because I don’t want them thinking I actually have beef with them or I come off too serious or something if that makes sense