I have a mast cell disorder (the cells that release histamines) and one of my triggers is peppercorns. I am a total wimp when it comes to black pepper especially. My lips swell up and such. I avoid it like the plague.
Conversely, cayenne pepper is totally fine and I use it in place of peppercorns when cooking.