It's not nice to be told we can't do something, or know we can't do some things other people can. It is, to be fairly obvious, limiting.
Anyone who has overload related to an illness or condition, will probably find in literature or be told by doctors not to do certain things. These are often numerous and the fun things we want to do, so it can seem horribly unfair. It can be avoiding bright lights, busy and noisy places, all kinds of tasty foods along with beer, wine and spirits, oh and coffee and chocolate and lots more. So that night out looks very different when all elements of a dinner date could cause migraine, anxiety, a Meniere's attack or trigger other things.
An additional problem to navigate is that sometimes 'it' is OK, but not always, and not in ways you can predict. So it becomes a gamble, to risk doing the normal thing you want to do, like meeting friends in a pub vs. staying at home isolating and insulating yourself. Not doing normal and social things has the potential to lead to more physical and mental health problems, so getting the balance right is the key; finding the limits yourself to keep physical and mental health optimal.
There are some jobs, friends or lovers and some environments that may no longer be compatible with these new limits to keep an optimal balance. Making significant life changes shouldn't be taken lightly and serious thought needs to be given, so responsibilities, loved ones and dependents aren't negatively affected. It can be hard to make different choices, but health and happiness can be enhanced and overload reduced when these elements are in balance.
So limits are not for others to impose on us, but for us to choose for ourselves and they needn't be limiting, just healthier, happier choices. There are some events and activities worth the gamble of migraine, anxiety or sensory/information overload, because the benefits outweigh the costs. The rest of the time we can make more choices to stay on the right side of our limits, because on the other side is overload or worse.