It's all about how each partner is feeling, and how much energy they have available for sex performance. Ideally having couple of times a day would work for many men, that is once in the morning once at night. But, men can have sex many more times than 2–3 times in one day if so inclined. Also, Women can handle many more encounters, but they do get sore and tired after a while.
Orgasms are not a huge deciding factor when it comes to sex, especially good sex. Just because most men can't have multiple orgasms and most women can doesn't mean that women can automatically have more sex. So the definition of sex is not “everyone had an orgasm”.
I'd say that on average 2–3 times a day is about the maximum for both men and women. Just keep in mind that averages are basically worthless.
A guy has to actually get hard to have sex.. a woman plain and simple has a hole. it does not need to get hard.. maybe wet.. but you can take care of that. so they physically can have sex 24 hours a day.
Yes, there are limits. For men, there would come a point when they could no longer get an erection. There is also a point at which they can no longer reach orgasm. For many (most?) 20-something men these limits are well beyond 2 or three times a day.
There are limits for women too, even if it’s just physical exhaustion or being too sore to take any more. I suspect many would be tired of it before hitting a physical limit.
Last, men can learn to control their ejaculation - to last as long as they or their partner want, and can learn to have orgasms without ejaculation. No ejaculation, no refractory period, so you can have as many orgasms as you want.
The real question is why would anyone push themselves to those limits? Sex should be far more than having an orgasm. Yeah, that’s a lovely part of sex, but it’s not all there is. Great sex is about enjoying another person; it’s part of a larger relationship. Done well, once can be so good you don’t need or even want any more.
Some people will argue that quality over quantity is more important on how many times they should have sex. Personally 3 time’s a day is a good amount if I have nothing on and I’m horny. When I was 18 I was able to go 3–7 times a day but it was only quick cums. It’s better to have quality over quantity. Sex once per day is a good amount.