People drink beer to have fun and nobody calls that selfish. People play games and sports with one another to have fun and nobody calls that selfish. But when two persons have sex with one another just to have fun many people call that selfish. Does that make any sense?
No. There isn't anything intrinsically selfish about sharing fun a deux, whether it is singing duets, riding a tandem, or sex. Of course, the sex-for-fun might be cheating (if you are in a relationship) or unwise (if you don't know where s/he has been) or against professional ethics (if you are a doctor, s/he your patient) or a bad idea for other reasons. But then again, it very likely will be just fine.