This is an excellent point. The judgement lies with the individual absolutely. But when the two people got married they both agreed, often before their God, that they would honour, love, respect and be FAITHFUL to each other. So when one of those spouses steps out, they are breaking those vows and dishonouring themselves, their spouses and frankly the person they are cheating with.
The single person in the equation (the Other) lives with their own value system as you have mentioned. But the couple have agreed on a value system and have made the agreement legal.
If we all decided to have just a free for all in all aspects of our lives, not just sex, then imagine the state, country world we would live in. I personally do not want to live in a free-for-all because I need to know that certain things that I hold sacred will be respected by those I bring close to me. I couldn't live happily by watching my back because someone is eyeballing my stuff, my body, or my spouse and will take it because they don't share the same morality as I have.