Divorce due to fornication?

Matthew 5:32 KJV

[32] But I say unto you, That whosoever shall put away his wife, saving for the cause of fornication, causeth her to commit adultery: and whosoever shall marry her that is divorced committeth adultery.

Matthew 5:28 KJV

[28] But I say unto you, That whosoever looketh on a woman to lust after her hath committed adultery with her already in his heart.

What man has not fornicated by looking on a woman to lust after her? Most men have probably done this hundreds of times.