I started riding in just shorts and a shirt, first cheap kit was worlds better. It's been developed for that purpose for over a century, they know what they're doing.I'm pretty certain this is why most people wear it.
Don't discount the importance of the ass to saddle interface. On a properly fitted road bike all of the weight on the saddle is actually borne by two little bony "nubs" on the bottom of the pelvis onto the part of the saddle where the nose transitions into the body. A normal four-way seam will compress your taint. This is a bad thing.
The ischial tuberosities are what you're sitting on, not your ass.