Un-padded lycra for cycling, should I care ?

After getting in good shape from the gym and aerobics, I started taking my cycling a bit more seriously last month, and now am up to around 40-50 mile rides twice a week. However, I had been wearing standard fabric shorts for the first three weeks, I finally gave in after reading so many cycling blogs about lycra shorts, although when I tried the padded ones, I found them uncomfortable, then I tried the ones with a camos I think it's called, and also found they chaffed. So now I only wear un-padded normal lycra shorts. Now for the obvious issue, I soon realised that i could not wear anything under these shorts either and this has presented the problem of certain things showing more so than before. Now I don't mind my friends and family having a giggle when they see me wearing the shorts as they can deal with it (I hope), but it's the ever increasing looks I get from strangers. I only cycle either on the road or on trails/dirt track etc...so I thought that there would be no real issue of anyone noticing, or being offended as I am above average endowed and it does show wether I am seated or standing up while cycling. Should I let this bother me, or just keep up what I have achived.