We made this mistake too for the longest time. The towels would also never full dry which left a mildew stink on them... I was constantly blamed for leaving the towels in the dryer too long UNTIL we figured out that fabric softener does not go on the towels.
Use vinegar.