To make one hat match several different outfits, all you need to do is to be able to change out your different color flowers.
Then use a yarn needle and some thread or yarn and sew on to your hat a button or two where you would like your flower or other embellishment to be.
Gently slip your flower(s) over your buttons on your hat. Change out flowers to match the outfit.
Perfect and Simple!