On early season commutes, most of my load going in is lunch. Most of it coming home is the layers I've shed because the weather warmed up.
Not complaining, just observing.
Maybe I'm faster in mid-season not because I'm in better shape but because I don't have all that laundry on the home-bound leg. That means less weight and less wind resistance.
My commute brackets the average work day, so at the requisite times of year I start around sunrise and finish at dusk. The morning is always chilly, except in the height of summer. The evening segment could be pleasantly tropical or bring on the sharp chill of a cold night ahead. Usually a chilly evening will still start out warmer than the morning preceding it.
A scheduled ride like a commute makes you deal with conditions as you find them, because you can't pick the best part of the day. It feels really good when the variations of weather fit neatly around ride time. It makes up a little for the times when challenging conditions line up to make things harder.