the fragrant perfume of the flowers can convince the speaker to tell a "summer's story" (line 7). That is, the speaker has no desire to speak in a light or entertaining or humorous way. He is too heavy, too sad.
In the third quatrain the speaker looks at the beauty of two flowers, the lily and the rose; but their beauty only reminds him of the greater beauty of the young man. The speaker states such flowers are "figures of delight drawn after you [the young man]." In other words, the flowers are a lesser symbol of beauty that attempt to emulate or copy the greater beauty of the young man. For the speaker, such lesser beauty has no appeal for him. His thoughts are on greater beauty. His thoughts are on the young man.
In the couplet, the speaker returns to a concept that he introduced in the earlier sonnet: even though the season is actually a mild and pleasant one, it seems like winter to him. The speaker compares (in a simile) the beauties of springtime to the shadow of the young man. The shadow of the young man is a dark and insubstantial copy of the young man's beauty. Thus, the speaker is implying that the beauties of nature and springtime are dark and insubstantial: they are cold and meaningless to the speaker. But those are all he has to look at or "play" with. In other words, he cannot play. And he cannot feel any joy.