Well, these are the most famous outdoor gardens in the Loire. If you're never going to be there again, and if the predicted (!) outdoor weather isn't going to be miserably cold and windy, I'd consider getting a look at them just the same.
You didn't mention what the indoor(?) alternatives might be. We went to the Trogodyte caves, for example, but Villandy was much better-and believe it or not, less "touristy". And the chateau interior tour (is it open in the winter?) will get you out of the open air, if not necessarily "toasty". OTOH, Abbaye de Fontevraud has lots indoors, but it's going to be drafty, and maybe just a bit out of the way. Again, there's a big property to walk around in, if the weather turns out to be bearable.
I'd hate to miss a glimpse of the exteriors of these iconic buildings. There's a lot to see in the Loire, but it's often outdoors. I suggest you look at Rick's "France", and check online temperature and weather records.