I have a very different take on Avignon than most of the comments above. Our family really loved our time there.
I will say that if what is in your mind's eye for Provence is fields of lavender and the warm hues of sandstone at golden hour, Avignon will not line up. It is more medieval in character. A bit severe, but not less interesting in my view.
Regarding the Pope's Palace, I concede that it's not the best if you are looking for art and decor. But I felt both the architecture and history were worth the time spent exploring. For us it was relatively more engaging than say, the arena in Arles (which we also enjoyed). It's a unique window into the distant past. There are plenty of interior images online. If they look compelling to you, I wouldn't hesitate to include it in your itinerary.
The park across the bridge can be a great place for photography, although if on foot the walk there is not so magical. You will be alongside a busy car bridge. I believe there is also a ferry which may be a nicer way to cross. If this is not an issue for you, my recommendation is to wait for nice light, then head NE of Le Pont to compress the scene.
I will also throw in that our accommodations were one of the most special and characterful places that we have ever stayed. It was a medieval church that was converted into a luxury apartment, right around the corner form a charming square with a bakery and cafe for morning coffee.