We went to Avignon yesterday driving from Saint Remy de Provence.
We visited the Pope's Palace and the "Pont de Avignon". It takes approximately two hours. After that we took the little train that leaves from the Pope's Palace every half an hour; it takes 45 minutes and gives you a nice view of the city. The cost of the ride is 9 euros per person. We feel that we covered what we wanted to see in the city. It was a nice experience! We arrived in Avignon by 11:00 AM and left at 5:00 PM. I have a rental car and have no information about public transportation. I understand that there are some good train connections to different villages in the area.
If it helps, we also visited Arles. At the Tourist office you can get a ticked for a tour on a taxi bike for two people that takes one hour, and covers the nice city sites including Van Gogh information(55 euros). After you get a general idea of the city you can visit any specific sites you like. I was planning to use the walking tour Rick has in his Guide but it was hot and my wife preferred to take the taxi bike. We enjoyed it. If you don't have too much time this is a good option..