Best of Istrian Peninsula

Besides Rovinj you could drive to Porec. There is a beautiful cathedral and nice restaurants along the water.
Don’t bother driving to Opatija. Back in the 70’s and early 80’s it was a lovely little coastal town. Since they built it up and added nightclubs and casinos it isn’t worth the drive. We have an apt in Rijeka (here now) which is just a short, city bus ride away and we don’t go any longer.

Kanenjak peninsula. The very tip of Istria and the famous and unusual Safari bar. Not far from Pula. Easy drive. Many nice beaches.

Motovun. Hilltop town with views but congested. Maybe not worth the drive.
Rovinj, nice but also congested with parking problems.

I'll second skipping Opatija. We stayed there as a stopover one night and it was one night too many. Crowded, noisy bars and casinos. Not our preference. We enjoyed Buzet and Groznjan. Both were lovely, quiet villages, and we had a couple nice lunches (but of course I can't remember the names of the restaurants). Personally, I think Motovun is very highly overrated. Yes, it has nice views, but it's a scrum to get to them. Busloads of people, and the town is boring. I was there once and didn't understand why everyone thought that it's so great. So, I went back a couple years later. If anything, it was worse. Rovinj is a parking challenge, no doubt, but we really enjoyed it. For 3 days, we just wandered the town, took in the sights and ate. Since you've only got 3 days, I'd balance a day in Rovinj with a few villages like Buzet and Groznjan. There are others, too.

Agree about Motovun too. Croatia doesn’t have too many specificly described hill towns, like Italy, because practically every city along the coast is hilly. I know, Split and Zadar are flat. Anyway, it was a big bore for us too. Half the town was up for sale when we were there 2 summers ago. I doubt much has changed.

First off, enjoy Pula and get out on the water. If you have time, Motovun and Rovinj are musts.

For an under-the-radar, low tourist pressure spot, spend a few hours in Bale.

Try to visit Grožnjan instead. It is further north but worth the trip. We’ll be visiting there within the next 2 weeks.

Do a loop trip up to Motovun, but find Oportalj as well. Like Grosznjan it is small and beautiful and even has a taverna /konoba for lunch under the chestnut tree looking down the valley. Motovun is much more developed.
Another day trip from Pula could take in Rovinj on the west coast. Very walkable if you stay at harbour level, and there are winding ways up to the cathedral of St Eufemia fro views and prayer. Get a bit lost in the old town for an hour or so. Cafes in all directions on the main square and waterfront as well as side streets. Rovinj to Pula is straightforward and well paved.
Any tourist office will give you maps for gastro and vino tours but be careful not to drink and drive.
Do avoid Opatija ad the busy east coast.
Have a coffee in Pula with James Joyce. His statue sits at a corner table. He was much younger and crazier when he and Nora stopped in Pulato teach English for Berlitz and write his Dubliners stories. The statue is an epic bronze good for photos.
The amphitheatre and its underground museum are worth the entry fee. Still remarkable well preserved, as probably you are too. Happy travels in Istria. Ciao!

Thank you to everyone for the great answers. There is nothing like first-hand advice. Looks like we have plenty to do in Istria,

An update - we visited Groznjan and Oprtalj a few days ago. Close enough to visit both.

Groznjan - so many small art shops with the artists creating every day. I believe they all moved here from Motovun, lol.

Oprtalj - very deserted town but worth the drive for the views and the restaurant mentioned above - Konoba Oprtalj. Can’t miss it as it is the first place you see walking into the town after the school building and parking lot. The food was beyond fabulous. We had the meat (it could be veal, beef, or pork depending on what they purchased) cooked under the Peka, which is a ceramic domed cooker they put right on the grill. Also, the homemade pasta was so delicious, especially the tagliatelle. It is a husband and wife operation and the place was packed.

The best part about these two hilltowns is that you drive up and there is ample parking - small fee in Groznjan, free in Oprtalj. No shlepping up a massive hill side. Definitely better places to visit than Motovun. An added bonus, neither place was crowded, except for the restaurant.

I second the opinion of visiting Bale. We are here now and it is really an interesting little town with some great walking, and old hill top castle. Not a lot of museums but a good place to take a break to see an old old village