I'd plan a day trip but not necessarily go to Kotor. Depends how much time you truly have in Dubrovnik itself. You could do something quicker like a day trip to Lokrum Island or a bus (or ferry?) to the town of Cavtat (neither of which I've done myself).
Yes, if you go to Kotor, rent a car and drive yourself so you can stop long the bay on the way to Kotor and take pictures - you can, and you'll want to. (Hint: there's a detour through Konfin at the border to Montenegro that could be less busy - tour buses can't use it, so no line of traffic probably. If not a busy time of year, maybe not worth worrying about.) Driving in Croatia is pretty easy; less fun driving in Dubrovnik itself. You could also hire a guide to drive you I guess; might cost more obviously but if you can stop as you drive...
You can take a guided day tour or a public bus or something, but it just wouldn't be the same as driving in my opinion. This is a case of "the journey is the destination." Kotor itself is OK, not as nice as Dubrovnik. The view above from the "walls" (a very steep climb up above town) is breathtaking and a must-do if you are in shape for it. "Walking the walls" in Dubrovnik (also very worth doing) is a completely different experience - truly you are walking around Dubrovnik vs. climbing up above it.
Stop in little Perast on the drive over to Kotor.