Are you trying to do both in the morning, before lunch? Rosenborg - it depends on how long you like to linger, especially around the gardens. The castle itself, though absolutely worth it, is not that large. I don't think we spent more than maybe an hour at the castle, and then walked around the gardens a bit. The National Museum - also great - is at least 2 hours, I think maybe closer to three, and again, it depends on how long you spend at the various exhibits. The two are about a 20 minute walk apart.
We were there in early October last year and did not wait for tickets at either. Note also that if you purchase the Copenhagen Card, admission to both is free. Whether the Card is worth it depends on your full itinerary (and especially if you are going to be using a lot of public transportation around Zealand).
Also - if it's a nice day, the Kongens Have (the gardens and park outside of Rosenborg) makes for a terrific place to have a picnic lunch.