Inverness is generally considered by most to be a jumping off point for nearby adventures (e.g., Culloden battlefield right outside of town; Loch Ness a short drive away; Fort George is an interesting historical site basically in town; etc.) rather than a must-see place itself.
However, we found Inverness to be charming and well worth spending an afternoon or evening in town, as the River Ness has some lovely paths along its banks which are very near the busy restaurants and hotels in the center of Inverness. It might lack a must-see-sight in the city itself and probably doesn't have as much polished charm as tiny Pitlochry, but I think it is well worth a short visit of a day/night. You could spend part of your day at Culloden or Loch Ness, and have an evening back in Inverness - and still keep your toddler happy. A car makes those trips out of town easier of course.
While I can't be much more help with your other questions, you really can't go wrong - Scotland is wonderful. Enjoy!