I would stay in Rutland. There are some very pretty historical towns such as Oakham and Uppingham and Stamford is only 10 miles away (but in Lincolnshire not Rutland).
If you feel energetic, you can hire a bike and cycle round Rutland Water (about 12 miles and very easy going) or if you are there May to September, take a boat ride on the Rutland Belle (weekends only late March and April). Lots of smaller pretty stone villages and great tea shops!
The museum in Oakham is worth a visit plus see the Castle there - free entry and known for its collection of horse shoes (more interesting than it sounds as many have been given by royalty going back hundreds of years).