if you like to sketch at all, go to the Musee Rodin. I could probably spend three days there sketching - and they have convenient chairs and benches to sit on. I was not the only one sketching by any means!
For walking and taking in the culture try walking from Musee d'Orsay to the Musee Rodin and getting lost, or (at least during the day) walking down from Gare du Nord via Boulevard du Magenta to Rue du Faubourg St Denis until you hit the big arch - tons of street grocers and middle eastern/north african restaurants and cafes, plus right by the arch is a great place to get a petit dejeuner for a decent price (yes they have menus in all different languages, but seriously it's a good price and has a nice atmosphere) and a metro stop to take you to your next location, with a view of an awesome giant arch out the window (Porte Saint-Denis is its name, erected in 1672, if you were wondering).
Also if you like hiking try exploring Montmartre on foot - get away from the Sacre Coeur and metro stops and walk into a random bakery and you can get an amazing pastry along with the locals getting their daily loaf of bread.
I have no suggestions for your anniversary, but happy anniversary anyway!