A few weeks ago I stayed for a few nights in Ingolstadt, which was surprisingly attractive. Beautiful Cathedral of Our Lady. Assam Church (like that in Munich, but without the visitors). Well preserved historical centre to wander around. And a good base for some attractions around. Neuburg (15 mins away) is a lovely tiny town with ainteresting Renaissance palace (schloß) with a good picture collection and a magnificent chapel. Eichstätt (45 mins, including a 9-minute journey on what with 5 kilometers must be one of the world's shortest railway lines) is a small town, which was the residence of prince-bishops, with again a beautiful cathedral and some peaceful baroque squares. It's all very low key, and most agreeable in a very German way, and almost totally devoid of foreign tourists.
The train from Lucerne takes around 6 hrs and it is 2,5 to Passau, which is hardly more than a direct Lucerne to Passau train.