I think that's a very good idea. First of all: if you like architecture you will love Görlitz. The town was never destroyed (even the synagogue survived), and the different architectural styles surround the centre like onion rings, from Gothic to Commie architecture. There are quite a few cool hotels in preserved Renaissance or Baroque buildings, like Tuchmacher, Börse or Frenzelhof. The cuisine is also unique. Görlitz belonged to Silesia for some time, and after 1945 many expelled Silesians settled there. Today it's the only place where you can get authentic Silesian food, which is largely forgotten in Polish Silesia now. Or you can cross the border and try Polish food if you like. Speaking of Silesia: there is also the Silesian museum, a state musem run by the government and the state of Saxony which keeps a largely forgotten world alive.
I think you can easily spend a full day in Görlitz. Explore the old town, and don't forget the Nikolai quarter a bit outside with the Baroque cemetery and the Holy Sepulchre, climb up the Reichenbach tower, visit a museum and a restaurant (like the Dreibeiniger Hund = three-legged dog), see the huge Gründerzeit quarters, or maybe walk to the former Upper Lusatian hall of fame, which looks like the Reichstag in Berlin, on the now Polish side.
You could actually spend a lot more time there, since Upper Lusatia, the historic region around Görlitz, is very unique and beautiful. You could, for instance, rent a bike and follow the Neisse river to the south until you reach the baroque monastery of St. Marienthal. Very popular among the locals. One hour away from Görlitz is Bautzen, the historic capital of Upper Lusatia and the Sorbs, a Slavic minority. Bautzen offers a beautiful baroque old town surrounded by massive medieval fortifications. One hour to the north is the park of Bad Muskau, which is a World Heritage Site. One hour to the south is Zittau, a mostly preserved town with two huge and unique medieval lenten veils. From Zittau you can take the nostalgic steam train to Oybin in the Zittau mountains. In Oybin you can visit the romantic ruins of a huge castle, planned as the old age residence of emperor Charles IV, on a mountain that looks like a beehive. And with a car you could see some of the cute villages in the area, which offer some very unique timber-framed houses (called Umgebinde) that exist only in this part of Germany.
Here is a nice gallery of this area (Görlitz and many places I mentioned):
http://www.skyscrapercity.com/showthread.php?t=869852
Umgebinde houses:
http://en.wikipedia.org/wiki/UpperLusatianhouse
And a pic of St. Marienthal:
http://media.belocal.de/127616/2000x1339_0,0c.jpg