Yes, its crazy but so what. Why do you feel that you have to leave around noon on the train? There are trains departing Rome Termini for Assisi as early as 5:50am that will get you to Assisi at 8:55am 2.5hr trip). On this run, you hit Spoleto at 7:17am. A bit early to be sure. But, on the R trains, you can hop off and hop back on a later train. The later train could be an issue because it could come through 2hrs later. The next train out of Rome departs at 7:55am with an arrival time in Assisi of 10:16am. CLICK HERE to see the morning runs. These train runs do not stop at Orvieto.
If you arrive on the train in Assisi, you are still some distance from the Assisi hill town. You go next door to the little shop and buy a round-trip ticket on the Line C bus. I suggest that you take it to the end of the line which is the top of the town. Assisi streets can be quite steep. Starting at the top, you easily walk down to the Basilica. Tour there, catch the bus back to the train station and train back to Rome.
Bus runs from Rome to Assisi are limited. The train runs are more flexible. CLICK HERE to see the afternoon and evening runs. You can see the fares as well.
Note that most nice restaurants in Assisi close from 4-7.