I've not done it, but I'm going to weigh in anyway. For the reasons you've identified it's less than ideal. But it is practical and since you're very keen, I'd go for it.
When planning, don't forget you need to get to/from your Madrid hotel to the Atocha railway station. Similarly, Sevilla's main station where you arrive is some distance from the main sights - so I'd suggest a taxi.
But (in a very non-specific sense), if you left your hotel at around 0630 then you should be in the centre of Sevilla by about 1000. Then if you left central Sevilla at about 2100, then you'd be back in your hotel around 0030 the next morning. That assumes you use the first and last AVE (fast) train services of the day.
So that gives you time to visit the Alcazar (definitely pre-book), walk around Santa Cruz and have lunch, visit the Cathedral and walk near the Plaza Nueva area and have tapas before heading home. Even though there is much else to see, I wouldn't stray much further than these suggestions in the time you have available.
Sevilla is Spain's best city - and the 3rd best in Europe - so I suspect the main outcome from your day-trip will be a longing to return. What you plan wouldn't be my choice, but if you want to do it then definitely it is practical.