Does this seem like a reasonable Paris itinerary or add a day?

Day three add Luxembourg Gardens and possibly the Cluny medieval museum as both are in area .

You shouldn’t drag suitcases to restaurants so check into hotel first on day one then go for dinner .

We will be going as minimal as possible so only backpacks

Have you been to a restaurant in Paris before ? The tables are often on top of each other , your backpacks will be in the way unless you plan a burger at a fast food place type meal stop .

So yes, the suitcases will be an issue...check in to your hotel first then go to dinner.
Day 2 will be tight and you won't get to St Chapelle by 3.
Luckily you have unplanned time built in so you can possibly do one of the Day 2 things, then.
Day 3 - the Pantheon (if you also go to the crypt you won't spend more than an hour there) is just up the street from Luxembourg Gardens. Highly recommend taking a stroll there. Just head straight down the hill and across the street.
Have lunch at Le Procope...about a 5 - 10 minute walk from the gardens - classic Parisian restaurant with many literary connections.
After that you can walk towards the Seine through St. Germain.
With older kids I would suggest...
Deyrolle - over the top taxidermy shop that's been there for 150+ years. Go upstairs.
Musee des Artes et Metiers - amazing science / technology museum
Paris catacombs
If it's summer, spend time walking along the "Paris plage", along the quay of the Seine.
Pompidou Museum - modern art; my son loved it there when he was an older kid.
Petite Palais - the free municipal museum of the city of Paris. Amazing collection both upstairs and downstairs. Great garden cafe to have a coffee and take a break.

OK so I guess check into hotel first then head out to get dinner and maybe do some stuff. My daughter doesn't like dead/creepy things so we will likely skip the crypt and catacombs however my son would love Deyrolle so we might hit that up, looks fun. Maybe we push St. Chapelle into day 3?

Any suggestions around the Eiffel Tower for dinner that is local hole in the wall food not tourist food that you can see the tower from decently?

If you can see the ET it will be filled with other tourists who also want to see the ET.

I havent eaten there yet.. but often see Les Ombres well recommended.. and yes.. with a nice view.. but may be more than you wish to spend.. you can look on website and decide anyways.

There are several nice restaurants on rue St. Dominique, not far from the Eiffel Tower, though they do not have views of it. The area directly around the ET is residential, not filled with cafes. However if you cross the river to the Trocadero area, I recommend Carette tea room, for tea, pasties and light lunches. Not sure if you can see the Tower from their outside seating, but perhaps you can. There are other cafes there also.

Day 3 - we didn’t find the Pantheon in Paris to be very exciting. Since you have kids with you, I might head up to Montmatre neighborhood for the morning. You could see the Cluny Museum in the afternoon with the Luxembourg Garden or go through the Opera House and also walk through the old-time covered shops. I definitely recommend ending that day with an evening cruise (not dinner cruise) on the Seine - amazing city lit up at night!

As others have said - Day 2 is too much. Arc de Triomphe -> Sainte-Chapelle isn't possible in your time frame. But I would advise against skipping Ste Chapelle. The stained glass is amazing. And if you have the Paris Museum Pass - that'll help get you in quicker.

As someone else mentioned - I'm not sure I'd include the Pantheon in this itinerary. I'd do the D'Orsay or the Rodin Museum well before the Pantheon. For something different - I'd recommend this: https://www.atelier-lumieres.com/ My wife and I visited this in December for a Klimt-themed production. And we're going back next month for the Van Gogh one. And if you find something to do in the morning of Day 3 (or stay with the Pantheon) - I echo what someone else said about at least going up to Montmartre. Plenty of lunch spots up there and you can see Sacre Coeur.

Day 4 - If the Arc de Triomphe is something you really want to see - I'd add it to this day since its not terribly far from the Eiffel Tower.

As for a restaurant near the Eiffel Tower - anything you get with a 'view' is likely going to be a cookie-cutter cafe. We've been at this restaurant twice (last time was New Year's Eve last year). We already have a reservation for our next visit to Paris next month.
The people are great and the food is wonderful. And its only about a 5 minute or less walk 'around the corner' to see the Tower. https://www.restaurant7emevin.fr/

And I'd add an extra day. Or 12. But that's just me. ;) Whatever you decide to do - just have a great time!! Don't stress out on having to 'see' everything. Enjoy just 'being' there too.

Regarding Day 2, go to St. Chapelle after Louvre Museum and lunch since St. Chapelle is less than a mile from Louvre Museum and a nice walk over the Seine onto the island where the church is. Save Arc de Triomphe and a stroll down the Champs Elysee for another unplanned afternoon. Maybe then visit the Conciergerie (we enjoyed it) when visiting St. Chapelle. Musee de Cluny might be nice to see stained glass up close and check out the tapestries.

I decided to add a day to spread things out some and booked the hotel, now I just need to fill in the gaps with what we want to do along with some of these suggestions.

I'm in the skip-the-Pantheon camp. I went mainly for the rooftop tour which was very good and there are great views. If it's available, do it. For several years they closed the rooftop to tourists.

The Louvre is huge. Plan in advance what you want to see. The audio guides are very good. You need to leave a photo ID (driver's license is fine) as a deposit. Remember where you took it from (there's a desk at each of the 3 gallery entrances) so you know where to return it. Look at the website's "Thematic Trails" for self-guided tours.

Take a 1-hour Seine cruise with Les Vedettes du Pont Neuf - it's a short walk from Sainte-Chapelle/Conciergerie. You can save a few euros by buying anytime tickets on their website. The best time is right around sunset.

Look at the Paris Walks schedule. Most of the guides are quite good.