Driving vs public transportation

How many people in your trip? Have you checked airfare between Milan - Vienna? I checked June and it is $115 round trip direct per person. You can travel by train to Swiss from Milan very easily. You will not travel each day of your trip, so perhaps map out the days and then check sbb.ch for ticket prices. If you don't want to travel with all of your luggage in Swiss, there are storage lockers at most larger train stations.

The car rental is that high because you're picking up and dropping off in different countries. If you really want a car for a part of the trip, try to keep it within one country. Are there particular villages in your plan that you know would be easier to drive to than to use public transport?

Hard for me to guess which rail pass you're looking at without knowing the number of travelers. Since you mention 17 days, it's likely that you're looking at prices for a Eurail Global pass, when a 2-country Select pass for Switzerland and Austria might be a better fit, and offers a cheaper 2nd-class version. Compare Swiss options at https://www.ricksteves.com/travel-tips/transportation/trains/switzerland-rail-passes. Italian trains between Milan or Novara and the border are cheap enough that you can leave them out of the equation. Also, most people don't want to be on a train (or on the road) every day of their trip. Just count the days that you would move between towns.

Most people no longer recommend getting a Eurail pass as a default. It can be cheaper to buy point to point tickets ahead of time but you will lose flexibility as you are restricted to a certain date and time. Switzerland sells some of their own passes but it really depends on where you are going and for how long to determine if those are worthwhile. Switzerland in general is one of the most expensive countries to visit.

Until you provide more info about your trip - an itinerary - it's difficult to give you advice. Renting a car in one country and dropping it in another always incurs a hefty drop off fee. If you were to rent a car in Italy, you could drop it at the rail station in Milan and take a 2 1/2 hr train trip to Lugano where you could rent a car to use in Switzerland. I don't from whom you received the auto rental quotes but I would call AutoEurope on their toll free line (1-888-223-5555) and speak with one of their customer service associates. They are knowledgeable and extremely helpful.

As far as a rail pass goes, rail travel in Italy is extremely affordable and, generally, less expensive than using a rail pass. Though I have never used one in either Switzerland or Austria, the Swiss passes may be beneficial. Note that there are extra fees if you want to take trains like the Glacier Express. You also only need a pass only for your actual travel days, not the length of time from first trip to last. And then it might be cheaper to by point to point tickets for short hops that you may take, saving the passes for longer journeys.

My initial thoughts are that you've chosen to make so many stops for so few days in Europe. And many of the places you've chosen are not commonly visited--not that that means anything.
By now, you realize that dropping cars in other countries is not economical as the rental company must pay to have the car taken back to its home country. Most people pickup and drop cars in the same country even if it means catching a train or budget airline to the next country.
You should consider making fewer stops. The easiest way would be to skip Switzerland--a very expensive country to travel through and to stay in.
When we want to enjoy mountain vistas, our time's spent in the Dolomites of Northern Italy and Western Tirol in Austria. You simply cannot find a more beautiful place than the Grossglockner High Alpine highway north of Lienz, Austria--south of Zell-am-See. We also enjoy Salzburg when in the area.
I don't know if you've traveled Northern Italy before, but you've deleted the most popular stops from your itinerary.

Thanks for everyone's help!

As for my itinerary, I specifically chose locations that are not overly popular because I enjoy going to places with less tourists. I also enjoy spending time in non-major cities so there's only a couple big cities included in my trip. And it's because it's easier to fly in and out of big cities.