Worth the trip?

Yes...Norway can be quite expensive. We were there for a month last summer and had to be very careful to not completely overextend our cash! We found that grocery stores were not that much more expensive than those at home so that might be a strategy to keep your prices down. Stay away from alcohol...it is crazy expensive.
Norway is worth your time though! The challenge is that although things aren't terribly far apart, the topography makes it challenging to get from one place to another in a timely and cheap way. You should focus on small areas of the country...your Bergen plan is perfect but consider only spending one day in the city itself. There really are only a few things to see and one day is plenty IMHO. Take the two days and see the fjords and mountains in the area...they really are stunning.
Other pockets of the country that are worth spending time in are the Alesund area and the region around the Lofoten Islands, much further to the north. Both of these areas have several days worth of things to see and do. Tromso is a popular destination in the dark of winter as many people go here to view the northern lights.

There are maybe one or two places in Europe that are more expensive than Norway. I'd go for 3-4 days now rather than think about a longer stay later - when it won't be any cheaper. Unless you expect to come into an inheritance in the meantime.

I agree with Anita, you can see the sights of Bergen in a day, then a 2 day cruise (if there is one) to the fjords.

Weather may be more favorable the closer you are to the end of your trip.

Scandinavian hostels are among the nicest and camping cabins can be another budget option along the fjords (but a better deal with more than one person).