We spent a little over 3 weeks in Norway this summer on a home exchange and yes, it is quite expensive. If your rent an apartment and cook for yourself then that will cut your costs considerably. Food at the grocery store is comparable to what I pay here in southern CA. Eating in restaurants is a whole different story...we never got out of a meal for our family of three for less than about $60-70.
For the amount of time you want to spend I would allot 3 days (4 nights) to Oslo - it's one of my favorite cities in Europe...very warm, inviting and low-key with plenty to see and do.
Take the train from Oslo to Bergen...amazing trip!
Bergen really doesn't warrant more than 1 day (2 nights) IMHO - after you've seen the old Brygge buildings along the wharf with all of their touristy shops, walked around town, and taken a ride up the funicular, there isn't a lot left to do in town. There are a couple of historic homes to see outside just of town, most notably, Edvard Grieg's house. If you like to hike then there are many trails from the top of the funicular with some pretty spectacular views at the top!
Most of our time this summer - about 3 weeks - was spent in Alesund. What a spectacular area! The town itself is small - you can spend one full day exploring. The town is easily walkable and is Art Deco in style...the whole place burned down in the early 1900s and was rebuilt. Nice shops, antiquing, restaurants, and a great view from the top of the funicular. There is a beautiful little outdoor folk museum just outside of town on the fjord with old buildings from that part of Norway that have been brought in and reconstructed, as well as Viking ships that have been dug out of the ground locally.
The Alesund area has a lot to offer. There are many fjords nearby including the Geirangerfjord - spectacular! The drive from the town of Geiranger over the Trollstigen Pass road is not to be missed with it's spectacular cataracts, waterfalls, and the pass itself that has a modernist designed lookout and visitor complex that is, by itself, worth seeing.
Runde Island is one of the largest bird sanctuaries in Europe - windswept and beautiful. You can hike all over it or take a boat around to see the birds and cliffs.
There are many other small islands to see in the area, most of which are connected to the mainland by bridges or tunnels. There are farms, small churches (don't expect them to be open that early in the year), beautiful boathouses on the water, lighthouses - lots of opportunity for photography if you like to take pictures.
Where there aren't bridges or tunnels, an excellent car ferry system connects the whole area so that you don't spend too much time driving around the huge bodies of water that are everywhere. Most of the ferries are used by the locals so they run year-round - a few, like the Geirangerfjord ferry, are seasonal, tourist ferries, so you would want to check to see if they are running in late May-early June.
Besides those things, the hiking and skiing in that area is also very good. You might just catch the tail-end of the ski season. The Norweigians hike even in the snow...they take their skis and just head out all over the place, taking along little cookers and making pancakes in the woods. The certainly know how to embrace their environment!
Now that I've said all of that my only worry is that you are traveling early in the season so you will run into snow and rain - even in summer there are just as many rainy days as sunny days! It won't be warm.