Germany, the Swiss Alps, or Austria in winter

Beilstein is very small and most of the tourist infrastructure is probably closed in the winter. And also not very close to the rest of the destinations you mentioned.

If you're interested in skiing and going to the Berner Oberland (although you'll pass dozens of other mountain resort areas along the way), stay in Grindelwald or Wengen. If you're just there for the scenery, go for Murren. Gimmelwald is pretty snowed in and lifeless in the winter.

If you fly into Munich, consider staying the first night at the amazing Therme Erding resort. It's a 15 minute taxi ride from the airport and would make a great first night stay to recover from jetlag...and for the whole honeymoon romance factor.

"We have already decided to visit Augsburg and Munich..."

You probably do want to stay with cities of some size. Augsburg isn't bad but it's an unusual choice. It's quite close to Munich, actually. Unless you have specific grounds for Augsburg (which you may have noticed lies on the Romantic Road but is really pretty unromantic,) I think I would look into some place a little more distant. Beilstein is distant but ridiculously tiny. If you're thinking "honeymoon," what about staying in a real castle? Not too far from Beilstein there are several castle-hotels.

Auf Schoenburg in the Rhine River town of OBERWESEL
Ehrenburg in the Mosel River town of Brodenbach

The villages in this area are pretty romantic too.
Braubach (Marksburg Castle is open year-round for tours.)
Bacharach
These and other similar towns are in the Middle Rhine Valley, a UNESCO World heritage site. There are also some larger cities in the area worth visiting or staying in - especially TRIER and MAINZ.

Closer to Munich: consider Burg Colmberg and BAMBERG.

We are very much adventuring types so visiting a castle(s) sounds fun but we would rather stay at a local BnB etc and explore the towns, countryside, and wilderness as it were. That is why the smaller towns appealed to us. As more of a quiet cultural getaway. However we dont want to be bored and snowed in the entire time. As for doing Munich and Augsburg, that will be done in tandem given that we have family and friends in Augsburg that we will visit.

"We are very much adventuring types so visiting a castle(s) sounds fun but we would rather stay at a local BnB etc and explore the towns, countryside, and wilderness as it were. That is why the smaller towns appealed to us."

I very much like Germany's small towns and BnB's so your idea sounds good to me in theory but small towns on the scale of Beilstein (population 145) or even Bacharach (2,000) Oberwesel (3,000) Cochem (5,000) or Boppard (15,000) in winter can be surprisingly quiet during the day and downright dull when evening settles in and it's just you and the TV in your room. You probably won't be snowed in in this temperate region, but days will still be quite cold and hikes in the woods and hills, though very scenic, will likely mean lots of mud, so go prepared for that. Don't expect the getaway to be "cultural" if that word means interacting with locals, which will be tricky in these small places. If you want to stay in this area for its castles and hiking, give it a go. Your best bet for a small town with a little more going on is Boppard, which doesn't seem like 15,000 because the population is scattered into neighboring settlements outside the main town. There should be a restaurant and bar or two open in the evenings. From Boppard you can visit the other villages very easily by train or via hiking trails like the Rhine Castle trail. The good winter thing about Boppard is that while it's a lovely town in a scenic spot on the river, it's also just 15 train minutes from Koblenz (100,000) where you will easily find things to do in the evenings if you wish.

Boppard BnB and other accommodations

BAMBERG at 70,000 is maybe a good compromise between small city and too tiny. You can hike there too. Lots of great brew pubs there, maybe a good place to interact with others.

explore the towns, countryside, and wilderness as it were.

Allow me to comment on this statement. Sounds like you want to do some hiking. If you really want to hike in the winter, you need to go with a different set of expectations. As Russ noted, except in the Alps, you can expect a better than even chance that temperatures will be a little above freezing, meaning that the ground will be quite muddy in spots. It never really dries out in this part of Europe from October until late spring or summer. So, prepared to get a little dirty.

Along with the ground, the atmosphere is usually very damp. Don't expect the brilliant spring and summer colors you see in travel videos, everything in the countryside mostly looks hazy and gray. And the sun sets at a rather early hour during the first two months of winter.

Some of the Alpine resorts maintain groomed winter walking trails. These might offer a more reliable hiking option at that time of year if you don't ski. I'm not sure about which resorts have these trails in Germany and Austria, but I know the Berner Oberland and Flumserberg in Switzerland offer them.

Russ mentioned Bamberg. The region to the south and east of this worthwhile town is known as "Franconian Switzerland (Fränkische Schweiz)" because apparently the eroded stone formations of the low mountains around here reminded Romantic era travelers of Switzerland. Looks nothing at all like any region of Switzerland that I've seen, but it is scenic. At least in the summer, if offers ample opportunities to explore the countryside, which is dotted with countless old villages and a fair sprinkling of castles. If you take up Russ's suggestion of Bamberg and the weather gives you good opportunity, you could do far worse than to spend a day wandering around.

When I lived in Germany, I hiked during all four seasons... but I knew the terrain, I had good maps, and I had ready access to a washing machine in my house after a long day of slogging through the mud. I'm not saying don't try hiking in the winter, but don't count on it as a given either. And definitely let the short-term weather forecast once you get on the ground be your guide, rather than trying to pinpoint an exact day months in advanced.

RE Tom's note: ...Some of the Alpine resorts maintain groomed winter walking trails.

Unlike Tom, my Germany & Austria winter experience is pretty limited. We're XC skers but did observe an extensive network of winter walking tracks in the PillerseeTal five village area of Austira's Tyrol. The town of Fieberbrunn in the PillerseeTal is about a 2 hour regional train ride from Salzburg. The five villages are served by a free local bus. We didn't have or need a car to ski the PillerseeTal. The area actually groomed more kilometers of walking track than ski track. The tracks are packed firm enough that walkers seem to walk in regular hiking boots, no snowshoes needed. Many walkers did use poles.

http://www.kitzbueheler-alpen.com/en/pillerseetal/winter/winter-hiking-snow-shoe-hiking.html

Winter hiking and snow-shoe hiking in PillerseeTal

On around 100 km of prepared winter hiking trails non-skiers and
occasional hikers are very well catered for. Countless routes take you
to the loveliest nooks in PillerseeTal through woodlands and across
fields, past farms, hamlets and streams. At the end of many of the
routes a warm snug in a cosy alpine inn, or snack station awaits.
Enjoy a romantic time in Fieberbrunn, Hochfilzen, St. Jakob in Haus,
St. Ulrich am Pillersee and Waidring, as thick snowflakes make for an
enchanting landscape in the Kitzbüheler Alpen.