If you are staying in Bayeux why not take the train to Bayeux rather than Caen? That price seems very high but again, it depends on the day you are traveling. I booked a ticket from Paris (Gare St Lazare) direct to Bayeux for June 2 for only 9 euros. The trip will take 2 hours 47 minutes. Look on the official French rail site. Oui.sncf or trainline. The schedule and the price will be the same on both. Again, not sure when you are traveling but if you can wait a while you may want to check daily to see if there are additional trains. When I first started looking prices were very high and there were no direct trains. I held off and all of a sudden one day there were many more choices! If you are traveling soon that may not be an option:(
If you decide to take a taxi you also need to consider space for 4 people plus driver and luggage for 4 could possibly be an issue. Someone elso on the forum may be able to speak to that. Also, while the taxi ride may be 3 hours or so without traffic, it can be much longer if you are traveling during busy times.
Good luck with your plans!!