I loved Morocco and hope to return and see more. I'd have no hesitancy in going with children of that age. How much time will you have?
I haven't visited the coast except in transit to Spain so can't help there.
My favorite cities were Fes and Marrakech, because they felt the most exotic to me. Rabat, Meknes, Taroudant and Ouarzazate were also interesting. You'll have to travel through Casablanca since it's the international gateway, but you needn't spend any time in that city.
My trip was a long time ago, and I am not up to date on transportation options, but I think moving around will be a challenge for you. Since you are four, I wonder about the possibility of a private tour to avoid some of the logistical issues, or perhaps multiple tours covering different parts of the country. It might be very affordable.
My mother and I enjoyed our very local experiences on the buses (some of which were very ramshackle back around 1990) and the "grand taxis", which are large cars used for unscheduled shared rides between towns. Those were not fast ways to get around, however.
One thing I believe you will observe is that any time you are not accompanied by a guide, you will be approached by young males (some looking under 15), wanting to be your guide. Life will be a lot easier if you agree on a fee and say "Yes", because that will stop the others from continually approaching you.