Tivoli is closed in February.
For the four year old, there is a really neat museum/center setup on Paper Island (Papiroen) right along one of the canals called Experimentarium City. It's a science center with games and stuff to do, fun for both kids and adults (my wife and I are in our 30s and both enjoyed it). Right next door is a food hall with tons of different trucks and vendors, it was delicious. You can go back and forth between the two. Experimentarium City isn't cheap if I recall though, but we had the Copenhagen Card so it was covered.
The National Museum (very close to Tivoli, and free) was excellent. Would highly recommend that.
I'm not sure if it would hold a four year old's attention, but we really enjoyed the Ny Carlsberg Glyptotek, an art museum very close to Tivoli. Be sure to see the Rosenborg Slot and the Little Mermaid statute (the latter is tiny and touristy, but seemingly every visitor to Copenhagen goes to see her).
For day trips, consider: (1) Helsingor (with its famous Krongborg Castle (Hamlet) and a great maritime museum, right along the water across from Sweden), also the Louisiana Museum for Modern Art in Humlebaek on your way back to Copenhagen; (2) Roskilde (truly awesome viking ship museum as well as a UNESCO World Heritage cathedral where Danish monarchs are buried); or (3) Hillerod (Fredericksborg Castle, lovely little old town).
Not sure how long you will be there for. With the little one, I think Roskilde might be the best day trip. The viking ship museum would be great for a kid.