When a Magsafe is plugged into your Macbook, most likely it will run warmer. That's because it would be trying to charge the battery as well as run the laptop. The charging is not 100% efficient and creates some heat on its own.
However, if by “overheating” you're referring to it feeling burn your fingers, that is not normal and to be unexpected.
Check if your MacBook charging adapter has been buried under a book, sat out in the hot sun, or otherwise left in an area with high temperatures, it could be overheating.
Above pitfalls can be easily avoided. If the adapter is continually getting very hot, turn off your computer for 60 seconds or so, and unplug the connector from the wall outlet. Let it cool down completely, then plug it back in to the wall outlet and turn on the laptop. Monitor the temperature of the adapter for a while, see if it gets hot again. Or take out the battery, let it sit for a minute or so, and then put it back in. Unfortunately, you can’t remove the battery using the same method on newer MacBooks, so you’ll have to skip to the next skip.
The other factor may cause this issue is the DC-in Board pins 2&4 (2nd from either side). Magsafe connector 2&4 pins are pressed down and not fully popped out. Hence these pins while touching DC-in Board connector, do not make full contact. They are making loose contact which in turn makes arcing/sparking which generates heat to make Magsafe hot. It is a typical phenomenon you see on household electric plugs getting fused and burnt. We suggest you clean both sides with alcohol and there is high chance that it should be working good.
At last, if it becomes hot again, unplug it immediately and ask for help from a licensed professional.