Headers are inexpensive means to connect wires to Arduinos or other circuit boards, while still being able to disconnect a pcb from your prototype. Always use plugs if you can. It keeps things modular, and you can exchange an Arduino with another if you have doubts that your Arduino is broken.
A header can be used for low voltages, and currents under 1 Ampère.
Be aware of the pitch (distance between the centers of the pins). Most used is 1/10 inch = 2.54 mm but also 2 mm pitch is frequently used.
Headers come in two forms: pin and socket.