Basically, it is limestone that has gone through a chemical cycle to break it down and make it workable. The cycle is somewhat complicated, but it boils down to this:
Lime plaster starts as limestone, which is quarried out of the ground and burnt in lime kilns at very high temperatures. This drives the carbon dioxide from the calcium carbonate. What's left is calcium oxide, or "quicklime." These lumps of lightweight calcium oxide are then mixed with water ("slaked") to become a thick, white putty of calcium hydroxide, or "hydrated lime," which is its most common form. This putty is then mixed with two to three parts sand to make a plaster.
This calcium hydroxide combines with the carbon dioxide in the air, replacing that which was driven off in the kiln, and becomes calcium carbonate again. It is the completion of this chemical process - from calcium carbonate to calcium oxide to calcium hydroxide and back to calcium carbonate - that makes it hard.