Pearls are formed by the mantle.
The mantle is the outer tissue of a mollusc that in some species, secretes chemicals that form its shell. A pearl is formed when the mantle tissue gets irritated, as a side-product of its defence mechanism.
The irritation is caused by either damage to the tissue, or little trespassers such as parasites.
The mantle forms a layer of crystallized calcium carbonate and conchiolin, which normally builds the shell. When this formation occurs around the irritation, it's called the nacre. The nacre stays between the shell and the mantle. As the mantle keeps secreting calcium carbonate, it adds more layers around the nacre, and the nacre grows. I guess this explains why pearls have an onion-like layered peelable structure.
Most nacres end up with slightly irregular shapes. Some end up perfectly smooth and spherical. It is sometimes possible to peel off the irregular layers to get to the smoother ones.
It's weird how I never questioned what a pearl really was until recently.