“Masteries” are a category of skills that grant direct power bonuses to one or more of your skills and spells. For example, ‘mastery of medicine’ boosts the amount healed by all ‘heal’ spells by a large percentage.
Although the effects of masteries can vary wildly, they almost always give far greater benefits than what can be had by 100% ability in what they modify. In most cases a mastery skill will add at least an extra 50% to the power of what they influence when maxed out.
Some masteries will also cause special effects or extra abilities to become available. For example ‘knowledge of identification’ will cause the identify spell to give highly detailed bonus information about materials.
Between changes to the game, guild customizations, reputation systems, special bonuses, and passive skill modifiers, there is actually a lot of gray area as to what skills are actually “masteries” in the classic sense of the term. For example ‘Essence of corrosion’ is an extremely useful modifier of acid spells, but is not their mastery in the classic sense, but a newer reputation bonus for acid mages.
As a rule, true masteries always give a straight power boost. They’re usually named accordingly: Skills named “theory of …”, “mastery of ..”, “knowledge of …” are generally classic mastery skills. They’re a mechanic that is older than the ‘new’ reputation systems, and will tend to be less dependent on how long you’ve been in any given guild.