The short answer, speaking as a mathematician, is that one should say "fewer" when one is comparing countable quantities (or discrete quantities) and "less" when comparing continuous quantities.
Less formally, things like weight are continuous, because I can have an object that weighs 1 gm, 1000 kg, *or anything in between*. The weight of an object doesn't have to be an integer number of grams: it could be 1 gm, 2 grams, or 1.5 grams, 1.25 grams, or 1.125 grams. (Replace kilos and grams with pounds and ounces, if you will.) For continuous quantities, one uses 'less'. So I could say "My bag weighs less than your bag."
On the other hand, you sometimes want to count two sets of things, and decide which set is larger. Here, the number of items in the set must take a value like 1,2,3, etc. For countable quantities, one uses "fewer". So I would say "my class has fewer students than your class": I can't have 13.5 students, or 13.125 students, after all; I have 13 or 14. Similarly, I could say "I have fewer apples than you do."
Finally, when directly comparing numbers (without units), one normally uses 'less'. So one would say "Two is less than three", but "Two apples are fewer than three apples".
(And for the pedantic (a group I often belong to), that doesn't describe usage for quantities that are discrete, but not integral. Thankfully, those don't come up very often in real life.)