It does take time, depending on what tool you are using, or whether you're doing it by hand. Every item has it's own assigned "tool" which will mine it faster depending on the tool's quality. For example, you can dig dirt by hand, but you can dig a lot faster with a shovel. Then there are ones which are near impossible to dig by hand (like rock), which require the correct tool to dig it in a reasonable time. Also, the item won't drop in some cases unless you are holding the right tool, and really hard stuff won't drop unless the tool is higher quality. (stone doesn't drop unless dug by a pickaxe, diamond can only be mined with an iron or diamond pick)
Some things, like flowers, simply get drop the moment you hit them with anything, while others take about 10 seconds with the best tool possible, like obsidian. Then there is bedrock, which simply cannot be destroyed. (To prevent you from going below y=0, y=64 is sea level, for reference, and they used to prevent placing of blocks above y=128, but it's been extended to y=256 in a recent patch.)
Also, with the semi-recent introduction of hunger (beta 0.8), mining blocks does require a small amount of energy, and will make you hungrier. If you become too hungry, your hp will drop until you either reach the minimum hp hunger can put you on for your difficulty, or die if you're playing on hard. You regenerate hp over time if your food bar is full. However, even if you're running on empty, you can still destroy blocks.