Reach for a doorknob and watch your own arm. It moves fast for most of the distance and then slows down at the end. That slow part isn't caution, it's correction — your hand overshoots slightly and pulls back, overshoots less, pulls back less, until it lands. The travelling is cheap. The correcting is what costs you, and it costs more the smaller the thing is.
Paul Fitts measured this in 1954 and found it was predictable enough to write down. Double the distance and you pay roughly a fixed amount more. Halve the size and you pay about the same amount again. The two trade off against each other, which is the useful part: a big button far away can be faster than a small one close by.
You don't have to take my word for it, and you shouldn't. Hit the targets below and the page will plot your own times.
What it costs you
Big beats pretty. A 32-pixel icon button with 8 pixels of padding around it is a 32-pixel target, not a 48-pixel one, unless the padding is inside the hit area. Half the icon buttons I have shipped failed this and looked fine in review.
Distance is measured from where the cursor already is, not from the middle of the screen. If someone has just used a control, the next control they need should be near it. This is the whole argument for context menus, and most of the argument against toolbars.
The four edges of the screen are infinitely deep. You cannot overshoot them — the pointer stops. A target pinned to an edge behaves as if it were enormous, which is why the Mac menu bar is fast and a menu three pixels below the top of a window is not.
The worst target is small, far, and beside something destructive. Each of those is survivable alone. Together they are how people delete work they meant to keep, and then blame themselves for it.
On a phone the pointer is a fingertip. About nine millimetres wide, arriving from below, covering the thing it is aiming at. Everything above is still true and all the numbers are worse.
None of this is hard to know. It is hard to keep holding onto at the point in a project where someone senior wants the button smaller.