if you think it's a bad question or not worth your time then just leave it, rather than taking the time to downvote it?
NO.
I'm not sorry. No.
I don't know how you use Stack Exchange sites or what you think their intended usage is, but I use it as a library*. When I browse this library, looking for something, I want good results. That means good answers. And that almost always means good questions. That means maintaining the sites: cleaning up useless crap, polishing the useful stuff, and ensuring that over time, there is more useful stuff than crap.
That means encouraging good questions ...
And...
Discouraging bad questions.
If there were no disincentive to posting crap, we'd be drowning it. Check my vote count on Ask Ubuntu:
5,088 up
6,167 down
But! That's my total vote count, including deleted posts. My surviving downvotes, on posts that haven't been deleted, is:
2,033 votes cast
That means a good two-thirds of the posts I have downvoted end up deleted. (For comparison, my surviving upvote count is 4,968 - some 2% have been deleted.) Now, you could say that some of the deleted posts I have downvoted may not have deserved it, but I say that some of the surviving downvoted posts do deserve deletion**, and these two balance each other out.
How much useful would you think the site would be, if for every useful post you see here, there's two crap posts? The search is bad enough as it is, but it would be far worse if you couldn't even use question score as a way to filter out sand.
Downvoting is, as pomsky says, a quality-control mechanism. There's a reason why downvoting questions is free (downvoting answers costs 1 rep): stem the tide of bad questions, and the number of bad answers that you then have to clean up automatically go down.
Aside from the roomba mechanism, which came rather late in the game, voting is the core quality control mechanism of Stack Exchange. Other systems depend on it: rate limiting, warnings, question bans. There's a lot that works with a signal as simple as a downvote.
* There's a reason I have posted something like ten times the answers on this site alone than I have posted questions on all SE sites combined. Most questions I have usually have been answered, or I have found answers I can adapt.
** Sometimes the roomba doesn't clean up posts that I have downvoted because some well-meaning idiot gave a sympathy upvote.