I like Jorge's answer.  I would also add, though, that rep isn't just a trust measure either.  It's also a metric for understanding how the system works.  

So, if you only ask questions - but you've been around a while - then you probably understand when to stop with comments and invite to chat, when voting for closure/flagging are appropriate, and other things.  
  
If you don't allow non-answering people to do this, then you put even more load on your people who are capable of answering - and they stop doing both.