I like Jorge's answer. I would also add, though, that rep isn't just a trust measure either. It's also a metric for understanding how the system works.
So, if you only ask questions - but you've been around a while - then you probably understand when to stop with comments and invite to chat, when voting for closure/flagging are appropriate, and other things.
If you don't allow non-answering people to do this, then you put even more load on your people who are capable of answering - and they stop doing both.
Let me use myself as an example:
I'm all over the SE network... just check my profile.
Would I just randomly close things as OT over on programmers.se, because I don't necessarily grasp the scope of their site? No. If there were another power-user of incredibly high rep that popped in to chat and asked me to contribute some closevotes - not to dogpile question - but to solve a problem - Then I might do it.
Will I ever answer a lot over on programmers? Probably not.
Does this make me less aware of how the system works? Certainly not.