If the question owner didn't accept an answer, and the bounty expires, half the reputation goes to the highest rated answer (score >=2) posted after the bounty was started. If no answer meets that criterion then the bounty just dies off without being awarded. The bounty poster doesn't get the reputation back, and the bounty expires.
That said, there is a case where a new answer posted after the bounty is posted, which receives two upvotes, will be awarded the bounty. Assuming it hasn't already expired.
