There's several different aspects here of "sockpuppeting" being addressed, but what you're talking about isn't necessarily sockpuppetting but targeted voting. So let's address each part individually.
But before we go into this into detail, with regards to your "wrongly detected" comment...
It should be made known that, to Moderators, it's very very easy to identify sockpuppetry and using them to bolster one's reputation. It is almost impossible for one who is receiving upvotes that are just targeted/serial in nature to be wrongly accused of sockpuppetry.
Now onto our regularly scheduled, overly-detailed explanation and answer.
Sockpuppeting
Sock puppets are duplicate accounts made by an individual. On their own, sock puppets which don't break any rules or have any mutual voting to boost one's rep are perfectly fine - I myself have one for testing things in a non-privileged manner and to run a chat bot in a private channel for moderation purposes.
Sock puppets which, however, have a very strong record of voting for each other, result in bans, account destruction of the socks, and invalidation of votes. And each subsequent offense of it can increase the bans significantly and cause issues.
These types of things are fairly easy to spot because they have very suspicious voting trends, and usually only vote for each other. It's harder to initially detect, but once found you can start to trace the 'master' to see future incidents of this.
This doesn't apply to you in this case. Ultimately the 'beneficiary' of sockpuppeting is the puppet master of the socks. (and they get banned as well in increasing severity for repeat offenses)
Targeted Voting / Serial Voting
Now, i mention targeted voting above for sockpuppets, but this is different because no socks are involved, and it's usually one person voting for another but not being the same person.
While this is understandably possible of happening, a handful of votes here or there from one person to another isn't necessarily against the rules. The system, also, has protections to detect serial upvoting in a way that is a violation of the rules, and as such those cases have automatic voting reversals within some period of time (not instantaneously).
Like for sockpuppets, though, it can result in suspensions for voting irregularities for the person who is doing the voting. The person who 'benefited' from the targeted upvoting isn't punished other than the loss of the reputation they had gained as a result of voting invalidation.
Ultimately I wouldn't be too worried about this. I'd suggest you consider reading up on serial voting for more info on the subject, but if you are just being given all the upvotes and it isn't a sockpuppeting issue, you'll just lose the rep when the system invalidates the reputation (either automatically or as a result of moderators requesting that it happen due to voting irregularities).
Why you shouldn't worry about the occasional set of upvotes from someone
I personally get a handful of upvotes in small clusters and batches - usually a few of my most popular answers and sometimes in rapid succession I get 50 - 100 reputation. The problem is, though, votes are by default anonymous so not even moderators can go and see who all voted for a given question and what their votes were, so tracing one-off cases of serial voting is VERY difficult.
If it's a violation of the rules then either the system will autoreverse the voting, or when we invalidate votes for a given serial upvoter the system will indicate you lost rep because of vote invalidation.
