Malicious/Dead link removal Policy, Process, Product requested

Question

As per this AU meta question we've seen links to external sites turn malicious over the years.

So could we have:

A Policy that this will not be allowed any more on the entire SE network?
A Process that removes Malicious/dead links after the fact just like LMGTFY links are refused before the fact)
a Product (batch job that runs nightly/weekly that checks whether or not domain names have turned malicious so that the community user can just edit them out?) that implements the above Process?

Because now it depends solely on People!

Please upvote if you think this is a good idea and downvote if it's a bad one so we can go / not go bark up the SE developer's tree..

Upvote my comment if you agree that regardless of the outcome we expect Fabby to 1. find that tree. 2. start barking. — Rinzwind, Commented May 29, 2018 at 12:18
As far i know malicious links are already not allowed on SE network, but for the rest i really would see something like that. maybe even checking for dead links and putting those answers to a review Que. — Videonauth, Commented May 29, 2018 at 14:08
I almost jumped in to defend codeplex. It used to be a decent code sharing and tutorial site. Went there again, and mother of god. All spam and ugliness. — RobotHumans, Commented May 29, 2018 at 16:45
Is there a reason you don't just flag it for moderator attention and let us go purge such links? There's no automatic way to determine if a link is actually malicious or not, nor could you write such a system in a way that is foolproof. — Thomas Ward, Commented May 29, 2018 at 18:40
Nothing is foolproof, but even I would not be evil enough to request the batch job to start spamming the poor overworked mods even more... (Sometimes it's so nice to be able to misinterpret someones words and then just turn them around) 99.9% is good enough in this case, whereas doing nothing is just bad PR overall @ThomasWard — Fabby, Commented May 29, 2018 at 18:46
If you need another example, the same happened to a post of mine a while back, until another editor edited out the malicious link: askubuntu.com/posts/173094/revisions — Dan, Commented May 29, 2018 at 19:21

Fabby · Accepted Answer · 2018-05-31 09:21:03Z

Update: I have a Product: linkcheck.py.

_{(Up to StackExchange to come up with the site-wide Policy and Process)}

It works but how do we know what a bad domain is?

enter image description here

So rather than wait around, I saw an opportunity to write some code and took it. It downloads the blacklists that Smoke Detector uses on new posts and edits and applies them against our entire database. You'll need to download a data dump, and install pypy. All in the readme.

After some more tweaking, it handles ~13,000 posts per second on my moderately powerful desktop so can chunk out the entire site in ~45s. The following is the output (and the matched domain):

Do as you will.

Because now it depends solely on People!

It does, but before we get too puffed up, what's the history of links-going-evil like here? How much work is it actually to:

Spot that the domain/URL is now bad
Do a search for posts with that domain
Flag/edit/etc

I also have to consider that step 1 of the above process might not be happening. How often do you spider the site checking for dodgy links?

My first thought is that Smoke Detector should be doing this. It already keeps track of bad domains/urls and their usage on new posts and new edits. As far as I know it doesn't retroactively search when a new "bad link" is added.

Maybe they could add this sort of feature. It's pretty hardcore though.

I just want to see what's involved with a DIY solution. I'm downloading the Ask Ubuntu data dump from March and I'll try to script something together to use Smoke Detector's bad-*.txt regex files and scan every post on the site. Of course that's possible, but performance may be... Interesting.

This would be outside the scope of Smoke Detector, we'd have to have a dedicated utility that would do this. SmokeDetector has no mechanisms to retroactively test posts, and if it did we'd have no API queries left. (It takes exponentially more requests to do retroactive testing, and I doubt SE would give the Charcoal team an API key with greater than 20000 requests a day anyways, which is what we have now) — Thomas Ward, Commented May 30, 2018 at 17:20
Cool! I'll check it out when I get back to my Ubuntu machine... Upvoted already! — Fabby, Commented May 30, 2018 at 18:21
"As far as I know it doesn't retroactively search when a new "bad link" is added." - the logic behind this is that if the post containing $BadDomain doesn't get bumped, then it's not really going to be a problem. If it does get bumped, it'll get scanned again and picked up. — ArtOfCode, Commented May 30, 2018 at 22:16
Worth noting also that almost all of the authors of these posts have too much reputation to be detected. (50 rep is the maximum for the blacklist) — a spaghetto, Commented May 30, 2018 at 22:31
Moderately powerful desktop??? On my Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, I get 60-64 it/s... — Fabby, Commented May 30, 2018 at 23:41
@Fabby I upgraded the script yesterday to not use regexes and used fixed strings (I manually expanded and converted the list from Smokey). This flies along (hence the output above). I'll get it uploaded as soon as I get a chance. — Oli, Commented May 31, 2018 at 8:30
I saw someone started at the top, so I started at the bottom... — Fabby, Commented May 31, 2018 at 9:22
I tried on Linux.stackexchange.com and there was only one there, so not posting on meta.se Thanks again! — Fabby, Commented Jun 6, 2018 at 23:26

RobotHumans · Accepted Answer · 2018-05-31 00:41:05Z

1

I think it's a good general practice to try and clean up these links, but we may or may not ever run in to the problem of someone poisoning the bad domain list or a heuristic picking up on some blog that does a kernel module programming tutorial as a downloadable rootkit or something.

I'm not sure if I'ld delete them or just put a don't click this/click at your own risk thing next to them.

answered May 31, 2018 at 0:41

RobotHumans

29.9k24 silver badges38 bronze badges

I'm editing them out one by one or flagging as SPAM if too bad or a comment I can't edit.
– Fabby
Commented May 31, 2018 at 0:49
If you want to trust google chrome malicious site links, you can use developers.google.com/safe-browsing/v4/update-api - which allows offline storage. That would also let you flag a domain as poisoned but OK.
– RobotHumans
Commented May 31, 2018 at 1:52

Add a comment |

Stack Exchange Network

Malicious/Dead link removal Policy, Process, Product requested

2 Answers 2

Update: I have a Product: linkcheck.py.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
feature-request
.

Linked

Hot Network Questions

Malicious/Dead link removal Policy, Process, Product requested

2 Answers 2

Update: I have a Product: linkcheck.py.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged feature-request.

Linked

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
feature-request
.