11

As per this AU meta question we've seen links to external sites turn malicious over the years.

So could we have:

  • A Policy that this will not be allowed any more on the entire SE network?
  • A Process that removes Malicious/dead links after the fact just like LMGTFY links are refused before the fact)
  • a Product (batch job that runs nightly/weekly that checks whether or not domain names have turned malicious so that the community user can just edit them out?) that implements the above Process?

Because now it depends solely on People!

Please upvote if you think this is a good idea and downvote if it's a bad one so we can go / not go bark up the SE developer's tree..

  • 10
    Upvote my comment if you agree that regardless of the outcome we expect Fabby to 1. find that tree. 2. start barking. – Rinzwind May 29 '18 at 12:18
  • 2
    As far i know malicious links are already not allowed on SE network, but for the rest i really would see something like that. maybe even checking for dead links and putting those answers to a review Que. – Videonauth May 29 '18 at 14:08
  • 1
    I almost jumped in to defend codeplex. It used to be a decent code sharing and tutorial site. Went there again, and mother of god. All spam and ugliness. – RobotHumans May 29 '18 at 16:45
  • @Videonauth Great idea! Comment incorporated into Question – Fabby May 29 '18 at 18:31
  • 1
    Is there a reason you don't just flag it for moderator attention and let us go purge such links? There's no automatic way to determine if a link is actually malicious or not, nor could you write such a system in a way that is foolproof. – Thomas Ward May 29 '18 at 18:40
  • Nothing is foolproof, but even I would not be evil enough to request the batch job to start spamming the poor overworked mods even more... (Sometimes it's so nice to be able to misinterpret someones words and then just turn them around) 99.9% is good enough in this case, whereas doing nothing is just bad PR overall @ThomasWard – Fabby May 29 '18 at 18:46
  • 1
    If you need another example, the same happened to a post of mine a while back, until another editor edited out the malicious link: askubuntu.com/posts/173094/revisions – Dan May 29 '18 at 19:21
  • @Dan Added... :-) – Fabby May 29 '18 at 19:30
7

Update: I have a Product: linkcheck.py.

(Up to StackExchange to come up with the site-wide Policy and Process)

It works but how do we know what a bad domain is?

enter image description here

So rather than wait around, I saw an opportunity to write some code and took it. It downloads the blacklists that Smoke Detector uses on new posts and edits and applies them against our entire database. You'll need to download a data dump, and install pypy. All in the readme.

After some more tweaking, it handles ~13,000 posts per second on my moderately powerful desktop so can chunk out the entire site in ~45s. The following is the output (and the matched domain):

https://askubuntu.com/q/44668/ rhubcom.com
https://askubuntu.com/q/168301/ cadsoftusa.com
https://askubuntu.com/q/211202/ code-industry.net
https://askubuntu.com/q/223107/ code-industry.net
https://askubuntu.com/q/304193/ pornhub.com
https://askubuntu.com/q/369358/ downloadavideo.net
https://askubuntu.com/q/656940/ code-industry.net
How to reformat pdf (change page size) using Master PDF Editor or ___ code-industry.net
https://askubuntu.com/q/718011/ manualbirds.com
https://askubuntu.com/q/802974/ videograbber.net
https://askubuntu.com/q/813665/ videograbber.net
How to identify malicious .deb files / sources? code-industry.net
https://askubuntu.com/q/81099/ official-driver
https://askubuntu.com/q/435500/ cadsoftusa.com
https://askubuntu.com/q/563947/ yolasite.com
Master PDF Editor won't start after installation code-industry.net
converting images to pdf without quality loss issuu.com
How to import a pdf in libreoffice? under ubuntu, all pages are blank code-industry.net
https://askubuntu.com/q/525310/ official-driver
https://askubuntu.com/q/303874/ code-industry.net
https://askubuntu.com/q/402201/ official-driver
https://askubuntu.com/q/428857/ code-industry.net
https://askubuntu.com/q/563543/ official-driver
https://askubuntu.com/q/622636/ code-industry.net
https://askubuntu.com/q/666310/ code-industry.net
https://askubuntu.com/q/666816/ listoffreeware.com
https://askubuntu.com/q/709665/ code-industry.net
https://askubuntu.com/q/718360/ code-industry.net
https://askubuntu.com/q/756653/ code-industry.net
https://askubuntu.com/q/936828/ code-industry.net
https://askubuntu.com/q/969431/ keyboardtester.co
Problem installing Adobe Reader with wine code-industry.net
https://askubuntu.com/q/244253/ code-industry.net
Ubuntu and ASUS G750JM, GTX860M / Nvidia problems official-driver
phpmyadmin not found //adf.ly/
https://askubuntu.com/q/531620/ code-industry.net
https://askubuntu.com/q/542931/ code-industry.net
Installing Eagle on 14.04 x64 cadsoftusa.com
https://askubuntu.com/q/614626/ code-industry.net
https://askubuntu.com/q/888913/ code-industry.net
https://askubuntu.com/q/893022/ code-industry.net
Cannot update after install master-pdf-editor code-industry.net
Strange URLs showing up in SysLog file pornhub.com
https://askubuntu.com/q/942413/ code-industry.net
https://askubuntu.com/q/956221/ code-industry.net
https://askubuntu.com/q/1001745/ code-industry.net
https://askubuntu.com/q/435413/ cadsoftusa.com
https://askubuntu.com/q/824570/ backup-utility.com
Is it possible to open a pdf that requires Adobe Reader 8 without using acroread? code-industry.net
https://askubuntu.com/q/160613/ official-driver
https://askubuntu.com/q/256642/ official-driver
https://askubuntu.com/q/335185/ //adf.ly/
https://askubuntu.com/q/355390/ coolutils.com
https://askubuntu.com/q/560963/ code-industry.net
https://askubuntu.com/q/617605/ code-industry.net
Error running Master PDF Editor 4 - missing libQtSvg.so.4 code-industry.net
https://askubuntu.com/q/922158/ code-industry.net
https://askubuntu.com/q/424886/ code-industry.net
https://askubuntu.com/q/777592/ code-industry.net
https://askubuntu.com/q/303874/ code-industry.net
https://askubuntu.com/q/86682/ official-driver
https://askubuntu.com/q/87526/ official-driver
https://askubuntu.com/q/424021/ rhubcom.com
https://askubuntu.com/q/442080/ code-industry.net
https://askubuntu.com/q/523646/ code-industry.net
https://askubuntu.com/q/648024/ code-industry.net
https://askubuntu.com/q/675507/ code-industry.net
https://askubuntu.com/q/893641/ code-industry.net
https://askubuntu.com/q/907089/ code-industry.net
https://askubuntu.com/q/982165/ code-industry.net
https://askubuntu.com/q/110312/ official-driver
https://askubuntu.com/q/303874/ code-industry.net
https://askubuntu.com/q/695976/ code-industry.net
https://askubuntu.com/q/863429/ serverbasket.com

Do as you will.


Because now it depends solely on People!

It does, but before we get too puffed up, what's the history of links-going-evil like here? How much work is it actually to:

  1. Spot that the domain/URL is now bad
  2. Do a search for posts with that domain
  3. Flag/edit/etc

I also have to consider that step 1 of the above process might not be happening. How often do you spider the site checking for dodgy links?


My first thought is that Smoke Detector should be doing this. It already keeps track of bad domains/urls and their usage on new posts and new edits. As far as I know it doesn't retroactively search when a new "bad link" is added.

Maybe they could add this sort of feature. It's pretty hardcore though.

I just want to see what's involved with a DIY solution. I'm downloading the Ask Ubuntu data dump from March and I'll try to script something together to use Smoke Detector's bad-*.txt regex files and scan every post on the site. Of course that's possible, but performance may be... Interesting.

  • This would be outside the scope of Smoke Detector, we'd have to have a dedicated utility that would do this. SmokeDetector has no mechanisms to retroactively test posts, and if it did we'd have no API queries left. (It takes exponentially more requests to do retroactive testing, and I doubt SE would give the Charcoal team an API key with greater than 20000 requests a day anyways, which is what we have now) – Thomas Ward May 30 '18 at 17:20
  • 1
    Cool! I'll check it out when I get back to my Ubuntu machine... Upvoted already! – Fabby May 30 '18 at 18:21
  • 3
    "As far as I know it doesn't retroactively search when a new "bad link" is added." - the logic behind this is that if the post containing $BadDomain doesn't get bumped, then it's not really going to be a problem. If it does get bumped, it'll get scanned again and picked up. – ArtOfCode May 30 '18 at 22:16
  • 1
    Worth noting also that almost all of the authors of these posts have too much reputation to be detected. (50 rep is the maximum for the blacklist) – a spaghetto May 30 '18 at 22:31
  • Moderately powerful desktop??? On my Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz, I get 60-64 it/s... – Fabby May 30 '18 at 23:41
  • @Fabby I upgraded the script yesterday to not use regexes and used fixed strings (I manually expanded and converted the list from Smokey). This flies along (hence the output above). I'll get it uploaded as soon as I get a chance. – Oli May 31 '18 at 8:30
  • The pornhub edit approval above was particularly annoying. – Oli May 31 '18 at 8:32
  • I saw someone started at the top, so I started at the bottom... – Fabby May 31 '18 at 9:22
  • I tried on Linux.stackexchange.com and there was only one there, so not posting on meta.se Thanks again! – Fabby Jun 6 '18 at 23:26
1

I think it's a good general practice to try and clean up these links, but we may or may not ever run in to the problem of someone poisoning the bad domain list or a heuristic picking up on some blog that does a kernel module programming tutorial as a downloadable rootkit or something.

I'm not sure if I'ld delete them or just put a don't click this/click at your own risk thing next to them.

  • I'm editing them out one by one or flagging as SPAM if too bad or a comment I can't edit. – Fabby May 31 '18 at 0:49
  • If you want to trust google chrome malicious site links, you can use developers.google.com/safe-browsing/v4/update-api - which allows offline storage. That would also let you flag a domain as poisoned but OK. – RobotHumans May 31 '18 at 1:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .