Can someone whip up a quick script for UWN?

Question

I would like to see the Ubuntu Weekly News add a section for SE questions. The thing is that it's run by non-technical people and since the data we need is already in SE I figured someone could whip up a quick script to grab the right info and then spit it out in MoinMoin wiki format (That they use) so the editor can just fire off the command, and then copy and paste the moin-compliant results:

• Grab the top five from here: https://askubuntu.com/?tab=week
• Top contributors for the week: http://stackexchange.com/leagues/31/week/ubuntu
• The moin formatting information is here: http://moinmo.in/HelpOnFormatting

It would look like this, but with links to the individual questions and users:

 == Top User Questions ==

  * My processor is 64-bit - does that mean I need the amd64 image?
  * Why might I want to use a clipboard manager?
  * Why are notifications so low?
  * With a launcher for a terminal application, how can I keep the terminal 
    open after the program is complete?
  * Alternative desktop managers

  People contributing the best questions and answers this week: Oli, Richard 
  Holloway, Marco Ceppi, fluteflute, and Jan Claeys

Can someone help me out?

Answer 1

I noticed that this didn't feature in this week's UWN. I thought that might be because there was some problem with the code due to the domain switch but it still works.

Anyway, I've changed things so the links go directly to the right domain. I've also stripped out the clipboard output for now as it has a problem with some unicode characters that I (frankly) can't be bothered fixing at the moment.

Feel free to edit this post to fix the bug.

import urllib2, StringIO, gzip, json, time, os
from BeautifulSoup import BeautifulSoup

output = ' == Most Active Questions == \n'

# top active threads (votes + answers + comments)
for q in BeautifulSoup(urllib2.urlopen('http://askubuntu.com/?tab=week')).findAll('h3')[:5]:
    output = '%s\n * [[http://askubuntu.com%s|%s]]' % (output, q.contents[0]['href'], q.contents[0].string)


output = '%s\n\n == Top Voted New Questions == \n' % output

# top (by votes only) threads started in the past 7 days
api_url = 'http://api.askubuntu.com/1.0/questions?sort=votes&order=desc&pagesize=5&fromdate=%s' % int(time.time() - 604800)
source = urllib2.urlopen(api_url).read()
for q in  json.loads(gzip.GzipFile(fileobj=StringIO.StringIO(source)).read())['questions'][:5]:
    output = '%s\n * [[http://askubuntu.com/questions/%s/|%s]]' % (output, q['question_id'], q['title'])


contribs = ['[[%s|%s]]' % (u.contents[0]['href'], u.contents[0].string) for u in BeautifulSoup(urllib2.urlopen('http://stackexchange.com/leagues/31/week/ubuntu')).findAll('h2')[1:6]]
output = '%s\n\nPeople Contributing the best questions and answers this week: %s and %s' % (output, ', '.join(contribs[0:4]), contribs[4])

print output

Answer 2

Well I wrote a little Python script to do this. Relies on BeautifulSoup (package: python-beautifulsoup available from all good repositories right now for free).

Update: This version inserts hyperlinks for the questions and users. This should be more useful for people actually getting to the site but if you don't want to that, click the "edited at time" link and you can see the previous revisions.

Update 2: I've written a quick tutorial on the basics behind what I'm doing here, explaining what's going on in a slightly simplified model: Scraping websites with Python.

There's some pretty crazy python in here that beginners might not comprehend. I think in Python's list comprehension model now so things end up getting compacted pretty tightly. Roll on 8 lines of heaven!

from BeautifulSoup import BeautifulSoup
import urllib2

output = ' == Top User Questions == \n'

for q in BeautifulSoup(urllib2.urlopen('http://ubuntu.stackexchange.com/?tab=week')).findAll('h3')[:5]:
    output = '%s\n * [[http://ubuntu.stackexchange.com%s|%s]]' % (output, q.contents[0]['href'], q.contents[0].string)

contribs = ['[[%s|%s]]' % (u.contents[0]['href'], u.contents[0].string) for u in BeautifulSoup(urllib2.urlopen('http://stackexchange.com/leagues/31/week/ubuntu')).findAll('h2')[1:6]]
output = '%s\n\nPeople Contributing the best questions and answers this week: %s and %s' % (output, ', '.join(contribs[0:4]), contribs[4])

print output

Actual output:

oli@bert:~/Desktop$ python se-top.py 
 == Top User Questions == 

 * [[http://ubuntu.stackexchange.com/questions/3378/my-processor-is-64-bit-does-that-mean-i-need-the-amd64-image|My processor is 64-bit - does that mean I need the amd64 image?]]
 * [[http://ubuntu.stackexchange.com/questions/3553/if-apt-get-from-2020-existed-what-one-project-would-you-install|If apt-get-from-2020 existed, what one project would you install? ]]
 * [[http://ubuntu.stackexchange.com/questions/3335/why-might-i-want-to-use-a-clipboard-manager|Why might I want to use a clipboard manager?]]
 * [[http://ubuntu.stackexchange.com/questions/3326/why-are-notifications-so-low|Why are notifications so low?]]
 * [[http://ubuntu.stackexchange.com/questions/3541/can-not-paste-into-terminal|Can not paste into terminal]]

People Contributing the best questions and answers this week: [[http://ubuntu.stackexchange.com/users/449/oli|Oli]], [[http://ubuntu.stackexchange.com/users/458/richard-holloway|Richard Holloway]], [[http://ubuntu.stackexchange.com/users/41/marco-ceppi|Marco Ceppi]], [[http://ubuntu.stackexchange.com/users/866/fluteflute|fluteflute]] and [[http://ubuntu.stackexchange.com/users/935/jan-claeys|Jan Claeys]]

Edit 15: I saw Javier's comment about getting just this week's top-voted threads and challenged myself to do it. I also went a little further overboard... The script will now automatically copy the output to the clipboard and send you a notification to let you know when you can paste. Should make it possible to just be a launcher for somebody to double-click.

This version requires one more dependency (xsel for copying to the clipboard): sudo apt-get install python-beautifulsoup xsel

import urllib2, StringIO, gzip, json, time, os
from BeautifulSoup import BeautifulSoup

output = ' == Most Active Questions == \n' # (votes + answers + comments)
for q in BeautifulSoup(urllib2.urlopen('http://ubuntu.stackexchange.com/?tab=week')).findAll('h3')[:5]:
    output = '%s\n * [[http://ubuntu.stackexchange.com%s|%s]]' % (output, q.contents[0]['href'], q.contents[0].string)


output = '%s\n\n == Top Voted New Questions == \n' % output # top by votes only
api_url = 'http://api.ubuntu.stackexchange.com/1.0/questions?sort=votes&order=desc&pagesize=5&fromdate=%s' % int(time.time() - 604800)
source = urllib2.urlopen(api_url).read()
for q in  json.loads(gzip.GzipFile(fileobj=StringIO.StringIO(source)).read())['questions'][:5]:
    output = '%s\n * [[http://ubuntu.stackexchange.com/questions/%s/|%s]]' % (output, q['question_id'], q['title'])


contribs = ['[[%s|%s]]' % (u.contents[0]['href'], u.contents[0].string) for u in BeautifulSoup(urllib2.urlopen('http://stackexchange.com/leagues/31/week/ubuntu')).findAll('h2')[1:6]]
output = '%s\n\nPeople Contributing the best questions and answers this week: %s and %s' % (output, ', '.join(contribs[0:4]), contribs[4])

print output

# Why stop there?! Let's copy this to the clipboard and spring a notification!
os.popen('xsel', 'wb').write(output)

os.system("notify-send 'Yo! When you next paste, you will be pasting the awesome scores from the Ubuntu StackExchange website.'")

Output

 == Most Active Questions == 

 * [[http://ubuntu.stackexchange.com/questions/3553/if-apt-get-from-2020-existed-what-one-project-would-you-install|If apt-get-from-2020 existed, what one project would you install? ]]
 * [[http://ubuntu.stackexchange.com/questions/3378/my-processor-is-64-bit-does-that-mean-i-need-the-amd64-image|My processor is 64-bit - does that mean I need the amd64 image?]]
 * [[http://ubuntu.stackexchange.com/questions/3541/can-not-paste-into-terminal|Can not paste into terminal]]
 * [[http://ubuntu.stackexchange.com/questions/3690/linux-alternative-to-photoshop-or-paint-net|Linux Alternative to Photoshop or Paint.NET]]
 * [[http://ubuntu.stackexchange.com/questions/3697/how-to-install-fonts-in-ubuntu| How to install fonts in Ubuntu?]]

 == Top Voted New Questions == 

 * [[http://ubuntu.stackexchange.com/questions/3554/|How to make my proxy settings change depending on the network I connect to ?]]
 * [[http://ubuntu.stackexchange.com/questions/3503/|Best way to cache apt downloads on a LAN?]]
 * [[http://ubuntu.stackexchange.com/questions/3530/|What's happening with windicators?]]
 * [[http://ubuntu.stackexchange.com/questions/3539/|How can I make and distribute an Ubuntu screensaver?]]
 * [[http://ubuntu.stackexchange.com/questions/3750/|What precautions should I take when exposing my Ubuntu desktop directly to the internet?]]

People Contributing the best questions and answers this week: [[http://ubuntu.stackexchange.com/users/449/oli|Oli]], [[http://ubuntu.stackexchange.com/users/866/fluteflute|fluteflute]], [[http://ubuntu.stackexchange.com/users/41/marco-ceppi|Marco Ceppi]], [[http://ubuntu.stackexchange.com/users/458/richard-holloway|Richard Holloway]] and [[http://ubuntu.stackexchange.com/users/235/jorge-castro|Jorge Castro]]