You have removed the email account from the addresses in your login but Gmail still considers the site as authorized to access your Gmail data and log you in in Ask Ubuntu.
Go to your brother's Gmail account > Account Settings, look for Security > Application and sites authorization, click Edit. From the list of authorized sites look for askubuntu.com and click Remove Access Authorization.
I am assuming that you have used Gmail to login to Stack Exchange with his account, if you use another service this should be similar.