When a new user joins the site a profile picture is automatically generated for them, how is this image generated? And does SE do it, or a third-party application?
They're third party. Look at the
src URL of one. Here's muru's:
The jumble there is an MD5 hash of the user's email (failing back to network ID if no email set).
This can be overridden by the profile image. These are uploaded to Imgur (like other images here are) and takes preference if set.
Third party. The service is by Gravatar, which provides similar services for a number of sites, including Github, Disqus and Wordpress.