I'm stuck with galaxy telling me to update and everytime I try it fails saying:
There was a problem with updating the application. You can always download the latest version from "link"

How do I get out of this update loop?

I suggest downloading v1.2 and then manually deleting the updater file so that it can never update.

Just a workaround.

Close Galaxy and watch it is not running in the background. Download the installer from the web site and install the thing, overwriting the folder. If it keeps failing do the same but this time deleting the Galaxy exe from the Galaxy installation folder.

This should completely fix the issue, since there was a problem with the updater before that update.

I called it workaround, but in fact it is exactly the solution advised in the popup error window. Is doing exactly what it advised, brute force :) but If my memory does not fail, that's how I solved it.