Thanks for sharing the program. That's really useful. :-)

Glad you guys are liking it!

Yeah a "clean" command is on my todo list. What I've been doing for now, is running the "backup" command to a new folder, then deleting the old folder. This gives the same effect but obviously isn't as efficient, so I'm happy to add "clean" command to the todo list.

I originally named it gog-repo.py to make it a little more clear.. by Python doesn't really like dash character in file names.

I'll be on vacation for 2 weeks starting Sunday, but feel free to keep the feedback coming. I'll post an updated todo list before I go.

Cheers

You could use and link to the inbuilt issue tracker of GitHuh or anything else. :)
https://github.com/eddie3/gogrepo/issues

What I find most amazing about your project is that in recent times I've been thinking how awesome would it be to have a tool to check my backups and update as necessary.

And here you are, with such a tool.

Like the ancients said before me once: Supimpa!*

Though, for various reasons including a download cap, I'll only be able to try it out in the future. But please keep improving and updating it!

*This is an ancient word used in, I don't know, the 70s maybe, here in Brazil. It has the same meaning of nice or cool.

Just a friendly warning. This will not work.
I have seen enough times files that changed without their file size changing.
This happens regularly for large games with multiple parts for all parts except the first and last because the size of the middle parts is fixed (was ~1.5 GB, nowadays ~4GB).

Great project! Kudos!

I would *think* it's quite rare that there is a change to file content without changing the file size OR the file name. I do acknowledge such a thing is possible though.. and "gogrepo.py verify" command will catch this (at least for installer files which have MD5 data). This is why running verify once in a while is a good idea.

For bonus file which change content, but have same file size, same file name, and have no MD5 provided by GOG.. there's not much I can do to identify that the file has changed. You would have to manually erase the file so it gets downloaded again on your next update and download.
Yeah good point...This should make it easier to track. Feel free to add there too!

Where exactly do you find this MD5 data?

Right from GOG... add .xml to the end of the URL.

Code snippit:

<code>
# fetch the md5 file and chunk data for each game
for f in sorted(g.game_items, key=lambda g: g.name):
try:
if f.url is not None:
with request(f.url.replace('?', '.xml?')) as page:
shelf_etree = xml.etree.ElementTree.parse(page).getroot()
except urllib2.HTTPError:
warn("failed to fetch md5 data for %s. skipping..." % f.name)
continue
finally:
del f['url']
</code>

EDIT: How do I properly <pre> / <code> / {code} so indentation remains? Not sure where forum syntax is.

Thank you for the MD5 info. I was hoping to grab the information for MaGog, but since it involves an extra file access, I will avoid it (enough IP blocking problems as it is).

Do note that GOG blocks one's IP for ~18 hours if one does too many file accesses in a limited period of time (exact limit unknown. Probably in the range of 2000-3000 accesses per hour + some additional limit per 24 hours + forum accesses are also counted).

As for the forum question. Not possible, AFAIK.

Yes you're right about the banning, however I coded in delays to solve this (based on trial/error/testing)... however I'm wondering if this restriction was "relaxed" with the release of GOG Galaxy. Extra fetch for MD5 is annoying (makes update take even longer), but it works.

On the contrary, it was "stressed" with the release of GOG Galaxy. I had to recode some of MaGog's harvesting logic to work around it after being IP blocked day after day.

Interesting. I wonder if we could reach out to GOG support/dev to get more info about these limits.

Good luck. If you get an answer, do please let me know. I never got any.

P.S. I have a theory that the limits are not fixed but depend on the current GOG server load (i.e. higher load, more strict limits).

You tried opening a GOG support ticket? Yeah they may not want to share this info. Also, I assume they "outsource" the actual data storage / download servers.. so perhaps this is not GOG staff applying these temp IP bans.

I opened multiple tickets since December. I already gave up.

If you contact them, I recommend you do not mention MaGog. Many people have opened tickets on my behalf to get GOG to whitelist MaGog's IP and they might be a bit less forthcoming if they associate your question with MaGog.

I can't be sure, but I expect it is a GOG ban because I was able to trigger a ban by accessing the GOG forums (which I don't expect GOG outsources), not only the downloadable files.