Author Topic: Automatic downloaders - avoid being blocked  (Read 4488 times)


  • Administrator
  • Sr. Member

  • Offline
  • *****
  • 299
    • View Profile
    • Higher Intellect
Automatic downloaders - avoid being blocked
« on: June 07, 2013, 05:43:26 am »
With an archive of this size, some users may be tempted to use automated download software to grab everything they can all at once. If you set sane limits on the number of hits and bandwidth being used, this isn't an issue and you likely won't be flagged by our auditing scripts. If your IP is flagged as hitting multiple files/directories per second and the bandwidth usage exceeds what any normal person would be consuming, a temporary ban is implemented for a duration of 24-48 hours. If the audit scripts pick up the same IP after the temporary ban is lifted, a more permanent ban is put in place. Trying to mask the user agent won't help you either. We know what's going on.

Users of wget can use the --limit-rate flag to specify this. For other software you'd need to consult the manual.