Thwarting the Scraper Horde: Building Defenses Against Malicious Bots
In the battle against malicious bots and web scrapers, developers and DevOps engineers are on the front lines. In this talk, we at HGV will share our first-hand experiences with invasive data collectors that have targeted our web platforms on a significant scale, even causing temporary disruptions of our services. Our aim is to raise awareness of this potential threat and to empower our audience with the knowledge to deal with these situations. We’ll dive into insights on the tactics these unwelcome bots employ and the countermeasures we have deployed to tackle them – including analysing their infrastructure, blocking them with an application firewall, introducing API rate limiting and adapting the API design to impair their data collection capabilities.