# Traffic management and bot protection **URL:** \ **Category:** Infrastructure, Interoperability, Collaboration\ **Tags:** repositories, tools, bots, anubis, cloudfare\ **Created:** [18 June 2026 08:06 UTC](https://forum.edch.eu/t/traffic-management-and-bot-protection/945 "2026-06-18T08:06:26Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![mpaulhac](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.edch.eu/mpaulhac/32/48_2.png) [@mpaulhac](https://forum.edch.eu/u/mpaulhac)\ **Post date:** [18 June 2026 08:06 UTC](https://forum.edch.eu/t/traffic-management-and-bot-protection/945/1 "2026-06-18T08:06:26Z")
Hi everyone, we recently had an internal discussion about how to handle automated traffic, especially regarding the balance between “good bots” and “bad bots.” During that meeting, we talked about [Anubis](https://anubis.techaro.lol), so I am very glad to see some feedback from the community about this tool, as well as [Cloudflare](https://www.cloudflare.com). **I am sharing this insightful article by OAPEN: [Traffic management and bot protection for the OAPEN Library and DOAB: Implementing Cloudflare and Anubis](https://oapen.hypotheses.org/2287).** Here is a short summary of the key takeaways: > Automated traffic (bots and AI agents) now accounts for 60-80% of all web requests, surpassing human traffic for the first time. For open science infrastructures like OAPEN and DOAB, this surge threatens system stability, inflates costs, and distorts metrics. > To ensure sustainable and equitable access, a layered traffic management strategy has been implemented without resorting to blanket blocking: > > - DOAB uses Cloudflare to filter massive traffic and mitigate overload, virtually eliminating “503 - service unavailable” errors. > - OAPEN deployed Anubis, an open-source anti-bot tool that uses lightweight proof-of-work challenges to filter automated scraping while keeping access seamless for humans. > > The goal is to maintain a delicate balance, protecting infrastructure performance and stability while keeping allowlists updated so that “good bots” (scholarly indexers, search engines, and research AI tools) can continue to discover and reuse open content. ---
**Author:** ![ajnyga](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@ajnyga](https://forum.edch.eu/u/ajnyga)\ **Post date:** [7 July 2026 12:26 UTC](https://forum.edch.eu/t/traffic-management-and-bot-protection/945/2 "2026-07-07T12:26:49Z")
We have been using Anubis now for a few months in our services ([Journal.fi](http://Journal.fi) and [Edition.fi](http://Edition.fi)). The effect has been dramatic with 60-70% reduction in traffic and at the same time zero reports of actual visitors being blocked. This is of course in many ways a good thing, but also comes with problems. What I have not seen discussed yet is the challenge this imposes to using statistics as a metric of journal quality. Some funding for example requires the journals to show how much visitors they have. Now that using a service like Anubis reduces the traffic, the journals need to explain this in their new applications. Also journals are now reporting stats that can not be compared anymore with some using Anubis and some not. I think COUNTER standard is lagging behind here. It had the purpose of making statistics in different services easy to compare, but now with some using bot protection and some not we are probably far from comparable statistics. ---
**Author:** ![mpaulhac](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.edch.eu/mpaulhac/32/48_2.png) [@mpaulhac](https://forum.edch.eu/u/mpaulhac)\ **Post date:** [16 October 2026 12:01 UTC](https://forum.edch.eu/t/traffic-management-and-bot-protection/945/3 "2026-10-16T12:01:05Z")
Hi all, I’m coming back to this discussion because I just came across a resource that hasn’t been mentioned yet. COAR created the [Dealing With Bots Task Group](https://dealing-with-bots.coar-repositories.org/about/) to develop advice and supporting information for repository managers on handling this phenomenon. The website is the group’s primary output. It covers a variety of strategies for managing bots: some can be implemented directly on your repository system, while others may require additional hardware or software components. It is very repository-focused, and I’m not familiar enough with other types of infrastructure to say how relevant it would be for them, but it may still be an interesting read. > **[Home](https://dealing-with-bots.coar-repositories.org)**