Best practice for 429 handling in distributed crawlers?

[minzf581]

Crawlee’s retry and autoscaling model is solid.

In large distributed agent scenarios, 429s often correlate with identity reuse patterns rather than raw concurrency.

Do you recommend request-level proxy rotation, or task-level identity isolation to reduce cascading rate limits?

Curious how you model long-lived agents vs burst crawlers in terms of network strategy.