# Disallow particular paths for all user agents User-agent: * Disallow: Crawl-delay: 30 # Disallow entire user agents User-agent: SemrushBot Disallow: / User-agent: Semrush Disallow: / User-agent: SEMrush Disallow: / User-agent: semrush Disallow: / User-agent: Mappy Disallow: / User-agent: BUbiNG Disallow: / User-agent: Heart Disallow: / User-agent: MauiBot Disallow: / User-agent: Baiduspider Disallow: /