Free SEO Audit
Free Tool

Robots.txt Generator

Build a correct robots.txt without guessing the syntax — including per-crawler control over the AI bots most generators still ignore. Runs entirely in your browser.

AI crawlers

Allow them to be citable in AI answers; block them to keep your content out of training data.

Your robots.txt


              

                
                
                
              

              
    
            
    
          
    

        
    
      
    
    
    

    
    
      
    
        
    
          Read This First
          
    Robots.txt Controls Crawling, Not Indexing
    
          
    This trips up almost everyone. Blocking a URL in robots.txt stops crawlers from requesting it — but if other sites link to that URL, Google can still index it and show it without a description.
    
          
    Worse: if you block a page, Google can't read the noindex tag on it either, so blocking actively prevents removal. To take a page out of the index, leave it crawlable and add noindex, or return a 410.
    
        
    
        
    
          
    The mistake that erases traffic
    
          
    A single stray Disallow: / left over from a staging site will deindex an entire domain. It's one of the first things we check in an audit, and we still find it on live sites. After you upload, always open yourdomain.com/robots.txt and read it.
    
          
    More technical SEO checks →
    
        
    
      
    
    
    

    
    
      
    
        
    FAQ
    Robots.txt Questions
    
        
    
          
    
            What is a robots.txt file?
            
    A plain text file at the root of your domain telling crawlers which paths they may request. It controls crawling, not indexing — a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, use a noindex tag and leave the page crawlable.
    
          
    
          
    
            Where do I upload robots.txt?
            
    At the root of your domain, reachable at yourdomain.com/robots.txt. A file in a subfolder is ignored. Each subdomain needs its own, served as plain text.
    
          
    
          
    
            Should I block AI crawlers like GPTBot?
            
    It depends on your goal. Blocking protects your content from training use, but can also remove you from AI answers that cite sources — visibility you may want. Most businesses that rely on being discovered should allow them; publishers protecting original work more often block them. More on AI search →
    
          
    
          
    
            Does blocking a page remove it from Google?
            
    No. Blocking prevents crawling, but a linked URL can still be indexed without a description — and Google can't read a noindex tag on a blocked page. To remove a page, allow crawling and add noindex, or return a 410.
    
          
    
        
    
      
    
    
    

    
    
      
    
        
    Not Sure What You're Blocking?
    
        
    The free audit checks your live robots.txt against what's actually indexed — and flags the rules quietly costing you traffic.
    
        Get a Free Audit →