moz.com Open in urlscan Pro
2606:4700::6811:325f  Public Scan

Form analysis
2 forms found in the DOM

/search

<form class="mzr-mobile-search" action="/search">
  <input type="text" class="mzr-input-search" name="q" placeholder="Search...">
  <button aria-labelledby="mobile-menu-search">
    <svg class="mzr-icon-search" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 18 18">
      <title id="mobile-menu-search">Search</title>
      <path d="M18 16.94l-5.553-5.554C13.417 10.186 14 8.66 14 7c0-3.86-3.14-7-7-7S0 3.14 0 7s3.14 7 7 7c1.66 0 3.185-.584 4.386-1.553L16.94 18 18 16.94zM7 12.5c-3.032 0-5.5-2.467-5.5-5.5S3.968 1.5 7 1.5s5.5 2.467 5.5 5.5-2.468 5.5-5.5 5.5z"></path>
    </svg>
  </button>
</form>

/help/search

<form action="/help/search" class="my-4">
  <input type="text" name="q" placeholder="Search..." class="form-control form-control-sm">
</form>

Text Content

Skip to content
Moz logo Menu open Menu close
Search
 * Products
    * Moz Pro
    * Moz Pro Home
    * Moz Local
    * Moz Local Home
    * STAT
    * Mozscape API

 * Free SEO Tools
    * Link Explorer
    * Keyword Explorer
    * Competitive Research
    * Domain Analysis
    * MozBar
    * More Free SEO Tools

 * Learn SEO
    * Beginner's Guide to SEO
    * SEO Learning Center
    * Moz Academy
    * SEO Q&A
    * Webinars, Whitepapers, & Guides

 * Blog
 * Why Moz
    * Agency Solutions
    * Enterprise Solutions
    * Small Business Solutions
    * Case Studies
    * The Moz Story
    * New Releases

 * Log in
 * Log out

 * Products
    * Moz Pro
      
      YOUR ALL-IN-ONE SUITE OF SEO TOOLS
      
      The essential SEO toolset: keyword research, link building, site audits,
      page optimization, rank tracking, reporting, and more.
      
      Learn more
      Try Moz Pro free
      
    * Moz Local
      
      COMPLETE LOCAL SEO MANAGEMENT
      
      Raise your local SEO visibility with easy directory distribution, review
      management, listing updates, and more.
      
      Learn more
      Check my presence
      
    * STAT
      
      ENTERPRISE RANK TRACKING
      
      SERP tracking and analytics for SEO experts, STAT helps you stay
      competitive and agile with fresh insights.
      
      Learn more
      Book a demo
      
    * Mozscape API
      
      THE POWER OF MOZ DATA VIA API
      
      Power your SEO with the proven, most accurate link metrics in the
      industry, powered by our index of trillions of links.
      
      Learn more
      Get connected
      
    * Compare SEO Products

 * Free SEO Tools
    * Link Explorer
      
      POWERFUL BACKLINK DATA FOR SEO
      
      Explore our index of over 40 trillion links to find backlinks, anchor
      text, Domain Authority, spam score, and more.
      
      Get link data
      
    * Keyword Explorer
      
      THE ONE KEYWORD RESEARCH TOOL FOR SEO SUCCESS
      
      Discover the best traffic-driving keywords for your site from our index of
      over 500 million real keywords.
      
      Search keywords
      
    * Competitive Research
      
      COMPETITIVE INTELLIGENCE TO FUEL YOUR SEO STRATEGY
      
      Gain intel on your top SERP competitors, keyword gaps, and content
      opportunities.
      
      Find competitors
      
    * Domain Analysis
      
      FREE DOMAIN SEO ANALYSIS TOOL
      
      Get top competitive SEO metrics like Domain Authority, top pages, ranking
      keywords, and more.
      
      Analyze domain
      
    * MozBar
      
      FREE, INSTANT SEO METRICS AS YOU SURF
      
      Using Google Chrome, see top SEO metrics instantly for any website or
      search result as you browse the web.
      
      Try MozBar
      
    * More Free SEO Tools

 * Learn SEO
    * Beginner's Guide to SEO
      
      The #1 most popular introduction to SEO, trusted by millions.
      
      Read the Beginner's Guide
    * How-To Guides
      
      Step-by-step guides to search success from the authority on SEO.
      
      See All SEO Guides
    * SEO Learning Center
      
      Broaden your knowledge with SEO resources for all skill levels.
      
      Visit the Learning Center
    * Moz Academy
      
      Upskill and get certified with on-demand courses & certifications.
      
      Explore the Catalog
    * On-Demand Webinars
      
      Learn modern SEO best practices from industry experts.
      
      View All Webinars
    * SEO Q&A
      
      Insights & discussions from an SEO community of 500,000+.
      
      Find SEO Answers
   
   July 11-13, 2022
   
   JOIN US IN SEATTLE FOR MOZCON
   
   Snag tickets
 * Blog
 * Why Moz
    * Small Business Solutions
      
      Uncover insights to make smarter marketing decisions in less time.
      
      Grow Your Business
    * The Moz Story
      
      Moz was the first & remains the most trusted SEO company.
      
      Read Our Story
    * Agency Solutions
      
      Earn & keep valuable clients with unparalleled data & insights.
      
      Drive Client Success
    * Case Studies
      
      Explore how Moz drives ROI with a proven track record of success.
      
      See What's Possible
    * Enterprise Solutions
      
      Gain a competitive edge in the ever-changing world of search.
      
      Scale Your SEO
    * New Releases
      
      Get the scoop on the latest and greatest from Moz.
      
      See What’s New
   
   New Feature: Moz Pro
   
   SURFACE ACTIONABLE COMPETITIVE INTEL
   
   Learn More

 * 
 * Log in
 *  * Moz Pro
    * Moz Local
    * Moz Local Dashboard
    * Mozscape API
    * Mozscape API Dashboard
    * Moz Academy

 * Avatar
    * Moz Home
    * Notifications
    * Account & Billing
    * Manage Users
    * Community Profile
    * My Q&A
    * My Videos
    * Log Out


 * The Help Hub The Help Hub

 * What's New
 * Contact Help

 * What's New
   
 * Contact Help
   

Help Home Your Account
Getting Started
Activate Your Account Moz Pro Free Trial Your Moz Community Profile Export
Notifications
Manage Subscriptions
Cancel Moz Pro Base Plan Update Your Moz Pro Base Plan Purchasing Subscription
Add-Ons Removing Subscription Add-Ons Cancel Local Market Analytics Upgrade or
Downgrade Local Market Analytics Cancel Moz Local Update Your Moz Local
Subscription Manual Invoicing Non-Profit Subscriptions Managing Multiple
Subscriptions Moz Pro Plan & Pricing Changes
Preferences
Billing Info Email Settings Sharing User Access Troubleshooting Shared User
Access Shared User Permissions Change Account Owner Delete Your Moz Account
Update Billing & Account Information
How do I update my credit card? How do I remove my credit card? Can I transfer
my subscription to another account? I forgot my password How do I change my
password? How do I change the owner of my Moz account? How do I update my Moz
email or user ID? That email address already exists
Payment & Billing Issues
Why was my card declined at checkout? Why has Moz charged my card? Why am I
getting an error when signing up for a free trial? Why did my payment fail? Why
is my subscription suspended? How do I reactivate my Moz subscription? Accepted
Methods of Payment Sales Tax
Invoices & Receipts
Where do I find my billing history? Where do I find my invoices? How do I
download my invoice as a PDF? I need a receipt for my subscription Can I pay by
invoice? Where can I find a copy of your W-9?
Moz Pro Campaigns
Getting Started
Track a Site Troubleshooting Setting Up a Tracked Site Moz Pro Checklist Connect
Google Analytics Troubleshooting Google Analytics Connection Insights Campaign
Dashboard
Manage Campaigns
Campaign Settings Brand Rules Manage Your Campaigns Removing Campaigns Export
Campaign Data
Rankings
Rankings Overview Search Visibility Investigating Changes In Search Visibility
Track Competitors' Rankings Add Keywords Delete Keywords Track Local Keywords
Add Keyword Labels Remove Keyword Labels Analyze a Keyword SERP Features Landing
Pages Exporting Rankings to CSV
Page Optimization
Page Optimization Overview Troubleshooting Page Optimization
Site Crawl
Site Crawl Overview All Crawled Pages View Affected Pages Page Details Filter &
Sort Ignoring Crawl Issues Critical Crawler Issues Crawler Warnings Redirect
Issues Metadata Issues Content Issues Moz Can't Crawl Your Site Troubleshooting
Pages Not Crawled Fluctuations in Pages Crawled Spikes in Site Crawl Issues
Performance Metrics
Links
Links Overview Inbound Links Linking Domains Anchor Text Top Pages Discovered &
Lost Spam Score
Site Traffic
Site Traffic Overview Traffic Not Matching GA
Reporting
Custom Reports
Workflows
Identify Long-Tail Keywords See Rankings for Specific Pages Segmenting Keyword
Data Monitoring Local Markets
Competitive Research
True Competitor
True Competitor Overview
Keyword Gap
Keyword Gap Overview
Keyword Research
Getting Started
Keyword Explorer Overview Add to Campaign
Explore By Site
Site Overview Ranking Keywords
Explore by Keyword
Keyword Overview Keyword Suggestions SERP Analysis SERP Features
Keyword Lists
Keyword Lists Overview Rank Checking
Keyword Metrics
Monthly Volume Difficulty Organic CTR Priority My Score
Workflows
Investigating Keyword Cannibalization Find Your Difficulty Benchmark
Link Research
Getting Started
Link Explorer Overview How We Index The Web
Link Explorer
Inbound Links Linking Domains Anchor Text Top Pages Discovered & Lost Compare
Link Profiles Spam Score Link Intersect Link Tracking Lists Export Link Data Moz
Isn’t Finding Your Links Troubleshooting Lost Links Investigating Spam Score
Additional Research Tools
On-Page Grader
On-Page Grader Overview Troubleshooting On-Page Grader
On-Demand Crawl
On-Demand Crawl Overview
Rank Checker
Rank Checker Overview
MozBar
MozBar Overview Troubleshooting MozBar Page Optimization With MozBar
Domain Analysis
Domain Analysis Overview
Moz Local
Getting Started
Check Presence Getting Started Checklist Purchase a Single Location Purchase
Locations in Bulk Connect Google Business Profile Connect Facebook Service Area
Businesses (SABs) Troubleshooting Connecting Google Business Profile
Troubleshooting Connecting Facebook Settings & Digest Email
Manage Locations
Local Dashboard Profile Profile Completeness Data Cleansing Directories
Categories Customer Feedback (Reviews) Publish (Social Posting) Location Filters
Location & Sync Status Managing Multiple Locations
Links API
Getting Started
Overview & Authentication for Links API V2 Links API V1 vs V2 Get a Key Moz API
Legal
Making Calls With Links API V2
Anchor Text Final Redirect Global Top Pages Global Top Root Domains Index
Metadata Link Intersect Link Status Linking Root Domains Links Top Pages URL
Metrics Usage Data Weighted Endpoints Moz API for Sheets
Links API V1 Endpoints & Authentication
Links API V1 Authentication V1 Anchor Text V1 Index Metadata V1 Links V1 Top
Pages V1 URL Metrics V1 Spam Score V1 API Troubleshooting V1 Response Fields V1
Example API Call V1 Best Practices V1 Free API Access
Moz Procedures
Crawlers
Rogerbot Dotbot
Glossary
Pro Glossary Local Glossary
What is a Beta?
Walkthroughs & Demos
Moz Pro Walkthroughs
What to Expect Scheduling Your Walkthrough Preparing For Your Walkthrough
Attending Your Walkthrough
Moz Academy
Getting Started
Accessing Moz Academy SEO Certifications Moz Academy Billing Issues
Troubleshooting
Troubleshooting Moz Academy Access Troubleshooting Course Access
Workflows & Troubleshooting
Workflow Guides Troubleshooting Guides Video Directory
Moz Status
Maintenance & Outages
Need More Help?
Help With Moz Tools
Contacting the Moz Help Team Moz Pro Help Moz Local Help Links API Help Moz
Billing Help Taking Screenshots
SEO Help
Where to Find SEO Help Moz Q&A Forum

RELATED RESOURCES

 * Ranking & Visibility
 * On-Site SEO
 * Links & Link Building
 * Local SEO
 * Keywords & Keyword Research
 * Mobile SEO
 * International SEO
 * Crawling & Site Audits
 * Analytics & Reporting
 * Beginner's Guide to SEO
 * Free SEO Tools
 * Google Algorithm Change History
 * Workshops & Training

 1. Help
 2. Moz Procedures
 3. Crawlers
 4. Dotbot

DOTBOT


FREQUENTLY ASKED QUESTIONS

 * What is the difference between Rogerbot and Dotbot?
 * Dotbot collects data for our Link index which is available through Link
   Explorer and our Links API. Rogerbot crawls your site for Moz Pro Site Crawl;
   if you have a Moz Pro Campaign setup we'll crawl your site weekly.
 * Why am I not seeing expected links?
 * After being found, newly discovered links have the ability to be populated
   into our index in about 3 days. It may take longer for us to discover
   backlinks to your site based on factors like crawlability of the referring
   pages, as well as the quality of the links and the referring pages. It is a
   good idea to check to see if we've indexed the referring page on which that
   link is found. If we haven't indexed the referring page yet, you won't see
   your link in our index. You can use Link Tracking Lists to help monitor the
   discovery of your backlinks. Read more about how we index the web.


WHAT'S COVERED?

In this guide you’ll learn more about our crawler, Dotbot, which is used to
crawl for the Moz Link Index that powers the Moz API, Link Explorer, and the
Links section of your Campaign . For more information regarding our site audit
crawler, Rogerbot, please see our Rogerbot guide.


QUICK LINKS

 * Moz’s Link Index Crawler
 * Block Dotbot From Certain Areas of Your Site
 * Block Dotbot From Any Part of Your Site
 * Slow Dotbot Down
 * When will Dotbot see changes to my robots.txt file?


MOZ'S LINK INDEX CRAWLER

Dotbot is Moz's web crawler, it gathers web data for the Moz Link Index. This
data we collect through Dotbot is available in the Links section of your Moz Pro
campaign, Link Explorer, and the Moz Links API. Dotbot is different from
Rogerbot, which is our site audit crawler for Moz Pro Campaigns.


WHY DOES MOZ CRAWL THE WEB?

Some of our tools, like Link Explorer, require us to crawl websites. When this
happens, the user-agent, Dotbot, is used to identify our crawler. It's good to
keep in mind that you need a Moz Pro account to access most of the information
gathered. Members of our free online marketing community have limited access. To
see an example of the type of data we collect, enter a URL in the search box for
Link Explorer.



HOW TO BLOCK DOTBOT FROM CRAWLING YOUR SITE


If you don't want Dotbot crawling your site, we always respect the standard
Robots Exclusion Protocol (aka robots.txt). If you would like to block Dotbot,
all you need to do is add our user-agent string to your robots.txt file.


BLOCK DOTBOT FROM CERTAIN AREAS OF YOUR SITE

          
User-agent: dotbot

Disallow: /admin/

Disallow: /scripts/

Disallow: /images/
        


BLOCK DOTBOT FROM ANY PART OF YOUR SITE

          
User-agent: dotbot

Disallow: /
        


SLOW DOTBOT DOWN

          
User-agent: dotbot

Crawl-delay: 10
        


WHEN WILL DOTBOT SEE CHANGES TO MY ROBOTS.TXT FILE?

Dotbot only looks at your robots.txt file the first time it encounters that site
during a new index crawl. That means if Dotbot saw it was allowed on the site
once, any changes to that permission would not be looked at until the next time
we locate links to your site.



RELATED ARTICLES

 * Rogerbot
 * Link Research
 * Links API
 * How We Index The Web

--------------------------------------------------------------------------------


WAS THIS ARTICLE HELPFUL?

Yes! Amazing! Yes! It was what I needed. Meh. It wasn’t really what I was
looking for. No, it wasn’t helpful at all.

Thanks for the feedback.

Contact the Help Team if you have a specific question.
Cancel Submit


WOO! 🎉
THANKS FOR THE FEEDBACK.


GOT IT.
THANKS FOR THE FEEDBACK.

--------------------------------------------------------------------------------


Moz logo
 * Contact
 * Community
 * Free Trial
 * Terms & Privacy
 * Jobs
 * Help
 * News & Press

 * 
 * 
 * 
 * 
 * 

Copyright 2022 © Moz, Inc. All rights reserved.