WebDigga's Web and Site Audit Crawlers

WebDigga crawls the web in order to identify and classify web content, to discover backlinks, and find related sites. We will not index anything you would like to remain private.

Does WebDigga follow the robots.txt file?


What if I don't want WebDigga to crawl my particular page?

To prevent webdiggabot from visiting a particular page, include Robots <meta> tag which should look:

<meta name="robots" content="noindex, nofollow">

The webdiggabot crawler strictly adheres to Robots <meta> tag rules.

Can I add my site to WebDigga?

Yes, click here to add your site.

For more information regarding robots, meta, crawling, and robots.txt visit the Web Robots Pages at Robotstxt.org, an excellent source for the latest information on the Standard for Robots Exclusion.