klikwit
Fill in the fields below to generate a valid robots.txt file. This is a generator only — it does not change your live website.
A robots.txt file, placed at the root of a domain (e.g. example.com/robots.txt), tells well-behaved web crawlers which parts of a site they're allowed or asked not to crawl. It's a set of instructions, not an enforced lock.
It lets you point crawlers away from pages you don't want indexed — like an admin area or an internal dashboard — and can point them toward your sitemap to help them discover your content more efficiently.
Enter a user-agent (* applies the rules to all crawlers), list paths to allow and disallow, and optionally add your sitemap URL. Copy the generated text or download it, then upload it as robots.txt at the root of your site.
Disallow rules are requests, not guarantees — well-behaved crawlers (like major search engines) respect them, but robots.txt is not a security mechanism and shouldn't be relied on to keep sensitive content private.
Disallow: / when only one section should be blocked.This tool only generates text — it does not upload, validate against your live site, or verify that crawlers are respecting the rules. You're responsible for placing the file correctly on your own server.
Not reliably by itself. A disallowed page can sometimes still appear in search results (e.g. if other sites link to it) without its content being shown. Use a noindex meta tag for pages you want fully excluded.
At the root of your domain, so it's reachable at https://yourdomain.com/robots.txt — not inside a subfolder.
The asterisk (*) is a wildcard meaning "all crawlers." You can also target a specific crawler by name if needed.
No. It's a public file that crawlers may or may not respect. Never rely on it to hide sensitive or private data — use proper authentication for that.
klikwit's free URL shortener, QR codes and link analytics live right alongside these SEO tools.
Try the URL Shortener
klikwit
Free URL shortener, QR codes and SEO tools — no signup required.