Morrnaire
Book a ConsultationMorrnaire
Book a ConsultationA robots.txt generator creates the robots.txt file that tells Google and other search engines which pages of your Nigerian business website to crawl and index — essential for controlling your SEO and preventing duplicate content issues.
Start by selecting a CMS preset — WordPress, Next.js, Shopify, WooCommerce, or Laravel — to auto-fill common crawl rules for that platform, then customise the user-agent, disallow, and allow paths for each rule. Add a sitemap URL so Googlebot and Baidu know where to find your pages, and optionally set a crawl delay if your Nigerian shared hosting plan needs a breather between bot requests. Once your rules are configured, copy the generated text and upload it as robots.txt to the root of your website.
Every Nigerian website that wants to optimise its search engine crawl budget should have a robots.txt. It tells Googlebot, Baidu, Yandex, and other crawlers which parts of your site to index and which to ignore — preventing them from wasting bandwidth on admin pages, shopping carts, or API endpoints. For a Lagos e-commerce store running WooCommerce on shared hosting, blocking /cart/ and /my-account/ ensures Google focuses on product pages instead of burning through your monthly bandwidth allowance.
Upload it to the root directory of your domain — for example, https://www.yourdomain.com/robots.txt. If you use a hosting provider like Netlify or Vercel, place it in the public folder of your project so it is served at the root path.
Robots.txt instructs well-behaved crawlers not to access those pages, but it does not guarantee they will not be indexed. For truly sensitive content — like admin dashboards or customer data — always add authentication or password protection alongside your robots.txt rules.
Crawl-delay tells search engine bots how many seconds to wait between successive requests to your server. This is essential for shared hosting plans common among Nigerian small businesses — providers like Whogohost, SmartWeb, and QServers often limit concurrent connections, and aggressive crawling can slow down the site for real visitors or trigger bandwidth overage charges.
Google, Bing, Yahoo, and Baidu all respect robots.txt directives. Yandex is also common for Nigerian sites with Russian traffic. However, some lesser-known crawlers and malicious bots may ignore them entirely. For comprehensive protection, combine robots.txt with a web application firewall like Cloudflare and proper server-level access controls.
The Clean-param directive tells Google to ignore tracking parameters like utm_source, gclid, and fbclid when crawling your URLs. This prevents the same page from being crawled multiple times with different query strings, saving your crawl budget and reducing server load.
Generate SEO meta tags and JSON-LD for your site.
Generate structured data for Google rich results.
Generate privacy policies for legal compliance.
Generate favicons and PWA icons for your site.
Generate structured data to complement your sitemap.