Free Chrome extension for comprehensive Shopify SEO analysis. 50+ checks, real Core Web Vitals, site crawler, Google rank tracker, competitor comparison, internal link map, and PDF reports. No API keys, no signup, no cost.
Generate clean, AI-ready llms.txt files for your website or docs. Supports crawling, sitemaps, static builds, and framework-aware adapters (Next.js, Vite, Nuxt, Astro, Remix). Includes Markdown/MDX docs mode and robots.txt generator for LLM and search crawlers.
Omnipotent Master SEO & Security Auditor (Cyber Fusion Engine v4.0)
grandstage catalog data scraper
Site crawler used in Ethora platform as an option to import your specific business data into your AI agent chat bot.
🔍 Deliver fast, reliable web search results for LLM clients using multi-engine scraping and stealth browsing with scalable API support.
Add a description, image, and links to the site-crawler topic page so that developers can more easily learn about it.
To associate your repository with the site-crawler topic, visit your repo's landing page and select "manage topics."