How to get your articles indexed by Google, Bing, and AI search
Getting indexed fast comes down to four pieces of plumbing: a working sitemap referenced from a clean robots.txt, a Request Indexing in Search Console per article, IndexNow so Bing (and every AI engine that rides on Bing) hears about the URL the minute you publish, and robots.txt allowances for AI search agents so answer engines are permitted to cite you. Verify the foundation first — it takes ten seconds:
curl -s https://yoursite.com/robots.txt curl -sI https://yoursite.com/sitemap_index.xml | head -3 # You want: readable rules, a Sitemap: line, and HTTP 200 on the sitemap. # Everything else in this guide is throttled until these pass.
Below: the one-time setup, the three-minute per-article routine, and the AI-search layer most checklists skip entirely.
The foundation: a sitemap search engines can actually read
Indexing problems are usually discovery problems wearing a disguise. Before touching anything clever, confirm three boring facts: your robots.txt parses (a malformed file — doubled directives, word-processor characters — can degrade crawling site-wide), your XML sitemap returns 200, and robots.txt points to it. On WordPress with an SEO plugin, the real sitemap is usually an index (e.g. /sitemap_index.xml) — submit that URL in Search Console, not a guessed /sitemap.xml.
A known-good robots.txt (copy, adapt, verify)
Most robots.txt damage comes from three sources: doubled directives pasted in a hurry, invisible characters from editing in a word processor, and blocking bots you actually wanted. Here's a complete, sane WordPress baseline — permissive to search, explicit about AI search agents, sitemap declared:
# Search engines: crawl everything except admin User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # AI answer engines (live search + citations) — allowed User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot Allow: / # Optional: block AI TRAINING crawlers without affecting citations. # Uncomment if that's your policy: # User-agent: GPTBot # User-agent: ClaudeBot # User-agent: Google-Extended # User-agent: CCBot # Disallow: / Sitemap: https://yoursite.com/sitemap_index.xml
Check your robots.txt right here
Open yoursite.com/robots.txt in a browser (or run curl -s yoursite.com/robots.txt), paste the contents below, and hit Check. Everything runs in your browser — nothing is uploaded anywhere.
If the checker flags something, fix the file, save as plain UTF-8, re-upload to your web root, and confirm with curl. Then move on — the rest of the pipeline assumes this passes.
Google: the legitimate fast lane
For each new article, open Search Console → URL Inspection, paste the URL, and click Request Indexing. That queues a priority crawl — typically same-day to a few days. It's the only sanctioned way to nudge Google per-URL, and it's enough when the foundation above is healthy.
Bing + IndexNow: the back door into AI search
Here's the part most site owners skip, and it's the highest-leverage move on this page: Bing's index feeds ChatGPT Search, Microsoft Copilot, and DuckDuckGo. If your articles aren't in Bing, a large share of AI answer traffic can't ever cite you — regardless of how well Google treats you.
- Bing Webmaster Tools: sign in and use Import from Google Search Console — one click, no DNS records. Submit the same sitemap index.
- IndexNow: a shared protocol that pings participating engines the instant a URL is published or updated. On WordPress, Rank Math's Instant Indexing module (Dashboard → Modules) enables it with an auto-generated key — after that, every publish notifies Bing within minutes, automatically. Yoast and dedicated IndexNow plugins do the same job.
Set it once and the per-article cost is zero. This is the closest thing indexing has to a free lunch.
The AI search layer: agents, permissions, and llms.txt
AI platforms touch your site with two different kinds of bots, and conflating them causes expensive mistakes. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data for model training. Search-time agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages live to answer a user's question — these are the ones that produce citations and referral clicks. You can block training and still welcome citations, or allow both; just don't block the search agents while hoping AI assistants recommend you:
The full example above already grants these agents access — run your live file through the checker to confirm yours does too.
Then give assistants a map. llms.txt is an emerging convention — a plain-text index at your site root that points AI systems at your best pages by intent. It costs ten minutes and reads like this:
# Your Company Name > One-sentence description of what you do and where. ## Best pages by intent - Hire us: https://yoursite.com/ - Service X: https://yoursite.com/service-x/ ## Technical resources - [Article topic]: https://yoursite.com/article-slug/
Keep it ruthlessly concrete — real URLs, no placeholders. An unfinished llms.txt with [insert link] lines tells an AI system exactly nothing.
The three-minute per-article routine
- Publish, then Request Indexing in Search Console.
- IndexNow fires automatically (verify once in the plugin's log).
- Internal-link pass: add links to the new article from one or two established, frequently-crawled pages. Crawlers discover and prioritize URLs through links from pages they already visit often — a paginated blog index is a weak discovery path on its own.
- Append the article to
llms.txt. - Distribute (social, newsletter) — external paths compound discovery.
Then verify instead of hoping: a day or two later, URL Inspection should read "URL is on Google," and Bing's Site Explorer shows its side. Weekly, glance at the Pages report's Discovered – currently not indexed bucket — new articles piling up there means crawl priority is your bottleneck, which internal links fix.
The mistakes that keep articles invisible
- Broken foundation, clever tactics. A dead sitemap or malformed robots.txt quietly throttles everything downstream. Ten seconds of curl before anything else.
- Ignoring Bing. In the AI era, skipping Bing Webmaster Tools means opting out of ChatGPT Search and Copilot citations entirely.
- Blocking the wrong bots. Blanket-blocking every AI user-agent blocks the search agents that would have cited you. Decide training and citations separately.
- Publishing orphans. No internal links pointing at the new article means slow discovery and a weak importance signal, sitemap or not.
- Leftover noindex. Staging imports and template swaps sometimes carry a stray noindex. If a page won't index, check its meta robots first — per Google's own recrawl documentation, no amount of requesting overrides an explicit exclusion.
- Panicking at "Crawled – currently not indexed." Feeds, date archives, pagination, and asset files living in that bucket is Google working correctly. Act only when real content pages land there.
- Verify robots.txt + sitemap with curl before any indexing tactic — the foundation gates everything.
- Request Indexing in GSC is the only sanctioned per-URL fast lane for Google; the Indexing API is not for articles.
- Bing Webmaster Tools + IndexNow puts you in the index that feeds ChatGPT Search, Copilot, and DuckDuckGo — automatically, forever.
- Allow AI search agents in robots.txt and maintain a concrete llms.txt; training bots are a separate decision.
- Internal links from established pages are a discovery mechanism, not just an authority one — never publish an orphan.
Want your whole site's crawl and indexing audited?
Sitemaps, redirects, index bloat, and the AI-visibility layer are part of every SEO engagement we run — with a report you can actually act on.
FAQ
How long does it take Google to index a new article?
With a healthy sitemap, an indexing request, and internal links, typically one to three days — sometimes hours. Without those signals, discovery alone can take weeks, and low-priority pages may sit in the crawl queue indefinitely.
Does IndexNow submit to Google?
No — Google runs its own pipeline and doesn't participate in IndexNow. Use Request Indexing for Google and IndexNow for Bing and the engines that share its ecosystem. The two together cover the market.
Does llms.txt affect Google rankings?
No. It's not a ranking signal — it's a directory for AI systems deciding which of your pages best answers an intent. Think of it as structured hospitality for answer engines, separate from SEO.
Should I block GPTBot and other AI training crawlers?
That's a business decision, not a technical one — blocking training bots doesn't affect search rankings or AI citations. What you shouldn't do is block the search-time agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want AI assistants recommending your pages.
Why does Search Console say "Crawled – currently not indexed"?
Google saw the page and chose not to index it yet. For feeds, archives, and pagination that's correct behavior. For a real content page, it usually signals thin or duplicative content — strengthen its uniqueness and internal links, then re-request.
Owner, Thomas McKee Website Design & SEO Solutions — building, hosting, and ranking websites from Springfield, Missouri since 1996. 600+ projects, six straight years Best SEO Springfield MO. What we do