Many new sites that are missing from Google have an indexing problem, not a ranking one. The checks we run before a new or rebuilt site goes live, from robots.txt and redirects to hreflang for Arabic, sourced from Google's own documentation.
On this page
A new website can be well built and still be missing from Google weeks after launch. The cause is rarely ranking. It is indexing: Google was told not to crawl the site, told not to index it, or sent to the wrong version of it.
This is our pre-launch checklist for a new or rebuilt site, each point from Google's own documentation. Google's SEO starter guide says there is "no guarantee that any particular site will be added to Google's index", so none of this promises a ranking; it removes the reasons not to index.
robots.txt and noindex do different jobs
- robots.txt controls crawling. Google's robots.txt guide calls it "not a mechanism for keeping a web page out of Google"; a blocked URL can still appear in results, without a description.
- noindex controls indexing, as a robots meta tag or X-Robots-Tag header, but only where Google can crawl. Block the page in robots.txt and "the crawler will never see the noindex rule".
The staging block that goes live
Staging is often hidden with a site-wide "Disallow: /" or noindex, and on launch day the block ships with the code. Google's site-move guide warns: "Don't forget to remove any noindex or robots.txt blocks that were only needed for the migration". Nor is the fix instant: Google generally caches robots.txt for up to 24 hours.
So we password-protect staging, as the robots.txt guide suggests, and production has nothing to remove. We check production's response headers too, since a noindex header never shows in the page source. And robots.txt must return a file or a 404: on a server error, Google stops crawling for the first 12 hours.
A sitemap, submitted
Google's sitemap rules:
- Absolute URLs of the canonical pages you want in results.
- At most 50,000 URLs or 50 MB uncompressed per file.
- An accurate lastmod date. Google ignores priority and changefreq.
- A Sitemap line in robots.txt, and a submission in Search Console's Sitemaps report.
Google calls submitting one "merely a hint". We verify Search Console as a Domain property, which covers every subdomain and protocol.
Redirects from the old site
This is where rebuilds lose traffic. The site-move guide's rules:
- Map every old URL to its new equivalent, using the old sitemap, analytics and Search Console's links report.
- Use server-side permanent (301) redirects, straight to the final URL rather than through a chain. Google says they "don't cause a loss in PageRank".
- Don't send old URLs to the home page by default; Google says that "might be treated as a soft 404". Removed content should return a 404 or 410.
- Keep the redirects "for as long as possible, generally at least 1 year".
- Use the Change of Address tool only when the domain or subdomain changes.
Even done well, expect temporary ranking fluctuation: Google says a small to medium site "can take a few weeks for most pages to move".

Canonical tags
When a page is reachable at several URLs, with or without www or with tracking parameters, Google picks one to show. Its canonical guide rates redirects and rel="canonical" as strong signals and sitemaps as a weak one, and they should agree:
- A self-referencing, absolute canonical in every page's head.
- The same URL in canonical, sitemap and internal links.
- No noindex or robots.txt to choose a canonical.
- On a bilingual site, a canonical in the same language: an Arabic page canonicalised to the English one asks Google to show the English.
hreflang for the Arabic and English versions
Google's multilingual guidance asks for a separate URL per language and no automatic redirects by language: Googlebot usually crawls from the USA without an Accept-Language header, so it may never see the Arabic. Then hreflang:
- Each version lists itself and every other version. If two pages don't point to each other, "the tags will be ignored".
- Fully qualified https URLs.
- The language first, a region optionally: en, ar or ar-AE. A country code alone is not allowed, and "ae" is not the UAE: it is the ISO 639-1 code for Avestan.
- An x-default for a language selector or a home page that redirects.
Google detects a page's language from its visible content, not hreflang or the lang attribute, so the Arabic page needs Arabic content, not just Arabic menus. The design side is in what goes wrong with bilingual Arabic and English websites.
HTTPS, with no exceptions
The canonical guide says Google prefers the HTTPS version unless something contradicts it: an invalid certificate, insecure dependencies, a redirect through HTTP or a canonical pointing to HTTP. So every HTTP URL redirects to HTTPS; sitemaps, canonicals and hreflang list only HTTPS URLs; and the certificate covers both www and the bare domain.
Do it for security and the right canonical, not ranking. Google's page experience guidance counts secure serving as one aspect, but says those beyond Core Web Vitals "don't directly help your website rank higher".
Mobile-first indexing
Google indexes and ranks "the mobile version of a site's content, crawled with the smartphone agent". The mobile page needs the same content, headings, structured data, alt text and robots meta tags as desktop. Accordions and tabs are fine; content that loads only when someone taps, swipes or types is not seen. Responsive design, which Google recommends, removes most of the risk.
JavaScript rendering
Google renders JavaScript from a queue where a page may wait "a few seconds, but it can take longer than that", and still calls server-side or pre-rendering "a great idea". What we check:
- Content, title, canonical and hreflang in the HTML the server sends.
- Real links with an href, and History API routes rather than # fragments.
- No noindex in the initial HTML of a page meant to be indexed: Google may skip rendering, so a script that removes it may not work.
- Scripts and styles not blocked in robots.txt.
- Missing pages returning a real 404 status.
Structured data, and what it really does
Structured data can make a page eligible for rich results, but Google "does not guarantee that your structured data will show up in search results", and a structured data manual action costs a page its rich results, not its ranking. Features also disappear: FAQ rich results stopped showing in Google Search on 7 May 2026.
What we add, where it is true and visible on the page:
- Organization, plus LocalBusiness with address and hours for a business customers visit.
- BreadcrumbList for the site structure.
- Product for a shop, Article for the blog.
Not star ratings for your own business: Google supports LocalBusiness and Organization review snippets only on sites that review other businesses.
Core Web Vitals, in proportion
Google's targets are LCP (loading) within 2.5 seconds, INP (responsiveness) under 200 milliseconds and CLS (visual stability) under 0.1. The page experience guidance says "Core Web Vitals are used by our ranking systems", and also that Search "always seeks to show the most relevant content, even if the page experience is sub-par". Fix a slow site before launch, but don't expect speed to rescue thin content.
On a new site the Core Web Vitals report may show no data until enough real visits build up. More in Core Web Vitals are a business metric.
The first weeks in Search Console
On launch day, inspect the home page and one page per template in both languages, request indexing for the few that matter most, and submit the sitemap. On a rebuild, keep the old sitemap submitted too, to watch indexing move across. Then, weekly:
- Page indexing. Pages you want that are blocked by robots.txt or noindex are a launch bug. "Duplicate, Google chose different canonical than user" means Google preferred another URL to your canonical. Google says not to expect every URL to be indexed, only the canonical ones.
- Performance. Impressions and clicks moving to the new URLs.
Be sceptical of shortcuts. Google says crawling "can take anywhere from a few days to a few weeks", and that asking repeatedly "won't get it crawled any faster".
Almost all of this is decided before launch. A site that passes these checks has told Google what exists, which version to show and in which language; whether it ranks then depends on content and competition, which nobody can promise. We run this list on every site we build or rebuild.
Sources
- Google, SEO starter guide: no guarantee of indexing.
- Google, Introduction to robots.txt: crawling, password protection.
- Google, Block Search indexing with noindex: noindex and robots.txt.
- Google, How Google interprets the robots.txt specification: caching, server errors.
- Google, Build and submit a sitemap: sitemap rules.
- Search Console Help, Add a website property: Domain properties.
- Google, Site moves with URL changes: migrations and redirects.
- Google, How to specify a canonical URL: canonical signals, HTTPS.
- Google, Managing multilingual and multi-regional sites: language URLs.
- Google, Localized versions of your pages: hreflang.
- Library of Congress, ISO 639-2 code list: language codes.
- Google, Understanding page experience: ranking.
- Google, Mobile-first indexing best practices: mobile parity.
- Google, Understand JavaScript SEO basics: rendering.
- Google, General structured data guidelines: rich results.
- Google, Latest documentation updates: FAQ rich results.
- Google, Review snippet structured data: review eligibility.
- Google, Understanding Core Web Vitals: thresholds.
- Search Console Help, Core Web Vitals report: field data.
- Search Console Help, Page indexing report: indexing statuses.
- Google, Ask Google to recrawl your URLs: crawl timing.
- technical SEO
- Google Search Console
- site migration
- hreflang
- Dubai



