Unicode URLs and Internationalised Domain Names (IDN)

Definition

Unicode URLs arise when non-ASCII characters (letters from non-Latin scripts, or Latin characters with diacritical marks) appear in URL paths or domain names. SEO handling of Unicode URLs: (1) **Percent-encoding**: non-ASCII characters in URLs should be percent-encoded (e.g., "München" becomes "M%C3%BCnchen"). Browsers display the decoded form in the address bar, but the canonical URL in HTML should use the correctly encoded form. Mixing encoded and decoded URL forms creates duplicate content if both versions are accessible. (2) **Internationalised Domain Names (IDN)**: domain names in non-Latin scripts (e.g., Arabic or Chinese characters in the domain name) are technically represented in the DNS as Punycode — `xn--` prefixed ASCII strings. The IDN form (`مثال.com`) and the Punycode form (`xn--mgbh0fb.com`) can both be used in canonical tags, but consistency is important. (3) **Canonical configuration**: all Unicode URL variants (encoded, decoded, IDN vs. Punycode) should have canonical tags pointing to one consistent canonical form. If a page is accessible at both `/münchen/` and `/m%C3%BCnchen/`, both should canonical to one consistent form. (4) **Sitemap encoding**: sitemaps must use correctly encoded URLs — non-ASCII characters in sitemap URLs should be XML-entity-encoded or percent-encoded. (5) **Hreflang in multilingual URLs**: when combining hreflang with Unicode URLs (e.g., Arabic-script URL paths for an Arabic version of the site), the hreflang `href` attributes should use the correctly percent-encoded canonical URL form.

Why it matters for SEO

Unicode URL inconsistencies create duplicate content at scale for non-Latin-script sites — the same page accessible at the decoded Unicode URL and the percent-encoded form represents two indexable URLs unless correct canonical configuration resolves them to one. For internationalised sites targeting right-to-left language markets (Arabic, Hebrew, Persian) or East Asian markets (Chinese, Japanese, Korean), Unicode URL handling is a foundational technical SEO requirement.

How DeepSEOAnalysis checks this

DeepSEOAnalysis checks the canonical tag on submitted URLs — for pages with Unicode characters in the URL, confirming that the canonical uses a consistent encoding form (either consistently encoded or consistently decoded, not mixed). For IDN domains, the audit checks whether canonical and hreflang tags use the IDN form or Punycode form consistently. The audit reports the exact canonical URL found in the HTML, making encoding inconsistencies immediately visible.