Technical SEO in numbers 2026: only 48% of mobile sites pass Core Web Vitals, and AI bots reach robots.txt
91.5% of mobile pages now use HTTPS, but fewer than half pass Core Web Vitals and only 2.1% have an llms.txt file. We review Web Almanac 2025 data to see how healthy the web's technical foundation is.
By Paulina · SEO and digital strategy specialist
· 4 min read
Only 48% of mobile sites pass all three Core Web Vitals at once, compared with 56% on desktop, according to the SEO chapter of the Web Almanac 2025 by HTTP Archive. At the same time, 91.5% of mobile pages already use HTTPS, and more and more sites give instructions to artificial intelligence bots.
Technical SEO is the part of search optimization that makes sure search engines can crawl, understand and index a site: speed, security, instruction files for bots, canonical tags and structured data. In this study we measure how widespread those foundations are.
48%
Mobile sites passing Core Web Vitals
56% on desktop
91.5%
Mobile pages using HTTPS
91.7% on desktop
2.1%
Sites with an llms.txt file
mobile, 2025
4.2%
Sites naming GPTBot in robots.txt
up from 2.7% in 2024
Source: HTTP Archive, Web Almanac 2025
View the data as a table
| Mobile sites passing Core Web Vitals | 48% | 56% on desktop |
| Mobile pages using HTTPS | 91.5% | 91.7% on desktop |
| Sites with an llms.txt file | 2.1% | mobile, 2025 |
| Sites naming GPTBot in robots.txt | 4.2% | up from 2.7% in 2024 |
Methodology
The Web Almanac is an annual HTTP Archive report that analyzes millions of real websites, in their mobile and desktop versions. We used the SEO chapter of the 2025 edition, published in January 2026. Some figures refer to sites and others to pages, depending on how the report measures them; we note this in each case. Comparisons with 2024 come from the same report.
Finding 1: the basics are covered, the advanced parts are not
Almost every page has a title and HTTPS. But adoption drops for elements that take more technical work: only half use structured data, and one in five declares versions in other languages.
Use of the canonical tag, which tells Google which version of a page is the main one, rose from 65% to 67% on mobile. We cover titles and meta descriptions in detail in our on-page SEO guide.
Finding 2: Core Web Vitals remain the weak spot on mobile
Core Web Vitals measure loading (LCP), visual stability (CLS) and responsiveness (INP). On mobile, loading is the hardest metric; on desktop, almost every site passes responsiveness.
- Mobile
- Desktop
Source: HTTP Archive, Web Almanac 2025
View the data as a table
| Mobile | Desktop | |
|---|---|---|
| LCP (loading) | 62% | 74% |
| CLS (stability) | 81% | 72% |
| INP (responsiveness) | 77% | 97% |
| All three | 48% | 56% |
In our on-page SEO article we cited that in 2024 only 43% of mobile sites passed all three metrics, according to that year's Web Almanac performance chapter. The two figures come from different chapters, so the comparison is a reference rather than an exact measure of progress.
Finding 3: sites are starting to talk to AI bots
The robots.txt file tells each bot what it may crawl. Between 2024 and 2025, the number of sites naming AI companies' bots grew sharply:
- 2024
- 2025
Source: HTTP Archive, Web Almanac 2025
View the data as a table
| 2024 | 2025 | |
|---|---|---|
| GPTBot (OpenAI) | 2.7% | 4.2% |
| ClaudeBot (Anthropic) | 1.6% | 3.4% |
Mentions of GPTBot rose by about 55% and mentions of ClaudeBot nearly doubled, according to the report. Mentioning a bot doesn't always mean blocking it: the file can allow or deny access.
The llms.txt file, a proposal for giving AI models a summary of a site, is still marginal: it appears on 2.1% of sites. It's part of the same trend we analyze in our article on AI Overviews: AI-powered search is changing how traffic arrives, and sites are starting to adapt.
How to prioritize your site's technical SEO
- Check your Core Web Vitals with real-user data in Google Search Console. On mobile, loading (LCP) is the metric most sites fail.
- Make sure your robots.txt responds correctly and decide deliberately which AI bots you want to allow. Only a minority of sites do so today.
- Add structured data to your main pages: half of all pages still don't have it.
- Use the canonical tag on every page to avoid duplicate content.
- Measure the effect of each change with your web analytics tool, comparing organic traffic before and after.
Limitations of the study
- The Web Almanac measures the whole web, not a country. It doesn't break out results for Chile or Latin America.
- Sites and pages are not the same. Some metrics are calculated per site (robots.txt, llms.txt) and others per page (title, canonical).
- Core Web Vitals come from Chrome users, so they reflect their experience and not that of every browser.
What it means for you
- Mobile speed is still an advantage: more than half of mobile sites fail Core Web Vitals. Improving them puts you ahead of most.
- Deciding what to do with AI bots is now part of SEO: only about 4% of sites name GPTBot, but the figure is growing fast. Set a policy before it catches you off guard.
- Structured data still sets you apart: half of all pages don't use it, and it helps search engines and AI models understand your content.
Sources
- technical SEO
- Core Web Vitals
- robots.txt
- llms.txt
- structured data