• The way content and businesses are discovered online is changing rapidly. • In the past, traffic originated from traditional search engines, and SEO determined who got found first. • Now the traffic is increasingly coming from AI crawlers and agents that demand structured data within the often-unstructured Web that was built for humans. • As a business, to continue to stay ahead, now is the time to consider not just human visitors, or traditional wisdom for SEO-optimization, but start to treat agents as first-class citizens. Why markdown is important Feeding raw HTML to an AI is like paying by the word to read packaging instead of the letter inside. • A simple ## About Us on a page in markdown costs roughly 3 tokens; its HTML equivalent â About Us â burns 12-15, and that’s before you account for the wrappers, nav bars, and script tags that pad every real web page and have zero semantic value. • This blog post youâre reading takes 16,180 tokens in HTML and 3,150 tokens when converted to markdown.

Article Summaries:

  • The way content and businesses are discovered online is changing rapidly. In the past, traffic originated from traditional search engines, and SEO determined who got found first. Now the traffic is increasingly coming from AI crawlers and agents that demand structured data within the often-unstructured Web that was built for humans. As a business, to continue to stay ahead, now is the time to consider not just human visitors, or traditional wisdom for SEO-optimization, but start to treat agents as first-class citizens. Why markdown is important Feeding raw HTML to an AI is like paying by the
  • Cloudflare has launched “Markdown for Agents,” a feature that lets AI systems request web pages in markdown instead of raw HTML. By adding an Accept: text/markdown header, agents receive a lightweight, token‑efficient markdown version of the content, cutting token usage by up to 80 % (e.g., a 16,180‑token HTML page becomes 3,150 tokens). The network fetches the original HTML from the origin, converts it in real time, and serves the markdown to the client. Developers can test the feature with a simple curl command or via a fetch call in Workers, and the response includes a x‑markdown‑tokens header. This streamlines AI pipelines by eliminating costly HTML parsing and intent analysis.

Sources: