Nobody knows this about OpenAI.

Thanks! Share it with your friends!

You disliked this video. Thanks for the feedback!

Added 3 months ago by admin

11 Views

Nobody knows this about OpenAI.

OpenAI's web crawlers are horribly inefficient.

ChatGPT spends 34% of its fetches on 404 pages.

Googlebot, in comparison, which crawls way more of the web, only spends 8% of its fetches on 404 pages.

ChatGPT spends an additional 14% of fetches on redirects.

For Googlebot, it's only 1.5%.

All the top AI crawlers can't execute JavaScript, whereas Googlebot can.

And these AI models probably rely a lot more on cached data or training data even when they've claimed to have fetched the latest information.

If you have a JavaScript heavy website, make sure you're using server side rendering.

The same actually goes for Googlebot. Even though Googlebot can execute JavaScript, it usually doesn't do so unless the website is getting a lot of traffic.

Since these AI crawlers are so inefficient, try to reduce the junk and clutter on your website.

Even if you don't have a JavaScript heavy website, make sure to put your sitemap in your robots.txt file.

And be skeptical of the information you get from AI bots because it could be out of date.

Data comes from Vercel - The rise of the AI crawler.

#openai #artificialintelligence #webdevelopment

Category: Artificial Intelligence & Business

Post your comment

Comments

Be the first to comment

Sign in

Create your account

Nobody knows this about OpenAI.

Post your comment

Comments

Up Next