New Google Model Ranked ‘No. 1 LLM’, But There’s a Problem

A new and mysterious Gemini model appears at the top of the leaderboard, but is that the full story? I dig behind the headline to show you some anti-climactic results, give some context with leaks in the last 48 hours of diminishing returns to scaling, and add the response of Altman, OpenAI and co. The future is about to look a lot stranger.

80,000 hours Podcast + Channel: https://open.spotify.com/show/2WzJwXWBDnn4iZ7odKwDib
https://www.youtube.com/@eightythousandhours/videos

You can now gift memberships to AI Insiders (my Patreon w/ exclusive vids, network): https://www.patreon.com/AIExplained/gift

Chapters:
00:00 - Introduction
01:25 - LM Leaderboard
02:35 - Benchmarks and Leaks
05:31 - Low EQ
07:37 - Other labs have issues too though
10:31 - OpenAI claim and counter-claim
14:13 - Other news

‘There is no wall’: https://x.com/sama/status/1856941766915641580
simple-bench.com
https://x.com/vedantmisra/status/1857148554105544708
Gemini Ranking: https://lmarena.ai/?leaderboard
API not yet up: https://x.com/OfficialLoganK/status/1857106844805681153
‘Just Die Chat’: https://x.com/koltregaskes/status/1856754648146653428
https://gemini.google.com/share/6d141b742a13
Google CEO tweet: https://x.com/sundarpichai/status/1857114106928718329
Sutskever Quote: https://www.reuters.com/technology/artificial-intelligence/openai-rivals-seek-new-path-smarter-ai-current-methods-hit-limitations-2024-11-11/
Another OpenAI Staffer Leaves: https://x.com/RichardMCNgo/status/1856843040427839804
Bloomberg Report: https://www.bloomberg.com/news/articles/2024-11-13/openai-google-and-anthropic-are-struggling-to-build-more-advanced-ai?s=09
Noam Brown on what OpenAI Researchers Believe: https://x.com/polynoamial/status/1855037689533178289
Clive Chan: https://x.com/itsclivetime/status/1855704120495329667
Chollet Responds to Altman: https://x.com/fchollet/status/1857060079586975852
https://x.com/sama/status/1856940152460869718
Altman Emails: https://x.com/TechEmails/status/1857285960997712356
Change of Heart: https://sd11.senate.ca.gov/news/senator-wiener-responds-openai-opposition-sb-1047
Amodei on ‘Empirical Regularities’: https://lexfridman.com/dario-amodei-transcript/
Verge Report: https://www.theverge.com/2024/10/25/24279600/google-next-gemini-ai-model-openai-december
OpenAI Agents in January: https://www.bloomberg.com/news/articles/2024-11-13/openai-nears-launch-of-ai-agents-to-automate-tasks-for-users?srnd=phx-ai


The 8 Most Controversial Terms in AI: https://imp.i384100.net/m57g3M

Non-hype Newsletter: https://signaltonoise.beehiiv.com/

Podcast: https://aiexplainedopodcast.buzzsprout.com/

I use Descript to edit my videos: https://get.descript.com/ldgxfuj2bhnb