Llama 405b: Full 92 page Analysis, and Uncontaminated SIMPLE Benchmark Results

Llama 3.1 is here, and if anything, it’s paper is even more impressive. It’s like Meta want to reveal the secret sauce of LLMs. I go through the highlights of all 92 pages and test Llama 405B on the SIMPLE bench, my new private, vetted general intelligence benchmark, against GPT 4o and Turbo, Gemini 1.5 Pro and Claude 3.5 Sonnet.

Weights and Biases Link: https://wandb.me/ai_explained 

AI Insiders: https://www.patreon.com/AIExplained


Llama 3.1 Paper: https://scontent-lhr6-2.xx.fbcdn.net/v/t39.2365-6/452387774_1036916434819166_4173978747091533306_n.pdf?_nc_cat=104&ccb=1-7&_nc_sid=3c67a6&_nc_ohc=t6egZJ8QdI4Q7kNvgEv-5bU&_nc_ht=scontent-lhr6-2.xx&oh=00_AYA6GoJPv0IACGeU2etgEvigNDflXdkYS7rn8n78COiDzg&oe=66A67B0D

Llama 405B Web Release: https://ai.meta.com/blog/meta-llama-3-1/

Zuckerberg Money Run Out Interview: https://www.youtube.com/watch?v=Vy3OkbtUa5k

Zuckerberg Llama 4: https://www.youtube.com/watch?v=YuIc4mq7zMU

OpenAI Losses: https://www.theinformation.com/articles/why-openai-could-lose-5-billion-this-year?rc=sy0ihq

Nvidia Nims: https://build.nvidia.com/explore/discover?snippet_tab=Python#llama-3_1-405b-instruct

Open Source Definition: https://opensource.org/deepdive/drafts/the-open-source-ai-definition-draft-v-0-0-8

Data Disappearing: https://www.nytimes.com/2024/07/19/technology/ai-data-restrictions.html

Infinite Bench: https://arxiv.org/pdf/2402.13718

ARC-A12 Challenge: https://www.semanticscholar.org/reader/88bb0a28bb58d847183ec505dda89b63771bb495

Adversarial Examples: https://arxiv.org/pdf/1707.07328

‘Systemic Risk’: https://x.com/deanwball/status/1815826885663658445/photo/1

Zuckerberg-Hawley Letter: https://www.hawley.senate.gov/hawley-and-blumenthal-demand-answers-meta-warn-misuse-after-leak-metas-ai-model/

Groq Demo: https://x.com/JonathanRoss321/status/1815777714642858313

Altman Letter: https://www.washingtonpost.com/documents/2ea97cb4-34df-4bdd-a100-3572e93fdba1.pdf?itid=lk_inline_manual_4

Lmsys Response: https://x.com/lmsysorg/status/1816010015494529540

Scale Private Leaderboard: https://scale.com/leaderboard



AI Insiders: https://www.patreon.com/AIExplained

Non-hype Newsletter: https://signaltonoise.beehiiv.com/
GenAI Hourly Consulting: https://www.theinsiders.ai/