In this video, I explain the softmax function and the concept of temperature in the context of language models and neural networks. I start by revisiting the Markov chain coding challenge, then break down how softmax converts raw neural network outputs (logits) into...
In this video, I explain the softmax function and the concept of temperature in the context of language models and neural networks. I start by revisiting the Markov chain coding challenge, then break down how softmax converts raw neural network outputs (logits) into probabilities, and how temperature can be used to adjust those probabilities making outputs more random (higher temperature) or more deterministic (lower temperature). I also briefly cover top-K and top-P sampling. Everything is demonstrated with p5.js sketches and a transformers.js language model running in the browser. Code: https://thecodingtrain.com/tracks/transformers-js/transformers-js/softmax-temperature
🚀 Watch this video ad-free on Nebula https://nebula.tv/videos/codingtrain-softmax-and-temperature-explained
p5.js Web Editor Sketches:
🕹️ Softmax and Temperature Visualization: https://editor.p5js.org/codingtrain/sketches/G39vTMSkz
🕹️ Markov Chain with Temperature: https://editor.p5js.org/codingtrain/sketches/SXqlv6-zI
🕹️ Markov Chain with Softmax and Temperature: https://editor.p5js.org/codingtrain/sketches/WNvGNuJHD
🕹️ LLM with Temperature: https://editor.p5js.org/codingtrain/sketches/36caiecXD
🎥 Previous: https://youtu.be/KR61bXsPlLU
🎥 Next: https://youtu.be/s2jm2Z22ibA
References:
📓 Softmax function on Wikipedia: https://en.wikipedia.org/wiki/Softmax_function
📓 Markov Chains explained visually: https://setosa.io/ev/markov-chains/
📓 Web Trigrams by Chris Harrison: https://www.chrisharrison.net/index.php/Visualizations/WebTrigrams
📓 Euler's number on Wikipedia: https://en.wikipedia.org/wiki/E_(mathematical_constant)
📓 p5.js exp() reference: https://p5js.org/reference/p5/exp/
💻 Transformers.js Documentation: https://huggingface.co/docs/transformers.js
🤗 SmolLM2 1.7B on Hugging Face: https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B
📕 Nature of Code: Genetic Algorithms: https://natureofcode.com/genetic-algorithms/
Videos:
🚂 https://youtu.be/KR61bXsPlLU
🚂 https://youtu.be/eGFJ8vugIWA
🚂 https://youtu.be/ETphJASzYes
🚂 https://youtu.be/kwcillcWOg0
Related Coding Challenges:
🚂 https://youtu.be/eGFJ8vugIWA
🚂 https://youtu.be/s2jm2Z22ibA
Timestamps:
0:00:00 Hello!
0:02:50 Markov chains as language models
0:07:48 From logits to probabilities
0:10:47 The softmax formula
0:14:18 Implementing softmax in p5.js
0:15:59 Weighted selection (sampling)
0:20:20 Temperature
0:23:02 Applying softmax and temperature to the Markov chain
0:28:01 Temperature with a Transformers.js language model
0:31:55 Top-K and Top-P sampling
0:33:38 Goodbye!
Editing by Mathieu Blanchette
Animations by Jason Heglund
Music from Epidemic Sound
🚂 Website: https://thecodingtrain.com/
👾 Share Your Creation! https://thecodingtrain.com/guides/passenger-showcase-guide
🚩 Suggest Topics: https://github.com/CodingTrain/Suggestion-Box
💡 GitHub: https://github.com/CodingTrain
💬 Discord: https://thecodingtrain.com/discord
💖 Membership: http://youtube.com/thecodingtrain/join
🛒 Store: https://standard.tv/codingtrain
🖋️ Twitter: https://twitter.com/thecodingtrain
📸 Instagram: https://www.instagram.com/the.coding.train/
🎥 https://www.youtube.com/playlist?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH
🎥 https://www.youtube.com/playlist?list=PLRqwX-V7Uu6Zy51Q-x9tMWIv9cueOFTFA
🔗 p5.js: https://p5js.org
🔗 p5.js Web Editor: https://editor.p5js.org/
🔗 Processing: https://processing.org
📄 Code of Conduct: https://github.com/CodingTrain/Code-of-Conduct
This description was auto-generated. If you see a problem, please open an issue: https://github.com/CodingTrain/thecodingtrain.com/issues/new
#softmax #temperature #languagemodels #machinelearning #transformersjs #markovchain #ngram #logits #probability #weightedselection #sampling #topk #topp #neuralnetworks #eulersnumber #javascript #p5js #transformersjs
