In this talk, I go over the rise of small language models (SLMs) and how they can benefit your business or day to day life. Talk date: March 12, 2026 at the Queensland AI Meetup. We'll look at case studies such as Sunny, an iOS application which uses a fine-tuned version of...
In this talk, I go over the rise of small language models (SLMs) and how they can benefit your business or day to day life.
Talk date: March 12, 2026 at the Queensland AI Meetup.
We'll look at case studies such as Sunny, an iOS application which uses a fine-tuned version of MedGemma to privately track skin health on-device.
We'll break down why on-device inference matters (privacy, offline access, zero ongoing cost) and compare the economics of local models versus cloud API pricing at scale.
Then we'll discuss the hardware and software optimizations required to run a model on a compute constrained device.
Finally, we'll get hands-on and fine-tune a small language model live.
We'll walk through how to build a custom dataset, set up supervised fine-tuning using Hugging Face's SFT Trainer, and fine-tune a small model, Gemma 3 270M, in about two minutes on an RTX 6000 Blackwell GPU on Google Colab.
We'll see how the base model's outputs to the fine-tuned version side by side, showing how even a small model can be customized to know specific people, handle edge cases, and refuse to answer questions it shouldn't.
Links:
Colab Notebook we used - https://dbourke.link/qldai-colab-notebook-llm-fine-tune-march-2026
End-to-end Small Language Model Fine-tuning Tutorial - https://www.learnhuggingface.com/notebooks/hugging_face_llm_full_fine_tune_tutorial
Courses I teach:
Learn AI/ML (beginner-friendly course) - https://dbourke.link/ZTMMLcourse
Learn Hugging Face - https://dbourke.link/ZTMHuggingFace
Learn TensorFlow - https://dbourke.link/ZTMTFcourse
Learn PyTorch - https://dbourke.link/ZTMPyTorch
Connect elsewhere:
Download Nutrify (my startup) - https://apple.co/4ahM7Wc
My website - https://www.mrdbourke.com
X/Twitter - https://www.twitter.com/mrdbourke
LinkedIn - https://www.linkedin.com/in/mrdbourke/
Get email updates on my work - https://dbourke.link/newsletter
Read my novel Charlie Walks - https://www.charliewalks.com
Timestamps:
0:00 - Intro
2:19 - About me
4:09 - Case study: Sunny
7:55 - Benefits of small language models running on-device
8:44 - Cost savings of on-device models
9:29 - Case study: Sunny (hardware overview)
10:55 - Current best practice for running VLMs on iPhone
12:35 - Case study: Sunny (memory usage in Xcode)
13:56 - Case study: Sunny (workflow overview)
15:25 - Jeff Dean on precision
16:46 - Precision breakdown
17:28 - Effects of quantization on model size footprint
17:48 - Saving memory by reducing token usage
19:28 - Before and after different on-device experiments
20:29 - Case studies for other small but useful language models
24:00 - Case study for private VLM-based surveillance
25:04 - Small language models features and benefits
25:47 - How to pick a model for your use case
26:06 - Question: What hardware is required for getting started?
27:21 - Prompting vs fine-tuning vs RAG
28:08 - Live LLM fine-tuning problem overview
28:50 - How I made a dataset for fine-tuning Gemma 3
33:08 - Live fine-tuning code begins in Google Colab
36:56 - Data = a guide for what you want your model to do
40:37 - Question: How do you know if your fine-tuned model is performing well?
44:24 - Comparing the base model to the fine-tuned model
53:23 - Demo'ing our fine-tuned model on Hugging Face Spaces
56:00 - Haiku
57:33 - Contact me
59:05 - Q&A
