An aws engineer discovered a 50% regression in postgres throughput while testing the new Linux 7.0 kernel. The cause turns out to be massive TLB and page faults exacerbated by Postgres process-based design. In this backend engineering show episode I dive deep into how this...
An aws engineer discovered a 50% regression in postgres throughput while testing the new Linux 7.0 kernel. The cause turns out to be massive TLB and page faults exacerbated by Postgres process-based design.
In this backend engineering show episode I dive deep into how this was discovered, the root cause and the possible fixes and workarounds.
Intermediate and Advanced Backend Engineering Course Bundle
https://courses.husseinnasser.com/bundle
My Book, Root Cause: Stories and Lessons from Two Decades of Backend Engineering Bugs
https://amzn.to/4cKfZhe
0:00 Intro
2:30 The Discovery
6:30 Spinlocks
9:25 Preemption
13:00 Root Cause
17:00 How Postgres Processes exacerbated the problem
22:30 Is the fix easy?
25:50 Summary
Stay Awesome,
Hussein
A discussion about why many engineers still love the struggle, the mistakes, and the process of figuring things out themselves. This is how we grow and get better and stronger. Letting AI do everything (even though it can’t) robs us this feeling.. Intermediate and Advanced...
A discussion about why many engineers still love the struggle, the mistakes, and the process of figuring things out themselves. This is how we grow and get better and stronger. Letting AI do everything (even though it can’t) robs us this feeling..
Intermediate and Advanced Backend Engineering Course Bundle
https://courses.husseinnasser.com/bundle
My Book, Root Cause: Stories and Lessons from Two Decades of Backend Engineering Bugs
https://amzn.to/4cKfZhe
My Book, Root Cause: Stories and Lessons from Two Decades of Backend Engineering Bugs paperback https://amzn.to/4cKfZhe ebook https://amzn.to/4cfQjJj I wrote a new book that has been in the works for years. It is called Root Cause, and it is for those who enjoy the art of...
My Book, Root Cause: Stories and Lessons from Two Decades of Backend Engineering Bugs
paperback
https://amzn.to/4cKfZhe
ebook
https://amzn.to/4cfQjJj
I wrote a new book that has been in the works for years. It is called Root Cause, and it is for those who enjoy the art of backend engineering.
Early in my career, 20 years ago, I built backend and database applications without fully grasping their inner mechanics. Performance issues, race conditions, bugs, and even data corruption often left me lost.
Since that day, I resolved to truly understand how systems work. From networking protocols and intermediary proxies to backend services and various database engines. I made it a habit to follow every request on its journey through the dark alleys of the network, down to the bowels of the database engine, meanwhile interacting with various kernel data structures in the process at every hop, and back.
I became obsessed with understanding what happens behind the scenes in software. Not just what breaks, and how but also why and what was the source of the bleed.
Root Cause is a collection of the most interesting bugs I encountered, ranging from performance bottlenecks and non-deterministic crashes to subtle data inconsistencies and incorrect results.
This book is for anyone curious about how production backend systems really behave under pressure, and how to debug them when they don’t. Even when you don’t have access to the source code.
Root cause consists of 15 chapters, each is a story about a backend bug, with investigation, diagrams, a section of a fundamental concept until the root cause is revealed.
Intermediate and Advanced Backend Engineering Course Bundle
https://courses.husseinnasser.com/bundle
Backend, Database, OS Courses
https://courses.husseinnasser.com
Fundamentals of Database Engineering
https://database.husseinnasser.com
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Fundamentals of Backend Engineering
https://backend.win
Follow me on Substack
https://hnasr.substack.com
Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
Fundamentals of Operating Systems Course https://oscourse.win NodeJS Internals and Architecture https://node.win Backend Troubleshooting Course https://performance.husseinnasser.com Fundamentals of Backend Engineering https://backend.win Fundamentals of Networking for...
Fundamentals of Operating Systems Course
https://oscourse.win
NodeJS Internals and Architecture
https://node.win
Backend Troubleshooting Course
https://performance.husseinnasser.com
Fundamentals of Backend Engineering
https://backend.win
Fundamentals of Networking for Effective Backends udemy course (link redirects to udemy with coupon)
https://network.husseinnasser.com
Fundamentals of Database Engineering udemy course (link redirects to udemy with coupon)
https://database.husseinnasser.com
Follow me on Medium
https://medium.com/@hnasr/membership
Introduction to NGINX (link redirects to udemy with coupon)
https://nginx.husseinnasser.com
Python on the Backend (link redirects to udemy with coupon)
https://python.husseinnasser.com
Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join
Buy me a coffee if you liked this
https://www.buymeacoffee.com/hnasr
Arabic Software Engineering Channel
https://www.youtube.com/channel/UChWZsjdoRvZ0T9QWZOD6UpA
🔥 Members Only Content
https://www.youtube.com/playlist?list=UUMO_ML5xP23TOWKUcc-oAE_Eg
🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com
💾 Database Engineering Videos
https://www.youtube.com/playlist?list=PLQnljOFTspQXjD0HOzN7P2tgzu7scWpl2
🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068
Stay Awesome,
Hussein
