System Security: Understanding the Ring Model (Part 1)

Share:

Part 1 of the System Security Series. Deep dive into OS architecture, Protection Rings, and how User Mode interacts with Kernel Mode.

os-security kernel protection-rings

Who Watches the Watchmen?

Your operating system (Windows, Linux, macOS) is the master controller of your hardware. It decides which process gets CPU time, which app gets to read the hard drive, and which user can delete files.

But if the OS controls everything, what stops a malicious app (like a virus) from simply telling the OS, “Delete System32” or “Send me the admin passwords”?

The answer lies in Privilege Levels, visually represented as Protection Rings.

---

The Ring Model (x86 Architecture)

The CPU hardware itself enforces security. It divides code execution into four concentric rings, numbered 0 to 3.

• Ring 0 (The Kernel): Absolute Power. Code here can run any CPU instruction and access any memory address. This is where the OS Kernel and device drivers live.
• Ring 1 & 2: Reserved for device drivers (rarely used in modern OSs).
• Ring 3 (User Mode): Zero Power. Code here cannot touch hardware directly. It cannot access memory belonging to other apps. This is where Chrome, Word, and your games live.

The Barrier

If a program in Ring 3 tries to run a privileged instruction (like “disable interrupts” or “access hardware”), the CPU hardware throws an exception (a “General Protection Fault”). The OS catches this and kills the program. This is why your browser crashing doesn’t crash your whole computer.

---

Crossing the Border: System Calls

So, if Ring 3 apps have no power, how do they write to a file or display an image? They must ask the Kernel nicely.

This mechanism is called a System Call (Syscall).

1. Request: Chrome wants to write to a file. It puts the request in a specific CPU register.
2. The Switch: It executes a special instruction (SYSCALL in x64 or INT 0x80 in legacy Linux) that triggers a context switch.
3. Validation: The CPU jumps to Ring 0. The Kernel checks: Does Chrome have permission to write to this file?
4. Execution: If yes, the Kernel writes the data to the disk.
5. Return: The Kernel switches the CPU back to Ring 3 and returns the result to Chrome.

The Attack Vector: System Calls are the gateways. If there is a bug in the Kernel code that handles the Syscall, a malicious User Mode app can exploit it to trick the Kernel into doing something it shouldn’t—like granting Admin privileges. This is called Privilege Escalation.

---

Memory Protection: ASLR and DEP

Modern Operating Systems don’t rely on Rings alone. They use aggressive randomization and memory tagging.

1. DEP / NX (Data Execution Prevention)

Imagine a “No Homers” club, but for code. DEP marks areas of memory as either “Executable” (Code) or “Writable” (Data).

• You can write data to the stack, but you can’t execute it.
• This stops classic Buffer Overflow attacks where hackers inject shellcode into variables.

2. ASLR (Address Space Layout Randomization)

In the old days, system libraries (like system()) were always at the same memory address. Hackers hardcoded these addresses into their malware.

• ASLR shuffles the memory locations of key data areas and libraries every time you reboot or launch a program. The hacker is shooting in the dark.

The Continuous War

Attackers invent ways to bypass these (ROP chains, Heap Spraying), and defenders invent new shields (CFG, PAC). In this series, we will study these wars.

Next Part: We will explore Buffer Overflows—how they work, how they bypass protections, and how to exploit them in a controlled lab environment.

Discussion