Kernel
The privileged core of the OS: syscalls, scheduling, virtual memory, drivers.
Level: intermediate
What is it?
The kernel is the part of an operating system that runs with elevated CPU privileges and owns core mechanisms such as scheduling, virtual memory, interrupt handling, and many device interfaces. Its exact structure varies: monolithic kernels, microkernels, and hybrid designs place responsibilities at different boundaries.
Why does it matter?
Kernel code sits at a high-trust boundary. A kernel bug can crash the machine or undermine isolation for every process, so the kernel's mechanisms must validate inputs and keep shared state consistent.
Mental model
Think of the kernel as a small, highly privileged control plane. User programs submit requests; the kernel decides whether they are legal, coordinates shared resources, and records the state needed to resume work.
How it works
- A syscall entry path changes execution from user privilege to a controlled kernel entry point.
- The scheduler chooses a runnable thread and saves/restores execution state when switching contexts.
- Virtual-memory code maintains address-space mappings and handles page faults.
- Interrupt handlers react to hardware events; complex work is often deferred out of the immediate handler.
- Kernel designs trade a smaller trusted core against communication overhead and the complexity of crossing protection boundaries.
Steps
- A thread issues a syscall or receives an interrupt.
- The CPU enters a defined kernel entry path and saves enough state to handle the event safely.
- The kernel checks parameters, permissions, and current process state before changing shared resources.
- The kernel returns a result, schedules another thread, or handles a fault that cannot be recovered.
Real-world usage
- Understanding why drivers, syscall interfaces, and scheduling are security-sensitive.
- Using tools such as strace or perf to understand where applications spend time.
- Evaluating kernel-module, microkernel, and user-space-driver trade-offs.
Common mistakes
- Calling every privileged OS service a separate executable; many fundamental mechanisms live in the kernel.
- Assuming kernel code can trust user pointers or sizes; user-controlled input must be checked.
- Assuming an interrupt handler should do unbounded work; long handling can increase latency.
Interview questions
Why must the kernel validate syscall arguments?
Arguments originate in a less-privileged context and may be malformed, unmapped, or changed concurrently. Validation preserves isolation and kernel invariants.
What is the difference between an interrupt and a system call?
A system call is a deliberate request by software through a defined interface; an interrupt is typically an asynchronous hardware event. Both can transfer control to privileged handling code.
Why might a microkernel move drivers into user space?
It can reduce the trusted kernel and contain some driver failures, at the cost of IPC, context switches, and more complex coordination.