System Calls
The trap into the kernel: the only legal way a user program asks for privileged work.
Level: intermediate
What is it?
A system call is a controlled entry from a user process into the kernel to request an operation that needs privileged access, such as creating a process, reading a file, changing memory mappings, or sending data through a socket. The ABI specifies how the call number, arguments, return value, and errors are represented.
Why does it matter?
System calls define a security and compatibility boundary. The kernel must treat arguments as untrusted, enforce permissions, and preserve a stable interface for user programs.
Mental model
A syscall resembles a guarded service counter: cross the boundary with a request, let the kernel verify it and perform permitted work, then return a result. The boundary is explicit; not every ordinary function call crosses it.
How it works
- A high-level library function may wrap one syscall, several syscalls, or no syscall at all.
- The CPU's syscall/trap instruction enters a configured privileged handler; the exact mechanism is architecture-specific.
- The kernel validates pointers, lengths, file descriptors, permissions, and current state before using them.
- Results and errors follow the platform ABI; POSIX interfaces often report -1 and set errno in the libc wrapper.
- Frequent syscalls have overhead, so buffered I/O and batching can reduce boundary crossings when semantics allow.
Steps
- In C, open(path, flags) is commonly a libc wrapper around the open/openat syscall family.
- The wrapper places arguments in the platform-defined locations and enters the kernel.
- The kernel resolves the path, checks credentials and mount/file permissions, then creates or returns a file descriptor.
- The return value is translated into the API's result and error convention.
Real-world usage
- Reading syscall traces to diagnose unexpected disk, process, or network behavior.
- Designing APIs that batch work instead of making thousands of tiny calls.
- Understanding the boundary enforced by seccomp, capabilities, and permission checks.
Common mistakes
- Assuming each standard-library function maps one-to-one to a syscall.
- Treating a file descriptor as a path; it is a process-local handle to an open kernel-managed object.
- Passing user pointers to kernel code without safe copying or validation.
Interview questions
Does printf always make a system call?
No. It formats output in a user-space buffer and may call write() when the buffer flushes; behavior depends on buffering and the stream.
Why can two libc versions implement the same API with different syscalls?
The user-facing API and kernel ABI are separate layers. A libc may use a newer syscall, fall back to an older one, or combine operations while preserving the documented API.
What security checks belong at a syscall boundary?
Validate values and memory ranges, check object handles and permissions, handle concurrent changes safely, and return consistent errors without exposing kernel memory.