Mark Russinovich & Windows Internals: Observability & Reliability

Why Mark Russinovich Still Matters to Windows Operations

If you run Windows in production—on laptops, servers, VDI, or cloud VMs—Mark Russinovich’s work still shows up in day-to-day operations. Not because of personality or nostalgia, but because he helped popularize an evidence-first approach to troubleshooting: look at what the OS is actually doing, then explain symptoms with proof.

Three plain-English ideas

Observability means you can answer “what is happening right now?” using signals the system produces (events, traces, counters). When a service slows down or logons hang, observability is the difference between guessing and knowing.

Debugging is turning a vague problem (“it froze”) into a specific mechanism (“this thread is blocked on I/O,” “this process is thrashing the page file,” “this DLL injection changed behavior”).

Reliability is the ability to keep working under stress and to recover predictably—fewer incidents, faster restores, and safer changes.

Why internals knowledge speeds up incidents

Most “mystery outages” aren’t mysteries—they’re Windows behaviors you haven’t mapped yet: handle leaks, runaway child processes, stuck drivers, DNS timeouts, broken auto-start entries, or security tooling that adds overhead. A basic grasp of Windows internals (processes, threads, handles, services, memory, I/O) helps you recognize patterns quickly and collect the right evidence before the problem disappears.

What this article will help you do

We’ll focus on practical, operations-friendly workflows using:

• Sysinternals tools (especially Process Explorer and Process Monitor) for fast, low-friction visibility
• ETW tracing when logs aren’t enough and you need high-fidelity “what happened” timelines
• WinDbg and crash/hang dumps to convert failures into actionable root causes

The goal isn’t to turn you into a kernel engineer. It’s to make Windows incidents shorter, calmer, and easier to explain—so fixes are safer and repeatable.

Windows Internals as a Troubleshooting Superpower

Windows “internals” is simply the set of mechanisms Windows uses to do real work: scheduling threads, managing memory, starting services, loading drivers, handling file and registry activity, and enforcing security boundaries. The practical promise is straightforward: when you understand what the OS is doing, you stop guessing and start explaining.

That matters because most operational symptoms are indirect. “The machine is slow” might be CPU contention, a single hot thread, a driver interrupt storm, paging pressure, or an antivirus filter blocking file I/O. “It hangs” could be a deadlock, a stuck network call, a storage timeout, or a service waiting on a dependency. Boot issues might be a broken autorun entry, a failing driver load, or a policy script that never finishes. Internals knowledge turns vague complaints into testable hypotheses.

User mode vs. kernel mode (just enough to be useful)

At a high level, user mode is where most apps and services run. When they crash, they typically take down only themselves. Kernel mode is where Windows itself and drivers run; problems there can freeze the whole system, trigger a bugcheck (blue screen), or quietly degrade reliability.

You don’t need deep theory to use this distinction—just enough to choose evidence. An app pegging CPU is often user mode; repeated storage resets or network driver issues often point toward kernel mode.

Evidence-first troubleshooting

Russinovich’s mindset—reflected in tools like Sysinternals and in Windows Internals—is “evidence first.” Before changing settings, rebooting blindly, or reinstalling, capture what the system is doing: which process, which thread, which handle, which registry key, which network connection, which driver, which event.

Once you can answer “what is Windows doing right now, and why,” fixes become smaller, safer, and easier to justify—and reliability work stops being reactive firefighting.

The Sysinternals Approach: Make the Invisible Visible

Sysinternals is best understood as a “visibility toolkit” for Windows: small, portable utilities that reveal what the system is actually doing—process by process, handle by handle, registry key by registry key. Instead of treating Windows as a black box, Sysinternals lets you observe the behavior behind symptoms like “the app is slow,” “CPU is high,” or “the server keeps dropping connections.”

Trust but verify: don’t guess, measure

A lot of operational pain comes from reasonable-sounding guesses: it must be DNS, it’s probably antivirus, Windows Update is stuck again. The Sysinternals mindset is simple: trust your instincts enough to form a hypothesis, then verify it with evidence.

When you can see which process is consuming CPU, which thread is waiting, which file path is being hammered, or which registry value keeps getting rewritten, you stop debating opinions and start narrowing causes. That shift—from narrative to measurement—is what makes internals knowledge practical, not academic.

Why Sysinternals shines during live incidents

These tools are built for the “everything is on fire” moment:

• Low friction: many tools run without installation and launch quickly.
• Fast feedback: you can validate or reject a theory in minutes.
• Focused visibility: each utility answers a specific class of questions (processes, startup items, network endpoints, memory usage).

That matters when you can’t afford a long setup cycle, a heavy agent rollout, or a reboot just to collect better data.

Safe usage principles

Sysinternals is powerful, and power deserves guardrails:

• Run as needed: start with read-only observation; elevate privileges only when required.
• Document what you do: record timestamps, filters, and any actions taken so findings are repeatable.
• Minimize disruption: prefer capturing evidence (screenshots, logs, exported traces) over “trying fixes” mid-incident.
• Change carefully: if you must alter a setting or kill a process, note the reason and expected outcome, then verify results.

Used this way, Sysinternals becomes a disciplined method: observe the invisible, measure the truth, and make changes that are justified—not hopeful.

Process Explorer & Process Monitor: The Everyday Debug Pair

If you only keep two Sysinternals tools in your admin toolkit, make it Process Explorer and Process Monitor. Together they answer the most common “what is Windows doing right now?” questions without requiring an agent, a reboot, or a heavy setup.

Process Explorer: fast answers in seconds

Process Explorer is Task Manager with x-ray vision. When a machine is slow or unstable, it helps you pinpoint which process is responsible and what it’s tied to.

It’s especially useful for:

• CPU and threads: Which process is burning CPU, and is it one hot thread or many?
• Parent/child relationships: What launched the process (a service, scheduled task, updater, or user action)?
• DLLs and handles: What modules are loaded, and what files/registry keys/pipes the process is holding open?

That last point is a reliability superpower: “Why can’t I delete this file?” often becomes “This service has an open handle to it.”

Process Monitor: the full activity trail

Process Monitor (Procmon) captures detailed events across file system, registry, and process/thread activity. It’s the tool for questions like: “What changed when the app hung?” or “What is hammering the disk every 10 minutes?”

Before you hit Capture, frame the question:

• What is the symptom (slow logon, high disk, crash, access denied)?
• When does it happen (on startup, at 09:00, after sleep)?
• Which machine and user context (only one server, only one user profile, only on VPN)?

Capture only what you need (noise is the enemy)

Procmon can overwhelm you unless you filter aggressively. Start with:

• Filter to a specific Process Name or PID.
• Use Include rules for the path you care about (e.g., a config folder) and exclude the rest.
• Capture for a short window around the symptom, then stop.

What you get out of it

Common outcomes are very practical: identifying a misbehaving service repeatedly querying a missing registry key, spotting a runaway “real-time” file scan touching thousands of files, or finding a missing DLL load attempt (“NAME NOT FOUND”) that explains why an app won’t start on one machine but works on another.

Autoruns, TCPView, RAMMap: Fast Clues Without Heavy Setup

When a Windows machine “feels off,” you often don’t need a full monitoring stack to get traction. A small set of Sysinternals tools can quickly answer three practical questions: What starts automatically? What is talking on the network? Where did the memory go?

Autoruns: reliability starts at boot

Autoruns is the fastest way to understand everything that can launch without a user explicitly running it: services, scheduled tasks, shell extensions, drivers, and more.

Why it matters for reliability: startup items are frequent sources of slow boots, intermittent hangs, and CPU spikes that only appear after login. One unstable updater, legacy driver helper, or broken shell extension can degrade the whole system.

Practical tip: focus on entries that are unsigned, recently added, or failing to load. If disabling an item stabilizes the box, you’ve turned a vague symptom into a specific component you can update, remove, or replace.

TCPView: confirm who’s listening, who’s chattering

TCPView gives you an instant map of active connections and listeners, tied to process names and PIDs. It’s ideal for quick sanity checks:

• Unexpected LISTENING ports (especially on servers that should be quiet)
• A single process owning an unusually high number of connections
• Rapid connection churn that correlates with CPU or latency complaints

Even for non-security investigations, this can uncover runaway agents, misconfigured proxies, or “retry storms” where the app looks slow but the root cause is network behavior.

RAMMap: memory pressure without guesswork

RAMMap helps you interpret memory pressure by showing where RAM is actually allocated.

A useful baseline distinction:

• Working sets: memory actively used by running processes
• Cache / standby: Windows keeping data around to speed things up (not inherently “bad”)

If users report “low memory” while Task Manager looks confusing, RAMMap can confirm whether you have true process growth, heavy file cache, or something like a driver consuming nonpaged memory.

Optional: Handle and VMMap when leaks are suspected

If an app slows down over days, Handle can reveal handle counts growing without bound (a classic leak pattern). VMMap helps when memory usage is odd—fragmentation, large reserved regions, or allocations that don’t show up as simple “private bytes.”

A repeatable first 15 minutes checklist

1. Autoruns: scan for new/unsigned entries; disable one suspicious item at a time.
2. TCPView: verify expected listeners; identify top connection owners.
3. RAMMap: check whether pressure is working set growth vs. cache/standby.
4. If symptoms are time-based: capture a quick “before/after” snapshot (counts, ports, memory totals).
5. If growth is obvious: use Handle/VMMap to confirm a leak pattern.
6. Write down the suspected component and the evidence so the fix is targeted, not guesswork.

From Logs to ETW: Building Real Observability on Windows

Windows operations often starts with what’s easiest to grab: Event Viewer and a few screenshots of Task Manager. That’s fine for breadcrumbs, but reliable incident response needs three complementary signal types: logs (what happened), metrics (how bad it got), and traces (what the system was doing moment-to-moment).

Event logs: great clues, imperfect coverage

Windows event logs are excellent for identity, service lifecycle, policy changes, and app-level errors. They’re also uneven: some components log richly, others log sparsely, and message text can be vague (“The application stopped responding”). Treat them as a timeline anchor, not the whole story.

Common wins:

• Service start/stop and crash events
• Authentication and authorization events
• Application exceptions (when apps actually log them)

Metrics during outages: the few that usually matter

Performance counters (and similar sources) answer, “Is the machine healthy?” During an outage, start with:

• CPU: sustained high CPU, ready time (VMs), per-process CPU
• Disk: queue length, read/write latency, IOPS, free space
• Memory: committed bytes, commit limit, hard faults/sec, pool usage
• Network: retransmits, errors, bytes/sec, connection counts

Metrics won’t tell you why a spike happened, but they’ll tell you when it started and whether it’s improving.

ETW in plain terms: structured, high-volume tracing

Event Tracing for Windows (ETW) is Windows’ built-in flight recorder. Instead of ad-hoc text messages, ETW emits structured events from the kernel, drivers, and services at high volume—process/thread activity, file I/O, registry access, TCP/IP, scheduling, and more. This is the level where many “mystery stalls” become explainable.

Choosing signals (without collecting everything)

A practical rule:

• Use logs for discrete events (crash, restart, auth failure).
• Use metrics to detect and quantify impact (latency, saturation).
• Use ETW when you need causality (what was blocking, which I/O, which call path).

Avoid “turn on everything forever.” Keep a small always-on baseline (key logs + core metrics) and use short, targeted ETW captures during incidents.

Time correlation is the superpower

The fastest diagnoses come from aligning three clocks: user reports (“10:42 it froze”), metric inflections (CPU/disk spike), and log/ETW events at the same timestamp. Once your data shares a consistent time base, outages stop being guesses and start becoming narratives you can verify.

Sysmon Telemetry: Security Signals That Help Reliability Too

Windows’ default event logs are useful, but they often miss the “why now?” details operators need when something changes unexpectedly. Sysmon (System Monitor) fills that gap by recording higher-fidelity process and system activity—especially around launches, persistence, and driver behavior.

What Sysmon adds (beyond default logs)

Sysmon’s strength is context. Instead of just “a service started,” you can often see which process started it, with full command line, parent process, hashes, user account, and clean timestamps for correlation.

That’s valuable for reliability work because many incidents begin as “small” changes: a new scheduled task, a silent updater, a stray script, or a driver that behaves badly.

Minimal configuration: start narrow on purpose

A “log everything” Sysmon config is rarely a good first move. Start with a minimal, reliability-focused set and expand only when you have clear questions.

Good early candidates:

• Process creation (unexpected launches, suspicious command lines)
• Driver load (new or changing kernel components)
• Image/DLL load (use selectively for dependency problems)
• Service and scheduled-task related activity (persistence and background changes)
• Network connections / DNS (enable only for specific investigations to manage volume)

Tune with targeted include rules (critical paths, known service accounts, key servers) and carefully chosen exclude rules (noisy updaters, trusted management agents) so the signal stays readable.

Reliability use cases you’ll actually see

Sysmon often helps confirm or rule out common “mystery change” scenarios:

• A new helper process spawning under a service account right before CPU spikes
• A service binary changing paths or start type after a patch cycle
• A driver update coinciding with new hangs, bugchecks, or storage/network resets

Operational cautions

Test impact on representative machines first. Sysmon can increase disk I/O and event volume, and centralized collection can get expensive quickly.

Also treat fields like command lines, usernames, and paths as sensitive. Apply access controls, retention limits, and filtering before broad rollout.

Complements, doesn’t replace, the rest of observability

Sysmon is best as high-value breadcrumbs. Use it alongside ETW for deep performance questions, metrics for trend detection, and disciplined incident notes so you can connect what changed to what broke—and how you fixed it.

WinDbg and Dumps: Turning Crashes and Hangs into Answers

When something “just crashes,” the most valuable artifact is often a dump file: a snapshot of memory plus enough execution state to reconstruct what the process (or the OS) was doing at the moment of failure. Unlike logs, dumps don’t require you to predict the right message ahead of time—they capture the evidence after the fact.

What crash dumps are (and why you want them)

• App crash dumps (user mode) record a single process. They’re ideal when one service dies but the machine stays up.
• Kernel dumps (system-wide) are used for bugchecks (BSODs) and capture OS-level state, drivers, and kernel threads.

Dumps can point to a specific module, call path, and failure type (access violation, heap corruption, deadlock, driver fault), which is hard to infer from symptoms alone.

WinDbg basics: symbols, stacks, and “what failed”

WinDbg turns a dump into a story. The essentials:

• Symbols map raw addresses to function names and line info. Without correct symbols, analysis quickly becomes guesswork.
• Stack traces show the call sequence leading to the crash or the current state of a “stuck” thread.
• The goal is to identify the failing component: your code, a dependency DLL, a driver, an antivirus shim, a graphics stack, etc.

A typical workflow is: open the dump → load symbols → run an automated analysis → validate by checking top stacks and involved modules.

Crash vs. BSOD vs. hang: don’t mix the categories

• Bugcheck (BSOD): the whole system stops. Expect kernel dumps and driver/root-cause work.
• App crash: one process terminates. Expect user-mode dumps and an exception code.
• Hang: nothing “crashes,” but work stops. You need proof of what threads are waiting on.

Hangs need evidence: stacks, waits, and locks

“It’s frozen” is a symptom, not a diagnosis. For hangs, capture a dump while the app is unresponsive and inspect:

• Thread stacks to see what each thread is doing.
• Wait reasons (I/O, RPC, mutex/critical section, network).
• Locks/contention patterns—often the “hung” UI thread is waiting on a worker thread that’s blocked elsewhere.

Realistic expectations: self-diagnose vs. escalate

You can often self-diagnose clear-cut issues (repeatable crashes in one module, obvious deadlocks, strong correlation to a specific DLL/driver). Escalate when dumps implicate third-party drivers/security software, kernel components, or when symbols/source access is missing—then a vendor (or Microsoft) may be needed to interpret the full chain.

Common Failure Patterns and How Internals Explains Them

A lot of “mysterious Windows issues” repeat the same patterns. The difference between guessing and fixing is understanding what the OS is doing—and the Internals/Sysinternals mental model helps you see it.

Memory leaks: working set vs. commit

When people say “the app is leaking memory,” they often mean one of two things.

Working set is the physical RAM currently backing the process. It can go up and down as Windows trims memory under pressure.

Commit is the amount of virtual memory the system has promised to back with either RAM or the page file. If commit keeps climbing, you have a real leak risk: eventually you hit the commit limit and allocations start failing or the host becomes unstable.

A common symptom: Task Manager shows “available RAM,” but the machine still slows down—because commit, not free RAM, is the constraint.

Handle leaks: slow failure that looks random

A handle is a reference to an OS object (file, registry key, event, section, etc.). If a service leaks handles, it may run fine for hours or days, then start failing with odd errors (can’t open files, can’t create threads, can’t accept connections) as per-process handle counts grow.

In Process Explorer, watch handle count trends over time. A steady upward slope is a strong clue the service is “forgetting to close” something.

Disk and file system issues: latency, retries, filter drivers

Storage problems don’t always show as high throughput; they often show as high latency and retries. In Process Monitor, look for:

• Repeated CreateFile/ReadFile operations
• Long-duration I/O events
• Lots of NAME NOT FOUND / PATH NOT FOUND noise (misconfigured paths)

Also pay attention to filter drivers (AV, backup, DLP). They can insert themselves into the file I/O path and add delay or failures without the application “doing anything wrong.”

CPU spikes: one hot process vs. contention

A single hot process is straightforward: one executable burns CPU.

System-wide contention is trickier: CPU is high because many threads are runnable and fighting over locks, disk, or memory. Internals thinking pushes you to ask: “Is the CPU doing useful work, or spinning while blocked elsewhere?”

Network problems: who owns the connection?

When timeouts happen, map process → connection using TCPView or Process Explorer. If the wrong process owns the socket, you’ve found a concrete culprit. If the right one owns it, look for patterns: SYN retries, long-established connections stuck idle, or an explosion of short-lived outbound attempts suggesting DNS/firewall/proxy trouble rather than “the app is down.”

A Practical Workflow: Observe → Capture → Explain → Fix

Reliability work gets easier when every incident follows the same path. The goal isn’t to “run more tools”—it’s to make better decisions with consistent evidence.

1) Reproduce (or define the trigger)

Write down what “bad” looks like in one sentence: “App freezes for 30–60 seconds when saving a large file” or “CPU spikes to 100% every 10 minutes.” If you can reproduce, do it on demand; if you can’t, define the trigger (time window, workload, user action).

2) Observe (lightweight first)

Before collecting heavy data, confirm the symptom and scope:

• Is it one machine or many?
• One process or the whole host?
• Performance issue, crash, or hang?

This is where quick checks (Task Manager, Process Explorer, basic counters) help you choose what to capture next.

3) Capture (build a good case file)

Capture evidence like you’re handing it to a teammate who wasn’t there. A good case file usually includes:

• Timestamps (start/end, time zone, frequency)
• Versions (Windows build, app version, driver versions)
• Configuration (feature flags, policies, environment variables, security tooling)
• Traces (Procmon filters, ETW session name, duration)
• Dumps (hangs/crashes: full vs. mini, which process, how it was triggered)

Keep captures short and targeted. A 60-second trace that covers the failure window beats a 6-hour capture nobody can open.

4) Explain (turn data into a story)

Translate what you collected into a plain narrative:

• What changed? (new build, policy, driver, load)
• What is the system doing instead? (retries, contention, blocked I/O, timeouts)
• What is the likely cause? (one or two hypotheses, ranked)

If you can’t explain it simply, you probably need a cleaner capture or a narrower hypothesis.

5) Fix, confirm, and reduce MTTR next time

Apply the smallest safe fix, then confirm with the same reproduction steps and a “before vs. after” capture.

To reduce MTTR, standardize playbooks and automate the boring parts:

• One script/command to start a trace, one to stop and zip results
• A consistent folder structure and naming convention
• A checklist for what to collect per symptom (crash vs. hang vs. slowdown)

Post-incident learning: add missing signal

After resolution, ask: “What signal would have made this obvious earlier?” Add that signal—Sysmon event, ETW provider, a performance counter, or a lightweight health check—so the next incident is shorter and calmer.

Making It Stick: Safer Fixes and Long-Term Reliability

The point of Windows internals work isn’t to “win” a debugging session—it’s to turn what you saw into changes that keep the incident from returning.

Turn findings into concrete actions

Internals tools usually narrow a problem to a small set of levers. Keep the translation explicit:

• Config change: a service account permission, a registry value, a pool size, a scheduled task cadence.
• Patch: OS cumulative update, .NET update, or vendor hotfix that matches the call stack or driver version you observed.
• Driver update (or rollback): if Procmon/ETW shows stalls around file/network/filter drivers, treat driver versions as first-class dependencies.
• Rollback: if the fix is risky, plan to revert quickly (known-good package, previous GPO, older driver bundle).

Write down the “because”: “We changed X because we observed Y in Process Monitor / ETW / dumps.” That sentence prevents tribal knowledge from drifting.

Guardrails: change windows, validation, rollback

Make your change process match the blast radius:

• Use a change window with reduced traffic if possible.
• Define validation steps (what counters, event IDs, or user journeys must improve).
• Prepare a clear rollback plan with an owner and a time limit (“If errors don’t drop in 15 minutes, revert”).

Reliability patterns you can apply repeatedly

Even when the root cause is specific, durability often comes from reusable patterns:

• Timeouts to prevent thread starvation and stuck dependency chains.
• Rate limiting/backoff to stop retry storms.
• Service recovery options (restart actions, failure reset period) for expected transient faults.
• Health checks that detect hangs, not just crashes.

Data hygiene for captures and telemetry

Keep what you need, and protect what you shouldn’t collect.

Limit Procmon filters to suspected processes, scrub paths/usernames when sharing, set retention for ETW/Sysmon data, and avoid payload-heavy network capture unless necessary.

Operationalizing playbooks (where Koder.ai can help)

Once you have a repeatable workflow, the next step is to package it so others can run it consistently. This is where a vibe-coding platform like Koder.ai can be useful: you can turn your incident checklist into a small internal web app (React UI, Go backend with PostgreSQL) that guides responders through “observe → capture → explain,” stores timestamps and artifacts, and standardizes naming and case-file structure.

Because Koder.ai builds apps through chat using an agent-based architecture, teams can iterate quickly—adding a “start ETW session” button, a Procmon filter template library, snapshot/rollback of changes, or an exportable runbook generator—without rebuilding everything in a traditional dev pipeline. If you’re sharing internal reliability practices, Koder.ai also supports source-code export and multiple tiers (free through enterprise), so you can start small and scale governance later.

A small weekly practice plan

Once a week, pick one tool and a 15-minute exercise: trace a slow app start with Procmon, inspect a service tree in Process Explorer, review Sysmon event volume, or take one crash dump and identify the failing module. Small reps build the muscle memory that makes real incidents faster—and safer.