“It Is DNS”: a company-wide problem that grows in front of you
A junior engineer should not be expected to solve a core-DNS outage alone. This scenario shows how an early comparison detects that the issue is wider than a normal ticket, triggers fast communication and escalation, and prevents a risky local fix from wasting the first useful minutes.
The point: the ticket was not proof that payroll had failed. It was the first report of failed name-based access to several services.
Step 1 — Define the first symptom without shrinking it
At 09:07, an accounts user reports that the payroll portal will not open. The browser says the site cannot be reached. A second user reports that Teams links and the intranet are also failing. Email is still working on some laptops. The first report is not “a payroll issue”; it is two users reporting failed name-based access to several services.
The junior records the exact messages, users, locations and time. They also check impact: payroll approval and customer-service systems may be affected, while email behaviour is inconsistent. There is no safe basis yet for restarting a browser, resetting an account or declaring a vendor outage.
Step 2 — Compare working and failing paths, then widen scope
A known internal application can be reached by IP address but not by its normal hostname. On one affected device, the default gateway responds, but the configured DNS resolver does not answer a lookup. Similar reports arrive from another floor and from a VPN user. That comparison moves the likely boundary from one user, laptop or application toward shared name resolution.
| Observation | What it suggests | What it does not establish |
|---|---|---|
| Default gateway responds | Local network reachability exists on this device | Every network path is healthy |
| Internal service works by IP but not hostname | Name resolution is a strong candidate | The DNS server itself is necessarily the cause |
| Multiple offices and VPN users see the pattern | Impact is broader than one segment | Every employee and service is affected |
The junior now treats this as a potentially company-wide service degradation. They notify the duty lead using the approved channel and update the ticket with the observed scope. That is escalation based on impact and evidence, not on the label someone used.
Step 3 — Keep the DNS theory honest
“It is DNS” becomes the leading hypothesis because multiple name lookups fail while basic IP reachability still works. It is not yet a confirmed root cause. A firewall policy, routing path to resolvers, DHCP configuration, resolver forwarding path or an upstream dependency could produce a similar symptom.
The evidence register is deliberately short: successful gateway ping; failed lookup against the configured resolver; affected users in separate locations; timestamped reports; and any monitoring alert. The junior avoids turning an unverified observation such as “the DNS dashboard looks red” into proof without checking its time and scope.
Step 4 — Choose the smallest safe test that separates explanations
The useful test is a bounded query for the same approved public name and record type against each configured resolver from an approved diagnostic host, alongside a comparison query to a known external resolver where policy permits. The test is read-only. It can strengthen or weaken the explanation that the failure is confined to one workstation versus a shared resolver path; it does not identify the failing component or repair anything.
The primary and secondary internal resolvers both time out from the diagnostic host. The external comparison resolves a public domain, while internal names remain unavailable. This strengthens the shared internal-resolver-path hypothesis without ruling out a contributing workstation issue or identifying the failing component.
Boundary: direct-IP access is a reachability comparison only where the approved service supports it. HTTPS certificate identity, SNI, virtual hosting and authentication can depend on the hostname. Do not change workstation DNS settings across the fleet; it creates inconsistent state and might bypass names that should remain internal.
Step 5 — Stabilise service without confusing recovery with diagnosis
The network or platform owner reports that a failed maintenance change interrupted both internal resolvers’ access to directory-integrated zone data. They roll back the approved change and confirm resolver responses return. The junior’s role is to keep impact and evidence current, communicate the approved workaround if one exists, and verify user tasks once recovery is declared.
At 09:32, hostname lookups succeed from the diagnostic host. At 09:35, users in accounts and on VPN can open the payroll portal and intranet. That establishes recovery of the user-facing symptom. It does not by itself prove every causal detail of the maintenance failure.
Step 6 — Escalate with the useful record, not a pile of screenshots
The escalation and incident update contains: start time; impacted business tasks; affected locations; IP-versus-name comparison; resolver query results; actions deliberately not taken; owner of the rollback; and user verification after recovery. It names the next decision: investigate why a maintenance change removed both resolver paths or their common dependency.
The junior did not need to be an expert DNS administrator. They needed to recognise that the safe next step had moved beyond their authority, preserve a small set of discriminating evidence, and prevent five different people from “fixing” their own workstation at the same time. That gives the incident owner useful evidence and a clear next decision.
Step 7 — Verify, analyse and prevent recurrence
Because the issue was broad, business-impacting and linked to a change, it needs more than “DNS restarted and people are happy again.” A subsequent post-incident review should test whether the triggering maintenance change, lack of staged validation, shared resolver dependency, alerting coverage, and rollback checkpoint contributed to the impact. Those are review questions, not findings established by the recovery alone.
Corrective actions might include a pre-change resolver query check, staged rollout with a stop condition, independent-path health monitoring, and a tested rollback. Each action needs an owner and a verification method. Root cause analysis identifies causes and contributing conditions; the corrective actions change the conditions that allowed one change to produce company-wide impact.
Scenario wrap-up — The response widened as the evidence changed
The first user report looked like an application ticket. The comparisons changed that: gateway reachable, names failing, resolver unavailable, multiple locations affected. Each result made a broader response more justified. The outcome was not “the junior fixed DNS”; it was a faster transition from a local report to a managed outage response, supported by evidence.
Want the full framework? The complete guide includes contrasting payroll and intermittent-fault scenarios, decision drills, a worksheet, handover record and desk reference.