The Problem This Solves
It's 2am. Your production database isn't responding. Three engineers are on the bridge. Nobody knows who's leading, everyone is trying different things at the same time, and the manager keeps pinging on WhatsApp asking for an ETA.
That chaos isn't a talent problem. It's a process problem.
ITIL Incident Management exists for exactly that scenario: restore the service as fast as possible with minimum chaos, maximum communication, and a record that lets you learn afterward.
What Incident Management Actually Is
An incident is any unplanned interruption or degradation of a service. It doesn't matter if it affects 1 user or 10,000 — if something broke or is working poorly, it's an incident.
Incident Management is the process for:
- 🔍 Detecting and logging the incident
- 🏷️ Classifying it by severity and type
- 👥 Assigning it to the right team
- 🔧 Restoring the service as fast as possible
- 📝 Documenting the cause and actions taken
- 📚 Learning from it to prevent recurrence
What Incident Management is NOT: the deep root cause analysis (that's Problem Management, ITIL P3), or implementing permanent fixes (that's Change Management).
Severity Classification
Without a clear severity matrix, every engineer decides if something is P1 or P3 based on their own gut feeling. That creates chaos in communication and priorities.
| Severity | Criteria | Response Time | Escalation |
|---|---|---|---|
| SEV-1 / P1 | Production service down, total or massive impact | Immediate (< 5 min) | CTO / Management |
| SEV-2 / P2 | Severe degradation, critical functionality affected | < 15 min | Tech Lead / Manager |
| SEV-3 / P3 | Partial impact, workaround available | < 1 hour | On-Call Team |
| SEV-4 / P4 | Minor impact, no operational urgency | Business hours | L1 Support |
💡 Rule of thumb: If you're unsure between P1 and P2, classify as P1. Better to over-escalate than under-escalate.
The Process: 6 Steps
Step 1 — Detection and Logging
The incident can come from monitoring (automated alert), a user report, or the ops team itself. Regardless of the source, the first action is to open a ticket.
The ticket must have from minute one:
- ⏰ Exact timestamp of incident start
- 📋 Description of the observed symptom (no assumptions yet)
- 🖥️ Affected system(s) and environment (PROD / QA / DEV)
- 👤 Who reported it and the channel
- 🏷️ Initial severity (adjustable as you learn more)
# Example: Initial ticket log
Incident ID : INC-2024-0347
Start : 2024-11-14 02:17 CST
Symptom : Oracle WebLogic Managed Server MS-01 not responding.
Health check returning HTTP 503 since 02:15.
System : app-prod-01 / Domain WL_PROD / RHEL 9.3
Severity : P1 (payments service down)
Reported by : Zabbix Monitoring — Alert #44832
Step 2 — Classification and Initial Diagnosis
With the ticket open, the first responder does a quick diagnosis (max 5-10 min for P1/P2) to understand the real scope and confirm or adjust severity.
Key questions at this stage:
- How many users / systems are affected?
- Is this an isolated component or something systemic?
- Was there a recent change? (deployment, config, patch)
- Is there an immediate workaround available?
⚠️ Critical: The initial diagnosis is for making escalation decisions, not for solving the problem. Don't spend 30 minutes investigating when you should be escalating first.
Step 3 — Escalation and Assignment
For P1 and P2, escalation must happen in parallel with diagnosis, not after. The Incident Commander (IC) — who leads the process — defines who does what.
Minimum roles for a major incident:
| Role | Responsibility |
|---|---|
| Incident Commander (IC) | Leads the process. Makes decisions. Does NOT execute changes. |
| Technical Responder | Diagnoses and executes technical actions. |
| Communicator | Updates stakeholders. Writes status updates. |
| Scribe (optional) | Documents everything in real-time in the ticket. |
Step 4 — Resolution
This phase can take minutes or hours. The goal is to restore the service, not necessarily to understand why it failed. Root cause analysis comes after.
Resolution principles:
- Try the fastest workaround first if one exists (e.g., restart the service)
- Every action must be documented in the ticket with a timestamp
- If an action isn't working after X minutes, stop and change approach
- Don't execute changes without communicating them to the IC first
- If you need an infrastructure change, follow the Emergency Change process
# Example: Action log during resolution
02:22 - [jlopez] Restarted Managed Server MS-01. No success.
02:28 - [jlopez] Log review: OutOfMemoryError on heap.
02:31 - [ic: gvargas] Escalating to Senior Java/WL — paging mherrera.
02:35 - [mherrera] Adjusted -Xmx in startManagedWebLogic.sh. Restarting.
02:41 - [mherrera] MS-01 UP. Health check OK. Monitoring.
02:45 - [ic: gvargas] Payments service restored. Closing P1.
Step 5 — Stakeholder Communication
For P1 and P2, stakeholders must receive proactive updates while the incident is active — not when they ask. Silence = panic in management.
Recommended update format (every 20-30 min for P1):
[ACTIVE INCIDENT] INC-2024-0347 — Payments Service
Time: 02:30 CST | Status: IN RESOLUTION
Situation: WebLogic Managed Server MS-01 experiencing an OutOfMemoryError
on the heap. Team is adjusting JVM configuration.
Impact: Payment transactions unavailable since 02:15 CST.
ETA: 15-20 minutes to resolution.
Next update: 02:50 CST or sooner if status changes.
Step 6 — Closure and Post-Incident Review
An incident closes when the service is restored and verified, not when the team is tired of working on it. Before closing, the ticket must have:
- ✅ Preliminary root cause (even if provisional)
- ✅ Actions taken to restore the service
- ✅ Total resolution time (MTTR)
- ✅ Estimated impact (users affected, downtime duration)
- ✅ Next steps: is a Problem being opened for deeper analysis?
For P1 and P2, a formal Post-Incident Review (PIR) is recommended within 48-72 hours. The goal of a PIR is not to find blame — it's to learn and improve.
The 4 Metrics That Matter
| Metric | What It Measures |
|---|---|
| MTTA | Mean Time to Acknowledge — how fast the team picks up the alert |
| MTTD | Mean Time to Detect — how late you found out |
| MTTR | Mean Time to Restore — resolution speed |
| Recurrence Rate | % of incidents that happen again without resolved root cause |
🔍 A high MTTR doesn't always mean a slow team — it can mean systems without proper monitoring, outdated runbooks, or missing production access. Fix the root causes, not just the numbers.
5 Common Mistakes (and How to Avoid Them)
1. No clear Incident Commander When everyone is responsible, no one is. In a P1 with 5 engineers on the call, time is wasted in discussions and overlapping actions. Assign an IC from minute one.
2. Resolving before documenting "I'll fix it first and write it up later" guarantees an incomplete post-mortem. Real-time documentation takes 2 extra minutes — and is worth weeks of analysis later.
3. Confusing incident with problem Opening a P1 and spending 3 hours on root cause analysis while the service is still down. The incident is to restore the service. The problem is to prevent recurrence.
4. Updating stakeholders only when they ask If management has to ask, there's already been a communication failure. Proactive updates every 20-30 min keep people informed and reduce pressure on the technical team.
5. No documented severity matrix Without clear criteria, the on-call team spends 10 minutes deciding if it's P1 or P2 while the service is down. Define the matrix in calm — not during the incident.
Real-World Example: WebLogic Production Outage
Scenario (anonymized): Financial platform, WebLogic 14c, Rocky Linux 9, production environment with 2,000 active users.
02:17 — Detection Zabbix alert: HTTP 503 on the Managed Server health check. On-call acknowledges in 3 minutes (MTTA: 3 min) and opens ticket INC-2024-0347 as immediate P1.
02:20 — Diagnosis and Escalation
Log review: java.lang.OutOfMemoryError. Heap configured with -Xmx512m — insufficient for the overnight batch load. On-call escalates to the WebLogic Senior and activates the communication bridge.
02:35 — Resolution
Senior adjusts -Xmx to 2048m in startManagedWebLogic.sh and restarts the Managed Server. At 02:41 the health check returns 200 OK. Total MTTR: 24 minutes.
02:45 — Close and Next Steps P1 closed with complete documentation. Problem PRB-2024-0089 opened to analyze why the heap was not flagged as insufficient before the incident, and to review JVM parameters across all Managed Servers.
Template Available
If you want to implement this process in your team without starting from scratch, the IT Incident Management Kit includes:
- 📊 Severity Matrix (Excel) — Criteria and response times by severity
- 📋 Escalation Runbook (Word) — Step-by-step guide for the Incident Commander
- ✉️ Stakeholder Communication Templates (Word) — Status updates ready to use
- 📈 Incident Log Tracker (Excel) — Log with automatic metrics (MTTR, MTTA)
- 🔍 Post-Incident Review Guide (Word) — Blameless post-mortem structure
→ Get the IT Incident Management Kit
Tested on production environments. RHEL 9 / Rocky Linux 9. WebLogic 14c, JBoss/WildFly, Oracle environments. Questions? → gvargas.devops@gmail.com