Build a NOC Around Exceptions, Not Wallboard Noise
A practical operating model for turning telemetry into owned, prioritized network work.
A practical operating model for turning telemetry into owned, prioritized network work.
A screen is not an operating model
A busy dashboard can still leave a team blind. The NOC becomes useful only when every important signal can become a scoped decision: what changed, who is affected, who owns the response and when the next update is due.
Start with service objects
Monitor objects that match how the ISP fails: site, power domain, router, uplink, OLT, PON, RADIUS service and payment dependency. Attach subscribers to those objects. A red device icon then becomes “312 services at Site North may be affected,” not an isolated ping failure.
| Signal | Useful context | Likely owner |
|---|---|---|
| Router unreachable | site power, upstream path, last config | NOC |
| Many ONUs offline | shared PON, splitter, first failure time | Fiber team |
| Authentication errors | NAS, rejection reason, recent policy change | Access systems |
| Collection callback delay | gateway, queue depth, affected payments | Billing operations |
Alert on symptoms with consequence
Do not page for every failed poll. Combine persistence, redundancy and customer impact. A single missed probe can be recorded; a router failing from two independent probes while sessions fall should open an incident. Use separate thresholds for notification and paging.
Group related alarms
If an aggregation router fails, downstream OLT and customer checks will also fail. Preserve those observations but group them beneath the probable parent event. Correlation should explain its evidence and remain reversible; an incorrect parent must not hide a second fault.
- Detect. Record the raw observation and source.
- Enrich. Add topology, maintenance and subscriber impact.
- Correlate. Join alarms sharing a plausible failure domain.
- Assign. Set owner, priority and response clock.
- Resolve. Verify recovery and record the actual cause.
Make maintenance a first-class state
A maintenance window should suppress expected notifications without deleting telemetry. Store scope, owner, approval, start and end. When it expires, normal evaluation must resume automatically and any continuing fault must surface.
Measure the loop
- Time from first customer impact to detection.
- Time from detection to a named owner.
- Incidents reopened after false recovery.
- Pages with no operator action.
- Repeat incidents without a corrective task.
Research basis: event-correlation and reliability practices adapted to ISP access, transport and business-service dependencies.