CrewAI SLA Incident Responders with Auto-Recovery & Human-in-the-Loop Safeguards

Automating Incident Response with CrewAI

By Deepak Bagada, CEO at SaaSNext & Principal AI Architect.

In modern Site Reliability Engineering (SRE), downtime is measured in thousands of dollars per minute. The integration of Autonomous AI agents into incident response workflows has shifted from experimental to critical. CrewAI offers a phenomenal framework for orchestrating role-playing agents that can triage, diagnose, and remediate system alerts automatically.

This article details the construction of a CrewAI-based Incident Responder team, specifically focusing on SLA adherence, automated recovery, and crucial Human-in-the-Loop (HITL) safeguards.

The Incident Response Crew

Our virtual SRE team consists of three distinct CrewAI agents:

• Triage Specialist: Ingests PagerDuty/Datadog alerts, classifies severity, and identifies affected services.
• Root Cause Analyst: Queries logs (via Elasticsearch/Splunk tools) and metrics to pinpoint the anomaly.
• Remediation Engineer: Proposes and executes fix scripts (e.g., restarting pods, rolling back deployments) with HITL gates.

Architecture Flow

sequenceDiagram
    participant Alerting(Datadog)
    participant Triage(Triage Agent)
    participant Analyst(RCA Agent)
    participant Remediation(Remediation Agent)
    participant SRE(Human SRE)
Alerting->>Triage: High CPU Alert
Triage->>Analyst: Service X is failing. Investigate.
Analyst->>Analyst: Query Logs & Metrics
Analyst->>Remediation: Memory Leak in pod Y. Needs restart.
Remediation->>SRE: Request Approval (Restart Pod Y)
SRE-->>Remediation: Approved
Remediation->>Remediation: Execute kubectl restart
Remediation->>Alerting: Resolve Alert

Implementation Codebase

1. Environment Config (.env)

OPENAI_API_KEY=sk-...
DATADOG_API_KEY=dd_...
PAGERDUTY_TOKEN=pd_...
SLACK_WEBHOOK_URL=https://hooks.slack.com/...

2. Custom Tools (tools.py)

Agents need tools from our MCP Tools Directory to interact with infrastructure.

from crewai.tools import tool
import subprocess
@tool("Query Logs")
def query_logs(service_name: str, time_range: str) -> str:
"""Queries system logs for a given service and time range."""
# Mock implementation
return f"[{time_range}] {service_name}: Exception OutOfMemoryError in worker thread."
@tool("Execute Kubernetes Command")
def execute_kubectl(command: str, approval_token: str) -> str:
"""Executes a kubectl command. REQUIRES HUMAN APPROVAL TOKEN."""
if approval_token != "APPROVED_BY_SRE":
return "Error: Action requires human approval."
# VERY DANGEROUS: Validate strictly in production
safe_commands = ["get", "describe", "logs", "rollout restart"]
if not any(cmd in command for cmd in safe_commands):
    return "Error: Command not in safe list."
    
try:
    result = subprocess.check_output(f"kubectl {command}", shell=True, text=True)
    return result
except Exception as e:
    return str(e)

3. Agent Definitions (agents.py)

from crewai import Agent
from tools import query_logs, execute_kubectl
triage_agent = Agent(
role='SRE Triage Specialist',
goal='Accurately classify incoming alerts and determine impacted systems.',
backstory="You are the first responder for production incidents. Speed and accuracy are vital.",
verbose=True,
allow_delegation=True
)
rca_agent = Agent(
role='Root Cause Analyst',
goal='Analyze logs and metrics to find the underlying cause of an incident.',
backstory="You are an expert at finding needles in haystacks within distributed system logs.",
tools=[query_logs],
verbose=True
)
remediation_agent = Agent(
role='Remediation Engineer',
goal='Apply fixes to restore service SLA safely.',
backstory="You execute changes to infrastructure. You are cautious and ALWAYS ask for human approval before destructive actions.",
tools=[execute_kubectl],
verbose=True
)

4. Task and Crew Execution (main.py)

from crewai import Task, Crew, Process
from agents import triage_agent, rca_agent, remediation_agent
alert_data = "CRITICAL: API Gateway latency > 2000ms. Error rate 15%."
task1 = Task(
description=f"Review this alert: {alert_data}. Identify the system.",
expected_output="A structured summary of the alert and impacted service.",
agent=triage_agent
)
task2 = Task(
description="Using the triage report, query logs to find the root cause.",
expected_output="A root cause analysis report.",
agent=rca_agent
)
task3 = Task(
description="Based on the RCA, propose a fix. If a pod restart is needed, output the exact command, then WAIT for approval.",
expected_output="Remediation plan and execution result.",
agent=remediation_agent,
human_input=True # ENABLES HUMAN IN THE LOOP
)
incident_crew = Crew(
agents=[triage_agent, rca_agent, remediation_agent],
tasks=[task1, task2, task3],
process=Process.sequential,
verbose=True
)
result = incident_crew.kickoff()
print("######################")
print("INCIDENT RESOLVED:")
print(result)

Human-In-The-Loop (HITL) Safeguards

In task3, we set human_input=True. When the CrewAI process reaches this task, it will pause execution and prompt the console (or a Slack integration in a real deployment) for human feedback. The SRE can review the RCA agent's findings and the Remediation agent's proposed command. If safe, the SRE types the approval, providing the necessary context for the agent to proceed with the execute_kubectl tool.

SLA Adherence Strategies

To ensure SLAs are met, agents must be configured with specific timeout limits. Using asynchronous execution for non-dependent tasks (like querying logs across multiple microservices simultaneously) significantly reduces Time-To-Resolution (TTR). Furthermore, utilizing smaller, faster models (like Llama 3.1 8B or GPT-4o-mini) for the Triage Agent ensures rapid initial classification, reserving heavy reasoning models only for the RCA phase.

Implementing CrewAI for incident response not only reduces engineer burnout but creates a self-healing infrastructure layer that dramatically improves overall system uptime.