No GPUs. No cloud scaling. Just Linux, CPUs, and solid systems engineering.

Large Language Models (LLMs) are often associated with expensive GPUs and cloud infrastructure. However, for development, research, privacy-sensitive environments, and cost-controlled setups, running LLMs entirely on CPU is not only possible — it’s practical.

This post is a complete, end-to-end guide to running large models (13B–27B+) on CPU-only hardware, using modern quantization techniques and efficient runtimes like llama.cpp.

By the end, you’ll understand:

• Why CPU-based LLMs exist
• What quantization and inference really mean
• How to set up a Linux system for CPU inference
• How large models can realistically run without GPUs
• A reproducible workflow you can use immediately

Why Run LLMs on CPU?

Let’s address the obvious question first.

If GPUs are faster, why bother with CPU?

Because speed is not the only constraint.

CPU-based LLMs are ideal when you need:

• Low-cost experimentation (no $10K GPUs)
• Offline or air-gapped environments
• Privacy & compliance (healthcare, finance, legal)
• On-prem developer tooling
• Edge or internal R&D systems
• Predictable, reproducible environments

For many teams, “fast enough” beats “fastest possible.”

Core Concepts (No ML Background Required)

1. What Is Inference?

Inference is the act of using a trained model to generate text.

• Training = learning weights (expensive, GPU-heavy)
• Inference = reading weights + predicting tokens (much cheaper)

This guide is only about inference.

2. Why Large Models Don’t Fit on CPU (By Default)

A 27B model in FP16 format:

• 27B parameters × 2 bytes ≈ 54 GB RAM

That’s before runtime overhead.

This is why quantization exists.

Quantization Explained (Simply)

What Is Quantization?

Quantization reduces the precision of model weights to save memory and speed up inference.

Quantization:

• Reduces RAM usage by 4–6×
• Improves CPU cache efficiency
• Makes CPU inference viable

Why GGUF Format?

Modern CPU runtimes use GGUF, a binary format that:

• Packs weights + tokenizer together
• Is optimized for memory-mapped loading
• Avoids Python overhead
• Works directly with C/C++ inference engines

Think of GGUF as:

“Docker images for LLM weights.”

The CPU Inference Stack

Here’s the minimal, production-grade stack:

Raw Model Weights (HF / Google)
↓
Conversion → GGUF
↓
Quantization (Q4/Q5/Q6)
↓
CPU Runtime (llama.cpp)
↓
Optimized Linux Execution

No PyTorch runtime is needed at inference time.

System Requirements (Realistic)

Minimum Recommended Specs

• Ubuntu 22.04+
• x86_64 CPU with AVX2 (AVX512 preferred)
• 128 GB RAM for 27B models
• 16–32 CPU cores
• SSD storage

Tip: CPUs with high memory bandwidth matter more than clock speed.

Step-by-Step Quick Start

1. Install System Dependencies

sudo apt update && sudo apt install -y \
  build-essential cmake git wget \
  python3 python3-venv python3-pip \
  numactl htop perf libopenblas-dev

2. Build the CPU Inference Engine

llama.cpp is the gold standard for CPU LLM inference.

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make LLAMA_AVX2=1 LLAMA_AVX512=1 LLAMA_BLAS=1 -j$(nproc)

Verify CPU features:

lscpu | grep AVX

3. Download Model Weights

Download official model weights (example shown):

mkdir -p models/raw
cd models/raw
wget <official-model-url>

Always respect model licenses.

4. Convert to GGUF

python llama.cpp/tools/convert-hf-to-gguf.py \
  models/raw/model.safetensors \
  models/gguf/model.gguf

This step:

• Aligns tokenizer
• Normalizes weight layout
• Ensures runtime compatibility

5. Quantize the Model

./quantize models/gguf/model.gguf models/quantized/model-q4.gguf q4_k_m
./quantize models/gguf/model.gguf models/quantized/model-q5.gguf q5_k_m
./quantize models/gguf/model.gguf models/quantized/model-q6.gguf q6_k

Start with Q4. Move up if quality is insufficient.

6. Run Inference (Optimized)

numactl --cpunodebind=0 --membind=0 \
./main -m models/quantized/model-q4.gguf \
--threads 16 \
-p "Explain CPU-based LLM inference"

Tune:

• –threads

• NUMA binding

• Batch size

• Performance Expectations (Honest)

For a 27B model on CPU:

This is not chatGPT speed — but it is:

• Stable

• Cheap

• Private

• Predictable

Common Pitfalls

❌ Tokenizer mismatch

• Always convert with the correct tokenizer.

❌ Running out of memory

• Use lower quantization or fewer threads.

❌ Poor performance

Check:

• AVX support

• NUMA locality

• BLAS enabled

Final Thoughts

CPU-based LLM inference is not a workaround — it’s a legitimate engineering choice.

With the right:

• Quantization
• Runtime
• Linux tuning
• Documentation

You can run surprisingly large models on commodity hardware.

And most importantly — you understand exactly how it works.

Further Reading

• llama.cpp GitHub
• GGUF specification
• CPU vectorization (AVX2 / AVX512)
• NUMA performance tuning

1. Design & Architecture

1.1 Clear Agent Purpose

# ✅ Good: Clear, single responsibility
class IncidentAnalysisAgent:
    """
    Agent responsible for:
    - Fetching incidents from API
    - Analyzing incident data
    - Extracting actionable insights
    - Providing recommendations
    """

    def __init__(self, api_client):
        self.api_client = api_client
        self.config = config
        self.logger = logging.getLogger(__name__)

    def analyze_incidents(self):
        """Analyze single incident"""
        pass
    
    def batch_analyze(self, incidents):
        """Analyze multiple incidents"""
        pass

1.2 Separation of Concerns

# ✅ Good: Separate components
class IncidentAgent:
    def __init__(self, fetcher, parser, analyzer, recommender):
        self.fetcher = fetcher          # Fetch data  
        self.parser = parser            # Parse data
        self.analyzer = analyzer        # Analyze data
        self.writer = writer            # Write results
    
# ❌ Bad: Everything mixed together
class MonolithicAgent:
    def __init__(self, api_client):
        # Fetch, parse, analyze, and write all in one class
        pass

2. Error Handling & Resilience

2.1 Comprehensive Error Handling

import logging
from typing import Optional, Dict, Any
from functools import wraps
import time

logger = logging.getLogger(__name__)

class AIAgent:
    def __init__(self, max_retries=3, timeout=30):
        self.max_retries = max_retries
        self.timeout = timeout
    
    def retry_with_backoff(func):
        ### Decorator for retrying with exponential backoff ###
        @wraps(func)
        def wrapper(self, *args, **kwargs):
            for attempt in range(self.max_retries):
                try:
                    logger.info(f"Attempt {attempt + 1}/{self.max_retries}")
                    return func(self, *args, **kwargs)
                
                except Exception as e:
                    if attempt == self.max_retries - 1:
                        logger.error(f"All {self.max_retries} attempts failed: {e}")
                        raise
                    
                    wait_time = 2 ** attempt # Exponential backoff
                    logger.warning(f"Attempt {attempt + 1} failed. Retrying in {wait_time}s: {e}")
                    time.sleep(wait_time)
        return wrapper
    
    @retry_with_backoff
    def fetch_data(self, url):
        """Fetch data from a URL with retries and timeout"""
        response = requests.get(url, timeout=self.timeout)
        response.raise_for_status()
        return response.json()

2.2 Graceful Degradation

# ✅ Good: Provide fallback when API fails
class RobustIncidentAgent:
    def get_incidents(self):
        try:
            return self.fetch_from_primary_api()
        except Exception as e:
            logger.warning(f"Primary API failed: {e}. Falling back to secondary API.")
            return self.fetch_from_cache()      # Fallback to cached data
    
    def analyze_with_fallback(self, incident):
        try:
            return self.advanced_analysis(incident)
        except Exception as e:
            logger.warning(f"Advanced analysis failed: {e}. Using basic analysis.")
            return self.basic_analysis(incident) # Fallback to basic analysis

3. Input Validation & Simulation

3.1 Strict Input Validation

from pydantic import BaseModel, ValidationError, Field
from typing import Optional

class IncidentData(BaseModel):
    ### Validate incident data structure ###
    incident_id: str
    target_name: str
    severity: str
    timestamp: Optional[str] = None

    @validator('incident_id')
    def incident_id_not_empty(cls, v):
        if not v and len(v) == 0:
            raise ValueError('incident_id must not be empty')
        return v
    
    @validator('severity')
    def severity_valid(cls, v):
        valid_severities = {'low', 'medium', 'high', 'critical'}
        if v not in valid_severities:
            raise ValueError(f'severity must be one of {valid_severities}')
        return v
    
class IncidentAgent:
    def process_incident(self, data: dict) -> bool:
        try:
            # Validate data structure
            incident = IncidentData(**data)
            logger.info(f"Processing incident: {incident.incident_id}")
            return True
        except ValidationError as e:
            logger.error(f"Invalid incident data: {e}")
            return False

3.2 Sanitize External Inputs

import html
import re
from urllib.parse import quote

def sanitize_input(user_input: str) -> str:
    """Sanitize user input to prevent injection attacks"""
    # Remove potentially dangerous characters
    sanitized = html.escape(user_input)
    sanitized = re.sub(r'[<>\"\'%;()&+]', '', sanitized)
    sanitized = sanitized.strip()
    return sanitized

def sanitize_file_path(file_path: str) -> str:
    ### Prevent directory traversal attacks ###
    path = file_path.replace("..", "")
    path = path.replace("//", "/")
    return path

4. Logging & Monitoring

4.1 Comprehensive Logging

import logging
import json
from datetime import datetime

class StructuredLogger:
    ### Structured logging for AI agents ###

    def __init__(self, name: str):
        self.logger = logging.getLogger(name)
        self.setup_handlers()

    def setup_handlers(self):
        # Console handler
        console_handler = logging.StreamHandler()
        console_handler.setLevel(logging.INFO)

        # File handler
        file_handler = logging.FileHandler('agent.log')
        file_handler.setLevel(logging.DEBUG)

        # Formatter
        log_formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

        console_handler.setFormatter(log_formatter)
        file_handler.setFormatter(log_formatter)

        self.logger.addHandler(console_handler)
    
    def log_action(self, action: str, details: dict):
        ### Log structured action data ###
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "action": action,
            "details": details
        }
        self.logger.info(json.dumps(log_entry))

# Usage details
logger = StructuredLogger(__name_)
logger.log_action('fetch_incidents', {
    'status': 'success',
    'count': 10,
    'duration_ms': 1234
})

4.2 Monitoring & Metrics

from prometheus_client import Counter, Histogram, Guage
import time

class MonitoredAgent:
    """ Agent with monitoring and metrics """

    def __init__(self):
        # Metrics
        self.request_count = Counter(
            'agent_requests_total',
            'Total requests',
            ['method', 'status']
        )

        self.request_duration = Histogram(
            'agent_request_duration_seconds',
            'Request duration',
            ['method']
        )

        self.active_requests = Guage(
            'agent_active_requests',
            'Active requests'
        )
    
    def fetch_data(self, url):
        ### Fetch with monitoring ###
        start_time = time.time()
        self.active_requests.inc()

        try:
            response = requests.get(url)
            response.raise_for_status()

            self.request_count.labels(
                method='fetch',
                status='success'
            ).inc()

            return response.json()
        
        except Exception as e:
            self.request_count.labels(
                method='fetch',
                status='raise'
            ).inc()
            raise
        
        finally:
            duration = time.time() - start_time
            self.request_duration.labels(method='fetch').observe(duration)
            self.active_requests.dec()

5. Testing & Quality Assurance

5.1 Unit Testing

import pytest
from unittest.mock import Mock, patch

class TestIncidentAgent:
    @pytest.fixture
    def agent(self):
        ### Create test agent ###
        mock_api = Mock()
        return IncidentAgent(api=mock_api)
    
    def test_extract_target_name(self, agent):
        ### Test target name extraction ###
        description = "Target Name: xyz.example.com"
        result = agent.extract_target_name(description)
        assert result == "xyz.example.com"
    
    def test_invalid_incident_data(self, agent):
        ### Test invalid incident data handling ###
        invalid_data = {'incident_id': ''} # Empty ID
        with pytest.raises(ValueError):
            agent.process_incident(invalid_data)
    
    @patch('requests.get')
    def test_fetch_with_proxy(self, mock_get, agent):
        ### Test fetch with proxy ###
        mock_get.return_value.json.return_value = {'result': []}

        result = agent.fetch_incidents()

        assert mock_get.called
        mock_get.assert_called_with(
            timeout=30,
            proxies={'http': 'http://proxy:8080'}
        )

5.2 Integration Testing

import pytest
import responses

class TestAgentIntegration:
    @responses.activate
    def test_full_incidents_processing(self):
        ### Test complete incidents processing flow ###
        
        # Mock API response
        responses.add(
            responses.GET,
            'https://api.example.com/incidents',
            json=[{
                'id': 'INC123',
                'description': 'Target Name: xyz.example.com\nTarget Type: server',
            }]
            status=200
        )

        # Create agent and process
        agent = IncidentAgent()
        incidents = agent.fetch_incidents()

        # Assertions
        assert len(incidents) == 1
        assert inciedents[0]['incident_id'] == 'INC123'
        assert incidents[0]['target_name'] == 'xyz.example.com'

6. Configuration Management

6.1 Environment-Based Configuration

import os
from dataclasses import dataclass
from dotenv import load_dotenv

load_dotenv()  # Load .env file

@dataclass
class Config:
    """ Agent configuration """

    # API settings
    api_url: str = os.getenv('API_URL', 'https://api.example.com')
    api_timeout: int = int(os.getenv('API_TIMEOUT', '30'))
    max_retries: int = int(os.getenv('MAX_RETRIES', '3'))

    # Proxy settings
    proxy_enabled: bool = os.getenv('PROXY_ENABLED', 'false').lower() == 'true'
    proxy_url: str = os.getenv('PROXY_URL', 'https://proxy:8080')

    # Logging
    log_level: str = os.getenv('LOG_LEVEL', 'INFO')
    log_file: str = os.getenv('LOG_FILE', 'agent.log')

    # Security
    api_token: str = os.getenv('API_TOKEN', '')
    verify_ssl: bool = os.getenv('VERIFY_SSL', 'true').lower() == 'true'

    def validate(self):
        """ validate configuration """
        if not self.api_url:
            raise ValueError("API_URL must be set")
        if not self.api_token:
            raise ValueError("API_TOKEN must be set")
# Usage
config = Config()
config.validate()

6.2 .env file

# API Configuration
API_URL=https://api.example.com
API_TIMEOUT=30
MAX_RETRIES=3

# Proxy Configuration
PROXY_ENABLED=true
PROXY_URL=http://proxy:8080

# Logging
LOG_LEVEL=DEBUG
LOG_FILE=agent.log

# Security
API_TOKEN=your_api_token_here
VERIFY_SSL=true

7. Security Best Practices

7.1 Secure Credential Management

from cryptography.fernet import Fernet
import os

class SecureCredentials:
    """ Manage credentials securely """

    def __init__(self):
        # Never hardcode key - use environment variable
        key = os.getenv('ENCRYPTION_KEY')
        if not key:
            raise ValueError("ENCRYPTION_KEY must be set in environment variables")
        
        self.cipher = Fernet(key)
    
    def encrypt(self, token: str) -> str:
        """ Encrypt sensitive token """
        return self.cipher.encrypt(token.encode()).decode()
    
    def decrypt(self, encrypted_token: str) -> str:
        """ Decrypt sensitive token """
        return self.cipher.decrypt(encrypted_token.encode()).decode()

# ❌ BAD: Hardcoding credentials
api_token = "secret_token_123"

# ✅ GOOD: Using secure credential management
api_token = os.getenv('API_TOKEN_ENCRYPTED')

7.2 Rate Limiting

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import time

class RateLimitedAgent:
    """ Agent with rate limiting """

    def __init__(self, requests_per_second=1):
        self.requests_per_second = requests_per_second
        self.min_interval = 1.0 / requests_per_second
        self.last_request_time = 0
    
    def _wait_if_needed(self):
        """ Wait to maintain rate limit """
        elapsed = time.time() - self.last_request_time
        if elapsed < self.min_interval:
            time.sleep(self.min_interval - elapsed)
    
    def fetch(self, url):
        """ Fetch with rate limiting """
        self._wait_if_needed()
        self.last_request_time = time.time()
        return requests.get(url)

8. Performance Optimization

8.1 Caching

from funtools import lru_cache
from datetime import datetime, timedelta

class CachedAgent:
    """ Agent with intelligent caching """

    def __init__(self):
        self.cache = {}
        self.cache_ttl = 300 # 5 minutes
    
    def get_incidents(self, force_refresh=False):
        """ Get incidents with caching """

        cache_key = 'incidents'

        # Check cache
        if cache_key in self.cache and not force_refresh:
            cached_data, timestamp = self.cache[cache_key]
            if datetime.now() - timestamp < timedelta(seconds=self.cache_ttl):
                return cached_data
        
        # Fetch fresh data
        logger.info("Fetching fresh incidents")
        incidents = self._fetch_from_api()

        # Update cache
        self.cache[cache_key] = (incidents, datetime.now())

        return incidents
    
    def _fetch_from_api(self):
        """ Fetch from API """
        pass

8.2 Batch Processing

from typing import List, Iterator

class BatchedAgent:
    """ Process incidents in batches """

    def process_in_batches(self, incidents: List[dict], batch_size: int = 10) -> Iterator[List[dict]]:
        """ process incidents in batches to reduce memory """
        for i in range(0, len(incidents), batch_size):
            batch = incidents[i:i + batch_size]
            logger.info(f"Processing batch {i // batch_size + 1}")
           
            for incident in batch:
                yield self.process_incident(incident)

9. Documentation & Code Quality

9.1 Type Hints

from typing import List, Dict, Optional, Tuple

class DocumentedAgent:
    """ Well-documented agent """

    def extract_target_name(self, description: str) -> Optional[str]:
        """ Extract target name from incident description

        Args:
            description (str): Incident description text

        Returns:
            The Target name if found, None Otherwise

        Raises:
            ValueError: If description is empty
        
        Example:
            >>> agent = DocumentedAgent()
            >>> agent.extract_target_name("Target Name: xyz.example.com")
            'xyz.example.com'
        """
        if not description:
            raise ValueError("Description must not be empty")
        
        match = re.search(r'Target Name:\s*(\S+)', description)
        return match.group(1) if match else None
    
    def batch_process(
        self,
        incidents: List[Dict[str, Any]],
        parallel: bool = False
    ) -> Tuple[List[Dict], List[str]]:
        """ 
        Process multiple incidents

        Args:
            incidents: List of incident dictionaries
            parallel: Whether to process in parallel

        Returns:
            Tuple of (processed_incidents, error_ids)
        """
        pass

10. Observability & Debugging

10.1 Debugging Mode

import logging

class DebuggableAgent:
    """ Agent with debug mode """

    def __init__(self, debug=False):
        self.debug = debug
        self.setup_logging()
    
    def setup_logging(self):
        """ Setup logging based on debug mode """
        level = logging.DEBUG if self.debug else logging.INFO
        logging.basicConfig(level=level)
        self.logger = logging.getLogger(__name__)

    def fetch_data(self, url):
        """ Fetch with debug output """
        if self.debug:
            self.logger.debug(f"Fetching data from URL: {url}")
            self.logger.debug(f"Using proxies: {self.proxies}")
        
        response = requests.get(url)

        if self.debug:
            self.logger.debug(f"Response status: {response.status_code}")
            self.logger.debug(f"Response size: {len(response.content)} bytes")
        
        return response

10.2 Health Checks

class HealthCheckAgent:
    """ Agent with health checks """

    def health_check(self) -> Dict[str, str]:
        """ check agent health """
        return {
            'api_connectivity': self._check_api(),
            'proxy_connectivity': self._check_proxy(),
            'database_connection': self._check_database(),
            'cache_health': self._check_cache()
        }
    
    def _check_api(self) -> str:
        """ Check API connectivity """
        try:
            response = requests.get(f"{self.api_url}/health", timeout=5)
            return response.status_code == 200
        except Exception:
            return False

✅ Quick Refence Checklist

• ✅ Single responsibility principle
• ✅ Comprehensive error handling with retries
• ✅ Input validation and sanitization
• ✅ Structured logging
• ✅ Unit & Integration testing
• ✅ Configuration management
• ✅ Security best practices
• ✅ Rate limiting and Caching
• ✅ Type hints and documentation
• ✅ Health checks and monitoring
• ✅ Gradeful degradation
• ✅ Retry logic with backoff
• ✅ Environment-based configuration
• ✅ Secrets management
• ✅ Performance optimization

🚀 Complete Production-Ready Agent Template

import logging
import os
import time
from dataclasses import dataclass
from typing import List, Dict, Optional
from functools import wraps

@dataclass
class Config:
    api_url: str = os.getenv('API_URL', 'https://api.example.com')
    api_timeout: int = int(os.getenv('API_TIMEOUT', '30'))
    max_retries: int = int(os.getenv('MAX_RETRIES', '3'))
    log_level: str = os.getenv('LOG_LEVEL', 'INFO')

class ProductionAgent:
    """ Production-ready AI Agent """

    def __init__(self, config: Config):
        self.config = config or Config()
        self.logger = self._setup_logging()
        self.cache = {}
    
    def _setup_logging(self):
        """ Setup structured logging """
        logger = logging.getLogger(__name__)
        logger.setLevel(self.config.log_level)

        handler = logging.StreamHandler()
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

        handler.setFormatter(formatter)
        logger.addHandler(handler)
        return logger

    def retry(self, func):
        """ Retry decorator with backoff """
        @wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(self.config.max_retries):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == self.config.max_retries - 1:
                        self.logger.error(f"All retries failed: {e}")
                        raise
                    wait_time = 2 ** attempt
                    self.logger.warning(f"Attempt {attempt + 1} failed. Retrying in {wait_time}s: {e}")
                    time.sleep(wait_time)
        return wrapper
    
    @retry
    def fetch_data(self, url: str) -> Dict:
        """ Fetch data with retry logic """
        self.logger.info(f"Fetching from: {url}")
        response = requests.get(url, timeout=self.config.api_timeout)
        response.raise_for_status()
        return response.json()
    
    def process(self, data: Dict) -> Optional[Dict]:
        """ Process data safely """
        try:
            self.logger.info("Processing data")
            # Your processing logic here
            return data
        except Exception as e:
            self.logger.error(f"Processing failed: {e}")
            return None

# Usage
if __name__ == "__main__":
    agent = ProductionAgent(config)
    data = agent.fetch_data(f"https://api.example.com/incidents")
    processed = agent.process(data)

This comprehensive guide covers all aspects of developing professional AI agents! 🎯

Create a closed-loop incident management system where New Relic automatically creates ServiceNow incidents when CPU usage is high and resolves them when usage returns to normal.

Architecture:

1. Monitor: New Relic Infrastructure Agent monitors a local machine.
2. Trigger: NRQL Alert Policy detects CPU > 5%.
3. Transport: New Relic Workflow sends a Webhook POST request.
4. Logic: ServiceNow Scripted REST API receives the request, checks for a correlation_id, and decides whether to insert() (Create) or update() (Close).

Part I: Environment Setup

Before configuring the integration, you need to establish your “Sandbox” environment.

1. Get a ServiceNow Personal Developer Instance (PDI)

ServiceNow does not run locally; you must use their cloud sandbox.

1. Go to the ServiceNow Developer Site.

2. Sign up for a free account.

3. Click Request Instance.

4. Wait a few minutes. You will receive:

   • URL: e.g., https://devXXXXX.service-now.com
   • Username: e.g., admin
   • Password: e.g., XXXXXX(Save this securely)

5. Log in to your instance using the provided URL, username, and password.

2. Get New Relic & Install Agent

1. Sign up for a free New Relic account at New Relic Sign Up.
2. Once logged in, go to Add Data > Linux (or MacOS/Windows depending on your local machine).
3. Follow the instructions to run the installation command in your terminal.
   • Example (linux):

     curl -Ls https://download.newrelic.com/install/newrelic-cli/scripts/install.sh | bash && sudo NEW_RELIC_API_KEY=YOUR_API_KEY NEW_RELIC_ACCOUNT_ID=YOUR_ACCOUNT_ID newrelic install
     

4. Verify data is flowing by going to Infrastructure > Hosts in New Relic.

Part II: ServiceNow Configuration(The Listener)

We will build a “Smart Endpoint” that accepts the alert data.

1. Create the API Container

1. In your ServiceNow PDI, use the Filter Navigator (top left) to search for “Scripted REST APIs”.
2. Click New .
   • Name: NewReliWebhook
   • API ID: newrelic_handler
3. Click Submit .

2. Create the Resource(The Endpoint)

1. Open the NewRelicWebhook record you just created.
2. Scroll down to the Resources tab and click New .
   • Name: HandleAlert
   • HTTP Method: POST
   • Relative Path: /incident

3. The Script: Delete the default code in the Script field and paste the following “Traffic Cop” logic:

    (function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
        // 1. Parse the incoming JSON from New Relic
        var requestBody = request.body.data;
        var nrIssueId = requestBody.issue_id;
        var nrState = requestBody.current_state;
        var nrTitle = requestBody.title;
        var nrCpu = requestBody.cpu_value;
   
        // 2. Look for an existing Incident with this New Relic Issue ID
        var gr = new GlideRecord('incident');
        gr.addQuery('correlation_id', nrIssueId);
        gr.query();
   
        if (gr.next()) {
            // --- SCENARIO: UPDATE EXISTING INCIDENT ---
            if (nrState === 'CLOSED') {
                gr.state = 6; // Set to 'Resolved' (Standard ID)
                gr.close_code = 'Solved (Permanently)';
                gr.close_notes = 'Cloased automatically by New Relic. Alert cleared.';
                gr.work_notes = "New Relic reports issue is closed.";
                gr.update();
   
                return { status: "Incident Resolved", number: gr.number };
            } else {
                // Optional: Update work notes if it's just an update
                gr.work_notes = "Received update from New Relic. Current CPU: " + nrCpu + "%";
                gr.update();
                return { status: "Incident Updated", number: gr.number };
            }
        } else {
            // --- SCENARIO: CREATE NEW INCIDENT ---
            if (nrState == 'ACTIVATED') {
                var newInc = new GlideRecord('incident');
                newInc.initialize();
                newInc.short_description = "New Relic Alert: " + nrTitle;
                newInc.description = "Automatic alert from New Relic.\nIssue ID: " + nrIssueId + "\nCPU Usage: " + nrCpu + "%";
                newInc.correlation_id = nrIssueId; // THIS IS CRITICAL
                newInc.caller_id = gs.getUserID(); // Assign to API user
                newInc.urgency = 2; // High
                newInc.insert();
   
                return { status: "Incident Created", number: newInc.number };
            }
        }
    })(request, response);
   

4. Click Submit.
5. Important: Note your full Endpoint URL. It will be: https://devXXXXX.service-now.com/api/x_scope_id/newrelic_handler/incident

Part III: New Relic Configuration(The Trigger)

1. Configure the Destination

1. In New Relic, go to Alerts & AI > Destinations.
2. Select Webhook .
   • Endpoint URL: Your ServiceNow Scripted REST API URL from above.
   • Authorization Type: Select Basic Authentication. Use your PDI admin username and password.
3. Name it “ServiceNow PDI”.

   2. Configure the Workflow (Payload)

4. Go to Alerts & AI > Workflows .
5. Click Add Workflow .
   • Filter: Policy Name = 'Low CPU Test' ( we will create policy in the next step)
6. Notify: Select your “ServiceNow PDI” destination.
7. Payload Template: This defines exactly what data is sent to script. Paste this JSON:

    {
        "issue_id": "",
        "title": "",
        "current_state": "",
        "cpu_value": "N/A"
    }
   

   3. Create the Alert Policy

8. Go to Alerts & AI > Conditions (Policies) .
9. Create a New Policy named Local CPU Test.
10. Create a Condition:
    • Product: NRQL
    • Query:

        SELECT average(cpuPercent) FROM SystemSample WHERE `hostname = 'YOUR_HOSTNAME'
      

    • Threshold: Critical if query returns value above 5 (Low threshold for easy testing) for at least 1 minute.
    • Advanced Signal Settings: “Close open incidents on signal loss” -> set to 5 minutes.

Part IV: Execution & Testing

Test A: Unit Test via cURL (Command Line)

Verify your serviceNow script works without waiting for New Relic.

1. Create Incident(Simulate Alert)

curl "https://<YOUR_INSTANCE>.service-now.com/api/<YOUR_SCOPE_ID>/newrelic_handler/incident" \
--request POST \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--user 'admin:<YOUR_PASSWORD>' \
--data '{
  "issue_id": "TEST12345",
  "title": "Test High CPU Alert",
  "current_state": "ACTIVATED",
  "cpu_value": "95"
}'

Result</b>: Check ServiceNow incidents table. You should see a new ticket.

2. Resolve Incident(Simulate Recovery)

curl "https://<YOUR_INSTANCE>.service-now.com/api/<YOUR_SCOPE_ID>/newrelic_handler/incident" \
--request POST \
--header "Accept: application/json" \
--header "Content-Type: application/json" \
--user 'admin:<YOUR_PASSWORD>' \
--data '{
  "issue_id": "TEST12345",
  "title": "Test High CPU Alert",
  "current_state": "CLOSED",
  "cpu_value": "20"
}'

Result</b>: The ticket created in step 1 should now be marked as “Resolved”.

Test B: End-to-End Integration (“Smoke Test”)

Now, use the strees tool to forced a real CPU spike on your local machine.

1. Install Stress Tool:
   • Linux: sudo apt-get install stress
   • MacOS: brew install stress
2. Run Stress:
   • stress --cpu 8 --timeout 180 (Runs for 3 minutes).
3. Monitor:
   • T+0 to T+1 min: New Relic detects high CPU.
   • T+1 min: New Relic triggers Alert -> Webhook -> ServiceNow creates Incident.
   • T+3 min: Stress command ends, CPU drops.
   • T+5 to T+8 min: New Relic closes Alert -> Webhook -> ServiceNow resolves Incident.

Appendix: Troubleshooting

An alert fires at 2am. An engineer wakes up, SSH’s into a server, and begins the manual loop — pulling logs, scanning for errors, guessing what to check next. This loop can take 15 to 45 minutes before the real diagnosis even begins. Multiply that by every incident across every team in your organisation, and you have thousands of engineering hours lost every year to work that is repetitive, stressful, and largely automatable.

I’ve been on that on-call rotation. I know what it costs — not just in time, but in cognitive load, in missed context, and in the compounding pressure of an active incident. So I built incopilot: a CLI tool that automates the entire first-pass triage so engineers can skip straight to actual problem-solving.

This post walks through the architecture, the design decisions, and exactly how to build it yourself. Everything is open source at https://github.com/AutoShiftOps/incopilot.

Architecture

Project structure

incopilot/
  __init__.py
  cli.py          # argument parsing + console output
  collectors.py   # journalctl, docker logs, file, bundle
  analyzer.py     # pattern detection + line normalization
  reporter.py     # report.md / report.json generation
  config.py       # patterns, golden-signal map, safe-command list
scripts/
  demo_generate_sample_logs.py
posts/
requirements.txt
pyproject.toml
README.md

Setup

git clone https://github.com/AutoShiftOps/incopilot.git
cd incopilot
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Quick test (no real services needed)

python scripts/demo_generate_sample_logs.py
python -m incopilot file --path sample.log
ls out/

Systemd journal triage

python -m incopilot journal --unit nginx --since "30 min ago"

Docker triage

python -m incopilot docker --container my-api --since 1h

Both sources (bundle)

python -m incopilot bundle \
  --unit nginx \
  --container my-api \
  --since-journal "30 min ago" \
  --since-docker 1h

What you get

out/report.md — paste into your incident doc
out/report.json — attach to a ticket or POST to a webhook

What to improve next

• Per-service pattern packs (nginx, postgres, java, node)
• Slack/Teams webhook posting (--webhook <url>)
• Unit tests + GitHub Actions CI
• Scheduled timer (systemd timer unit) for proactive reports

Sudhakar Sajja is an Application Architect at TechMahindra with 13 years of experience across protocol testing, SDET, DevOps, and cloud architecture. He specialises in AI-powered DevOps operations — building tools that use LLMs to replace manual incident response and query diagnostics. He writes weekly at AutoShiftOps (autoshiftops.com) and built QueryTuner (querytuner.com), an AI-driven SQL query analysis tool. Based in Mississauga, Canada.

Introduction

At AutoShiftOps, we don’t build demos — we build systems that survive real-world uncertainty.

AI trading is often presented as a shortcut to profits:

“Train a model, predict prices, make money.”

That’s not how markets work.

This post walks through how to design an AI-powered stock trading bot using Python, focusing on engineering discipline, backtesting, and system thinking, rather than hype.

You’ll also find links to other published versions of this post on Medium, Substack, Patreon, and DEV.to at the bottom.

What Is an AI Trading Agent?

An AI trading agent is not just a model.

It’s a system that:

• Observes market data
• Generates signals
• Applies trading rules
• Manages risk
• Evaluates performance

Remove any one of these pieces, and the system breaks.

AutoShiftOps Trading Agent Architecture

Market Data
↓
AI Prediction Engine
↓
Strategy & Risk Rules
↓
Backtesting Engine
↓
Broker Execution Layer

This separation keeps the system testable, explainable, and production-ready.

Step 1: Market Data

We start with historical stock prices (for example, from Yahoo Finance).

Clean, consistent data matters more than fancy indicators at this stage.

Goals:

• Stable inputs
• Reproducible experiments
• Minimal noise

Step 2: AI Model (Why LSTM?)

Markets are time-series data.

LSTM (Long Short-Term Memory) models work well because they:

• Capture temporal patterns
• Handle noisy signals better than linear models

⚠️ Reminder: The model provides signals, not decisions.

Step 3: Prediction Is Not Trading

Prediction alone does not generate profits.

Trading requires:

• Rules
• Constraints
• Risk limits

At AutoShiftOps, AI output is treated as input, not authority.

Step 4: Strategy Rules

Example simple rule layer:

Doing nothing is a valid action.
Over-trading kills strategies faster than bad models.

Step 5: Backtesting

Backtesting is non-negotiable.

If you skip it:

• You are guessing
• You are curve-fitting
• You are gambling

Backtesting reveals:

• Drawdowns
• Trade frequency
• Capital erosion
• Market regime sensitivity

Many “great” models fail here — and that’s a good thing.

Lessons Learned

• AI didn’t make me profitable — it made me disciplined
• Consistency comes from rules, not intelligence
• Engineers have an advantage because they think in systems, failure modes, and feedback loops
• Trading is a software system problem first, finance second

Cross-Published Versions

This post is also available on:

• Medium: Read on Medium
• Substack: Read on Substack
• Patreon: Read on Patreon
• DEV.to: Read on DEV

Disclaimer

This content is for educational purposes only.
It is not financial advice.

About AutoShiftOps

AutoShiftOps explores the intersection of:

• AI agents
• Automation
• DevOps
• Real-world engineering systems

We focus on practical AI — systems that survive production, not demos.

Over time, SRE and DevOps teams add more checks, more monitors, more “smart” tools. The result is often alert fatigue: hundreds of alerts, dozens of dashboards, and still no clear answer to the question “Do we wake someone up for this?”

In this post we’ll do two things:

• Design a simple Alert Decision Layer – an explicit step between alerts and humans.
• Build a small, working CLI in Python called alertdecider that you can run today.

By the end you’ll have:

• A clear architecture for routing alerts to page, ticket, aggregate, or suppress.
• A CLI that reads alerts from JSON, applies transparent rules, and outputs a Markdown + JSON report.
• A foundation you can later extend with AI without turning your incident pipeline into a black box.

The problem: alerts without decisions

Alert fatigue has been written about a lot, but the symptoms are consistent:

• On-call engineers get spammed by low-value alerts, become desensitized, and miss the important ones.
• Teams add more tools and even AI summaries, but humans still have to manually decide “page vs ticket vs ignore” for each signal.
• Runbooks and SLOs exist, but the mental model that turns raw alerts into decisions lives in people’s heads.

What’s missing is a small decision layer that sits between your alert sources and your incident response.

That layer should be:

• Transparent – decisions come from rules everyone can understand and modify.
• Context-aware – severity alone isn’t enough; you need service tier, environment, and history.
• Composable – easy to extend later with AI-based explanations instead of replacing human judgment.

That’s what we’ll build.

Design principles

Before touching code, let’s describe the design as an architect:

1. Normalize first, then decide
   Every alert source has its own JSON schema. We’ll normalize into a simple Alert model (id, name, service, severity, environment, fingerprint, timestamps) before any rules.

2. Service profiles matter
   A critical alert on a tier1, SLO-critical service is not the same as a critical on a best-effort internal tool. We encode this in a ServiceProfile model loaded from services.yml.

3. History influences action
   A fingerprint fired once is different from a fingerprint that fired 40 times today. We’ll use a small History model to treat flapping/noisy alerts differently.

4. Rules are code, not magic
   The AlertDecisionEngine is a small, explicit rule engine. The first version has no AI – just clear if/else policies you can review in code.

5. Decisions must be explainable
   Every decision carries a reason string. This is where AI can plug in later to generate richer explanations, but today it’s hand-written and predictable.

Architecture

Here’s the high-level architecture of the Alert Decision Layer:

1. Sources
   • Prometheus Alertmanager, PagerDuty, or any system that can export alerts as JSON.
2. Normalizer (loader.py)
   • Reads raw JSON and maps each item into an Alert dataclass with a consistent shape.
3. Context loaders
   • services.yml → ServiceProfile per service (tier, SLO criticality, owner).
   • history.json → History per fingerprint (how often we’ve seen this alert recently).
4. Decision engine (engine.py)
   • For each Alert, looks up the service profile and history and decides one of: page, ticket, aggregate, suppress.
5. Reporters (reporter.py)
   • Writes a human-friendly decision_report.md and a machine-friendly decision_report.json.
   • Prints a table to the terminal so you can sanity-check the decisions quickly.

This is intentionally simple: a single CLI binary that can be run locally, in CI, or as a cron/systemd job.

Project structure

alertdecider-agent/
  __init__.py
  __main__.py
  cli.py          # CLI entry point
  models.py       # Alert, ServiceProfile, History dataclasses
  loader.py       # Load alerts/services/history from JSON/YAML
  engine.py       # Decision rules (AlertDecisionEngine)
  reporter.py     # Console + Markdown/JSON output
examples/
  alerts.json     # Example alerts
  services.yml    # Service risk profiles
  history.json    # Simple alert history

You can clone the repo and run it against the example data to see how everything works before wiring it into your own tooling.

Setup

git clone https://github.com/AutoShiftOps/alertdecider.git
cd alertdecider
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The dependencies are minimal:

• rich for nice CLI tables.
• PyYAML for services.yml.

Modelling alerts, services, and history

We start by defining three small dataclasses in models.py:

• Alert – normalized alert coming from any source (Prometheus, PagerDuty, etc.).
• ServiceProfile – tier and SLO information for a service.
• History – how often an alert fingerprint has fired in the last 24 hours.

This gives us a clean domain model to talk about decisions.

Implementing the decision engine

engine.py contains the AlertDecisionEngine with a handful of clear rules.

Examples of policies encoded:

• Low-severity non-prod noise
  info / debug in non-prod → suppress.

• Critical on tier1, SLO-critical services
  critical + tier1 + slo_critical=true → page.

• Flapping alerts
  count_24h >= 20 → aggregate (don’t page for every occurrence; treat as noisy/frequent).

• Warnings on tier1
  warning on tier1 → ticket (not a page, but still important to track).

• Default behavior
  Prod alerts without a specific rule → ticket.
  Non-prod alerts without a specific rule → suppress.

The whole point is that you can read this engine like a policy document.

Running the CLI with example data

With the repo set up, try the example:

python -m alertdecider-agent   --alerts examples/alerts.json   --services examples/services.yml   --history examples/history.json   --out-dir out

cat out/decision_report.md

You’ll see a table in your terminal and a Markdown report with each alert’s decision and reason.

With the provided examples, you should observe:

• A critical alert on a tier1, SLO-critical service in prod → page.
• A warning on a tier1 service → ticket.
• A flapping alert with high count_24h → aggregate.
• A low-severity info alert in staging → suppress.

From here, you can plug in your own alerts JSON and adjust services.yml and engine.py to match your reality.

Where AI fits in (later)

Right now, alertdecider-agent is deliberately rule-based and transparent.

Once you’re happy with the structure, you can start experimenting with AI in safe, incremental ways:

• Richer explanations
  Feed the normalized alert + decision into an LLM to generate a human-friendly explanation and suggested next steps.

• Runbook suggestions
  Use alert name, service name, and history to suggest a runbook link or dashboard.

• Rule tuning hints
  Analyze real decision logs to recommend new rules (e.g., “this pattern is always suppressed but often escalated manually”).

The important part is: the control plane (what gets paged vs ticketed vs suppressed) remains in code you own.

Next steps for your team

If you try this out, here are some directions to evolve it:

• Add time-of-day and on-call load
  Don’t page at 03:00 for something that can safely be a ticket until business hours.

• Persist history more robustly
  Replace history.json with a small SQLite table or a lightweight time-series store.

• Integrate with your actual alert pipeline
  Wire alertdecider into Alertmanager/PagerDuty via webhooks, or run it as part of your incident ingestion path.

• Measure impact
  Track how many alerts are suppressed or aggregated, and how many pages you avoided without missing true incidents.

This is the kind of small, opinionated tool that can pay off quickly for SRE/DevOps teams drowning in alerts – and it’s a great foundation for more advanced AI-assisted incident management.

This post shows how to build a lightweight bash watchdog that:

1. Monitors container health via Docker health checks
2. Restarts unhealthy containers
3. Integrates with systemd for daemon-like behavior
4. Logs everything for incident review

The Self-Healing Architecture

Docker has built-in restart policies (--restart unless-stopped), but they don’t respect health checks. A container can be “running” but unhealthy (app crashed, dependencies down, etc.).

Our script loops every 30 seconds:

1. Query Docker API for container health
2. Restart unhealthy containers
3. Log actions to systemd journal
4. Repeat

Step 1: Docker Health Checks (the foundation)

First, ensure your containers have proper health checks in docker-compose.yml or docker run:

services:
  web:
    image: nginx
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

This marks containers healthy, unhealthy, or starting.

Step 2: The Self-Healing Watchdog Script

Save this as /usr/local/bin/docker-autoheal.sh:

#!/usr/bin/env bash
set -euo pipefail

# Config
CHECK_INTERVAL=30
LOG_FILE="/var/log/docker-autoheal.log"
CONTAINER_LABEL="autoheal=true"  # Label your containers

log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $*" | tee -a "$LOG_FILE"
}

heal_containers() {
  local unhealthy=($(docker ps --filter "health=unhealthy" --filter "label=$CONTAINER_LABEL" --format ""))
  
  for container in "${unhealthy[@]}"; do
    log "RESTARTING UNHEALTHY CONTAINER: $container"
    
    # Graceful stop first
    docker stop "$container" || true
    
    # Hard kill after timeout
    docker kill "$container" || true
    
    # Restart with original command
    docker start "$container"
    
    log "RESTARTED: $container"
  done
}

# Main loop
log "Docker Auto-Heal started (check interval: ${CHECK_INTERVAL}s)"
while true; do
  heal_containers
  sleep "$CHECK_INTERVAL"
done

Step 3: Run as a Systemd Service (production-ready)

Create /etc/systemd/system/docker-autoheal.service:

[Unit]
Description=Docker Container Auto-Healer
After=docker.service
Requires=docker.service

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/docker-autoheal.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable docker-autoheal.service
sudo systemctl start docker-autoheal.service

Check status:

sudo systemctl status docker-autoheal.service
sudo journalctl -u docker-autoheal.service -f

Step 4: Label Your Containers

Tag containers you want to auto-heal:

docker run -d \
  --label "autoheal=true" \
  --restart unless-stopped \
  --health-cmd="curl -f http://localhost:8080/health || exit 1" \
  nginx

Step 5: Advanced Features

Grace Period (avoid restart loops)

# Add to script before restart
if docker inspect "$container" --format '' | grep -q "10"; then
  log "SKIPPING: $container restart count too high (restart loop?)"
  continue
fi

Webhook Alerts

curl -X POST -d "{\"container\":\"$container\",\"action\":\"restart\"}" "$WEBHOOK_URL"

Multi-Host (Docker Swarm)

Use labels + a central orchestrator or run the script on each host. ​

Testing It (safely)

Spin up a container with a failing health check:

docker run -d \
  --name test-fail \
  --label "autoheal=true" \
  --health-cmd="exit 1" \
  alpine sleep infinity

Watch it get restarted:

journalctl -u docker-autoheal.service -f

Limitations + When to Upgrade

This works great for:

• Small deployments / homelabs
• Edge services / single-host apps
• Dev/staging environments

Upgrade to:

• Kubernetes: liveness/readiness probes + pod disruption budgets
• Docker Swarm: service replicas + constraints
• Nomad: health checks + restart stanzas

Summary Table (Copy/Paste)

This post is a field guide for the three most common container “murders”: Memory (OOMKilled), CPU Throttling, and I/O Saturation. We’ll diagnose each using the docker stats + Linux host tools workflow we established last week.

Scenario 1: The “Silent” Death (OOMKilled)

Symptom: The container restarts randomly. No error logs in the application output because it was killed instantly by the kernel.

1. Confirm it was an OOM Kill

Docker knows why the container died. Ask it:

docker inspect <container> --format ''
# Output: true

Or check the specific exit code (137 = 128 + 9 SIGKILL):

docker inspect <container> --format ''
# Output: 137

2. Find the “Smoking Gun” in Kernel Logs

If Docker confirms it, see exactly when the kernel snapped. Run this on the host:

dmesg -T | grep -i "killed process"

You’ll see a line like: Out of memory: Killed process 1234 (node) total-vm:2048kB, anon-rss:1024kB.

3. The Fix

• Immediate: Bump the memory limit if the host has capacity.

    docker update --memory 2g <container>
  

• Root Cause: Check your application for memory leaks. If it’s Java, check the heap settings (-Xmx). If it’s Node, check the GC behavior.

Scenario 2: The “Slow” Death (CPU Throttling)

Symptom: App is running but incredibly slow. Latency spikes. Health checks time out. ​

1. Check if it’s throttling

Linux cgroups enforce CPU limits by “pausing” your process when it uses its quota. It doesn’t kill the app; it just freezes it for milliseconds at a time.

Check docker stats first:

docker stats --no-stream

If CPU % is consistently near 100% of your configured limit (e.g., if you gave it 0.5 CPUs and it’s at 50%), you are being throttled.

2. Verify Throttling in cgroups

Look at the raw cgroup metrics (works on cgroup v1/v2):

# Find the container ID
docker inspect <container> --format ''

# Check throttle stats (path varies by distro, commonly:)
cat /sys/fs/cgroup/cpu/docker/<long-id>/cpu.stat

Look for nr_throttled and throttled_time. If these numbers are rising, your app is gasping for air. ​

3. The Fix

• Remove the limit temporarily to prove it’s the bottleneck.

    docker update --cpus 0 <container>
  

• Tune requests: If the app needs that CPU, increase the limit. If it’s a bug (infinite loop), profile the app. ​

  Scenario 3: The “Gridlock” (Disk I/O Saturation)

  Symptom: The container becomes unresponsive, docker ps hangs, or logs stop writing. ​

  1. Identify the I/O Hog

  Is it the container or the neighbor?

# Check host I/O
iostat -x 1 5

If %util is >80%, the disk is saturated.

2. Blame the Container

Use pidstat (part of sysstat) to find which process is thrashing the disk:

pidstat -d 1

Look for the PID with high kB_rd/s or kB_wr/s. Match that PID back to a container:

docker inspect --format '' <container>

3. The Fix

• Limit the blast radius: Set a Block I/O limit on the greedy container so it doesn’t kill the host.

    docker update --blkio-weight 100 <container>  # Low priority (default 500)
  

• Move logs: Ensure your app isn’t logging debug data to the container’s JSON log driver (which writes to disk). Use a log shipper or write to stdout sparingly. ​

Bonus: Network Connectivity Issues

Symptom: “Connection refused” or timeouts between containers.

1. The “Can I reach it?” Check

Don’t guess. Enter the container’s namespace:

docker exec -it <source-container> sh
# Inside:
ping <target-container-name>
nc -zv <target-container-name> <port>

2. If DNS fails

Docker has its own internal DNS. Check /etc/resolv.conf inside the container:

cat /etc/resolv.conf

It should point to Docker’s embedded DNS server (usually 127.0.0.11). If it’s missing or wrong, check your daemon config. ​

Summary Checklist (Copy/Paste)

Next Steps

Now that you can debug containers manually, how do you automate this? Next week, we’ll build a “Self-Healing” Bash Script that detects these states and alerts you automatically.

This guide shows a practical monitoring stack you can run from any Docker host:

1. Docker-level commands (docker stats, docker inspect, docker logs)
2. Host Linux tools (ps/top/free/df/iostat/ss/journalctl)
3. Kernel primitives: cgroups (resource limits/accounting) and namespaces (isolation)

1) Start with docker stats (the fastest signal)

docker stats streams runtime metrics for containers, including CPU%, memory usage/limit, network I/O, and block I/O.

docker stats

Common workflows:

docker stats --no-stream          # Snapshot (good for scripts)
docker stats <container_name>     # Focus on one container

How to interpret it (in plain language)

• CPU%: who’s burning compute right now.
• MEM USAGE / LIMIT: how close you are to the memory ceiling.
• NET I/O: traffic spikes, retries, or unusual egress.
• BLOCK I/O: slow disks, chatty logging, or heavy read/write workloads.

2) Jump from “container name” → “what is it?”

Once you identify a hot container, immediately gather identity + configuration.

docker ps
docker inspect <container> | less

Useful inspect questions:

• What image/tag is running?
• What env vars/config are set?
• What ports and volumes are attached?
• Are there memory/CPU limits configured?

3) Logs: confirm symptoms fast

docker logs --tail 200 <container>
docker logs -f <container>

This is often enough to spot:

• crash loops
• OOM errors / memory pressure
• upstream timeouts
• DB connection exhaustion

4) Understand why it’s happening: cgroups + namespaces (the mental model)

Docker relies on Linux kernel features:

• Namespaces isolate views of processes, networking, mounts, etc.
• cgroups control and account for resources like CPU, memory, and I/O.

Why this matters during incidents:

• A container can be “slow” because it’s CPU-throttled, not because the app code suddenly got worse.
• A container can restart because it hit its memory limit and the kernel’s OOM behavior targeted its processes.

5) Host-level confirmation (tie back to your Linux monitoring toolkit)

When docker stats shows a spike, verify on the host to avoid false conclusions.

CPU hogs

ps aux --sort=-%cpu | head -15

Memory pressure

free -h

Disk full / log explosions

df -h
du -sh /var/lib/docker/* 2>/dev/null | sort -h | tail -10

Disk I/O saturation

iostat -x 1 3

Unexpected listeners / traffic patterns

ss -tuln

These host checks help you decide whether you’re dealing with a single container or a node-wide saturation problem. ​

6) What to do with the data (action mapping)

Use the shortest safe path to stability:

• CPU high + latency rising

  • If CPU is legitimately needed: scale out / add capacity.
  • If CPU is throttled: revisit limits/requests (or container CPU shares).

• Memory near limit

  • If memory leak suspected: restart as mitigation + open an issue with heap profiling.
  • If limit too low for normal peaks: adjust limit carefully and monitor.

• Block I/O high

  • Check log volume and disk saturation; reduce noisy logs or move logs off disk.
  • Consider storage performance constraints and workload patterns.

• Network I/O abnormal

  • Look for retries, timeouts, DDoS/abuse patterns, or upstream issues.

7) Copy/paste triage sequence (5 minutes)

# 1) Find the hot container
docker stats --no-stream

# 2) Identify it
docker ps
docker inspect <container> | less

# 3) Check symptoms
docker logs --tail 200 <container>

# 4) Confirm on host (avoid guessing)
ps aux --sort=-%cpu | head -10
free -h
df -h
iostat -x 1 3
ss -tuln

Next in the series

Next post: Container Troubleshooting Playbook (common symptoms → commands → safe mitigations → root cause follow-ups).

Incidents are stressful when the team is improvising. A simple runbook reduces MTTR by making response repeatable, not heroic. ​ This post provides a ready to use incident response runbook template plus a practical Linux triage checklist you can run from any box.

What this runbook optimizes for

• Fast acknowledgement and clear ownership (Incident Commander + roles).
• Early impact assessment and severity assignment to avoid under/over‑reacting.
• Communication cadence and “known/unknown/next update” structure that builds trust.
• Evidence capture (commands + logs) to support post‑incident review.

The incident runbook template

Copy this into your internal wiki, README, Notion, or ops repo. ​

1. Trigger

   Triggers:

   • Monitoring alert / SLO breach
   • Customer report escalated
   • Internal detection (logs, latency spikes, error spikes)
2. Acknowledge (0–5 minutes)
   • Acknowledge page/alert in your paging system.
   • Create an incident channel: #inc-YYYYMMDD-service-shortdesc.
   • Assign Incident Commander (IC) and Comms Lead.
   • Start an incident document: timeline + links + decisions. ​

3. Assess severity (5–10 minutes)

   Answer quickly:

   • What’s impacted (service, region, feature)?
   • How many users / revenue / compliance impact?
   • Is impact ongoing and spreading?

   Suggested severity:

   • SEV1: Major outage / severe user impact; immediate coordination.
   • SEV2: Partial outage / significant degradation; urgent but controlled.
   • SEV3: Minor impact; can be handled async.

4. Stabilize first (10–30 minutes)

   Goal: stop the bleeding before chasing root cause.

   Typical mitigations:

   • Roll back the last deploy/config change.
   • Disable a feature flag.
   • Scale up/out temporarily.
   • Fail over if safe.
   • Rate-limit or block abusive traffic.

5. Triage checklist (host-level) Run these to establish the baseline quickly (copy/paste friendly).

   CPU

    ps aux --sort=-%cpu | head -15
   

   Alert cue: any process >50% sustained.

   Memory

    free -h
   

   Alert cue: available <20% total RAM.

   Disk

    df -h
    du -sh /var/log/* 2>/dev/null | sort -h | tail -10
   

   Alert cue: any filesystem >90%. ​

   Disk I/O

    iostat -x 1 3
   

   Alert cue: %util >80%, await >20ms.

   Network listeners

    ss -tuln
   

   Alert cue: unexpected listeners/ports. ​ Logs (example: nginx)

    journalctl -u nginx -f
   

   Alert cue: 5xx errors spiking.

6. Comms cadence (keep it boring)
   

   SEV1: updates every 10–15 minutes.​
   SEV2: updates every 30 minutes.
   SEV3: async updates acceptable.
   
   ​ Use this structure:

   • What we know
   • What we don’t know
   • What we’re doing now
   • Next update at: TIME ​
7. Verify resolution
   • Confirm user impact is gone (synthetic checks + error rate + latency).
   • Confirm saturation is back to normal (CPU/memory/disk/I/O).
   • Watch for 30–60 minutes for regression.
8. Close and learn (post-incident)
   • Write a brief timeline (detection → mitigation → resolution).
   • Capture what worked, what didn’t, and what to automate.
   • Create follow-ups: alerts tuning, runbook updates, tests, guardrails.

Bonus: “Golden signals” lens for incidents

When you’re lost, anchor on the four golden signals:

• Latency (are requests slower?)
• Traffic (is demand abnormal?)
• Errors (is failure rate rising?)
• Saturation (are resources hitting limits?)

This keeps triage focused on user impact and system limits, not vanity metrics. ​

Download / reuse

If you reuse this template internally, make one improvement immediately: add links to dashboards, logs, deploy history, and owners for each service. Your future self will thank you.