# HEARTBEAT — System Health Monitoring Configuration

> The pulse of the GhostFoundry-Syndicate platform. If the heartbeat flatlines, something is wrong.

---

## Overview

The Heartbeat system continuously monitors the health of every component in the platform: agents, databases, APIs, integrations, workflows, and the Dark Factory. It provides real-time status, alerts, and automated responses to degradation.

## Component Types

| Type | Description | Examples |
|------|-------------|----------|
| `system` | Core platform infrastructure | Node.js runtime, Next.js server, memory |
| `agent` | Ghost Agents | Tes, Ledger, Harmony, Pipeline, etc. |
| `database` | Data stores | PostgreSQL, Prisma client |
| `api` | API endpoints | REST routes, webhooks |
| `integration` | External services | Telegram, Twilio, ElevenLabs, GitHub |
| `workflow` | Business processes | Approval chains, data pipelines |
| `factory` | Dark Factory | Code generation pipeline stages |

## Health Statuses

| Status | Meaning | Color |
|--------|---------|-------|
| `healthy` | Operating normally | 🟢 Green |
| `degraded` | Functional but impaired | 🟡 Yellow |
| `critical` | Severe issues, intervention needed | 🔴 Red |
| `offline` | Component unreachable | ⚫ Black |
| `unknown` | Status cannot be determined | ⚪ Gray |

## Default Configuration

```yaml
heartbeat:
  interval_ms: 30000        # Check every 30 seconds
  timeout_ms: 5000           # 5 second response timeout
  max_missed_beats: 3        # Degraded after 3 missed
  critical_threshold: 5      # Critical after 5 missed
  offline_threshold: 10      # Offline after 10 missed
```

## Component Health Metrics

Each component reports:

| Metric | Type | Description |
|--------|------|-------------|
| `status` | HealthStatus | Current health state |
| `statusMessage` | string | Human-readable status detail |
| `latency` | number (ms) | Response time |
| `uptime` | number (%) | Availability percentage |
| `memoryUsage` | number (%) | Memory consumption |
| `cpuUsage` | number (%) | CPU consumption |
| `errorRate` | number (%) | Error rate in recent window |
| `requestCount` | number | Requests in current window |
| `lastHeartbeat` | timestamp | Last successful check |
| `missedBeats` | number | Consecutive missed heartbeats |
| `consecutiveFails` | number | Consecutive failure count |
| `customMetrics` | Record | Component-specific metrics |

## System Health Summary

The overall system health is computed as:

```
overallStatus = worst status across all components
healthScore = weighted average (0-100)
alerts = { critical: count, warning: count }
metrics = {
  agentCount: total registered agents,
  activeAgents: agents currently running,
  healthyAgents: agents in healthy state,
  workflowsActive: running workflows,
  workflowsFailed: failed workflows,
  tasksQueued: pending Dark Factory tasks,
  tasksFailed: failed Dark Factory tasks,
  avgLatency: average response time,
  errorRate: system-wide error rate
}
```

## Heartbeat Events

| Event Type | Trigger | Action |
|------------|---------|--------|
| `pulse` | Regular heartbeat check | Log metrics |
| `status_change` | Component status transitions | Alert + log |
| `flatline` | Component goes offline | Alert + escalate |
| `recovery` | Component returns from degraded/critical | Log + notify |

## Monitoring Targets

### Agents to Monitor

| Agent | ID | Critical? |
|-------|----|-----------|
| Tes (CEO) | `agent-ceo` | ✅ Yes |
| Ledger (Finance) | `agent-finance` | ✅ Yes |
| Harmony (HR) | `agent-hr` | No |
| Pipeline (Sales) | `agent-sales` | ✅ Yes |
| Stockwatch (Inventory) | `agent-inventory` | No |
| Sentinel (Compliance) | `agent-compliance` | ✅ Yes |
| Milestone (Projects) | `agent-projects` | No |
| Conductor (Operations) | `agent-operations` | No |
| Nexus (CRM) | `agent-crm` | No |

### Integrations to Monitor

| Integration | Health Check | Critical? |
|-------------|--------------|-----------|
| PostgreSQL | Connection test | ✅ Yes |
| Telegram Bot | API ping | No |
| Twilio | Account status check | No |
| ElevenLabs | API key validation | No |
| HeyGen | API health check | No |
| GitHub | API rate limit check | No |
| OpenAI | Model availability check | ✅ Yes |
| S3 Storage | Bucket accessibility | ✅ Yes |

### Dark Factory Stages to Monitor

| Stage | Health Indicator |
|-------|------------------|
| Intent Parser | Parse success rate |
| Schema Generator | Schema validation rate |
| Code Generator | Code compilation rate |
| Test Generator | Test generation success rate |
| Validator | Validation pass rate |
| Deployer | Deployment success rate |
| Orchestrator | Pipeline completion rate |

---

## Alert Escalation

1. **Warning** (degraded) → Log + dashboard indicator
2. **Critical** → Log + Telegram alert to Scott + Sentinel notified
3. **Offline** → Log + Telegram + SMS alert + all agents notified + Sentinel investigation triggered

---
