Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

DaemonEye Operator Guide

This guide provides comprehensive instructions for operators managing DaemonEye in production environments. It covers day-to-day operations, troubleshooting, and advanced configuration.

Table of Contents

System Overview

Component Status

Check the overall health of your DaemonEye installation:

# Overall system health
daemoneye-cli health

# Component-specific health
daemoneye-cli health --component procmond
daemoneye-cli health --component daemoneye-agent
daemoneye-cli health --component database

Expected Output:

System Health: Healthy
├── procmond: Running (PID: 1234)
├── daemoneye-agent: Running (PID: 1235)
├── database: Connected
└── alerting: All sinks operational

Service Management

Start Services:

# Linux (systemd)
sudo systemctl start daemoneye

# macOS (launchd)
sudo launchctl load /Library/LaunchDaemons/com.daemoneye.agent.plist

# Windows (Service)
sc start "DaemonEye Agent"

Stop Services:

# Linux (systemd)
sudo systemctl stop daemoneye

# macOS (launchd)
sudo launchctl unload /Library/LaunchDaemons/com.daemoneye.agent.plist

# Windows (Service)
sc stop "DaemonEye Agent"

Restart Services:

# Linux (systemd)
sudo systemctl restart daemoneye

# macOS (launchd)
sudo launchctl unload /Library/LaunchDaemons/com.daemoneye.agent.plist
sudo launchctl load /Library/LaunchDaemons/com.daemoneye.agent.plist

# Windows (Service)
sc stop "DaemonEye Agent"
sc start "DaemonEye Agent"

Basic Operations

Querying Process Data

List Recent Processes:

# Last 10 processes
daemoneye-cli query "SELECT pid, name, executable_path, collection_time FROM processes ORDER BY collection_time DESC LIMIT 10"

# Processes by name
daemoneye-cli query "SELECT * FROM processes WHERE name = 'chrome'"

# High CPU processes
daemoneye-cli query "SELECT pid, name, cpu_usage FROM processes WHERE cpu_usage > 50.0 ORDER BY cpu_usage DESC"

Process Tree Analysis:

# Find child processes of a specific parent
daemoneye-cli query "
SELECT
    p1.pid as parent_pid,
    p1.name as parent_name,
    p2.pid as child_pid,
    p2.name as child_name
FROM processes p1
JOIN processes p2 ON p1.pid = p2.ppid
WHERE p1.name = 'systemd'
"

# Process hierarchy depth
daemoneye-cli query "
WITH RECURSIVE process_tree AS (
    SELECT pid, ppid, name, 0 as depth
    FROM processes
    WHERE ppid IS NULL
    UNION ALL
    SELECT p.pid, p.ppid, p.name, pt.depth + 1
    FROM processes p
    JOIN process_tree pt ON p.ppid = pt.pid
)
SELECT pid, name, depth FROM process_tree ORDER BY depth, pid
"

Suspicious Process Patterns:

# Processes with suspicious names
daemoneye-cli query "
SELECT pid, name, executable_path, command_line
FROM processes
WHERE name IN ('malware.exe', 'backdoor.exe', 'trojan.exe')
   OR name LIKE '%suspicious%'
   OR executable_path LIKE '%temp%'
"

# Processes with unusual parent-child relationships
daemoneye-cli query "
SELECT
    p1.pid as parent_pid,
    p1.name as parent_name,
    p2.pid as child_pid,
    p2.name as child_name
FROM processes p1
JOIN processes p2 ON p1.pid = p2.ppid
WHERE p1.name = 'explorer.exe'
  AND p2.name NOT IN ('chrome.exe', 'firefox.exe', 'notepad.exe')
"

Data Export

Export to Different Formats:

# JSON export
daemoneye-cli query "SELECT * FROM processes WHERE cpu_usage > 10.0" --format json > high_cpu_processes.json

# CSV export
daemoneye-cli query "SELECT pid, name, cpu_usage, memory_usage FROM processes" --format csv > process_metrics.csv

# Table format (default)
daemoneye-cli query "SELECT * FROM processes LIMIT 5" --format table

Export with Filters:

# Export processes from last hour
daemoneye-cli query "
SELECT * FROM processes
WHERE collection_time > (strftime('%s', 'now') - 3600) * 1000
" --format json > recent_processes.json

# Export by user
daemoneye-cli query "
SELECT * FROM processes
WHERE user_id = '1000'
" --format csv > user_processes.csv

Process Monitoring

Real-time Monitoring

Watch Process Activity:

# Monitor new processes in real-time
daemoneye-cli watch processes --filter "name LIKE '%chrome%'"

# Monitor high CPU processes
daemoneye-cli watch processes --filter "cpu_usage > 50.0"

# Monitor specific user processes
daemoneye-cli watch processes --filter "user_id = '1000'"

Process Statistics:

# Process count by name
daemoneye-cli query "
SELECT name, COUNT(*) as count
FROM processes
GROUP BY name
ORDER BY count DESC
LIMIT 10
"

# CPU usage distribution
daemoneye-cli query "
SELECT
    CASE
        WHEN cpu_usage IS NULL THEN 'Unknown'
        WHEN cpu_usage = 0 THEN '0%'
        WHEN cpu_usage < 10 THEN '1-9%'
        WHEN cpu_usage < 50 THEN '10-49%'
        WHEN cpu_usage < 100 THEN '50-99%'
        ELSE '100%+'
    END as cpu_range,
    COUNT(*) as process_count
FROM processes
GROUP BY cpu_range
ORDER BY process_count DESC
"

# Memory usage statistics
daemoneye-cli query "
SELECT
    AVG(memory_usage) as avg_memory,
    MIN(memory_usage) as min_memory,
    MAX(memory_usage) as max_memory,
    COUNT(*) as process_count
FROM processes
WHERE memory_usage IS NOT NULL
"

Process Investigation

Deep Process Analysis:

# Get detailed information about a specific process
daemoneye-cli query "
SELECT
    pid,
    name,
    executable_path,
    command_line,
    start_time,
    cpu_usage,
    memory_usage,
    executable_hash,
    user_id,
    collection_time
FROM processes
WHERE pid = 1234
"

# Find processes with the same executable
daemoneye-cli query "
SELECT
    executable_path,
    COUNT(*) as instance_count,
    GROUP_CONCAT(pid) as pids
FROM processes
WHERE executable_path IS NOT NULL
GROUP BY executable_path
HAVING COUNT(*) > 1
ORDER BY instance_count DESC
"

# Process execution timeline
daemoneye-cli query "
SELECT
    pid,
    name,
    collection_time,
    cpu_usage,
    memory_usage
FROM processes
WHERE name = 'chrome'
ORDER BY collection_time DESC
LIMIT 20
"

Alert Management

Viewing Alerts

List Recent Alerts:

# Last 10 alerts
daemoneye-cli alerts list --limit 10

# Alerts by severity
daemoneye-cli alerts list --severity high,critical

# Alerts by rule
daemoneye-cli alerts list --rule "suspicious-processes"

# Alerts from specific time range
daemoneye-cli alerts list --since "2024-01-15 10:00:00" --until "2024-01-15 18:00:00"

Alert Details:

# Get detailed information about a specific alert
daemoneye-cli alerts show <alert-id>

# Export alerts to file
daemoneye-cli alerts export --format json --output alerts.json

# Export alerts with filters
daemoneye-cli alerts export --severity high,critical --format csv --output critical_alerts.csv

Advanced Alert Queries:

# Alerts affecting specific processes
daemoneye-cli query "
SELECT
    a.id,
    a.title,
    a.severity,
    a.alert_time,
    a.affected_processes
FROM alerts a
WHERE JSON_EXTRACT(a.alert_data, '$.pid') = 1234
ORDER BY a.alert_time DESC
"

# Alerts by hostname
daemoneye-cli query "
SELECT
    a.id,
    a.title,
    a.severity,
    a.alert_time,
    JSON_EXTRACT(a.alert_data, '$.hostname') as hostname
FROM alerts a
WHERE JSON_EXTRACT(a.alert_data, '$.hostname') = 'server-01'
ORDER BY a.alert_time DESC
"

# Alert frequency by rule
daemoneye-cli query "
SELECT
    rule_id,
    COUNT(*) as alert_count,
    MAX(alert_time) as last_alert
FROM alerts
GROUP BY rule_id
ORDER BY alert_count DESC
"

Alert Response

Acknowledge Alerts:

# Acknowledge a specific alert
daemoneye-cli alerts acknowledge <alert-id> --comment "Investigating"

# Acknowledge multiple alerts
daemoneye-cli alerts acknowledge --rule "suspicious-processes" --comment "False positive"

# List acknowledged alerts
daemoneye-cli alerts list --status acknowledged

Alert Suppression:

# Suppress alerts for a specific rule
daemoneye-cli alerts suppress --rule "suspicious-processes" --duration "1h" --reason "Maintenance"

# Suppress alerts for specific processes
daemoneye-cli alerts suppress --process 1234 --duration "30m" --reason "Known good process"

# List active suppressions
daemoneye-cli alerts suppressions list

Rule Management

Rule Operations

List Rules:

# List all rules
daemoneye-cli rules list

# List enabled rules only
daemoneye-cli rules list --enabled

# List rules by category
daemoneye-cli rules list --category "malware"

# List rules by severity
daemoneye-cli rules list --severity high,critical

Rule Validation:

# Validate a rule file
daemoneye-cli rules validate /path/to/rule.sql

# Validate all rules
daemoneye-cli rules validate --all

# Test a rule with sample data
daemoneye-cli rules test /path/to/rule.sql --sample-data

Rule Management:

# Enable a rule
daemoneye-cli rules enable suspicious-processes

# Disable a rule
daemoneye-cli rules disable suspicious-processes

# Update a rule
daemoneye-cli rules update suspicious-processes --file /path/to/new-rule.sql

# Delete a rule
daemoneye-cli rules delete suspicious-processes

Rule Development

Create a New Rule:

# Create a new rule file
cat > /etc/daemoneye/rules/custom-rule.sql << 'EOF'
-- Detect processes with suspicious names
SELECT
    pid,
    name,
    executable_path,
    command_line,
    collection_time
FROM processes
WHERE
    name IN ('malware.exe', 'backdoor.exe', 'trojan.exe')
    OR name LIKE '%suspicious%'
    OR executable_path LIKE '%temp%'
ORDER BY collection_time DESC;
EOF

# Validate the rule
daemoneye-cli rules validate /etc/daemoneye/rules/custom-rule.sql

# Enable the rule
daemoneye-cli rules enable custom-rule

Rule Testing:

# Test rule against current data
daemoneye-cli rules test custom-rule --live

# Test rule with specific time range
daemoneye-cli rules test custom-rule --since "2024-01-15 00:00:00" --until "2024-01-15 23:59:59"

# Test rule performance
daemoneye-cli rules test custom-rule --benchmark

Rule Import/Export

Export Rules:

# Export all rules
daemoneye-cli rules export --output rules-backup.tar.gz

# Export specific rules
daemoneye-cli rules export --rules "suspicious-processes,high-cpu" --output selected-rules.tar.gz

# Export rules by category
daemoneye-cli rules export --category "malware" --output malware-rules.tar.gz

Import Rules:

# Import rules from file
daemoneye-cli rules import rules-backup.tar.gz

# Import rules with validation
daemoneye-cli rules import rules-backup.tar.gz --validate

# Import rules with conflict resolution
daemoneye-cli rules import rules-backup.tar.gz --resolve-conflicts

System Health Monitoring

Performance Metrics

System Performance:

# View system metrics
daemoneye-cli metrics

# CPU usage over time
daemoneye-cli metrics --metric cpu_usage --duration 1h

# Memory usage over time
daemoneye-cli metrics --metric memory_usage --duration 1h

# Process collection rate
daemoneye-cli metrics --metric collection_rate --duration 1h

Database Performance:

# Database status
daemoneye-cli database status

# Database size
daemoneye-cli database size

# Database performance metrics
daemoneye-cli database metrics

# Database maintenance
daemoneye-cli database maintenance --vacuum

Log Analysis

View Logs:

# Recent logs
daemoneye-cli logs --tail 50

# Logs by level
daemoneye-cli logs --level error

# Logs by component
daemoneye-cli logs --component procmond

# Logs with filters
daemoneye-cli logs --filter "error" --tail 100

Log Analysis:

# Error frequency
daemoneye-cli logs --analyze --level error

# Performance issues
daemoneye-cli logs --analyze --filter "slow"

# Security events
daemoneye-cli logs --analyze --filter "security"

Configuration Management

Configuration Files

View Configuration:

# Show current configuration
daemoneye-cli config show

# Show specific configuration section
daemoneye-cli config show alerting

# Show configuration with defaults
daemoneye-cli config show --include-defaults

Update Configuration:

# Update configuration value
daemoneye-cli config set app.scan_interval_ms 60000

# Update multiple values
daemoneye-cli config set alerting.sinks[0].enabled true

# Reload configuration
daemoneye-cli config reload

Configuration Validation:

# Validate configuration file
daemoneye-cli config validate /etc/daemoneye/config.yaml

# Validate current configuration
daemoneye-cli config validate

# Check configuration for issues
daemoneye-cli config check

Environment Management

Environment Variables:

# Set environment variables
export DAEMONEYE_LOG_LEVEL=debug
export DAEMONEYE_DATABASE_PATH=/var/lib/daemoneye/events.redb

# View environment configuration
daemoneye-cli config show --environment

Service Configuration:

# Update service configuration
sudo systemctl edit daemoneye

# Reload service configuration
sudo systemctl daemon-reload
sudo systemctl restart daemoneye

Troubleshooting

Common Issues

Service Won't Start:

# Check service status
sudo systemctl status daemoneye

# Check logs for errors
sudo journalctl -u daemoneye -f

# Check configuration
daemoneye-cli config validate

# Check permissions
ls -la /var/lib/daemoneye/

Database Issues:

# Check database status
daemoneye-cli database status

# Check database integrity
daemoneye-cli database integrity-check

# Repair database
daemoneye-cli database repair

# Rebuild database
daemoneye-cli database rebuild

Alert Delivery Issues:

# Check alert sink status
daemoneye-cli alerts sinks status

# Test alert delivery
daemoneye-cli alerts test-delivery

# Check network connectivity
daemoneye-cli network test

# View delivery logs
daemoneye-cli logs --filter "delivery"

Debug Mode

Enable Debug Logging:

# Set debug log level
daemoneye-cli config set app.log_level debug

# Restart service
sudo systemctl restart daemoneye

# Monitor debug logs
daemoneye-cli logs --level debug --tail 100

Component Debugging:

# Debug procmond
sudo daemoneye-cli debug procmond --verbose

# Debug daemoneye-agent
daemoneye-cli debug daemoneye-agent --verbose

# Debug database
daemoneye-cli debug database --verbose

Performance Issues

High CPU Usage:

# Check process collection rate
daemoneye-cli metrics --metric collection_rate

# Reduce scan interval
daemoneye-cli config set app.scan_interval_ms 60000

# Check for problematic rules
daemoneye-cli rules list --performance

High Memory Usage:

# Check memory usage
daemoneye-cli metrics --metric memory_usage

# Reduce batch size
daemoneye-cli config set app.batch_size 500

# Check database size
daemoneye-cli database size

Slow Queries:

# Check query performance
daemoneye-cli database query-stats

# Optimize database
daemoneye-cli database optimize

# Check for slow rules
daemoneye-cli rules list --slow

Best Practices

Security

  1. Regular Updates: Keep DaemonEye updated to the latest version
  2. Access Control: Limit access to DaemonEye configuration and data
  3. Audit Logging: Enable comprehensive audit logging
  4. Network Security: Use secure connections for remote management
  5. Backup: Regularly backup configuration and database

Performance

  1. Resource Monitoring: Monitor CPU, memory, and disk usage
  2. Rule Optimization: Optimize detection rules for performance
  3. Database Maintenance: Regular database maintenance and cleanup
  4. Alert Tuning: Tune alert thresholds to reduce noise
  5. Capacity Planning: Plan for growth in process count and data volume

Operations

  1. Documentation: Document custom rules and configurations
  2. Testing: Test rules and configurations in non-production environments
  3. Monitoring: Set up comprehensive monitoring and alerting
  4. Incident Response: Develop procedures for security incidents
  5. Training: Train operators on DaemonEye features and best practices

Maintenance

  1. Regular Backups: Backup configuration and database regularly
  2. Log Rotation: Implement log rotation to prevent disk space issues
  3. Database Cleanup: Regular cleanup of old data
  4. Rule Review: Regular review and update of detection rules
  5. Performance Tuning: Regular performance analysis and tuning

This operator guide provides comprehensive instructions for managing DaemonEye in production environments. For additional help, consult the troubleshooting section or contact support.