Splunk Enterprise Deployment — PostEx (Production)

Problem

PostEx operated with fragmented security visibility. Local network, global WAN connections, core routers, and 500+ infrastructure assets each generated logs in isolation with no centralized collection, correlation, or alerting. Security events were missed because there was no single platform to aggregate and analyze data across the entire infrastructure stack.

Environment

• 500+ endpoints: Windows workstations, Linux servers, VMware ESXi, Proxmox VE hypervisors
• Network infrastructure: MikroTik routers, Cisco switches, Juniper firewalls
• Multi-site architecture: Primary data center (Lahore) + remote site connections via WAN/VPN
• Fintech application stack: Payment processing APIs, customer portal, internal ERP systems

Architecture

Data Sources
  Windows Endpoints -> Universal Forwarder -> Splunk Indexer Cluster
  Linux Servers     -> Universal Forwarder -> Splunk Indexer Cluster
  MikroTik Routers  -> Syslog UDP 514 -> Heavy Forwarder -> Indexer
  Cisco Switches    -> Syslog UDP 514 -> Heavy Forwarder -> Indexer
  VMware/Proxmox    -> Universal Forwarder -> Indexer
  Web App Logs      -> Filebeat -> Heavy Forwarder -> Indexer

Splunk Components:
  Indexer Cluster (3 nodes, RF=2)
  Search Head (Splunk ES app)
  Heavy Forwarder (network + high-volume sources)
  Deployment Server (forwarder config management)
  License Manager

Tools & Technologies

Methodology

Phase 1: Architecture & Planning

Profiled log volumes from all sources over 2 weeks to size the indexer cluster and estimate daily GB licensing requirements. Designed index schema by data type and retention policy to optimize storage costs and search performance.

Phase 2: Core Infrastructure

Deployed 3-node indexer cluster with replication factor 2 for redundancy. Installed Splunk Enterprise Security app on the dedicated search head. Configured Deployment Server for centralized forwarder management at scale.

Phase 3: Endpoint Coverage

Deployed Universal Forwarders to 500+ Windows endpoints via Group Policy software installation and to Linux servers via Ansible playbooks. Configured inputs.conf to collect Windows Security Event Logs (critical Event IDs: 4624, 4625, 4672, 4688, 4698, 4702) and Linux auditd/syslog.

Phase 4: Network Device Integration

Configured syslog forwarding on all MikroTik routers and Cisco switches to the Heavy Forwarder. Implemented netflow collection for bandwidth monitoring and lateral movement detection. Added VMware ESXi and Proxmox VE log collection.

Phase 5: Detection Rules & Dashboards

Engineered 40+ MITRE ATT&CK-mapped detection rules in Splunk SPL. Built 3 primary dashboards: Security Overview, Infrastructure Health, and Threat Hunting. Configured email and automated alerting for critical detections.

Detection Logic

Brute Force Detection (T1110.001)

index=windows_security EventCode=4625
| bucket _time span=5m
| stats count by _time, TargetUserName, IpAddress
| where count >= 10
| eval severity="HIGH", technique="T1110.001"

Router Anomaly Detection

index=network_syslog sourcetype=mikrotik
| rex field=message "src-address=(?P<src_ip>[\d.]+)"
| stats count by src_ip, _time
| eventstats avg(count) AS avg_cnt, stdev(count) AS std_cnt by src_ip
| where count > avg_cnt + (3 * std_cnt)
| eval alert="Anomalous router traffic from " . src_ip

MITRE ATT&CK Mapping

Challenges & Solutions

• Challenge: High false positive rate from initial detection rules. Solution: 2-week audit mode baseline per rule before enabling alerts. Reduced FP by 40%.
• Challenge: MikroTik syslog format parsing. Solution: Custom Logstash filter and Splunk props/transforms for RouterOS message normalization.
• Challenge: License sizing under budget. Solution: Implemented summary indexing and data acceleration to minimize raw data ingestion for high-volume low-value sources.

Results & Metrics

• Unified visibility across 500+ endpoints, 4 network segments, and all core infrastructure
• Mean time to notify reduced to under 2 minutes for critical alerts
• False positive rate reduced by 40% through systematic rule tuning
• First ransomware precursor activity detected and blocked within 3 weeks of deployment
• 100% compliance with log retention requirements for financial regulatory audit

Lessons Learned

• Profile data volumes before deployment — accurate sizing prevents expensive mid-project license upgrades
• Start with detection coverage for your most critical assets, not all assets simultaneously
• Build dashboards that answer business questions ("Are we under attack?") not just technical questions ("How many events?")
• Invest in Deployment Server from day one — forwarder management without it does not scale