Splunk Enterprise SIEM Deployment: Complete Production Guide with Universal Forwarders

Introduction

Splunk Enterprise is one of the most powerful SIEM platforms available. When properly architected, it provides unparalleled visibility across your entire IT environment. This guide documents the complete deployment executed at PostEx, covering architecture decisions through production alerting and dashboards.

At PostEx, we deployed Splunk Enterprise to consolidate log collection from four primary sources: the local network, the global WAN network, core routers (MikroTik and Cisco), and all infrastructure assets including Windows endpoints, Linux servers, VMware hosts, and Proxmox hypervisors.

Splunk's true power is correlating logs across sources, time, and context to surface threats that individual log sources would never reveal alone.

Deployment Architecture

• Indexer Cluster - 3 indexers for redundancy and search performance, replication factor 2
• Search Head - Dedicated search head with Splunk Enterprise Security app
• Heavy Forwarder - For parsing, filtering, routing of high-volume sources (firewall, NetFlow)
• Universal Forwarders - Deployed on every Windows and Linux endpoint
• Deployment Server - Centralized configuration management for all forwarders

Universal Forwarder Deployment at Scale

Windows - PowerShell Silent Install

# Silent install via PowerShell
$WAZUH_MANAGER = "splunk-indexer01.internal"
msiexec.exe /i SplunkUniversalForwarder.msi AGREETOLICENSE=Yes SPLUNKUSERNAME=splunkfwd SPLUNKPASSWORD=SecurePass! RECEIVING_INDEXER="splunk-indexer01.internal:9997" /quiet

# inputs.conf for Windows Event Logs:
[WinEventLog://Security]
index = windows_security
disabled = false
evt_resolve_ad_obj = 1

[WinEventLog://System]
index = windows_system
disabled = false

Linux - Ansible Playbook

# ansible playbook: deploy_splunk_uf.yml
---
- name: Deploy Splunk Universal Forwarder
  hosts: linux_servers
  become: yes
  tasks:
    - name: Download UF installer
      get_url:
        url: https://download.splunk.com/products/universalforwarder/releases/9.2.0/linux/splunkforwarder-9.2.0-linux-2.6-amd64.deb
        dest: /tmp/splunkforwarder.deb
    - name: Install UF
      apt:
        deb: /tmp/splunkforwarder.deb
    - name: Deploy inputs.conf
      template:
        src: templates/inputs.conf.j2
        dest: /opt/splunkforwarder/etc/system/local/inputs.conf
    - name: Start and enable
      service:
        name: SplunkForwarder
        enabled: yes
        state: started

Network Device Monitoring

All MikroTik routers and Cisco switches send syslog to the Splunk Heavy Forwarder for centralized network visibility.

# MikroTik RouterOS syslog config
/system logging action
add name=splunk remote=192.168.1.50 remote-port=514 target=remote
/system logging
add action=splunk topics=critical,warning,firewall

# Cisco IOS syslog config
logging on
logging trap informational
logging host 192.168.1.50

Index Design Strategy

# indexes.conf design
[windows_security]    # 365-day retention - compliance
[windows_system]      # 90-day retention
[network_syslog]      # 180-day retention - router/switch logs
[network_firewall]    # 180-day retention
[linux_syslog]        # 90-day retention
[infrastructure]      # 365-day retention
[splunk_summary]      # Summary indexes for dashboards

Detection Use Cases in SPL

Brute Force Followed by Success

index=windows_security EventCode=4625
| bucket _time span=10m
| stats count AS failures by _time, TargetUserName, IpAddress
| where failures >= 5
| join TargetUserName, IpAddress [
    search index=windows_security EventCode=4624
    | stats count AS successes by TargetUserName, IpAddress]
| where successes >= 1
| eval alert="Brute force then success", severity="CRITICAL"

DNS Exfiltration - Long Subdomain

index=network_dns
| eval query_len=len(query)
| where query_len > 50
| stats count, dc(query) AS unique_q by src_ip
| where unique_q > 100
| eval severity="HIGH", technique="T1048.003"

Dashboard Design

We built three primary dashboards: Security Overview (real-time event volume, alert distribution, authentication failures), Infrastructure Health (router CPU/memory, interface status, bandwidth), and Threat Hunting (process timeline, network connections heatmap, user behavior anomalies).

Key Lessons from Production

• Profile data volumes before deployment - daily GB drives licensing cost
• Use summary indexes for dashboard searches to reduce search load
• Deploy Deployment Server from day one for forwarder config management at scale
• Use Splunk CIM for normalization across data sources
• Always test alert rules in audit mode for 2 weeks before enabling notifications