Over time, a Splunk environment can become unstable due to environment complexity and configuration concerns. Common issues include distributed deployments, misconfigurations, configuration file creep, partially correct data on-boarding, poor query construction, improperly sized systems, log files containing multiple formats, date/time skew, user rights, and many other factors.
Splunk environments should be reviewed on a periodic basis to ensure that data is on boarded and retained correctly, that sufficient operational and security controls are enabled, and that the Splunk Enterprise deployment is operating at peak performance. The following areas should be reviewed:
- Splunk architecture and growth strategy analysis
- Base configurations on the system
- License usage and forecast for future growth
- Performance metrics and bottlenecks in the system
- Internal messages produced by the system
- Applications deployed on the system
- Index retention policy and health
- Universal and heavy forwarder health
- Integrity of the data stored in Splunk
- Security and data governance issues within the system
To help keep your Splunk environment running smoothly, Perficient offers a streamlined and automated Splunk Health Assessment, which can be completed in two weeks.
LEVEL 1: THE CHECKUP Review diagnostic logs using the Splunk Health Assessor computer-assisted audit tool (CAAT) with a brief focus review by a Certified Splunk Architect. Level 1 reviews should be performed quarterly to ensure that the Splunk environment is ingesting and retaining data correctly and is running at its peak operational capacity.
LEVEL 2: THE EKG Performance metrics are collected from the running Splunk Enterprise environment to collect information regarding CPU, memory, disk, queuing, process, network, license usage, forwarder usage, indexer retention and bucket usage. The collected information is reviewed by the Splunk Health Assessor CAAT.
LEVEL 3: THE BIOPSY Samples of data are collected and/or reviewed to investigate an area of concern that is detected with the data stored in Splunk or with the operation of the system. The data is reviewed against the configuration and is assessed to determine the risk associated with this condition and to recommend a course of action.
LEVEL 4: EXPLORATORY SURGERY Access the running system, investigate high-risk situations, and perform corrective measures or recommend a course of action.
Splunk Health Assessor Application
The Perficient Splunk Health Assessor App is a computer-assisted audit tool (CAAT) designed to assist with the assessment of an Enterprise Splunk environment. The app’s primary objectives are to produce meaningful reports that depict the state of a Splunk Environment, tools to perform deep-dive forensic investigations, a framework that enables automated checks for known conditions, and a historical reference point to measure the rate of change within a Splunk environment.
A complete assessment can be performed using only the diagnostic dumps provided by all core Splunk servers in the environment. Plugins are available to review samples of customer data, performance metrics over time, and diagnostic information from key universal forwarders in the environment.
