A telecommunications provider faced significant challenges managing its vast and intricate IT infrastructure, due to teams working across various time zones and a lack of comprehensive system monitoring.
The absence of proactive monitoring tools led to difficulty in tracking IT service performance metrics, such as failure rates and response times, complicating the prioritization of issues and root cause analysis.
Proactive Measures for IT Efficiency
Without the capability to identify the source of issues, it was challenging for the provider's IT team to implement preventative measures.
This lack of insight significantly impacted service performance and restricted executives from making informed business decisions. The company sought a solution to link key performance indicators (KPIs) with critical service components, enabling real-time issue tracking and investigation.
Introducing Splunk IT Service Intelligence
Splunk ITSI analyzes existing data to predict potential issues and troubleshoot existing problems
Following a successful deployment of Splunk Enterprise Security, RiverSafe was engaged again to implement Splunk's IT Service Intelligence (ITSI) tool. Utilizing machine learning, Splunk ITSI analyzes existing data to predict potential issues and troubleshoot existing problems.
The deployment aimed to enhance monitoring across the IT environment, providing the team with comprehensive observability and rapid access to service insights through refined data collection on platforms such as Kubernetes.
Rapid Implementation and Outcomes
RiverSafe rolled out Splunk ITSI in under a week, enabling the team to gain actionable insights quickly. By integrating data from existing KPIs, the operations team could identify trends and preemptively address anomalies.
A key feature was the custom glass table visualizations, which facilitated swift issue detection and resolution by providing an easy-to-read overview of web portal performance metrics like ticket status, login attempts, and API performance.
Streamlining Event Management
Splunk ITSI significantly reduced false positives in event management, lowering overall event volume by 40%
The implementation allowed centralization of events from previously separate solutions into one interface, which was critical in prioritizing and reacting swiftly to infrastructure events. Splunk ITSI significantly reduced false positives in event management, lowering overall event volume by 40%.
This enabled operators to concentrate on crucial events, resulting in an average reduction of event processing time by eight minutes, improving Service Level Agreement (SLA) performance.
Optimized Operations with Best Practices
Overall, the addition of Splunk ITSI increased operational visibility and efficiency, enabling swift identification and resolution of workflow bottlenecks.
RiverSafe also advised on best practices for utilizing Splunk ITSI, including data collection methods and alternatives to save storage space when logging.
The client, a large telecommunications provider, had teams working across multiple time zones and operated a sprawling and complex IT system. The scale and density of this system made gaining visibility into IT services extremely difficult, leaving the security team blind to what was happening with its IT environment.
With little-to-no operations monitoring tools in place to proactively monitor these systems, the team had no visibility on the availability of its IT services or KPIs such as failure rates, send request times, and response times. This lack of oversight made it difficult for the team to prioritise issues — and nearly impossible for them to find the root cause of any problem.
Proactive measures
Without the ability to investigate the source of issues, the team was unable to take proactive measures to prevent them from reoccurring, severely impacting service performance.
This was not only a problem for IT teams, but also for executives, who lacked the insight into IT business operations that would help them make decisions.
The solution
The company needed a solution that would map KPIs to critical service components, enabling the operations team to effectively drill down into issues in real time and conduct in-depth investigations to find resolutions.
RiverSafe had previously implemented Splunk Enterprise Security for the customer and given the success of the implementation and the positive client feedback on the platform, RiverSafe was again engaged to deploy Splunk’s IT Service Intelligence (ITSI) tool to help it tackle its visibility problem.
Data collection metrics
Splunk ITSI uses machine learning to analyse existing data and predict future issues. As well as forecasting potential services bottlenecks, it can also troubleshoot problems and help users resolve issues fast. Delivering comprehensive monitoring across the entire IT environment, Splunk ITSI would also give the team full observability of their IT infrastructure.
In particular, the engineering team wanted to gather metric data relating to the Kubernetes platform. RiverSafe reconfigured and implemented data collection metrics used elsewhere in the IT environment in Splunk to allow engineers to collate and access this information from different data sources in one place.
The outcome: Actionable insights in days, not weeks
In less than a week, the RiverSafe team implemented Splunk ITSI and began running monitoring services.
With data from existing KPIs already indexed by the Splunk platform, the team are now able to access service insights even faster. These KPIs allow the operations team to identify trends, detect patterns, and proactively address any anomalies that occur before issues arise.
Instant visibility with glass table visualisations
To enable rapid and proactive issue resolution, RiverSafe implemented custom glass table visualisations in Splunk ITSI. This enables the team to navigate large volumes of data and reduce the time needed to identify and resolve problems.
This simple and accessible dashboard gives the team an instant, digestible overview of its web portal performance metrics.
These KPIs included the number of open tickets and failed login attempts, memory usage, API call success rates, average response times, and overall health of container services.
Event analytics in Splunk ITSI
As a result of RiverSafe’s work, the team has been able to centralize events from all its previously siloed solutions into a single interface with Splunk ITSI.
The event analytics in Splunk ITSI help to prioritise responses and react more quickly to customers’ infrastructure events, empowering them to provide a better service. This is thanks in part to Splunk ITSI’s ability to identify and filter out false positives from the event management process. Excluding these invalid events reduced the total event volume by 40%, helping operators focus on the events that really matter.
With fewer events to process, a single interface to work from, and a streamlined event analytics framework in Splunk ITSI, operators now process events eight minutes faster on average. This boost in efficiency has led to a major improvement in the company’s SLA performance.
Best practices for using Splunk ITSI
Overall, Splunk ITSI has delivered enhanced operational visibility, meaning the team can locate bottlenecks in workflows quickly and deliver fast recovery and troubleshooting solutions.
Along the way, RiverSafe also provided best practices for using Splunk ITSI and recommended the most effective ways to collect data, including proposing an alternative metric type that would save on storage space when collecting logs.
