The Department of Work and Pensions: Proactive Monitoring with the Elastic Stack

The Department for Work and Pensions (DWP) in the UK, responsible for welfare, pensions, and child maintenance policy, implemented a Digital Experience Monitoring (DXM) project to enhance the performance monitoring of its Siebel system. The initiative aimed to provide the Live Service Support team with more proactive capabilities in ensuring system performance by utilizing the Elastic Stack for log parsing, data visualization, and real-time analysis. An internal squad was formed in 2017, working in Agile sprints to develop the architecture through discovery, alpha, beta, and live phases. The Elastic Stack, comprising Logstash, Elasticsearch, and Kibana, was chosen for its scalability, cost-effectiveness, and ability to deliver insights into system performance and issues. The DXM system processes data from various sources, enabling the DWP to identify and solve potential performance concerns before they affect users. The success of this project, which includes creating dashboards for performance visualization and automation, has been acknowledged by senior leaders and shared across the DWP to support similar monitoring solutions. Upgrades to the Elastic Stack are underway to incorporate machine learning for anomaly detection.