IDC

Download White Paper
Region Focus: Worldwide

An Executive Blueprint for an Observability Platform: Driving Operational Excellence and Business Outcomes through Analytics and Automation

May 2023 | us50716823
Stephen Elliot

Stephen Elliot

Group Vice President, I&O, Cloud Operations, and DevOps

Mark Leary

Mark Leary

Research Director, Network Analytics & Automation

Product Type:
IDC: White Paper
Sponsored by: Cisco

IDC Opinion

The digital business model has digitalized processes, services, and products; significantly transforming customer engagement models. Observability continues to garner significant attention from business and IT executives as detailed visibility, precise control, and automated actions become core operating tenets within the digital infrastructure. Business executives now demand critical customer experience knowledge to create and successfully execute growth strategies.  Executives are adopting observability platforms to deliver high performing digital services that drive great customer experiences and revenue growth.  

This wide use and deep impact are reflected in IDC’s view of a comprehensive Observability Platform as a:

“Platform that collects (in real-time) high volume and high cardinality data at scale, and utilizes advanced analytic models across network, application, cybersecurity, cloud and digital experience domains with extensibility that enables packaged solutions for technology, business, and the partner ecosystem.  In addition, the platform is extensible to take advantage of the data and analytics allowing for custom solutions to be developed for additional use cases.”  

An observability platform is an evolution of monitoring capabilities that empowers technology teams to move from being reactive to proactive by unifying, viewing, analyzing, and automating the entire end-to-end top-to-bottom digital infrastructure, encompassing core computing and cloud infrastructure, network, applications, security, and digital experience data to prevent downtime and negative impacts to customers.  It enables technology teams (application, security, networking, platform engineering, and Site Reliability Engineering (SREs) to collaborate with business executives on understanding the customer journey and measuring and managing the moments that matter for revenue growth and customer experiences. It offers a consolidated path across technology, teams, and processes to deliver lower costs, automated actions, and team and process efficiencies that directly corelate to revenue generating services that support customers and business growth.    

75% of the respondents agree that their CEO and business leadership understand that complete visibility and control over infrastructure, network, applications, security, and digital experience, is critical to digital business success. And with so much riding on the digital infrastructure, 47% of the respondents cite their average cost of an hour of downtime at $250,000 or more.  Executives realize that employees, partners, and customers demand high performing services. Business architectures are now technology architectures, and delivering great experiences are a major factor in generating sustainable revenue growth, customers adoption and innovation.  IT organizations are recognizing that the digital infrastructure must be viewed as more of a concert than a collection of instruments. As such, comprehensive observability is fast developing into a core tenet when building out a resilient and responsive digital infrastructure.

The research further suggests (and often quantifies) critical themes that IT and business leadership teams should consider as drivers for Observability platform investments.  These are: 

  • Collaboration and improved proactive control points across security, IT operations, and SRE teams offers a pathway to improving and accelerating an organization’s security posture and service performance at the same time. 
  • Observability platform adoption offers technology capabilities that unifies traditionally fragmented data onto a single platform, and applies analytic models to improve team and process efficiencies that reduce business and customer risks.  
  • The importance of solution extensibility that empowers customers with the ability to ask and solve questions from a unified observability data model, unlocking data value and new use cases that are unique to every organization. 
  • The rising influence and collaboration of Security Operations teams with IT Operations and Site Reliability Engineering teams when using Observability platforms and improving an organization security posture.
  • Observability provides IT leadership teams with a direct path for business leadership collaboration on customer journey and experience conversations and value streams.  
  • Service Partners have opportunities with Observability platforms through new use case development, and high interest in SaaS-delivered services.  

With most existing IT management tools, staff, and practices focused on specific siloed technologies, moving toward a more concerted monitoring and management effort presents many challenges. Successfully applying comprehensive observability practices and capabilities involves people and organizational constructs, processes, and technology. Through unification, a comprehensive observability platform offers IT organizations an opportunity to satisfy complex management requirements. Meeting these management requirements drives operational excellence, empowers business and technology teams to work faster and smarter, and deliver rich, reliable digital experiences to workers, partners, and customers. 

An overall analysis of IDC worldwide survey results focused on the use, challenges, and returns relating to observability yields detailed views into current state, solution expectations, and future strategies (see Figure 1). 

Figure 1: Full Stack Observability Survey: Summary Focal Points – State, Solutions, Strategies

StateSolutionsStrategies
Observability well established as key tactical and strategic function with vital benefits, exec support, rising budgets.Data. Data. Data. Comprehensive sourcing, Complex correlation. Cross-org sharing. A focus on a single source of truth!High value assigned outcomes (vs tech capabilities) – Digital innovation, IT productivity, operational efficiency, Automation, Reduced business risk.
Observability is a primary tool to boost IT teamwork — esp. with SecOps.Intelligent analytics linked to precise automated actions. Building realization that Al/ML-driven analysis can drive big gains in service quality and staff productivity.Supplier role intensifying – Customization, Customer success, Technology innovation, Cloud/Multicloud management, Ecosystem.
IT uses an abundance of observability tools… and management gaps persist.
Ready integration and extensibility viewed as valuable across IT AND the business.

Observability is a hierarchical Top-Down capability within IT. ITOps drives use and unification. Heightening role for Security.

Significant problems: High TCO, Data access/use, IT productivity, Delayed resolution/mitigation efforts, limited flexibility, narrow use cases…

Consolidation, convergence, and unification leads towards platforms vs best-of-breed solutions.

Operational responsibilities for observability spread rather equally across customer, MSP, and mix of both. For staff, automation, security, and data skills grow.

Positive perceptions in data collection/correlation, digital experience mgmt, and observability use can be a mismatch to challenges, limitations, and expected returns.

Equal emphasis on proactive and reactive management capabilities. Match to industry movement to predict, prevent, prescribe, and protect.

Observability to increase influence over decision-making in IT and lines of business.

Methodology

This white paper is based on a global IDC web-based survey of organizations (see Appendix). It included 2,062 respondents represented by titles that are IT operations (44%), network operations (17%), IT executives (13%), observability engineers (10%), DevOps (10%), system manager (9%), Cloud operations (7%), AppOps (7%), SecOps (5%), Platform Engineering (5%), and Site Reliability Engineering (5%). Organizations were categorized as small (500–999 employees, 20%), medium sized (1,000–1,999 employees, 30%), large-sized (2000-4,999 employees, 30%) and very-large (5,000 or more employees, 19%). Respondents were asked about the role and value of observability in their organization; how it impacts security, operations, development, and SRE teams (among others) and processes; and business executive involvement and value. The survey spanned core themes across people, process, and technology impact, and the role that observability can play in the future of business outcome realization.

Respondents represented 12 countries: Australia, Brazil, Canada, Germany, France, India, Japan, Mexico, Netherlands, Singapore, the United Kingdom, and the United States. Organizations represented a range of industries spread across financial services, construction, professional services, telecommunications, manufacturing, healthcare, retail, transportation, and others.

This survey was conducted in March 2023 by IDC and was commissioned by Cisco.

In this White Paper

This IDC white paper uses a global IT survey to confirm trends and provide unique insights for organizations responding to critical resiliency and responsiveness challenges to achieve (or even exceed) desired business outcomes in today’s competitive, customer-focused, and complex environment. This white paper covers observability perspectives from the lens of core buyers and practitioners, to understand the transformational opportunities a comprehensive observability platform can provide. It provides a unique and quantified in-depth perspective into the critical areas that are driving business outcomes and competitive advantages for organizations around the world.

These critical areas are:

  • Collaboration between security, IT operations, SRE, and other teams
  • Observability adoption and maturity and related challenges
  • The importance and value of high cardinality data, advanced analytics, data/tool integration, and solution extensibility
  • The impact on team collaboration and new models for working to boost operational excellence and IT service delivery
  • Data driven results and use cases that bolster observability tactics, IT strategies, and business outcomes

This white paper provides quantification of expected value and benefits associated with investment in comprehensive observability platforms to support the current challenges and future opportunities from security, operations, SRE, and related teams as they embrace evolving observability technologies and practices. This research probes deeply into the business results executives are obtaining from their cross-team investments. Most importantly, the white paper provides executives with a road map of activities and supporting data that they can use to optimize business returns from their comprehensive observability investments.

Situation Overview

Observability as a practice and set of capabilities requires a platform foundation based on data collection, intelligent analysis, and actionable insights that can be applied and shared in-context across multiple IT and business domains. But how do executives perceive (or define) observability today? The research indicates that 40% of the respondents aligned their understanding of observability as applied to monitoring as “tools and practices that enable the sharing of intelligence and insights across multiple IT teams.” And current and future spending in the segment is strong. 75% said they spend $1 million or more, and of that group, 21% spend $5 million or more. Reflecting the current and potential impact observability is expected to have on both IT and business initiatives, 46% or respondents are planning to increase their related spending over the next two years. 42% plan to stay the same.

Critical Needs: Consolidation, Collaboration, and Proactive Control


What is the reality of observability within the IT organization today? IT organizations are increasingly experiencing tool chain fatigue, resulting in high costs, unneeded complexity, delayed responses, security vulnerabilities, and integration barriers (see Figure 2). 40% of the survey respondents use 2- 10 monitoring/observability tools, 28% 11-20 tools, and 18.5% use 21-40 tools. Incredibly, another 10% indicate use of 41-100 tools! With so many tools in use, it should come as no surprise that 74% of respondents said data collection and correlation is difficult. The many respondent citations relating to needed improvement in staff productivity and team collaboration are certainly driven by the over- abundance of tools gathering data, issuing alarms, providing conflicting insights, and misdirecting staff actions.

Figure 2:
Using Multiple Tools Creates Many Challenges, Leading with High TCO

Q. What are the biggest challenges of utilizing multiple monitoring/observability tools for managing infrastructure, network, applications, cybersecurity, and digital experience?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

However, despite the plethora of tools deployed, critical measurements and coverage gaps persist. For example, with multiple tools to apply and a wide mix of IT domains (e.g., operations, engineering, security, networking, etc.) and lines of business involved, 82% of the survey respondents said it’s difficult to deliver a flawless end-to-end digital experience to internal employees and external business partners; 77% said the same for end customers. If the digital infrastructure must be presented as a concert, the audience experience goes unmeasured and, certainly, undermanaged.

Only 17% of organizations stated they are using an observability solution that meets all their needs, and 25% of organizations believe they have optimized their observability journey. The research indicates the maturation of observability solutions and practices and the movement from reactive monitoring to proactive observability can further align people, process, technology, and automation in unique ways. Figure 3 indicates how IT, and business teams are considering the use of observability to further unify data with collaboration and automation for a positive business impact. IT organizations are certainly looking to achieve greater results with a comprehensive observability approach and platform.

Figure 3:
IT Unification, Optimization, Automation, and Protection Core to Observability

Q. Which of the following best aligns with your understanding of observability applied to monitoring solutions?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

Data unification across critical IT management domains is another pressing requirement. In fact, 53% of organizations stated they need to unify observability across critical IT management domains such as infrastructure, network, applications, cybersecurity, and digital experience. The top three leading drivers of unifying observability data are:

  • Improving business and technology use case capabilities using observability data and analytics
  • Using observability data to improve teamwork across various IT domains
  • Increasing cybersecurity and application security threats.

37% stated data collection and correlation across critical IT management domains such as infrastructure, network, applications, cybersecurity, and digital experience as their #1 most important observability solution capability.

The Rising Influence of SecOps on Observability

The involvement and rising collaboration from Security Operations (SecOps) stakeholders with observability practices cannot be denied. In fact, increasing collaboration with SecOps teams and adjacent IT teams is a key value proposition for observability (see Figure 4).

Figure 4:
Increasing Collaboration with SecOps is a Key Value Proposition

Q. Which combination of teams do you expect to improve their collaboration by using observability capabilities?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

As a growing influencer in observability decision making, SecOps teams are demanding greater visibility and control; underscoring the growing complexity of securing application architectures and the infrastructure platforms that support modern, containerized applications. Because of the increased influence held by both developer and security teams today, operations teams must increasingly view their counterparts’ teams as internal “customers” of their services and platforms — building solutions that meet the fast-moving needs called for by oncoming DevSecOps practices.

IT organizations are reacting positively to heightened demand for SecOps involvement in observability. 61% of the respondents agree that the security team will play a larger role in the observability solution selection, use, and collaboration. In fact, SecOps teams ranked second (only ITOps was higher) for who should be responsible for establishing, controlling, and advancing observability efforts within the organization (see Figure 5).

Figure 5:
ITOps and SecOps Lead the Way for Observability with IT and Business Exec Support

Q. Who should be responsible for establishing, controlling, and advancing observability efforts within your organization?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

Over the next two years, the survey data also indicates that security influence is expected to grow with observability decisions. 99% of the respondents stated that observability will be more influential to their security decision making when compared to today.

Observability Benefits

Organizations tend to have broad expectations for returns from observability investments. Figure 6 shows that leading benefits are varied, spanning technology, people, and process. The top-ranked benefits focus on productivity, innovation, digital experience, cybersecurity, and extensibility across business lines.

Figure 6:
Leading Benefits: Improved Productivity, Responsiveness, Experiences, and Security

Q. What are the biggest benefits when applying comprehensive observability capabilities to the management of infrastructure, network, applications, cybersecurity, and digital experience?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

IT organizations want several additional value streams from observability solutions, including the ability to have both broad and deep data collection (including standardized Open Telemetry data), effective intelligent analytics, and the ability to use a single source of truth across all IT domains (see Figure 7). It is interesting to note that respondents indicated in both Figure 7 and in response to other survey questions preferences for observability solutions that promote customization and extension of observability functions across IT and business use cases. There is strong recognition that solution flexibility, data sharing, and tool integration deliver real value when tailoring observability solutions to specific organizational demands and environments.

Figure 7:
Intelligence and Insights Combine to Deliver a Vital Single Source of Truth

Q. In examining observability solutions adopted or to be adopted within your organization, to what

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

Observability Drives Business and Line of Business Involvement

Observability is increasingly being valued as a business resource. When asked to cite the driving forces behind comprehensive observability and a more unified approach to digital infrastructure management, the #1 response highlighted improved business AND technology use cases for observability data and analytics. The intelligence and insights provided by observability solutions are cited across the survey results: (1) drive outcomes (2) reduce risk (3) accelerate innovation (4) deliver positive digital experiences (5) provide proactive management in support of business results. These and more are cited as key capabilities and benefits of comprehensive observability.

This realization of the observability benefits to business is tightening the bonds between IT and business teams. When specifying requirements, defining outcomes, and evaluating solutions, 97% of IT respondents involve their line of business partners in observability initiatives. And for many organizations, this involvement is significant. 25% of organizations have equal involvement and influence of both IT and business teams. For 16% of organizations, the lines of business are leading their observability thrusts. It’s probably no coincidence that when asked who in their organization should be responsible for observability, over 14% of respondents (remember all respondents come from IT roles) indicated that business leadership or lines of business should be in charge.

And this linkage between observability and the business is not limited to only lines of business. Owing to its potential broad-based business impact, it extends all the way up through the business executive chain of command. 75% of survey respondents agree that their CEO and business leadership understand that detailed visibility and complete control over infrastructure, network, applications, security, and digital experience is critical to digital business success. That C-suite understanding lends significant support for the increased observability investments seen into the future.

Observability Provides Ample Service Partner Opportunities

For the service partner ecosystem (e.g., managed service providers, outsourcing vendors, and systems integrators), observability provides significant opportunities to deliver value added management services and support. 67% of respondents agree that service partners are an acceptable or even preferred alternative for their observability initiatives (see Figure 8). Often daunting observability challenges and heightening requirements are driving many IT organizations to demand service-based deployment (and payment) options and a platform perspective for observability. In addition, these service partners provide much-needed observability expertise, integration capabilities, and best practices to their service clients.

Figure 8:
IT Organizations Seek Deployment Options — and Outside Help — with Observability

Q. How does your organization prefer to utilize a monitoring/observability solution?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

In addition, Figure 9 indicates the breath of opportunities for service partners to augment specific skillsets that IT organizations prioritize when developing and hiring staff with observability responsibilities. Indicated staff shortages, skill gaps, and constrained teamwork are all pointers to service partner opportunities. Notably, significant areas that service partners can add value for IT organizations include comprehensive observability platform extensions focused on automation, security, data management, and cross-IT engineering and operations.

Figure 9:
Prioritizing IT Skills: Automation, Security, Data Mgmt., and Multiple Technologies

Q. Below are some observability-related technical skills needed for the management of infrastructure, network, applications, cybersecurity, and digital experience. Thinking about your own organization, what are the staff skills you will prioritize when promoting, hiring, and developing IT staff?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

Future Outlook

In laying out their plans for observability, respondents offered answers directly matching often cited strategic business priorities. Productivity, impact, efficiency, effectiveness, and collaboration are all assigned high placement in observability plans (see Figure 10).

Figure 10:
Automation, Consolidation, and Managed Services Driving Observability Plans

Q. Considering your organization’s approach to observability across infrastructure, network, applications, cybersecurity, and digital experience management, to what extent do you agree with each of the following statements about IT solutions?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

When examining specific observability capabilities to be prioritized now and into the future, respondents reflect a movement to a more proactive management approach. Reactive management capabilities such as identify problems and threats and provide rapid remediation are assigned almost the exact same importance as proactive management functions such as predictive analysis, problem prevention, and digital acceleration (see Figure 11). And again, we see further evidence of the importance of a single source of truth and extensible observability solution – bolstered by support for standardized technologies (e.g., Open Telemetry).

Figure 11:
Problems, Predictions, and Precision Drive Future Observability Adoption

Q. In examining observability solutions adopted or to be adopted within your organization, to what extent do you agree with each of the following statements about observability solutions?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

When examining observability practices, future plans are aligned to multiple priorities. There is little difference among the top-ranked impact areas. Once again, high value is placed on cross-IT teamwork, IT automation, unified management, intelligent analytics, and reduced cost and complexity (see Figure 12). Organizations do indeed aspire to accomplish much with their future comprehensive observability platforms and efforts.

Figure 12:
Top Expectations Equally Ranked, but Team Collaboration Tops the List

Q. In examining observability solutions adopted or to be adopted within your organization, to what extent do you agree with each of the following statements about observability practices?

Source: Full Stack Observability Thought Leadership Survey, IDC, March 2023, n = 2,062

Challenges/Opportunities

Opportunities abound for organizations to reap benefits from the deployment of a comprehensive observability platform. Of course, opportunities are frequently made possible because of challenges that disrupt the way things operated in the past. Specific challenges and opportunities associated with a comprehensive observability approach include:

  • Customer Experience challenge: Customer Experience is the new growth engine and delivering a flawless and exceptional user-experience is directly associated with positive business outcomes.
  • Opportunity: there is a critical need to monitor and track real-time digital experiences for users and applications as a business requirement.
  • Security challenge: Security threats are rising while, at the same time, security vulnerabilities are increasing.
  • Opportunity: Security is pivoting from a surround-and-defend approach to an approach where security becomes intrinsic to each component of a digital infrastructure – from core systems and services to business data and applications. As such, comprehensive observability must support a strengthening of security postures and practices across all possible attack vectors and cyber targets.
  • Cross-domain challenge: Security, Operations, Network, and SRE executives face the need to create a collaborative culture that measures success based on data, using common metrics that drive behaviors, and a blameless culture that empowers teams to fail fast and learn without unnecessary career risks.
  • Opportunity: Organizations have a once-in-a-lifetime opportunity to fundamentally rethink how teams are organized around observability platforms, how teams integrate and collaborate, and how decision making is carried out.
  • Digital Infrastructure challenge: Organizations are struggling with organizational and technology complexity as their digital infrastructure evolves to support accelerating business demands, intensifying user demands, and an expanding web of IT systems and services – e.g., multiple clouds, zero trust, sustainability requirements and distributed and diverse networks.
  • Opportunity: IT organizations can reduce process and tool complexity by standardizing on a comprehensive observability platform that allows for consistent, policy-driven management across business workloads, multicloud environments, hyper-connected infrastructure, and digital experiences.

Conclusion

Many IT and business leaders have created false perceptions around the adoption and use of observability solutions and the full evaluation of the value they provide. Pressures from business teams and modern development practices are forcing security, networking, IT operations, and SREs to work faster and to shift their approach in observability from a reactive set of limited capabilities based on fragmented tools, processes, and disconnected teams to a proactive, automated and unified management posture. The comprehensive observability platform enables teams (i) to share critical data, views, and insights; (ii) boost operational efficiency and service quality via high data cardinality, deep impact analysis, and directed actions; and (iii) enable effective capabilities that serve diverse IT staff roles (e.g., security, networking, cloud, development…) and organizations at any level of management maturity and responsibility. In addition, a comprehensive observability platform enables IT organizations and their solution and service partners to create their own complementary observability capabilities to expand use cases and business value opportunities. This is the art of the possible for a comprehensive observability platform that transforms the way IT, business, and partner executives can adopt and drive value from observability — end-to-end and top-to-bottom — across the entire digital infrastructure.

Survey Demographics

Respondents per country

US406
Canada154
Mexico105
Brazil103
United Kingdom262
France157
Germany154
Netherlands101
India152
Singapore160
Japan153
Australia155

Primary Industry

Department

Company Size

Knowledge/Authority

An Executive Blueprint for an Observability Platform: Driving Operational Excellence and Business Outcomes through Analytics and Automation