India's largest platform for AI & Analytics leaders, professionals & aspirants

Sign in

India's largest platform for AI & Analytics leaders, professionals & aspirants

3AI Digital Library

Top four focus areas to help you shape a data-driven enterprise

March 24, 2022

Author: Praveen Reddy, Vice President (Digital – data, analytics, and cloud) | Genpact

As we navigate the world of data, some significant trends are manifesting with respect to data ownership and utilization. There are multiple triggers for these trends – ever-rising complexity of data, lack of proper intelligence concerning data assets, need for accelerated digital transformation, and data democratization. This list could go on. And the only way to respond to these challenges is by enabling organizations to become data driven.

Domain data fabric

The lack of data ownership is the most critical challenge here. If data is the holy grail that drives transformation, then there must be a proper context, ownership, and governance around it. It must be treated as an end-to-end (E2E) product rather than a commodity. To achieve this, it is not enough to build monolithic architectures such as data warehouses and data lakes and populate them through conventional ETL processes to meet consumption needs. We must move to a distributed architecture framework with the domain as the pivot for distribution and focus on self-sustenance, in terms of platform and ownership. This results in a mesh type of architecture called the data mesh (term defined by Zhamak Dehghani, a ThoughtWorks consultant).

But the important concern here is how should we define a domain. Is it based on the source data that is usually closely aligned to business functions or operational systems or is it based on consumer needs that are primarily driven by a set of use cases? Consumer-driven data sets are required to undergo different transformations like normalization, denormalization, aggregation and more.

For a domain data fabric to be successful, you need to meet the following checkboxes:

  • Complete product mindset (more on the topic here)
    • Vision
    • Roadmap
    • Life cycle management
    • Key performance indicators (KPIs)
    • Versioning
    • Monetization
  • Easy accessibility
  • Effective governance
  • Ownership

Advanced and active metadata management solutions

While there is tremendous progress in terms of management of modern data platforms and tools, we cannot say the same when it comes to speed, efficiency, and capability with respect to ingestion, transformation, and consumption of metadata. There is a lot that remains to be investigated as far as semantics, governance, reliability, and trust are concerned. With the rise in complexity and use of IT systems, the need to govern the different types of assets, stakeholders, and use cases to be supported, has expanded significantly. Thus, metadata management solutions have fallen behind in addressing industry requirements.

To address these gaps, modern solutions need to be scalable, flexible and must allow a high degree of collaboration capabilities, given that the size and complexity of the metadata that needs to be stored, processed, and connected have also increased. Further, all types of metadata (technical, operational, and business) across different platforms and tools need to be linked in a much more contextual and intelligent manner to meet expectations. In fact, some of the capabilities of modern IT systems (infinite scalability of cloud platforms, embedded collaboration, advances in semantics) will help in addressing these challenges.

Gartner made a big change last year by scrapping Magic Quadrant for Metadata Management Solutions and replacing it with a market guide for active metadata. This helps in identifying a broader set of platforms/tools that significantly enables active metadata use in organization’s transformation journey.

Unified and integrated metrics repository

Though surprising, organizations are still struggling to create a metadata-driven simple, flexible, scalable, and unified metrics layer. In the existing scenario, most of the metrics are pre-defined, designed, and tightly embedded in an organization’s BI platforms. While BI platforms are good at abstracting technical complexity from users (for example, users can avoid writing complex SQLs and can easily create and visualize simple metrics based on underlying data), these metrics can’t be extended beyond BI tools across other platforms and departments within an organization. Also, these can’t be clubbed with other metrics to come up with a set of integrated metrics without significant engineering effort.

One possible new approach Headless BI (introduced by Base Case) can solve this to some extent and the solution lies in separating visualization from metrics definition and ownership. This way, the team that owns the metrics, defines and catalogs them only once. And different tools/platforms and departments can access these metrics based on their needs. This goes a long way and helps organizations in addressing their metrics problems.

Data observability and data quality

When an organization goes through digital transformation and becomes data-driven, the reliability of its E2E pipelines and the quality of the data across all the pipelines become extremely important. This, in a way, changes how we think about data quality, reliability, and usage. Earlier, these issues were more application or process specific. But as we embrace a data-driven and data as a product mindset, forecasting, and real-time monitoring, tracking, and triaging of issues (E2E of this is called data observability) have become critical to any organization and are tightly linked to the emerging trends of today.

Therefore, it is crucial to have an integrated and intelligent data platform (a single platform/tool or a combination of platforms and tools) that can provide the following:

  • Organize all data sets into domain-based, self-sustaining, and distributed repositories
  • View, link, and tag all the assets, based on context-sensitive metadata, quality, and trust
  • Adopt a product mindset
  • Unify semantic framework for organization-wide metrics
  • Focus on real-time data observability, stewardship, and correctness

Path forward

Organizations that successfully establish these capabilities, frameworks, and mindset, have a better chance of accelerating and controlling their digital transformation initiatives for better, faster, and expected outcomes. This will help businesses to become more resilient and handle any unexpected scenario like COVID-19 and avoid getting disrupted or going out of the market.   

    3AI Trending Articles

  • IIT Delhi Offers Free Online Course on AI

    IIT Delhi is offering a free online 12-week programme on Artificial Intelligence with a certificate on completion. Indian Institute of Technology (IIT) Delhi is inviting registrations from those who would like to enrol for a 12-week long free online course on Artificial Intelligence. This programme will be offered on the National Programme on Technology Enhanced […]

  • Microsoft Announces Limited Access to its Custom Neural Voice

    Microsoft announced limited access to its neural text-to-speech AI called Custom Neural Voice. The service allows developers to create custom synthetic voices. . The Custom Neural Voice is a Text-to-Speech (TTS) feature of Speech in Azure Cognitive Services that allows users to create a one-of-a-kind customized synthetic voice for their brand.  Since the preview last year in September, the […]

  • Reimagine Business Strategy & Operating Models with AI : The CXO’s Playbook

    Author: Sameer Dhanrajani, CEO at AIQRATE advisory & consulting | AI advisor, business builder | Author | Speaker |Columnist| President at 3AI | Ex-CSO: Fractal, Global Business Leader: Cognizant, Country Head: Fidelity National Financial AlphaGo caused a stir by defeating 18-time world champion Lee Sedol in Go, a game thought to be impenetrable by AI […]

  • Daimler India opens new GCC

    Daimler India Commercial Vehicles (DICV) has opened a new Global Capability Centre in Chennai with 165 seats to meet growing demand for Indian-based shared services. Spread over 20,000 sq.ft., the facility is designed to house the company’s ‘shared services’ business stream. Shared services refers to the various services DICV exports to other Daimler entities around […]