India's largest platform for AI & Analytics leaders & professionals

Sign in

India's largest platform for AI & Analytics leaders & professionals

3AI Digital Library

Top four focus areas to help you shape a data-driven enterprise

3AI March 24, 2022

Author: Praveen Reddy, Vice President (Digital – data, analytics, and cloud) | Genpact

As we navigate the world of data, some significant trends are manifesting with respect to data ownership and utilization. There are multiple triggers for these trends – ever-rising complexity of data, lack of proper intelligence concerning data assets, need for accelerated digital transformation, and data democratization. This list could go on. And the only way to respond to these challenges is by enabling organizations to become data driven.

Domain data fabric

The lack of data ownership is the most critical challenge here. If data is the holy grail that drives transformation, then there must be a proper context, ownership, and governance around it. It must be treated as an end-to-end (E2E) product rather than a commodity. To achieve this, it is not enough to build monolithic architectures such as data warehouses and data lakes and populate them through conventional ETL processes to meet consumption needs. We must move to a distributed architecture framework with the domain as the pivot for distribution and focus on self-sustenance, in terms of platform and ownership. This results in a mesh type of architecture called the data mesh (term defined by Zhamak Dehghani, a ThoughtWorks consultant).

But the important concern here is how should we define a domain. Is it based on the source data that is usually closely aligned to business functions or operational systems or is it based on consumer needs that are primarily driven by a set of use cases? Consumer-driven data sets are required to undergo different transformations like normalization, denormalization, aggregation and more.

For a domain data fabric to be successful, you need to meet the following checkboxes:

  • Complete product mindset (more on the topic here)
    • Vision
    • Roadmap
    • Life cycle management
    • Key performance indicators (KPIs)
    • Versioning
    • Monetization
  • Easy accessibility
  • Effective governance
  • Ownership

Advanced and active metadata management solutions

While there is tremendous progress in terms of management of modern data platforms and tools, we cannot say the same when it comes to speed, efficiency, and capability with respect to ingestion, transformation, and consumption of metadata. There is a lot that remains to be investigated as far as semantics, governance, reliability, and trust are concerned. With the rise in complexity and use of IT systems, the need to govern the different types of assets, stakeholders, and use cases to be supported, has expanded significantly. Thus, metadata management solutions have fallen behind in addressing industry requirements.

To address these gaps, modern solutions need to be scalable, flexible and must allow a high degree of collaboration capabilities, given that the size and complexity of the metadata that needs to be stored, processed, and connected have also increased. Further, all types of metadata (technical, operational, and business) across different platforms and tools need to be linked in a much more contextual and intelligent manner to meet expectations. In fact, some of the capabilities of modern IT systems (infinite scalability of cloud platforms, embedded collaboration, advances in semantics) will help in addressing these challenges.

Gartner made a big change last year by scrapping Magic Quadrant for Metadata Management Solutions and replacing it with a market guide for active metadata. This helps in identifying a broader set of platforms/tools that significantly enables active metadata use in organization’s transformation journey.

Unified and integrated metrics repository

Though surprising, organizations are still struggling to create a metadata-driven simple, flexible, scalable, and unified metrics layer. In the existing scenario, most of the metrics are pre-defined, designed, and tightly embedded in an organization’s BI platforms. While BI platforms are good at abstracting technical complexity from users (for example, users can avoid writing complex SQLs and can easily create and visualize simple metrics based on underlying data), these metrics can’t be extended beyond BI tools across other platforms and departments within an organization. Also, these can’t be clubbed with other metrics to come up with a set of integrated metrics without significant engineering effort.

One possible new approach Headless BI (introduced by Base Case) can solve this to some extent and the solution lies in separating visualization from metrics definition and ownership. This way, the team that owns the metrics, defines and catalogs them only once. And different tools/platforms and departments can access these metrics based on their needs. This goes a long way and helps organizations in addressing their metrics problems.

Data observability and data quality

When an organization goes through digital transformation and becomes data-driven, the reliability of its E2E pipelines and the quality of the data across all the pipelines become extremely important. This, in a way, changes how we think about data quality, reliability, and usage. Earlier, these issues were more application or process specific. But as we embrace a data-driven and data as a product mindset, forecasting, and real-time monitoring, tracking, and triaging of issues (E2E of this is called data observability) have become critical to any organization and are tightly linked to the emerging trends of today.

Therefore, it is crucial to have an integrated and intelligent data platform (a single platform/tool or a combination of platforms and tools) that can provide the following:

  • Organize all data sets into domain-based, self-sustaining, and distributed repositories
  • View, link, and tag all the assets, based on context-sensitive metadata, quality, and trust
  • Adopt a product mindset
  • Unify semantic framework for organization-wide metrics
  • Focus on real-time data observability, stewardship, and correctness

Path forward

Organizations that successfully establish these capabilities, frameworks, and mindset, have a better chance of accelerating and controlling their digital transformation initiatives for better, faster, and expected outcomes. This will help businesses to become more resilient and handle any unexpected scenario like COVID-19 and avoid getting disrupted or going out of the market.   

    3AI Trending Articles

  • The Role of Emotional Intelligence in an AI-Driven World: A Crucial Human Skill for Symbiotic Relationships

    Featured Article: Author: Anees Merchant, EVP, Global Growth and Client Success, Course5 Intelligence The relentless progress of artificial intelligence (AI) is indisputable, and it is rapidly transforming many sectors worldwide, from healthcare and education to commerce and manufacturing. The term “AI-driven world” is no longer futuristic but a present reality. However, amidst this remarkable advancement, […]

  • Commodity Price Forecasts using ML driven Insights

    Featured Article: Author: Tarana Chauhan, Procurement Analyst, AB InBev Dependency on Commodities and Associated Risks: Companies with Agricultural commodities as their core raw material face several risks in supply security. Agricultural commodities not only suffer from the risks associated with market dynamics like all other commodities but are also impacted by environmental factors making them […]

  • Sebi is setting up a Cybersecurity Fusion Centre

    Establishing a cybersecurity fusion centre or a cyber lab is part of Sebi’s three-tier structure for monitoring cybersecurity-related events in the securities markets. . NEW DELHI: Markets regulator Sebi is in the process of setting up a cybersecurity fusion centre, a move aimed at detecting cyber threats faster and resolve such incidents efficiently and effectively. […]

  • Property Registration in Karnataka to use Blockchain Technology

    Property registration in Karnataka is all set to become more secure and hassle-free, with the state government developing a system based on blockchain technology for online property documentation. The new system, developed in collaboration with IIT-Kanpur, promises an immutable electronic storage of property data through blockchain. In other words, data once stored cannot be changed, […]