India's largest platform for AI & Analytics leaders, professionals & aspirants

Sign in

India's largest platform for AI & Analytics leaders, professionals & aspirants

3AI Digital Library

Top four focus areas to help you shape a data-driven enterprise

March 24, 2022

Author: Praveen Reddy, Vice President (Digital – data, analytics, and cloud) | Genpact

As we navigate the world of data, some significant trends are manifesting with respect to data ownership and utilization. There are multiple triggers for these trends – ever-rising complexity of data, lack of proper intelligence concerning data assets, need for accelerated digital transformation, and data democratization. This list could go on. And the only way to respond to these challenges is by enabling organizations to become data driven.

Domain data fabric

The lack of data ownership is the most critical challenge here. If data is the holy grail that drives transformation, then there must be a proper context, ownership, and governance around it. It must be treated as an end-to-end (E2E) product rather than a commodity. To achieve this, it is not enough to build monolithic architectures such as data warehouses and data lakes and populate them through conventional ETL processes to meet consumption needs. We must move to a distributed architecture framework with the domain as the pivot for distribution and focus on self-sustenance, in terms of platform and ownership. This results in a mesh type of architecture called the data mesh (term defined by Zhamak Dehghani, a ThoughtWorks consultant).

But the important concern here is how should we define a domain. Is it based on the source data that is usually closely aligned to business functions or operational systems or is it based on consumer needs that are primarily driven by a set of use cases? Consumer-driven data sets are required to undergo different transformations like normalization, denormalization, aggregation and more.

For a domain data fabric to be successful, you need to meet the following checkboxes:

  • Complete product mindset (more on the topic here)
    • Vision
    • Roadmap
    • Life cycle management
    • Key performance indicators (KPIs)
    • Versioning
    • Monetization
  • Easy accessibility
  • Effective governance
  • Ownership

Advanced and active metadata management solutions

While there is tremendous progress in terms of management of modern data platforms and tools, we cannot say the same when it comes to speed, efficiency, and capability with respect to ingestion, transformation, and consumption of metadata. There is a lot that remains to be investigated as far as semantics, governance, reliability, and trust are concerned. With the rise in complexity and use of IT systems, the need to govern the different types of assets, stakeholders, and use cases to be supported, has expanded significantly. Thus, metadata management solutions have fallen behind in addressing industry requirements.

To address these gaps, modern solutions need to be scalable, flexible and must allow a high degree of collaboration capabilities, given that the size and complexity of the metadata that needs to be stored, processed, and connected have also increased. Further, all types of metadata (technical, operational, and business) across different platforms and tools need to be linked in a much more contextual and intelligent manner to meet expectations. In fact, some of the capabilities of modern IT systems (infinite scalability of cloud platforms, embedded collaboration, advances in semantics) will help in addressing these challenges.

Gartner made a big change last year by scrapping Magic Quadrant for Metadata Management Solutions and replacing it with a market guide for active metadata. This helps in identifying a broader set of platforms/tools that significantly enables active metadata use in organization’s transformation journey.

Unified and integrated metrics repository

Though surprising, organizations are still struggling to create a metadata-driven simple, flexible, scalable, and unified metrics layer. In the existing scenario, most of the metrics are pre-defined, designed, and tightly embedded in an organization’s BI platforms. While BI platforms are good at abstracting technical complexity from users (for example, users can avoid writing complex SQLs and can easily create and visualize simple metrics based on underlying data), these metrics can’t be extended beyond BI tools across other platforms and departments within an organization. Also, these can’t be clubbed with other metrics to come up with a set of integrated metrics without significant engineering effort.

One possible new approach Headless BI (introduced by Base Case) can solve this to some extent and the solution lies in separating visualization from metrics definition and ownership. This way, the team that owns the metrics, defines and catalogs them only once. And different tools/platforms and departments can access these metrics based on their needs. This goes a long way and helps organizations in addressing their metrics problems.

Data observability and data quality

When an organization goes through digital transformation and becomes data-driven, the reliability of its E2E pipelines and the quality of the data across all the pipelines become extremely important. This, in a way, changes how we think about data quality, reliability, and usage. Earlier, these issues were more application or process specific. But as we embrace a data-driven and data as a product mindset, forecasting, and real-time monitoring, tracking, and triaging of issues (E2E of this is called data observability) have become critical to any organization and are tightly linked to the emerging trends of today.

Therefore, it is crucial to have an integrated and intelligent data platform (a single platform/tool or a combination of platforms and tools) that can provide the following:

  • Organize all data sets into domain-based, self-sustaining, and distributed repositories
  • View, link, and tag all the assets, based on context-sensitive metadata, quality, and trust
  • Adopt a product mindset
  • Unify semantic framework for organization-wide metrics
  • Focus on real-time data observability, stewardship, and correctness

Path forward

Organizations that successfully establish these capabilities, frameworks, and mindset, have a better chance of accelerating and controlling their digital transformation initiatives for better, faster, and expected outcomes. This will help businesses to become more resilient and handle any unexpected scenario like COVID-19 and avoid getting disrupted or going out of the market.   

    3AI Trending Articles

  • How Augmented Analytics is Transforming the Analytics Ecosystem

    Author:  Sidharth Sivasailam, Vice President – Products, Course5 Intelligence | LinkedIn – The world of Business Analytics is at an inflection point. Trillions of bytes of data are being generated every day; however, companies continue to struggle with harmonizing this data, analyzing the data of various shapes and sizes they are storing, determining what’s most […]

  • IIT Delhi Offers Free Online Course on AI

    IIT Delhi is offering a free online 12-week programme on Artificial Intelligence with a certificate on completion. Indian Institute of Technology (IIT) Delhi is inviting registrations from those who would like to enrol for a 12-week long free online course on Artificial Intelligence. This programme will be offered on the National Programme on Technology Enhanced […]

  • New 2D tag with AI authentication to spot counterfeit products Faster & Accurately

    The authentication process takes under 3.5 minutes to complete, and involves scanning the tags under an electronic microscope to obtain the PUF pattern, which is sent to the AI-driven software for validation. A new anti-counterfeiting technique uses two dimensional (2D)-material tags along with artificial intelligence (AI)-driven authentication software, and promises to deliver faster, more accurate […]

  • Improving Customer Experience – AI/ML Driven Issue Resolution

    Author: Ashish Vashishta, VP – Technology, eClerx Customers across industries typically face issues with their purchased products/services. A troubleshooting system which can use latest AIML technologies to resolve these issues faster and more effectively can lead to a much better customer experience. As an example, a customer may have purchased a bundled product offering – […]