Revolutionizing Cloud Data Management for the era of AI

Generative AI has captured the imagination of everyone, and organizations are eager to use it to drive better business outcomes. Although percolating for decades, its sudden traction reflects three major developments:

• A massive proliferation of data
• Availability of compute scale capacity
• Machine learning (ML) innovations

According to McKinsey, the economic potential of generative AI can add trillions of dollars in value to the global economy. This research is based on 63 use cases, each of which ties back to data and analytics.¹ With such potential tied to the unreliability of this data foundation, chief data officers (CDOs) are facing a sobering proposition. Nearly 72% of leading organizations acknowledge that data management is their top challenge, preventing them from scaling AI and analytics use cases.²

Further, most organizations are burdened with a maze of disconnected and disparate services. Unfortunately, this lack of cohesion, along with poor quality data, can lead to inaccurate insights. If your data is not high quality data, your business is not ready for generative AI and modern analytics.

For generative AI and large language model (LLM)-managed services to improve outcomes, there needs to be a massive supply of clean high-quality data available through a data management platform that is deeply integrated in a modern analytics system. This is where Informatica comes in.

To help customers foster an intelligent enterprise, Informatica has established deep integrations with modern analytics platforms, like Microsoft Fabric, which addresses virtually every aspect of your organization’s analytics platform needs.

Informatica, an ISV Design Partner of Microsoft Fabric, has been working closely with the Microsoft Fabric engineering teams to offer the power of AI-powered Informatica Intelligent Data Management Cloud (IDMC) as a native application for all Microsoft Fabric users. With IDMC deeply integrated as a unified experience on Microsoft Fabric, customers can discover and use key data management solutions, including data profiling, data quality, data observability and data integration, to build trusted data that feeds Microsoft Fabric analytics workloads. By delivering these key functions through a software as a service (SaaS) experience, Informatica helps integrate and optimize data management into Microsoft Fabric’s data analytics.

IDMC is powered by CLAIRE, our AI engine, and helps connect, manage and unify data across virtually any multi-cloud or hybrid system. This enables organizations to democratize data and advance their business strategies, a must in today’s competitive climate. IDMC also helps organizations build specific functions and capabilities into the data architecture (i.e., data integration patterns, data catalog, detailed data lineage, data governance, master data management, etc.) that support the broadest set of use cases and the entire data lifecycle within the organization.

Managing the dynamic and intricate data lifecycle for an analytics platform like Microsoft Fabric requires a new approach to the user experience. Let’s review what’s involved in making this a differentiated experience.

Trusted data is the foundation for trusted analytics and generative AI. Without trust and data quality, the insights you drive from the analytics platform is questionable and not fit for business use. Informatica addresses data usability through three critical IDMC capabilities that are embedded in Microsoft Fabric, which makes for a seamless experience.

1. Data profiling. Data profiling is the process of examining and analyzing the data available on Microsoft Fabric (OneLake) and collecting statistics and an informative summary about the data. With profiling, users can get a high-level overview of the distribution of data, discover quality issues, see trends and more. This function can be invoked directly from your Microsoft Fabric experience.
2. Data quality. Once the data is profiled, users can view the quality statistics for the data and apply rules to cleanse, preview and validate the data. These rules are powered by the Informatica CLAIRE engine through a natural language processing (NLP)-based approach to data quality rules and cleansing. The ability to identify, cleanse, validate and apply the quality rules ensures that the datasets in Microsoft Fabric are usable and provide trusted data for trusted insights. Best part? Users can access data quality functions of IDMC within Microsoft Fabric without ever having to leave the Microsoft Fabric screens.
3. Data observability. With data observability, users can observe data in real time and can quickly identify and resolve data quality issues so organizations can maintain standards and expectations throughout the entire data lifecycle. This allows Microsoft Fabric users to confidently make decisions based on accurate, timely and reliable information.

Figure 1: IDMC as a Native Microsoft Fabric Application.

In addition to the value Informatica adds to the Microsoft Fabric platform with data profiling, data quality and data observability, it also brings the full power of IDMC to Microsoft Fabric. This includes the ability for users to ingest data with virtually any patterns — extract, load, transform (ELT) and extract, transform, load (ETL), real-time, batch, change data capture — using a rich set of connectors. Along with supporting Azure Databricks, IDMC provides three new connectors for Microsoft Fabric —  One Lake, Lakehouse and Data Warehouse — so users can have seamless ingestion with support for both Delta Parquet and SQL endpoints.

The strategic partnership between Informatica and Microsoft is built on a foundation of co-innovation, deep collaboration and a shared passion to deliver world class experiences in data, AI and analytics. 

The relationship expands beyond Informatica being a Design Partner for Microsoft Fabric. CLAIRE is built upon Azure OpenAI, which powers the GPT and copilots of the Informatica AI engine. IDMC is also the only data management solution offered as an Azure Native ISV Service. This gives Azure customers a simple unified experience to seamlessly discover, access, provision, onboard, configure and manage IDMC on Azure from within the Azure Portal.

"Informatica’s Cloud Data Governance Catalog has been key in providing us with a firm foundation in how we understand our data and how it flows across our enterprise. We are excited to see Informatica’s upcoming developments on this platform that will enable us to better understand our data relationships and holistically connect our data universe in our enterprise so that we may deliver true value to all our stakeholders."

- Shoby Kelliher, Global Head of Enterprise Data, Janus Henderson Investors

Combined, Informatica and Microsoft have thousands of joint customers across different segments and industries that drive success and help transform data and analytics in the AI era.

Get ready to revolutionize cloud data management for the era of AI:

• Visit Informatica at Microsoft Ignite in Seattle November 14–17 or virtually on November 15–16 for demos, giveaways and more.
• Sign up for a private preview of Informatica on Microsoft Fabric.

_{¹https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier
²https://www.technologyreview.com/2023/10/05/1080618/laying-the-foundation-for-data-and-ai-led-growth/}