What Is a Data Catalog?
A data catalog helps data users identify which data assets are available and provides relevant context about that data, allowing them to assess the data for use. Data catalogs help you organize and evaluate information about your data, including:
Data Catalog Examples
Data catalogs have emerged as a foundational need for modern data-driven organizations. As a result, you’ll find them in many data management solutions. But not all data catalogs are created equal. Some are part of other tools, such as business intelligence tools (like Tableau) or analytics tools (like Databricks). These data catalogs only scan and catalog datasets and reports within that environment for a limited use case. The large cloud platform providers offer their own data catalogs as well. Microsoft Azure, Amazon Web Services (AWS) and Google Cloud offer catalogs focused on their cloud ecosystems. These data catalogs do not cover data on-premises or in other cloud ecosystems, which can result in vendor lock-in.
True enterprise-scale data catalogs can scan, catalog and inventory data of virtually all types across virtually all data sources for cloud and on-premises environments. This allows them to remain more platform- and vendor-agnostic. Some enterprise-scale catalogs take it a step further, offering the ability to scan and ingest metadata from other data catalogs. Because these solutions act as a “catalog of catalogs,” they can provide a comprehensive metadata system of record for the organization.
How Users Interact with a Modern Data Catalog
Here is a real-world example of how users interact with a data catalog.
An organization’s marketing team wants to create personalized campaigns to support cross-sell and upsell opportunities. The team contacts a data analyst within the organization to help find the relevant data to support their project.
The data analyst searches for order data within the organization’s data catalog solution. Upon selecting a data asset called “Orders,” she can quickly review the asset’s elements, quality, lineage and related policies. She determines that the data asset is appropriate for the marketing team to use for their personalized campaign initiative.
Watch this demo to see this example in action:
In this data catalog example, a data analyst uses Informatica Cloud Data Governance and Catalog (CDGC) to help a marketing analyst gain context about the data needed for analytics to help generate personalized customer campaigns.
Data Catalog Template
“What should be in our data catalog?” is an important question. The more businesses across your organization use and add context to the data, the more valuable the data becomes. This increases the likelihood that the data catalog can help data stewards, data analysts, data scientists and other data consumers efficiently locate and assess data across the organization.
Modern data catalogs automatically harvest metadata (data about data) from data across on-premises and cloud environments, allowing users to enrich the metadata further. This helps improve productivity and operational efficiency by streamlining or automating critical tasks.
The types of metadata in a data catalog include:
- Technical metadata, including database schemas, mappings and code, transformations and quality checks
- Business metadata, including glossary terms, data governance processes and application and business context metadata
- Operational and infrastructure metadata, including run-time stats, time stamps, volume metrics, log information and other system metadata
- Usage metadata, including user ratings, comments and access patterns
Modern data catalogs leverage this metadata to enable automated and AI-powered capabilities such as:
- Bulk import of data assets via data catalog templates
- Automated data discovery
- Semantic search
- Domain and entity recognition
- Automated data classification
- Automated association of glossary terms to technical data assets
- Data profiling
- Inferred data lineage and relationships
- Data recommendations
To learn more about critical features in a data catalog, read this blog, “Which Data Catalog Features Do I Really Need?”
Data Catalog Use Cases
Modern AI-powered/machine learning-enabled data catalogs can support many use cases, such as:
- Data discovery
- Data lineage and impact analysis
- Common business understanding of business terms and policies
- Data governance and modernization of cloud data warehouses and data lakes
- Sensitive data classification and management
- Policy compliance
- Environmental, social and governance (ESG) reporting
- Data quality monitoring and improvement
- Master data governance
- AI and analytics governance
How Do I Create a Data Catalog?
Another common question is, “How do I develop a data catalog?” To get started with a data catalog, follow these best practices:
- Formulate a data governance and data catalog program strategy: Set up your program for success by making sure it reflects your organization’s overall vision and business objectives.
- Define a pilot project: Start small by defining a pilot project focused on critical data sets informed by top business imperatives at your organization.
- Get started: Demonstrate value with limited resources to rally the organization and accelerate further adoption.
- Lay the governance foundation: Build a robust, holistic and adaptive foundation to help ensure your data governance and cataloging program succeeds.
- Build momentum and scale: Grow adoption for your data catalog solution by incrementally developing use cases that align business and technical needs.
- Refine and expand: Review the program with sponsors and contributors and take time to reflect and fill in any gaps.
- Track usage and solicit feedback: You can’t improve what you don’t measure. Ensure that your success criteria are driven by business goals linked to data.
- Train and enable users: Users need training to understand the benefits of data governance and cataloging and how to use these new capabilities. Organizations must develop a plan to train their staff to support the solution and then ultimately teach end users how to use the solution in a way that supports their business.
For more details on how to get started with building a data catalog, check out this article: 4 Ways to Start with a Data Catalog.
What Makes a Good Data Catalog?
At a minimum, a good data catalog should be able to meet the following requirements:
- Data Intelligence Powered by AI and Machine Learning
With the sheer volume of data across the modern enterprise, manually discovering and classifying all your business-critical data becomes impossible. An AI-powered data catalog with a machine learning-based discovery engine delivers the data intelligence that modern data-driven organizations need. It automates necessary processes, including curation, classification, detection of similar data and association of business terms to physical datasets. With AI/ML, not only can your data catalog provide end-to-end visibility, but you can guide users to the most relevant and trusted data for their business requirements.
- Broad and Deep Connectivity
It’s important to have a data cataloging solution that can intelligently inventory data as the size and complexity of organizations’ data landscapes continue to increase. Your data catalog should also be able to inventory metadata across a variety of sources and applications. Ensure your data catalog tool has broad and deep connectivity across cloud and on-premises systems and applications. Some data catalog solutions are vendor-specific, limiting the solution's efficacy. A catalog of catalogs — a data catalog with universal metadata connectivity — provides a centralized, comprehensive view of your data across sources and is essential for getting value from this data. Additionally, your data catalog should be able to scan across a wide range of business, technical, operational and usage metadata to discover all your critical data.
- End-to-End Data Lineage
Data lineage is a visual representation of how data flows from its origin to its destination. This helps with understanding how data changes along its journey. Many data catalogs need help tracing end-to-end lineage across systems or when data moves from on-premises to the cloud.
A data catalog solution that automates data stewardship activities can help ensure trustworthy data is available for users by aligning business and technical stakeholders around data purpose. Automated governance helps improve data quality and supports the management of appropriate data use.
Additional Resources and Tools
- What is a data catalog?
- Watch our Back to Basics: Data Catalog webinar to learn more about the fundamentals of data catalog tools.
- Learn more about the benefits of a machine-learning data catalog.
- Experience Cloud Data Catalog in action with our interactive demo.
