What is Data Profiling and Why Profile Your Data?

Data profiling is the process for assessing the quality and structure of data sources so you have a complete, 100-percent-accurate picture of your data. Data profiling verifies that data columns are populated with the types of data you expect. If a profile reveals problems in data, you can define steps in your data quality project to fix those problems. Data profiling promotes good data governance.

Nothing puts a project at risk faster than starting with data that’s been compromised. Industry experience has shown that application modernization and data integration projects are prone to the same challenges and problems that are common to all types of IT projects: they suffer from time and budget overruns, tradeoffs between quality and deadlines, and outright project failures because they are based upon an inaccurate or incomplete understanding of the source data. A recent article by McKinsey noted that “Data quality is a significant, and often underestimated, challenge that should be addressed early in any digital effort.”

This happens because databases and applications are complex, the volumes data can be vast and difficult to unravel, and the process of understanding source data can be laborious and prone to error. Before data can be integrated or used in a cloud data warehouse, CRM, ERP, or business analytics application, its content, quality and structure must be understood.

A deeper look into the reasons why projects overrun or fail shows that most application modernization and data integration initiatives rely on external information to provide an understanding of the data. Much of this information—documentation, source programs, existing data models, and staff experience—is often outdated, incorrect, or missing. If the data is invalid, then it may take many iterations to fix the data, develop a process to ensure improvements, and validate that it is indeed correct (i.e., that it actually represents the source data). In this scenario, as much as 50 percent of a total project’s labor budget may be wasted on manual and outdated data analysis and diagnosis techniques, while poor understanding of the source data jeopardizes the project’s overall success.

So, whether your IT team is building a new cloud data warehouse or your management needs information so they can make strategic and trusted decisions, you need thorough data profiling to understand the quality, shape, and characteristics of your source data.

Here are some of the important questions that you can answer with good data profiling:

• Do we have the data necessary to complete the project on time and on budget?
• Does the data definition support our business requirements?
• Will the project be able to cost-effectively produce and maintain the information required by the business?
• Does the data consistently and accurately represent the business needs?
• Will the relationship between the data elements support the business requirements?
• Will we be able to integrate, consolidate, aggregate, cross-reference, and pivot the data for usable reports?
• What data needs to be cleansed?
• What data needs to be transformed?
• Will the data be correct, consistent, and stable?

The goal of the profiling process is to provide accurate metadata and complete metrics for understanding the data as it actually is—rather than how it was designed years or decades ago.The data will certainly have been altered Since its initial design, and the existing documentation may no longer accurately describe the content and structure of the data source. Using the following steps, the data analyst can quickly discover the true content and quality of the source data and take necessary actions to ensure the data is fit for purpose in the target system.

The first step in data investigation and profiling is to prepare the data source to be analyzed. Informatica Cloud Data Quality can access hundreds of millions of rows of data for analysis, enabling users to profile data from almost many data sources including Azure Synapse, Amazon S3, Snowflake, Google Big Query, ODBC, Oracle, Salesforce, delimited files, and more.

In the data profiling phase—an interactive process between the user and the software—Informatica Cloud Data Profiling analyzes data to discover the true content, structure, and quality of the data. The user reviews the results generated by the Cloud Data Profiling, enabling them to arrive at a model that is both consistent with the source data and is meaningful in a business context. In addition, data analysts can perform What-If scenarios by integrating data profiling with Data Quality assets, such as data quality rules, cleansing and standardization rules, address verification, and parsers.

Many organizations that have used profiling have tended to implement tactical solutions to improve quality within a single application or within a single business process. While this approach may mitigate the problem for part of the organization in the short term, these sorts of limited initiatives generally fail to achieve long-term data quality improvement. To solve the data quality issue for good requires an enterprise-wide approach that includes both IT and business.

By empowering data stewards, business analysts, and line-of-business managers, Informatica Cloud Data Profiling allows ownership of the data quality process so business can maximize the return on trusted data. Users can compare profile runs to check if the quality of the data is improving over time. And since Informatica Cloud Data Profiling offers the flexibility to filter and drill down on specific records for better detection of problems, the process can be applied as new data sources come on stream.

If your organization is involved in any type of application modernization, data migration, data integration, or data consolidation initiative that relies on an accurate understanding of the data sources then you need Informatica Cloud Profiling (part of Informatica Cloud Data Quality). Key benefits of Informatica Cloud Data Profiling include:

• Delivers accurate source system knowledge
• Improves corporate data quality and accuracy
• Enables efficient application modernization, data migration, and integration
• Helps expedite integration of multiple, disparate data sources
• Mitigates risk and reduces costly rework in data management projects
• Minimizes overruns in enterprise applications projects
• Improves productivity of data management projects
• Reduces development costs by fully understanding data content, quality, and structure

Why not try Cloud Data Quality for free for 30 days and get an accurate picture of your data today?