Co-authored with Sam Tawfik, Principal Product Marketing Manager, Cloud Integration.
In Deloitte’s recent 2020 retail industry outlook, retailers are advised to focus on the following two critical success factors (among others)—technology/automation and partnerships. Lack of agility in the supply chain has recently plagued many manufacturing and retail businesses, including even big giants like Amazon, who have found themselves short on products like masks, gloves, and sanitizers. There are obvious gaps in manufacturing and transportation which have contributed to this shortfall.
However, part of the problem also lies in the lack of digitization. Even if a hat manufacturer quickly transforms itself into a mask manufacturer, if the company’s technology wasn’t integrated with that of Amazon’s or Walmart’s marketplaces—so that it could automatically receive orders that flow straight through to billing, fulfillment, and shipping—a lot of time and resources would be wasted in every step of the process. If, on the other hand, the manufacturer has its order data and applications integrated with its customers’, the orders will automatically show up, be managed, and, as a result, be considerably accelerated.
Now that we have talked about business operations, lets discuss analytics. Businesses want to be able to analyze data. In the above example, for a manufacturer, this would entail the number of orders received within any given timeframe, its trend, order volume by customer, fulfillment SLA attained, etc. If business analysts or management want to inspect order volume, its longitudinal behavior, etc., they need to have easy and quick access through self-service analytics.
There are multiple infrastructure technologies for the type of process automation referenced above, including application integration and data integration. As a business process moves forward, for the control to be passed from one application to another, the applications need to be connected to each other. Now imagine that the above-mentioned hat/mask manufacturer starts processing orders 24 hours after reception. The order request contains basic information about the order, and the complete order data is loaded into the data warehouse using periodic batch jobs. In this case, data integration is used to extract the data from the order system and integrate it with the rest of the business systems for customer, product, and sales analytics, as well as for supply chain planning.
As you can see, organizations rely on both application and data integration to automate processes and share real-time and batch data. These are two distinct but essential capabilities, the yin and yang that businesses need to achieve automation and digital transformation.
Let’s start with the yin of digital transformation, application integration—it is used to connect different applications and is performed in (near) real-time to complete a business process or real-time data integration, such as receiving a customer order or to provide immediate customer data to a customer service representative for credit approval.
Here is a bit of a sidebar on API management in the context of integration: APIs have evolved as one of the most popular ways to integrate applications. This is because APIs are light, loosely coupled, and can be secured and controlled. The advent of the technology called API management—which provides the ability to publish, subscribe, secure, monitor, and manage APIs—was inevitable. After all, no one wants a rampant proliferation of unmanaged APIs. API management, however, doesn’t integrate applications. Rather, it’s an extension of application integration because app integration technologies provide the ability to create and author APIs. API management provides a means to control them.
Back to integration: Even ten years back, enterprises would use ESBs, integration servers, and message brokers to integrate their applications. However, with the advent of cloud, integration platform as a service (iPaaS) has become the defacto technology for application integration. It has quickly evolved to include data integration for multiple clouds and on-prem deployments.
Analysts joke about the birth of iPaaS being driven by the need to integrate customers’ on-prem SAP implementation with their cloud-based Salesforce CRM. This is indeed the most basic use case that iPaaS technology needs to support. In addition, iPaaS technology should support advanced data management needs (we’ll talk about those a bit later in this post); provide a plethora of out-of-the-box connectivity options to popular cloud and on-prem systems and data stores; and offer authoring tools to define a process, an API, or a data mapping easy enough to be used by developers, citizen integrators, and analysts alike. Typical iPaaS platforms also come with developer and operations collaboration tools for version control, CICD, and DevOps.
The yang of digital transformation is data integration, which is used to load or replicate new data from heterogenous data sources into a central repository such as a data warehouse. This provides a unified view of the business for analytics and reporting. Analytics includes dashboards, data mining, predictive analytics, and self-service analytics.
Data integration is also used to migrate data from legacy systems to modern environments or consolidate data from operational systems into a data warehouse to reduce the impact on the operational systems. Data integration tools are categorized as data extraction, transformation, and loading (ETL) tools. ETL tools are used to ingest or extract the data from a variety of sources, provide data transformation capabilities to standardize or cleanse the data, and load the data into the target system(s). The ETL tools include a visual designer to help developers create data pipeline mappings to extract, transform, and load the data into the data warehouse.
Application integration is typically used for process or API-centric integration and data integration is used for data-centric integration via batch jobs that are processed periodically—weekly, daily, hourly, or as needed. This data-centric approach is designed for collecting data for the purpose of historical analysis and consists of integrating millions or billions of sales transactions, orders, insurance claims, clinical tracking activities, machine productions, and other types of data. With digital transformation, more and more data integration solutions provide the capability to integrate streaming and very high-volume, real-time data to process time-sensitive data. This includes sensor data to avoid production disruptions, store transactions to prevent fraud while it is happening, and supply chain routing to avoid weather delays or to optimize inventory levels. As on the applications side, the majority of the analytics solutions have been migrated to the cloud with cloud data warehouse solutions but still rely on data integration as the key technology to manage their data.
Hopefully, we have made it abundantly clear that for businesses to achieve digital transformation they need both data and application integration. Moreover, organizations want these tools to be part of the same platform, which makes absolute sense. According to Gartner, “Organizations want to have data integration and application integration capabilities in their data and analytics architecture, along with the flexibility of using any number of solutions and using the same talent to optimize resources and remove redundancies.”
Gartner adds, “Technology and service providers in the data integration market should support through internal product development, or through partnerships, the ability to interoperate data integration with application integration capabilities, such as enterprise service bus (ESB), API integration and integration platform as a service (iPaaS).”
However, having both data and application integration capabilities in one unified platform is just the beginning. Data management is the next frontier, because what good is data if it can’t be trusted or secured? It’s not very useful if you can’t consolidate the data to provide a comprehensive view of your customer, is it? And then there is artificial intelligence, which harnesses the power of your data and metadata to automate your processes, making these more efficient and intelligent. This is why Gartner says, “Technology and service providers in the data integration market should start providing capabilities for data quality and metadata management throughs strategic partnerships and strong out-of-the-box integration with the top stand-alone vendors in those markets initially. At the same time, start adding these capabilities incrementally through in- house development and mergers and acquisitions (M&As).”
No blog (or thought process, really) these days is complete without a nod to the changes in business models, supply chain, customer behavior, and regulations that enterprises are experiencing with the new 2020 world order. Focusing on business continuity is paramount in today’s climate, which calls for “lights-out” operations. But think about it—if you didn’t have your data and applications connected, would you be able to achieve continuous operations?
Going back to the example above: Unless you integrate your applications, you will be manually moving your business process from one system to another, perhaps via an email notification. We have all heard scary stories of manual data sharing through spreadsheets, which makes the entire process slow, unreliable, and insecure. Now think about a business process “auto-correcting” for an order that halted due to an apparent lack of inventory (an error), even with substantial inventory at-hand—the result of application integration and data integration running on the same platform.
How? A job running on your data integration service failed (the reason for the failure is not important) resulting in incorrect inventory data. Since the application integration service is running on the same platform, it has immediate knowledge of this failure. It waits for the inventory update to complete successfully and in the meantime even sends an email to the customer informing them of a delay in their order fulfillment. Now think about the result if these two services operated on independent platforms —this would perhaps require all kinds of manual intervention and correcting for errors, resulting in delays and, in the worst case, a loss of revenue. Do you still think it’s a good idea for application and data integration to work independently?
For more information on what to look for in an iPaaS, read Modernizing Data Management with Next-Gen iPaaS. Find more details on Informatica’s Cloud Application Integration and Cloud Data Integration services. And experience a 30-day free trial of Informatica’s Application Integration and Data Integration services—sign up now.
 Source: Gartner, Market Trends: The Impact of 3 Convergence Points on the Data Integration Tools Market” by Sharat Menon, 16 April 2020