Continuous data delivery is not rocket science—it’s common sense

Keep pace with business user demand with these three steps, adapted from the Manifesto for Agile Software Development.

“When the business person and developer can talk the same language, they develop the right things and reduce the number of iterations between IT and the business.”

—David Lyle, vice president of research & development in the Office of the CTO, at Informatica

Architecting for continuous delivery means business users can have visibility into their data faster than ever before. This can happen without spending weeks to validate requirements, prototype a solution, and test it with users. You don't have time for that, and neither does the business. You need to match the speed of information delivery with the pace of the always-on business.

“With the continuous delivery movement, we're talking about hours to validate requirements and build a prototype. We're not passing the ball to the business, emailing something, and hoping for a response—we are jointly holding the ball at the same time,” said David Lyle, vice president of research & development in the Office of the CTO, at Informatica, during the “Great Data by Design I: Agile Data Integration and Holistic Data Stewardship” Informatica Talk.

With these steps, adapted from the Manifesto for Agile Software Development, you can accelerate the flow of information throughout your organization:

  1. Provide the right tools to support an agile, iterative approach to data integration. 
    Business users are waking up to the value of data as an asset, and many want to participate in the data integration development process. Provide tools so business users can collaborate naturally with your data integration team, enabling IT to respond to changes from the business side more nimbly. 

    “When the business person and developer can talk the same language, they develop the right things and reduce the number of iterations between IT and the business,” said Lyle.
  2. Insist that data quality monitoring is central to the data integration development process. 
    Don't let developers wait until integration to discover issues with data quality. Establish these policies to ensure that data quality problems don't become an integration bottleneck:
    • Monitor early. Plan for data quality monitoring as part of the first stage of investigating data. It is less expensive to fix data quality problems or develop business rules to filter out anomalies at this stage.
    • Monitor everywhere. Data quality monitoring must be pervasive across the data integration environment in order to catch all errors. Don’t forget sources outside your company, such as social and machine data.
    • Continue monitoring. Create a policy for ongoing data quality monitoring.
    “You might think the data is of good quality at a particular stage in its life cycle, but we have found that the quality of data deteriorates if it's not taken care of on a daily basis,” said John Poonen, director of product engineering at Quintiles, during the “Great Data by Design II: How to Get Started with Next-Gen Data Integration” Informatica Talk.
  3. Make sure each new project creates reusable software components. 
    Software reuse is a core tenet of agile iterative development, and it should be used to speed your project team's data integration efforts. Take data profiling, for example. You might have results from data profiling from six months ago. Another team member could use that as a starting point to gather requirements for a new project. 

    “Now you've also built a logical asset about how that data comes together, which now you can use again and again,” said Jared Hillam during the Informatica Talk. Hillam, author of the Intricity blog and an enterprise information management practice director, added, “You can use it for persistence or a data warehouse in the future.”
  4. An enterprise data architecture that adopts these three steps will achieve an integrated vision of the data. More importantly, these steps help create a shared understanding of the information the data represents. This allows your team to not only show faster results, but more relevant results that accurately reflect the ever-evolving needs of the business.

    You can read the manifesto in its entirety, as well as its “Twelve Principles,” in 40 different languages at AgileAlliance.org.

Related content


Amr Awadallah: Win big with big data

Use the vision of a big data pioneer to understand why you need to “walk before you run” to build the foundation of an enterprise data hub.


Architecting for big data

The Internet of Things produces data at rates beyond human comprehension. It may also overwhelm your data environment.


Embrace change in the interest of integration standards

Fighting change can be a waste of time and energy. Instead, elevate your role in the organization by focusing on fundamental business problems.