In today’s modern software development world, adoption of DevOps and CI/CD have become widespread, but their definitions are often confused and consequently they can be misused. In this post, I define each process and explain how Informatica Big Data Management (BDM) integrates with CI/CD and DevOps pipelines in one integrated environment.
Continuous Integration (CI), is a software development practice in which all developers merge code changes in a central repository multiple times a day. Continuous Delivery (CD), adds the practice of automating the entire software release process.
With CI, each change in code triggers an automated build-and-test sequence for the given project, providing feedback to the developer who made the change.
CD includes infrastructure provisioning and deployment, which may be manual and consist of multiple stages. What’s important is that all these processes are fully automated, with each run fully logged and visible to the entire team.
DevOps is a culture shift or a movement that encourages communication and collaboration to build better-quality software more quickly with more reliability.
DevOps organizations break down the barriers between Operations and Engineering by cross-training each team in the other’s skills. This approach leads to higher quality collaboration and more frequent communication.
For Agile methodology which focuses on collaboration, customer feedback, and rapid releases, DevOps plays a vital role in bringing development and operations teams together.
Today’s development according to Agile practices couldn’t be considered without CI. It became easier to make changes within the software development through infrequent version releasing as development and operations teams can collaborate easily with CI.
Informatica Big Data Management provides support to all the components in the CI/CD pipeline. It supports version control for versioning and use of the infacmd command line utility to automate the scripts for deploying. When you build a CI/CD pipeline, consider automating three different aspects of development: Versioning, Collaboration, and Integration & Delivery.
Informatica developers create and update objects like mappings, workflows, applications and data objects as part of their daily activities. When you have multiple developers working on the same objects, it is critical for developers to version control these objects.
In the recent releases, Big Data Management can integrate multiple version control applications like Git, SVN, Perforce, etc.
When you enable version control in Big Data Management, objects in the model repository service (MRS) are version controlled, enabling developers to check in and check out objects to the version control repository. When version control is enabled on MRS, developers can use the following functionality:
Big Data Management offers simple drag-and-drop functionality. It is easy for developers to build a complex mapping with a lot of transformations and mapping logic. As the mapping grows over time with multiple steps integrated, it is common for multiple users to work on the same mapping or workflow. With version control, multiple developers work on the same set of objects without any conflicts.
You can achieve the following goals for better collaboration:
Continuous delivery is an important aspect that allows users to automate the end-to-end pipeline for CI/CD. Using Big Data Management, users can create an archive file for the BDM application that can be deployed on the data integration service (DIS) in any environment. Big Data Management provides these functionalities for Continuous Delivery:
Objects in a Big Data Management development project are classified into design time objects and runtime objects. Design time objects are the objects that the developer builds in the Developer tool and stores in the MRS. The objects that are deployed from MRS to DIS are called the run-time objects. You can build, update, and view design-time objects on the Informatica Developer client. You can view and manage run-time objects on the Informatica Administrator tool.
Design-time objects consist of projects, folders, mappings, and workflows. Run-time objects consist of deployed applications, mappings, and workflows inside the applications.
To deploy, you select the objects to package them into an Application Archive File (.iar). Informatica provides command line utilities like infacmd which can be used to create the .iar files. These infacmd commands can be used in your automation scripts for DevOps.
Once you create the .iar file, you can deploy the .iar file to any DIS environments like Dev, QA, Prod, etc.
Below is the CI/CD Deployment best practice architecture for Big Data Management:
As explained in the deployment process, the DevOps flow for Big Data Management has two steps: build process flow and deployment process flow.
In the following build and deployment process flows, we use Jira as the ticketing system, Jenkins as the build system, and Git as the version control. You can build these flows with any available source control, ticketing, and build tools in the market.
In an Agile environment, developers are assigned Jira tickets for their daily activities. When the developer creates/updates an application, he/she updates the Jira ticket with relevant details like the domain name, model repository name, application name, and build number. The development or build manager can review the changes the developer made and approve the changes. Once the status of the Jira ticket is changed, Jenkins can identify this change on Jira and trigger a build. In the Jenkins build, we can have scripts to create a corresponding .iar file for the application the developer worked on. Jenkins can automatically pull the application name and MRS details from the Jira ticket.
A Jenkins script can also push this .iar file to version control.
When the application is ready to be deployed to the DIS, a developer can create a deployment request on Jira. In the Jira ticket, the developer can identify the application, DIS, and the build number to deploy against.
The deployment manager can review this request and approve this ticket on Jira. Once the status of Jira is changed to “Approved,” Jenkins can automatically trigger a job to pull the .iar file from version control and deploy the .iar to the DIS that the developer gave in the Jira ticket.
CI/CD and DevOps are critical for delivering faster results in an efficient way for any modern-day software development project. It is important to understand what each piece means in CI/CD and how DevOps can be applied to successfully implement CI/CD pipelines. Informatica Big Data Management provides the best of solutions for customers to be compliant to any CI/CD and DevOps tools available in the market. CI/CD and DevOps is a critical area that Informatica is investing heavily in. With the latest releases of Informatica Big Data Management, customers can create and deploy the most complex CI/CD pipelines with the various utilities that comes with the product.
Check out the following resources for more information on DevOps and CI/CD for Big Data Management.