Architecting for big data

The Internet of Things produces data at rates beyond human comprehension. It may also overwhelm your data environment.  

“The winners in the IoT arena are going to be the ones who provide the most user- friendly data controls with a tight lockdown on who gets to see what.”

—Brian Anderson, general manager, Notionovus

Some data architects have already confronted the sheer scale of data produced by the Internet of Things (IoT) through their work with private networks of intelligent devices. Integration expert Brian Anderson is general manager at Notionovus, a software and process development firm. He started as an embedded systems programmer at Caterpillar Inc. At that time, he built applications for a network of robotic vehicles on the factory floor.

During Anderson’s 25-year career at Caterpillar, he also served as engineer supervisor, manufacturing engineer, and 6 Sigma Black Belt. In other words, he was dealing with the data challenges of the IoT before the term became popular.

Potential at Work asked Anderson what architects can do to ready their existing environments for the massive scale and myriad integrations inherent to the IoT.

Do you have advice for architects who are trying to lay the foundation for a structured approach to data integration in the IoT?
Anderson: An integration professional needs to look at four key process indicators (KPIs): speed, reliability, cost, and security. These are the four customer-facing metrics that crop up when people talk about good or bad integration experiences. But when we discuss the IoT, the one key indicator to focus on is security. Devices are going to have owners, and owners are going to have privacy concerns.

The winners in the IoT arena are going to be the ones who provide the most user-friendly data controls with a tight lockdown on who gets to see what. The failures in the IoT will be with companies that surreptitiously use the data that their devices generate to essentially spy on their users. Also, look for tools that capitalize on reuse and leverage libraries of existing code. This reduces the need for highly specialized personnel.

The architects’ job should involve more big-picture thinking and anticipation of future integration needs so they can build flexibility into their planning models. For example, when your data architect is busy with the data integration professional on a particular problem with a particular interface, that’s a sign that you are wasting money rather than planning for the future.

How do you apply lean methodologies to something as unstructured as the IoT?
Anderson: Stick with tools that help build transparent integrations. End-to-end visibility into a message's value chain is crucial for measuring its KPIs. If continuous improvement is your goal, you can't work with black box systems. Avoid the development of redundant applications and reinvention of established connection points.

Patches, updates, and bug fixes are non-customer and value added, so concentrate on minimizing the complexity of released code. Since bad data is so expensive, building data validation into software is value added. Validation is only non-value added when systems and their users produce error-free data.

How should data architects adapt their enterprise data management strategy for the IoT?
Anderson: An enterprise data management strategy needs to revisit the archiving, security, and use cases for future and existing data flows. All of those increased data flows from the IoT are going to have a huge impact on your current operations. You need to put an archiving and data retention strategy in place to maximize the utility of this massively increased data flow while minimizing the data management system footprint.


Related content


Continuous data delivery is not rocket science--it's common sense

Continuous data delivery is not rocket science—it’s common sense.


Amr Awadallah: Win big with big data

Use the vision of a big data pioneer to understand why you need to “walk before you run” to build the foundation of an enterprise data hub.


Can you architect your way from sluggishness to speed?

Infrastructure complexity and poor coordination across teams are slowing some IT functions to a crawl. Enterprise architects should consider Lean Integration as a framework to architect for speed and agility while minimizing waste.