Accelerate Data Analytics with DataOps

By Harvinder Atwal, Head of Data Strategy and Advanced Analytics, MoneySuperMarket.com

Harvinder Atwal, Head of Data Strategy and Advanced Analytics, MoneySuperMarket.com

It is an amazing time to be working with data. Exponential growth in data collection, advances in artificial intelligence (AI) algorithms, an explosion in software libraries for working with large quantities and varieties of data, and improvements in big data technologies for storing and processing data are ushering in a digital transformation. At the heart of the revolution is a data analytics ecosystem of skills to exploit new opportunities that include data science, data engineering and the broader fields of data and business analytics.

The data-driven digital revolution is beginning to transform every aspect of our lives from personalised health care or financial management, to how we interface with machines, whether it is a self-driving car or virtual assistants. However, we are also only at the start of the journey to understand the best processes required to deliver desired outcomes from raw data. Modern data analytics is still in the comparable transition phase between bespoke handcrafted production and mechanised automation that manufacturing confronted during the early nineteenth century. Unfortunately, the lack of maturity means there is plenty of evidence that returns on substantial investment in data capability is not uniformly positive. According to Forrester Research, only 22% of companies see a significant return from data analytics expenditures.

Despite the high failure rate, the prescriptions and discussions remain the same. Big data technology vendors promote the latest storage and processing solutions, but adding new technology to legacy processes results in much more expensive legacy processes. Most data scientists talk about how to create machine learning or AI models, but not many speak about getting them off laptops into scalable production workflows where they can affect customers. Solutions vendors talk about the latest platforms but not how to overcome organisational barriers to using data effectively.

Most approaches for managing and consuming data were developed in a world where data was scarce; computer resource was expensive, storage constrained, access restricted, the opportunity to test and learn minimally, and digital automation non-existent.

We now live a world with fewer constraints and expect data-driven decisions to be made everywhere and in smaller and smaller increments from auto-enhancement in a photo-editing mobile app to sentence autocomplete in webmail. It is also now cheaper than ever before to generate empirical data to learn and reduce the uncertainty of decision-making. Nevertheless, most organisations still approach data analytics as a series of large bespoke, waterfall research projects with artificial constraints in the provision of data, software, and systems. Poor returns on data analytics investment are often a result of applying 20th-century thinking to 21st-century opportunities.

Manufacturing and software development have also faced similar challenges as data analytics. To add value as quickly and efficiently as possible, yet deal with high levels of complexity and uncertainty, they created revolutionary approaches, Lean thinking, Agile practices, and DevOps culture. DataOps is an emerging methodology that applies these approaches to the unique needs of data analytics:

Poor returns on data analytics investment are often a result of applying 20th-century thinking to 21st-century opportunities

• Agile collaboration breaks down data and organisational silos to ensure we work on the "right things" that add value for the "right people."

• Lean manufacturing like focus eliminates waste and bottlenecks, continuously improves speed and quality, monitors data flows and makes it cheap to put data into the hands of consumers.

• DevOps mindset allows teams to move at rapid speed using highly optimised self-service tools and automated processes across the whole data lifecycle to continuously deliver what customers need.

The name DataOps is a portmanteau of Data and Operations and was first introduced by Lenny Liebmann in a 2014 blog post. Since then, interest has snowballed, and DataOps made Gartner's "Hype Cycle" for Data Management in 2018. The goal of DataOps is to turn unprocessed data into a valuable data product for customers through a rapid, scalable and repeatable process. Numerous times a day we are customers of a data product whether it is a Google Maps route request, a Netflix product recommendation, or even a business report. It is not a one-off project; a data product is in continuous production with constant monitoring, iteration based on experimentation and feedback leads to improvement, it is reproducible, and it brings customer and organisational benefit.

DataOps brings together self-contained teams with data analytics, data science, data engineering, DevOps skills and line of business expertise in close collaboration. The DataOps team view the analytical pipelines between raw data and data product as a lean manufacturing line, and they orchestrate the data, code, and environments from start to end. Self-service access to tools and data allows DataOps teams to deliver quickly but requires trust; trust in data and trust in the users of data. Trust in data occurs from constant monitoring and testing for quality as data moves through pipelines. Trust in users of data comes through robust data governance and data management practices that respect agility.

DataOps practices encourage organisations to take a collaborative end-to-end view of their data flows to increase data analytics speed to market and value. With DataOps, organizations remove many of the challenges faced in delivering modern data analytics allowing them to thrive in the data-driven digital revolution.