Are You Ready For Data Science?

By Hervé Schnegg, Principal Data Scientist, The Telegraph

Hervé Schnegg, Principal Data Scientist, The Telegraph

For the last 10 years [1][2], companies have been complaining on how hard it is to hire data scientists. In the meantime, most data scientists keep changing jobs on a regular basis and invoke a lack of challenges as a reason for leaving. What is happening? My opinion is that companies are not hiring the right profiles at the right time and that they should start addressing a few issues before calling their favourite recruiters. Here are the main stages leading to a level of data maturity justifying the need for data scientists. At each stage, I describe the kind of professionals required to move to the next level.

Stage 1 - Business Preparation

First of all, companies need to assess how ready they are to start the journey that will transform them into data-driven businesses. Building a vision of what success looks like should be the starting point. Understanding the strengths, weaknesses, opportunities, and threats [3] of the environment in which they operate is still a valid approach. This initial analysis is what will feed into the definition of key performance indicators [4]. At that stage, working with experienced business consultants and business analysts is probably the best decision.

Further, another important assessment also has to be performed at the business level. The critical point here is to decide whether the organisation is ready to base its decisions on data analysis or even recommendations of data models [5]. For traditional businesses, used to rely on the sole experience of domain experts, this might be the hardest challenge. Many experienced managers will struggle to go against their instinct and trust decisions coming from teams of new data experts lacking a deep experience of their industry. Much careful communication is required to explain that change is required and that using data can help finding new paths leading to success. Such an exercise should definitely start before the first data scientists are hired.

Stage 2 - Technical Preparation

Technical questions needs to be addressed in parallel with those business considerations.

Is the organisation capturing the data required to start performing data analysis? Some legacy systems might have been designed to help running some aspects of the business (eg. sales, logistics…), but do they expose data? Having access to the data related to any business process is critical to the success of the data science effort. Once again, hiring data scientists too early might prove extremely unproductive and frustrating. At that stage, working with internal system developers - or vendors - is required.

The next question covers the processing and storage of the data. Once companies start capturing data, questions on how to make the data available for reporting and analysis have to be addressed. Traditionally, companies have built data warehouses for that purpose [6]. In our age of big data, data lakes tend to replace them [7]. Data scientists should be able to work with any chosen data store and the choice of the right architecture should be left in the hands of data architects and data engineers. The key point here is to have a centralised repository of all the important data.

Stage 3 - Data Democratisation

Governance considerations will arise from the collection and centralisation of data [8]. Managing the availability, usability, integrity, and security of the data is critical and has to be in the hands of a senior manager. Could it be that a chief data officer has to be hired before a data scientist [9]?

Implementing what has been described so far should allow any person in the business to have access to past data. Reports showing how business decisions influenced results should now be generated by data analysts and business intelligence developers. At this stage of democratisation of the data, more and more advanced questions should be brought to the data analysts and the need for more advanced analytics should be growing [10]. This is probably the best stage to start thinking about hiring data scientists.

Stage 4 - Data Maturity

The arrival of data scientists in an environment where data is available and where people starts understanding the kind of questions that analytics can answer is ideal. They should be involved in many projects and help assessing the potential of new ideas. The organisation should rapidly starts feeling the benefits of data-driven decisions and data scientists should be excited by the variety of projects brought to them. This situation should lead to an evolution where data products based on machine learning start to be developed. Machine learning engineers might be required at this stage to help with the release of more data products - some of them being branded AI [11]!

Following this process [Fig. 1] should ensure that organisations hire the right data professionals at the right time and avoid building teams of data scientists not fully engaged as they do not have access to the data they need and lack the management support allowing them to contribute building a data-driven organisation. Are you ready for the journey?