Big Data is on everyone’s mind these days. Creating an analytical environment involving Big Data technologies is exciting and complex. New technology, new ways of looking at the data which is otherwise remained dark or not available. The exciting part of implementing the Big Data solution is to make it a production ready solution.
Once the enterprise comes to rely on the solution, dealing with typical production issues is a must. Expanding the data lakes and creating multiple applications accessing, changing and deploying new statistical learning solutions can hit the overall platform performance. In the end-user experience and trust will become an issue if the environment is not managed properly. Models which used to run in minutes may turn into hours and days based on the data changes and algorithm changes deployed. 
Having the right DevOps process framework is important to the success of Big Data solutions.
In many organizations the Data Scientist reports to the business and not to IT. Knowing the business and technological requirements and setting up the DevOps process is key to make the solutions production ready.
Key DevOps Measures for Big Data environment:
- Data acquisition performance (ingestion to creating a useful data set)
- Model execution performance (Analytics creation)
- Modeling platform / Tool performance
- Software change impacts (upgrades and patches)
- Development to Production – Deployment Performance (Application changes)
- Service SLA Performance (incidents, outages)
- Security robustness / compliance
One of the top key issue is Big Data security. How secured is the data and who has the access and the oversight of the data? Putting together a governance framework to manage the data is vital for the overall health and compliance of the Big Data solutions. Big Data is just getting the traction and much of best practices for Big Data DevOps scenarios yet to mature.
