Perficient is at the Hadoop Summit in San Jose, CA and we’re tracking the best of the conference. Here’s the top 5 lessons from day 1:
- Apache Atlas for managing your business catalog is almost ready for prime time! It is not, however, ready to be a full fledged Records Management solution (no policy management, no proper disposition automation, although expiry is there… stay tuned).
- Microsoft ‘s Big Data and intelligence suite centered around HDInsight, MS R Server (formerly Revolution) and Power BI has a name, “Cortana Intelligence Suite” and they’ve got a lot to say about the future of the Connected Data Platform. Hortonworks is planning to prioritize HDInsight go-forward as their Premier Cloud Solution.
- Assemblies are the future. The next grand evolution in cluster management is coming soon (how soon is as yet unknown). We’ll see the marriage of Slider (long running apps on YARN) and Docker (containerized apps). This will enable the deployment of end-to-end data apps with only a few clicks into a YARN+HDFS cluster. The demo showed Credit Fraud Detection, Log Search and Metron. You will be able to deploy all parts of a logical App (eg. for Log Search, that’s Zookeeper, Solr and a UI) with ease through an Ambari integrated Assembly Store.
- Metron is going to be big. With early adopters seeing ROI for security threat detection less than two months from go-live. This is one of the great use-cases for the whole ecosystem, Storm, Spark+ML, Solr/Elastic, Hive, HBase and NiFi. I’m hearing stories about 6-8 months to implement, 3-6 weeks to see results, it’s truly remarkable and coming soon to the mainstream.
- Business Value. The ecosystem is trying to pivot, as is the conference, to tell the story of big data in terms of business value to business users. They will be the drivers of new innovation and items 1, 3 and 4 on my list are all signs of this change.
The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
We’ll be following up on “Big Data and You (the enterprise IT Leader)” with more thoughts on Data Governance, Records Management and Metron. I’ll be back tomorrow with more from the conference.
BONUS: Check this out… Big Data is going to try and cure cancer through genomic research!
