Regulated industries, such as financial and healthcare companies, often need to make hard choices when it comes to balancing innovation and compliance. Most technology companies are focused on cloud-first, if not entirely cloud-native, offerings, particularly in the search and data space. I was recently working with a large financial services company that wanted to consolidate and modernize their enterprise search offering.
The Enterprise Data Platform offered Elastic as a shared service, but access to features and upgrades was limited in order to ensure the highest level of stability and resilience for all of the users. Also, since some data was sensitive, there were no plans to move to the cloud. As expected, this led to a growth of shadow IT among the more sophisticated user base, who needed more advanced functionality but were less able to manage licensing, security, and disaster recovery than the formal IT offering. We ended up installing a hybrid Elastic Cloud Enterprise solution.
Elastic Cloud Enterprise
Elastic Cloud Enterprise (ECE) is the same product that underpins the popular Elastic Cloud hosted service, providing you with the flexibility to install it on hardware and in an environment of your choice. ECE offers flexibility and convenience, as all services are containerized using Docker. This means that it can be installed on public/private clouds, virtual machines, or even directly onto dedicated hardware. The key feature for ECE is the concept of deployment.
A deployment in ECE is a separate instance of an Elasticsearch cluster with its own configuration settings, resources, and data. This means that multiple deployments can coexist within the same Elastic Cloud Enterprise environment, each with their own unique set of settings, resources, and data. For example, an organization might create separate deployments for different business units, projects, or applications, each with its own dedicated resources and data. Additionally, each deployment can be independently managed and scaled up or down based on its own needs. For example, one deployment might require more nodes or storage capacity than another, and these resources can be allocated or adjusted as needed without affecting the other deployments.
Solutions in Elastic Cloud Enterprise
The Future of Big Data
With some guidance, you can craft a data platform that is right for your organization’s needs and gets the most return from your data capital.
Since each deployment can be managed and maintained as a distinct entity within the ECE cluster, and since ECE can be installed on any platform, we were able to address the core issues that were limiting the enterprise elastic offering and pushing so many departments into rolling their own solution. Also, we were able to address their cloud restrictions. At a high level, if your data contained PII or other sensitive information, it remained on premise in dedicated VMWare pods. Otherwise, your search indices could be stored on the cloud at a substantially lesser cost. While it’s not always the case that cloud offerings are less expensive in the long run, the way this organization’s infrastructure was tiered, the dedicated VMWare pod structure’s memory and storage options were not ideal for Elasticsearch loads.
We were able to provide the users with the search environment they needed through ECE’s concept of deployments. We were able to scale this offering because of the unified administrative functionality. No matter how many deployments or whether you are using all on-premise, hybrid, and/or multi-cloud, there is one administration interface. This means that you can enforce a single, coherent operating model using a single UI/API interface to maintain consistent application of operations management, security, disaster recovery, and monitoring.
Operations Management
ECE provides a centralized management interface simplifying operations, ensuring consistency, and reducing the risk of errors. ECE supports integration with popular DevOps and collaboration tools such as Ansible, Terraform, and GitLab, enabling teams to manage and deploy their infrastructure and applications through familiar workflows and processes. If you have a choice, using the official Ansible playbooks for installation and provisioning is the easiest method. ECE provides built-in notification and alerting capabilities, which can be configured to trigger on events such as cluster status changes, resource usage thresholds, and system updates. These notifications can be sent via email, Slack, or other popular communication tools.
Security
The ECE operations model provides several features to help organizations comply with regulatory requirements. Elastic is a large enterprise and they have all of the alphabet soup regulations and compliance certificates covered: Payment Card Industry Data Security Standard (PCI DSS), General Data Protection Regulation (GDPR), SOC 2, ISO 27001, and HIPAA. Standard enterprise authentication and authorization features including role-based access control, multi-factor authentication, and integration with LDAP, Active Directory, and kerberos are available.
Communication among the ECE components uses Transport Layer Security (TLS) encryption for encryption in transit but you need to bring your own encryption-at-rest solution. If you are using the cloud, you can configure secure network connections through Virtual Private Clouds (VPCs). ECE simplifies security and safety management with customizable policies that allow organizations to determine and enforce their own standards across the board. These guidelines include limits on accessing certain elements, the requirement of multiple authentication measures, plus confirmation checks to verify all settings adhere to regulatory rules. While we did not use this feature, there is a data residency options that enable customers to choose where their data is stored and processed.
Availability
ECE provides features such as automatic failover and load balancing, which can help ensure high availability and minimize downtime. You need to provide your own load-balancing solution. It also supports data replication across multiple nodes and datacenters, which can help achieve redundancy and minimize the impact of a single node or datacenter failure. Cross-cluster replication (CCR) ensures that you can efficiently replicate indices from one remote cluster to an additional cluster regardless of their geographical position. Snapshots offer an emergency backup in the event of failure if not enough availability zones have been put into use to ensure top-level uptime, as well as providing a safeguard against potential accidental deletion. You can use any cloud storage offering or, if on-premise, an object store such as Miro.
Monitoring
ECE offers a suite of real-time monitoring metrics, including CPU consumption, memory usage, disk utilization, and other key performance indicators that are compiled into actionable summaries. By leveraging ECE’s alerting technology, organizations can easily customize threshold values for any key performance indicators and be immediately notified when these thresholds are breached. This helps detect potential issues at the earliest stage possible, enabling corrective action. Not surprisingly, it’s very easy to analyze logs, including application logs and system logs, to help identify the root cause of issues. Audit logging capabilities include tracking and analyzing all administrative actions taken on the system, including user login attempts, cluster creation, and configuration changes, and user role changes.
Summary
There are challenges in building robust solutions for regulated clients while maintaining strict compliance with internal and external regulations, standards, and best practices. By leveraging services as Docker containers, Elastic Cloud Enterprise gave us solutions where we could create isolated deployments within a cluster targeting specific department use cases. We were able to offer all of the features and functionality power users were getting with their isolated elastic installations while maintaining the strict governance and controls that were in place in their enterprise offering.
