Author: from Elastic Ken Exner
Today, we are thrilled to announce Search AI Lake and Elastic Cloud Serverless. Search AI Lake is a groundbreaking cloud-native architecture optimized for real-time applications, combining scalable storage capacity with low-latency queries and the powerful search and AI-related capabilities of Elasticsearch. This architecture powers the entirely new Elastic Cloud Serverless offering, eliminating all operational overhead so you can get started quickly and scale your workloads seamlessly.
The Next Era of Search
For over a decade, Elasticsearch has delivered fast, scalable solutions even for the most complex data. Elastic customers succeed because search is designed to deliver real-time insights from data without requiring explicitly defined schemas or predictable query patterns. While No-SQL databases and other solutions require more structured data, schemas, or queries, Elastic can quickly search all data by default. Search speed drives critical outcomes ranging from faster threat detection to improved operational efficiency and greater user engagement. This makes search the best solution for scenarios where data is messy and constantly changing, and where users do not know the exact attributes they want to query but still need to quickly search any content across their data in real time.
This is why Elasticsearch is so widely used for real-time analysis of both structured and unstructured data, whether for log analytics, SIEM, or a wide range of search AI-powered applications. This is also why search is the foundational layer for generative AI experiences. Search is critical for efficiently encoding, retrieving, and synthesizing large volumes of data to generate accurate, contextually appropriate responses with large language and other models. We recognized that as AI and real-time workloads scale in production, a new architecture is needed that can deliver low-latency query performance across all data, without sacrificing scalability, relevance, or cost-effectiveness.
We have seen the emergence and evolution of data lakes, which attempt to address the problem of rapidly growing data scale by separating compute and storage. However, by design, these architectures are optimized for storage rather than performance. For example, object storage inherently prioritizes scalability over speed, leading to unacceptable latency for interactive queries. This makes data lakes impractical for real-time applications that require low-latency queries and access to all data, regardless of size or complexity.
A low-latency, AI-powered future requires a new lake architecture.
No Compromises: Search AI Lake, a New Architecture for Real-Time, Low-Latency Applications
Today, we are excited to launch the first Search AI Lake of its kind. It is a cloud-native architecture optimized for real-time, low-latency applications including search, Retrieval-Augmented Generation (RAG), observability, and security. It brings together the scalable storage capacity of data lakes and the low-latency queries, powerful search, and AI-related capabilities of Elasticsearch. This capability is available today in technical preview.
Search AI Lake creates new opportunities to quickly and interactively search almost unlimited volumes of data on demand at efficient storage costs. For search applications, this enables RAG to leverage large datasets seamlessly and cost-effectively. For security, this means better threat protection by enabling easy access to massive volumes of security data for investigations and threat hunting. Security teams can enhance anomaly detection with immediate, unrestricted access to previously siloed data, significantly improving their security posture. SREs gain deeper solution insights — they can search years of data on application performance without rehydration (rehydrate refers to reading and mounting data from a snapshot), and access high-resolution datasets for predictive analysis, trend analysis, and proactive detection. Search AI Lake delivers many unique advantages:
- Unlimited scale, decoupled compute and storage: Storage and compute are fully decoupled to enable easy scalability and reliability using native cloud storage, while our dynamic caching supports high throughput, frequent updates, and interactive queries over large data volumes. Leveraging cloud-native object storage delivers high data durability while balancing storage costs at any scale. This eliminates the need to replicate indexing operations across multiple servers, reducing indexing costs and data duplication.
- Real-time, low latency: Even when data is stored securely on object storage, multiple enhancements maintain excellent query performance. This includes the introduction of segment-level query parallelization, which reduces latency by enabling faster data retrieval and allowing more requests to be processed faster. Better reuse is achieved through more efficient caching and optimized use of the Lucene index format.
- Independently scalable indexing and querying: By separating indexing and search at a low level, the platform can scale independently and automatically to meet a wide range of workload demands.
- Generative AI optimization: Native inference and vector search: Use RAG and proprietary data to customize generative AI experiences for your business. Leverage a native suite of powerful AI relevance, retrieval, and re-ranking capabilities, including a native vector database fully integrated into Lucene, open inference APIs, semantic search, and first- and third-party transformer models that work seamlessly with a full range of search capabilities.
- Powerful querying and analytics: It includes Elasticsearch's powerful query language ES|QL, which enables you to transform, enrich, and streamline investigations with fast concurrent processing, regardless of data source and structure. It also includes full support for accurate, efficient full-text search and time series analysis to identify patterns in geospatial analysis. Data indexing leverages "schema on write" (schema defined at write time) for scale and speed, and "schema on read" (schema defined at read time) to deliver flexibility and faster time to value.
- Native machine learning: Build, deploy, and optimize machine learning directly across all your data for superior predictive performance. For security analysts, this means pre-built threat detection rules can easily run even across years of historical information. Similarly, you can run unsupervised models to perform near-real-time anomaly detection on data spanning much longer time periods than other SIEM platforms.
- Truly distributed: Cross-region, multi-cloud, or hybrid: Query data across regions and data centers from a single interface. Cross-Cluster Search (CCS) eliminates the requirement for centralization or synchronization. This means that within seconds after ingestion, any data format is standardized, indexed, and optimized for extremely fast queries and analytics. This also reduces data transfer and storage costs. Full CCS support will be available on Search AI Lake in the near future.
Introducing Elastic Cloud Serverless — Get Started and Scale Instantly
Elastic Cloud Serverless is built on Search AI Lake, delivering worry-free management, fast onboarding, and an optimized product experience — all tailored to leverage Search AI Lake's impressive speed and scale. In technical preview, serverless projects are fully simplified to eliminate operational overhead and handle scaling and management automatically. Everything from monitoring and backups to configuration and scaling is managed by Elastic. You never have to worry about underlying clusters, nodes, versions, or scaling — just bring your data and start using any Elastic solution.
Serverless complements existing cloud deployments, giving you a choice: use serverless for greater simplicity, or use existing Elastic Cloud Hosted for greater control. Currently, serverless projects run on AWS (us-east-1) by default, with support for additional cloud providers and regions coming in the near future. Serverless projects are available today in technical preview.
Serverless projects use a new solution-specific pricing model. Simplified pricing makes it easy to understand and forecast your spending for search, observability, or security. Credits are unified across deployment options, giving you flexibility to pay for and use either Elastic Cloud Hosted or Elastic Cloud Serverless.
With the new product experience and underlying Search AI Lake architecture, serverless projects make getting started easier than ever.
- Instant provisioning: Quickly spin up a new, fully configured serverless project.
- Project-based: Use the new workflow to easily create projects optimized for the unique requirements of each use case, from vector search and APM to endpoint security and more.
- Guided onboarding: Skip the learning curve with dedicated steps that guide you to faster results with in-product resources and tools.
- Worry-free operations: Free your team from operational responsibilities — no need to manage backend infrastructure, perform capacity planning, carry out upgrades, or scale data.
- Unlimited scaling: Serverless allows your workloads to scale seamlessly by automatically responding and adapting to changes in demand, minimizing latency and ensuring the fastest possible response times.
Simplified from the Start: Streamlined Product Experience, Packaging, and Pricing
Serverless delivers a simplified product experience and guided onboarding for search, observability, and security, speeding up time to results and optimizing for each use case. New simplified pricing and packaging make it easy to manage your spending and get the full benefit of everything Elastic has to offer. This includes a range of new pricing and packaging options for Elasticsearch, Elastic Observability, and Elastic Security.
Elasticsearch Serverless enables developers to quickly deliver impactful AI search experiences without worrying about speed or scale. Hardware-optimized projects for general search or vector search and time series are coming soon. Build search applications quickly with a streamlined developer experience that includes extensive inline documentation and code examples to guide your search project. Spend less time managing infrastructure and more time developing — no need to configure or manage clusters, shards, versions, or scaling. Accelerate development of generative AI experiences by accessing Elastic's latest AI capabilities, including vector search, Elastic Learned Sparse EncodeR (ELSER), semantic search, machine learning (ML), and AI model integration and management.
Search AI Lake puts you in control to balance search performance and storage cost-effectively. The separation of compute and storage, as well as indexing and querying, allows any workload to scale independently quickly and reliably, without impacting performance. Even when using extensive indexing-time features for better relevance, search performance is not affected.
Pricing and packaging: Elasticsearch Serverless introduces a single product tier that provides access to all search features and building blocks for programmatically developing search applications. Pricing is simplified and metered based on the compute resources used for ingestion, search, and machine learning, as well as data retention and data egress (outbound data transfer). For more details, visit the Elasticsearch Serverless pricing page.
Elastic Observability Serverless delivers a worry-free experience with all the benefits of full-stack observability, without the overhead of managing the stack or scaling dynamic workloads. Streamlined workflows and guided onboarding minimize time to insight and make it easy to pivot between signals without losing critical context. Additionally, with over 350 integrations, managed ingestion services, and an OpenTelemetry-first approach, ingesting observability data into Elastic is easier than ever.
Search AI Lake enables faster analytics than ever before with extremely fast queries and machine learning jobs, delivering insights in minutes even for petabyte-scale datasets. With unmatched speed and scale, you can now analyze all your business and operational data to proactively detect issues, accelerate problem resolution, and deliver business outcomes.
Pricing and packaging: Elastic Observability Serverless introduces a single product tier that provides access to all Observability capabilities. Pricing is simple: you pay for what you ingest and retain. Additional options include synthetic monitoring browser tests priced per test run, and lightweight tests priced per test per region. For more details, visit the Elastic Observability Serverless pricing page.
Elastic Security Serverless provides a new cloud deployment option for security analysts for security analytics and SIEM use cases. This entirely new, fully managed cloud offering delivers a curated security solution that you can start using quickly. Deploying Elastic Security this way eliminates the overhead of managing cloud and SIEM infrastructure, and allows security teams to focus on protecting, investigating, and responding to threats across your organization.
Security analysts can leverage Search AI Lake to seamlessly analyze all security-related data, including months or even years of historical data, to deliver insights in minutes. Search AI Lake supports threat hunting, automated detection, and AI-powered security analytics capabilities, including attack discovery and an AI assistant.
Pricing and packaging: Elastic Security Serverless offers two carefully curated tiers of functionality for common security operations.
- Security Analytics Essentials includes everything required to run a traditional SIEM for most organizations.
- Security Analytics Complete adds advanced security analytics and AI-powered capabilities that many organizations require when modernizing or replacing legacy SIEM systems.
- Optional add-ons: Endpoint and cloud protection are available as optional add-ons. Selecting one or both options adds additional protection capabilities beyond the security analytics/SIEM functionality included with all Elastic Security Serverless projects.
Serverless pricing for Elastic Security is simple: you pay for the data you ingest and retain. For more details, visit the Elastic Security Serverless pricing page.
Explore the Full Power of Search and AI
The future of low-latency applications is here, and it does not require compromises on speed, scale, or cost. Elastic invites security analysts, SREs, and developers to experience serverless and Search AI Lake to unlock new opportunities with your data. Learn more about the possibilities of serverless, or get started today with a free trial in technical preview.
The release and timing of any features or functionality described in this article remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.
In this blog post, we may use or reference third-party generative AI tools that are owned and operated by their respective owners. Elastic has no control over third-party tools, and we accept no responsibility for their content, operation, or use, nor for any loss or damage that may result from your use of such tools. Please exercise caution when using AI tools to process personal, sensitive, or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that the information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tool prior to using it.
Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine, and their respective marks are trademarks, logos, or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos, or registered trademarks of their respective owners.
Original article: Announcing Search AI Lake and Elastic Cloud Serverless to scale low latency search | Elastic Blog
This is a discussion topic separated from the original thread at https://juejin.cn/post/7369117760461439014