Why and How to Migrate from Solr to Amazon OpenSearch Service in 2024

This guide is for organizations currently using Solr and considering a migration to Amazon OpenSearch Service. We will explore the benefits of migrating, the process, and the key challenges, offering insights from real-world migrations to help you make an informed decision.

Introduction: The Evolution of Enterprise Search

In today’s fast-paced digital landscape, having a scalable and easily manageable search infrastructure is essential for smooth user experiences and efficient operations. Apache Solr and Amazon OpenSearch Service are two robust platforms offering advanced search capabilities.

Though Apache Solr has been a trusted solution for years, there is growing interest in cloud-native platforms like Amazon OpenSearch Service.

This shift reflects a broader demand for managed services that simplify operations, making OpenSearch an attractive choice for organizations looking to modernize their search infrastructure.

This guide is for organizations currently using Solr and considering a migration to Amazon OpenSearch Service. We will explore the benefits of migrating, the process, and the key challenges, offering insights from real-world migrations to help you make an informed decision.

Why Migrate?

Migrating from Solr to OpenSearch offers several compelling benefits, especially for AWS-centric organizations. The advantages of moving to Amazon OpenSearch Service include:

• Simplified Management: OpenSearch is a fully managed service, reducing the burden of maintaining your search infrastructure.
• Cloud-Native Scalability: OpenSearch’s integration with other AWS services allows for smooth scaling and performance optimization without additional effort.
• Advanced Features: OpenSearch provides out-of-the-box solutions such as ML integrations, anomaly detection, real-time analytics, and powerful dashboards for intuitive data visualization—features that extend beyond the capabilities of Solr.
• Security and Compliance: Amazon OpenSearch Service integrates with AWS Identity and Access Management (IAM) and offers encryption at rest and in transit, among other security features.

Migration Process: From Solr to Amazon OpenSearch

1. Assess Your Priorities, Train Your Team, and Choose a Migration Strategy

Before diving into technical details, it’s crucial to define your migration priorities and ensure your team is prepared for the transition to OpenSearch. Whether you’re opting for a lift-and-shift migration or planning a modernization strategy, it’s essential that your team understands how to use and manage OpenSearch effectively.

Start by organizing training sessions or workshops to familiarize your team with OpenSearch’s features, query language, and management tools. This will help ensure a smoother transition and enable your team to take full advantage of OpenSearch’s capabilities from the outset. tecRacer offers tailored workshops that can help your team get up to speed quickly.

Once your team is prepared, you can decide between:

• Minimal Downtime Migration: Opt for an active-active setup where both Solr and OpenSearch run in parallel, ensuring a smooth cutover with zero downtime.
• Downtime-Tolerant Migration: If your application can tolerate some downtime, you can simplify and expedite the migration process.

Additionally, for cases where Solr isn’t the system of record, consider rebuilding indices directly in OpenSearch. This gives you the opportunity to redesign schemas, mappings, and search functionalities, allowing for a cleaner, modern implementation.

AWS recommends modernization for long-term scalability and flexibility. From our experience, modernization can be achieved rapidly, with most of the complexity typically residing in the application layer rather than the search engine.

2. Cluster Sizing and Cost Estimation

Once you’ve committed to modernization and your team is trained, the next step is to right-size your OpenSearch cluster. Start by assessing your current Solr setup in terms of performance and cost, then project these needs into OpenSearch. Consider how scaling requirements will affect performance and costs in a cloud-native environment.

After this assessment, plan a Proof of Concept (PoC) after completing the schema mapping and key configurations (discussed in step 5). This PoC will help validate your cluster sizing and performance estimates, ensuring you’re working with the right setup before the actual migration.

3. Core Functionality and Future Needs

Review the core functionalities of your existing search infrastructure. Identify which features are critical to your business and explore ways OpenSearch’s advanced capabilities can improve your search infrastructure. For example, OpenSearch offers vector search, semantic search, hybrid search, and more, which could open new opportunities for your organization.

Also, plan for future scaling and feature needs. tecRacer can offer workshops and training to help your team fully leverage OpenSearch’s potential.

At this point, focus on planning and be prepared to conduct the following tests after the initial setup:

• Relevance and Accuracy: Ensuring that the migrated queries produce high-quality search results.
• Search Latency: Measuring response times under typical and peak usage conditions.
• User Experience (UX): Validating that the search results align with user expectations in terms of speed and relevance.

4. Proof of Concept (PoC), Sizing Decisions, and Final Testing

With your migration strategy in place, you can now perform a Proof of Concept (PoC). During the PoC, test different cluster configurations and query behaviors to ensure they align with your performance needs and cost projections. This phase allows you to fine-tune your cluster size and performance settings based on real-world workloads.

Key considerations for the PoC:

• Performance Testing: Simulate typical and peak traffic scenarios to ensure your OpenSearch cluster scales efficiently and maintains the required performance levels.
• Data Quality and UX Testing: Evaluate the quality of search results, ensuring accuracy and relevance.
• Cost Efficiency: Ensure your cluster sizing and performance optimizations meet both operational and cost requirements, factoring in future growth.

Based on the PoC results, finalize your decision on whether the cluster is sized appropriately and whether OpenSearch can handle your workload effectively. If all criteria are met, you can confidently move forward with the migration.

5. Execution: Schema Mapping and Best Practices

Once you have the correct sizing from the PoC, the next step is to begin the actual migration process, starting with schema mapping. OpenSearch uses a more flexible, dynamic mapping system than Solr, allowing you to optimize field types and indexing behavior.

During this phase, focus on:

• Schema-to-Mapping Translation: Convert your Solr schema into OpenSearch mappings, leveraging OpenSearch’s flexible field definitions.
• Analyzers and Normalizers: Choose and configure the appropriate analyzers and normalizers for your data to ensure proper indexing and search functionality.
• Index and Shard Management: Plan the number of indices and shards carefully to optimize performance and resource allocation.

Thoroughly test your mapping and indexing configurations to ensure they align with your business needs before migrating production data.

6. Security and Compliance Best Practices

As you set up the OpenSearch environment, make security a priority. OpenSearch integrates seamlessly with AWS Identity and Access Management (IAM), allowing granular permission control. Additionally, you can secure your cluster within a VPC and monitor it with Amazon CloudWatch.

Key steps include:

• IAM Roles and Permissions: Implement the appropriate permissions for your OpenSearch cluster.
• Encryption: Enable encryption for data both at rest and in transit.
• Monitoring: Set up CloudWatch to monitor security and performance metrics.

As security configurations can vary between Solr and OpenSearch, thoroughly test your security settings to ensure compliance before migrating live data.

7. Final Migration and Continuous Operations Setup

With the PoC validated, schema mappings tested, and security in place, it’s time for the actual migration. Depending on your chosen strategy (active-active or downtime-tolerant), you’ll either phase in OpenSearch or switch over entirely.

During the final migration, you need to implement:

• Workload Runbooks: Define procedures for handling day-to-day operations, covering search queries, index management, and performance monitoring.
• Alarms and Monitoring: Set up alarms in CloudWatch to alert your team of any anomalies in performance or security.
• Performance Tuning: Continuously monitor and fine-tune your OpenSearch cluster to ensure it operates efficiently, adjusting based on real-world workloads.
• Backup and Disaster Recovery Plans: Establish regular snapshot schedules, and ensure that your disaster recovery plan is robust and regularly tested.

By ensuring that monitoring and automation are in place, you can maintain a smooth-running system post-migration. Regular performance and cost evaluations, as well as ongoing query and data quality improvements, will help keep your OpenSearch cluster optimized.

Through this structured process—starting with clear migration priorities, validating configurations via a PoC, training your team, and finally executing a carefully monitored migration—you can ensure a successful transition from Solr to OpenSearch with minimal disruption and maximum long-term benefits.

Terminology Comparison

Schema and Query Translation: Key Differences

While Solr and OpenSearch both leverage Lucene, the two platforms handle schemas and queries differently, which presents challenges during migration.

Schema Translation

Solr uses a global schema for each collection, which simplifies management but can lead to limitations when different documents require varied settings. OpenSearch, on the other hand, employs index-based mappings, offering flexibility in handling diverse data types across multiple indices.

Query Translation

OpenSearch and Solr differ significantly in their querying approaches, but OpenSearch provides a wide range of powerful options, which you can explore in its API reference. For instance, the simple_query_string query has a syntax similar to Solr’s, making it an easy starting point for users less familiar with OpenSearch’s DSL. In many cases, it can perform well and simplify the migration process.

That said, we’ve observed with one of our customers that using simple_query_string in OpenSearch is not exactly the same as in Solr. While it may seem similar at first glance, there are key differences in behavior, such as handling term-specific boosts.

Here’s an example of where these differences come into play:

In Solr, you can easily boost individual terms in a multi-term query using the caret (^) symbol. But in OpenSearch’s simple_query_string, this functionality is not fully supported, and queries relying on heavy term-specific boosting may need more advanced solutions like the bool query.

Solr Query with Term-Specific Boosting:

q=title:(OpenSearch^2 AWS) OR description:(search engine)

In this Solr query, only the term OpenSearch is boosted within the title field, while AWS is left unboosted. This term-specific boosting within a multi-term query is not natively supported in OpenSearch.

Attempted OpenSearch Equivalent:

{
  "query": {
    "simple_query_string": {
      "fields": ["title", "description"],
      "query": "title:(OpenSearch^2 AWS) OR description:(search engine)"
    }
  }
}

While this query will run, the boost on OpenSearch (^2) will be ignored because simple_query_string in OpenSearch does not support boosting individual terms within a multi-word query.

To accurately replicate this behavior, you would need to use more advanced query types like bool or match, where you can apply specific boosts to each term individually.

OpenSearch bool Query with Term-Specific Boosting:

{
  "query": {
    "bool": {
      "should": [
        { 
          "match": { 
            "title": { 
              "query": "OpenSearch", 
              "boost": 2 
            } 
          } 
        },
        { 
          "match": { 
            "title": { 
              "query": "AWS" 
            } 
          } 
        },
        { 
          "match": { 
            "description": { 
              "query": "search engine" 
            } 
          } 
        }
      ]
    }
  }
}

Here, each term (OpenSearch and AWS) is treated separately, with explicit boosting applied only to OpenSearch. This approach is required because OpenSearch does not support inline term-specific boosts within a single multi-word query in the way Solr does.

Thus, while simple_query_string can be a helpful starting point for migrating basic queries from Solr, more complex queries—especially those involving term-specific boosting—require transitioning to OpenSearch’s full Query DSL for accurate query behavior and control..

More Query Translation Examples

1. Basic Field Query

   • Solr: q=title:OpenSearch
   • OpenSearch:

   {
     "query": {
       "match": {
         "title": "OpenSearch"
       }
     }
   }
   

2. Range Query

   • Solr: q=price:[10 TO 100] AND popularity:[5 TO *]
   • OpenSearch:

   {
     "query": {
       "bool": {
         "must": [
           {
             "range": {
               "price": {
                 "gte": 10,
                 "lte": 100
               }
             }
           },
           {
             "range": {
               "popularity": {
                 "gte": 5
               }
             }
           }
         ]
       }
     }
   }
   

3. Fuzzy Search

   • Solr: q=title:OpenSerch~2
   • OpenSearch:

   {
     "query": {
       "fuzzy": {
         "title": {
           "value": "OpenSerch",
           "fuzziness": 2
         }
       }
     }
   }
   

4. Phrase Query with Slop

   • Solr: q="open source search"~3
   • OpenSearch:

   {
     "query": {
       "match_phrase": {
         "content": {
           "query": "open source search",
           "slop": 3
         }
       }
     }
   }
   

Use Case: MemoMeister’s Solr Migration to Amazon OpenSearch

Freiraum GmbH, based in Stuttgart, manages the digital documentation platform MemoMeister. Recognizing that efficient enterprise search was critical for its customers, the company decided to migrate its search infrastructure from Solr on EC2 to Amazon OpenSearch Service.

Key Objectives of the Migration:

• Reduce operational overhead through a fully managed service.
• Improve search performance and scalability.
• Leverage OpenSearch’s integration with other AWS services for enhanced resilience and ease of management.

The Proof of Concept (PoC) focused on testing OpenSearch in a Multi-AZ environment, ensuring the solution met high availability and performance benchmarks. The results demonstrated that OpenSearch not only met but exceeded expectations in terms of ease of use, resilience, and drastically reduced operational overhead.

Jan Schurkus, CEO bei Freiraum GmbH

During the Proof of Concept Project for the migration of Solr to Amazon OpenSearch Service, we were not only be able to assess the viability of this solution but also initiated the migration process itself. tecRacer helped us to untangle the knots in our minds and unleash the full potential of the service. The impulses they provided were crucial in developing a solution aligned with best practices. We work with a tecRacer on various cloud improvement projects simultaneously, which serves as building blocks for achieving greater success with our customers.

Migration Highlights:

• Savings on Total Cost of Ownership (TCO): Operational overhead was reduced by 50%x, thanks to OpenSearch’s ease of management.
• Cluster Stability: The stability provided by OpenSearch’s managed service ensured a seamless migration experience.
• Optimized Search Capabilities: Despite initial concerns, OpenSearch offered all the necessary search functionalities, along with improved resource efficiency.

Overcoming Challenges in Migration

Migrating from Solr to OpenSearch can involve various challenges, including:

• Schema Differences: Addressing the differences between Solr’s global schema and OpenSearch’s index-specific mappings.
• Query Language Adjustments: Rewriting queries to use OpenSearch’s Query DSL.
• Terminology Translation: Understanding and adapting the differences in terminology between Solr and OpenSearch.
• Data Integrity: Ensuring that field mappings and data ingestion processes don’t lead to inconsistencies during the migration process.

Conclusion

Migrating from Solr to Amazon OpenSearch Service offers numerous long-term benefits, including simplified management, enhanced search functionalities, and better scalability. With the right tools and planning, the migration process can be seamless and rewarding. By adopting a proof of concept approach, organizations can ensure that they are set up for success, both in terms of technical performance and operational efficiency.

Whether you’re looking to reduce operational overhead, improve search capabilities, or leverage AWS’s ecosystem, migrating to OpenSearch is a forward-thinking choice for enterprises.

References

• AWS Documentation on OpenSearch
• AWS Blog: Migrate from Apache Solr to OpenSearch
• tecRacer Training Amazon OpenSearch Service Fundamentals
• MemoMeister
• OpenSearch Documentation
• OpenSearch Java Client
• OpenSearch JavaScript Client
• OpenSearch Python Client
• Solr Documentation