Index Partitions in Axcelerate Projects
Index engines are automatically created with at least two partitions. This makes it possible to treat larger document numbers and to add further partitions when needed, and speeds up index engine activities such as indexing or learning.
Tip: An index engine should be configured with at least one partition before the first document is added to it. This gives you the flexibility to add more partitions later.
Axcelerate database
By default, an index engine is created with 2 index partitions. If the expected number of documents exceeds 1,000,000, a third partition is created and, for more than 1,500,000 expected documents, a fourth one. The additional third and fourth index partition are only created automatically if you insert the respective number of expected documents during application creation.
For or index engines, a maximum of 4 partitions is recommended. For fast document access, the overall number of documents per index engine should not exceed 10,000,000, i.e. 2,500,000 documents per partition.
Tip: For larger document quantities, add more index engines to the application.
Axcelerate Review & Analysis
By default, an index engine is created with 2 index partitions.
For Axcelerate Review & Analysis, it is recommended to use up to 6 index partitions. It is better to have more than 2.5 million documents per partition than to create additional partitions. For more information or to discuss special cases contact Recommind support.
Rules and Recommendations
- You need one core per index partition (in addition to one core for the index engine main process).
- Index partitions must use the same server as the index engine. 1The configuration actually allows you to place index partions on different servers, but this will lead to bad performance.
- The default heap size used by each partition is 256 MB. Depending on your system and the number of documents to be processed with one index engine, it may be useful to configure a larger heap size, in order to optimize the document access and the processing of very large document numbers. Ask Recommind support for further details.
- You can add further partitions, if required, but the best document distribution on partitions is guaranteed if you create all partitions before loading documents into or , respectively before publishing.
- You cannot change the configured number of documents per partition once there are documents in the index engine.
- Using no partitions is only recommended for index engines that treat less than 500,000 documents in or or 800,000 documents in Axcelerate Review & Analysis. The advantage over having an index engine with one partition is that you save the RAM needed for the Java Heap Space of the index partition process. The disadvantage is that this cannot be changed once documents are in the engine and no additional partitions can be added to increase the capacity of the engine if needed.
- An index engine that does not allow partitions has the value internal in the Names of index partitions parameter. Do not change this value after documents have been loaded to the index engine! Doing so will break the index engine.