Scaling Using Index Partitions
Index partitions are a way to utilize more system cores in parallel, while not duplicating central data structures, such as the word map. This comes with the penalty of having some communication overhead and additional coordination. For this reason, using more than 4 partitions per Axcelerate database index engine and 6 partitions for the Axcelerate Review & Analysis index engine is not practical.
Advantages of scaling using index partitions
- Avoids duplicating data structures between index engines, such as the word map.
- Reduces memory overhead for Java Heap space (typically 1GB per index engine).
- Increases the utilization
of the CPU:
- Flush will be
ktimes faster (k= number of partitions). - Query stack
computation (which is a part of each search request) will be
ktimes faster. - Count computation will
be slightly faster. (Not a factor of
kbecause the counts from the partitions need to be merged on the index engine and also sorted there.)
- Flush will be
- Speeds up concept group creation, as this is typically only done on the first partition, while the rest is folded in.
Disadvantages of scaling using index partitions
- Single document metadata or text changes are slower due to communication overhead with the index partition that stores the particular document.
- Splitting partitions across physical servers is not advised, due to the communication overhead. If you need to distribute across physical servers, use separate index engines with a meta engine instead.