Package org.elasticsearch.index.engine

Class InternalEngine

java.lang.Object
org.elasticsearch.index.engine.Engine
org.elasticsearch.index.engine.InternalEngine
All Implemented Interfaces:
Closeable, AutoCloseable
public class InternalEngine extends Engine

  • Constructor Details

    • InternalEngine

      public InternalEngine(EngineConfig engineConfig)

  • Method Details

    • completionStats

      public CompletionStats completionStats(String... fieldNamePatterns)
      Description copied from class: Engine
      Returns the CompletionStats for this engine
      Specified by:
      completionStats in class Engine

    • restoreLocalHistoryFromTranslog

      public int restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) throws IOException
      Description copied from class: Engine
      This method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.
      Specified by:
      restoreLocalHistoryFromTranslog in class Engine
      Returns:
      the number of translog operations have been recovered
      Throws:
      IOException

    • fillSeqNoGaps

      public int fillSeqNoGaps(long primaryTerm) throws IOException
      Description copied from class: Engine
      Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.
      Specified by:
      fillSeqNoGaps in class Engine
      Parameters:
      primaryTerm - the shards primary term this engine was created for
      Returns:
      the number of no-ops added
      Throws:
      IOException

    • recoverFromTranslog

      public InternalEngine recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo) throws IOException
      Description copied from class: Engine
      Performs recovery from the transaction log up to recoverUpToSeqNo (inclusive). This operation will close the engine if the recovery fails.
      Specified by:
      recoverFromTranslog in class Engine
      Parameters:
      translogRecoveryRunner - the translog recovery runner
      recoverUpToSeqNo - the upper bound, inclusive, of sequence number to be recovered
      Throws:
      IOException

    • skipTranslogRecovery

      public void skipTranslogRecovery()
      Description copied from class: Engine
      Do not replay translog operations, but make the engine be ready.
      Specified by:
      skipTranslogRecovery in class Engine

    • isTranslogSyncNeeded

      public boolean isTranslogSyncNeeded()
      Description copied from class: Engine
      Checks if the underlying storage sync is required.
      Specified by:
      isTranslogSyncNeeded in class Engine

    • ensureTranslogSynced

      public boolean ensureTranslogSynced(Stream<Translog.Location> locations) throws IOException
      Description copied from class: Engine
      Ensures that all locations in the given stream have been written to the underlying storage.
      Specified by:
      ensureTranslogSynced in class Engine
      Throws:
      IOException

    • syncTranslog

      public void syncTranslog() throws IOException
      Specified by:
      syncTranslog in class Engine
      Throws:
      IOException

    • readHistoryOperations

      public Translog.Snapshot readHistoryOperations(String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws IOException
      Creates a new history snapshot for reading operations since the provided seqno. The returned snapshot can be retrieved from either Lucene index or translog files.
      Specified by:
      readHistoryOperations in class Engine
      Throws:
      IOException

    • estimateNumberOfHistoryOperations

      public int estimateNumberOfHistoryOperations(String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws IOException
      Returns the estimated number of history operations whose seq# at least the provided seq# in this engine.
      Specified by:
      estimateNumberOfHistoryOperations in class Engine
      Throws:
      IOException

    • getTranslogStats

      public TranslogStats getTranslogStats()
      Specified by:
      getTranslogStats in class Engine

    • getTranslogLastWriteLocation

      public Translog.Location getTranslogLastWriteLocation()
      Description copied from class: Engine
      Returns the last location that the translog of this engine has written into.
      Specified by:
      getTranslogLastWriteLocation in class Engine

    • getHistoryUUID

      public String getHistoryUUID()
      Description copied from class: Engine
      returns the history uuid for the engine
      Specified by:
      getHistoryUUID in class Engine

    • getForceMergeUUID

      @Nullable public String getForceMergeUUID()
      returns the force merge uuid for the engine

    • getWritingBytes

      public long getWritingBytes()
      Returns how many bytes we are currently moving from indexing buffer to segments on disk
      Specified by:
      getWritingBytes in class Engine

    • get

      public Engine.GetResult get(Engine.Get get, MappingLookup mappingLookup, DocumentParser documentParser, Function<Engine.Searcher,Engine.Searcher> searcherWrapper)
      Specified by:
      get in class Engine

    • assertPrimaryCanOptimizeAddDocument

      protected boolean assertPrimaryCanOptimizeAddDocument(Engine.Index index)

    • assertPrimaryIncomingSequenceNumber

      protected boolean assertPrimaryIncomingSequenceNumber(Engine.Operation.Origin origin, long seqNo)

    • generateSeqNoForOperationOnPrimary

      protected long generateSeqNoForOperationOnPrimary(Engine.Operation operation)

    • advanceMaxSeqNoOfUpdatesOrDeletesOnPrimary

      protected void advanceMaxSeqNoOfUpdatesOrDeletesOnPrimary(long seqNo)

    • index

      public Engine.IndexResult index(Engine.Index index) throws IOException
      Description copied from class: Engine
      Perform document index operation on the engine
      Specified by:
      index in class Engine
      Parameters:
      index - operation to perform
      Returns:
      Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown
      Throws:
      IOException

    • planIndexingAsNonPrimary

      protected final InternalEngine.IndexingStrategy planIndexingAsNonPrimary(Engine.Index index) throws IOException
      Throws:
      IOException

    • indexingStrategyForOperation

      protected InternalEngine.IndexingStrategy indexingStrategyForOperation(Engine.Index index) throws IOException
      Throws:
      IOException

    • delete

      public Engine.DeleteResult delete(Engine.Delete delete) throws IOException
      Description copied from class: Engine
      Perform document delete operation on the engine
      Specified by:
      delete in class Engine
      Parameters:
      delete - operation to perform
      Returns:
      Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown
      Throws:
      IOException

    • deletionStrategyForOperation

      protected InternalEngine.DeletionStrategy deletionStrategyForOperation(Engine.Delete delete) throws IOException
      Throws:
      IOException

    • planDeletionAsNonPrimary

      protected final InternalEngine.DeletionStrategy planDeletionAsNonPrimary(Engine.Delete delete) throws IOException
      Throws:
      IOException

    • assertNonPrimaryOrigin

      protected boolean assertNonPrimaryOrigin(Engine.Operation operation)

    • maybePruneDeletes

      public void maybePruneDeletes()
      Description copied from class: Engine
      Tries to prune buffered deletes from the version map.
      Specified by:
      maybePruneDeletes in class Engine

    • noOp

      public Engine.NoOpResult noOp(Engine.NoOp noOp) throws IOException
      Specified by:
      noOp in class Engine
      Throws:
      IOException

    • preFlightCheckForNoOp

      protected Optional<Exception> preFlightCheckForNoOp(Engine.NoOp noOp) throws IOException
      Executes a pre-flight check for a given NoOp. If this method returns a non-empty result, the engine won't process this NoOp and returns a failure.
      Throws:
      IOException

    • refresh

      public void refresh(String source) throws EngineException
      Description copied from class: Engine
      Synchronously refreshes the engine for new search operations to reflect the latest changes.
      Specified by:
      refresh in class Engine
      Throws:
      EngineException

    • maybeRefresh

      public boolean maybeRefresh(String source) throws EngineException
      Description copied from class: Engine
      Synchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.
      Specified by:
      maybeRefresh in class Engine
      Returns:
      true if the a refresh happened. Otherwise false
      Throws:
      EngineException

    • writeIndexingBuffer

      public void writeIndexingBuffer() throws EngineException
      Description copied from class: Engine
      Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.
      Specified by:
      writeIndexingBuffer in class Engine
      Throws:
      EngineException

    • syncFlush

      public Engine.SyncedFlushResult syncFlush(String syncId, Engine.CommitId expectedCommitId) throws EngineException
      Description copied from class: Engine
      Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.
      Specified by:
      syncFlush in class Engine
      Parameters:
      syncId - id of this sync
      expectedCommitId - the expected value of
      Returns:
      true if the sync commit was made, false o.w.
      Throws:
      EngineException

    • shouldPeriodicallyFlush

      public boolean shouldPeriodicallyFlush()
      Description copied from class: Engine
      Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.
      Specified by:
      shouldPeriodicallyFlush in class Engine

    • flush

      public Engine.CommitId flush(boolean force, boolean waitIfOngoing) throws EngineException
      Description copied from class: Engine
      Flushes the state of the engine including the transaction log, clearing memory.
      Specified by:
      flush in class Engine
      Parameters:
      force - if true a lucene commit is executed even if no changes need to be committed.
      waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.
      Returns:
      the commit Id for the resulting commit
      Throws:
      EngineException

    • rollTranslogGeneration

      public void rollTranslogGeneration() throws EngineException
      Description copied from class: Engine
      Rolls the translog generation and cleans unneeded.
      Specified by:
      rollTranslogGeneration in class Engine
      Throws:
      EngineException

    • trimUnreferencedTranslogFiles

      public void trimUnreferencedTranslogFiles() throws EngineException
      Description copied from class: Engine
      checks and removes translog files that no longer need to be retained. See TranslogDeletionPolicy for details
      Specified by:
      trimUnreferencedTranslogFiles in class Engine
      Throws:
      EngineException

    • shouldRollTranslogGeneration

      public boolean shouldRollTranslogGeneration()
      Description copied from class: Engine
      Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
      Specified by:
      shouldRollTranslogGeneration in class Engine
      Returns:
      true if the current generation should be rolled to a new generation

    • trimOperationsFromTranslog

      public void trimOperationsFromTranslog(long belowTerm, long aboveSeqNo) throws EngineException
      Description copied from class: Engine
      Trims translog for terms below belowTerm and seq# above aboveSeqNo
      Specified by:
      trimOperationsFromTranslog in class Engine
      Throws:
      EngineException
      See Also:
      Translog.trimOperations(long, long)

    • forceMerge

      public void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments, String forceMergeUUID) throws EngineException, IOException
      Description copied from class: Engine
      Triggers a forced merge on this engine
      Specified by:
      forceMerge in class Engine
      Throws:
      EngineException
      IOException

    • acquireLastIndexCommit

      public Engine.IndexCommitRef acquireLastIndexCommit(boolean flushFirst) throws EngineException
      Description copied from class: Engine
      Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.
      Specified by:
      acquireLastIndexCommit in class Engine
      Parameters:
      flushFirst - indicates whether the engine should flush before returning the snapshot
      Throws:
      EngineException

    • acquireSafeIndexCommit

      public Engine.IndexCommitRef acquireSafeIndexCommit() throws EngineException
      Description copied from class: Engine
      Snapshots the most recent safe index commit from the engine.
      Specified by:
      acquireSafeIndexCommit in class Engine
      Throws:
      EngineException

    • getSafeCommitInfo

      public SafeCommitInfo getSafeCommitInfo()
      Specified by:
      getSafeCommitInfo in class Engine
      Returns:
      a summary of the contents of the current safe commit

    • maybeFailEngine

      protected boolean maybeFailEngine(String source, Exception e)
      Description copied from class: Engine
      Check whether the engine should be failed
      Overrides:
      maybeFailEngine in class Engine

    • getLastCommittedSegmentInfos

      protected org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()
      Specified by:
      getLastCommittedSegmentInfos in class Engine

    • writerSegmentStats

      protected final void writerSegmentStats(SegmentsStats stats)
      Overrides:
      writerSegmentStats in class Engine

    • getIndexBufferRAMBytesUsed

      public long getIndexBufferRAMBytesUsed()
      Description copied from class: Engine
      How much heap is used that would be freed by a refresh. Note that this may throw AlreadyClosedException.
      Specified by:
      getIndexBufferRAMBytesUsed in class Engine

    • segments

      public List<Segment> segments(boolean verbose)
      Description copied from class: Engine
      The list of segments in the engine.
      Specified by:
      segments in class Engine

    • closeNoLock

      protected final void closeNoLock(String reason, CountDownLatch closedLatch)
      Closes the engine without acquiring the write lock. This should only be called while the write lock is hold or in a disaster condition ie. if the engine is failed.
      Specified by:
      closeNoLock in class Engine

    • getReferenceManager

      protected final org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getReferenceManager(Engine.SearcherScope scope)
      Specified by:
      getReferenceManager in class Engine

    • activateThrottling

      public void activateThrottling()
      Description copied from class: Engine
      Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call to Engine.deactivateThrottling().
      Specified by:
      activateThrottling in class Engine

    • deactivateThrottling

      public void deactivateThrottling()
      Description copied from class: Engine
      Reverses a previous Engine.activateThrottling() call.
      Specified by:
      deactivateThrottling in class Engine

    • isThrottled

      public boolean isThrottled()
      Description copied from class: Engine
      Returns the true iff this engine is currently under index throttling.
      Specified by:
      isThrottled in class Engine
      See Also:
      Engine.getIndexThrottleTimeInMillis()

    • getIndexThrottleTimeInMillis

      public long getIndexThrottleTimeInMillis()
      Description copied from class: Engine
      Returns the number of milliseconds this engine was under index throttling.
      Specified by:
      getIndexThrottleTimeInMillis in class Engine

    • commitIndexWriter

      protected void commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog, @Nullable String syncId) throws IOException
      Commits the specified index writer.
      Parameters:
      writer - the index writer to commit
      translog - the translog
      syncId - the sync flush ID (null if not committing a synced flush)
      Throws:
      IOException - if an I/O exception occurs committing the specfied writer

    • onSettingsChanged

      public void onSettingsChanged(org.elasticsearch.core.TimeValue translogRetentionAge, ByteSizeValue translogRetentionSize, long softDeletesRetentionOps)
      Overrides:
      onSettingsChanged in class Engine

    • getMergeStats

      public MergeStats getMergeStats()
      Overrides:
      getMergeStats in class Engine

    • getLastSyncedGlobalCheckpoint

      public long getLastSyncedGlobalCheckpoint()
      Description copied from class: Engine
      Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)
      Specified by:
      getLastSyncedGlobalCheckpoint in class Engine

    • getProcessedLocalCheckpoint

      public long getProcessedLocalCheckpoint()

    • getPersistedLocalCheckpoint

      public long getPersistedLocalCheckpoint()
      Specified by:
      getPersistedLocalCheckpoint in class Engine
      Returns:
      the persisted local checkpoint for this Engine

    • markSeqNoAsSeen

      protected final void markSeqNoAsSeen(long seqNo)
      Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value.

    • hasBeenProcessedBefore

      protected final boolean hasBeenProcessedBefore(Engine.Operation op)
      Checks if the given operation has been processed in this engine or not.
      Returns:
      true if the given operation was processed; otherwise false.

    • getSeqNoStats

      public SeqNoStats getSeqNoStats(long globalCheckpoint)
      Specified by:
      getSeqNoStats in class Engine
      Returns:
      a SeqNoStats object, using local state and the supplied global checkpoint

    • newChangesSnapshot

      public Translog.Snapshot newChangesSnapshot(String source, MapperService mapperService, long fromSeqNo, long toSeqNo, boolean requiredFullRange) throws IOException
      Description copied from class: Engine
      Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw an IllegalStateException.
      Specified by:
      newChangesSnapshot in class Engine
      Throws:
      IOException

    • hasCompleteOperationHistory

      public boolean hasCompleteOperationHistory(String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws IOException
      Description copied from class: Engine
      Checks if this engine has every operations since startingSeqNo(inclusive) in its history (either Lucene or translog)
      Specified by:
      hasCompleteOperationHistory in class Engine
      Throws:
      IOException

    • getMinRetainedSeqNo

      public final long getMinRetainedSeqNo()
      Returns the minimum seqno that is retained in the Lucene index. Operations whose seq# are at least this value should exist in the Lucene index.
      Specified by:
      getMinRetainedSeqNo in class Engine
      Returns:
      the minimum retained sequence number

    • acquireHistoryRetentionLock

      public Closeable acquireHistoryRetentionLock(Engine.HistorySource historySource)
      Description copied from class: Engine
      Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed
      Specified by:
      acquireHistoryRetentionLock in class Engine

    • refreshIfNeeded

      protected final void refreshIfNeeded(String source, long requestingSeqNo)
      Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint.

    • getMaxSeenAutoIdTimestamp

      public final long getMaxSeenAutoIdTimestamp()
      Description copied from class: Engine
      Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard via Engine.updateMaxUnsafeAutoIdTimestamp(long). Notes this method returns the auto_id_timestamp of all append-only requests, not max_unsafe_auto_id_timestamp.
      Overrides:
      getMaxSeenAutoIdTimestamp in class Engine

    • updateMaxUnsafeAutoIdTimestamp

      public final void updateMaxUnsafeAutoIdTimestamp(long newTimestamp)
      Description copied from class: Engine
      Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at most newTimestamp.
      Specified by:
      updateMaxUnsafeAutoIdTimestamp in class Engine

    • getMaxSeqNoOfUpdatesOrDeletes

      public long getMaxSeqNoOfUpdatesOrDeletes()
      Description copied from class: Engine
      Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number from Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long). An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.

      A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:

      1. I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
      2. I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.

      When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).

      However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.

      Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.

      Specified by:
      getMaxSeqNoOfUpdatesOrDeletes in class Engine
      See Also:
      Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long)

    • advanceMaxSeqNoOfUpdatesOrDeletes

      public void advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary)
      Description copied from class: Engine
      A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.
      Specified by:
      advanceMaxSeqNoOfUpdatesOrDeletes in class Engine

    • getRawFieldRange

      public ShardLongFieldRange getRawFieldRange(String field)
      Specified by:
      getRawFieldRange in class Engine
      Returns:
      a ShardLongFieldRange containing the min and max raw values of the given field for this shard if the engine guarantees these values never to change, or ShardLongFieldRange.EMPTY if this field is empty, or ShardLongFieldRange.UNKNOWN if this field's value range may change in future.