KAFKA-21234: Fix bug in KRaft where the metadata layer can receive records past the high watermark - #23737

Issue Description

When the metadata layer (e.g. MetadataLoader) tries to read a slice of records from KRaft which is greater than KafkaRaftClient#MAX_BATCH_SIZE_BYTES, the RecordsIterator can return records which are past the HWM. As it iterates through this slice of records, FileRecords#readInto reads from disk the minimum of the bytes until end of the file and the remaining space in the associated buffer. In the case of the former, this can lead to a metadata error where the node tries to snapshot with records past the HWM.

For the metadata error to occur, the following three pre-conditions must be met:

1. One committed read handed to a listener is larger than 8 MiB.
2. The segment file has bytes after the HW, meaning records this node already appended but that aren't committed yet.
3. For the error to actually show up, the snapshot generator emits a snapshot from the resulting image before the HW catches up.

What Changed

Add FileRecords#readUntil which reads from the underlying file to the minimum of:

• the remaining space in the associated buffer, which in the case of RecordsIterator is the minimum of the FileRecords' record size and the batch size defined by KRaft.
• the end byte position of the FileRecords object, rather than the end of the underlying file. This ensures when a caller constructs a FileRecords slice of [start position, HWM), no bytes past the HWM are read into memory.

Testing

Added unit tests in RecordsIteratorTest and FileRecordsTest

Alternatives

• Limiting the buffer in RecordsIterator prior to calling FileRecords#readInto. The motivation behind this is that other callers may want to this functionality in the future, so the change belongs in FileRecords, not a caller of that class.
• Changing the implementation of FileRecords#readInto. The main concern I had with this was that there are many other callers of this method outside of the raft layer, and it is not clear to me the impact of this change on them. Adding an additional method does not impact existing behavior. The other users of readInto are AbstractFetcherThread, LogSegment, Cleaner, CoordinatorLoaderImpl, and TransactionStateManager.

Reviewers: José Armando García Sancio jsancio@apache.org