feat(membership): retain local health history - #10502

src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs:56

• ElapsedSinceLastResponse reads _lastSuccessfulResponseTimestamp as a long? without synchronization. Nullable<long> is not an atomic read/write (it is larger than 64-bits), so this can tear when ProbeDirectly/ProbeIndirectly update the field concurrently with ProbingSiloHealthMonitor reading it, resulting in incorrect/negative elapsed values or sporadic HasValue flips. Consider storing a non-nullable long timestamp and reading it atomically via Volatile.Read (the field is always initialized in the ctor anyway).

        public TimeSpan? ElapsedSinceLastResponse
            => _lastSuccessfulResponseTimestamp is { } timestamp ? _timeProvider.GetElapsedTime(timestamp) : null;

src/Orleans.Runtime/MembershipService/LocalSiloHealthMonitor.cs:593

• SubtractTimestamp converts TimeSpan to a timestamp delta via duration.TotalSeconds * TimestampFrequency. For very small durations (eg TimeSpan.FromTicks(1)), the double conversion can round down and produce a 0 delta, making lookback queries subtly wrong at sub-second granularity. Using integer arithmetic based on duration.Ticks avoids precision loss.

        private long SubtractTimestamp(long timestamp, TimeSpan duration)
            => timestamp - (long)(duration.TotalSeconds * _timeProvider.TimestampFrequency);

src/Orleans.Runtime/MembershipService/SiloHealthMonitor.cs:296

• ProbeDirectly calls roundTripTimer.GetElapsedTime(...) twice (once when formatting the OperationCanceledException message and again when computing roundTripTime). If time advances between those calls, the exception message and logged round-trip time can disagree, and it does extra timestamp reads.

            catch (OperationCanceledException exception)
            {
                failureException = new OperationCanceledException(
                    $"The ping attempt was cancelled after {roundTripTimer.GetElapsedTime(out _)}. Ping #{id}",
                    exception);
            }
            catch (Exception exception)
            {
                failureException = exception;
            }
            var roundTripTime = roundTripTimer.GetElapsedTime(out _);

src/Orleans.Runtime/MembershipService/LocalSiloHealthMonitor.cs:52

• LocalSiloHealthStatus.Complaints currently includes any event with a non-null Complaint, even when the event has Score=0 (for example, informational events like GarbageCollectionPause). This can add noise to degradation logs since the complaint list can include non-degrading observations.

    internal readonly record struct LocalSiloHealthStatus(int Score, ImmutableArray<LocalSiloHealthEvent> Events)
    {
        public ImmutableArray<string> Complaints
            => [.. Events.Where(static status => status.Complaint is not null).Select(static status => status.Complaint!)];