feat(runtime): refresh type manifests on hot reload - #10947

This PR adds runtime support for refreshing Orleans’ generated type/grain manifests after a .NET Hot Reload metadata update, so newly added [GenerateSerializer] types, grain interfaces, and grain classes become usable without restarting silos/clients.

Changes:

• Introduces a MetadataUpdateHandler-driven refresh pipeline with per-container participants (serialization first, then grain metadata) and opt-in registration when MetadataUpdater.IsSupported.
• Rebuilds/merges manifest-derived state (codecs, type-name allow/alias maps, well-known types, grain class maps, RPC proxy mappings, and local cluster/client manifests).
• Adds unit + end-to-end cluster tests covering refresh behavior and idempotency.

• Files reviewed: 27/27 changed files
• Comments generated: 9
• Review effort level: Lite

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/Orleans.Serialization/TypeSystem/CompoundTypeAliasTree.cs:5

• MergeFrom publishes _children using Volatile.Write, but _children itself is not declared volatile, so reads in TryGetChild are not guaranteed to have acquire semantics. Marking _children as volatile ensures safe publication of the copy-on-write dictionary on weak memory architectures.
  src/Orleans.Serialization/Hosting/HotReloadMetadataUpdateHandler.cs:8
• using Orleans.Serialization.Hosting; is unnecessary here (the file is already in namespace Orleans.Serialization.Hosting;) and can trigger IDE0005 (unnecessary using) when code style is enforced as errors.

• Files reviewed: 26/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

Suppressed comments (1)

test/Orleans.Serialization.UnitTests/HotReloadRefreshTests.cs:92

• ScenarioTypeNameFilter currently denies all matching type names permanently, which prevents this test from ever recovering after refresh. Make the filter behavior switchable so the test can simulate a filter policy change (or external allow-list update) across a hot reload batch.

    private sealed class ScenarioTypeNameFilter : Orleans.Serialization.ITypeNameFilter
    {
        public bool? IsTypeNameAllowed(string typeName, string assemblyName)
            => typeName is not null && typeName.Contains(ScenarioNamespaceFragment, StringComparison.Ordinal) ? false : null;
    }

• Files reviewed: 26/26 changed files
• Comments generated: 2
• Review effort level: Lite

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs:83

• OnLocalManifestUpdated reads the evolving _current manifest using a plain read. Since _current is concurrently updated via Interlocked.Exchange in the AsyncEnumerable onPublished callback, using Volatile.Read here avoids stale reads that can cause TryPublishManifest to repeatedly propose a version which is not actually > the latest published manifest (and therefore fail the updateValidator), potentially preventing the local manifest update from being published.

• Files reviewed: 26/26 changed files
• Comments generated: 0 new
• Review effort level: Lite

• Files reviewed: 26/26 changed files
• Comments generated: 2
• Review effort level: Lite

• Files reviewed: 27/27 changed files
• Comments generated: 2
• Review effort level: Lite

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/Orleans.Serialization/Hosting/HotReloadMetadataUpdateHandler.cs:8

• using Orleans.Serialization.Hosting; is redundant here since the file is already in the Orleans.Serialization.Hosting namespace, and the directive is unused. Removing it avoids unnecessary imports and potential IDE0005 diagnostics during build.
  test/Orleans.DefaultCluster.Tests/HotReloadGrainTests.cs:20
• Several using directives at the top of the file appear to be unused (e.g., Microsoft.Extensions.Options, Orleans.GrainReferences, Orleans.Metadata, Orleans.Runtime.Versions, and the extra Orleans.Serialization namespaces). Removing them helps keep the test clean and avoids potential build-time style diagnostics.

• Files reviewed: 27/27 changed files
• Comments generated: 0 new
• Review effort level: Lite

• Files reviewed: 27/27 changed files
• Comments generated: 2
• Review effort level: Lite

• Files reviewed: 27/27 changed files
• Comments generated: 0 new
• Review effort level: Lite

• Files reviewed: 27/27 changed files
• Comments generated: 0 new
• Review effort level: Lite

• Files reviewed: 27/27 changed files
• Comments generated: 1
• Review effort level: Lite

• Files reviewed: 27/27 changed files
• Comments generated: 2
• Review effort level: Lite

Suppressed comments (8)

src/Orleans.Core/Manifest/ClientHotReloadRefresher.cs:72

• Writing the new local manifest here is visible to ClientClusterManifestProvider.RunAsync; if a gateway response is processed concurrently, it can publish a cluster manifest containing the new interfaces before _rpcProvider, the interface resolver, and _grainVersionManifest have been refreshed. That creates a transient window where callers see metadata newer than the client caches. Update the dependent caches first and publish the local client manifest last.

                _clientClusterManifestProvider.OnLocalManifestUpdated(_clientManifestProvider.ClientManifest);

src/Orleans.Runtime/Manifest/SiloHotReloadRefresher.cs:74

• Publishing the refreshed provider does not update ManagementGrain, which captures both LocalGrainManifest and Current in readonly fields when its activation is created (src/Orleans.Runtime/Core/ManagementGrain.cs:46-47). Any existing management activation will therefore continue rejecting GetActivationAddress and interface checks for grain metadata added by hot reload. Make management operations read the live provider or refresh/recreate those snapshots as part of this update.

                _clusterManifestProvider.OnLocalManifestUpdated(_siloManifestProvider.SiloManifest);

src/Orleans.Runtime/Manifest/SiloHotReloadRefresher.cs:77

• OnLocalManifestUpdated makes the new manifest visible to ClusterManifestProvider before _rpcProvider, _grainInterfaceTypeToGrainTypeResolver, and _grainVersionManifest have been refreshed. A concurrent request (or a peer which observes the new minor version) can therefore use the new manifest while this silo still has the old proxy, interface-resolution, or version state. Refresh these local consumers first and publish the cluster manifest last so the version bump is the visibility boundary.

                _clusterManifestProvider.OnLocalManifestUpdated(_siloManifestProvider.SiloManifest);
                _rpcProvider.OnManifestUpdated();
                _grainInterfaceTypeToGrainTypeResolver.OnManifestUpdated();
                _grainVersionManifest.OnLocalManifestUpdated(_siloManifestProvider.SiloManifest);

src/Orleans.Serialization/Hosting/SerializationHotReloadRefresher.cs:79

• TypeManifestProviderAttribute is also valid for user-authored providers, and the provider-discovery contract keeps those providers out of reflection activation because they are configured through DI (test/Orleans.Core.Tests/ProviderRegistrationResolverTests.cs:81-103). This refresh path calls Activator.CreateInstance for every attributed provider, so a custom provider with constructor dependencies is silently skipped and a parameterless one is instantiated outside its configured lifetime. Resolve registered providers through the service provider, or restrict reflective activation to generated providers.

                        if (Activator.CreateInstance(attribute.ProviderType, nonPublic: true) is IConfigureOptions<TypeManifestOptions> provider)
                        {
                            providers.Add(provider);
                        }

src/Orleans.Serialization/Hosting/SerializationHotReloadRefresher.cs:160

• Generated hot-reload assemblies can contain open generic codec types such as Codec_GenericWithCtor<T> whose mutable getField_/setField_ fields are included by this scan. FieldInfo.SetValue cannot set a field on a declaring type which still contains generic parameters, so the first such field throws and the enclosing catch stops clearing accessors for every remaining generated type in the assembly. Handle generic type definitions separately (or clear through generated helpers) and continue per field/type instead of aborting the whole assembly.

                    if (field.Name.StartsWith("setField_", StringComparison.Ordinal) || field.Name.StartsWith("getField_", StringComparison.Ordinal))
                    {
                        field.SetValue(null, null);

src/Orleans.Serialization/Serializers/CodecProvider.cs:115

• These caches are cleared without coordinating with in-flight TryGetCodec/copier calls. A call which started before the manifest swap can finish creating an AbstractTypeSerializer (or another fallback) and execute GetOrAdd after this clear, leaving the stale result cached and causing the newly registered generated codec to remain unused until a later refresh. Publish caches with a generation check or synchronize cache creation with manifest refresh so pre-refresh work cannot repopulate the new cache.

            _untypedCodecs.Clear();
            _typedCodecs.Clear();
            _typedBaseCodecs.Clear();
            _untypedCopiers.Clear();
            _typedCopiers.Clear();

src/Orleans.Serialization/TypeSystem/TypeConverter.cs:170

• An in-flight Parse can pass the filter while the type is still absent, then be preempted before InspectTypeNameFilters writes its false verdict. If refresh runs and purges the dictionary during that gap, the old parse resumes and reinserts the negative entry, so the newly manifested type can still be rejected after refresh. Use a copy-on-write/versioned allow-list cache or synchronize verdict writes with OnManifestUpdated.

            foreach (var entry in _allowedTypes)
            {
                if (!entry.Value)
                {
                    _allowedTypes.TryRemove(entry.Key, out _);

src/Orleans.Serialization/TypeSystem/TypeConverter.cs:153

• ConsumeMetadata(options) formats newly registered serializer and invokable types, and those generated invokables can use a new CompoundTypeAlias branch (for example, the alias generated for a newly added grain interface). ResolveCompoundAliasType still reads the previous _compoundTypeAliases tree during that call, so it can throw TypeLoadException before the refresh publishes the new manifest. Make the new alias tree available to metadata consumption without exposing a partially applied state.

        _compoundTypeAliases = options.CompoundTypeAliases;

• Files reviewed: 27/27 changed files
• Comments generated: 3
• Review effort level: Lite

Suppressed comments (10)

src/Orleans.Core/Core/GrainInterfaceTypeToGrainTypeResolver.cs:42

• Clear() can race with the write at the end of an in-flight generic lookup: a call which captured the old cluster manifest can resume after this line and add its old result back into _genericMapping. A subsequent lookup can then use that stale mapping even though the refreshed manifest contains a different implementation. Publish a generation-aware cache or prevent lookups from inserting results from an older manifest.

        internal void OnManifestUpdated() => _genericMapping.Clear();

src/Orleans.Core/Manifest/ClientHotReloadRefresher.cs:67

• Calling DefaultGrainTypeOptionsProvider.Configure on the already-final _grainTypeOptions also repopulates interfaces which a client configuration intentionally removed. For example, test/Orleans.Core.Tests/ClientBuilderTests.cs:149-160 clears options.Interfaces to reject startup; after this refresh a hot-reload event adds all generated interfaces back, changing the client's configured manifest and proxy surface. Rebuild a scratch options object through the complete configuration pipeline, or otherwise preserve existing exclusions while merging new interfaces.

                        defaultProvider.Configure(_grainTypeOptions);

src/Orleans.Runtime/Manifest/ClusterManifestProvider.cs:104

• If a membership or peer-repair publication wins all three races, this method simply returns without publishing localManifest. LocalGrainManifest has already been replaced, but Current still contains the old local manifest and no later path necessarily retries because the membership major version is unchanged; peers can therefore permanently miss the newly added grain metadata. The update needs a serialized publication/reconciliation path or a retry which cannot silently give up.

            // Publish with a retry: a concurrently published manifest (e.g. built from membership processing
            // before this update) can win the race while still carrying the previous local manifest.
            for (var attempt = 0; attempt < 3; attempt++)
            {
                var current = _current;

src/Orleans.Runtime/Manifest/SiloHotReloadRefresher.cs:69

• Calling DefaultGrainTypeOptionsProvider.Configure directly on the already-final _grainTypeOptions repopulates every generated grain class that startup configuration intentionally removed. GrainTypeOptions is used for per-silo exclusions such as options.Classes.Remove(...) and even options.Classes.Clear() (for example, test/Benchmarks/Ping/PingBenchmark.cs:37-40 and test/Orleans.Core.Tests/SiloBuilderTests.cs:295-304), so any hot-reload update can silently change activation and placement behavior. Rebuild a scratch options object through the complete configuration pipeline, or otherwise merge only newly discovered entries while preserving existing exclusions.

                        defaultProvider.Configure(_grainTypeOptions);

src/Orleans.Runtime/Manifest/SiloHotReloadRefresher.cs:74

• GrainMigratabilityChecker is a singleton which captures clusterManifestProvider.LocalGrainManifest at construction and caches the result of IsImmovable. Refreshing SiloManifestProvider and ClusterManifestProvider here does not update that checker, so a newly added ordinary grain is absent from the captured manifest, classified as immovable, and cached as non-migratable for rebalancing/repartitioning. The refresh must invalidate or update this consumer as well.

                _siloManifestProvider.OnManifestUpdated();
                _clusterManifestProvider.OnLocalManifestUpdated(_siloManifestProvider.SiloManifest);

src/Orleans.Serialization/Hosting/HotReloadMetadataUpdateHandler.cs:64

• The new tests call SerializationHotReloadRefresher.Refresh and SiloHotReloadRefresher.Refresh directly, so UpdateApplication itself is never exercised. That leaves the assembly filtering, phase ordering, weak-participant snapshot, and the documented no-throw behavior unverified; add a focused handler test which supplies manifest and non-manifest updates and a throwing participant.

    public static void UpdateApplication(Type[]? updatedTypes)
    {
        try

src/Orleans.Serialization/Hosting/SerializationHotReloadRefresher.cs:18

• This changes the supported restart boundary, but the current guidance in docs/site/src/content/docs/grains/code-generation.md:45-47 still says a restart is required when an update introduces types through polymorphic or container members and does not describe the new grain-interface/class workflow. Update the hot-reload documentation to cover the refreshed manifest behavior and its stated limitations before merging.

/// Refreshes the serialization layer after a .NET Hot Reload update: re-runs the updated assemblies'
/// generated manifest providers, merges the result into the live <see cref="TypeManifestOptions"/>, and
/// refreshes the caches derived from it.

src/Orleans.Serialization/Serializers/CodecProvider.cs:97

• This lookup is the only place a serializer-only container subscribes the refresher, but it runs only from Initialize, which is reached on the first codec lookup. An application can construct Serializer/CodecProvider, receive a metadata update, and only then perform its first serialization; no participant is registered for that update, so the live manifest is never merged and the newly generated type remains unavailable. The refresher needs to be subscribed before metadata updates, independently of first codec use.

                    _ = _serviceProvider.GetService<Hosting.SerializationHotReloadRefresher>();

src/Orleans.Serialization/TypeSystem/TypeCodec.cs:43

• ConcurrentDictionary.Clear() does not fence an in-flight GetOrAdd factory. A type-name lookup which started before the manifest update can finish after these clears and insert the old TypeKey, so subsequent writes can continue using the pre-refresh alias/name indefinitely. Cache population needs to be synchronized with refresh or versioned so stale factories cannot repopulate the current cache.

        internal void ClearCaches()
        {
            _typeCache.Clear();
            _typeKeyCache.Clear();

src/Orleans.Serialization/TypeSystem/TypeConverter.cs:170

• Removing the current negative entries is not sufficient when parsing can run concurrently with this refresh. A parse which observed the old denying filter can set _allowedTypes[type] = false after this loop has removed it, leaving the stale denial in place and causing the newly allowed type to remain rejected. Coordinate the purge with verdict publication or associate cached verdicts with a manifest generation.

            foreach (var entry in _allowedTypes)
            {
                if (!entry.Value)
                {
                    _allowedTypes.TryRemove(entry.Key, out _);

• Files reviewed: 27/27 changed files
• Comments generated: 2
• Review effort level: Lite