3.14: Windows on Arm build fails since windows-11-arm migrated to windows-11-vs2026-arm

@StanFromIreland

The Windows Arm64 jobs in the CI recently started failing on 3.14 (for example). Noted by @StanFromIreland in #158188 (comment): the _freeze_module

cpython/PCbuild/pcbuild.proj

Lines 20 to 21 in 9def455

    
           <FreezeProjects> 
        
             <Platform>$(PreferredToolArchitecture)</Platform>

can no longer be compiled for PreferredToolArchitecture=x86 and PlatformToolset=v143:

The build tools for Visual Studio 2022 (Platform Toolset = 'v143') cannot be found.

PreferredToolArchitecture=x86 is chosen, because we use the x86 version of msbuild in CI:

"C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin\msbuild.exe"

which is chosen by build.bat if no MSBUILD environment variable is set and msbuild is not in the path. ¹

Compiling the _freeze_module for arm64 on an arm64 host by just using the arm64 version of msbuild would solve this, by e.g. just adding

      env:
        MSBUILD: ${{ inputs.arch == 'arm64' && 'C:\Program Files\Microsoft Visual Studio\18\Enterprise\MSBuild\Current\Bin\arm64\msbuild.exe' || '' }}

to

cpython/.github/workflows/reusable-windows.yml

Lines 38 to 40 in e16b12b

    
               - name: Build CPython 
        
                 run: >- 
        
                   .\\PCbuild\\build.bat

because then PreferredToolArchitecture=arm64 and also the arm64 version of the compiler is chosen.

I think running the _freeze_module natively on an arm64 host is anyway the better fit?

noticed already in https://github.com/python/cpython/issues/153668#issuecomment-5304195008 ↩

Note, we cannot override PreferredToolArchitecture if we use the x86 version of msbuild via

build.bat "/p:PreferredToolArchitecture=arm64"

due PROCESSOR_ARCHITECTURE=x86 in this case, and via C:\Program Files\Microsoft Visual Studio\18\Community\MSBuild\Microsoft\VC\v180\Microsoft.Cpp.ToolsetLocation.props:

  <PropertyGroup>
    <!-- UseNativeEnvironment and _IsNativeEnvironment was used in prev versions to define if we want to use 64-bit tools when building for x64 platform. -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and ('$(UseNativeEnvironment)' == 'true' or '$(_IsNativeEnvironment)' == 'true')">x64</PreferredToolArchitecture>

    <!-- By default we use the same bitness as the hosting platform -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and ('$(PROCESSOR_ARCHITECTURE)' == 'AMD64' and '$(Platform)' == 'x64')">x64</PreferredToolArchitecture>
    <!-- prefer arm64 tools when on arm64 -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == '' and '$(PROCESSOR_ARCHITECTURE)' == 'ARM64' and '$(HostARM64Suported)' != 'false'">arm64</PreferredToolArchitecture>

    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == ''">x86</PreferredToolArchitecture>

    <!-- If OS is not x64, we cannot use x64 tools even if preferred -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == 'x64' and ('$(PROCESSOR_ARCHITECTURE)' != 'AMD64' and '$(PROCESSOR_ARCHITEW6432)' != 'AMD64' and '$(PROCESSOR_ARCHITECTURE)' != 'ARM64')">x86</PreferredToolArchitecture>

    <!-- If OS is not arm64, we cannot use arm64 tools even if preferred -->
    <PreferredToolArchitecture Condition="'$(PreferredToolArchitecture)' == 'arm64' and '$(PROCESSOR_ARCHITECTURE)' != 'ARM64'">x86</PreferredToolArchitecture>
  </PropertyGroup>

we always get PreferredToolArchitecture=x86. This is also the default, hence in all builds so far:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostX86\x86\CL.exe /c /IC:\a\cpython\cpython\PCbuild\obj\314win32_Debug\_freeze_module\

Why this does not work anymore for the latest windows-11-arm image is not clear to me, especially because these compilers are part of it

Toolset     HostX86_X86 HostX86_ARM64 HostARM64_X86 HostARM64_ARM64
-------     ----------- ------------- ------------- ---------------
14.29.30133        True          True         False           False
14.44.35207        True          True         False            True
14.51.36231        True          True          True            True

so 14.44.35207\bin\HostX86\x86 should be found.

Details

=== Compiler paths ===
--- 14.29.30133 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\arm\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.29.30133\bin\HostX86\x86\cl.exe
--- 14.44.35207 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostarm64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostarm64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\Hostx86\x86\cl.exe
--- 14.51.36231 ---
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostarm64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x86\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\arm64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\x64\cl.exe
C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.51.36231\bin\Hostx86\x86\cl.exe

See https://github.com/chris-eibl/cpython/actions/runs/36241148789/job/108401760423 in my fork.

@zooba

Short summary:

Running the _freeze_module natively on an arm64 host could be done in main as well and would fix 3.14 without the need of migrating to v145, which was done to get 3.14 CI green via backporting:

[3.14] GH-144679: When building with VS 2026 on Windows, use PlatformToolset v145 by default (GH-144680) #158188
- including also Silence experimental coroutine deprecation warnings #150788
[3.14] gh-153668: Temporarily disable unicodectype.c optimization on WinARM64 due to compiler bug (GH-153669) #158221

which could be reverted again.

Otherwise:

The current situation in 3.14 CI is that

v145 is used for WoA builds
but still v143 for AMD64 and x86, because they use windows-2022 that does not have VS2026.

Regarding official releases, AFAIU we could achieve the same (or even stick with v143 for Arm) since we cross-compile and only run the PGO step on @zooba's Azure VM (at least that's my understanding): we'd simply have to use the respective image, that is windows-2022 instead of windows-2025. For the first we get v143, for the latter v145, since this is chosen when VS 2026 is available now on 3.14. Previously, we've used v143 in this case, too.

Compiling and running the _freeze_module natively works on the new windows-11-arm image, tested here:
https://github.com/chris-eibl/cpython/actions/runs/36245087168/job/108412696751
note the

 C:\Program Files\Microsoft Visual Studio\18\Enterprise\VC\Tools\MSVC\14.44.35207\bin\HostArm64\arm64\CL.exe

@zware

@zware I have not yet found out why on your ARM64 Windows 3.x bot

bin\HostArm64\arm64\CL.exe

is chosen to build the _freeze_module, see e.g. https://buildbot.python.org/#/builders/729/builds/11426/steps/2/logs/stdio.

On my D4ps v6 VM in Azure, bin\HostX86\x86\CL.exe is taken by default like in CI.

Only one of

setting the environment variable MSBUILD to the arm64 version
adding the arm64 msbuild path to PATH
calling vcvarsarm64.bat

tricks me into bin\HostArm64\arm64\CL.exe there.

Have you done anything of the above or maybe something even simpler?

I do in fact have MSBUILD set on that worker; see line 19 for example. I found that was the only way to get a build to succeed at all with available RAM.

Yupp, I forgot to emphasize that using the native msbuild would of course use the native compiler for the whole build, which is an even stronger motivation.

Another way to achive that: react on PROCESSOR_ARCHITECTURE in build.bat to prefer the native msbuild which then uses the native compiler tool chain.

Btw, for the same reason, in the whole non-arm CI currently HostX86 is used - and most probably all build bots, except they do set MSBUILD or PreferredToolArchitecture (which I think is close to zero).

Using the native compiler tool chain would also give us faster compilation and lesser likelihood that the compiler runs out of memory.

I think it's totally fine for buildbots to override either of those variables to get a native MSBuild/native build tools.

I also don't have a problem with the use of a newer toolset, provided it's going to be used for release builds as well and the vcruntime tag in cpython-bin-deps gets updated.

Keeping the older toolsets in some buildbots is a nice way to make sure we don't adopt brand new features too eagerly, but compatibility has been good enough that I'm not concerned about whether a contributor will probably be able to use their existing v143 while buildbots are running v145.

Navigation Menu

3.14: Windows on Arm build fails since windows-11-arm migrated to windows-11-vs2026-arm #158244

Short summary:

Otherwise:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	<FreezeProjects>
	<Platform>$(PreferredToolArchitecture)</Platform>

3.14: Windows on Arm build fails since windows-11-arm migrated to windows-11-vs2026-arm #158244

Description

Footnotes

Activity

chris-eibl commented on Sep 26, 2026

chris-eibl commented on Sep 26, 2026

Short summary:

Otherwise:

chris-eibl commented on Sep 26, 2026

zware commented on Sep 27, 2026

chris-eibl commented on Sep 27, 2026

zooba commented on Sep 28, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions