BUG: CIBW is ignoring MACOS arch

[bashtage]

Description

I am trying to build for only x86_64 and CIBW is downloading arm wheels.

Really just need the top and the bottom. architectures: x86_64 is correct, but at the bottom you see wheel download like Downloading numpy-2.1.1-cp312-cp312-macosx_14_0_arm64.whl (5.1 MB).

cibuildwheel version 2.20.0
Build options:
  platform: macos
  allow_empty: False
  architectures: x86_64
  build_selector: 
    build_config: cp312-*
    skip_config: pp* *-win32 cp38-musllinux* cp39-musllinux* *musllinux_aarch64*
    requires_python: >=3.9
    prerelease_pythons: False
    free_threaded_support: False
  output_dir: /Users/runner/work/statsmodels-wheels/statsmodels-wheels/wheelhouse
  package_dir: /Users/runner/work/statsmodels-wheels/statsmodels-wheels/statsmodels
  test_selector: 
    skip_config: *-*linux_{aarch64,ppc64le,s390x} *pyodide*
  before_all: 
  before_build: git submodule foreach git checkout  main
  before_test: 
  build_frontend: None
  build_verbosity: 1
  config_settings: 
  container_engine: docker
  dependency_constraints: pinned
  environment: 
    CGLAGS="-std=c99"
  manylinux_images: None
  musllinux_images: None
  repair_command: delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}
  test_command: python -c "import statsmodels; statsmodels.test(['--skip-examples','--skip-slow','-n','2'], exit=True)"
  test_extras: 
  test_requires: 
    pytest
    pytest-xdist
Cache folder: /Users/runner/Library/Caches/cibuildwheel
Here we go!
Building cp312-macosx_x86_64 wheel
CPython 3.12 macOS x86_64
Installing Python cp312...
                                                              ✓ 0.03s
Setting up build environment...
                                                              ✓ 3.65s
Installing build tools...
                                                              ✓ 1.08s
Running before_build...
                                                              ✓ 0.49s
Building wheel...
  + python -m pip wheel /Users/runner/work/statsmodels-wheels/statsmodels-wheels/statsmodels --wheel-dir=/private/var/folders/n3/x0zngmn15fzg2jfkglj4ys4w0000gn/T/cibw-run-2tmpazp3/cp312-macosx_x86_64/built_wheel --no-deps -v
  Processing ./statsmodels
    Installing build dependencies: started
    Running command pip subprocess to install build dependencies
    Using pip 24.2 from /Users/runner/Library/Application Support/virtualenv/wheel/3.12/image/1/SymlinkPipInstall/pip-24.2-py3-none-any/pip (python 3.12)
    Collecting setuptools>=69.0.2
      Obtaining dependency information for setuptools>=69.0.2 from https://files.pythonhosted.org/packages/cb/9c/9ad11ac06b97e55ada655f8a6bea9d1d3f06e120b178cd578d80e558191d/setuptools-74.1.2-py3-none-any.whl.metadata
      Downloading setuptools-74.1.2-py3-none-any.whl.metadata (6.7 kB)
    Collecting cython<4,>=3.0.10
      Obtaining dependency information for cython<4,>=3.0.10 from https://files.pythonhosted.org/packages/43/39/bdbec9142bc46605b54d674bf158a78b191c2b75be527c6dcf3e6dfe90b8/Cython-3.0.11-py2.py3-none-any.whl.metadata
      Downloading Cython-3.0.11-py2.py3-none-any.whl.metadata (3.2 kB)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11)
    Collecting numpy<3,>=2.0.0
      Obtaining dependency information for numpy<3,>=2.0.0 from https://files.pythonhosted.org/packages/34/f2/1316a6b08ad4c161d793abe81ff7181e9ae2e357a5b06352a383b9f8e800/numpy-2.1.1-cp312-cp312-macosx_14_0_arm64.whl.metadata
      Downloading numpy-2.1.1-cp312-cp312-macosx_14_0_arm64.whl.metadata (60 kB)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/99/f1/c00d6be56e1a718a3068079e3ec8ce044d7179345280f6a3f5066068af0d/scipy-1.6.2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/fe/fd/8704c7b7b34cdac850485e638346025ca57c5a859934b9aa1be5399b33b7/scipy-1.6.3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/bb/bb/944f559d554df6c9adf037aa9fc982a9706ee0e96c0d5beac701cb158900/scipy-1.7.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.10'): https://files.pythonhosted.org/packages/47/33/a24aec22b7be7fdb10ec117a95e1e4099890d8bbc6646902f443fc7719d1/scipy-1.7.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.10)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/0e/23/58c4f995475a2a97cb5f4a032aedaf881ad87cd976a7180c55118d105a1d/scipy-1.7.2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/61/67/1a654b96309c991762ee9bc39c363fc618076b155fe52d295211cf2536c7/scipy-1.7.3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.7,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/c0/ad/e3c052ed4e0027a8abef0a5e8441a044427d252d17d9aee06d56e62fc698/scipy-1.8.0rc1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/29/d2/151a54944b333e465f98804dced31dab1284f3c37b752b9cefa710b64681/scipy-1.8.0rc2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/e4/26/83dd1c6378513a6241d984bda9f08c512b6e35fff13fba3acc1b3c195f02/scipy-1.8.0rc3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/22/78/056cc43e7737811b6f50886788a940f852773dd9804f5365952805db9648/scipy-1.8.0rc4.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/b4/a2/4faa34bf0cdbefd5c706625f1234987795f368eb4e97bde9d6f46860843e/scipy-1.8.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.11'): https://files.pythonhosted.org/packages/26/b5/9330f004b9a3b2b6a31f59f46f1617ce9ca15c0e7fe64288c20385a05c9d/scipy-1.8.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.11)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/87/82/8b9bf8bb8030f1eef281ad2af87cc648e06e4a4974c2b9201a57d991b12c/scipy-1.9.0rc1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/5a/87/27f9ee75c32138d6eaabff2b91744c97a3fdbbbdd78948372dac93e0079f/scipy-1.9.0rc2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/57/72/484c38e5b814b48b41ac6432ef69077f2e7c8094925d3ce17bc3d61359b0/scipy-1.9.0rc3.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/a8/e3/4ec401f609d34162b7023a09[165](https://github.com/MacPython/statsmodels-wheels/actions/runs/10836200460/job/30069432088#step:4:171)da491630879e4cfa2336667fe2102cd06/scipy-1.9.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
      Link requires a different Python (3.12.6 not in: '>=3.8,<3.12'): https://files.pythonhosted.org/packages/db/af/[169](https://github.com/MacPython/statsmodels-wheels/actions/runs/10836200460/job/30069432088#step:4:175)06139f52bc6866c43401869ce247662739ad71afa11c6f18505eb0546/scipy-1.9.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:>=3.8,<3.12)
      Link requires a different Python (3.12.6 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/a2/ce/2592c3b550cf8f68879d4ff2159f3c689ee6f032f8fc9059022074f7bd75/scipy-1.10.0rc1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
      Link requires a different Python (3.12.6 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/6b/a3/7de758a32569e06877a6b298bdf7b529ce97a33c55deb2b1637b0162fd9d/scipy-1.10.0rc2.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
      Link requires a different Python (3.12.6 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/d6/bd/2d13a273d95f7b7d9903c906c486040b0aebb85e008f93a5dd0891f21f1f/scipy-1.10.0.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
      Link requires a different Python (3.12.6 not in: '<3.12,>=3.8'): https://files.pythonhosted.org/packages/84/a9/2bf119f3f9cff1f376f924e39cfae18dec92a1514784046d[185](https://github.com/MacPython/statsmodels-wheels/actions/runs/10836200460/job/30069432088#step:4:191)731301281/scipy-1.10.1.tar.gz (from https://pypi.org/simple/scipy/) (requires-python:<3.12,>=3.8)
    Collecting scipy<2,>=1.13
      Obtaining dependency information for scipy<2,>=1.13 from https://files.pythonhosted.org/packages/66/67/6ef192e0e4d77b20cc33a01e743b00bc9e68fb83b88e06e636d2619a8767/scipy-1.14.1-cp312-cp312-macosx_14_0_arm64.whl.metadata
      Downloading scipy-1.14.1-cp312-cp312-macosx_14_0_arm64.whl.metadata (60 kB)
    Collecting setuptools_scm<9,>=8 (from setuptools_scm[toml]<9,>=8)
      Obtaining dependency information for setuptools_scm<9,>=8 from https://files.pythonhosted.org/packages/a0/b9/[190](https://github.com/MacPython/statsmodels-wheels/actions/runs/10836200460/job/30069432088#step:4:196)6bfeb30f2fc13bb39bf7ddb8749784c05faadbd18a21cf141ba37bff2/setuptools_scm-8.1.0-py3-none-any.whl.metadata
      Downloading setuptools_scm-8.1.0-py3-none-any.whl.metadata (6.6 kB)
    Collecting packaging>=20 (from setuptools_scm<9,>=8->setuptools_scm[toml]<9,>=8)
      Obtaining dependency information for packaging>=20 from https://files.pythonhosted.org/packages/08/aa/cc0[199](https://github.com/MacPython/statsmodels-wheels/actions/runs/10836200460/job/30069432088#step:4:205)a5f0ad350994d660967a8efb233fe0416e4639146c089643407ce6/packaging-24.1-py3-none-any.whl.metadata
      Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
    Downloading setuptools-74.1.2-py3-none-any.whl (1.3 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 35.9 MB/s eta 0:00:00
    Downloading Cython-3.0.11-py2.py3-none-any.whl (1.2 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 53.9 MB/s eta 0:00:00
    Downloading numpy-2.1.1-cp312-cp312-macosx_14_0_arm64.whl (5.1 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 77.2 MB/s eta 0:00:00
    Downloading scipy-1.14.1-cp312-cp312-macosx_14_0_arm64.whl (23.1 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.1/23.1 MB 79.9 MB/s eta 0:00:00

Build log

No response

CI config

No response

[joerick]

This is as designed, unfortunately. The Xcode toolchain can cross-compile, but it's not supported to tell Pip that you want to install x86_64 wheels when running on a arm64 machine.

The best thing to do is compile on the target architecture, if you can. But in this case, because you're going from arm64 to x86_64, there might also be a workaround - you could use the Rosetta emulation to do the compilation, by using arch -x86_64 cibuildwheel

Our macOS multi-arch support was created when x86_64 runners were common, so cross-compilation was our only option. However, as arm64 becomes more common on the host side, it might make sense to do the x86_64 compilation in emulation... 🤔

[henryiii]

Also, when you download a package, you often want to run it on the host (such as calling numpy.f2py.get_include()); so even if you could tell pip to download the cross-compiled wheels, it might not be what you want! The ideal thing would be to download both and fuze them into a univeral2 wheel, I suppose. Might be something you could talk the uv folks into supporting. ;)

[henryiii]

In fact, symbols are mangled in wheels, so do you ever do a link to them? Usually you want Python code and headers, which you want the native wheel for, not the cross-compiled wheels.

This is certainly true for NumPy, which is the example given here. Is it actually causing a problem, or are is it just that it's printing this in the logs? Tests should work since that's emulated.

[bashtage]

Yes, NumPy provides arch-specific math libraries which need to match the target build. In our build there is a warning from ld about finding arm libraries when linking x86_64 binaries. When these are run they raise with symbol not found.

This behavior is currently breaking statamodels on MacOS.

The short term fix is to revert guthub's MacOS rubber from macos-latest to macos-13. Latest runs on M1 while 13 is still hosted on x86_64. This restores the correct behavior for building both arm and x86_64.

[bashtage]

it might make sense to do the x86_64 compilation in emulation...

This would seem to be the logical way forward so that both could be correctly compiled irrespective of host arch. For now it only works correctly when the host is x86_64.

[henryiii]

Okay, I'd strongly recommend macos-13 for Intel builds and macos-14/macos-latest for ARM builds for now. That's fastest, and actually runs your tests (Intel can't run ARM tests). As the last Intel runner, I expect macos-13 will be around for a while (macos-12 was just deprecated).

Why is the Intel one working cross compiling to ARM?

[bashtage]

, it might make sense to do the x86_64 compilation in emulation...

It is just how we always did it. CIBW is only used for releases so changes like the asymmetry introduced by the different runners is not easy to detect. I imagine we will eventually move to distinct runners for the different arch builds.

[bashtage]

I wonder if CIBW should refuse to compile x86_64 on arm64 since it doesn't have a mechanism to know it can build correctly. An early exit would have simplified finding out things were broken.

[henryiii]

The tests failed, right? It can't know beforehand that either your build system can't cross-compile, or that some wheel is going to do something specific (in this case, it seems numpy doing something strange with ARM math libraries). (To be fair, not many projects are cross compiling Intel from ARM yet, that's probably going to be somewhat rare until macos-13 is deprecated, maybe in a year or so).

Do you have the exact error message? A bit curious, though I use pybind11 which doesn't require NumPy at all at build time.

[henryiii]

You are manually trying to link to numpy/_core/lib/libnpymath.a, which is being pulled from the ARM wheel rather than the target platform. Is that public? It seems you have to manually construct the path by going up and then back down from the include path you are given. If you could rebuild it, it would be fine. And if you didn't need to link to it, it would also be fine.

This shouldn't work the other direction, though? Why can the ARM build link to libnpymath.a on Intel?

[henryiii]

From the docs: https://numpy.org/doc/stable/reference/c-api/coremath.html#linking-against-the-core-math-library-in-an-extension

Note

Keep in mind that when you are cross compiling, you must use the numpy for the platform you are building for, not the native one for the build machine. Otherwise you pick up a static library built for the wrong architecture.

So it sounds like cross-compiling ARM from Intel is not supported either; if you use this library, you have to avoid cross-compiling.

There's also this:

Note

An effort is underway to make npymath smaller (since C99 compatibility of compilers has improved over time) and more easily vendorable or usable as a header-only dependency. That will avoid problems with shipping a static library built with a compiler which may not match the compiler used by a downstream package or end user. See gh-20880 for details.

So it's also incorrect to use this with a different compiler than NumPy was built with...