Metadata in pyproject.toml

.github/workflows/release.yml

@@ -20,7 +20,7 @@ jobs:
         python-version: "3.14"
 
     - uses: astral-sh/setup-uv@v7
-    - run: uvx --with tox-uv tox -e py,docs,style
+    - run: uvx --with tox-uv tox -e py,style
     - run: uv build
     - name: Publish to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1

.github/workflows/tests.yml

@@ -56,4 +56,4 @@ jobs:
         python-version: "3.14"
 
     - uses: astral-sh/setup-uv@v7
-    - run: uvx --with tox-uv tox -e docs,style
+    - run: uvx --with tox-uv tox -e style

.gitignore

@@ -25,3 +25,4 @@ root/
 venv*/
 
 pp*.yml
+uv.lock

ARCHITECTURE.md

@@ -0,0 +1,74 @@
+## Architecture
+
+### Core Classes Hierarchy
+
+```
+ModuleBuilder (abstract base)
+├── PythonBuilder (abstract, extends ModuleBuilder)
+│   └── Cpython (concrete implementation in cpython.py)
+└── External modules (in external/xcpython.py, xtkinter.py)
+    ├── Bdb, Bzip2, Gdbm, LibFFI, Mpdec, Openssl, Readline
+    ├── Sqlite, Uuid, Xz, Zlib, Zstd, TkInter
+
+BuildSetup (coordinates overall compilation)
+├── BuildContext (handles isolation hacks on macOS)
+├── Folders (manages build directory structure)
+└── ModuleCollection (manages module selection and dependencies)
+
+PPG (Global state singleton)
+├── config: Config (YAML configuration)
+├── target: PlatformId (target OS/arch)
+├── cpython: CPythonFamily (available versions)
+└── families: dict (extensible family support)
+
+Config
+├── Folders (build/dist/sources/logs directories)
+├── target platform detection
+├── YAML config merging (platform-specific overrides)
+
+PythonInspector
+├── Inspect Python installation for portability
+├── LibAutoCorrect (rewrite paths to be relative)
+└── Report system and shared lib detection
+
+Tracker/Trackable
+├── Categorizes found issues/objects by type
+└── Provides detailed reports
+```
+
+### Key Design Patterns
+
+1. **Hierarchical Module Building**: External modules are compiled first, then Python itself, using the same build framework.
+
+2. **Environment Variable Injection**: `xenv_*` methods dynamically provide environment variables (CPATH, LDFLAGS, PATH, etc.) for compilation.
+
+3. **Platform Abstraction**: `PPG.target` (PlatformId) encapsulates platform logic. Compile methods named `_do_<platform>_compile()` dispatch to platform-specific implementations.
+
+4. **Configuration Precedence**: YAML config supports platform-specific overrides (windows.ext, macos.env, etc.). Most specific setting wins.
+
+5. **Folder Masking (macOS)**: On macOS, `/usr/local` is temporarily masked with a RAM disk to prevent accidental dynamic linking.
+
+6. **Build Isolation**: All external modules compiled to a shared `build/deps/` folder, Python finds them via CPATH/LDFLAGS.
+
+7. **Lazy Version Fetching**: `VersionFamily` caches available versions, fetching from python.org on first access.
+
+8. **Telltale Detection**: Modules check for marker files (`m_telltale`) to determine if they're already available on the system (as shared libs).
+
+9. **Log Aggregation**: Each module logs to a separate file (`01-openssl.log`, `02-cpython.log`, etc.) under `build/logs/`.
+
+
+## CI/CD
+
+### GitHub Actions
+
+**tests.yml** (main branch & PRs):
+- Matrix test on py3.10, 3.11, 3.12, 3.13, 3.14
+- Runs: `uvx --with tox-uv tox -e py`
+- Coverage upload to coveralls.io (parallel, then finish)
+- Linter job: docs + style checks on 3.14
+
+**release.yml** (version tags v*):
+- Triggers on `v[0-9]*` tags
+- Runs all tests + docs + style
+- Builds distribution with `uv build`
+- Publishes to PyPI via trusted publishing (OIDC)

CLAUDE.md

@@ -0,0 +1,76 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What This Project Does
+
+portable-python is a CLI tool and Python library for compiling portable CPython binaries from source. Binaries are statically linked so they can be extracted to any folder and used without installation. Supports Linux and macOS (not Windows).
+
+## Common Commands
+
+```bash
+# Run all tests (tox manages envs for py310-314, coverage, docs, style)
+tox
+
+# Run tests for a single Python version
+tox -e py313
+
+# Run a single test
+pytest tests/test_build.py::test_build_rc -vv
+
+# Lint check / auto-fix
+tox -e style
+tox -e reformat
+
+# Run tests directly (from venv)
+pytest tests/
+
+# CI uses: uvx --with tox-uv tox -e py
+```
+
+## Architecture
+
+**Key classes:**
+
+- `PPG` (versions.py) — Global singleton holding config, target platform, and version families. All modules access shared state through this.
+- `BuildSetup` (__init__.py) — Coordinates overall compilation: downloads sources, builds external modules, then CPython.
+- `ModuleBuilder` (__init__.py) — Abstract base for anything that gets compiled. Both external C modules and Python itself extend this.
+- `PythonBuilder` (__init__.py) — Extends ModuleBuilder for Python implementations.
+- `Cpython` (cpython.py) — Concrete PythonBuilder that handles CPython's configure/make/install, optimization, and finalization.
+- `Config` (config.py) — Loads and merges YAML configuration (portable-python.yml) with platform-specific overrides.
+- `PythonInspector` (inspector.py) — Validates portability of a built Python by checking shared library dependencies and paths.
+
+**External modules** (external/xcpython.py): `Bdb`, `Bzip2`, `Gdbm`, `LibFFI`, `Mpdec`, `Openssl`, `Readline`, `Sqlite`, `Uuid`, `Xz`, `Zlib`, `Zstd` — each is a ModuleBuilder subclass that compiles a C library statically before CPython is built.
+
+**Key patterns:**
+
+- Platform-specific compile logic uses `_do_linux_compile()` / `_do_macos_compile()` method dispatch.
+- Environment injection: `xenv_*` methods provide CPATH, LDFLAGS, PATH etc. for compilation.
+- On macOS, `/usr/local` is masked with a RAM disk (`FolderMask`) to prevent accidental dynamic linking.
+- External modules compile to a shared `build/deps/` prefix; CPython finds them via CPATH/LDFLAGS.
+- Telltale detection: modules check for marker files (`m_telltale`) to determine if system already has the library.
+- No patches to upstream CPython source — relies solely on configure flags.
+
+**runez** is the foundational utility library (file ops, system info, CLI decorators, logging, Version/PythonSpec types). Check runez before reimplementing anything.
+
+**Additional pointers:**
+
+- `ModuleCollection.selected` contains only the modules chosen for a build — not all candidates.
+- Build logs go to `build/logs/NN-modulename.log` (e.g. `01-openssl.log`, `02-cpython.log`).
+- YAML config supports platform-specific overrides and path templates — see CONFIGURATION.md.
+- See ARCHITECTURE.md for class hierarchy and design patterns, DEVELOP.md for common tasks and dependencies.
+
+## Testing
+
+- pytest with 100% code coverage target
+- Tests mock `runez.run()` to avoid actual compilation — uses `--dryrun` mode
+- `conftest.py` provides a `cli` fixture (from runez) and forbids HTTP calls (`GlobalHttpCalls.forbid()`)
+- Sample YAML configs in `tests/sample-config*.yml` for testing configuration parsing
+
+## Linting
+
+Ruff handles all linting and formatting. Key settings in pyproject.toml:
+- Line length: 140
+- McCabe complexity: max 18
+- Security checks (S rules) disabled in tests
+- Numpy-style docstrings

CONFIGURATION.md

@@ -0,0 +1,53 @@
+# Configuration (portable-python.yml)
+
+portable-python is configured via a YAML file (default: `portable-python.yml` in current directory, override with `--config`).
+
+
+## Key Sections
+
+### folders
+
+- `build`, `dist`, `sources`, `logs`, `destdir`, `ppp-marker`
+- All support path templates: `{build}`, `{version}`, `{abi_suffix}`
+
+### cpython-modules (CSV)
+
+Which external modules to auto-select.
+Default is configured in `DEFAULT_CONFIG` (see `src/portable_python/config.py`).
+Examples: openssl, zlib, xz, sqlite, bzip2, gdbm, libffi, readline, uuid
+
+### cpython-configure (list)
+
+Extra `./configure` args for CPython.
+Default includes: `--enable-optimizations`, `--with-lto`, `--with-ensurepip=upgrade`
+
+### cpython-clean-1st-pass (list)
+
+Files to remove before `compileall` — removes test files, idle, 2to3 (~94 MB savings).
+
+### cpython-clean-2nd-pass (list)
+
+Files to remove after `compileall` — removes pycaches for seldom-used libs (~1.8 MB savings).
+
+
+## Per-module config
+
+Each external module can be customized with these keys (replace `{module}` with the module name, e.g. `openssl`):
+
+| Key | Purpose |
+|-----|---------|
+| `{module}-version` | Version to use |
+| `{module}-url` | URL to download from |
+| `{module}-src-suffix` | File extension if not in URL |
+| `{module}-configure` | Custom configure args |
+| `{module}-http-headers` | HTTP headers for download |
+| `{module}-patches` | File patches to apply |
+| `{module}-debian` | Package name on Debian (for dependency detection) |
+
+
+## Platform-specific overrides
+
+Configuration supports platform-specific sections (e.g. `windows.ext`, `macos.env`, etc.).
+The most specific setting wins.
+
+`MACOSX_DEPLOYMENT_TARGET` defaults to 13 (Ventura).

DEVELOP.md

@@ -3,9 +3,7 @@
 Create a dev venv:
 
 ```shell
-uv venv
-uv pip install -r requirements.txt -r tests/requirements.txt
-uv pip install -e .
+uv sync
 ```
 
 You can then run `portable-python` from that venv:
@@ -66,3 +64,48 @@ Now inside docker, you run a build:
 ```shell
 portable-python build 3.13.2
 ```
+
+
+# Key Dependencies
+
+| Package | Version | Purpose |
+|---------|---------|---------|
+| click | <9 | CLI framework |
+| pyyaml | <7 | Configuration parsing |
+| requests | <3 | HTTP downloads |
+| runez | <6 | Utilities (file ops, system, logging) |
+| urllib3 | <3 | HTTP transport |
+| pytest-cov | - | Coverage reporting (dev only) |
+
+**runez** is central: provides file ops (`ls_dir`, `touch`, `compress`/`decompress`), system info (platform detection), CLI decorators (`@click.group`), logging, and version handling (`Version`, `PythonSpec`).
+
+
+# Common Tasks
+
+## Add a New External Module
+
+1. Create class in `src/portable_python/external/xcpython.py` extending `ModuleBuilder`
+2. Set `m_name`, `m_telltale`, `m_debian`, `version` property
+3. Implement `url` property (or override `_do_linux_compile()` / `_do_macos_compile()`)
+4. Add to `Cpython.candidate_modules()` if it's a CPython sub-module
+5. Add tests in `tests/test_setup.py`
+
+## Add a New Config Option
+
+1. Update `DEFAULT_CONFIG` in `src/portable_python/config.py`
+2. Use `PPG.config.get_value("key")` to retrieve it in code
+3. Add tests to `tests/test_setup.py`
+
+## Fix a Portability Issue
+
+1. Run `portable-python inspect <PATH>` to diagnose
+2. If lib is being dynamically linked, add to module list or update isolation
+3. Use `LibAutoCorrect.run()` logic (or extend it) to fix paths
+4. Add test case to `tests/test_inspector.py`
+
+## Bump Python Support
+
+1. Update `pyproject.toml` classifiers and `requires-python`
+2. Update `.github/workflows/tests.yml` matrix
+3. Update `CPythonFamily.min_version` if needed
+4. Run full test matrix with `tox`

MANIFEST.in

@@ -1,5 +1,2 @@
 include LICENSE
 include README.rst
-include DEVELOP.md
-include SECURITY.md
-include requirements.txt

README.rst

@@ -128,6 +128,64 @@ Note that you can use ``--dryrun`` mode to inspect what would be done without do
     Would tar build/3.9.7 -> dist/cpython-3.9.7-macos-x86_64.tar.gz
 
 
+CLI reference
+-------------
+
+Main entry point::
+
+    portable-python [OPTIONS] COMMAND [ARGS]
+
+**Global options**:
+
+- ``--config PATH``: Config file (default: ``portable-python.yml``)
+- ``--quiet``: Turn off DEBUG logging
+- ``--dryrun`` / ``-n``: Show what would be done
+- ``--target PLATFORM``: Override detected platform (for testing)
+
+
+**build** ``<PYTHON_SPEC>`` - Build a portable Python binary::
+
+    portable-python build 3.13.2 -m openssl,zlib
+
+- ``--modules, -m CSV``: External modules to include
+- ``--prefix, -p PATH``: Use ``--prefix`` (non-portable)
+
+
+**build-report** ``[PYTHON_SPEC]`` - Show module status and what will be compiled:
+
+- ``--modules, -m CSV``: Specific modules to check
+- Validates that modules can be built
+
+
+**inspect** ``<PATH>`` - Check if a Python installation is portable::
+
+    portable-python inspect /usr/bin/python3
+
+- ``--modules, -m MODULES``: Which modules to inspect
+- ``--verbose, -v``: Show full ``.so`` report
+- ``--prefix, -p``: Built with ``--prefix`` (not portable)
+- ``--skip-so, -s``: Don't check all ``.so`` files
+
+
+**list** ``[FAMILY]`` - List available versions (default: cpython)::
+
+    portable-python list cpython
+
+- ``--json``: Output as JSON
+
+
+**diagnostics** - Show system diagnostics
+
+
+**recompress** ``<PATH> <EXT>`` - Re-compress existing binary tarball (for comparing compression sizes)
+
+
+**lib-auto-correct** ``<PATH>`` - Auto-correct exes/libs to use relative paths:
+
+- ``--commit``: Actually perform changes (dryrun by default)
+- ``--prefix, -p PATH``: Expected ``--prefix`` from build
+
+
 Library
 -------