Define themes as JSON data, loadable from files and packages - #15356

Themes were Python dicts mapping pygments token objects to style strings, built eagerly at IPython.utils.PyColorize import. Adding one meant editing IPython's source — which docs/source/config/details.rst already apologised for:

while currently you need to modify source code to add a theme, it should be possible to load theme from Json, Yaml, or any other declarative file type.

This does that.

Themes are data

Each theme is a JSON file in IPython/utils/themes/. Token types are dotted paths with the leading Token. dropped:

{
  "name": "gruvbox-dark",
  "base": "gruvbox-dark",
  "extra_style": { "Prompt": "#689D6A", "Prompt.Continuation.L1": "#D79921" },
  "symbols": { "top_line": "─", "arrow_body": "─", "arrow_head": "▶" }
}

Theme.from_dict / Theme.from_file do the conversion, resolving tokens only when a theme is actually looked up.

The shipped themes are unchanged. The JSON was generated from the existing definitions rather than transcribed, then all nine themes were diffed against a pre-change snapshot: zero differences.

Three places a theme can come from

1. Bundled — the JSON above.
2. Your IPython directory — drop ~/.ipython/themes/my-theme.json and %colors my-theme works, no code to write. Listed on each lookup, so a file added mid-session is found without restarting.
3. An installed package — via the new ipython.themes entry point group. The entry point is named after the theme; its value says where the JSON lives:

[project.entry-points."ipython.themes"]
solarized-dark = "my_themes"              # my_themes/solarized-dark.json
solarized-light = "my_themes:data"        # my_themes/data/solarized-light.json

A theme package contains no code — JSON files and an empty __init__.py. IPython reads the data and never imports an object from it or calls into it.

Neither external source is consulted until a lookup misses the built-ins, so the ~25 ms importlib.metadata scan and the IPython directory both stay off the startup path. Precedence is bundled → IPython directory → installed package, so %colors linux always means the linux IPython ships.

Validation

A theme is now data that can arrive from elsewhere, so it is checked before it loads and refused with a warning if it fails.

Token paths must match [A-Z]\w* per segment — what pygments itself tests to decide whether an attribute access names a subtoken. Without it, a key like "split.__globals__" or "__class__.__base__" resolved to whatever getattr found and walked off the token graph into the pygments module globals. Not exploitable (attribute reads alone give no call or subscript primitive), but a theme file should be inert by construction rather than by argument.

Symbols are written straight to the terminal, and make_arrow repeats arrow_body besides, so they must be at most 20 printable characters — otherwise a theme could smuggle in an escape sequence and set the window title, move the cursor, or read back the clipboard via OSC 52.

str.isprintable() is the obvious check and is wrong here: it rejects the private use area, where Powerline separators and Nerd Font glyphs live, which is exactly what someone wants an arrow head to be. Rejecting unicode categories Cc, Cf, Cs, Cn, Zl, Zp covers what an escape sequence is built from while leaving private use alone; Cf also blocks the bidi overrides.

Theme names become file names, so they must not name a directory — files(pkg) / "../../../x.json" resolves outside the package rather than failing.

Notes for review

• theme_table is a lazy Mapping rather than a dict. Lookup, iteration and .keys() work as before; it can no longer be mutated with __setitem__, which nothing in the tree did.
• The former module-level theme objects (linux_theme, neutral_theme, …) stay reachable through a module __getattr__, without building every theme on import. Nothing in-tree used them.
• Deliberately not a performance change: pygments.token costs 0.56 ms and building all nine themes well under 1 ms. The two expensive imports (pygments.styles at 27 ms, pygments.formatters at 29 ms) were already deferred. This is for extensibility.
• Packaging: pyproject.toml gains "IPython.utils" = ["themes/*.json"]; MANIFEST.in's graft IPython covers the sdist.

Verified end to end against a genuinely pip-installed theme package and a real IPYTHONDIR, not just stubs. Tests are in tests/test_themes.py; the validation guards were checked by mutation — disabling any one of them fails between 3 and 16 tests.