Repository navigation
Improve speed of stdlib functions by replacing re uses #130167
Description
We can often find the module re in the standard library modules but it can be replaced (if it is possible). I don't suggest removing it everywhere, there are places where its use is appropriate, but there are also places where it is an unnecessary solution and leads to unpleasant consequences (they can be found below)
Cons of regular expressions and reasons to replace regular expressions with functions and methods:
- We spend time to compile re pattern (one time, but anyway we spend it)
- In most cases simple string methods are faster (according to my benchmarks about 2x)
- We can remove
import rewhich will affect import time - Additionally: I think for those who don't know regular expressions, the code is more difficult to read and therefore difficult to maintain.
Important
For those who want to work on the issue, please:
- Read https://devguide.python.org/getting-started/pull-request-lifecycle/ before anything else.
- Select one function to improve. It's easier to review and possibly backport.
- Always report benchmarks using
pyperf,hyperfine, andtunatogether with-X importtimeto compare import times and execution time. - Open a pull request with the following title:
gh-130167: Improve speed of `module.function` by replacing `re`
Linked PRs
- gh-130167: Improve speed of
difflib.IS_LINE_JUNKby replacingre#130170 - gh-130167: Improve speed of
inspect.formatannotationby replacingre#130242 - gh-130167: Improve speed of
ftplib.parse150by replacingre#130243 - gh-130167: Optimise
textwrap.dedent()#131919 - gh-130167: Minor
textwrap.dedent()optimization #131925 - gh-130167: Add a What's New entry for changes to
textwrap.{de,in}dent#131924 - gh-130167: Improve speed of
_pydecimal._all_zerosand_pydecimal._exact_halfby replacingre#132065 - gh-130167: Improve the error case for
textwrap.dedent#132666 - [3.14] gh-130167: Add a What's New entry for changes to
textwrap.{de,in}dent(GH-131924) #143292
Activity
[-]Improve speed of functions that use `re` of various stdlib modules[/-][+]Improve speed of functions by replacing `re` uses[/+][-]Improve speed of functions by replacing `re` uses[/-][+]Improve speed of stdlib functions by replacing `re` uses[/+]terryjreedy commentedon Feb 16, 2025
replaceandtranslateto consider?Unless you find and list examples of useful replacements in an stdlib module, this is not properly an issue. If you have or do, make a PR. Or suggest a revision of the HOWTO section.
donbarbos commentedon Feb 16, 2025
@terryjreedy thank you for your comment and I know about the counterarguments you brought up but it doesn't change the increase in speed and the fact that we can get rid of unnecessary imports (sometimes or often, I don't know the statistics yet :)
I've only submitted one PR so far but I'm open to finding more examples since we use
rein the stdlib all the timepicnixz commentedon Feb 16, 2025
I removed the paragraph for new contributions as this issue is not necessary something we want to address immediately. Before opening a PR, benchmarks must be given and be convincing. If we lose the readability at the cost of an improved import time, I'm not sure it's worth unless we're like 5 or 6 times faster and if it's an important module that is usually imported and that
reis not usually imported.donbarbos commentedon Feb 16, 2025
Usually, i think, the readability of code without regular expressions is better. And from a performance standpoint, I think the speed of the function is the main improvement, although import time is also important.
you can find
import revery often and only one usepicnixz commentedon Feb 16, 2025
If you can remove
rewith simpler alternatives, then I'm fine. What I don't want to see is essentially PRs that would move lots ofreimports to local ones if they hurt readability and don't improve start up time by much.Note that it's also important to possibly plan for future extensions of the regex usage. Like getting the matched substring could be useful later. Now, I'm not against improving the various
reusages whenever possible but benchmarks must always be given (with sufficient coverage).picnixz commentedon Feb 16, 2025
I've restored your guidelines but added the requirements of having benchmarks
picnixz commentedon Feb 16, 2025
Considering I wrote #128983, I think we can treat it as a legit improvement. But let's only focus on changing re usages, not changing the imports or something else.
gh-130167: Optimise ``textwrap.dedent()`` (#131919)
gh-130167: Improve the error case for ``textwrap.dedent`` (#132666)
gh-130167: Improve ``difflib.IS_LINE_JUNK`` performance by using stri…
gh-130167: Add a What's New entry for changes to ``textwrap.{de,in}de…
pythongh-130167: Add a What's New entry for changes to ``textwrap.{de…
[3.14] gh-130167: Add a What's New entry for changes to ``textwrap.{d…
pythongh-130167: Add a What's New entry for changes to ``textwrap.{de…