Skip to content
Navigation Menu
Sign in
Appearance settings
Platform
AI CODE CREATION
GitHub Copilot
Write better code with AI
GitHub Copilot app
Direct agents from issue to merge
MCP Registry
Integrate external tools
DEVELOPER WORKFLOWS
Actions
Automate any workflow
Codespaces
Instant dev environments
Issues
Plan and track work
Code Review
Manage code changes
Code Quality
Enforce quality at merge
APPLICATION SECURITY
GitHub Advanced Security
Find and fix vulnerabilities
Code security
Secure your code as you build
Secret protection
Stop leaks before they start
EXPLORE
Why GitHub
Documentation
Blog
Changelog
Marketplace
View all features
Solutions
BY COMPANY SIZE
Enterprises
Small and medium teams
Startups
Nonprofits
BY USE CASE
App Modernization
DevSecOps
DevOps
CI/CD
View all use cases
BY INDUSTRY
Healthcare
Financial services
Manufacturing
Government
View all industries
View all solutions
Resources
EXPLORE BY TOPIC
AI
Software Development
DevOps
Security
View all topics
EXPLORE BY TYPE
Customer stories
Events & webinars
Ebooks & reports
Business insights
GitHub Skills
SUPPORT & SERVICES
Documentation
Customer support
Community forum
Trust center
Partners
View all resources
Open Source
COMMUNITY
GitHub Sponsors
Fund open source developers
PROGRAMS
Security Lab
Maintainer Community
Accelerator
GitHub Stars
Archive Program
REPOSITORIES
Topics
Trending
Collections
Enterprise
ENTERPRISE SOLUTIONS
Enterprise platform
AI-powered developer platform
AVAILABLE ADD-ONS
GitHub Advanced Security
Enterprise-grade security features
Copilot for Business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Pricing
Type
/
to search
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading.
Please reload this page
.
python
/
cpython
Public
Uh oh!
There was an error while loading.
Please reload this page
.
Notifications
You must be signed in to change notification settings
Fork
35.2k
Star
74.3k
Code
Issues
5k+
Pull requests
2.5k
Actions
Projects
Security and quality
0
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Security and quality
Insights
Commits
Breadcrumbs
History for
cpython
Python
Python-tokenize.c
on
main
User selector
All users
All time
Commit history
Commits on Mar 24, 2026
gh-128341: Use _Py_ABI_SLOT in stdlib modules (#145770)
Show description for 1887a95
befeleme
and
vstinner
authored
1887a95
View commit details
Copy full SHA for 1887a95
View code at this point
Browse repository at this point
Commits on Mar 11, 2026
GH-145247: Use _PyTuple_FromPair in Parser and Python (#145842)
Show description for d19de37
sergey-miryanov
authored
d19de37
View commit details
Copy full SHA for d19de37
View code at this point
Browse repository at this point
Commits on Mar 19, 2025
gh-111178: Fix function signatures for test_types (#131455)
vstinner
authored
bbe5baa
View commit details
Copy full SHA for bbe5baa
View code at this point
Browse repository at this point
Commits on Oct 9, 2024
gh-115754: Use Py_GetConstant(Py_CONSTANT_EMPTY_STR) (#125194)
Show description for b9a8ca0
vstinner
authored
b9a8ca0
View commit details
Copy full SHA for b9a8ca0
View code at this point
Browse repository at this point
Commits on Jul 16, 2024
gh-120317: Lock around global state in the tokenize module (#120318)
Show description for 8549559
lysnikolaou
and
pablogsal
authored
8549559
View commit details
Copy full SHA for 8549559
View code at this point
Browse repository at this point
Commits on Jun 12, 2024
gh-120343: Fix column offsets of multiline tokens in tokenize (#120391)
lysnikolaou
authored
4b5d3e0
View commit details
Copy full SHA for 4b5d3e0
View code at this point
Browse repository at this point
Commits on Jun 11, 2024
gh-120343: Do not reset byte_col_offset_diff after multiline tokens (#120352)
Show description for 1b62bce
lysnikolaou
and
blurb-it[bot]
authored
1b62bce
View commit details
Copy full SHA for 1b62bce
View code at this point
Browse repository at this point
Commits on May 29, 2024
gh-119704: Fix reference leak in the ``Python/Python-tokenize.c`` (#119705)
Eclips4
authored
c0faade
View commit details
Copy full SHA for c0faade
View code at this point
Browse repository at this point
Commits on May 28, 2024
gh-119118: Fix performance regression in tokenize module (#119615)
Show description for d87b015
lysnikolaou
and
pablogsal
authored
d87b015
View commit details
Copy full SHA for d87b015
View code at this point
Browse repository at this point
Commits on May 3, 2024
gh-116322: Add Py_mod_gil module slot (#116882)
Show description for c2627d6
swtaarrs
authored
c2627d6
View commit details
Copy full SHA for c2627d6
View code at this point
Browse repository at this point
Commits on Dec 11, 2023
gh-112943: Correctly compute end offsets for multiline tokens in the tokenize module (#112949)
pablogsal
authored
a135a6d
View commit details
Copy full SHA for a135a6d
View code at this point
Browse repository at this point
Commits on Nov 2, 2023
Remove unnecessary includes (GH-111633)
serhiy-storchaka
authored
a12f624
View commit details
Copy full SHA for a12f624
View code at this point
Browse repository at this point
Commits on Oct 11, 2023
gh-104169: Refactor tokenizer into lexer and wrappers (#110684)
Show description for 01481f2
3 people
authored
01481f2
View commit details
Copy full SHA for 01481f2
View code at this point
Browse repository at this point
Commits on Jul 26, 2023
gh-107015: Remove async_hacks from the tokenizer (#107018)
pablogsal
authored
da8f87b
View commit details
Copy full SHA for da8f87b
View code at this point
Browse repository at this point
Commits on Jun 9, 2023
gh-105564: Don't include artificial newlines in the line attribute of tokens (#105565)
pablogsal
authored
d7f46bc
View commit details
Copy full SHA for d7f46bc
View code at this point
Browse repository at this point
Commits on Jun 7, 2023
gh-105390: Add explicit type cast (#105466)
Eclips4
authored
264a011
View commit details
Copy full SHA for 264a011
View code at this point
Browse repository at this point
gh-105435: Fix spurious NEWLINE token if file ends with comment without a newline (#105442)
pablogsal
authored
7279fb6
View commit details
Copy full SHA for 7279fb6
View code at this point
Browse repository at this point
gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (#105399)
pablogsal
authored
ffd2654
View commit details
Copy full SHA for ffd2654
View code at this point
Browse repository at this point
Commits on Jun 6, 2023
gh-105259: Ensure we don't show newline characters for trailing NEWLINE tokens (#105364)
pablogsal
authored
c0a6ed3
View commit details
Copy full SHA for c0a6ed3
View code at this point
Browse repository at this point
Commits on May 30, 2023
gh-105042: Disable unmatched parens syntax error in python tokenize (#105061)
lysnikolaou
authored
70f315c
View commit details
Copy full SHA for 70f315c
View code at this point
Browse repository at this point
gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070)
pablogsal
authored
9216e69
View commit details
Copy full SHA for 9216e69
View code at this point
Browse repository at this point
Commits on May 28, 2023
gh-105017: Include CRLF lines in strings and column numbers (#105030)
Show description for 96fff35
mgmacias95
and
pablogsal
authored
96fff35
View commit details
Copy full SHA for 96fff35
View code at this point
Browse repository at this point
Commits on May 26, 2023
gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (#104980)
Show description for 46b52e6
pablogsal
authored
46b52e6
View commit details
Copy full SHA for 46b52e6
View code at this point
Browse repository at this point
gh-104972: Ensure that line attributes in tokens in the tokenize module are correct (#104975)
pablogsal
authored
3fdb55c
View commit details
Copy full SHA for 3fdb55c
View code at this point
Browse repository at this point
Commits on May 24, 2023
gh-104825: Remove implicit newline in the line attribute in tokens emitted in the tokenize module (#104846)
pablogsal
authored
c8cf9b4
View commit details
Copy full SHA for c8cf9b4
View code at this point
Browse repository at this point
Commits on May 22, 2023
gh-104741: Add line number attribute to indentation error exception (#104743)
mgmacias95
authored
729b252
View commit details
Copy full SHA for 729b252
View code at this point
Browse repository at this point
gh-102856: Tokenize performance improvement (#104731)
mgmacias95
authored
8817886
View commit details
Copy full SHA for 8817886
View code at this point
Browse repository at this point
Commits on May 21, 2023
gh-102856: Python tokenizer implementation for PEP 701 (#104323)
Show description for 6715f91
mgmacias95
and
pablogsal
authored
6715f91
View commit details
Copy full SHA for 6715f91
View code at this point
Browse repository at this point
Commits on May 5, 2023
gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)
Show description for a9c6e06
ericsnowcurrently
authored
a9c6e06
View commit details
Copy full SHA for a9c6e06
View code at this point
Browse repository at this point
Commits on Apr 19, 2023
gh-102856: Initial implementation of PEP 701 (#102855)
Show description for 1ef61cf
5 people
authored
1ef61cf
View commit details
Copy full SHA for 1ef61cf
View code at this point
Browse repository at this point
Commits on Oct 6, 2022
gh-97973: Return all necessary information from the tokenizer (GH-97984)
Show description for cbf0afd
lysnikolaou
authored
cbf0afd
View commit details
Copy full SHA for cbf0afd
View code at this point
Browse repository at this point
Commits on Aug 11, 2022
gh-90928: Statically Initialize the Keywords Tuple in Clinic-Generated Code (gh-95860)
Show description for 6f6a4e6
ericsnowcurrently
authored
6f6a4e6
View commit details
Copy full SHA for 6f6a4e6
View code at this point
Browse repository at this point
Commits on Feb 11, 2022
bpo-46613: Add PyType_GetModuleByDef to the public API (GH-31081)
Show description for 2049469
encukou
and
vstinner
authored
2049469
View commit details
Copy full SHA for 2049469
View code at this point
Browse repository at this point
Commits on Oct 13, 2021
bpo-45434: Mark the PyTokenizer C API as private (GH-28924)
Show description for 713bb19
vstinner
authored
713bb19
View commit details
Copy full SHA for 713bb19
View code at this point
Browse repository at this point
Commits on Oct 3, 2021
Remove trailing spaces. (GH-28706)
serhiy-storchaka
authored
a5a5615
View commit details
Copy full SHA for a5a5615
View code at this point
Browse repository at this point
Previous
Next
You can’t perform that action at this time.