Skip to content

Use RestrictedPython instead of eval to evaluate python code on API specification#798

Merged
Pradhvan merged 9 commits into
scanapi:mainfrom
Pradhvan:Issue-51
Aug 20, 2025
Merged

Use RestrictedPython instead of eval to evaluate python code on API specification#798
Pradhvan merged 9 commits into
scanapi:mainfrom
Pradhvan:Issue-51

Conversation

@Pradhvan
Copy link
Copy Markdown
Member

@Pradhvan Pradhvan commented Aug 12, 2025 •
edited
Loading

Description

Security: Replace unsafe eval() with RestrictedPython

  • Use compile_restricted() + sandboxed execution context
  • Limit available modules/functions via allowlists
  • Enable safe generator expressions

Motivation behind this PR?

The CodeEvaluator class uses unsafe eval(), which poses security risks when evaluating user-provided Python code in API specifications.

What type of change is this?

Could be a Breaking Change since we refactored CodeEvaluator class. Would need to test thoroughly.

Checklist

  • A changelog entry was added, or this PR does not require one. Instructions
  • Unit tests were added or updated as needed, or not required for this change. Instructions
  • All unit tests pass locally. Instructions
  • Docstrings or comments were added or updated as needed, or no documentation changes were required. Instructions
  • This PR does not significantly reduce code or docstring coverage.
  • Code follows the project’s style guidelines.
  • ScanAPI was run locally and the changes were manually verified, or this was not necessary. Instructions
  • Commits were squashed, or squashing was not necessary (e.g., only one commit). Instructions

Issue

Closes #51

@Pradhvan
Add RestrictedPython dependency
6495e1b 
- pyproject.toml: Restrict Python version range to >=3.10,<3.14 to ensure
  compatibility with RestrictedPython package requirements.
@Pradhvan
Update CodeEvaluator to use RestrictedPython
5e24842 
- compile_restricted() prevents dangerous code compilation
- safe_globals/safe_builtins limit available functions
- Sandboxed execution environment for `mode='eval'`
@Pradhvan
Dynamically import modules for API spec evaluation
3fb112a 
- Remove top imports with `# noqa: F401`
- Created ALLOWED_MODULES to store configurable modules to import
- Add `_get_allowed_modules()` helper method
@Pradhvan Pradhvan marked this pull request as ready for review August 14, 2025 15:10
@Pradhvan Pradhvan requested review from a team as code owners August 14, 2025 15:10
@Pradhvan Pradhvan requested a review from camilamaia August 14, 2025 15:11
camilamaia
camilamaia reviewed Aug 15, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@camilamaia camilamaia left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, this looks great! I can't believe this is finally happening. Thanks a lot for the implementation, @Pradhvan 🙇‍♀️ I was feeling a bit rusty with ScanAPI, so it took me a while to jump in, but glad I did!

Thinking about the tests — maybe we could expand test_code_evaluator.py with some extra cases, like:

  • Allowed modules
    Confirm only the modules in ALLOWED_MODULES are usable.
    Examples: datetime.datetime.now(), math.sqrt(16), random.randint(1, 10), re.match("a", "abc"), time.time(), uuid.uuid4()

  • Disallowed modules
    Ensure importing/using modules not in ALLOWED_MODULES raises InvalidPythonCodeError.
    Examples: import os; os.system("ls"), import sys

  • Safe built-ins only
    Verify that dangerous built-ins are blocked.
    Examples: open("file.txt"), exec("print(1)"), eval("2+2")

  • Restricted attributes
    Block access to dunder and non-whitelisted attributes/methods.
    Examples: __import__('os').system('ls'), ().__class__.__mro__, (lambda:0).__globals__

  • Side effects
    Prevent unintended state mutation or persistence between runs.
    Examples: globals()["x"] = 1, setting a var and checking if it persists across executions

  • Error messages
    Ensure raised errors include both a helpful message and the offending code — and it looks like the offending code part isn’t currently covered by tests.

  • Unicode handling
    Guarantee correct behavior with Unicode and non-ASCII content.
    Examples: u"áéíóú"[::-1], identifiers like á = 42

  • Isolation
    Ensure evaluations are independent between runs.
    Examples: len("a") then len("bb") without state leaking

Also, we can open an issue at the https://github.com/scanapi/examples for adding some examples that would test some of these cases (I can work on it if you like the idea).

What do you think?

Comment thread scanapi/evaluators/code_evaluator.py
Comment thread scanapi/evaluators/code_evaluator.py
@Pradhvan
Copy link
Copy Markdown
Member Author

Pradhvan commented Aug 18, 2025

Wow, this looks great! I can't believe this is finally happening. Thanks a lot for the implementation, @Pradhvan 🙇‍♀️ I was feeling a bit rusty with ScanAPI, so it took me a while to jump in, but glad I did!

Thinking about the tests — maybe we could expand test_code_evaluator.py with some extra cases, like:

  • Allowed modules
    Confirm only the modules in ALLOWED_MODULES are usable.
    Examples: datetime.datetime.now(), math.sqrt(16), random.randint(1, 10), re.match("a", "abc"), time.time(), uuid.uuid4()
  • Disallowed modules
    Ensure importing/using modules not in ALLOWED_MODULES raises InvalidPythonCodeError.
    Examples: import os; os.system("ls"), import sys
  • Safe built-ins only
    Verify that dangerous built-ins are blocked.
    Examples: open("file.txt"), exec("print(1)"), eval("2+2")
  • Restricted attributes
    Block access to dunder and non-whitelisted attributes/methods.
    Examples: __import__('os').system('ls'), ().__class__.__mro__, (lambda:0).__globals__
  • Side effects
    Prevent unintended state mutation or persistence between runs.
    Examples: globals()["x"] = 1, setting a var and checking if it persists across executions
  • Error messages
    Ensure raised errors include both a helpful message and the offending code — and it looks like the offending code part isn’t currently covered by tests.
  • Unicode handling
    Guarantee correct behavior with Unicode and non-ASCII content.
    Examples: u"áéíóú"[::-1], identifiers like á = 42
  • Isolation
    Ensure evaluations are independent between runs.
    Examples: len("a") then len("bb") without state leaking

What do you think?

Sounds like a plan @camilamaia 🙌🏾, I can start with updating unit tests from the above mentioned lists. Post that I can take a poke at updating integration tests with the examples repo.

Also, we can open an issue at the https://github.com/scanapi/examples for adding some examples that would test some of these cases (I can work on it if you like the idea).

Feel free to open up an issue and I can update it once we are done with this PR.

@Pradhvan
test_code_evaluator.py: Update unit tests
f9fa0f9 
- Updated Class InvalidPythonCodeError to have `self.expression = code`
- Added unit tests to check updated code_evaluators changes.
@Pradhvan
Copy link
Copy Markdown
Member Author

Pradhvan commented Aug 18, 2025 •
edited
Loading

@camilamaia unit tests updated and logging added. 🚀

For the logging part do you want to review this in the same PR ? or we can open up a new issue for it and work on it separately.

@Pradhvan Pradhvan requested a review from camilamaia August 18, 2025 12:38
camilamaia
camilamaia approved these changes Aug 19, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@camilamaia camilamaia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NICEEEEEEEEEE, let's merge it!! 😍

@Pradhvan Pradhvan merged commit 0f45246 into scanapi:main Aug 20, 2025
9 checks passed
@camilamaia camilamaia mentioned this pull request Aug 20, 2025
Open
5 tasks
@camilamaia
Copy link
Copy Markdown
Member

camilamaia commented Aug 20, 2025

Niceee! I've just added the example issue here: scanapi/examples#32

@dinalivia dinalivia mentioned this pull request May 11, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@camilamaia camilamaia camilamaia approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Stop using eval to evaluate python code on API specification

2 participants

@Pradhvan @camilamaia