Hi, thanks for the note. I would highly recommend not restricting the memory limit but reducing parallelization if you face memory issues, as it can lead to many false negatives in the grading.

Thanks for your reply.

1. The issue is not caused by parallelization. Even with 1 process, if the generated codes contain any zombie part, the IDLE issue will show up, and LCB cannot kill such zombie processes clearly.
2. Besides timeout, memory limit is also an important criteria in most coding competitions. If the generated code needs unlimited memory to pass tests, then the code may not be the correct answer.
3. Below is the result of Qwen-2.5-Coder-7B-Instruct (codegen 2025-01-01 -- 2025-05-01) under different memory limits, no significant differences are shown.

Based on above, I still suggest to add memory limit.

The issue is not caused by parallelization. Even with 1 process, if the generated codes contain any zombie part, the IDLE issue will show up, and LCB cannot kill such zombie processes clearly.

Thanks for following up. This is interesting, I think this behavior was not common for earlier models (since they used to perform poorly on hard problems) when I designed the autograder.

Besides timeout, memory limit is also an important criteria in most coding competitions. If the generated code needs unlimited memory to pass tests, then the code may not be the correct answer.

Yes, but I am not entirely sure what a good memory limit is when computing the results, given the small variance across memory limits (particularly when you do parallel execution). I will investigate this. Thanks for your thoughts!

Hi, I think this idea is helpful and I do face many OOM issues when testing it with gpt-4o-mini.