Check and change the Python recursion limit (e.g. sys.setrecursionlimit)

Money and Business

In Python, there is an upper limit to the number of recursions (the maximum number of recursions). To execute a recursive function with a large number of calls, it is necessary to change the limit. Use the functions in the sys module of the standard library.

The number of recursions is also limited by the stack size. In some environments, the resource module of the standard library can be used to change the maximum stack size (it worked on Ubuntu, but not on Windows or mac).

The following information is provided here.

  • Get the upper limit of the current number of recursions:sys.getrecursionlimit()
  • Change the upper limit of the number of recursions:sys.setrecursionlimit()
  • Change the maximum size of the stack:resource.setrlimit()

The sample code is running on Ubuntu.

Get the current recursion limit: sys.getrecursionlimit()

The current recursion limit can be obtained with sys.getrecursionlimit().

import sys
import resource

print(sys.getrecursionlimit())
# 1000

In the example, the maximum number of recursions is 1000, which may vary depending on your environment. Note that the resource we are importing here will be used later, but not on Windows.

As an example, we will use the following simple recursive function. If a positive integer n is specified as an argument, the number of calls will be n times.

def recu_test(n):
    if n == 1:
        print('Finish')
        return
    recu_test(n - 1)

An error (RecursionError) will be raised if you try to perform recursion more than the upper limit.

recu_test(950)
# Finish

# recu_test(1500)
# RecursionError: maximum recursion depth exceeded in comparison

Note that the value obtained by sys.getrecursionlimit() is not strictly the maximum number of recursions, but the maximum stack depth of the Python interpreter, so even if the number of recursions is slightly less than this value, an error (RecursionError) will be raised.

再帰限界は、再帰の限界ではなく、pythonインタープリタのスタックの最大深度です。
python – Max recursion is not exactly what sys.getrecursionlimit() claims. How come? – Stack Overflow

# recu_test(995)
# RecursionError: maximum recursion depth exceeded while calling a Python object

Change recursion limit: sys.setrecursionlimit()

The upper limit of the number of recursions can be changed by sys.setrecursionlimit(). The upper limit is specified as an argument.

Allows deeper recursion to be performed.

sys.setrecursionlimit(2000)

print(sys.getrecursionlimit())
# 2000

recu_test(1500)
# Finish

If the specified upper limit is too small or too large, an error will occur. This constraint (upper and lower limits of the limit itself) varies depending on the environment.

The maximum value of limit depends on the platform. If you need deep recursion, you can specify a larger value within the range supported by the platform, but be aware that this value will cause a crash if it is too large.
If the new limit is too low at the current recursion depth, a RecursionError exception is raised.
sys.setrecursionlimit() — System-specific parameters and functions — Python 3.10.0 Documentation

sys.setrecursionlimit(4)
print(sys.getrecursionlimit())
# 4

# sys.setrecursionlimit(3)
# RecursionError: cannot set the recursion limit to 3 at the recursion depth 1: the limit is too low

sys.setrecursionlimit(10 ** 9)
print(sys.getrecursionlimit())
# 1000000000

# sys.setrecursionlimit(10 ** 10)
# OverflowError: signed integer is greater than maximum

The maximum number of recursions is also limited by the stack size, as explained next.

Change the maximum size of the stack: resource.setrlimit()

Even if a large value is set in sys.setrecursionlimit(), it may not be executed if the number of recursions is large. A segmentation fault occurs as follows.

sys.setrecursionlimit(10 ** 9)
print(sys.getrecursionlimit())
# 1000000000

recu_test(10 ** 4)
# Finish

# recu_test(10 ** 5)
# Segmentation fault

In Python, the resource module in the standard library can be used to change the maximum stack size. However, the resource module is a Unix-specific module and cannot be used on Windows.

With resource.getrlimit(), you can get the limit of the resource specified in the argument as a tuple of (soft limit, hard limit). Here, we specify resource.RLIMIT_STACK as the resource, which represents the maximum size of the call stack of the current process.

print(resource.getrlimit(resource.RLIMIT_STACK))
# (8388608, -1)

In the example, the soft limit is 8388608 (8388608 B = 8192 KB = 8 MB) and the hard limit is -1 (unlimited).

You can change the limit of the resource with resource.setrlimit(). Here, the soft limit is also set to -1 (no limit). You can also use the constant resource.RLIM_INFINIT to represent the unlimited limit.

Deep recursion, which could not be performed due to segmentation fault before the stack size change, can now be performed.

resource.setrlimit(resource.RLIMIT_STACK, (-1, -1))

print(resource.getrlimit(resource.RLIMIT_STACK))
# (-1, -1)

recu_test(10 ** 5)
# Finish

Here, the soft limit is set to -1 (no limit) for a simple experiment, but in reality, it would be safer to limit it to an appropriate value.

In addition, when I tried to set an unlimited soft limit on my mac as well, the following error occurred.ValueError: not allowed to raise maximum limit
Running the script with sudo didn't help. It may be restricted by the system.

A process with the effective UID of a superuser can request any reasonable limit, including no limit.
However, a request that exceeds the limit imposed by the system will still result in a ValueError.
resource.setrlimit() — Resource usage information — Python 3.10.0 Documentation

Windows doesn't have a resource module, and mac couldn't change the maximum stack size due to system limitations. If we can increase the stack size by some means, we should be able to solve the segmentation fault, but we have not been able to confirm this.