Getting Original Line Number For Exception In Concurrent.futures
Solution 1:
I was in your same situation and I really needed to have the traceback of the raised exceptions.
I was able to develop this workaround which consists in using the following subclass of the
ThreadPoolExecutor
.
import sys
import traceback
from concurrent.futures import ThreadPoolExecutor
classThreadPoolExecutorStackTraced(ThreadPoolExecutor):
defsubmit(self, fn, *args, **kwargs):
"""Submits the wrapped function instead of `fn`"""returnsuper(ThreadPoolExecutorStackTraced, self).submit(
self._function_wrapper, fn, *args, **kwargs)
def_function_wrapper(self, fn, *args, **kwargs):
"""Wraps `fn` in order to preserve the traceback of any kind of
raised exception
"""try:
return fn(*args, **kwargs)
except Exception:
raise sys.exc_info()[0](traceback.format_exc()) # Creates an# exception of the# same type with the# traceback as# message
If you use this subclass and run the following snippet:
deff(x):
return x * x
data = [1, 2, 3, None, 5]
with ThreadPoolExecutorStackTraced(max_workers=len(data)) as executor:
futures = [executor.submit(f, n) for n in data]
for future in futures:
try:
print future.result()
except TypeError as e:
print e
the output will be something like:
149
Traceback (most recent call last):
File "future_traceback.py", line 17, in _function_wrapper
return fn(*args, **kwargs)
File "future_traceback.py", line 24, in f
return x * x
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'25
The problem is in the usage of sys.exc_info()
by the futures
library. From the
documentation:
This function returns a tuple of three values that give information about the exception that is currently being handled. [...] If no exception is being handled anywhere on the stack, a tuple containing three None values is returned. Otherwise, the values returned are (type, value, traceback). Their meaning is: type gets the exception type of the exception being handled (a class object); value gets the exception parameter (its associated value or the second argument to raise, which is always a class instance if the exception type is a class object); traceback gets a traceback object which encapsulates the call stack at the point where the exception originally occurred.
Now, if you look at the source code of futures
you can see by yourself why the traceback is
lost: when an exception raises and it is to be set to the Future
object only
sys.exc_info()[1]
is passed. See:
https://code.google.com/p/pythonfutures/source/browse/concurrent/futures/thread.py (L:63) https://code.google.com/p/pythonfutures/source/browse/concurrent/futures/_base.py (L:356)
So, to avoid losing the traceback, you have to save it somewhere. My workaround is to wrap
the function to submit into a wrapper whose only task is to catch every kind of exception and
to raise an exception of the same type whose message is the traceback. By doing this, when an
exception is raised it is captured and reraised by the wrapper, then when sys.exc_info()[1]
is assigned to the exception of the Future
object, the traceback is not lost.
Solution 2:
I think the original exception traceback gets lost in the ThreadPoolExecutor code. It stores the exception and then re-raises it later. Here is one solution. You can use the traceback module to store the original exception message and traceback from your function f into a string. Then raise an exception with this error message, which now contains the line number etc of f. The code that runs f can be wrapped in a try...except block, which catches the exception raised from ThreadPoolExecutor, and prints the message, which contains the original traceback.
The code below works for me. I think this solution is a little hacky, and would prefer to be able to recover the original traceback, but I'm not sure if that is possible.
import concurrent.futures
import sys,traceback
deff(x):
try:
return x * x
except Exception, e:
tracebackString = traceback.format_exc(e)
raise StandardError, "\n\nError occurred. Original traceback is\n%s\n" %(tracebackString)
data = [1, 2, 3, None, 5] # line 10with concurrent.futures.ThreadPoolExecutor(len(data)) as executor: # line 12try:
futures = [executor.submit(f, n) for n in data] # line 13for future in futures: # line 14print(future.result()) # line 15except StandardError, e:
print"\n"print e.message
print"\n"
This gives the following output in python2.7:
149Error occurred. Original traceback is
Traceback (most recent call last):
File "thread.py", line 8, in f
return x * x
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'
The reason your original code gives the right location when run in Python 3 and not 2.7 is that in Python 3 exceptions carry the traceback as an attribute, and when re-raising an exception, the traceback is extended rather than replaced. The example below illustrates this:
defA():
raise BaseException("Fish")
defB():
try:
A()
except BaseException as e:
raise e
B()
I ran this in python 2.7 and python 3.1. In 2.7 the output is as follows:
Traceback (most recent calllast):
File "exceptions.py", line 11, in<module>
B()
File "exceptions.py", line 9, in B
raise e
BaseException: Fish
i.e. the fact that the exception was originally thrown from A is not recorded in the eventual output. When I run with python 3.1 I get this:
Traceback (most recent call last):
File "exceptions.py", line 11, in <module>
B()
File "exceptions.py", line 9, in B
raise e
File "exceptions.py", line 7, in B
A()
File "exceptions.py", line 3, in A
raise BaseException("Fish")
BaseException: Fish
which is better. If I replace raise e
with just raise
in the except block in B, then python2.7 gives the complete traceback. My guess is that when back-porting this module for python2.7 the differences in exception propagating were overlooked.
Solution 3:
Taking inspiration from the first answer, here it is as a decorator:
import functools
import traceback
defreraise_with_stack(func):
@functools.wraps(func)defwrapped(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
traceback_str = traceback.format_exc(e)
raise StandardError("Error occurred. Original traceback ""is\n%s\n" % traceback_str)
return wrapped
Just apply the decorator on the executed function:
@reraise_with_stackdeff():
pass
Post a Comment for "Getting Original Line Number For Exception In Concurrent.futures"