How To Run A Python Script With Dependencies In A Virtual Environment In Nifi?

Is there a way in Nifi to run a python script which has modules imported from a different folder, requirements specified in a pipfile and has arguments to pass? In short, how to ex

Solution 1:

To perform follow-on processing on the flowfile using Python, you can use the ExecuteStreamCommand or ExecuteScript/InvokeScriptedProcessor processors.

The ExecuteStreamCommand processor will run an external shell command, like python3 my_python_script.py -arg1 string -arg2 213, which can wrap custom Python code and uses STDIN to pass the existing flowfile content and STDOUT to capture the new flowfile content. Populate the Command Arguments and Command Path properties of the processor to locate your python executable and provide CLI arguments, including flowfile attributes via NiFi Expression Language. See this answer for an example.

The ExecuteScript processor runs Jython code (Python but without access to native libraries, only Python 2.7 compatibility, and some other restrictions due to JSR-223) in the same JVM as NiFi. You can process the flowfile attributes and content directly with Python code. See this answer or this answer for more details.