Import Error Loading Module Using Bash Script On EC2 Instance (AWS)

I am trying to run a bash script through the user_data argument of the spot_fleet_request method. The script executes all the installations and connects to the filesystem, but when

Solution 1:

I think the recommended method is of course to try to package your code as a proper module and install it prior with pip, either at the user/system level or a pythonenv style environment. This of course, can be highly difficult and time consuming depending on how much code you have to change to accommodate this, and its recommended that you try to do this at the beginning of making projects to avoid this headache.

Short of the doing the nontrivial amount of work that would be though, I actually have encountered this specific error you're having on Amazon's EC2 LTS ubuntu. No idea what causes it, but its like python just side steps the normal environment settings, and doesn't work as you expect. In my case, I just went for the dirty fix of:

import os

os.environ['PYTHONPATH'] = '{}:{}'.format('my/addition/to/pythonpath', os.environ.get('PYTHONPATH', ''))
print(os.environ['PYTHONPATH'])
# prints my/addition/to/pythonpath:existing/things/on/pythonpath

so for your usecase, try:

import os

os.environ['PYTHONPATH'] = '{}:{}'.format('/home/ec2-user/efs/Day-Trader/day_trader/', os.environ.get('PYTHONPATH', ''))
print(os.environ['PYTHONPATH'])
# prints /home/ec2-user/efs/Day-Trader/day_trader/:

you can of course do some pretty nasty things with this, such as:

python -c "import os; os.environ['PYTHONPATH'] = '/my/addition/to/pythonpath'; import mymodule; mymodule.doThing()'

if for some reason you dislike everyone who has the displeasure of working with this. Jokes aside, I really did use things like this in production, if you every figure out what the hell causes these env issues, I'd be interested to know even though I no longer work on the machines that had these issues.