How Do I Stop Elastic Beanstalk (apparently) Removing My Downloaded File?

I have these container commands in an .ebextensions .config file: container_commands: download_geography_data_01: command: 'python scripts/download_geography_data.py' downl

Solution 1:

AWS Elastic Beanstalk have never removed your file, except for the app directory.

As AWS Documentation said in Customizing Software on Linux Servers:

You can use the container_commands key to execute commands for your container. The commands in container_commands are processed in alphabetical order by name. They run after the application and web server have been set up and the application version file has been extracted, but before the application version is deployed. They also have access to environment variables such as your AWS security credentials. Additionally, you can use leader_only. One instance is chosen to be the leader in an Auto Scaling group. If the leader_only value is set to true, the command runs only on the instance that is marked as the leader.

You put your SQLite database inside your app directory. When you update to a new one, your change into your old directory will be destroyed.

Before deploying:

[ec2-user@ip-172-31-52-184python]$ll/opt/python/total20drwxr-xr-x2rootroot4096 Jul3009:09bindrwxr-xr-x3rootroot4096 Jul3009:12bundlelrwxrwxrwx1rootroot20Jul3009:12current->/opt/python/bundle/3drwxr-xr-x2rootroot4096 Jul3009:09etcdrwxrwxr-x3rootwsgi4096 Jul3009:10logdrwxr-xr-x4rootroot4096 Jul3009:10run

On deploying:

[ec2-user@ip-172-31-52-184 python]$ ll /opt/python/
total 20
drwxr-xr-x 2 root root 4096 Jul  30 09:09 bin
drwxr-xr-x 4 root root 4096 Jul  30 09:18 bundle
lrwxrwxrwx 1 root root   20 Jul  30 09:12 current -> /opt/python/bundle/3
drwxr-xr-x 2 root root 4096 Jul  30 09:09 etc
drwxrwxr-x 3 root wsgi 4096 Jul  30 09:10 log
lrwxrwxrwx 1 root root   20 Jul  30 09:18 ondeck -> /opt/python/bundle/4

After deployed:

[ec2-user@ip-172-31-52-184python]$ll/opt/python/total20drwxr-xr-x2rootroot4096 Jul3009:09bindrwxr-xr-x3rootroot4096 Jul3009:18bundlelrwxrwxrwx1rootroot20Jul3009:18current->/opt/python/bundle/4drwxr-xr-x2rootroot4096 Jul3009:09etcdrwxrwxr-x3rootwsgi4096 Jul3009:10logdrwxr-xr-x4rootroot4096 Jul3009:10run

Do you notice your /opt/python/current/app changed? While on deploying, your /opt/python/current/app is pointed to /opt/python/bundle/3. And then after deployed, it is pointed to /opt/python/bundle/4. And then, the /opt/python/bundle/3 (your old app) is removed.

So, the solution might be:

• Don't put your database inside your app directory. Just put anywhere outside your app directory.
• Run the command after deployed (see here).

UPDATE: First time deployment and first time update

TLDR: If you create a new environment, the /opt/python/current was pointed to /opt/python/bundle/1 and then it will be updated to /opt/python/bundle/2. So, before your app is deployed, there is a "default" app inside the instance.

I create a simple .ebextensions script to list all object inside /opt/python:

.ebextensions/env-check.config

files:
  "/opt/env-checker/env-checker.sh" :
    content : |
      #!/usr/bin/env bash
      TIMESTAMP=`date +%Y%m%d_%H%M%S_%N`
      if [ -d /opt/python ]
      thenecho"PWD: $PWD" >> /opt/env-checker/${TIMESTAMP}_${1}ls -al /opt/python >> /opt/env-checker/${TIMESTAMP}_${1}elseecho"PWD: $PWD" >> /opt/env-checker/${TIMESTAMP}_${1}ls -al /opt >> /opt/env-checker/${TIMESTAMP}_${1}fi
    mode : "000755"
    owner : root
    group : root
  "/opt/elasticbeanstalk/hooks/appdeploy/post/post-check.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      /opt/env-checker/env-checker.sh 02_post_deploy

commands:
  000_create_post_dir:
    command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
  100_commands_envc:
    command: /opt/env-checker/env-checker.sh 00_commands

container_commands:
  101_container_commands_envc:
    command: /opt/env-checker/env-checker.sh 01_container_commands

In the first deployment (create a new environment using the above .ebentensions), I got this:

20150730_141343_068692315_00_commands:

PWD:/total28drwxr-xr-x7rootroot4096 Jul3014:13.drwxr-xr-x6rootroot4096 Jul3014:13..drwxr-xr-x2rootroot4096 Jul3014:13bindrwxr-xr-x3rootroot4096 Jul3014:13bundlelrwxrwxrwx1rootroot20Jul3014:13current->/opt/python/bundle/1drwxr-xr-x2rootroot4096 Jul3014:13etcdrwxrwxr-x3rootwsgi4096 Jul3014:13logdrwxr-xr-x4rootroot4096 Jul3014:13run

20150730_141350_596104854_01_container_commands

PWD:/opt/python/bundle/2/apptotal28drwxr-xr-x7rootroot4096 Jul3014:13.drwxr-xr-x6rootroot4096 Jul3014:13..drwxr-xr-x2rootroot4096 Jul3014:13bindrwxr-xr-x4rootroot4096 Jul3014:13bundlelrwxrwxrwx1rootroot20Jul3014:13current->/opt/python/bundle/1drwxr-xr-x2rootroot4096 Jul3014:13etcdrwxrwxr-x3rootwsgi4096 Jul3014:13loglrwxrwxrwx1rootroot20Jul3014:13ondeck->/opt/python/bundle/2drwxr-xr-x4rootroot4096 Jul3014:13run

20150730_141355_731024404_02_post_deploy

PWD:/total28drwxr-xr-x7rootroot4096 Jul3014:13.drwxr-xr-x6rootroot4096 Jul3014:13..drwxr-xr-x2rootroot4096 Jul3014:13bindrwxr-xr-x3rootroot4096 Jul3014:13bundlelrwxrwxrwx1rootroot20Jul3014:13current->/opt/python/bundle/2drwxr-xr-x2rootroot4096 Jul3014:13etcdrwxrwxr-x3rootwsgi4096 Jul3014:13logdrwxr-xr-x4rootroot4096 Jul3014:13run

In the second deployment (first update), I got this:

20150730_141535_631911395_00_commands

PWD:/total28drwxr-xr-x7rootroot4096 Jul3014:13.drwxr-xr-x6rootroot4096 Jul3014:13..drwxr-xr-x2rootroot4096 Jul3014:13bindrwxr-xr-x3rootroot4096 Jul3014:13bundlelrwxrwxrwx1rootroot20Jul3014:13current->/opt/python/bundle/2drwxr-xr-x2rootroot4096 Jul3014:13etcdrwxrwxr-x3rootwsgi4096 Jul3014:13logdrwxr-xr-x4rootroot4096 Jul3014:13run

20150730_141542_258594223_01_container_commands

PWD:/opt/python/bundle/3/apptotal28drwxr-xr-x7rootroot4096 Jul3014:15.drwxr-xr-x6rootroot4096 Jul3014:13..drwxr-xr-x2rootroot4096 Jul3014:13bindrwxr-xr-x4rootroot4096 Jul3014:15bundlelrwxrwxrwx1rootroot20Jul3014:13current->/opt/python/bundle/2drwxr-xr-x2rootroot4096 Jul3014:13etcdrwxrwxr-x3rootwsgi4096 Jul3014:13loglrwxrwxrwx1rootroot20Jul3014:15ondeck->/opt/python/bundle/3drwxr-xr-x4rootroot4096 Jul3014:13run

20150730_141547_336930605_02_post_deploy

PWD:/total28drwxr-xr-x7rootroot4096 Jul3014:15.drwxr-xr-x6rootroot4096 Jul3014:13..drwxr-xr-x2rootroot4096 Jul3014:13bindrwxr-xr-x3rootroot4096 Jul3014:15bundlelrwxrwxrwx1rootroot20Jul3014:15current->/opt/python/bundle/3drwxr-xr-x2rootroot4096 Jul3014:13etcdrwxrwxr-x3rootwsgi4096 Jul3014:13logdrwxr-xr-x4rootroot4096 Jul3014:13run

From the above result and some checking inside EC2 instance, I can conclude that:

• There is default app /opt/python/bundle/1 directory before your app come into /opt/python.
• The commands in .ebextensions will be executed before your app file is in /opt/python.
• The working directory of commands is in /.
• The container_commands in .ebextensions will be executed after your app file is extracted in /opt/python.
• The working directory of container_commands is in your latest app directory in /opt/python.
• EB will remove your old app directory after the app is deployed.

I thought, another solution: you can put your SQLite database using current working directory of your script.