AWS TIPS AND TRICKS: Automatically create a cron job at Instance creation

by | Jun 20, 2016

AWS
As mentioned in my previous blog AWS TIPS AND TRICKS: TIME ZONE I have had to do a few things I didn’t know how to do when setting up an EC2 Linux server and I thought I would share these learnings.

How to automatically create a cron job at Instance creation.

I needed to create a schedule to run a small python script every 5 min, the example I was using suggested:

echo “*/5 * * * * python my-python-script” >> /etc/crontab

I included this line in my user data section of the instance however when the instance started the script wasn’t running.
Taking a step back and I tried running the line of code manually, and no matter what I did (adding sudo etc) I always ended up with the same error:

/etc/crontab: Permission denied

Taking another step back, I asked:
Could I display/list the crontab for my user?

crontab -l

returned:

no crontab for ec2-user

Could I edit the crontab for my user?

crontab -e

Added the line:

*/5 * * * * python my-python-script

And saved, noted it returned:

crontab: installing new crontab

At this point the script was running, however, this was set-up manually how could I do automatically
One final step back, during my “googling” of my issues I noted there was mention of another way of using cron to schedule and it was by placing a file containing your cron code/schedule (with the addition of which user to run as) in the folder:

/etc/cron.d/

From what I can tell that this is a centralised way of maintaining cron schedules especially if you have multiple admin users rather than each admin adding to their own user crontab; they could add their schedule files in the cron.d folder which makes it easier for the other admins to find etc (I could be totally wrong with this understanding so let me know if I am).
As I was already uploading some files from s3 when the instance was created (via s3 sync command – another blog to be written) I saw using the cron.d folder as a potential option.
I created a file called ec2userschedule (note no use of special characters or even a file extension; it didn’t like it when it did) containing the schedule line with the addition of ec2-user

*/5 * * * * ec2-user python my-python-script

I added this file to my s3 bucket that contained the other files I was uploading.
The final thing I had to do was add a file move command to the instance user data after my s3 sync:

mv /home/ec2-user/ec2userschedule /etc/cron.d/

And to my surprise, my python script was run every 5 minutes, yay.
Cron job set at instance start-up. Job done.
If you are looking there is plenty of information in the land of google on how to use cron (all be it on different Linux flavours), I found what I needed by searching ‘cron syntax’.
Barry, Preventer of Chaos
Barry blogs about how to stop chaos in your systems.
You can read all of Barry’s blogs here.
Don’t forget, we can train your team in the art of agile business intelligence at any time!

4 Comments
  1. Maheswata

    I am trying to load the data from AWS S3 to redshift using data pipeline. But everytime I am getting an error, Syntax error or any other issue.

    Reply
  2. Barry Stevens

    Hi Maheswata,
    Thanks for taking the time to read my post, unfortunately without knowing your set-up or the error I am unable to help. May I suggest logging a call with AWS Support, they may be able to assist.
    Regards
    Barry

    Reply
  3. j

    Assuming sudo is installed, did you try sudo sh -c ‘echo “*/5 * * * * python my-python-script” >> /etc/crontab’, and if it isn’t installed, you can write the file to file readable by ec2-user and then run crontab . This would install the crontab for the user.

    Reply
  4. Barry Stevens

    Hi J. Thanks for taking the time to read my blog post and making this suggestion. Regards Barry

    Reply

Trackbacks/Pingbacks

  1. AWS TIPS AND TRICKS: Moving files from s3 to EC2 instance | OptimalBI - […] on from my previous post AWS TIPS AND TRICKS: Automatically create a cron job at Instance creation I mentioned I…
  2. Importing CSV file from AWS S3 into Redshift | OptimalBI - […] can read Barry’s blog, AWS TIPS AND TRICKS: Automatically create a cron job at Instance creation, or all of Barry’s…
Submit a Comment

Your email address will not be published. Required fields are marked *