How I Managed to cURL a File with cron

Herein, I recount how I managed to accomplish the simple task of setting up a recurring process to copy a file from a remote server. It does not offer any sort of overview on the tools I used. But the reader can consult information about cron and cURL. This document merely notes the resources I consulted in learning how to accomplish the task and the concrete steps I took to implement the solution. My hope is that, if someone faced the exact same problem as I did, they would be able to simply follow the steps I outline in the final section. Should the reader wish to know more about what is being done, then they can consult the resources listed and/or read my brief notes.

Resources I Consulted

Command to Copy a Remote File

cURL is “A command line tool for getting or sending files using URL syntax.” This tool is accessed from the cli using the curl command. I will also make use of the unix command >, which “sends the output of an executable to a file instead of putting it on the screen. This is called output redirection” (UNIX Commands).

The basic syntax of my command, if used at the command line, is as follows:

curl *[file to copy from]* > *[file to copy to]*

However, running this command produces additional informaton about the process, which I don’t to deal with when it’s run by cron, therefore, following ~cmr’s advice, I passed the command two flags: one to tell it to keep quite (--silent abbreviated to -s) the other to tell it to speak up if it encounters an error (--show-error abbreviated to -S). When run at the command line, the full command is (using the short forms of the flags in conjunction):

curl -Ss http://tilde.club/~pfhawkins/othertildes.json > /home/um/public_html/json/othertildes.json

Scheduling the Task with crontab

“Cron a time-based job scheduler in Unix-like computer operating systems.” Cron lets a schedule specified to process to be run at specified times. This schedule is kept in a users’ crontab file (short for “cron table”). crontab is the command used to edit this file.

To edit the user’s crontab file, or create a new one if none exists already, use the command

crontab -e

The syntax for scheduling cron jobs is dead simple:

*[time specification]* *[command or script to execute]*

The time specification can detail the minute, hour, day of month, month of year, and day of the week, but for tasks that don’t need such precision, we can use predefined scheduling definitions. This process should recur daily, so I will use the predefined value @daily. My crontab entry for this task is

@daily /usr/bin/curl -Ss http://tilde.club/~pfhawkins/othertildes.json > /home/um/public_html/json/othertildes.json

Note, after /usr/bin/curl, this is identical to the cli command given above. However, when scheduling tasks for cron, apparently we use the full path to the executable instead of the shorthand command. Thus, we substitute /usr/bin/curl for curl.

The Solution

The learning took awhile, but the doing was very simple. Here are all of the concrete steps needed to accomplish this task. (Commands entered into the terminal look like this: $ command args):