This example CrushTask shows how to setup variables which facilitate in running multiple jobs servers in different data centers with full redundancy.#

---

For this example we have two data centers, DC1, and DC2. In each data center there are two instances of CrushFTP running, Main1 and Main2.

Requirements:#

All four servers are using replication and cross pointing at each other.
Each pair of Main servers in each DC has ServerBeat configured using the "JOB" VIP (or a real VIP) so that only one Main server in each DC is actually running jobs.

Expectations:#

We have jobs that should only ever run in DC1, and jobs that should only ever run in DC2...never on the opposite
We have jobs that should run in DC1, unless DC1 is down...then they should run in DC2.
We have jobs that should run in DC2, unless DC2 is down...then they should run in DC1.

---

To accomplish this we will use variables for the "enabled" flag in all our jobs....specifically global variables. There will be another job that runs every minute which is configuring variables for the current status of things.


The "controller" job on these variables utilizes the hostname of the current machine to determine if its running in a dc1 or dc2 environment. This is "row 1" area of the job. This is critical that the hostname has some marker you can use to distinguish which DC you are running in. The very first part of the job simply looks at the current hostname to decide which one we are in and sets a variable for future decisions. This controller job runs frequently, so you should turn off saving state between steps and saving state at the end. Maybe keep them enabled which configuring, but then turn those off to not have a lot of job pollution.


OK, finally...now into the job logic. The logic is "simple" and yet complex at the same time. Please follow along.
(Row 2 area)
1.) We check if we are in DC1. If we are, we enable the flags for dc1_only and dc1_unless to be true. (We are in DC1, so those are definitely true.)
2.) If we are not in DC1, then we need to check if DC1 is online or not...is a server in DC1 responding? We do this by checking the list of reporting hostnames in the variable: server_active_replicated_hostnames
2.a.) If we see a server responding from DC1, then we set the flag dc1_unless to false.
2.b.) If we don't see any servers from DC1, then we set the flag dc1_unless to true.

Now we repeat for the opposite scenario of DC2.
(Row 3 area)
1.) We check if we are in DC2. If we are, we enable the flags for dc2_only and dc2_unless to be true. (We are in DC2, so those are definitely true.)
2.) If we are not in DC2, then we need to check if DC2 is online or not...is a server in DC2 responding? We do this by checking the list of reporting hostnames in the variable: server_active_replicated_hostnames
2.a.) If we see a server responding from DC2, then we set the flag dc2_unless to false.
2.b.) If we don't see any servers from DC2, then we set the flag dc2_unless to true.

And here is the overall job flow:

Sample job.XML can be downloaded here: attachments