Kubernetes CronJob monitoring
kubectl get cronjobs shows when a CronJob last ran, but nothing pages you when it stops. A heartbeat check does: the Job pings QuickStatus.dev when it succeeds, and if the ping doesn't arrive on time, you get an alert. Because the alert comes from outside the cluster, it still fires when the whole cluster, or your Prometheus, is down.
How a CronJob fails quietly
- The pod exits non-zero until the Job hits its
backoffLimit. - The image tag was deleted and the pod sits in
ImagePullBackOff. - Someone set
suspend: trueduring an incident and forgot. - The controller missed the start time by more than
startingDeadlineSecondsand skipped the run. - The previous run is still going and
concurrencyPolicy: Forbidskips the next one. - The node pool, or the whole cluster, is gone.
Every one of these ends the same way: no successful Job, so no ping.
Set it up
- In QuickStatus.dev, click Add check, then Heartbeat. Name it after the CronJob.
- Pick Expected every to match the schedule, for example 1 day for
0 3 * * *. - Pick Grace to cover the job's run time plus its retries, for example 1 hour.
- Copy the ping URL (it's shown only once) and store it as a Secret:
kubectl create secret generic quickstatus-heartbeat \
--from-literal=url='<ping URL>'
Then run your job as an init container and the ping as the main container. Kubernetes only starts the main container after the init container exits 0, so a failed run never pings.
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-report
spec:
schedule: "0 3 * * *"
timeZone: "Etc/UTC"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 600
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
template:
spec:
restartPolicy: Never
initContainers:
- name: job
image: ghcr.io/acme/nightly-report:1.4.2
args: ["run"]
containers:
- name: heartbeat
image: curlimages/curl:8.11.1
env:
- name: PING_URL
valueFrom:
secretKeyRef:
name: quickstatus-heartbeat
key: url
args: ["-fsS", "-m", "10", "--retry", "3", "$(PING_URL)"]
Replace the init container's image and args with your own job. Your job image doesn't need curl. If it does have curl and a shell, you can skip the init container and ping from the job's own command, which also reports failures straight away:
if your-job; then curl -fsS -m 10 --retry 3 "$PING_URL"; else curl -fsS -m 10 --retry 3 "$PING_URL/fail"; exit 1; fi
What it catches
- Jobs that fail until
backoffLimit - Pods stuck in
ImagePullBackOfforPending - A CronJob left on
suspend: true - Runs skipped by
startingDeadlineSecondsorconcurrencyPolicy: Forbid - Jobs killed by
activeDeadlineSeconds - A cluster, or an in-cluster monitoring stack, that is down
Pricing
The Free plan gives you 10 checks of any type, heartbeats included, plus email and Telegram alerts (2 chats) and 14 days of results, with no card needed. Pro is $15 a month for 25 checks, a 1-minute minimum for Expected every, 90 days of results, and Slack, Discord, PagerDuty, Pushover, webhook and unlimited Telegram alerts. See pricing
Want to watch the services in the cluster too? Add HTTP or TCP checks for your ingress hosts in the same account; Kubernetes ingress monitoring from outside shows which ones. Turn a schedule into a CronJob snippet with the cron expression generator, and see cron job monitoring for other schedulers.
Kubernetes CronJob FAQ
How do I get alerted when a Kubernetes CronJob fails?
Have the Job ping a heartbeat URL only when it succeeds. If the ping doesn't arrive within Expected every plus Grace, QuickStatus.dev alerts you, whatever the reason the Job didn't succeed.
Why not just use Prometheus and kube-state-metrics?
If you already run them, alerting on kube_cronjob_status_last_successful_time works well. They run inside the cluster, though, so they can't tell you the cluster itself is down. A heartbeat from outside covers that, and it's a good second signal even with Prometheus.
Does my job image need curl?
No. In the example, a separate curlimages/curl container sends the ping after your job's container succeeds.
What about the CronJob's time zone?
Set spec.timeZone to an IANA name such as “Etc/UTC”. Without it, the schedule follows the time zone of the kube-controller-manager.
Can I monitor a weekly CronJob?
Not yet. Expected every goes up to 1 day, so heartbeat checks fit CronJobs that run at least once a day.