Welcome back to our k8s series. In this post, we’ll learn how Kubernetes Jobs and CronJobs work in practice.
· Understanding Jobs in Kubernetes
∘ Kubernetes Job Types
∘ Defining a Job resource
· CronJob in Kubernetes
∘ What are Kubernetes CronJobs
∘ Writing a CronJob spec
∘ How to Create CronJob in Kubernetes
· Spring Batch on Kubernetes
∘ Let’s Code
∘ Testing
· Conclusion
· References
This series of stories shows how to use Kubernetes in the Spring ecosystem. We work with a Spring Boot API and Minikube to have a lightweight and fast development environment similar to production.
- Lab1 (Spring Boot/K8S): Deploy Spring Boot application on Kubernetes
- Lab2 (Spring Boot/K8S): Kubernetes health probes with Spring Boot
- Lab3 (Spring Boot/K8S): Mastering ConfigMaps in Kubernetes
- Lab4 (Spring Boot/K8S): Using Kubernetes Secrets in Spring Boot
- Lab5 (Spring Boot/K8S): Understanding Kubernetes Resources Management
- Lab6 (Spring Boot/K8S): Persistent Volumes in Kubernetes
- 👉 Lab7 (Spring Boot/K8S): Spring Batch on Kubernetes — Jobs and CronJobs
A Kubernetes job or CronJob defines a specific task (called a Job) and ensures that a certain number of pods are stopped after the job is completed. These are two essential resources that enable efficient job scheduling within Kubernetes clusters.
Understanding Jobs in Kubernetes
Jobs are Kubernetes resources used to execute one-off tasks that must be reliably run to completion. A Job creates one or more Pods from a template and waits for at least a specified number of them to terminate successfully. The Job is then marked as complete.
A simple case is to create one Job object in order to reliably run one Pod to completion. The Job object will start a new Pod if the first Pod fails or is deleted (for example due to a node hardware failure or a node reboot).
Kubernetes Job Types
Kubernetes provides three main types of tasks suitable to run as a Job:
- Non-parallel Jobs — a Job that runs only one Pod unless the Pod fails. When the pod completes successfully, the Job is considered complete. There’s no parallelism involved.
- Parallel Jobs with a fixed number of completions — These are tasks that start and run several Pods in parallel. The Job continues running until a specified number of successful Pod
.spec.completionscompletions have occurred. When using.spec.completionMode="Indexed", each Pod gets a different index in the range 0 to.spec.completions-1. Another option is to set thespec.parallelismfield to define the number of parallel Jobs. - Parallel Jobs with a Work Queue — running multiple tasks in parallel. They’re used when the process consists of several tasks, but none are dependent on each other. The Pods must coordinate amongst themselves or an external service to determine what each should work on. For example, a Pod might fetch a batch of up to N items from the work queue. Once at least one Pod has terminated with success and all Pods are terminated, then the Job is completed with success.
Defining a Job resource
Here is an example of the Job manifest:
apiVersion: batch/v1
kind: Job
metadata:
name: batch-job-pi
spec:
template:
spec:
containers:
- name: pi
image: perl:5.34.0
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: NeverAs with all other Kubernetes configs, a Job needs apiVersion, kind, and metadata fields. Jobs are part of the batch API group and v1API version.
The YAML defines a resource of type Job that will run the perl:5.34.0 image, which computes π to 2000 places and prints it out. It takes around 10 seconds to complete.
$ kubectl apply -f batch-job-pi.yml
job.batch/batch-job-pi createdTo check job status with kubectl:
We can also check the status of the job through the Dashboard:
As we see if you leave both .spec.completions and .spec.parallelism unset. Both are defaulted to 1.
For a fixed completion count Job, you should set .spec.completions to the number of completions needed. You can set .spec.parallelism, or leave it unset and it will default to 1.
For a work queue Job, you must leave .spec.completions unset, and set .spec.parallelism to a non-negative integer.
CronJob in Kubernetes
What are Kubernetes CronJobs
Kubernetes CronJobs are a way to run a task on a time-based schedule and have been around for a long time in Linux and UNIX systems. They can be used to run recurring tasks such as backup jobs, triggering emails, report generation, or automating restarts of containers. They run a Job periodically on a given schedule, written in Cron format.
Writing a CronJob spec
The .spec.schedule field is required. The value of that field follows the Cron syntax:
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
# │ │ │ │ │ OR sun, mon, tue, wed, thu, fri, sat
# │ │ │ │ │
# │ │ │ │ │
# * * * * *For example, 0 0 13 * 5 states that the task must be started every Friday at midnight, as well as on the 13th of each month at midnight.
Other than the standard syntax, some macros like @monthly can also be used:
To generate CronJob schedule expressions, you can also use web tools like crontab.guru.
How to Create CronJob in Kubernetes
The following example of a CronJob manifest will print the current time and a hello message every minute.
apiVersion: batch/v1
kind: CronJob
metadata:
name: hello-cronjob
spec:
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox:1.28
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailureNote the kind is set to CronJob and the .spec.schedule is set to run the job every minute. The .spec.schedule is a required field of the .spec.
$ kubectl apply -f hello-cronjob.yml
cronjob.batch/hello-cronjob createdTo check job status with kubectl:
cronjobs run every minute and create associated jobs and pods. The ACTIVE field shows how many jobs are in progress, 0 meaning it has either already been completed or failed.
Spring Batch on Kubernetes
In this section, we’ll use my previous Spring batch story that read from MongoDB and generate CSV files.
Spring Batch 5 — Read from MongoDB and generate CSV files: Part 2
We will make some updates to the code, containerize it, and deploy it to Kubernetes with jobs. The goal is to learn how to use Kubernetes jobs to run Spring batch applications.
We’ll add a new feature that consists of sending the CSV report to email.
public void sendNotificationEmailReport(String fileToAttach) {
MimeMessagePreparator preparator = mimeMessage -> {
mimeMessage.setFrom(new InternetAddress(MAIL_FROM));
mimeMessage.setRecipient(Message.RecipientType.TO, new InternetAddress(MAIL_RECIPIENT));
mimeMessage.setSubject(MessageFormat.format("{0}{1}", MAIL_SUBJECT, fileDateFormat));
try {
FileSystemResource file = new FileSystemResource(new File(fileToAttach));
MimeMessageHelper helper = new MimeMessageHelper(mimeMessage, true);
helper.addAttachment(Objects.requireNonNull(file.getFilename()), file);
helper.setText("Hi Team, <br><br> Please find attached the daily report. <br> Regards.", true);
} catch (Exception ex) {
LOGGER.error(">> Unable to send report by email {} ", ex.getMessage());
}
};
mailSender.send(preparator);
}Let’s Code
The main code update is to remove the scheduler. Kubernetes Cronjob will replace it.
Here is the complete job manifest content:
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
# property-like keys; each key maps to a simple value
# replace the config values with yours
host: cluster0.xpkhnmq.mongodb.net
database: sample_training
option: retryWrites=true&w=majority
smtp-host: vps-3904db98.vps.ovh.net
smtp-port: "1025"
---
apiVersion: v1
kind: Secret
metadata:
name: app-secret
type: Opaque
data:
# replace the secret values with yours
username: ZGJ1c2VybmFtZQ==
password: ZGJwYXNzd29yZA==
smtp-username: YWRtaW4=
smtp-password: bXkgcGFzc3dvcmQ=
---
apiVersion: batch/v1
kind: Job
metadata:
name: spring-batch-job
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: spring-batch5-mongodb
image: spring-batch5-mongodb:latest
imagePullPolicy: Never
env: # array of environment variable definitions
- name: MONGODB_URL
valueFrom: # select individual keys in a ConfigMap
configMapKeyRef:
name: app-config
key: host
- name: MONGODB_DATABASE
valueFrom:
configMapKeyRef:
name: app-config
key: database
- name: MONGODB_OPTS
valueFrom:
configMapKeyRef:
name: app-config
key: option
- name: SMTP_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: smtp-host
- name: SMTP_PORT
valueFrom:
configMapKeyRef:
name: app-config
key: smtp-port
- name: MONGODB_LOGIN
valueFrom:
secretKeyRef:
name: app-secret
key: username
- name: MONGODB_PWD
valueFrom:
secretKeyRef:
name: app-secret
key: password
- name: SMTP_USERNAME
valueFrom:
secretKeyRef:
name: app-secret
key: smtp-username
- name: SMTP_PASSWORD
valueFrom:
secretKeyRef:
name: app-secret
key: smtp-passwordFor this lab, we use the Mongo database from Atlas.
Testing
$ kubectl apply -f k8s/spring-batch-job.yml
configmap/app-config created
secret/app-secret created
job.batch/spring-batch-job createdTo check job status.
Pod Output Logs:
-15:51:21.847 [main] INFO o.h.validator.internal.util.Version.<clinit> - HV000001: Hibernate Validator 8.0.1.Final
-15:51:23.207 [main] INFO c.b.s.SpringBatch5MongodbApplication.logStarted - Started SpringBatch5MongodbApplication in 13.411 seconds (process running for 16.759)
-15:51:23.249 [main] INFO o.s.b.a.b.JobLauncherApplicationRunner.run - Running default command line with: []
-15:51:23.415 [main] INFO o.s.b.c.l.support.SimpleJobLauncher.run - Job: [SimpleJob: [name=tripJob]] launched with the following parameters: [{'run.id':'{value=1, type=class java.lang.Long, identifying=true}'}]
-15:51:23.478 [main] INFO c.b.s.job.TripJobCompletionListener.beforeJob - -------------------------------------------------- JOB BATCH LABS - Spring Batch Version 5.1.0 ----------------------------------------------------
-15:51:23.479 [main] INFO c.b.s.job.TripJobCompletionListener.beforeJob - >> START JOB EXECUTION BATCH LABS - Spring Batch Version 5.1.0 AT 2024-07-14T15:51:23.478393313
-15:51:23.479 [main] INFO c.b.s.job.TripJobCompletionListener.beforeJob - -----------------------------------------------------------------------------------------------------------------------------
-15:51:23.507 [main] INFO o.s.batch.core.job.SimpleStepHandler.handleStep - Executing step: [tripJobCSVGenerator]
-15:51:23.513 [main] INFO c.b.s.job.step.TripStepListener.beforeStep - ################################################# STEP TRIP SUMMARY ###################################################
-15:51:23.545 [main] INFO c.b.s.job.step.TripStepListener.beforeStep - >> START STEP: Trip step process ...
-15:51:23.546 [main] INFO c.b.s.job.step.TripStepListener.beforeStep - >> START TIME: 2024-07-14T15:51:23.513612452
-15:51:23.547 [main] INFO c.b.s.job.step.TripStepListener.beforeStep - #############################################################################################################################
-15:51:28.260 [main] INFO c.b.s.job.step.TripItemWriter.afterStepExecution - logger null
-15:51:30.684 [main] INFO c.b.s.job.step.TripStepListener.afterStep - ################################################# STEP TRIP SUMMARY ###################################################
-15:51:30.686 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> TRIP STEP SUMMARY
-15:51:30.686 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> END TIME: 2024-07-14T15:51:30.685841891
-15:51:30.687 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> TOTAL DURATION: 7 seconds.
-15:51:30.687 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> ExitStatus: COMPLETED
-15:51:30.687 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> ReadCount: 3821
-15:51:30.688 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> ReadSkipCount: 0
-15:51:30.688 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> WriteCount: 3821
-15:51:30.689 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> WriteSkipCount: 0
-15:51:30.690 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> Summary: StepExecution: id=1, version=9, name=tripJobCSVGenerator, status=COMPLETED, exitStatus=COMPLETED, readCount=3821, filterCount=0, writeCount=3821 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=8, rollbackCount=0
-15:51:30.690 [main] INFO c.b.s.job.step.TripStepListener.afterStep - >> END STEP: Trip step process
-15:51:30.691 [main] INFO c.b.s.job.step.TripStepListener.afterStep - #############################################################################################################################
-15:51:30.696 [main] INFO o.s.batch.core.step.AbstractStep.execute - Step: [tripJobCSVGenerator] executed in 7s188ms
-15:51:30.710 [main] INFO c.b.s.job.TripJobCompletionListener.afterJob - -------------------------------------------------- JOB BATCH LABS - Spring Batch Version 5.1.0 ----------------------------------------------------
-15:51:30.711 [main] INFO c.b.s.job.TripJobCompletionListener.afterJob - >> END JOB EXECUTION BATCH LABS - Spring Batch Version 5.1.0
-15:51:30.713 [main] INFO c.b.s.job.TripJobCompletionListener.afterJob - >> Total trips: null
-15:51:30.714 [main] INFO c.b.s.job.TripJobCompletionListener.afterJob - >> TOTAL JOB PROCESSING DURATION BATCH LABS - Spring Batch Version 5.1.0 IN 7 second(s).
-15:51:30.714 [main] INFO c.b.s.job.TripJobCompletionListener.afterJob - >> EXECUTION OF JOB BATCH LABS - Spring Batch Version 5.1.0 COMPLETED SUCCESSFULLY
-15:51:30.715 [main] INFO c.b.s.job.TripJobCompletionListener.afterJob - -----------------------------------------------------------------------------------------------------------------------------
-15:51:30.723 [main] INFO o.s.b.c.l.support.SimpleJobLauncher.run - Job: [SimpleJob: [name=tripJob]] completed with the following parameters: [{'run.id':'{value=1, type=class java.lang.Long, identifying=true}'}] and the following status: [COMPLETED] in 7s250ms
-15:51:30.923 [main] INFO o.s.j.d.e.EmbeddedDatabaseFactory.shutdownDatabase - Shutting down embedded database: url='jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=false'The mail has been sent successfully with the CSV file.
The full CronJob manifest in case we want to schedule the job. In this case, the tasks will be executed every 10 minutes.
apiVersion: batch/v1
kind: CronJob
metadata:
name: spring-batch-cron-job
spec:
schedule: '10 * * * *'
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 5
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: spring-batch5-mongodb
image: 'spring-batch5-mongodb:latest'
imagePullPolicy: Never
env:
- name: MONGODB_URL
valueFrom:
configMapKeyRef:
name: app-config
key: host
- name: MONGODB_DATABASE
valueFrom:
configMapKeyRef:
name: app-config
key: database
- name: MONGODB_OPTS
valueFrom:
configMapKeyRef:
name: app-config
key: option
- name: SMTP_HOST
valueFrom:
configMapKeyRef:
name: app-config
key: smtp-host
- name: SMTP_PORT
valueFrom:
configMapKeyRef:
name: app-config
key: smtp-port
- name: MONGODB_LOGIN
valueFrom:
secretKeyRef:
name: app-secret
key: username
- name: MONGODB_PWD
valueFrom:
secretKeyRef:
name: app-secret
key: password
- name: SMTP_USERNAME
valueFrom:
secretKeyRef:
name: app-secret
key: smtp-username
- name: SMTP_PASSWORD
valueFrom:
secretKeyRef:
name: app-secret
key: smtp-passwordConclusion
In this story, we’ve looked at Kubernetes Jobs and Cronjobs. These are good choices for automating and scheduling tasks in your Kubernetes cluster.
The complete source code is available on GitHub.
Support me through GitHub Sponsors.
Thank you for Reading !! See you in the next story.
