Semra Kermo

When and why do we need background jobs? Nowadays, background job processing is indispensable in the world of web development. The need for background jobs stems from the fact that synchronous execution of time-consuming and resource-intensive tasks would heavily impact an application's  performance and user experience.  Even though Elixir is famous for its concurrency model and can easily handle multiple processes simultaneously, there are still several reasons to consider adding a third-party tool like Oban to manage and monitor their background processes. What is Oban and its advantages? What if a process fails or the server crashes? Oban provides elegant solutions for those and any similar problems, all with a convenient UI for managing and monitoring said processes and load. Oban is a highly efficient Ecto-based background job processing system. It is famous for its intuitiveness and reliability.  One of the advantages of Oban is that it uses a database (PostgreSQL or SQLite) to store and manage its background jobs. The aftereffect of that is the mentioned reliability. Because the jobs are persisted in the database, if something were to happen, like the server crashing or a job failing, the job data wouldn’t be lost. With that, a lot of the complexity regarding background job processing is already taken away by Oban. Other key features Oban offers are job scheduling, job prioritization, periodic and unique jobs, concurrency control, etc. Oban Integration With Oban being Elixir-based and built on top of PostgreSQL, and Elixir applications mostly using PostgreSQL in their stack anyway, the integration of Oban into your project should be very simple and straightforward. This article will explore Oban integration on an imaginary Elixir Airline application.  Essentially, when purchasing a ticket at the airline’s website, the airline sends an email confirmation to the user. The whole process is done consecutively, which is very unnecessary in this case, having that the email confirmation could easily be delegated to a background process, and thus, making the process of the purchase faster and giving the user a better experience at using the application. Let’s investigate how we would achieve this in our Elixir app using Oban. Getting started Firstly, we will need to add the Oban dependency into our project. To do so, we need to locate our mix.exs file and add the following:  # mix.exs def deps do   [     {:oban, "~> 2.16"}   ] end After adding the dependency, we will need to run mix deps.get. Because Oban uses the database to store jobs, we need to add a database migration to add the needed oban_jobs table. With this table, Oban will track everything it needs to know regarding our jobs, such as the state of the job, when it was attempted to execute the job, how many times it was attempted, etc. mix ecto.gen.migration add_oban_jobs_table defmodule Airline.Repo.Migrations.AddObanJobsTable do   use Ecto.Migration   def up do     Oban.Migration.up(version: 11)   end   def down do     Oban.Migration.down(version: 1)   end end Now, we need to run the migration using mix ecto.migrate. Configurations We have successfully added Oban as a dependency in our project. The next step is adjusting some configurations. In our config.exs file, the next configuration is needed: # config/config.exs config :airline, Oban,   repo: Airline.Repo,   plugins: [Oban.Plugins.Pruner, max_age: 300],   queues: [purchase_ticket: 10] The queues option marks the maximum number of concurrent jobs allowed in the queue (purchase_ticket queue in this case).  The Pruner allows us to state for how much time we want to keep completed, cancelled or discarded jobs in our database. This prevents our database from growing indefinitely. By default, the Pruner will retain jobs only for 60 seconds, but we can change this by adding the max_age option (300 seconds in this case, or 5 minutes) or it can also be specified by the number of rows we want to keep, for instance: max_len: 3000 means we only store the most recent 3000 jobs. Also worth mentioning is that the Pruner will never delete a new job, a scheduled job or a job which failed and will be retried. The only thing between us and setting up a new Oban job is adding Oban to our supervisor. In our application.ex we have to add it to the list of supervised children: # lib/my_app/application.ex def start(_type, _args) do   children = [     Airline.Repo,     {Oban, Application.fetch_env!(:airline, Oban)}   ] end Defining Workers Intuitively, the Oban module which does the work of processing jobs is called a Worker. To use it, we need to define our own module, let’s say PurchaseTicketWorker and use the Oban.Worker module. In the module, we call which process we want to call in the background in the purchase_ticket queue. Each Worker must define a perform/1 function. It should accept an Oban.Job struct with a list of arguments. The arguments can be used in a variety of ways, for example, differentiating what should be done in the perform function and they will be stored in the database. We will not use them in this case. defmodule Airline.PurchaseTicketWorker do   use Oban.Worker, queue: :purchase_ticket   @impl Oban.Worker   def perform(%Oban.Job{args: _args} = job) do     # dbg(job)     with :ok <- Airline.PurchaseTicket.send_email() do       :ok     end   end end To handle errors and job retries, we just have to add another option to the job, max_attempts.  Now, if the job execution fails for whatever reason, and the number of attempts to execute the job is lower than max_attempts, the job will automatically be retried in the future and the error will be logged in the database. use Oban.Worker, queue: :purchase_ticket, max_attempts: 3 After executing our job, if dbg(job) is uncommented, we should get the following output: [lib/airline/purchase_ticket_worker.ex:6: Airline.PurchaseTicketWorker.perform/1] job #=> %Oban.Job{   __meta__: #Ecto.Schema.Metadata<:loaded, "public", "oban_jobs">,   id: 4,   state: "executing",   queue: "purchase_ticket",   worker: "Airline.PurchaseTicketWorker",   args: %{},   meta: %{},   tags: [],   errors: [],   attempt: 1,   attempted_by: ["DESKTOP-8JVA10V", "bd6da488-083b-484f-b688-996a63997048"],   max_attempts: 3,   priority: 0,   attempted_at: ~U[2024-01-28 21:29:07.048000Z],   cancelled_at: nil,   completed_at: nil,   discarded_at: nil,   inserted_at: ~U[2024-01-28 21:29:06.991971Z],   scheduled_at: ~U[2024-01-28 21:29:06.991971Z],   conf: %Oban.Config{     dispatch_cooldown: 5,     engine: Oban.Engines.Basic,     get_dynamic_repo: nil,     insert_trigger: true,     log: false,     name: Oban,     node: "DESKTOP-8JVA10V",     notifier: {Oban.Notifiers.Postgres, []},     peer: {Oban.Peers.Postgres, []},     plugins: [{Oban.Plugins.Pruner, []}],     prefix: "public",     queues: [purchase_ticket: [limit: 10]],     repo: Airline.Repo,     shutdown_grace_period: 15000,     stage_interval: 1000,     testing: :disabled   },   conflict?: false,   replace: nil,   unique: nil,   unsaved_error: nil } Enqueuing Now, to start using our job, we need to enqueue it. Job creation is rather simple. A Job is just an Ecto struct which gets enqueued by inserting it into the database. Oban workers provide a new/2 function which takes in a map of arguments (the same arguments that will be sent to the perform function) and converts it into an Oban.Job which can be inserted into the database using Oban.insert/1. # purchasing ticket code %{} |> Airline.PurchaseTicketWorker.new() |> Oban.insert() Now, after adding this to our code which handles ticket purchasing, whenever someone purchases a ticket, the email confirmation task will be offloaded to a separate process and handled in the background. With this, the process of ticket purchasing will not be waiting on the email confirmation task to be finished, thus becoming faster to execute. Another thing worth mentioning, when the need arises, as our application grows and we create multiple jobs, there will come a need to prioritize some jobs over the others. Usually, all jobs are executed in the order in which they were scheduled. But luckily, giving preference to one job over another is fairly simple, it can be defined while defining the queue in the Worker or we can give an individual job a priority when enqueueing it. use Oban.Worker, queue: :purchase_ticket, priority: 3 Oban.insert(Airline.PurchaseTicketWorker.new(%{}, queue: :purchase_ticket, priority: 2)) Jobs with the highest priority have a value of 0, and the lowest priority comes to the jobs which have the value of 9. Scheduling But, what happens if we want to send an email to all users who purchased a ticket for a specific flight 24 hours before the flight, reminding them to check in? Well, Oban also has a very useful feature which is called job scheduling. Again, a very intuitive name as it’s used to schedule jobs for the future. The syntax is very similar, we just need to specify when the job should execute, either with schedule_in which accepts seconds, or with scheduled_at which accepts a DateTime. In our example, we would use scheduled_at and subtract 24 hours from the flight’s departure time. Another important thing to mention is that job scheduling is always done in UTC. This means that we will have to shift our DateTime struct to use UTC. %{} |> Airline.PurchaseTicketWorker.new(scheduled_at:  DateTime.shift_zone!(day_before_departure_time, "Etc/UTC") ) |> Oban.insert() Periodic jobs If we wanted to send a daily email digest to every subscriber of our airline newsletter, standard scheduling wouldn’t cut it. That’s where periodic jobs come in. Oban’s Cron plugin can schedule and enqueue jobs automatically. We declare periodic jobs either as {cron, worker} or {cron, worker, opts}. So, to send out daily emails, the declaration would look something like this: config :airline, Oban,   repo: Airline.Repo,   plugins: [     {Oban.Plugins.Cron,     crontab: [       {"@daily", Airline.DailyWorker, max_attempts: 2}     ]}   ] To set a more specific periodic job, for example, to have a job on every Monday or every hour, you can read about the Cron Syntax on the official Oban documentation. "Background Jobs in Elixir - Oban" Tech Bite was brought to you by Semra Kermo, Junior Software Engineer at Atlantbh. (more…)