Overview

For this example I'll use what was created at BlueSky firehose. This site (code name icky-venus) runs a simple GenServer that registers for events on a registry. These events fire every time there is a post created on BlueSky. The GenServer holds a state of the total count for an indefinite amount of days.

The issue is if that the site ever shuts down, it' loses all of the metrics it has accumulated. We can solve this by hooking into the shutdown signals and persisting that state else where to be read from on its next start up.

There's a few core components to cover here:

• Data storage
  • Where the data is stored (file, memory cache, relational db, etc)
• Serializing
  • Data is in memory so we need to get that in a format ingestible for our data storage layer
• Crashing
  • If we crash we cannot persist

Data Storage

For this project I'm simply going to use a file to store the GenServer's state. However, whichever the data layer is being used the concept is the same. We want to serialize the data and ship it to our storage

Serialize

Also for this project I'm going to simple approach here. I'm going to JSON encode the entire state and then do a full replacement on the file used to persist. Which then on start up of the GenServer it'll do a read and decode from JSON into its state. Overall though, GenServer states should not be that big to begin with so this is a reasonable approach.

Crashing

If the application crashes then there goes all of its memory. Which if your GenServer's data state is important then periodic persists should be done. When running your application locally and you do ctrl + C to kill it (SIGKILL), that's an ungraceful shutdown and your hooks to handle termination will most likely not be ran. An invoke of :init.stop() is the only reliable way to handle terminations of the application (GenServers themselves can also be told to shutdown).

Changes for a GenServer

Starting from the GenServer with this current code.

There isn't much that needs to be done here so below is what the GenServer looks like after adding the changes need to restore state and persist state on terminate.

defmodule BlueSky.PostCreatedServer do
  use GenServer
  require Logger

  @state_file "post_created_state.json"

  def start_link(_) do
    GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
  end

  def init(_) do
    Process.flag(:trap_exit, true)
    state = read_state_from_file()
    Process.send_after(self(), :register_event, 1000)
    {:ok, state}
  end

  def handle_call(:get_totals, _from, state) do
    {:reply, state, state}
  end

  def handle_info(:register_event, state) do
    Registry.register(BlueSky.EventRegistry.PostCreated, :post_created, %{})
    {:noreply, state}
  end

  def handle_info(:broadcast, state) do
    count = Map.get(state, Date.to_string(Date.utc_today()), 0) + 1
    {:noreply, Map.put(state, Date.to_string(Date.utc_today()), count)}
  end

  def handle_info({:EXIT, _pid, _reason}, state) do
    {:stop, :normal, state}
  end

  def terminate(reason, state) do
    write_state_to_file(state)
    Logger.info("PostCreatedServer is terminating", reason: reason)
    {:ok, state}
  end

  defp read_state_from_file do
    state_file_path = get_state_file_path()

    if File.exists?(state_file_path) do
      case File.read(state_file_path) do
        {:ok, content} ->
          case Jason.decode(content) do
            {:ok, state} ->
              state

            {:error, reason} ->
              Logger.error("Failed to decode JSON from state file", reason: reason)
              %{}
          end

        {:error, reason} ->
          Logger.error("Failed to read state file", reason: reason)
          %{}
      end
    else
      %{}
    end
  end

  defp write_state_to_file(state) do
    state
    |> Jason.encode!()
    |> (&File.write!(get_state_file_path(), &1)).()
  rescue
    e in File.Error ->
      Logger.error("Failed to write state file", exception: e)
      File.mkdir_p!(Path.dirname(get_state_file_path()))
      Logger.info("Attempting to write state file again after creating directory")

      state
      |> Jason.encode!()
      |> (&File.write!(get_state_file_path(), &1)).()
  end

  defp get_state_file_path do
    config_dir = :filename.basedir(:user_config, to_string(:icky_venus))
    Path.join([config_dir, @state_file])
  end
end

The main functions are read_state_from_file/0 and write_state_to_file/1, they do as the name suggests and are expected to be called on GenServer startup (init/1) and shut down (terminate/2).

Periodic Scheduling of Persists

There is one issue with that in that persistance only happens on graceful shutdowns and not all shutdowns will be graceful. Therefore we also want to periodically persist state. This is easily done by scheduling messages to send to the GenServer every time interval. The GenServer itself can be the scheduler.

@hour_in_milliseconds 3_600_000

def init(_) do
  Process.flag(:trap_exit, true)
  state = read_state_from_file()
  Process.send_after(self(), :register_event, 1000)
  :timer.send_interval(@hour_in_milliseconds, :persist_state)
  {:ok, state}
end

def handle_info(:persist_state, state) do
  write_state_to_file(state)
  {:noreply, state}
end

By using :timer it will send a message to itself every hour. The message will be for :persist_state.