Improving the Writing Experience

As with writing code or documentation that is going to be viewed, it is desired to be able to view it as it is being iterated on. Currently for this site the processes to do so is as follows:

Run mix run --no-halt in a terminal
Then:
- Make a change
- Run mix build
- Refresh page
- Repeat

This process is kind of annoying, more so when trying to do HTML and CSS stuff. However, being able to watch for file system changes is pretty straight forward and we already have mix build that will do most of what we want on a change. So with that, I went down the path of extending mix build to re-build on file changes

File System Watching

I started off by searching for a library that can do file system watching. First result lead me to FileSystem. Quick look at this readme showed me everything I needed. So I started by writing a GenServer to hook into the file system events.

lib/tasks/file_system_watcher.ex

defmodule SourShark.FileSystemWatcher do
  use GenServer
  @root_dir File.cwd!()
  @watch_dir ["#{@root_dir}/lib/sour_shark/templates", "#{@root_dir}/priv/posts"]

  def start_link(args) do
    GenServer.start_link(__MODULE__, args)
  end

  def init(_args) do
    Mix.Tasks.Build.build_output(true)
    {:ok, watcher_pid} = FileSystem.start_link(dirs: @watch_dir, latency: 0)
    FileSystem.subscribe(watcher_pid)
    IO.puts("Watching for file changes at:\n\t- #{Enum.join(@watch_dir, "\n\t- ")}")
    {:ok, %{watcher_pid: watcher_pid}}
  end

  def handle_info({:file_event, watcher_pid, {path, events}}, %{watcher_pid: watcher_pid} = state) do
    IO.puts("File event: #{inspect(path)} -> #{inspect(events)}")
    IEx.Helpers.recompile()
    Mix.Tasks.Build.build_output(false)
    {:noreply, state}
  end

  def handle_info({:file_event, watcher_pid, :stop}, %{watcher_pid: watcher_pid} = state) do
    IO.puts("File watcher stopped")
    {:noreply, state}
  end
end

This also meant adding a new API to Mix.Tasks.Build to leverage this.

Note the usage of IEx.Helpers.recompile(), I did this because I could not find a way to invoke a new build of NimblePublisher. As far as I can tell there is no API provided, as it only runs on compilation.

Now I needed a host for this file system watcher GenServer, which I only have one good option for that. The Bandit server that runs when doing mix run --no-halt.

With that I updated the application.ex to start it up on its supervisor.

lib/sour_shark/application.ex

defmodule SourShark.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      Registry.child_spec(
        keys: :duplicate,
        name: SourShark.EventRegistry.FileChange
      ),
      {Bandit, scheme: :http, plug: SourShark.Router, port: 8080},
      SourShark.FileSystemWatcher
    ]

    opts = [strategy: :one_for_one, name: SourShark.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

What remains is updating the API on the mix build module.

lib/mix/tasks/build.ex

def run(_args) do
  build_output()
end

def build_output(should_clean? \\ false) do
  {micro, _} =
    :timer.tc(fn ->
      if should_clean?, do: clean_output_dir()
      download_highlight_js_assets()
      build_pages()
      build_blog()
      build_assets()
    end)

  ms = micro / 1000
  IO.puts("Finished building site in #{ms}ms")
end

I added a should_clean? flag as well, this definitely exposes us to defects when it comes to lifecycle events of files. Such as one of them being deleted. However, since this product is a user of one, I'm not going to bother improving it. What spurred the should_clean? flag is the download_highlight_js_assets() function. I didn't want to keep going to the CDNs and downloading the JavaScript files on every change, so I updated that function to check if what it was downloading already existed. But since clean_output_dir() was always being ran, it would always download. Hence the should_clean? flag.

Now when running mix run --no-halt a file system watcher is started and a new build is ran when changes are detected.

Live Reloading

The remaining thing to solve is the live reloading. At this point I have it so whenever there is a change to the markdown files or the HTML files a new build will be invoked then upon refresh on the web page the new content will load. Now I just need a way to force a refresh.

This can be done by opening a web socket connection to the Bandit server and when an event is sent on this connection, use JavaScript to refresh the page.

To begin I updated the Bandit server to have a web socket connection endpoint via the WebSockAdapter package. Then added a handler for the web socket connections:

lib/sour_shark/router.ex

get "/events/reload" do
  conn
  |> WebSockAdapter.upgrade(SourShark.HotReloader, [], timeout: 60_000)
  |> halt()
end

lib/sour_shark/hot_reloader.ex

defmodule SourShark.HotReloader do
  def init(_) do
    Process.send_after(self(), :register_event, 1000)
    {:ok, %{}}
  end

  def handle_in(_, state) do
    {:ok, state}
  end

  def handle_info(:broadcast, state) do
    {:reply, :ok, {:text, "reload"}, state}
  end

  def handle_info(:register_event, state) do
    Registry.register(SourShark.EventRegistry.FileChange, :file_change, %{})
    {:ok, state}
  end

  def handle_info({:EXIT, _pid, _reason}, state) do
    {:stop, :normal, state}
  end

  def terminate(_reason, state) do
    {:ok, state}
  end
end

Make note of the Register.register API call. This is because we need a way to connect the file system watcher process with the open web socket connection handler process.

This register call makes it so whenever something is dispatched it'll invoke a function. The dispatch exists on the file system watcher

lib/sour_shark/file_system_watcher

def handle_info({:file_event, watcher_pid, {path, events}}, %{watcher_pid: watcher_pid} = state) do
  IO.puts("File event: #{inspect(path)} -> #{inspect(events)}")
  IEx.Helpers.recompile()
  Mix.Tasks.Build.build_output(true, false)

  Registry.dispatch(SourShark.EventRegistry.FileChange, :file_change, fn entries ->
    for {pid, _} <- entries, do: send(pid, :broadcast)
  end)

  {:noreply, state}
end

The important part here is Registry.dispatch. This will result in whenever there is a file event anything registered for the :file_change key on SourShark.EventRegistry.FileChange registry will then invoke sending a message to :broadcast

Now what remains is injecting HTML when running locally that will have JavaScript code to open a connection to this new web socket endpoint.

lib/sour_shark/templates/root.html.eex

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title><%= Application.get_env(:sour_shark, :product_name) %></title>
  <link href="/css/app.css" rel="stylesheet">
  <%= @extra_head %>
  <%= @hot_reload %>
</head>

lib/sour_shark/templates/hot_reload.html.eex

<script>
  const socket = new WebSocket("ws://localhost:8080/events/reload");
  socket.onmessage = function(event) {
    window.location.reload();
  };
  socket.onopen = function(e) {
    socket.onclose = function(event) {
      if (!event.wasClean)
        window.location.reload();
    };
  };
</script>

Now need to dynamically inject this HTML into the built HTML files when running locally. To do this I updated the mix build module once more

lib/mix/tasks/build.ex

def build_output(add_hot_reload? \\ false, should_clean? \\ false) do
  {micro, _} =
    :timer.tc(fn ->
      if should_clean?, do: clean_output_dir()
      download_highlight_js_assets()
      build_pages(add_hot_reload?)
      build_blog(add_hot_reload?)
      build_assets()
    end)

  ms = micro / 1000
  IO.puts("Finished building site in #{ms}ms")
end

defp build_pages(add_hot_reload?) do
  hot_reload =
    if add_hot_reload?,
      do: File.read!("lib/sour_shark/templates/hot_reload.html.eex"),
      else: nil

  for source <- Path.wildcard("lib/sour_shark/templates/pages/*.html.eex") do
    target = source |> Path.basename() |> String.replace(".eex", "")
    IO.puts("Building #{source} -> #{target}")

    File.write!(
      "#{@output_dir}/#{target}",
      eval_file("lib/sour_shark/templates/root.html.eex",
        assigns: [
          hot_reload: hot_reload,
          extra_head: nil,
          content: eval_file(source, assigns: [])
        ]
      )
    )
  end
end

The add_hot_reload? is there because I didn't know of a better way to pass down the context to include the web socket connection HTML. Given that we don't want this HTML injected on non file system watch invocations of build. As in, we don't want deployed code at jack-develops.com to be trying to open a web socket connection to localhost.

At this point hot reloading works. Whenever there is a file change a web socket is sent to the client (browser page) and a page reload is done.