Michał Szajbe

Pogo - distributed supervisor for Elixir

2023-05-22T08:00:00+00:00

Typical architecture of systems written for Erlang VM consists of a top-level process, known as a supervisor, which spawns and monitors child processes that can either act as workers or be supervisors themselves. Such hierarchical process structure is known as supervision tree. Supervisors are fundamental builing blocks of fault-tolerant applications and embodiment of the famous “let it crash” philosophy which, in case of a serious failure, prefers that a process be fully restarted (by its supervisor) to a known good state rather than perform overextensive error handling. Supervisor processes are responsible for managing the lifecycle of their child processes, which includes starting, termination, monitoring and possibly restarting them when they crash.

Ease of building distributed systems is another main selling point of Erlang VM. All it takes for processes running on different nodes to communicate with each other is to ensure that their nodes are connected in a cluster. From code perspective, sending a message to a process running on another node is no different than sending it locally.

While supervision and distribution are first-class citizens of the VM, the standard library doesn’t provide a complete solution to the problem of process supervision in distributed environment. This article discusses why and how to use a distributed supervisor and demonstrates pogo library, developed at Telnyx, that provides one for Elixir applications.

What is a distributed supervisor

Unlike a typical supervisor, distributed supervisor is not a single process. It is many local supervisors running on different nodes, working in coordinated fashion. It abstracts away the complexity of scheduling and monitoring processes in distributed environment.

Scheduling child processes

When scheduling a process with a distributed supervisor no assumptions can be made about which node will actually start it, the only guarantee we get is that it will run “somewhere” in the cluster. Internally, pogo builds a consistent hash ring consisting of all participating nodes, calculates the search key by hashing the child process’ spec and determines which node in the ring “owns” the key. The local supervisor running on that node will then manage the process.

As a natural consequence of this approach, load will be spread over the cluster. Nodes participating in the ring own equally-sized keyspaces, so should theoretically have equal probabilities to be assigned a child process.

Synchronization

Each local supervisor periodically synchronizes with the cluster, inspecting cluster memberships and recalculating hashes. When local supervisor detects that it runs a child process that should be assigned to another node, that process is terminated with the assumption that it will be started on correct node when that node synchronizes.

Note that changes in cluster membership are automatically reflected in the structure of the node ring. When a node is added or removed, it is very likely that some child processes will get rescheduled to other nodes. Whether this behaviour is desired depends on the use case, of course, but it’s crucial to take it into account, especially when running the application in an unreliable network or with autoscaling.

Pogo has a configuration option to control how often synchronization happens. It can be set to a higher value in unstable networks to minimize the effects of aggressive rescheduling.

Pogo.DynamicSupervisor.start_link(sync_interval: 5_000)

Another important observation is that it is possible for a child process to be simultaneously running on multiple nodes, at least briefly. When rescheduling, the new node may be first to synchronize and start the child process before the old node terminates it. The consistency is eventually restored after full synchronization cycle.

Internal state

To maintain cluster-wide state, pogo uses battle-tested distributed named process groups available in Erlang’s standard library - :pg module¹. It differs in this aspect from other similar libraries, like swarm² or horde³. Pogo-based supervisors are completely reliant on process groups, they do not attempt to send any messages to each other.

When a local supervisor receives a request to start a child process via start_child/2, instead of starting it immediately, it propagates that request through process groups to supervisors running on other nodes. Termination of child processes via terminate_child/2 works in the same way. These requests are processed by local supervisors when they perform synchronization; once started, child process information is again propagated using process groups.

With :pg, process groups can be organised into named scopes that are completely independent of each other. It allows us operate multiple distributed supervisors in a single application, differing in configuration or even node memberships.

CAP theorem

In short, CAP theorem⁴ states that in the presence of network partition a distributed system can guarantee either consistency or availability, but not both. It should be now clear that pogo guarantees availability.

When network partition happens, our application will basically operate in two (or more) separated clusters, with each cluster running their own distributed supervisor on available nodes (even if it’s only a single node). Previously scheduled child processes will now be duplicated. Once network connectivity is restored, the clusters will merge and extraneous child processes will be terminated.

When to use distributed supervisor

Distributed supervisor is a good fit for clustered applications that need to effortlessly spread the load over available nodes while preserving all the benefits of typical Elixir’s Supervisor or DynamicSupervisor.

Real world example

The example below is based on one of the services we have at Telnyx. Its role is to monitor media servers, collect their metrics and expose them for other services to consume. Telnyx’s telephony platform runs across a global multi-cloud network and this service is deployed in multiple regions with multiple instances per region for redundancy and load distribution. For brevity, the code responsible for clustering of nodes is omitted⁵.

At startup, each node starts its own local supervisor under application’s supervision tree. As you can see, scope option is set to name of the region where the node is located, so our application effectively runs multiple distributed supervisors, one per region⁶.

Each local supervisor defines the same list of initial child processes to start under distributed supervisor, but these processes will of course be started only once per region, on their assigned nodes.

defmodule Monitor.Application do
  use Application

  def start(_type, _args) do
    pogo_opts = [
      name: Monitor.DistributedSupervisor,
      scope: get_region(),
      children: [
        {Monitor.ServiceDiscovery, "media-server"},
        {Monitor.ServiceDiscovery, "media-server-canary"}
      ]
    ]

    children = [
      {Pogo.DynamicSupervisor, pogo_opts}
      #... other children
    ]

    opts = [strategy: :one_for_one, name: Monitor.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

In our case these initial children are service discovery processes for media servers (both default and canary versions). They constantly watch Consul for registrations and unregistrations of media-server and media-server-canary service instances and respectively start or stop probing processes for each discovered media server instance.

defmodule Monitor.ServiceDiscovery do
  use GenServer

  # ...

  def handle_info({:register, service_id}, state) do
    # new media server instance discovered, start probing process for it
    Pogo.DynamicSupervisor.start_child(
      Monitor.DistributedSupervisor,
      {Monitor.Prober, service_id}
    )

    {:noreply, state}
  end

  def handle_info({:unregister, service_id}, state) do
    # media server instance was unregistered, stop its probing process
    Pogo.DynamicSupervisor.terminate_child(
      Monitor.DistributedSupervisor,
      {Monitor.Prober, service_id}
    )

    {:noreply, state}
  end
end

That’s it. We need only one service discovery process per region for each service type we want to monitor and we need only one prober for each monitored service instance. Both size of the cluster and health of individual instances of monitoring service are transparent to us⁷, because pogo does all the dirty work. The load is spread over available nodes and if one of them goes down its local processes will be automatically redistributed to healthy nodes.

Footnotes

See Distributed named process groups in Erlang’s official docs. ↩
Swarm uses Interval Tree Clock. ↩
Horde uses ∂-CRDT and also provides distributed registry. ↩
Read about CAP theorem on Wikipedia. ↩
If you’re interested, we use Consul for service discovery and libcluster. ↩
Pogo takes advantage of the fact that process groups can be organized in multiple scopes that are completely independent of each other, simply by passing scope argument to :pg function calls. ↩
Obviously we need to make sure there is at least one healthy node per region, but this is an operational problem. ↩

How to use Monaco editor with Phoenix LiveView and esbuild

2023-05-15T12:00:00+00:00

Monaco Editor is state-of-the-art code editor, packed with features like syntax coloring, IntelliSense¹, validation and much more. It powers VS Code and since it is browser-based, it can be integrated into web-apps too. If you ever used Elixir’s Livebook, you’ve seen it action.

This article documents step by step how to integrate Monaco editor with LiveView and Phoenix’s default bundler for the web, esbuild. The source code shown here can be found in GiHub repository: szajbus/phoenix_monaco_example. Let’s start.

Add dependency

First, we need to add the npm dependency. Phoenix keeps frontend assets in assets folder by default, so the path needs to be specified.

$ npm install monaco-editor --prefix assets

Create editor component

To properly initialize the editor and render it on the page, we need to execute some client-side JavaScript once its container is added to DOM and is mounted by LiveView. We can use phx-hook to achieve it.

Let’s start with the template. The editor’s container will be rendered at 100% width and height relative to its parent element.

Through phx-hook we connect the container to CodeEditor JavaScript object which will manage its lifecycle, we’ll define it in next step. We also want the Monaco to control the contents, so we specify phx-update="ignore" to prevent the LiveView from re-rendering the container.

Lastly we pass @language and @code from socket’s assigns via data attributes.

<div
  id="code-editor"
  class="flex w-full h-full"
  phx-hook="CodeEditor"
  phx-update="ignore"
  data-language={@language}
  data-code={@code}
>
  <div class="w-full h-full" data-el-code-editor />
</div>

Next, let’s create the hook object in assets/js/hooks/code_editor.js.

It defines mounted callback to create an instance of Monaco editor and render it when the component is mounted and destroyed callback to clean up when it’s not needed anymore. We access the actual element the hook is added to via this.el and the dynamic variables via its dataset.

We limit ourselves to some most basic configuration options here, the full list of available customization options is available in official docs.

import * as monaco from "monaco-editor";

const CodeEditor = {
  mounted() {
    const container = this.el.querySelector("[data-el-code-editor]");
    const { language, code } = this.el.dataset;

    this.editor = monaco.editor.create(container, {
      theme: "vs-dark",
      language: language,
      value: code,
      minimap: {
        enabled: false
      }
      // ... other options
    });
  },

  destroyed() {
    if (this.editor) this.editor.dispose();
  },
};

export default CodeEditor;

We also need to tell the LiveSocket to use our hook by adding the following to assets/app.js.

import CodeEditor from "./hooks/code_editor";

let liveSocket = new LiveSocket("/live", Socket, {
  hooks: {
    CodeEditor,
    // ... possibly other hooks
  },
  // ... rest of options
});

Bundle with esbuild

With many other libraries that should be enough, but with Monaco we need some adjustments to the bundling process.

First of all, when we import monaco-editor JavaScript code, we are also pulling in the accompanying CSS and other assets bundled with it, like fonts.

Esbuild is not configured to bundle fonts by default, so we need to change it. Let’s update the :default profile in config/config.exs and add file loader for ttf files. It is enough that the source file is simply copied to output directory without any processing. The loader will automatically embed the file name in the bundle as a string.

config :esbuild,
  version: "0.14.41",
  default: [
    args: ~w(
        js/app.js
        --bundle
        --target=es2017
        --outdir=../priv/static/assets
        --external:/fonts/*
        --external:/images/*
        --loader:.ttf=file
      ),
    cd: Path.expand("../assets", __DIR__),
    env: %{"NODE_PATH" => Path.expand("../deps", __DIR__)}
  ]

With the above configuration the editor should render correctly, but it is quite possible that the rest of application’s CSS is now broken. That’s because esbuild also bundles CSS and outputs the resulting file as app.css which conflicts with the app.css bundled by tailwind library. The CSS imported with monaco-editor and bundled by esbuild simply overwrite the CSS produced by tailwind.

To resolve this conflict, let’s rename our assets/css/app.css to assets/css/style.css and reconfigure tailwind in config/config.exs.

config :tailwind,
  version: "3.2.4",
  default: [
    args: ~w(
      --config=tailwind.config.js
      --input=css/style.css
      --output=../priv/static/assets/style.css
    ),
    cd: Path.expand("../assets", __DIR__)
  ]

Utilize web workers

Monaco provides syntax coloring for myriad of programming languages (it’s even possible to add one for your custom language if needed), so let’s support some popular ones here. For performance reasons we want to offload their work to web workers in order not to block the main thread of execution.

Web workers, naturally, are not imported by default, so we need to again adjust our esbuild configuration to include them. This time however, we’ll define a separate esbuild profile and we’ll do it for two reasons:

they will be loaded on demand, so we don’t want to bundle them with rest of the code
they are external files, so we don’t need to watch them for changes and rebuild in dev

Here’s the new esbuild profile.

config :esbuild,
  version: "0.14.41",
  default: [
    # default profile defined above
  ],
  monaco_editor: [
    args: ~w(
        node_modules/monaco-editor/esm/vs/editor/editor.worker.js
        node_modules/monaco-editor/esm/vs/language/css/css.worker.js
        node_modules/monaco-editor/esm/vs/language/html/html.worker.js
        node_modules/monaco-editor/esm/vs/language/json/json.worker.js
        node_modules/monaco-editor/esm/vs/language/typescript/ts.worker.js
        --bundle
        --target=es2017
        --outdir=../priv/static/assets/monaco-editor
      ),
    cd: Path.expand("../assets", __DIR__)
  ]

We also need to include it in assets build pipelines which are defined in mix.exs.

  defp aliases do
    [
      "assets.build": [
        "tailwind default",
        "esbuild default",
        "esbuild monaco_editor"
      ],
      "assets.deploy": [
        "tailwind default --minify",
        "esbuild default --minify",
        "esbuild monaco_editor --minify",
        "phx.digest"
      ],
      # ... other aliases
    ]
  end

At this point we should rebuild the assets locally because Phoenix will not do it for us (we don’t define watchers for these files in config/dev.exs).

$ mix assets.build

Finally, we must tell Monaco where to get these workers from and how to apply them. Let’s modify our component to define MonacoEnvironment.

const CodeEditor = {
  mounted() {
    self.MonacoEnvironment = {
      globalAPI: true,
      getWorkerUrl(_workerId, label) {
        switch (label) {
          case "css":
          case "less":
          case "scss":
            return "/assets/monaco-editor/language/css/css.worker.js";
          case "html":
          case "handlebars":
          case "razor":
            return "/assets/monaco-editor/language/html/html.worker.js";
          case "json":
            return "/assets/monaco-editor/language/json/json.worker.js";
          case "javascript":
          case "typescript":
            return "/assets/monaco-editor/language/typescript/ts.worker.js";
          default:
            return "/assets/monaco-editor/editor/editor.worker.js";
        }
      },
    };

    // the rest of the code may remain unchanged
  }
}

That’s it. We have a basic code editor that supports syntax coloring for several languages.

The source code presented in this article can be found in GiHub repository: szajbus/phoenix_monaco_example.

Footnotes

IntelliSense is a general term for various code editing features, such as: code completion, parameter info, quick info, and member lists. ↩

Understanding and fixing recompilation in Elixir projects

2020-04-14T08:00:00+00:00

:bulb: Do you need help optimizing your project's compilation time? Let's talk.

Our goal as programmers is to deliver value by writing code. It should be efficient, maintainable, but most importantly correct.

Correctness can be checked at multiple stages of the development process, but the most immediate feedback we can get is from the compiler, test suite or the REPL.

We rarely write large chunks of code in one go, then test it as a whole, because that means unnecessary risk of wasted effort if it turns out not to work as expected. We’d rather make small, deliberate changes, then quickly test and adjust if needed before going forward.

For that however, our feedback loops must be as short as possible. We strive for fast test suites and invest time in optimizing the compilation process, because they directly affect our workflow. Slow feedback contributes to lost focus at the very least.

Consider the rate of feedback as your speed limit. ¹

When working with compiled languages, and Elixir is no exception here, we are destined to spend some time waiting for the compiler. And there may be times when even the smallest of changes result in recompilation of significant parts of the codebase. These situations quickly get annoying.

Ever experienced this?

$ touch some/project/module.ex
$ mix test

Running tests...
Compiling 791 files (.ex)

That’s a long wait before tests even run! Let’s find why and how to fix this.

How compiler decides what to recompile

Elixir compiler uses lexical tracker to track references to modules, function dispatches, usage of aliases, imports and requires in the code, etc. It uses this information to build project modules’ dependency graph and ultimately optimize its own work.

When module changes, the compiler finds its dependants by analyzing the dependency graph and marks them as stale. Next, dependants of these are marked as stale too. The process is repeated until the whole dependency graph is traversed and all the stale modules are identified.

Whether a stale module will be recompiled depends on the type of dependency it has to a module that “made” it stale.

Understanding module dependencies

Basically, when module A uses module B in any way, we say it depends on B. Dependencies themselves are transitive. If A depends on B and B depends on C, then by implication A depends on C too.

There are three types of dependencies between Elixir modules.

Compile-time dependencies

Such dependecies are created when module A uses module B at compile time, for example by:

requiring module B
importing functions from B
using macros from B
delegating calls to B via defdelegate
implementing behaviour B
implementing protocol B

When B changes, A must be recompiled too.

Runtime dependencies

Runtime dependencies happen when module A interacts with B only at runtime, e.g. by calling its functions (either by fully qualified name or via an alias).

When B changes, A does not need to be recompiled.

Struct dependencies

This type of dependency is created when A uses %B{} struct.

A needs to be recompiled only when the definition of %B{} struct changes, because struct keys are checked at compile time. ²

Strategy for faster recompilation

Actually it’s not about the speed of the compiler, but the amount of work it has to do. The less cross-dependencies in the codebase, the less modules will need to be recompiled after something changes.

The obvious strategy would be to try to reduce compile-time dependencies, but reducing runtime and struct deps is equally important.

Consider following example:

defmodule A do
  @answer B.search()
  def get_answer, do: @answer
end

defmodule B do
  def search, do: C.search()
end

defmodule C do
  def search do
    # ...
  end
end

A has a compile-time dependency on B because it calls a function from that module at compile-time (when module attributes are defined).

B has only a runtime dependency on C because it calls a function from that module at runtime.

A doesn’t have a direct compile-time dependency on C, but if C changes, then A must be recompiled, even though B doesn’t have to!

$ touch lib/c.ex
$ mix compile --verbose
Compiling 2 files (.ex)
Compiled lib/c.ex
Compiled lib/a.ex

The process is as follows:

Compiler detects that C changed (on disk), marks it as stale and for recompilation.
Compiler sees that B depends on something stale (C), so marks it as stale as well, but doesn’t mark it for recompilation because it’s only a runtime dependency.
Compiler sees that A depends on something stale (B), marks it as stale, but this time it’s a compile-time dependency, so the file is marked for recompilation too.

Clearly A has an implicit compile dependency on C.

It was counter-intuitive to me at first ³. It happens because the compiler assumes that A may be using C’s code at compile time indirectly (through B).

This fact is going to shape our strategy when trying to avoid recompilation hell in our projects.

Fixing your project

In large projects, it’s not uncommon to see cycles in the dependency graph. If there happen to be a compile-time dependency between member modules of such cycle, any change will trigger a cascade-style recompilation of other modules in that cycle as well as ones depending on them and so on.

That’s why even changes that appear simple on the surface, sometimes get you few hundred files to recompile… and ruined workflow.

As removing cycles from the dependency graph is rarely a trivial task, it’s better to try to prevent them happening in the first place. ⁴

Nonetheless, there are some dependency-breaking techniques listed below that may apply to any project.

As a prerequisite, get familiar with xref tool.

It’s built into mix and can help you indentify super-connected modules in your project and modules that have deep subtrees of dependencies ⁵. Simply use mix help xref to start.

There’s also a short and practical overview of the tool written by Wojtek Mach on Dashbit’s blog.

Keep you library code clean

There will probably be some well-connected modules in your business layer, like those in User or Account contexts. In general, business layer code is very likely to contain a lot of cross-module dependencies, even cycles.

Library code on the other hand is supposed to be generic, it should not depend on any module from your business layer. Otherwise it would transfer all such dependencies to any place it’s referenced from.

Here’s an example from actual project:

# web/i18n.ex
defmodule MyApp.I18n do
  @moduledoc """
  Internationalization with a gettext-based API.
  """

  use Gettext, otp_app: :my_app

  def set_locale(%User{locale: locale}),
    do: Gettext.put_locale(__MODULE__, locale)

  # …
end

Pattern-matching on %User{} struct creates a compile-time dependency on User module. It didn’t seem to be a big deal until we realized that it creates a lot of indirect dependencies throughout the whole project because I18n module is very widely-used.

A simple change yielded a huge positive change in recompilation.

- def set_locale(%User{locale: locale}),
+ def set_locale(%{locale: locale}),
    do: Gettext.put_locale(__MODULE__, locale)

Don’t import everything

In Phoenix apps, web/router.ex builds a super-connected MyApp.Router.Helpers module. When you import it to use route helpers, you indirectly import a lot of its dependencies. To avoid that, alias it instead:

- import MyApp.Router.Helpers
+ alias MyApp.Router.Helpers, as: Routes

The same applies to any other well-connected module.

Change defdelegate to proxy functions

defdelegate defines functions via metaprogramming at compile time. Simple “proxy” functions would be runtime dependencies instead.

- defdelegate authorize(conn), to: Auth
+ def authorize(conn), do: Auth.authorize(conn)

Don’t define module attributes with remote functions

Module attributes are defined at compile-time, if they are set by using remote functions, compile-time dependencies are created. If you don’t need to use module attributes in guards, consider functions instead.

- @extension_whitelist FileExt.images()
+ defp extension_whitelist, do: FileExt.images()

Use remote types in typespec

Consider the example:

defmodule Hello do
  def say(%User{username: username}), do: "Hello, #{username}"
  def say(%Admin{name: name}), do: "Hello, #{name}"
end

Hello uses %User{} and %Admin{} structs, so we have just struct dependencies, as shown by xref.

$ mix xref graph
lib/hello.ex
├── lib/admin.ex (struct)
└── lib/user.ex (struct)

Now let’s add a pretty standard function @spec that list these structs as accepted argument types:

@spec say(%User{} | %Admin{}) :: binary()

Suddenly, we get more strict compile-time deps ⁶.

$ mix xref graph
lib/hello.ex
├── lib/admin.ex (compile)
└── lib/user.ex (compile)

In order to fix this, we should rather define remote types and use them instead of structs.

defmodule User do
  defstruct [:username]
  @type t() :: %__MODULE__{}
end

defmodule Admin do
  defstruct [:name]
  @type t() :: %__MODULE__{}
end

defmodule Hello do
  @spec say(User.t() | Admin.t()) :: any()
  def say(%User{username: username}), do: "Hello, #{username}"
  def say(%Admin{name: name}), do: "Hello, #{name}"
end

Summary

Sometimes a quick, small change may result in removal of a crucial dependency and break a cycle in your dependency graph, yielding tangible improvements in recompilation speed. Other times, some improvements may come at the expense of code readability and understandability, and simply will not be worth it.

Pay attention. The compiler, through slow recompilation, may be signalling problems in your code. It may prompt you to rethink your recent architectural decisions.

Issues are generally easier and cheaper to fix when detected early and recompilation that’s slowing down is a plainly visible warning sign you probably should take seriously.

:bulb: Do you need help optimizing your project's compilation time? Let's talk.

Footnotes

D. Thomas, A. Hunt. (2020). The Pragmatic Programmer, 20th Anniversary Edition. Pearson Education, Inc. ↩
As of Elixir 1.6. See Separate tracking structs from compile-time dependencies #6575 ↩
Until I received explanation from Jason Axelson. See Implicit compile-time dependencies in Elixir Forum. ↩
I personally look forward to using such tools as boundary which makes cross-module dependencies explicit. Umbrella projects can also be helpful in that aspect as they don’t allow cyclic dependencies between individual apps. ↩
Technically it’s a graph, not a tree, but xref displays it as such. ↩
I actually suspect this is a bug and plan to investigate it. ↩

Base64-encoded file uploads with Phoenix and Plug

2019-02-13T23:00:00+00:00

Handling external files is one of the basic tasks for many web applications. Profile pictures, attachments, resources fetched from remote services, all are examples of the above.

The most common way such files find their way into a web app is by means of an HTTP multipart request, usually performed through a web page with an HTML form and a file input tag or a client using app’s HTTP API directly. Phoenix and, more specifically, Plug handles those very well, by wrapping the uploaded file information in a convenient struct:

%Plug.Upload{
  content_type: "image/jpg",
  filename: "rainbow.jpg",
  path: "/var/folders/nk/_hgjvryd2ml40g0jpyllt2080000gn/T//plug-1548/multipart-440190-911831-1"
}

Plug writes uploaded data on disk as a temporary file and provides its path along with original filename and type encapsulated in a struct. As such, it is passed to Phoenix controller with the rest of request params.

The convenience of Plug.Upload goes beyond the definition of the above struct. It also implements a GenServer that tracks and manages all file uploads during the life cycle of the process ¹. This includes generation of unique file names, creation of intermediate directories in tmp folder, creating and writing the file itself and, probably most importantly, cleanup of all the temporary files when the process terminates.

However, not all the clients will find the HTTP multipart interface suitable. Instead of reimplementing mentioned Plug.Upload functionality, it would be better to simply re-use it when handling files obtained by other means.

Let’s explore some options.

Base64-encoded data

XML and JSON are just two examples of text-based formats widely used in service-to-service communication. They are easy to implement and very flexible, but are limited to text only. Neither of those formats, nor the protocols that make use of them, directly support mixing with non-text data.

The solution is to turn non-text (binary) data into plain text, that can be easily embedded in XML or JSON for transport and turned back into binary format at the destination. Base64 is probably the most common encoding scheme that helps to achieve this.

Consider the following JSON document, describing some person and embedding her picture as Base64-encoded string (truncated for readability):

{
  "name": "John Doe",
  "email": "john@example.org",
  "picture": "SGFuZGxpbmcgZXzEtMSIKfQpgYGA..."
}

In contrast to HTTP multipart requests, this time Phoenix controller will receive picture as string instead of %Plug.Upload{} struct. To plug into the above-mentioned conveniences of Plug.Upload we need to decode it and build the struct ourselves.

First, let’s define a helper function that writes arbitrary binary data into a temporary file and wraps it with a %Plug.Upload{} struct ².

def binary_to_upload(binary) do
  with {:ok, path} <- Plug.Upload.random_file("upload"),
       {:ok, file} <- File.open(path, [:write, :binary]),
       :ok <- IO.binwrite(file, binary),
       :ok <- File.close(file) do
    %Plug.Upload{path: path}
  end
end

The important part is the call to Plug.Upload.random_file/1, which creates a file in tmp directory and sets up its tracking by the GenServer so that it can be later cleaned up.

Now the handling of Base64-encoded files is straightforward.

def base64_to_upload(str) do
  with {:ok, data} <- Base.decode64(str) do
    binary_to_upload(data)
  end
end

Base64 encoding is quite common, but it’s not the only alternative. Let’s see another one.

Data URLs

Mobile and JavaScript single page apps, even when using regular HTTP APIs, may prefer to upload files as Data URLs that include raw or Base64-encoded file contents.

The format is very simple, it consists of data: prefix, optional MIME type indicator, optional base64 token (for non-textual data) and the data itself. For example, data:text/plain;base64,SGVsbG8gd29ybGQh is a "Hello World!" string encoded as Data URL. You can try pasting it in your web browser’s address bar and it should happily decode it.

The approach here is analogous to the previous example - our goal is to decode the original file and plug it into Plug.Upload.

The following code snippet uses ex_url package, which provides parser and decoder for Data URLs (note URL.Data.parse/1 function).

def data_url_to_upload(data_url) do
  with %{scheme: "data"} = uri <- URI.parse(data_url),
       %URL.Data{data: data} <- URL.Data.parse(uri) do
    binary_to_upload(data)
  end
end

Let’s see it in action:

iex> upload = data_url_to_upload("data:text/plain;base64,SGVsbG8gd29ybGQh")
%Plug.Upload{
  content_type: nil,
  filename: nil,
  path: "/var/folders/nk/_hgjvryd2ml40g0jpyllt2080000gn/T//plug-1548/base64-data-1548887956-746460942629208-8"
}

iex> File.read(upload.path)
{:ok, "Hello World!"}

Looks good!

Summary

The essence of abstractions is preserving information that is relevant in a given context, and forgetting information that is irrelevant in that context.

– John V. Guttag ³

Base64-encoded files may be a special case in your app, but with little work they can be handled as easily as regular files uploaded via HTML forms.

Plug.Upload nicely abstracts away the process of handling temporary files. At the same time, it provides a convenient struct around them to work with.

With it at your disposal, irrespective of where the external files come from, what format they’re in and what your application does with them, you can rely on a universal, stable interface to ease both feature development and testing.

Footnotes

Wherever in this article a word process is used, it refers to Erlang VM process. ↩
To keep the examples simple, I am not even trying to set content_type and filename fields in the structs. Depending on the situation they may or may not be available in the request data. ↩
Guttag, John V. (2013-01-18). Introduction to Computation and Programming Using Python (Spring 2013 ed.). Cambridge, Massachusetts: The MIT Press. ISBN 9780262519632. ↩