elixir – mattmower.com

Is a better Elixir REPL possible?

From the perspective of being an Elixir developer, one of the things I miss about Clojure is it’s REPL and the ability to eval code directly from the editor to the REPL session. For example, when working on a function, I can send the current code to the REPL to redefine the function and then play with it in the REPL. Elixir has a REPL, but you can’t do this.

Clojure has the concept of a “current namespace” (namespace is the nearest Clojure equivalent to Elixirs module). Anything evaluated in the REPL is done so in the context of this namespace. You can switch namespaces using the in-ns function. Elixir isn’t this loose, you can re-evaluate a function you need to specify it’s module, there’s no implicit module context.

def foo(), do: true
> ** (ArgumentError) cannot invoke def/2 outside module

defmodule X
  defn foo(), do: true
end
> X.foo() => true

I wonder if it would be possible to implement the concept of a “current module” and then automatically wrap any use of def with defmodule <current_module> do; …code…; end?

Wrapping your Elixir app in a Burrito

Probably not an original title but here I am talking about turning an Elixir command-line tool I built into a native executable.

When I needed to learn Elixir (that we’re using in AgendaScope) I picked a project I’d been thinking about for some time: a tool for building interactive fiction. I liked the problem particularly because it would exercise a lot of ground but be quite self contained.

It was a great learning experience and, along the way, I a couple of tools I’m quite proud of: a parser combinator library, Ergo, and a library for representing multiple files as a single contiguous file, LogicalFile.

So now I have this tool that is actually pretty close to being usable and I start thinking about how other people might test it out and that’s where my use of Elixir strikes me as not so good.

You see Elixir has the notion of an escript which is a script that embeds Elixir so a user doesn’t need Elixir installed. But Elixir depends upon Erlang and does need the Erlang runtime system (ERTS) installed. For the kind of people I am targeting that felt awkward.

Along comes Burrito which creates a native binary for macOS, Linux, and Windows. The binary that automatically installs ERTS for the platform and hides all that stuff in the background.

Some things that I struggled with:

I didn’t spend enough time looking at the example app. Don’t be like me.
When you create the releases() function in mix.exs you need to actually call it from your project() function as releases: releases() this does not magically happen. Without this mix release will function but Burrito won’t do anything. If you’re unfamiliar with mix release as I was you might see what it does as “working” and be quite confused.
In your releases() function the key (which is example_cli_app in their example) should be the app key from your project() function.
If you use MIX_ENV=prod mix release then beware that the binary will not include updated code no matter how much you mix compile unless you also bump the project version.
On macOS the cached deployment is in ~/Library/Application Support/.tinfoil which makes no real sense to me and was pretty hard to find. If you don’t fall foul of (4) above this may not matter to you.

All in all most of my issues were lack of observation on my part. The tool works (although I confess I haven’t been able to test the cross-platform binaries) and is a boon to anyone building command-line tools who wants to keep using their favourite language.

Without Burrito I’d probably be learning Zig or Nim right now.

Many’s the difficult parser

A while back I wrote a parser combinator library in the Elixir language called Ergo. I did it partly as a language learning exercise and partly because I had some things needing parsing that related to AgendaScope (which is also being written in Elixir).

Barring some performance issues, Ergo has actually turned into quite a nice tool for parsing and it’s getting a workout in two projects: one related to AgendaScope and one a fun side-project. But there is a problem I’ve hit that I haven’t quite thought my way around yet. The side project is easier to talk about so let’s use a problem example from there:

@game begin
  title: "The Maltese Owl"
  author: "Matt Mower"

  @place bobs_office begin
    description: """A dimly lit, smoky, office with a battered leather top desk and swivel chair."""
    w: #outer_office
    s: #private_room
  end

  @actor bob_heart begin
    name: "Bob"
    role: #detective
  end

  …
end

A parser combinator combines simple parsers together to create more complex parsers. A simple parser might be character that parsers things like ‘a’, ‘b’, ‘5’, ‘#’ and so on. Then there are parsers (called combinator parsers) that take other parsers and use a rule to apply them. It’s a bit like higher-order functions.

The sequence parser applies a series of parsers, in turn, and returns :ok if they all match, otherwise :error. The choice takes a list of parsers and returns :ok when one of them matches, otherwise :error. The many parser takes a single parser and applies it repeatedly. When a parser returns :error the input is rewound so that the next parser gets a chance to match on the input. So in the choice each parser gets an opportunity to parse the same input.

Let’s look at an example to parse the above syntax (ignoring whitespace & other issues to keep things clear). It might look something like:

def game() do
  sequence([
    literal("@game"),
    literal("begin"),
    many(
      choice([
        place(),
        actor()
    ]),
    literal("end")
  ])
end

def place() do
  sequence([
    literal("@place"),
    id(),
    literal("begin"),
    attributes(),
    literal("end")
  ])
end

# actor() will look very similar to place()

def attributes() do
  many(attribute())
end

def attribute() do
  sequence([
    name(),
    literal(":"),
    value()
  ])
)

Within game the choice parser attempts to parse a place and if that fails, rewinds the input, and attempts to match actor instead. The many parser would then repeat this choice over and over so that you could parse an infinite stream of places and actors, in any order.

The many parser can succeed zero or more times. It’s like the * operator in a regular expression. When its child parser returns :error (in our example, when the choice cannot match either a place or actor) it stops. But what does it return?

It turns out that many returns :ok and not :error. That’s because many is greedy and will keep trying to match forever so that not matching at some point is not only expected but required!

If we were parsing:

@game begin
  @place …
  end

  @actor …
  end

  @finally …
  end
end

We expect many to terminate because @final is not part of the sequence of places & actors. However, if we look at the example at the beginning, we have two attributes defined on bobs_office called n and w.

It turns out that an attribute name must be at least 3 characters long so these are invalid and attributes is consequently returning an :error and triggering the end of the many. But this is a different end than meeting @finally.

In this case the input is rewound and the next parser — literal("end") — gets applied. But it’s getting applied to input that should have been consumed by the place parser!

So the problem, as clearly as I can state it, is how to distinguish between situations in which many receiving an :error from its child parser means “end of the sequence” and when it means “I should have parsed this, but couldn’t because of a user error.”

I’m not sure what an elegant solution is to this problem. My first instinct is to add a “parser” called something like commit that specifies that an error after this point is a user-error.

So, for example:

def place() do
  sequence([
    literal("@place"),
    commit(),
    literal("begin"),
    …
    literal("end")
  ])
end

What this means is that if we have correctly parsed the input “@place” we know we are definitely in something that should match. If we can’t get place to match from here then the user has specified an incorrect input and rather than place returning :error it should return something like :fatal as an indicator to future parsers that the input cannot be correctly parsed, rather than attempting to continue parsing on on the wrong inputs.

Is the problem clear? Would this work? Is there something I’ve missed? Perhaps a more elegant solution to this problem?

Libraries can start processes too!

It’s the kind of thing that, if you are used to other languages, would make you very suspicious. Library code starting its own processes — you can almost feel the mess and sense the bugs just musing over it. Yet in Elixir (and, of course, Erlang too), this is a totally normal thing to do.

The Elixir approach to shared mutable state is wrapping it in a process. In this case, I needed a counter and the easiest way to implement it is to use an Agent which is a kind of process designed to handle simple state. In this case, the get_and_update function allows me to return the counter and increment it as an atomic operation.

To start an Agent you use the Agent.start_link function. But where to call it? Do I have to add some equivalent initializer function to my library? While not exactly onerous it felt awkward somehow like my implementation was leaking into the caller. Then again did I have to stop the agent process somewhere? Where would I do that?

Now I figured out how to manage the life-cycle of the agent process myself within the library. But it turns out to be unnecessary. All I had to was make one change to my mix.exs file and one addition to a module in my library.

def application do
  [
    extra_applications: [:logger]
  ]
end

becomes:

def application do
  [
    extra_applications: [:logger],
    mod: {Ergo, []}
  ]
end

along with changing the referenced library module, Ergo to look like:

defmodule Ergo
  use Application

  def start(_type, _args) do
    Supervisor.start_link([Ergo.AgentModule], strategy: :one_for_one)
  end

  …
end

This is enough that any application using my library (called Ergo, btw) knows will automatically start the Agent and manage its life-cycle. Without me, or the calling application, needing to know anything about it at all.

This is a pretty neat trick.

Elixir Protocols vs Clojure Multimethods

I am coming to appreciate José Valim’s creation, Elixir, very much. It is fair to say that Rich Hickey set a very high bar with Clojure and Elixir for the most part saunters over it. I think that is a reflection of two very thoughtful language designers at work. But there is some chafing when moving from Clojure to Elixir.

I wrote previously about how Clojure’s metadata feature lets one subtly piggyback data in a way that doesn’t require intermediate code to deal with it, or even know it exists. It creates a ‘functional’ backchannel. Elixir has no equivalent feature.

If your data is a map you can use “special” keys for your metadata. Elixir does this itself with the __struct__ key that it injects into the maps it uses as the implementation of structs. You mostly don’t know it’s there but would have to implement a special case if you ever treated the struct as a map.

However, if the value you want to attach metadata to is, say, an anonymous function then you’re out of luck. In that case, you have to convert your function to a map containing the function and metadata and then change your entire implementation. That could be a chore, or worse.

Today I hit the problem of wanting to define a function with multiple implementations depending on its arguments. Within a module, this is not hard to do as Elixir functions allow for multiple heads using pattern matching. It’s one of the beautiful things about writing Elixir functions. So:

defmodule NextWords do
  def next-word("hello"), do: "world"
  def next-word("foo"), do: "bar"
end

Works exactly as you would expect. Of course, the patterns can be considerably more complex than this and allow you to, for example, match values inside maps. So you could write:

def doodah(%{key: "foo"}), do: "dah"
def doodah(%{key: "dah"}), do: "doo"

And that would work just as you expect too! Fantastic!

But what about if you want the definitions of the function spread across different modules?

defmodule NextWords1 do
  def next-word("hello"), do: "world"
end

defmodule NextWords2 do
  def next-word("foo"), do: "bar"
end

This does not work because while NextWords1.next-word and NextWords2.next-word share the same name they are in all other respects unrelated functions. In Elixir a function is specified as a tuple of {module, name, arity} so functions in different modules are totally separate regardless of name & arity.

In Clojure, when I need to do something like this I would reach for a multi. A Clojure multi uses a function defined over its arguments to determine which implementation of the multi to use.

(defmulti foo [x y z] (fn [x y z] … turn x, y, z into a despatch value, e.g. :bar-1, :bar-2 or what have you))

(ns 'bar-1)
(defmethod foo :bar-1 [x y z] … implementation)

(ns 'bar-2)
(defmethod foo :bar-2 [x y z] … another implementation)

The dispatch function returns a dispatch value and the methods are parameterised on dispatch value. Different method implementations can live in different namespaces and a call with the right arguments will always resolve to the right implementation, regardless of where it was defined.

Now Elixir has an equivalent to multi, the Protocol. We can define the foo protocol:

defprotocol Fooish do
  def foo(x)
end

Now in any module, we can define an implementation for our Fooish protocol. But, and here’s the rub, an implementation looks like:

defmodule Foo-1 do
  defimpl Fooish, for: String
    def foo(s), do: …impl…
  end
end

So an Elixir protocol can only be parameterised on the type of its first argument! This means that there’s no way to dispatch protocol implementations based on lovely pattern matching. Disappointing.

It may even be that a smarter Elixir programmer than I could implement Clojure style multi-methods in Elixir. For now, I can find a work-around and I’m still digging Elixir a lot.

Value metadata is a subtle but useful language feature

Since I am using it to create AgendaScope, i’ve written quite a lot of Elixir code to get myself up to speed. Most recenty a parser combinator library, Ergo. And it’s in building Ergo that I’ve realised I really miss a feature from Clojure, metadata.

In short Clojure allows you to attach arbitrary key-value metadata to any value. While there is syntax sugar it boils down to:

meta(obj) to get the map of metadata key-value pairs for the value obj.

vary-meta(obj, f, args) to modify the meta-data associated with the value obj.

If you make a copy of obj the metadata is carried with it. In all other respects obj is unaffected. Why is this so useful?

Well I first used it to manage whether a map, representing a SQL row, comes from the database or not as well as associated data about the record. I could have put that in the map using “special” keys but this pollutes the map with stuff not strictly related to its purpose and more complex code with special cases to handle those keys. But what if obj wasn’t a map to which I could add special keys?

Clojure makes heavy use of it for self-documenting, tracking attributes, and tagging of all different kinds of structures.

What about Ergo? Well being a parser combinator library Ergo is about building functions, usually returning them as anonymous functions. When debugging a parser you’d really like to know more about it & how it was constructed but what you have is a function f.

In Clojure I’d just put all that in metadata of f and it would be no problem later to get at that metadata for debugging purposes by calling meta(f). The code of the parser itslef needing to know nothing about the metadata at all. But in Elixir… I am not sure what to do.

Elixir functions can have multiple heads so it occurs to me that I could return my parsers like:

fn
  %{Context} = ctx -> do parsing stuff here
  :info -> return some metadata instead
end

And this could work but also feels brittle to me. If a user defines a parser and doesn’t add an :info head then my debugging code that invokes f.(:info) is going to generate an error. It doesn’t smell right. Is this the best I can do?

I being to wish Elixir had an equivalent meta functionality to Clojure.

Connecting Elixir GenServer and Phoenix LiveView

My new company AgendaScope is building it’s product using the Elixir/Phoenix/LiveView stack and while I have spent some time in recent months learning Elixir and Phoenix, LiveView was entirely new to me. This weekend I decided it was time to crack it.

I had a little project in mind which is creating a dashboard of thermal info from my Mac. There’s a reason for that which is another post but suffice to say I needed information displayed on a second computer, preferably my iPad.

The way I chose to tackle this problem was to create a GenServer instance that would monitor the output of a few commands on a periodic basis (at the moment once every 5 seconds). The commands are:

thermal levels

powermetrics --samplers smc | grep -i "die temperature"

uptime

The Elixir Port module makes this a pretty trivial thing to do with the GenServer handle_info callback receiving the command output that I parse and store in the GenServer state.

On the other end it turns out LiveView is pretty simple. A LiveView template is an alternative to a regular static template (this part had not been clear to me before and I thought they sat, side-by-side).

The LiveView module in my case ThermalsLive stores things like CPU pressure, GPU pressure, & CPU die temp, in the Socket assigns. And a template is simple to create. LiveView takes care of all the magic of making things work over a web socket.

For example, in a LiveView template the markup <a href='#' phx-click='guess' phx-value-number='1'>1</a> generates a link that results in the running LiveView module receiving the handle_event callback with the value 1. Like this:

def handle_event("guess", %{"number" => guess} = data, %{assigns: %{score: score, answer: answer}} = socket) do
 …
end

Something I learned about LiveView is that each browser session gets its own stateful server side LiveView module that lasts throughout the session. When a LiveView event handler changes the assigns in the socket structure LiveView responds by re-rendering the parts of the template that depended on those particular assigns. This is good magic!

Now the question was: How does my GenServer (ThermalsService) notify the LiveView (ThermalsLive) when one of the variables it’s monitoring (cpu_pressure) has changed? This event is not coming from the browser so the regular LiveView magic isn’t enough.

I couldn’t find a good simple example and so what I’ve learned was gleaned from some books and Google searching. There short answer is Phoenix.PubSub, here’s the longer answer.

First we need a topic. When three friends are at a table for lunch their topic might be “politics” and one of them might tune out and not be listening. But when the topic changes to “cheese” they tune back in and hear those messages.

For our GenServer & LiveView the topic (defined as a module attribute @thermals_topic) is going to be thermals_update because I want them to talk about these and I don’t plan to use individual message types for different variables like cpu_pressure and gpu_die_temp.

On the LiveView end we need to subscribe to the same topic. The right place to do this appears to be in the mount callback where the socket assigns are first set.

def mount(_params, _session, socket) do
  if connected?(socket) do
    DashletWeb.Endpoint.subscribe(@thermals_topic)
  end
  …
end

We test if the socket is connected because it turns out a LiveView module calls mount twice. Once in response to the initial browser request where it returns the HTML page that starts a web socket connection back to the server. The second call to mount is in response to setting up the web socket and long-term communication between client and server. This seems to be the best point to subscribe to the topic.

It wasn’t obvious that the DashletWeb.EndPoint module had a subscribe method ready made for this purpose and I was mucking about the the Phoenix.PubSub module for a bit before I hit on this.

My GenServer process ThermalsService receives handle_info callbacks from the Port that contain the terminal output of the running command as a string. With a little bit of parsing & conversion we end up with something like cpu_pressure: 56 and as well as storing this in the GenServer state we need the LiveView module to know about it. We need to broadcast a message to the thermals_update topic.

This turned out to be pretty simple:

alias Phoenix.PubSub
@thermals_topic "thermals_update"

def handle_info({_port, {:data, text_line}}, state) do
  …
  PubSub.broadcast(Dashlet.PubSub, @thermals_topic, {"cpu_pressure", cpu_pressure)
  …
end

I confess I am still not sure what Dashlet.Pubsub is. By my Elixir knowledge it should be a module but I can’t find it defined anywhere. Anyway, it works.

The last piece of the puzzle that I couldn’t find stated but inferred from examples is that, for each broadcast, there is a call to the handle_info callback on all topic subscribers (in this case our LiveView module).

So, in my live view ThermalsLive I add:

def handle_info({"cpu_pressure", pressure}, socket) do
  {:noreply, assign(socket, cpu_pressure: pressure)}
end

This is super simple. It recevies the cpu_pressure message from the GenServer along with the pressure value and stores it in the LiveView socket assigns (the same way the handle_event callback would do in respond to clicking a browser link). This is enough for LiveView to take the hint and trigger a client update.

What foxed me was that regular LiveView events arrive via the handle_event callback that receives the LiveView socket instance. You need the socket to change its assigns to trigger an update. It didn’t occur to me that PubSub might ensure that the handle_info path would be equivalent.

And, there you have it, sending data from a GenServer via Phoenix PubSub to a LiveView module. I learned quite a bit via this exercise and I hope this might help anyone following the same path.

Stop aliasing String.t and integer

I’m learning Elixir and, a little unwillingly, learning about it’s system of type specifications. I will concede that being able to specify the types of return values is helpful but I am not sure about the rest. Nevertheless I am told the Dialyzer system (that analyses type information and looks for errors) can be very helpful so for now I am playing along.

While reading the documentation something caught my eye:

Defining custom types can help communicate the intention of your code and increase its readability.

defmodule Person do
   @typedoc """
   A 4 digit year, e.g. 1984
   """
   @type year :: integer

   @spec current_age(year) :: integer
   def current_age(year_of_birth), do: # implementation
end

I found myself puzzled by this statement. The argument in question year_of_birth is already clearly indicating it’s a year. And the type cannot enforce the rule “A 4 digit year, e.g. 1984”.

So what is the type spec adding? It seems to me that what it’s adding is a step of cognitive overhead in understanding that what is being passed is an integer.

I’ve seen other examples of creating types for username and password that alias the String.t type and again I find this unconvincing since the function being spec’d almost certainly calls it’s arguments username and password so what is being added?

Where a type adds useful information I buy it. A struct for example is already a kind of alias since it defines compound structure. But for primitive types like String.t and integer it seems like adding aliases is hiding information not adding to it.

Inserting associated models with Ecto

I’m building a “learning app” using the PETAL stack and it’s taxing some of the grey cells that haven’t worked since I was working with Ruby on Rails many years ago.

It’s hit a point where I need to insert two associated records at the same time. I am sure this involves Ecto.Multi but I’m also trying to understand how to build the form since the form_for expects an Ecto.Changeset and so far as I can see these are schema specific. Or, at least, in all the examples I’ve seen so far they have been.

I make a lot of introductions by email so I decided to build a little helper application to make it easier for me. My schema at the moment is quite simple:

Contact <-> Role <-> Company

At the moment I have a very simple CRUD approach where you create Contacts and Companies separately. But, of course, in practice when I create a Contact I want to create the Company and specify the Role at the same time. And that’s where I run into a problem. The common pattern is something like:

def new(conn, _params) do
  changeset = Introductions.change_contact(%Contact{})
  render(conn, "new.html", changeset: changeset)
end

In this case we are creating an Ecto.Changeset corresponding to a new Contact. Later when we want to build a form to edit the details we have:

<%= form_for @changeset, @action, fn f -> %>

Where the form relates the fields of the schema to the fields in the form.

So the question is how you create a “blended” or “nested” Changeset that can contain the details of each of the 3 schemas at work.

I’ve not seen any examples covering this case. I’m muddling my way through it but it would be great to have something to work from.