Scrubbing Get Params in Phoenix

Phoenix applications can be a mixture of pure REST API endpoints, as well as HTML endpoints rendering content dynamically using views, templates, and partials. However, some convenience methods that exist when creating HTML endpoints are missing when working on an API. One such example is input parameter validation and type coercion.

Introduction

This article is written for Elixir 1.2.5, Phoenix 1.1.4, and Ecto 1.1.7. Please note that future versions of Elixir, Phoenix or Ecto may change in behaviour and invalidate what’s written here. The sample code to this article is available on Github.

What is Phoenix?

Phoenix is a modern web framework, written in the Elixir programming language, which in turn executes on the Erlang Virtual Machine (BEAM). Phoenix could probably best described as an evolution of Ruby on Rails, taking many of the lessons learned from Rails and improving on them.

Phoenix for Rest APIs

Unlike Rails 5’s upcoming API module, Phoenix applications can be a mixture of pure REST API endpoints, as well as HTML endpoints rendering content dynamically using views, templates, and partials. However, when working with Phoenix, some convenience methods that exist when creating HTML endpoints are missing when working on an API.

One such example is input parameter validation and type coercion, which we will take a closer look at in this article.

Preparation

For the rest of this article we will be working with a small demo API that we are going to set up now using the mix phoenix.new command. If you haven’t installed the Hex Package Manager yet, run the command mix local.hex.

First, we create a new Phoenix application using the mix command. I am using the sqlite database here, the default is postgresql. Either is fine, for our sample app sqlite is sufficient.

$ mix phoenix.new --database sqlite sample_api  
...

Fetch and install dependencies? [Yn] Y
* running mix deps.get
* running npm install && node node_modules/brunch/bin/brunch build

We are all set! Run your Phoenix application:
    $ cd sample_api
    $ mix phoenix.server

You can also run your app inside IEx (Interactive Elixir) as:
    $ iex -S mix phoenix.server

Before moving on, configure your database in config/dev.exs and run:
    $ mix ecto.create

Next, let’s do what Phoenix tell us and create the database, and run:

$ mix ecto.create
...
The database for SampleApi.Repo has been created.

Let’s open up sample_api/web/router.ex and add a new endpoint /hello that will call HelloController.index/2.

defmodule SampleApi.Router do
  use SampleApi.Web, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", SampleApi do
    pipe_through :browser # Use the default browser stack

    get "/", PageController, :index
    get "/hello", HelloController, :index
  end
end

Finally we create HelloController under sample_api/web/controllers/hello_controller.ex as follows:

defmodule SampleApi.HelloController do
  use SampleApi.Web, :controller

  def index(conn, %{"name" => name, "age" => age} = params) do
     conn
     |> text("I see, #{name} is #{age} years old!")
  end
  def index(conn, _params) do
    conn
    |> put_status(400)
    |> text("Error, wrong parameters supplied!")
  end
end

HelloController.index/2 has two matches, as our goal is to enforce name and age as required GET parameters for our /hello API endpoint.

Let’s fire up the application using mix phoenix.server and visit the API endpoint, to get the expected results (I’m using HTTPie as my commandline tool):

$ http localhost:4000/hello

HTTP/1.1 400 Bad Request
Error, wrong parameters supplied!
$ http localhost:4001/hello\?name=John\&age=44

HTTP/1.1 200 OK
I see, John is 44 years old!

Using Ecto Models for Parameter Validation and Type Coercion

The naive implementation shown above gives us a very basic mechanism to ensure that required parameters are present in the GET request. From there on out, we would now have to implement our own logic for checking whether a specified parameter adheres to the specification.

For instance, if we specified in our API contract that age is an integer type with allowed valued between 1 and 120, we might end up adding additional validation code like this:

defmodule SampleApi.HelloController do
  use SampleApi.Web, :controller

  def index(conn, %{"name" => name, "age" => age} = params) do
    if valid_age?(age) do
      conn |> text("I see, #{name} is #{age} years old!")
    else
      conn
      |> put_status(400)
      |> text("Error, age parameter must be and integer between 1 and 120!")
   end
  end
  def index(conn, _params) do
    conn
    |> put_status(400)
    |> text("Error, wrong parameters supplied!")
  end

  def valid_age?(age) do
    if (is_integer(age)) do
      (age >= 1 && age <= 120)
    else
      case (Integer.parse(age)) do
        :error -> false
        {val, _} -> valid_age?(val)
      end
    end
  end
end

This kind of code will quickly become very convoluted and un-maintainable! Luckily, there is a much better way! We can use Ecto Models and the validation logic they offer. For this purpose we will create a new Ecto.Model called HelloParams. However, we won’t be using Ecto to persist HelloParams to any database! The main piece we want from the model is the concept of achangeset!

Let’s create sample_api/web/models/hello_params.ex:

defmodule SampleApi.HelloParams do
  use SampleApi.Web, :model

  schema "HelloParams" do
    field :name, :string
    field :age, :integer
  end

  @required_fields ~w(name age)
  @optional_fields ~w()

  def changeset(model, params \\ :empty) do
    model
    |> cast(params, @required_fields, @optional_fields)
    |> validate_number(:age, greater_than_or_equal_to: 1, less_than_or_equal_to: 120)
  end
end

Now we can simplify our HelloController logic significantly:

defmodule SampleApi.HelloController do
  use SampleApi.Web, :controller
  alias SampleApi.HelloParams

  def index(conn, params) do
    cs = HelloParams.changeset(%HelloParams{}, params)
    case cs do
      %{:params => %{ "age" => age, "name" => name }, :valid? => true} ->
        conn |> text("I see, #{name} is #{age} years old!")
      _ ->
        conn
        |> put_status(400)
        |> text("Error, wrong parameters supplied!")
    end
  end
end

Notice, how we are matching the changeset cs against %{:valid => true} which is the main logic that tells us whether all of our required parameters were present, and passed the validation and type checks. I’ve used a more detailed match in the example above, to conveniently extract the name and age values into variables that I can immediately use in the String interpolation, but other methods of working with the params is absolutely possible, they are available as cs.params.

Bonus: Error Details

A good API does not only render the proper HTTP status codes, but also gives appropriate error messages and error details. As an extra benefit in using Ecto.Model for our parameter validation and coercion, we can now add error details to our endpoint quite conveniently by parsing the changeset.errors construct:

defmodule SampleApi.HelloController do
  use SampleApi.Web, :controller
  alias SampleApi.HelloParams

  def index(conn, params) do
    cs = HelloParams.changeset(%HelloParams{}, params)
    case cs do
      %{:params => %{ "age" => age, "name" => name }, :valid? => :true} ->
        conn |> text("I see, #{name} is #{age} years old!")
      _ ->
        error_msgs = changeset_errors(cs)
        IO.inspect cs
        conn
        |> put_status(400)
        |> text("Error, wrong parameters supplied!\nDetails: #{error_msgs}")
    end
  end

  def changeset_errors(changeset) do
     changeset.errors
     |> Enum.map(fn {k, v} -> "Parameter #{k} #{render_detail(v)}" end)
  end

  defp render_detail({message, values}) do
    Enum.reduce values, message, fn {k, v}, acc ->
      String.replace(acc, "%{#{k}}", to_string(v))
    end
  end
  defp render_detail(message) do
    message
  end
end

Now calling our endpoint with invalid parameters will render proper error messages:

$ http localhost:4001/hello\?name=John
HTTP/1.1 400 Bad Request

Error, wrong parameters supplied!
Details: Parameter age can't be blank
$ http localhost:4001/hello\?name=John\&age=130
HTTP/1.1 400 Bad Request

Error, wrong parameters supplied!
Details: Parameter age must be less than or equal to 120
$ http localhost:4001/hello\?name=John\&age=two
HTTP/1.1 400 Bad Request

Error, wrong parameters supplied!
Details: Parameter age is invalid

Now that’s pretty amazing, isn’t it?!

Empty Strings vs. Nil

One little issue remains though - consider the following case where we call the API endpoint with the name parameter present, but equal to an empty String:

$ http localhost:4001/hello\?name=\&age=13
HTTP/1.1 200 OK

I see,  is 13 years old!

For HTML endpoints processing form data, Phoenix provides the Phoenix.Controller.scrub_params/2 function that we can plug in a Controller like so:

defmodule SampleApi.SomeController do
  user SampleApi.Web, :controller

  plug :scrub_params, "user" when action in [:new, :create]

  def new(conn, %{"user" => user} = params) do
    # More Code here
  end
end

This assumes that some post "/some/:user" route maps to the SampleApi.SomeController.new/2 function, posting a user form to the endpoint. The scrub_params/2 function will recursively walk through the user struct, replacing all Key-value pairs that look like {k, ""} with {k, nil} (see GitHub for more details). Unfortunately, we can’t use scrub_params/2 with our API endpoint above, where parameters are given as GET parameters.

Scrubber Plug

Let’s fix this by adding our own implementation of Param Scrubbing for ordinary GET params. First, we will create a new file sample_api/lib/sample_api_helpers.ex as follows:

defmodule SampleApi.Helpers do

  def changeset_errors(changeset) do
     changeset.errors
     |> Enum.map(fn {k, v} -> "Parameter #{k} #{render_detail(v)}" end)
  end

  defp render_detail({message, values}) do
    Enum.reduce values, message, fn {k, v}, acc ->
      String.replace(acc, "%{#{k}}", to_string(v))
    end
  end
  defp render_detail(message) do
    message
  end

  def scrub_get_params(conn, _opts) do
     params = conn.params |> Enum.reduce(%{}, &scrub/2 )
     %{conn | params: params}
  end
  defp scrub({k, ""}, acc) do
    Map.put(acc, k, nil)
  end
  defp scrub({k, v}, acc) do
    Map.put(acc, k, v)
  end

end

If you read the scrub_params/2 code on GitHub linked above, you will see that our implementation of scrub_get_params/2 is actually quite similar to the original. Our function scrub_get_params/2 is defined as a function plug, thus accepting a Plug.Conn.t and a Keyword List, and returning a Plug.Conn.t. We walk over all GET params with Enum.reduce/3, using an empty map as the intial accumulator and passing in the private scrub/2 function. That function has two matches, {k, ""} and {k,v}. In the first match, we add {k, nil} to the accumulator, in the second match, we add add {k, v} as it were.

We can now plug scrub_get_params/2 into our SampleApi.HelloController:

defmodule SampleApi.HelloController do
  use SampleApi.Web, :controller
  alias SampleApi.HelloParams
  import SampleApi.Helpers, only: [changeset_errors: 1, scrub_get_params: 2]

  plug :scrub_get_params

  def index(conn, params) do
    cs = HelloParams.changeset(%HelloParams{}, params)
    case cs do
      %{:params => %{ "age" => age, "name" => name }, :valid? => :true} ->
        conn |> text("I see, #{name} is #{age} years old!")
      _ ->
        error_msgs = changeset_errors(cs)
        conn
        |> put_status(400)
        |> text("Error, wrong parameters supplied!\nDetails: #{error_msgs}")
    end
  end
end

Now calling the API endpoint as before yields the proper error:

$ http localhost:4001/hello\?name=\&age=13
HTTP/1.1 400 Bad Request

Error, wrong parameters supplied!
Details: Parameter name can't be blank