Simple Api Server in Phoenix
Recently, I have been spending time learning about the Elixir programming language and its ecosystem. One large part of that ecosystem is the Phoenix Framework, a full stack web framework similar in scope to Python’s Django or Ruby’s Rails. I’m still new to working with these technologies, but I’m really enjoying it so far. Its clear that after you get a solid grasp of things, you can be quite productive. So in the spirit of better understanding I’m going to be walking through the process of building a small json api project in Phoenix.
Requirements
While working on a data processing related project I had the need to create a small service that would return json data at monthly granularity. It didn’t really matter to me what data was returned from the endpoint as long as I could query it monthly and get an array of daily metrics back. Since daily Bitcoin data is easy to find and it satisfies the requirement, I’m going to be using some daily Bitcoin candle data. The data will be static for this project and serving it should be straight forward. The end result should be that we have a small Phoenix based json api service that we can query for a month of Bitcoin data and get back a json array of daily Bitcoin candles. We also will want to serve the data using a database and package everything up so that we can run this service using docker compose. All the code for this project has been pushed to a github repository so feel free to dig deeper into the code there.
Getting started
This post assumes some basic knowledge in elixir, phoenix, ecto, and docker. I won’t go too deep into any of these here, however, there are a lot of great resources out there if you’re interested in taking a closer look.
The first step is to make sure Phoenix is installed and available. You can do that by following their installation documentation. Phoenix and Elixir both have excellent documentation so in most cases that should be your first stop for information when trying to figure things out. Next step is to actually create the project, which we can do by using the mix phx.new command.
# Create the project
mix phx.new phx_serve_btc --database sqlite3 --no-assets --no-dashboard --no-gettext --no-html --no-live --no-mailer
Since we are creating a json api service only, I have added a bunch of --no-... flags here to tell Phoenix that we don’t need those things for our specific application. I also chose to use sqlite3 here as our database. We could have just as easily gone with postgres but for small projects like this Sqlite is a great choice for its simplicity and ease of development. For more information on the various flags Phoenix provides (including the flags I used above) you can run mix help phx.new. Lets enter the project directory and get started on putting the api together.
# Enter the project
cd phx_serve_btc
At this point I will usually create a git repo and commit.
git init
git add .
git commit -m 'Init commit'
This is helpful for me as someone who is new to the framework. I can explore the project layout before we make any changes to familiarize myself with how Phoenix structures things. Phoenix provides many generator commands, some of which we will use in this post, and it is helpful to see all the diffs and changes Phoenix is making to the project at every step.
Since we are using a database, we need to configure it with ecto:
mix ecto.create
At this point you should have a functional application that can be started with mix phx.server. The service doesn’t currently do anything so lets fix that.
Api development
We can use a Phoenix generator to initialize the data and the endpoint for our application.
mix phx.gen.json CandleData Bitcoin btc_candles day:date:unique open:float high:float low:float close:float volume:float currency:string
This command will generate the controller, json view, and context for a json resource. You’ll notice we also give the schema for the backing data here as well. For more detailed info on all the parts of this command you can check out the docs. This will output the following.
* creating lib/phx_serve_btc_web/controllers/bitcoin_controller.ex
* creating lib/phx_serve_btc_web/controllers/bitcoin_json.ex
* creating lib/phx_serve_btc_web/controllers/changeset_json.ex
* creating test/phx_serve_btc_web/controllers/bitcoin_controller_test.exs
* creating lib/phx_serve_btc_web/controllers/fallback_controller.ex
* creating lib/phx_serve_btc/candle_data/bitcoin.ex
* creating priv/repo/migrations/20240326005152_create_btc_candles.exs
* creating lib/phx_serve_btc/candle_data.ex
* injecting lib/phx_serve_btc/candle_data.ex
* creating test/phx_serve_btc/candle_data_test.exs
* injecting test/phx_serve_btc/candle_data_test.exs
* creating test/support/fixtures/candle_data_fixtures.ex
* injecting test/support/fixtures/candle_data_fixtures.ex
Some of the generated database columns are unique. Please provide
unique implementations for the following fixture function(s) in
test/support/fixtures/candle_data_fixtures.ex:
def unique_bitcoin_day do
raise "implement the logic to generate a unique bitcoin day"
end
Add the resource to your :api scope in lib/phx_serve_btc_web/router.ex:
resources "/btc_candles", BitcoinController, except: [:new, :edit]
Remember to update your repository by running migrations:
$ mix ecto.migrate
As you can see Phoenix has generated a bunch of files to help get things off the ground. Some of these things we won’t use but it doesn’t hurt to keep them around since this is a test application. If this was more of a production application I would spend time trimming things down to essentials. Lets follow the output’s suggestions and add resources "/btc_candles", BitcoinController, except: [:new, :edit] to the router then run mix ecto.migrate. The migrate will apply our database changes and the resources will allow us to use some basic crud style routes on our app with the provided data schema. Feel free to run mix phx.server and execute some test queries against these routes. We won’t be using these generated routes but they are nice to have and its good to understand what the generators provide out of the box. Now is also a good time to check in these changes since we will be further updating the generated files.
Since we want to query for a month’s worth of Bitcoin data, lets get started on adding that new endpoint. First, lets add the new endpoint to our router.
get "/btc_month/:month", BitcoinController, :get_month
This is telling Phoenix to create an endpoint /btc_month/:month that accepts http get requests and routes those requests to the BitcoinController module using the :get_month function. The :month here is just a path variable that will be available to us in the function.
Next we need to create this get_month function in the BitcoinController. This is most definitely not a production ready piece of code but it will work for testing purposes.
def get_month(conn, _params) do
year_month = conn.params["month"]
# Validate iso month format YYYY-MM
true = Regex.match?(~r/20[0-9][0-9]-(0[0-9]|1[0-2])/, year_month)
[year, month] = String.split(year_month, "-")
start_date = Date.from_erl!({String.to_integer(year), String.to_integer(month), 1})
end_date = Date.add(start_date, Date.days_in_month(start_date))
btc_candles = CandleData.list_btc_candles_date_range(start_date, end_date)
render(conn, :index, btc_candles: btc_candles)
end
We also need the function for listing all candle data over a range of dates. This can be done with a fairly straight forward ecto query.
def list_btc_candles_date_range(start_date, end_date) do
Repo.all(
from bc in Bitcoin,
where: bc.day >= ^start_date and bc.day < ^end_date
)
end
Seeding the data
We are almost ready to start testing our endpoint. We just need some data to play with. Phoenix provides a seed script that we can use to inject some test data into our database.
First we will want to create a new module that can contain our seed code. We want to put this seeding code into a module so it can be re-used to seed the data in our output container later on. Lets take a look at the seed function within the PhxServeBtc.SeedCandleData module.
def seed(path) do
Repo.delete_all(Bitcoin)
File.stream!(path)
|> Stream.map(&String.trim_trailing(&1, "\n"))
|> Stream.map(&String.split(&1, ","))
|> Stream.drop(1)
|> Enum.map(fn _ = [day, open, high, low, close, volume, currency] ->
Bitcoin.changeset(
%Bitcoin{},
%{
"day" => day,
"open" => open,
"high" => high,
"low" => low,
"close" => close,
"volume" => volume,
"currency" => currency
}
)
end)
|> Enum.each(&Repo.insert!/1)
end
Here we just iterate through all the records in the csv and insert them into our database. No need to worry about batching our inserts here since our data is fairly small. Now that we have this seed function we can use it in the seed script at priv/repo/seeds.exs.
alias PhxServeBtc.SeedCandleData
SeedCandleData.seed("data/bitcoin.csv")
Now we can seed the data.
mix run priv/repo/seeds.exs
With our data seeded we should be able to test basic functionality of the api. With the server running, navigate to http://localhost:4000/api/btc_month/2022-01 and you should see all available Bitcoin candle data for 2022-01. Curl could also be used here, however, since Phoenix was built for the web, if you use a browser you will get much better errors by using the browser if anything goes wrong.
Deployment
Now that our api is working the way we want it, we can start getting our application ready for deployment. We’ll use a basic docker compose setup to handle our local simulation of a deployment. Phoenix makes this very straight forward. Again, we will rely on the generators to help us out. Lets generate the release using the --docker flag.
mix phx.gen.release --docker
Feel free to run docker build . to make sure the container is building without error. Having the docker file generated here is a huge time saver. I didn’t realize this option was available until after I took a stab at writing the docker file from scratch. The generators are faster and did a better job than I did at setting up the image. The above command gives some useful output.
* creating rel/overlays/bin/server
* creating rel/overlays/bin/server.bat
* creating rel/overlays/bin/migrate
* creating rel/overlays/bin/migrate.bat
* creating lib/phx_serve_btc/release.ex
21:23:01.280 [debug] Fetching latest image information from https://hub.docker.com/v2/namespaces/hexpm/repositories/elixir/tags?name=1.16.1-erlang-26.2.2-debian-bullseye-
* creating Dockerfile
* creating .dockerignore
Your application is ready to be deployed in a release!
See https://hexdocs.pm/mix/Mix.Tasks.Release.html for more information about Elixir releases.
Using the generated Dockerfile, your release will be bundled into
a Docker image, ready for deployment on platforms that support Docker.
For more information about deploying with Docker see
https://hexdocs.pm/phoenix/releases.html#containers
Here are some useful release commands you can run in any release environment:
# To build a release
mix release
# To start your system with the Phoenix server running
_build/dev/rel/phx_serve_btc/bin/server
# To run migrations
_build/dev/rel/phx_serve_btc/bin/migrate
Once the release is running you can connect to it remotely:
_build/dev/rel/phx_serve_btc/bin/phx_serve_btc remote
To list all commands:
_build/dev/rel/phx_serve_btc/bin/phx_serve_btc
As you can see this generator creates some helper scripts that can be used during deployment. We will be making use of the migrate script so that our migrations are properly applied within our container.
Now we can create the compose file that will run our migrations and stand up our service.
version: '3.8'
services:
phx_serve_btc_init:
build: .
container_name: phx_serve_btc_init
volumes:
- ./data:/app/data
command: bin/migrate
environment:
DATABASE_PATH: /app/data/phx_serve_btc_prod.db
# Generated with: mix phx.gen.secret
SECRET_KEY_BASE: JEX39XUFm6djBvkqbSxgO40Bp9uS7rCYX0coMYaKueYN1hDNbk9heLAj1NCfUT9t
phx_serve_btc:
build: .
container_name: phx_serve_btc
volumes:
- ./data:/app/data
environment:
DATABASE_PATH: /app/data/phx_serve_btc_prod.db
# Generated with: mix phx.gen.secret
SECRET_KEY_BASE: JEX39XUFm6djBvkqbSxgO40Bp9uS7rCYX0coMYaKueYN1hDNbk9heLAj1NCfUT9t
ports:
- "4000:4000"
depends_on:
phx_serve_btc_init:
condition: service_completed_successfully
Once the compose file is in place you should be able to run docker compose up --build. With the compose running you should be able to again see proper output at http://localhost:4000/api/btc_month/2022-01. However, we currently don’t have any data within the database. Lets fix that by adding a one-off seed script that can be run within our new service container.
We can add a custom bin script here rel/overlays/bin/seed_btc. With the following contents.
#!/bin/sh
set -eu
cd -P -- "$(dirname -- "$0")"
exec ./phx_serve_btc eval 'PhxServeBtc.Release.seed_btc("/app/data/bitcoin.csv")'
Here we are making use of the ./phx_serve_btx eval command which will evaluate the elixir code that you pass to it. We need to create this new function in PhxServeBtc.Release.
def seed_btc(path) do
load_app()
{:ok, _, _} =
Ecto.Migrator.with_repo(PhxServeBtc.Repo, fn _repo ->
PhxServeBtc.SeedCandleData.seed(path)
end)
end
This function loads our app, then with the repo active it runs our seed function that we had created earlier.
With docker compose restarted we can exec into the container and run our new seed script.
# Exec into the running container
docker exec -it phx_serve_btc bash
# Run the seed_btc script (make sure it is executable)
bin/seed_btc
This will go ahead an seed our database from the container. After that has been executed we can check http://localhost:4000/api/btc_month/2022-01 to validate we are getting the output we expect. Since we are writing our data out to a local directory you should only need to run this seed script once (unless you wipe the data). This is why we created a one-off script rather than add this process into a compose step.
We now have a fully up and running Phoenix service that meets all our original requirements. This app is far from production ready. We have no tests, our validation and organization should be fleshed out more, we would need to do some work to get processes in place to deploy this out to a vm, and we don’t have anything in place for updating/expanding our dataset. Since this is a test application meant for learning this is totally fine. It shouldn’t be much more work to get this into a more production ready state. However, that is outside the scope of this post.
I wanted to spend some time putting this post together so that I have a well documented approach for setting up a simple api server with Phoenix. Hopefully this is helpful to anyone else that is trying to get up and running with Elixir and Phoenix.
4/22/2024