Manjunath Reddy

REST API: How to avoid duplicate resource creation on concurrent requests.

Manjunath Reddy — Sun, 01 Jan 2023 16:03:35 GMT

Source: http://www.juimg.com/shiliang/201904/anniutubiao_1768500.html

As anyone working as a software engineer, we all might have come across dealing with API's in our lives. It's easy to consume them, however, designing one requires lots of experience and requires diligent effort and rigorous testing. Also, one of the topics which are always ambiguous in API design is Idempotency, which HTTP verbs are considered to be Idempotent, and which are not, and having that knowledge helps build effective APIs.

We are not going to delve into the Idempotency topic today, the topic is for another day :-). If you would like to explore further, here are some interesting articles

https://developer.mozilla.org/en-US/docs/Glossary/Idempotent

Preamble

In part of this write-up, I'm going to explore a problem I ran into dealing with concurrent requests which resulted in creating multiple resources with the same data, especially since such a problem is very critical if we are dealing with payments.

Problem

I was working on an integration with external payment gateway providers with their API for accepting payments into our system. As part of the gateway integration, we needed to provide a callback URL for the payment provider to a callback for the payment transaction state (was it successful, failed, rejected etc). End-to-end integration went well and we covered the happy path of accepting payment with all statuses.

However, At one point, we needed to test a transaction which was successful from the client side, and break the connection so that we did not get an immediate callback from the payment provider. Usually in such scenarios, Payment providers eventually call back at certain intervals to make sure our sever state for the transactions is processed correctly, and update the payment status accordingly. At a certain interval, we did get a callback from the payment provider, however, to our surprise, we got multiple callbacks for the same successful payment payload.

In the application, we end up having multiple records for the same payment and ended up updating users' payments multiple times (the number of times we got the callback requests). Imagine, the user wants to topup $100 and ended up having $300 in their wallet. In such scenarios, we can not possibly control external or third-party APIs and how they are designed. They might have retried or used a queue mechanism or reposted the resources accidentally.

Dealing with such a design will be crucial if we are dealing with payments or mission-critical applications.

Problem in detail

Let’s take an example of an application integrating with a payment gateway. Each customer who comes to our app uses to select the payment gateway of their choice to pay for a subscription. Internally in our API, we send a request to the payment gateway to create a new payment (POST /api/payments/payment). If the transaction is successful, we get status successful status from the Payment gateway with transaction details, and the transaction is recorded in our DB. This is the sweet, ideal scenario as shown in the below illustration.

On the other hand, if we encounter any network issues while waiting for the payment gateway to acknowledge us, there is no way to find out if the payment was successful or not as illustrated below.

After a while, the said payment gateway decided to callback our API to update the broken payment transaction. These calls could come in many means, could be by manually requesting the payment status or retries from the payment gateway itself or automatic callbacks. And if we do not have a proper method to check for duplicate transaction records in our database, we might end up updating customer payment as many times as we got the request from the gateway. How do we make sure we do not capture multiple payment records for the same transaction?

There is always more than one right way.

Solutions

Ideally, most of us already know how to deal with such operations. I will try to elaborate on the methods I applied and the outcome of each method and why I choose to go with a certain method.

Method 1: Data Validation

Customer payment transactions are recorded in a table called payments. The structure of the table looks like

CREATE TABLE `payments` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(11) NOT NULL,
  `order_id` varchar(50) NOT NULL DEFAULT '',
  `status` varchar(10) NOT NULL DEFAULT '',
  `amount` decimal DEFAULT NULL,
  `transaction_id` varchar(50) DEFAULT NULL,
  `gateway_response` json DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Each time get a response or callback from the payment gateway, query the DB to check if there is an existing transaction for the same user with similar order_id

SELECT * from payments where user_id = ? and order_id = ?;

Pseudocode for the payment validation looks like

const payment = getPayment(userId, orderId);
if (payment) {
  print 'Payment already exists!. Must be duplicate payment'
  throw Error('duplicate') or return;
}// elsevar newPayment = createPayment(userId, reqParams)

The problem with this method is that, when we are swamped with concurrent requests with the same payload in a matter of seconds, we still can not avoid duplication. Because it takes time to query and get the response from DB and during that period, we could have created multiple payment records already.

Method 2: Locking (Mutex) Mechanism

According to Wikipedia's definition

In computer science, a lock or mutex (from mutual exclusion) is a synchronization primitive: a mechanism that enforces limits on access to a resource when there are many threads of execution. A lock is designed to enforce a mutual exclusion concurrency control policy, and with a variety of possible methods, there exist multiple unique implementations for different applications. - Wikipedia

It is a well-known mechanism that certainly could be applied in many cases, however, not when we are dealing with concurrent requests.

Implementation is quite similar to the above method, however, instead of DB, we could choose to implement this in in-memory data stores such as Redis.

Pseudocode for the locking the payment to avoid duplication looks like

// make a unique reference key for each payment transaction
var paymentKey = 'PAYMENT' + user_id + order_id
var payment = getPayment(paymentKey) // assume that this method calls redis or any other in-memory store to get the key
if (payment) {
  print 'Payment already exists!. Must be duplicate payment'
  throw Error('duplicate') or return;
}setPayment(paymentKey) // assume that this methods sets the new payment reference in in-memory
var newPayment = createPayment(userId, reqParams)

From the above pseudocode, when we get the first request, we set the reference in in-memory and create the payment record. And for the subsequent payment records for the same transaction, we ignore them.

Unfortunately, even with this method, a query to in-memory was slow enough to record multiple payment records and still could not avoid duplicates

Here are a few interesting articles regarding locking mechanisms to explore.

https://stackoverflow.com/questions/129329/optimistic-vs-pessimistic-locking https://blog.couchbase.com/optimistic-or-pessimistic-locking-which-one-should-you-pick/

Method 3: Queuing

I did not try this approach since I did not have any queues implemented in the application. However, this could be a more reliable method and gives more flexibility for the application to deal with concurrent requests.

The idea here is to queue all the incoming requests into a queue and deal with them slowly using a consumer and validate each incoming request to make sure we capture only one request.

Method 4: Database table with composite UNIQUE constrain

In this method, we design the payments table with a composite unique key that ensures we have a unique record for each payment. The table design is as follows

CREATE TABLE `payments` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `user_id` int(11) NOT NULL,
  `order_id` varchar(50) NOT NULL DEFAULT '',
  `status` varchar(10) NOT NULL DEFAULT '',
  `amount` decimal DEFAULT NULL,
  `transaction_id` varchar(50) DEFAULT NULL,
  `gateway_response` json DEFAULT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `unique_payment_transaction` (`user_id`,`order_id`,`status`),
  KEY `user_id` (`user_id`),
  KEY `order_id` (`order_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Note: The DB structure I’m using here is MySQL, the structure may differ from other RDBS systems

From the above table schema, I made a composite key out of (user_id, order_id, status)

If we get the same payment request for the same order for a user, In MySQL, we would only have one row recorded, and for the subsequent insertion, MySQL throws a Duplicate entry error. In the application, we can either throw the error or explicitly capture the error from the DB engine and show a meaningful error message to the user.

I chose to go with this approach since it's more efficient than method 1 and 2 and it guarantees that there will not be duplicates and does not require to implementation of extra validation nor requires to use of any in-memory data stores.

Those who are resorting to NoSQL systems might need to look to the first three methods. Method 1 and 2 are not so efficient and queuing methods be the best option if your infrastructure supports them. I do not prefer to go with NoSQL databases as my primary data stores :-) but they are preferable choices as secondary storage.

Here is the GitHub Repo that demonstrates the Method 4 implementation written in NodeJs.

If there are other ways to deal with such operations, please feel to share, would love to get to know different ways of solving such operations.

Dockerize Phoenix + Tailwind application

Manjunath Reddy — Sat, 31 Dec 2022 00:30:17 GMT

Preamble

I'm hoping you already have a Phoenix application running If you happen to hop here directly.

If not, you may visit the previous two parts where we went through Creating a simple phoenix application, and how to set up Tailwind with phoenix.

Build a production-ready app and run it locally

In this article, I'm directly going to jump into setting up the application to make it production ready by

Testing it locally in production mode (env=prod)
Run the app in production mode to ensure our build process is correct,
And then use the same instructions to dockerize the application

There is an excellent series written by Miguel Cobá on how to prepare Phoenix application deployment with Elixir Releases and also, official elixir release documentation.

However, the main reason I'm writing this article is that I bump into some obstacles to configure a Dockerfile to build the production-ready application. Here, I will go through the issues I have had in detail.

Setup, Compile In prod mode

// Initial setup
$ mix deps.get --only prod
$ MIX_ENV=prod mix compile

With this, you may notice that there is a new folder created under _build/ called prod

// Compile assets
$ MIX_ENV=prod mix assets.deploy

// lets test the build with production mode
$ PORT=4000 MIX_ENV=prod SECRET_KEY_BASE=$(mix phx.gen.secret)  mix phx.server

// You will see the following instructions
╰─$ PORT=4000 MIX_ENV=prod SECRET_KEY_BASE=$(mix phx.gen.secret)  mix phx.server
00:21:54.461 [info] Running DockerPhoenixTailwindWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
00:21:54.468 [info] Access DockerPhoenixTailwindWeb.Endpoint at http://example.com

If you are wondering about the env variable SECRET_KEY_BASE which is a secret Phoenix uses to sign and encrypt important information.

Head over to http://localhost:4000 and ensure that the application is running.

Generate a release for a production-ready build

This is the step which I missed during my entire process of preparing a release and almost spent a day and a half to figure it out. Before generating a release, we need to uncomment the following in /config/runtime.exs

config :docker_phoenix_tailwind, DockerPhoenixTailwindWeb.Endpoint, server: true

Essentially, it instructs Phoenix to start the webserver from the release.

Once the above line is uncommented, run the following command to generate the actual release

╰─$ MIX_ENV=prod mix release                                                                        
* assembling docker_phoenix_tailwind-0.1.0 on MIX_ENV=prod
* using config/runtime.exs to configure the release at runtime
* skipping elixir.bat for windows (bin/elixir.bat not found in the Elixir installation)
* skipping iex.bat for windows (bin/iex.bat not found in the Elixir installation)

Release created at _build/prod/rel/docker_phoenix_tailwind

    # To start your system
    _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind start

Once the release is running:

    # To connect to it remotely
    _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind remote

    # To stop it gracefully (you may also send SIGINT/SIGTERM)
    _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind stop

To list all commands:

    _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind

As shown above, the release is created at _build/prod/rel/docker_phoenix_tailwind/bin/ folder,

Let's run it and see if the release works

╰─$ _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind start
ERROR! Config provider Config.Reader failed with:
** (RuntimeError) environment variable SECRET_KEY_BASE is missing.
You can generate one by calling: mix phx.gen.secret

    /Users/manju/Codes/github/docker_phoenix_tailwind/_build/prod/rel/docker_phoenix_tailwind/releases/0.1.0/runtime.exs:17: (file)
    (elixir 1.14.2) src/elixir.erl:309: anonymous fn/4 in :elixir.eval_external_handler/1
    (stdlib 4.2) erl_eval.erl:748: :erl_eval.do_apply/7
    (stdlib 4.2) erl_eval.erl:492: :erl_eval.expr/6
    (stdlib 4.2) erl_eval.erl:136: :erl_eval.exprs/6
    (elixir 1.14.2) src/elixir.erl:294: :elixir.eval_forms/4
    (elixir 1.14.2) lib/module/parallel_checker.ex:107: Module.ParallelChecker.verify/1
    (elixir 1.14.2) lib/code.ex:425: Code.validated_eval_string/3

That did not work?

It's because we need to make sure to run the above command with a SECRET_KEY_BASE a secret which is required in production mode to encrypt important information. Let's try it again

╰─$ SECRET_KEY_BASE=$(mix phx.gen.secret) _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind start
00:41:29.927 [info] Running DockerPhoenixTailwindWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
00:41:29.928 [info] Access DockerPhoenixTailwindWeb.Endpoint at http://example.com

Head over to the browser and test http://localhost:4000/. The application now is running through a release executable (As we will do the same in production)

Let's sum up all the commands

$ mix deps.get --only prod
$ MIX_ENV=prod mix compile
$ MIX_ENV=prod mix assets.deploy
$ MIX_ENV=prod mix release
$ SECRET_KEY_BASE=$(mix phx.gen.secret) _build/prod/rel/docker_phoenix_tailwind/bin/docker_phoenix_tailwind start

It's been quite a journey to get the phoenix app release to make it work locally. In the following section, let's prepare a Dockerfile with all the instructions we went through so far.

Let's Dockerize

Initially, I followed the Phoenix release documentation to generate the Dockerfile automatically and try it build the image, and run the container, however, the container did not run due to incorrect instructions to build the assets and also missing SECRET_KEY_BASE and it took me a couple of hours to debug the Dockerfile.

Then I stumbled upon two issues

Assets weren't compiling (Tailwindcss) and therefore website was broken
Was running docker container without SECRET_KEY_BASE

To address the above issues, I slightly modified the Dockerfile instructions to build the assets correctly. This is instructed quite well in the Dockerfile comments, which I did not give much attention to.

We are not going to explore the entire Dockerfile instruction, I hope the comments are self-explanatory.

https://github.com/manjufy/docker_phoenix_tailwind/blob/main/Dockerfile

An important change to notice from the auto-generated Dockerfile is to ensure asset compilation is after copying lib

Setup Asset compilation correctly

In the Dockerfile, we need to make sure to setup asset compilation correctly so that our tailwind is working correctly.

Before.

Notice that, in the following Docker instructions, we COPY the assets and then we immediately compile the assets, Which was causing assets not getting minified correctly.

# Copy assets
# note: if your project uses a tool like https://purgecss.com/,
# which customizes asset compilation based on what it finds in
# your Elixir templates, you will need to move the asset compilation
# step down so that `lib` is available.
COPY assets assets
# Compile assets
RUN mix assets.deploy

# Compile project
COPY lib lib

RUN mix compile

# Copy runtime configuration file
COPY config/runtime.exs config/

# Assemble release
COPY rel rel
RUN mix release

After

Moved instruction RUN mix assets.deploy after, COPY lib lib instruction. Otherwise, assets (JS and CSS) won't be properly minified.

# Copy assets
# note: if your project uses a tool like https://purgecss.com/,
# which customizes asset compilation based on what it finds in
# your Elixir templates, you will need to move the asset compilation
# step down so that `lib` is available.
COPY assets assets

# Compile project
COPY lib lib

# IMPORTANT: Make sure asset compilation is after copying lib
# Compile assets
RUN mix assets.deploy

RUN mix compile

# Copy runtime configuration file
COPY config/runtime.exs config/

# Assemble release
COPY rel rel
RUN mix release

Build the image and run it from a container

Copy and create the Dockerfle in the project root directory.

Build the image

╰─$ docker image build -t elixir/docker_phoenix_tailwind .

Run the container

╰─$ docker run -e SECRET_KEY_BASE="$(mix phx.gen.secret)" -p  4000:4000 elixir/docker_phoenix_tailwind
00:17:50.396 [info] Running DockerPhoenixTailwindWeb.Endpoint with cowboy 2.9.0 at :::4000 (http)
00:17:50.396 [info] Access DockerPhoenixTailwindWeb.Endpoint at http://example.com

Now head over to http://localhost:4000. We now have an application running from the container which you could use to deploy in production.

The main Dockerfile is based on Debian bullseye, however, I have also experimented with an alpine image which I have explained in the README.

The entire application source is here

https://github.com/manjufy/docker_phoenix_tailwind

For now, that completes the series. However, in the future, I would like to explore deploying by containerising the application on different platforms such as Digitalocean, and Fly.io.

Setting up Tailwind CSS In a Phoenix application

Manjunath Reddy — Fri, 30 Dec 2022 22:42:32 GMT

In the previous article, we explored setting up a basic Phoenix application. If you haven't read the first part yet, Just take a moment to read it before this post :-).

However, if you already have an existing phoenix application running, and if you would like to configure Tailwind with it, skip the first part.

Preamble

Install & Configure Tailwind Plugin

Let me be DRY, and not going to repeat what's been documented quite nicely by the Tailwind official website on how to set up Tailwind CSS in a Phoenix project. Steps are elegantly explained.

via GIPHY

Re-run the application. At this point, It looks like

I know it looks ugly :-). That completes the first part of this article. In the following section, let's beautify the application to make it look nice.

Theme your application

Let's create a simple page which uses tailwind.

Look for lib/docker_phoenix_tailwind_web/templates/layout/root.html.heex file and replace it with

html>
<html lang="en">
  <head>
    <meta charset="utf-8"/>
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
    <meta name="viewport" content="width=device-width, initial-scale=1.0"/>
    <%= csrf_meta_tag() %>
    <%= live_title_tag assigns[:page_title] || "", suffix: " · Micheal Schumacher" %>
    <link phx-track-static rel="stylesheet" href={Routes.static_path(@conn, "/assets/app.css")}/>
    <script defer phx-track-static type="text/javascript" src={Routes.static_path(@conn, "/assets/app.js")}>script>
  head>
  <body class="font-sans antialiased leading-normal tracking-wider text-gray-900 bg-cover">
    <%= @inner_content %>
  body>
html>

And look for our content page which is lib/docker_phoenix_tailwind_web/templates/page/index.html.heex and replace it with, which contains a simple layout

<div class="min-h-screen flex flex-col">
  <header class="p-2 bg-indigo-200">
    <p class="text-2xl font-bold text-center">Phoenix Frameworkp>
  header>
  <main class="flex-1 bg-indigo-50 p-2">
    <div class="mb-20">Content goes here....div>
  main>
  <footer class="p-2 bg-indigo-200">Footer sectionfooter>
div>

This is how my assets/css/app.css looks like

@tailwind base;
@tailwind components;
@tailwind utilities;

body {
  background-color: crimson;
}

And my theme assets/tailwind.config.js

// See the Tailwind configuration guide for advanced usage
// https://tailwindcss.com/docs/configuration

let plugin = require('tailwindcss/plugin')

module.exports = {
  content: [
    './js/**/*.js',
    '../lib/*_web.ex',
    '../lib/*_web/**/*.*ex'
  ],
  mode: 'jit',
  theme: {
    extend: {},
  },
  plugins: [    require('@tailwindcss/forms'),    plugin(({addVariant}) => addVariant('phx-no-feedback', ['&.phx-no-feedback', '.phx-no-feedback &'])),
    plugin(({addVariant}) => addVariant('phx-click-loading', ['&.phx-click-loading', '.phx-click-loading &'])),
    plugin(({addVariant}) => addVariant('phx-submit-loading', ['&.phx-submit-loading', '.phx-submit-loading &'])),
    plugin(({addVariant}) => addVariant('phx-change-loading', ['&.phx-change-loading', '.phx-change-loading &']))
  ]
}

With that, Re-run the application and you will have a simple tailwind themed page.

Please leave a comment if you happen to face any issues configuring Tailwind.

In the next section, we will explore running the Phoenix application locally with a production release build, so that we could use the same instructions to build our Dockerfile.

Creating Phoenix Application

Manjunath Reddy — Fri, 30 Dec 2022 20:22:22 GMT

I have been wanting to explore and create a simple blog with Phoenix and Tailwind for quite a long. However, I struggle a bit to get it up and running and also to be able to dockerise to make it ready for production deployment.

Preamble

This is the first part of the series, where I explore, creating a phoenix application, configuring tailwind, dockerizing the application and exploring creating a release, that we can run in production. Note that, we omitted using a Database in this series. I would explore that in a separate section. The goal for this series is to be able to prepare, dockerize, and deploy the phoenix application to production quickly.

What We'll need?

My development machine is macOS, and most of my instructions will be catered for mac, however, instructions could be closer to Linux machines as well.

First, we need to have elixir installed. Please follow the installation instructions from the official elixir website https://elixir-lang.org/install.html

Once you have elixir installed, Erlang will be automatically installed.

Let's check if we have elixir installed. Note that, Erlang will be automatically installed with Elixir.

╰─$ elixir -v                                                                                                                                                                                                     
Erlang/OTP 25 [erts-13.1.3] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]

Elixir 1.14.2 (compiled with Erlang/OTP 25)

Next, we need to install Phoenix application generated to be able to create initial Phoenix application.

╰─$ mix archive.install hex phx_new

// Check if its installed
╰─$ mix phx.new --version                                                                                                                                                                          
Phoenix installer v1.6.2

The above command would install phoenix generator. Just for info, it will be located under /Users/{username}/.mix/archives/phx_new-{version}

In conclusion, we need to have `Erlang > Elixir > Phoenix generator` to be able to create a new phoenix app.

Initialising the application

As I explained earlier, we create the phoenix application without a database. Use the following command to generate a basic phoneix application.

╰─$ mix phx.new --no-ecto docker_phoenix_tailwind
......
* running mix deps.get
[warning] :rebar is no longer supported in Mix, falling back to :rebar3

We are almost there! The following steps are missing:

    $ cd docker_phoenix_tailwind

Start your Phoenix app with:

    $ mix phx.server

You can also run your app inside IEx (Interactive Elixir) as:

    $ iex -S mix phx.server
// Which will create the project and also installs dependencies

// Then
╰─$ cd docker_phoenix_tailwind
╰─$ docker_phoenix_tailwind > mix phx.server

At the time of this writing, I get the following warning to remove the atom :gettext entry from mix.exs file

warning: the :gettext compiler is no longer required in your mix.exs.

Go ahead and remove this entry.

If you are wondering what is Gettext, it's a module to work with internationalised applications

mix.exs => projection section would look like
// Before
  def project do
    [
     ....
      compilers: [:gettext] ++ Mix.compilers(),
     ....
    ]
  end

// After  
def project do
    [
     ....
      compilers: [] ++ Mix.compilers(),
     ....
    ]
  end

Note: Do not remove the dependency gettext from deps secton. Only remove it from project compilers list.

Stop the application (`ctrc-c` twice), and Re-run the application again

╰─$ mix phx.server                                                                                  
Generated docker_phoenix_tailwind app
[info] Running DockerPhoenixTailwindWeb.Endpoint with cowboy 2.9.0 at 127.0.0.1:4000 (http)
[info] Access DockerPhoenixTailwindWeb.Endpoint at http://localhost:4000
[watch] build finished, watching for changes...

By default, applications run at port 4000, type http://localhost:4000 in your browser, and you will be greeted with the Phoenix Framework home page.

Monitoring and Debugging

Another cool thing about the Phoenix framework is that it comes with a default LiveDhasboard that provides real-time monitoring and debugging tools for the application. Head over to http://localhost:4000/dashboard to view the LiveDashboard

Source: https://hexdocs.pm/phoenix_live_dashboard/Phoenix.LiveDashboard.html

That's it in this episode. In the next article, we explore configuring Tailwind with our Phoenix application.