Distributed Phoenix Chat using Redis PubSub

January 23, 2019

X Follow Button

In the previous article, Create a High-Availability Kubernetes Cluster on AWS with Kops, we have seen how to create a Kubernetes cluster and how to deploy the Phoenix Chat app. The single node configuration worked well, but when we tried to scale out we saw that the messages were not sent to all the browsers.

In the image below, we see a configuration where the two chat servers are isolated. When the load-balancer routes the two WebSocket connections into two different servers, there is no way for the two browsers to send messages one another.

Two Phoenix Chat containers

This problem is often solved using an external component, like Redis, which helps to broadcast the messages to all the nodes in the cluster.

With Elixir is also possible to get rid of this component, purely relying on the communication between Elixir nodes (we will see this in the next article).

In this article we’ll see how we can scale the Phoenix Chat horizontally, running multiple servers and leveraging on Redis PubSub.

Redis PubSub

If you want to test this out on your local machine, you can find this version of the app on the poeticoding/phoenix_chat_example GitHub repo, under the pubsub_redis branch.

We are now going to use the Redis PubSub messaging implementation, so we can broadcast the messages to all the Chat nodes. But .. What is PubSub messaging? I think the AWS definition is simple and explanatory.

Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all of the subscribers to the topic.

AWS – Pub/Sub Messaging

To understand better how PubSub works, let’s see it in action with Redis. The easiest way to run a Redis server on our local machine is using Docker.

$ docker run --name redis -d redis -p 6379:6379

Bash

Copy

In this way we run Redis in background, starting a container we call redis. In the Docker container we’ve just started, we now find the redis-cli which we can use to do some experiments subscribing to a channel and publishing messages on it.

To execute the redis-cli process in the redis container, we use the docker exec command, passing the -it option to enable the interactive mode. The cli will connect to the local redis server by default.

$ docker exec -it redis redis-cli

Redis PubSub

In the image above, we run 4 different clients. Three of them subscribe to my_channel and one publishes a message to the channel. We see how Redis sends this message to all the clients subscribed to the channel.

Phoenix PubSub Redis adapter

Connecting Phoenix Chat servers to Redis

To be able to leverage on this Redis PubSub functionality in our app, we are going to use the phoenix_pubsub_redis adapter. This adapter is really easy to integrate: we just need to add it in the dependencies and configure it in our phoenix app.

#mix.exs
defp deps do
[\
  ...\
  {:phoenix_pubsub_redis, "~> 2.1"}\
]
end

Elixir

Copy

Elixir

Copy

In this case we just manually set the host and the port to localhost and 6379, but if you are running a different Redis setup you maybe need to change this setting. We also need to set a node_name for each chat server we run, so we differentiate the Elixir nodes inside the Redis PubSub channel. We’ll pass the node_name using the NODE environment variable.

That’s it, ready to roll! We start two chat servers, one on 4000 called n1 and the other one on 4001 called n2.

# Terminal 1
$ NODE=n1 PORT=4000 mix phx.server

# Terminal 2
$ NODE=n2 PORT=4001 mix phx.server

Bash

Copy

Messages Broadcasted correctly using Phoenix PubSub Redis

The messages are now correctly sent to all the browsers connected to the channel, regardless of whether they are connected to different chat servers or not.

Inspecting the Phoenix PubSub messages in Redis

To understand better what’s going on, we can inspect the messages sent to Redis by the nodes. All the chat nodes publish messages on the phx:Elixir.Chat.PubSub Redis channel, but these messages are binary encoded using the :erlang.term_to_binary function, so we can’t simply use the redis-cli to properly see them.

In the repo, always under the pubsub_redis branch and in the redis_print directory, I’ve put a super-small Elixir app we can use to subscribe and decode the binary messages.

defmodule RedisPrint do
  def subscribe(host,port,channel) do
    {:ok, pubsub} = Redix.PubSub.start_link(host: host, port: port)
    {:ok, ref} = Redix.PubSub.subscribe(pubsub, channel, self())
    receive_messages(pubsub,ref)
  end

def receive_messages(pubsub,ref) do
    receive do
      {:redix_pubsub, ^pubsub, ^ref, :message, %{channel: _, payload: payload}} ->
        :erlang.binary_to_term(payload) |> IO.inspect()
    end
    receive_messages(pubsub,ref)
  end
end

Elixir

Copy

The RedisPrint.subscribe/3function starts a PubSub process which connects to Redis and subscribes to a specific channel. It then start receiving messages, recursively calling receive_messages/2 and decoding the payload with :erlang.binary_to_term

Let’s test again the chat with two browsers and two servers, this time inspecting the messages in a separate iex session.

# Terminal 1
$ NODE=n1 PORT=4000 mix phx.server

# Terminal 2
$ NODE=n2 PORT=4001 mix phx.server

# redis_print
$ iex -S mix
...
%{
  __struct__: Phoenix.Socket.Broadcast,
  event: "new:msg",
  payload: %{body: "hello from u1", user: "u1"},
  topic: "rooms:lobby"
}

%{
  __struct__: Phoenix.Socket.Broadcast,
  event: "new:msg",
  payload: %{body: "hello from u2", user: "u2"},
  topic: "rooms:lobby"
}

Bash

Copy

When the user u1, connected to n1 on port 4000 , sends the message “hello from u1”, this message is sent through the WebSocket connection and once n1receives it, it’s then published to the phx:Elixir.Chat.PubSub Redis channel.

Wrap Up

As I said at the beginning, this is just one way to make our Phoenix Chat app distributed. We’ll see in further articles how, thanks to how Elixir nodes can communicate, we connect and broadcast the messages using another Phoenix PubSub adapter.

tempest.services.disqus.com is blocked

This page has been blocked by an extension

Try disabling your extensions.

ERR_BLOCKED_BY_CLIENT

Reload

This page has been blocked by an extension

Disqus Comments

We were unable to load Disqus. If you are a moderator please see our troubleshooting guide.

Join the discussion…

Comment

Log in with

or sign up with Disqus or pick a name

Disqus is a discussion network

Don't be a jerk or do anything illegal. Everything is easier that way.

Read full terms and conditions

This comment platform is hosted by Disqus, Inc. I authorize Disqus and its affiliates to:

Use, sell, and share my information to enable me to use its comment services and for marketing purposes, including cross-context behavioral advertising, as described in our Terms of Service and Privacy Policy, including supplementing that information with other data about me, such as my browsing and location data.
Contact me or enable others to contact me by email with offers for goods or services
Process any sensitive personal information that I submit in a comment. See our Privacy Policy for more information

Acknowledge I am 18 or older

2
Discussion Favorited!

Favoriting means this is a discussion worth sharing. It gets shared to your followers' Disqus feeds, and gives the creator kudos!

Find More Discussions

Tweet this discussion
- Share this discussion on Facebook
- Share this discussion via email
- Copy link to discussion
Best
- Newest
- Oldest

amazing!

Show more replies

Nice article man, very useful.

I think you got a typing error in the "Phoenix PubSub Redis adapeter" topic, you wrote "AdapEter".

Updated, Thanks a lot! :D

Nicely done! A big Thank You for your time and quality of material posted here.

I have a question regarding the second part, right after testing Redis on local cluster. Could you please give us some more details about steps required to start the two chat servers in Windows? I am using VS Code and Docker for Windows with the local Kubernetes cluster up an running. Up until now, everything worked just fine, but I don't know how that part works. At some point, I have installed Elixir and tried [ $env:NODE="n1"; $env:PORT="4000"; mix phx.server], which initially has built up a stack inside the source code folder, then it started, and now I am getting something like this:

" [error] unable to establish initial redis connection. Attempting to reconnect...

[info] Running Chat.Endpoint with Cowboy using http://0.0.0.0:4001

[error] Failed to connect to Redis (localhost:6379): connection refused"

Redis container is up, and I can reach it via redis-client and can also [" docker exec -it redis redis-cli -h localhost -p 6379 ping"] with a valid PONG response.

Environment wise I have these: [apm -v]

apm 2.1.3

npm 6.2.0

node 8.9.3 x64

atom 1.35.1

python 3.6.8

git 2.21.0.windows.1

visual studio 2015

Hi 😃

It seems that Docker for Windows doesn't bind ports in localhost, instead it uses a different ip. Try using docker-machine ip to get the ip to use instead of localhost.

You should set this ip in the redis adapter configuration

config :chat, Chat.Endpoint,
  ...
  pubsub: [\
    name: Chat.PubSub,\
    adapter: Phoenix.PubSub.Redis,\
    host: IP HERE, port: 6379,\
    node_name: System.get_env("NODE")\
 ]

Hi Alvise,

Thank you for the tip! I went through the Docker for Windows documentation and read a bit more about the networking stuff ( https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/linux-containers).

Basically, it behaves differently depending on your choice between Linux and Windows (LCOW) containers. Since I am running the latest Docker for Windows version, " docker-machine" is not required anymore.

I had no time to try out every single combination of settings, but here are some results:

1). Windows containers / Hyper-V Isolation (LCOW).

- It only works as expected if I switch the Daemon to " Experimental".

- The command I used to pull the exposed container IP: docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'

- I run the whole thing twice:

docker run --name redis -d redis / default; no explicit port publishing

PORTS: 6379/tcp

docker run --name redis -d -p 6379:6379 redis / with explicit port publishing

PORTS: 0.0.0.0:6379->6379/tcp

- Started the two chat servers:

$env:NODE="n1"; $env:PORT="4000"; mix phx.server

$env:NODE="n2"; $env:PORT="4001"; mix phx.server

CONCLUSION: It worked as expected regardless I used " localhost" or " container_IP" in the main config.exs file.

2). Linux containers / Linux VM.

- It does not require to have the Daemon in " Experimental" mode.

- I executed the exact same set of commands as with the Windows containers, but the results varied as shown below:

- docker run --name redis -d redis -> This will not work even if I switch the config.exs file to use the exposed container IP.

- docker run --name redis -d -p 6379:6379 redis -> This will only work if I have " localhost" in the config.exs file.

I chose to stay with the Linux containers, just because the environment it's more stable and supports most of the vanilla Docker functionality out of the box.

Once again, many thanks for your work here.

Cheers,

Luc

Thanks a lot Luc, a really great detailed explanation for Windows users!

Reading your examples I just noticed that in the example "-p 6379:6379" was missing 😅Sorry for that, my fault!

I agree with you to stay with linux containers, especially if you use linux in your production (cloud?) environment! I would use Windows containers only if I have to deploy on Windows Server..maybe..

Cheers,

Alvise

Very useful. Tanks

Load more comments

live.rezync.com