If you've ever had the frustrating experience of trying to debug the websocket configuration in Odoo (version 16+) then I have some information that might be useful for you.
If you just pull the official odoo docker image and run it locally, you might be all too familiar with this error:
File "/usr/lib/python3/dist-packages/odoo/addons/bus/controllers/websocket.py", line 20, in websocket
return WebsocketConnectionHandler.open_connection(request, version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/odoo/addons/bus/websocket.py", line 1009, in open_connection
raise RuntimeError(
RuntimeError: Couldn't bind the websocket. Is the connection opened on the evented port (8072)?
And it won't take you long to find the offending line of code:
request.httprequest._HTTPRequest__environ['socket']
And you'll see that the key 'socket' is not in the environ dictionary.
But why?
How do web-sockets work?
Web-sockets allow for 2-way communication over the TCP protocol.
Normal HTTP traffic goes over a TCP connection. The client sends the HTTP Request over TCP, then the server responds with the HTTP Response over the same TCP connection, which is then closed.
For 2-way traffic, typically we can use a technology like a web-socket. One way to initialise a connection is over HTTP initially, with one slight difference. It is not a normal HTTP request, we actually pass in some special headers requesting the server upgrade it to a web-socket connection.
If the server accepts this, it will send back the HTTP Response, but this time it will not close the TCP connection. Then, the same underlying TCP connection that was used to make the initial HTTP request, just gets used for the web-socket communication and stays open as long as required.
Note that the web-socket protocol actually allows for the possibility to initialise web-sockets not over HTTP, but typically this is how it is done. They designed the protocol this way to future-proof it, and also allow it to be backwards compatible with the existing HTTP infrastructure.
We can back-trace this
After a lot of logging and tracing execution through the server I have learned some interesting things.
While this error occurs in the following file:
odoo.addons.bus.websocket
there are 2 more files of interest here, these are:
odoo.http
but actually more relevant to us, is:
odoo.service.server
The underlying web server code is using the werkzeug library. The server.py file provides a lot of the configuration, and various wrappers around this lower-level web server library, and sets up the application entry point.
The http.py file provides the logic of how to actually process each request, and so does the routing and various other tasks. But the websocket file throws an error when there is a missing environment variable. Where does this environment variable get set?
If you inspect the server.py file, you will see this happens in two methods:
class RequestHandler(CommonRequestHandler):
...
def make_environ(self):environ = super().make_environ()
environ['socket'] = self.connection
if self.headers.get('Upgrade') == 'websocket':
self.protocol_version = "HTTP/1.1"
return environ
And in:
class GeventServer(CommonServer):
...
class ProxyHandler(WSGIHandler):
def get_environ(self):environ = super().get_environ()
environ['socket'] = self.socket
if self._connection_upgrade_requested():
environ['wsgi.input'] = self.rfile
environ['wsgi.input_terminated'] = False
return environ
The first class is the most illuminating. You'd think that this method would be called, as expected, and the socket key correctly set in the environment.
But if you put some logging statements in there, you will see that method is never called.
Why is that?
How Odoo handles requests
You can actually "fix" this error by changing this line in your odoo.conf:
workers = 2
to this:
workers = 0
So what does this configuration option do?
Threaded vs Pre-Fork architecture
If you want your server to be able to handle more than one client request at a time, you have to figure out some method of concurrently handling requests. There are a few ways of doing this, but ultimately you will need some concurrent processing model which will be some combination of either processes, threads or asynchronous event loops.
By using Linux fork call, you can copy the actual process that is running, so you end up with multiple versions of the server running, i.e. separate processes. The OS is then responsible for scheduling the processing workload on the CPU, in the same way you can also have postgresql running on the same machine as your web server. Apache uses this model.
By using threads, you can have a single process managing the concurrency. Each thread can run on a different CPU core, but the application has a bit more control over the threads as they are running inside the master process address space.
Using an async event loop, you can have multiple requests being handled by a single thread, where the async jobs basically share the thread, and whenever one needs to wait for something to happen (like read from disk) it yields control of the thread to the next async task. Node.js uses this architecture.
Web servers like nginx use a combination, where it has many forked process, but each process handles thousands of connections using async / await architecture.
The idea of "pre-forking", means that rather than waiting for an incoming connection to fork a new process to handle it, you pre-fork maybe 10-20 processes and leave them on stand-by. Then when new connections come in the master process can pass them over to one of the waiting processes in a pool.
Odoo Multi-threaded mode
For local development, it's ok to use multi-threaded mode.
This means you set the workers parameter to 0. In this case, Odoo will spawn a new thread for each incoming connection. If that is an HTTP request, or a long-running web-socket connection, or even a cron job. Doesn't matter.
This will then utilise the RequestHandler class, and you'll see that the 'socket' key gets correctly set in the make_environ method.
The browser will be trying to establish web-socket connections here:
localhost:8069/websocket
And you can test this in postman by trying to establish a web-socket connection here also.
This means you can get rid of those annoying messages on your local dev setup, but it is not recommended to use this in production.
Odoo Multi-process mode
Once you configure additional workers, Odoo will "pre-fork" these and run them on separate CPU cores (if available). This obviously gets better performance. It will however not run cron jobs on these workers, nor will it deal with web-sockets.
You can see the logic for that here:
if odoo.evented:
server = GeventServer(odoo.http.root)
elif config['workers']:
server = PreforkServer(odoo.http.root)
As far as I can tell, this method gets called twice with different parts of the config passed in each time, thereby setting up both the regular http server, and the web-socket gevent server.
What Odoo actually does is have each worker use the BaseWSGIServerNoBind class to handle the request. This is not bound to any network interface, and is sort of the base logic that werkzeug uses to process a request, minus all the networking.
This class will then call http.py and everything continues as in multi-threaded mode. However, this bypasses the RequestHandler class and the make_environ method is never called. So the socket key will be missing from environ.
So when you try and hit this endpoint:
localhost:8069/websocket
It will not work.
But let's think back to that second class, the GeventServer class. This had a get_environ method, and there it would set the socket key. So how do we trigger that?
Well the GeventServer is running on the gevent_port, which is usually 8072 by default.
If you fire up postman and try and establish a websocket connection to localhost:8072/websocket, then you will find this succeeds also. But now this response is being handled by the GeventServer class, which adds the socket field correctly (in the get_environ method) before handing it over to http.py for processing.
So in this scenario, we must have a reverse proxy sitting in front of the Odoo server, to accept the connection on localhost:8069/websocket endpoint, but in reality forward that connection request to localhost:8072/websocket.
It is recommended to have some sort of load balancer or reverse proxy running in front of Odoo anyway in production. If you are running this on your own server then you will need to have nginx sitting in front of it, configured this way.
However, if you are behind an AWS load balancer for example, you'd need a listener rule to intercept the /websocket url paths, and send them to a different target group that is pointed at port 8072 (or whatever port you are running gevent on).