Initial commit
parent
b501498401
commit
b1b0aeb69d
@ -0,0 +1,7 @@
|
||||
This project is open source, and your contributions are all welcome. There are mostly three different ways one can contribute to the project:
|
||||
|
||||
1. use Podop, either on test or on production servers, and report meaningful bugs when you find some;
|
||||
2. write and publish, or contribute to mail distributions based on Podop, like Mailu;
|
||||
2. contribute code and/or configuration to the repository (see [the development guidelines](https://mailu.io/contributors/guide.html) for details);
|
||||
|
||||
Either way, keep in mind that the code you write must be licensed under the same conditions as the project itself. Additionally, all contributors are considered equal co-authors of the project.
|
@ -0,0 +1,25 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2018 All Podop contributors at the date
|
||||
|
||||
This software consists of voluntary contributions made by multiple individuals.
|
||||
For exact contribution history, see the revision history available at
|
||||
https://github.com/Mailu/podop.git
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
@ -0,0 +1,112 @@
|
||||
Podop is a piece of middleware designed to run between Postfix or Dovecot
|
||||
on one side, any Python implementation of a table lookup protocol on the
|
||||
other side.
|
||||
|
||||
It is thus able to forward Postfix maps and Dovecot dicts to the same
|
||||
(or multiple) backends in order to write a single, more flexible backend
|
||||
for a mail distribution.
|
||||
|
||||
Examples
|
||||
========
|
||||
|
||||
- Connect Postfix to a DNS lookup so that every domain that has a proper MX
|
||||
record to your Postfix is actually accepted as a local domain
|
||||
- Connect both Postfix and Dovecot to an HTTP microservice to run a high
|
||||
availability microservice-based mail service
|
||||
- Use a single database server running any Python-compatible API for both
|
||||
your Postfix and Dovecot servers
|
||||
|
||||
Configure Podop tables
|
||||
======================
|
||||
|
||||
Podop tables are configured through CLI arguments when running the server.
|
||||
You must provide a ``--name`` for the table, a ``--type`` for the table and
|
||||
a ``--param`` that parametrizes the map.
|
||||
|
||||
URL table
|
||||
---------
|
||||
|
||||
The URL table will initiate an HTTP GET request for read access and an HTTP
|
||||
POST request for write access to a table. The table is parametrized with
|
||||
a template URL containing ``§`` (or ``{}``) for inserting the table key.
|
||||
|
||||
```
|
||||
--name test --type url --param http://microservice/api/v1/map/tests/§
|
||||
```
|
||||
|
||||
GET requests should return ``200`` and a JSON-encoded object
|
||||
that will be passed either to Postfix or Dovecot. They should return ``4XX``
|
||||
for access issues that will result in lookup miss, and ``5XX`` for backend
|
||||
issues that will result in a temporary failure.
|
||||
|
||||
POST requests will contain a JSON-encoded object in the request body, that
|
||||
will be saved in the table.
|
||||
|
||||
Postfix usage
|
||||
=============
|
||||
|
||||
In order to access Podop tables from Postfix, you should setup ``socketmap``
|
||||
Postfix maps. For instance, in order to access the ``test`` table on a Podop
|
||||
socket at ``/tmp/podop.socket``, use the following setup:
|
||||
|
||||
```
|
||||
virtual_alias_maps = socketmap:unix:/tmp/podop.socket:test
|
||||
```
|
||||
|
||||
Multiple maps or identical maps can be configured for various usages.
|
||||
|
||||
```
|
||||
virtual_alias_maps = socketmap:unix:/tmp/podop.socket:alias
|
||||
virtual_mailbox_domains = socketmap:unix:/tmp/podop.socket:domain
|
||||
virtual_mailbox_maps = socketmap:unix:/tmp/podop.socket:alias
|
||||
```
|
||||
|
||||
In order to simplify the configuration, you can setup a shortcut.
|
||||
|
||||
```
|
||||
podop = socketmap:unic:/tmp/podop.socket
|
||||
virtual_alias_maps = ${podop}:alias
|
||||
virtual_mailbox_domains = ${podop}:domain
|
||||
virtual_mailbox_maps = ${podop}:alias
|
||||
```
|
||||
|
||||
Dovecot usage
|
||||
=============
|
||||
|
||||
In order to access Podop tables from Dovecot, you should setup a ``proxy``
|
||||
Dovecot dictionary. For instance, in order to access the ``test`` table on
|
||||
a Podop socket at ``/tmp/podop.socket``, use the following setup:
|
||||
|
||||
```
|
||||
mail_attribute_dict = proxy:/tmp/podop.socket:test
|
||||
```
|
||||
|
||||
Multiple maps or identical maps can be configured for various usages.
|
||||
|
||||
```
|
||||
mail_attribute_dict = proxy:/tmp/podop.socket:meta
|
||||
|
||||
passdb {
|
||||
driver = dict
|
||||
args = /etc/dovecot/auth.conf
|
||||
}
|
||||
|
||||
userdb {
|
||||
driver = dict
|
||||
args = /etc/dovecot/auth.conf
|
||||
}
|
||||
|
||||
# then in auth.conf
|
||||
uri = proxy:/tmp/podop.socket:auth
|
||||
iterate_disable = yes
|
||||
default_pass_scheme = plain
|
||||
password_key = passdb/%u
|
||||
user_key = userdb/%u
|
||||
```
|
||||
|
||||
Contributing
|
||||
============
|
||||
|
||||
Podop is free software, open to suggestions and contributions. All
|
||||
components are free software and compatible with the MIT license. All
|
||||
the code is placed under the MIT license.
|
@ -0,0 +1,44 @@
|
||||
""" Podop is a *Po*stfix and *Do*vecot proxy
|
||||
|
||||
It is able to proxify postfix maps and dovecot dicts to any table
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from podop import postfix, dovecot, table
|
||||
|
||||
|
||||
SERVER_TYPES = dict(
|
||||
postfix=postfix.SocketmapProtocol,
|
||||
dovecot=dovecot.DictProtocol
|
||||
)
|
||||
|
||||
TABLE_TYPES = dict(
|
||||
url=table.UrlTable
|
||||
)
|
||||
|
||||
|
||||
def run_server(server_type, socket, tables):
|
||||
""" Run the server, given its type, socket path and table list
|
||||
|
||||
The table list must be a list of tuples (name, type, param)
|
||||
"""
|
||||
# Prepare the maps
|
||||
table_map = {
|
||||
name: TABLE_TYPES[table_type](param)
|
||||
for name, table_type, param in tables
|
||||
}
|
||||
# Run the main loop
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
loop = asyncio.get_event_loop()
|
||||
server = loop.run_until_complete(loop.create_unix_server(
|
||||
SERVER_TYPES[server_type].factory(table_map), socket
|
||||
))
|
||||
try:
|
||||
loop.run_forever()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
server.close()
|
||||
loop.run_until_complete(server.wait_closed())
|
||||
loop.close()
|
@ -0,0 +1,95 @@
|
||||
""" Dovecot dict proxy implementation
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
|
||||
class DictProtocol(asyncio.Protocol):
|
||||
""" Protocol to answer Dovecot dict requests, as implemented in Dict proxy.
|
||||
|
||||
There is very little documentation about the protocol, most of it was
|
||||
reverse-engineered from :
|
||||
|
||||
https://github.com/dovecot/core/blob/master/src/dict/dict-connection.c
|
||||
https://github.com/dovecot/core/blob/master/src/dict/dict-commands.c
|
||||
https://github.com/dovecot/core/blob/master/src/lib-dict/dict-client.h
|
||||
"""
|
||||
|
||||
DATA_TYPES = {0: str, 1: int}
|
||||
|
||||
def __init__(self, table_map):
|
||||
self.table_map = table_map
|
||||
self.major_version = None
|
||||
self.minor_version = None
|
||||
self.dict = None
|
||||
super(DictProtocol, self).__init__()
|
||||
|
||||
def connection_made(self, transport):
|
||||
logging.info('Connect {}'.format(transport.get_extra_info('peername')))
|
||||
self.transport = transport
|
||||
|
||||
def data_received(self, data):
|
||||
logging.debug("Received {}".format(data))
|
||||
results = []
|
||||
for line in data.split(b"\n"):
|
||||
logging.debug("Line {}".format(line))
|
||||
if len(line) < 2:
|
||||
continue
|
||||
command = DictProtocol.COMMANDS.get(line[0])
|
||||
if command is None:
|
||||
logging.warning('Unknown command {}'.format(line[0]))
|
||||
return self.transport.abort()
|
||||
args = line[1:].strip().split(b"\t")
|
||||
try:
|
||||
future = command(self, *args)
|
||||
if future:
|
||||
results.append(future)
|
||||
except Exception:
|
||||
logging.exception("Error when processing request")
|
||||
return self.transport.abort()
|
||||
logging.debug("Results {}".format(results))
|
||||
return asyncio.gather(*results)
|
||||
|
||||
def process_hello(self, major, minor, value_type, user, dict_name):
|
||||
self.major, self.minor = int(major), int(minor)
|
||||
logging.debug('Client version {}.{}'.format(self.major, self.minor))
|
||||
assert self.major == 2
|
||||
self.value_type = DictProtocol.DATA_TYPES[int(value_type)]
|
||||
self.user = user
|
||||
self.dict = self.table_map[dict_name.decode("ascii")]
|
||||
logging.debug("Value type {}, user {}, dict {}".format(
|
||||
self.value_type, self.user, dict_name))
|
||||
|
||||
async def process_lookup(self, key):
|
||||
logging.debug("Looking up {}".format(key))
|
||||
result = await self.dict.get(key.decode("utf8"))
|
||||
if result is not None:
|
||||
if type(result) is str:
|
||||
response = result.encode("utf8")
|
||||
elif type(result) is bytes:
|
||||
response = result
|
||||
else:
|
||||
response = json.dumps(result).encode("ascii")
|
||||
return self.reply(b"O", response)
|
||||
else:
|
||||
return self.reply(b"N")
|
||||
|
||||
def reply(self, command, *args):
|
||||
logging.debug("Replying {} with {}".format(command, args))
|
||||
self.transport.write(command)
|
||||
self.transport.write(b"\t".join(
|
||||
arg.replace(b"\t", b"\t\t") for arg in args
|
||||
))
|
||||
self.transport.write(b"\n")
|
||||
|
||||
@classmethod
|
||||
def factory(cls, table_map):
|
||||
""" Provide a protocol factory for a given map instance.
|
||||
"""
|
||||
return lambda: cls(table_map)
|
||||
|
||||
COMMANDS = {
|
||||
ord("H"): process_hello,
|
||||
ord("L"): process_lookup
|
||||
}
|
@ -0,0 +1,115 @@
|
||||
""" Postfix map proxy implementation
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
|
||||
class NetstringProtocol(asyncio.Protocol):
|
||||
""" Netstring asyncio protocol implementation.
|
||||
|
||||
For protocol details, see https://cr.yp.to/proto/netstrings.txt
|
||||
"""
|
||||
|
||||
# Length of the smallest allocated buffer, larger buffers will be
|
||||
# allocated dynamically
|
||||
BASE_BUFFER = 1024
|
||||
|
||||
# Maximum length of a buffer, will crash when exceeded
|
||||
MAX_BUFFER = 65535
|
||||
|
||||
def __init__(self):
|
||||
super(NetstringProtocol, self).__init__()
|
||||
self.init_buffer()
|
||||
|
||||
def init_buffer(self):
|
||||
self.len = None # None when waiting for a length to be sent)
|
||||
self.separator = -1 # -1 when not yet detected (str.find)
|
||||
self.index = 0 # relative to the buffer
|
||||
self.buffer = bytearray(NetstringProtocol.BASE_BUFFER)
|
||||
|
||||
def data_received(self, data):
|
||||
# Manage the buffer
|
||||
missing = len(data) - len(self.buffer) + self.index
|
||||
if missing > 0:
|
||||
if len(self.buffer) + missing > NetstringProtocol.MAX_BUFFER:
|
||||
raise IOError("Not enough space when decoding netstring")
|
||||
self.buffer.append(bytearray(missing + 1))
|
||||
new_index = self.index + len(data)
|
||||
self.buffer[self.index:new_index] = data
|
||||
self.index = new_index
|
||||
# Try to detect a length at the beginning of the string
|
||||
if self.len is None:
|
||||
self.separator = self.buffer.find(0x3a)
|
||||
if self.separator != -1 and self.buffer[:self.separator].isdigit():
|
||||
self.len = int(self.buffer[:self.separator], 10)
|
||||
# Then get the complete string
|
||||
if self.len is not None:
|
||||
if self.index - self.separator == self.len + 2:
|
||||
string = self.buffer[self.separator + 1:self.index - 1]
|
||||
self.init_buffer()
|
||||
self.string_received(string)
|
||||
|
||||
def string_received(self, string):
|
||||
pass
|
||||
|
||||
def send_string(self, string):
|
||||
""" Send a netstring
|
||||
"""
|
||||
self.transport.write(str(len(string)).encode('ascii'))
|
||||
self.transport.write(b':')
|
||||
self.transport.write(string)
|
||||
self.transport.write(b',')
|
||||
|
||||
|
||||
class SocketmapProtocol(NetstringProtocol):
|
||||
""" Protocol to answer Postfix socketmap and proxify lookups to
|
||||
an outside object.
|
||||
|
||||
See http://www.postfix.org/socketmap_table.5.html for details on the
|
||||
protocol.
|
||||
|
||||
A table map must be provided as a dictionary to lookup tables.
|
||||
"""
|
||||
|
||||
def __init__(self, table_map):
|
||||
self.table_map = table_map
|
||||
super(SocketmapProtocol, self).__init__()
|
||||
|
||||
def connection_made(self, transport):
|
||||
logging.info('Connect {}'.format(transport.get_extra_info('peername')))
|
||||
self.transport = transport
|
||||
|
||||
def string_received(self, string):
|
||||
space = string.find(0x20)
|
||||
if space != -1:
|
||||
name = string[:space].decode('ascii')
|
||||
key = string[space+1:].decode('utf8')
|
||||
return asyncio.async(self.process_request(name, key))
|
||||
|
||||
def send_string(self, string):
|
||||
logging.debug("Send {}".format(string))
|
||||
super(SocketmapProtocol, self).send_string(string)
|
||||
|
||||
async def process_request(self, name, key):
|
||||
""" Process a request by querying the provided map.
|
||||
"""
|
||||
logging.debug("Request {}/{}".format(name, key))
|
||||
try:
|
||||
table = self.table_map.get(name)
|
||||
except KeyError:
|
||||
return self.send_string(b'TEMP no such map')
|
||||
try:
|
||||
result = await table.get(key)
|
||||
return self.send_string(b'OK ' + str(result).encode('utf8'))
|
||||
except KeyError:
|
||||
return self.send_string(b'NOTFOUND ')
|
||||
except Exception:
|
||||
logging.exception("Error when processing request")
|
||||
return self.send_string(b'TEMP unknown error')
|
||||
|
||||
@classmethod
|
||||
def factory(cls, table_map):
|
||||
""" Provide a protocol factory for a given map instance.
|
||||
"""
|
||||
return lambda: cls(table_map)
|
@ -0,0 +1,26 @@
|
||||
""" Table lookup backends for podop
|
||||
"""
|
||||
|
||||
import aiohttp
|
||||
import logging
|
||||
|
||||
|
||||
class UrlTable(object):
|
||||
""" Resolve an entry by querying a parametrized GET URL.
|
||||
"""
|
||||
|
||||
def __init__(self, url_pattern):
|
||||
""" url_pattern must contain a format ``{}`` so the key is injected in
|
||||
the url before the query, the ``§`` character will be replaced with
|
||||
``{}`` for easier setup.
|
||||
"""
|
||||
self.url_pattern = url_pattern.replace('§', '{}')
|
||||
|
||||
async def get(self, key):
|
||||
logging.debug("Getting {} from url table".format(key))
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(self.url_pattern.format(key)) as request:
|
||||
if request.status == 200:
|
||||
result = await request.json()
|
||||
logging.debug("Got {} from url table".format(result))
|
||||
return result
|
@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
import argparse
|
||||
|
||||
from podop import run_server, SERVER_TYPES, TABLE_TYPES
|
||||
|
||||
|
||||
def main():
|
||||
""" Run a podop server based on CLI arguments
|
||||
"""
|
||||
parser = argparse.ArgumentParser("Postfix and Dovecot proxy")
|
||||
parser.add_argument("--socket", help="path to the socket", required=True)
|
||||
parser.add_argument("--mode", choices=SERVER_TYPES.keys(), required=True)
|
||||
parser.add_argument("--name", help="name of the table", action="append")
|
||||
parser.add_argument("--type", choices=TABLE_TYPES.keys(), action="append")
|
||||
parser.add_argument("--param", help="table parameter", action="append")
|
||||
args = parser.parse_args()
|
||||
run_server(
|
||||
args.mode, args.socket,
|
||||
zip(args.name, args.type, args.param) if args.name else []
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
from distutils.core import setup
|
||||
|
||||
setup(
|
||||
name="Podop",
|
||||
version="0.1",
|
||||
description="Postfix and Dovecot proxy",
|
||||
author="Pierre Jaury",
|
||||
author_email="pierre@jaury.eu",
|
||||
url="https://github.com/mailu/podop.git",
|
||||
packages=["podop"],
|
||||
scripts=["scripts/podop"]
|
||||
)
|
Loading…
Reference in New Issue