It looks like there is a problem with your dovecot installation. Because your fail2ban screenshot shows that there is no mail.log file. It's better to reinstall. And you have to create a fail2ban rule for dovecot definitely.
But for a quick hack, you can limit the dovecot service for consuming CPU and memory.
vi /lib/systemd/system/dovecot.service
Under Service paste following code:
MemoryLimit=2048M
CPUQuota=30%
And then
systemctl daemon-reload
service dovecot restart