Email notification debug

Software SW help
1

Hi guys,

TL;DR: how can I debug the Foris email notification sending?

Context:

I’m trying to get my Omnia to email notifications using my in-lab Exchange server. Unsuccesfully, so far.

I’m currently running 5.1.3, and I can remember it was working on 3.x (can’t really remember if it worked until 3.11.19, or it broke earlier). Anyway, I’ve succesfully sent emails over SMTP from that server, using STARTTLS on port 25. Including from the Omnia.

sudo tail -f /var/log/messages (I avoid running as root) doesn’t show anything when I try to send a test message. I get the error message in Foris though.

So. Any ideas about how I can enable logging? Anybody who’s ever looked at Exchange logs can testify that digging through those is a cruel and unusual punishment, so I’d very much prefer to do it from my Omnia instead.

Thanks guys,
Mircea

2

can you try to capture packets using tcpdump?

3

I probably could, but then again, that’d be as bad as digging through th Exchange logs :slight_smile:

Problem is, the stuff under /usr/lib/python3.7/site-packages/foris* is, to my C-trained eyes, an unholy mess (no offense intended - it’s just that the code I normally look at is… rather different).
And I was hoping somebody could point me to the exact place where the code sending notifications by email resides - I could take it from there.
Since I’m not much of a python guy, this is unlikely to be painless, but compared to digging through logs and/or IP packets, at least I’d learn something new :slight_smile:

Thanks anyway!

4

this is a good reason for request a better logging of e-mail notifications.

5

Hello.

No need to dig through Foris code.

To send a Foris notification via e-mail, you should set the severity to receive all notification (at least to start debugging):

Screenshot from 2020-11-07 23-45-38
Screenshot from 2020-11-07 23-45-38748×141 7.7 KB

If you want to use STARTTLS to send e-mails via a custom mail server, do not forget to set correct port 587 (or 25, depends on your mail server preferences):

Screenshot from 2020-11-07 23-52-13
Screenshot from 2020-11-07 23-52-13759×190 5.13 KB

(But you should prefer to use TLS/SSL method on port 465 which is recommended by latest standards/RFCs.)

Then, (since you are familiar with SSH and CLI), to send an instant testing notification simply run

create_notification -t -s error 'This is a testing notification'

While looking into logs in the other terminal:

tail -f /var/log/messages

If anything goes wrong, you should see it either on stderr or in the logs.

BTW ignore output like

cat: can't open '/.../message_en': No such file or directory

this is just a “warning by mistake” that would be suppressed in next release.

2 Likes
6

still not bad to log it into messages :slight_smile:

7

Actually, that was very useful, I found my problem:

msmtp: TLS certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.
msmtp: could not send mail (account default from /tmp/user_notify/.locked/msmtp.cfg.14673)
notifier: msmtp has failed to send e-mail notification

this makes a lot of sense, since I run my own CA.

For the time being, I renamed /usr/bin/msmtp to /usr/bin/msmtp-bin and created (755) /usr/bin/msmtp containing

#!/bin/sh
echo "$0 $*"
msmtp-bin --tls-certcheck=off $*

Sure enough, that worked like a charm.

Now, an option to disable TLS certificate checking wouldn’t hurt in the UI :slight_smile:

But for now, I’ve solved my problem. Thank you very much!

1 Like
8

Actually, I take back the “an option to disable TLS certificate checking wouldn’t hurt in the UI” bit.

It’d be great if an option existed in /etc/config/user_notify though.
But it shouldn’t be published in the UI. People who know what they’re doing should be comfy with vi /etc/config/user_notify; leaving that option around the UI for the average user to mess with makes little sense.

My reasoning is, you don’t want to expose a configuration option that
a) reduces security, if you don’t know what you’re doing, and
b) has maybe 3 users
but you don’t want those 3 users having to mess with /usr/bin everytime msmtp gets updated either (or even better, stop receiving notifications if they should forget to fixup their msmtp).

10

For an encrypted communication, it would not help a lot. (Like the OP, this was a TLS problem, tcpdump would be useful only for TCP or IP issues and I already checked the connectivity from a container with openssl)

11

Indeed. The complete lack of error messages when testing notifications in Foris or Reforis is a big problem.

12

issue created, hopefully in proper project

13

Thanks for for very useful command, at least now, I know what the problem is.

1 Like
14

The solution I use when the default TurrisOS does not know my CA. Example with CAcert :

  1. Copy the cert of the CA on the Turris, in /etc/ssl/certs/CAcert-class3.crt

  2. Add to the list: cat CAcert-class3.crt >> ca-certificates.crt

  3. May be (I’m not sure) create a symbolic link to the certificat:

cd /usr/share/ca-certificates
openssl x509 -hash -noout -in CAcert-class3.crt (on another machine, TurrisOS has no openssl)
ln -s CAcert-class3.crt RESULT-OF-ABOVE-COMMAND.0
15

That’d be great, if OpenWRT regenerated ca-certificates.crt instead of hardcoding, and it seems (sudo opkg files ca-certificates|grep ca-cert) that is the case indeed.

Btw, re. “(on another machine, TurrisOS has no openssl)”: sudo opkg update; sudo opkg install openssl-util; which openssl

EDIT: Worked like a charm. It also seems adding just the root CA is enough - the TLS CA, signed by the root CA, is not needed

$ cd /usr/ssl/certs
$ sudo openssl x509 -in ~/pki/ca/ca-root.crt -out My_Root_CA.crt
$ sudo ln -s My_Root_CA.crt $(openssl x509 -in My_Root_CA.crt -noout -hash).0
$ cat My_Root_CA.crt | sudo tee -a ca-certificates.crt
$ ls -l | grep My_Root_CA
lrwxrwxrwx    1 root     root            nn Nov 12 13:36 xxxxxxxx.0 -> My_Root_CA.crt
-rw-r--r--    1 root     root          nnnn Nov 12 13:36 My_Root_CA.crt
16

Be aware that the ca-certificates.crt file is managed by ca-bundle package so it would be overwritten when this package is updated.

I made quick search on OpenWrt wiki but I was not successful to find some more systematic approach.

What about to get a trusted certificate? It should not be so hard since we’ve got Let’s Encrypt here for a while. You would also welcome a trusted certificate for a public-accessible server.

this can be managed by c_rehash utility

1 Like
17

That’s very good to know, I assumed you’d find it in ca-certificates :expressionless:
This is exactly why I’m using Fedora for work instead of Ubuntu - sane certificate management.

Oh well. Think I’ll just drop a script to /etc/init.d to do this on my behalf, and symlink it to /etc/rc.d/S99...

test -f /etc/ssl/certs/my-ca.crt \
  ||( openssl x509 -in /root/ca/pki/ca-root.crt -out /etc/ssl/certs/my-ca.crt \
      && c_rehash )
grep $(head -2 /etc/ssl/certs/my-ca.crt | tail -1) /etc/ssl/certs/ca-certificates.crt \
  || cat /etc/ssl/certs/my-ca.crt >> /etc/ssl/certs/ca-certificates.crt

Dirty, but should do the trick. Anyway, it’s not worse than keeping msmtp a wrapper over the actual msmtp.

Ummm no. It’s a home lab. I want stuff to start throwing SSL errors unless I had installed my CA on the device, on purpose :slight_smile:

Thank you very much for both the ca_bundle and the c_rehash pointers!

18

Ubuntu have nothing to do with OpenWrt packages. Also AFAIK, Ubuntu is based on Debian where you can simply add custom certificates to /usr/local/share/ca-certificates which is included in system CA bundle on update. This feature, seems to be missing in OpenWrt.

Ok, I understand.

To deliver notifications, you can freely use our SMTP relay, which works out-of-the-box on any Turris router (and our servers have trusted certificates so this hacky script won’t be needed). This is achieved by selecting a “Turris servers” in (re)Foris configuration.

You’re welcome :wink: