[Bug 1580356] [NEW] OpenVPN causes reboot failure on Xenial in AWS

[Scott Crooks]

Public bug reported:

The problem actually has to do with the way `systemd` is configured in
the latest version of Ubuntu 16.04. Setting up a typical OpenVPN
configuration, and rebooting causes a Instance Status check failure in
AWS EC2. The reason is because the file
`/lib/systemd/system/***@.service` is missing the following line:

After=network.target network-online.target

The entire [Unit] section then looks like:

[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
After=network.target network-online.target
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

This ensures that routes needed for OpenVPN configurations get called
after the main networking stack.

See issues here:

- https://forums.aws.amazon.com/thread.jspa?messageID=719570
- https://sourceforge.net/p/openvpn/mailman/message/35078170/

** Affects: openvpn (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Simon Déziel]

Scott, could you please provide some information about the routing/IP
configuration before the reboot?

** Changed in: openvpn (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

Simon,

Sure. Would you prefer a Github Gist, or can I paste the outputs here
directly?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

Simon,

Also, which commands do you want to see? Just `ip route` ?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Simon Déziel]

If you could attach "ip addr" and "ip route" output here that would be
good, thanks.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

Simon,

The setup I have is running two OpenVPN daemons on the same port (1618)
but on different protocols (TCP/UDP). The post that Chris made in the
AWS forum above had a much simpler, single virtual interface setup.

***@ip-10-0-0-9:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
link/ether 06:cc:43:0f:66:e9 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.9/28 brd 10.0.0.15 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::4cc:43ff:fe0f:66e9/64 scope link
valid_lft forever preferred_lft forever
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
link/none
inet 10.255.248.1/23 brd 10.255.249.255 scope global tun0
valid_lft forever preferred_lft forever
4: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
link/none
inet 10.255.250.1/23 brd 10.255.251.255 scope global tun1
valid_lft forever preferred_lft forever



***@ip-10-0-0-9:~# ip route
default via 10.0.0.1 dev eth0
10.0.0.0/28 dev eth0 proto kernel scope link src 10.0.0.9
10.255.248.0/23 dev tun0 proto kernel scope link src 10.255.248.1
10.255.250.0/23 dev tun1 proto kernel scope link src 10.255.250.1


***@ip-10-0-0-9:~# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1618 0.0.0.0:* LISTEN 2875/openvpn
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1059/sshd
tcp6 0 0 :::22 :::* LISTEN 1059/sshd
udp 0 0 0.0.0.0:68 0.0.0.0:* 876/dhclient
udp 0 0 0.0.0.0:1618 0.0.0.0:* 2874/openvpn
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 8782 1/init /run/lvm/lvmpolld.socket
unix 2 [ ACC ] STREAM LISTENING 20818 2974/systemd /run/user/1000/systemd/private
unix 2 [ ACC ] SEQPACKET LISTENING 8864 1/init /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 8783 1/init /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 9033 1/init /run/lvm/lvmetad.socket
unix 2 [ ACC ] STREAM LISTENING 10909 1/init /run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 20850 3012/0 /tmp/ssh-zhPNN6x7d9/agent.3012
unix 2 [ ACC ] STREAM LISTENING 10899 1/init /run/snapd.socket
unix 2 [ ACC ] STREAM LISTENING 10824 1/init /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 10891 1/init /run/uuidd/request
unix 2 [ ACC ] STREAM LISTENING 10893 1/init /var/lib/lxd/unix.socket
unix 2 [ ACC ] STREAM LISTENING 13922 1074/iscsid @ISCSIADM_ABSTRACT_NAMESPACE
unix 2 [ ACC ] STREAM LISTENING 17650 1/init /var/run/pcscd/pcscd.comm
unix 2 [ ACC ] STREAM LISTENING 8778 1/init /run/systemd/private
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Simon Déziel]

Thanks Scott. Could you also attach the OpenVPN configs since they
apparently don't match the gist [1].

Thanks

1: https://gist.github.com/sc250024/001e05d64cd02fe746ae1772baccd24a
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

Simon,

The configuration was more or less the same, just different port
numbers. Here they are for documentation purposes. Again, the only
differences are the protocol, and the IP pools.

UDP Config:
port 1618
proto udp
dev tun
ca /etc/openvpn/openvpnudp-files/ca.crt
cert /etc/openvpn/openvpnudp-files/server.crt
key /etc/openvpn/openvpnudp-files/server.key
dh /etc/openvpn/openvpnudp-files/dh3072.pem
push "redirect-gateway def1 bypass-dhcp"
server 10.255.248.0 255.255.254.0
remote-cert-eku "TLS Web Client Authentication"
ifconfig-pool-persist /etc/openvpn/openvpnudp-files/ipp.txt
keepalive 10 120
comp-lzo yes
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 5
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"
plugin /usr/lib/openvpn/openvpn-auth-ldap.so "/etc/openvpn/openvpnudp-files/auth-ldap.conf"
username-as-common-name
cipher AES-256-CBC
topology subnet
log-append /var/log/openvpn.log
tls-auth /etc/openvpn/openvpnudp-files/tls.key 0
key-direction 0
tls-version-min 1.2
auth SHA512
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
crl-verify /etc/openvpn/openvpnudp-files/crl.pem

TCP Config:
port 1618
proto tcp
dev tun
ca /etc/openvpn/openvpntcp-files/ca.crt
cert /etc/openvpn/openvpntcp-files/server.crt
key /etc/openvpn/openvpntcp-files/server.key
dh /etc/openvpn/openvpntcp-files/dh3072.pem
push "redirect-gateway def1 bypass-dhcp"
server 10.255.250.0 255.255.254.0
remote-cert-eku "TLS Web Client Authentication"
ifconfig-pool-persist /etc/openvpn/openvpntcp-files/ipp.txt
keepalive 10 120
comp-lzo yes
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 5
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"
plugin /usr/lib/openvpn/openvpn-auth-ldap.so "/etc/openvpn/openvpntcp-files/auth-ldap.conf"
username-as-common-name
cipher AES-256-CBC
topology subnet
log-append /var/log/openvpn.log
tls-auth /etc/openvpn/openvpntcp-files/tls.key 0
key-direction 0
tls-version-min 1.2
auth SHA512
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
crl-verify /etc/openvpn/openvpntcp-files/crl.pem
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

Greetings,

Any status on this? The thread I opened on the openvpn-user mailing list
has confirmed as well that this is an issue. See here:
http://article.gmane.org/gmane.network.openvpn.user/36909
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Simon Déziel]

Hi Scott,

I just finished migrating my VPN to Xenial. I cannot reproduce your
issue using a static IP configuration in /etc/network/interfaces. I'll
try to set things up with DHCP and see how it goes.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

Simon,

Yes, the instance I used as a test was using DHCP. AWS is a bit odd in
that sense since, for all intents and purposes, your instance has a
static internal IP address that persists across reboots. However, the
instance thinks it's using DHCP. It would be as if you configured DHCP
on the OpenVPN server, but configured the LAN's DHCP server to assign
only one address to your OpenVPN server.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Robie Basak]

** Tags added: systemd-boot
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[WoJ]

I confirm that the issue is present with the AMI Ubuntu image
20160516.1. I systematically reproduce it by

- launching an instance of the AMI image on EC2
- apt update ; apt install openvpn
- configuring the OpenVN server with any configuration (static key for instance https://openvpn.net/index.php/open-source/documentation/miscellaneous/78-static-key-mini-howto.html)
- enable OpenVPN at boot
- reboot

The instance fails to configure its network for the reason described by
the OP.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Simon Déziel]

** Changed in: openvpn (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Samuli Seppänen]

This link might be of interest:

<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

Much of the same info can be viewed with

$ man 7 systemd.special

It seems that network-online.target should be used instead of
network.target. I think fixing this in openvpn.service instead of
***@.service should work, but I have not tried it, nor have I
actually tried reproducing this bug myself.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Harney]

I tested that theory of modifying openvpn.service with network-
online.target to no avail. Can confirm this issue in ubuntu/images/hvm-
ssd/ubuntu-xenial-16.04-amd64-server-20160610 (ami-d06a90b0) as well.

Will try modifying ***@service as suggested by OP to see if that
resolves.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Harney]

Doing as the OP suggested does work. the following modification of
lib/systemd/system/***@.service allows the AWS xenial instance to
reboot with openvpn service enabled

$ diff -u /lib/systemd/system/***@.service roles/openvpn_server/files/***@.service
--- /lib/systemd/system/***@.service 2016-02-02 06:33:30.000000000 -0600
+++ roles/openvpn_server/files/***@.service 2016-06-27 15:55:19.839586907 -0500
@@ -3,6 +3,7 @@
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
+After=network.target network-online.target
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Samuli Seppänen]

Upstream OpenVPN has two unit files, one for servers, one for clients:

<https://community.openvpn.net/openvpn/ticket/462>
<https://github.com/OpenVPN/openvpn/tree/master/distro/systemd>

Have you tested whether these unit files work better?

** Bug watch added: community.openvpn.net/openvpn/ #462
https://community.openvpn.net/openvpn/ticket/462
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Simon Déziel]

There is a nice explanation of the bug in the duplicate LP: #1598522.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Malte Zacharias]

*** This bug is a duplicate of bug 1598522 ***
https://bugs.launchpad.net/bugs/1598522

I can confirm the above reports as well as the description in #1595822.
OpenVPN is started too early in the boot process. Cloud-Init will default to configure tun0 instead of the guests main network interface via dhcp. This breaks the cloud-init process.

** This bug has been marked a duplicate of bug 1598522
openvpn starts before network is up in 16.04 this breaks networking in clouds
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Stas]

*** This bug is a duplicate of bug 1598522 ***
https://bugs.launchpad.net/bugs/1598522

I have just tried <https://github.com/OpenVPN/openvpn/tree/master/distro/systemd>
It works OK, however, they assume that server configuration is in /etc/openvpn/server/<name>.conf, in current ubuntu package it is: /etc/openvpn/<name>.conf that is the only issue. Everything else is great. So if you take those files from upstream, ppl will have to change directory structure a bit, not sure if that is a big issue or not.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[Scott Crooks]

*** This bug is a duplicate of bug 1598522 ***
https://bugs.launchpad.net/bugs/1598522

Is there a timeline for this fix? It seems like a pretty simple fix, and
it's a bit nerve-racking every time I have to reboot an OpenVPN server
:)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions

[David Sommerseth]

*** This bug is a duplicate of bug 1598522 ***
https://bugs.launchpad.net/bugs/1598522

I am the one who added the systemd unit files to upstream OpenVPN.
These files are now getting a few updates [1] and will ship as the
recommended unit files in the coming OpenVPN 2.4 release.

The OpenVPN team would appreciate if Ubuntu was shipping the upstream
version of these unit files. We try to make all systemd based
distributions use the same ones, as that gives less maintenance burden
on package maintainers as well as a consistent behaviour across all
distributions. Which again helps the support guys on #openvpn @
FreeNode when something isn't working.

With that said, I am happy to see that our versions resolves this issue.
If anyone is interested in helping out testing, fixing and improving
these unit files further, please do not hesitate getting in touch with
us on #openvpn-devel @ FreeNode.

[1] <https://www.mail-archive.com/openvpn-
***@lists.sourceforge.net/msg12764.html>
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1580356

Title:
OpenVPN causes reboot failure on Xenial in AWS

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1580356/+subscriptions