[Bug 1684295] [NEW] sssd fails with 'Exiting the SSSD. Could not restart critical service [tpad].

[Pamela Skutnik]

Public bug reported:

This is Ubuntu 16.04.2 LTS

sssd is configured to connect to two domains, our TPAD directory and
Active Directory. sssd starts up at boot time. As soon as I ssh login
(with any id, AD, TPAD or local), sssd fails with the error message in
the title. After that, we can only login with local ids, not TPAD or AD
ids.

****************
Here is the output from systemctl status sssd after the failure:
***@dcmilphlum128:~# systemctl status sssd
â sssd.service - System Security Services Daemon
Loaded: loaded (/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2017-04-19 16:40:08 EDT; 7min ago
Process: 119143 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS)
Main PID: 119145 (code=exited, status=1/FAILURE)

Apr 19 16:39:47 dcmilphlum128.edc.nam.gm.com sssd[be[119187]: Starting up
Apr 19 16:39:51 dcmilphlum128.edc.nam.gm.com sssd[be[119191]: Starting up
Apr 19 16:39:57 dcmilphlum128.edc.nam.gm.com sssd[be[119206]: Starting up
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119145]: Exiting the SSSD. Could not restart critical service [tpad].
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119149]: Shutting down
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119148]: Shutting down
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[be[119146]: Shutting down
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Unit entered failed state.
Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Failed with result 'exit-code'.

******************
Also, in kern.log I have four of these (I have retries set to 3):
Apr 19 16:39:59 dcmilphlum128 kernel: [ 6205.937807] sssd_be[12218]: segfault at 0 ip 00007fecb32b6b94 sp 00007ffce49a2230 error 4 in libsss_util.so[7fecb32a2000+6c000]
Apr 19 16:40:02 dcmilphlum128 kernel: [ 6206.980725] sssd_be[12253]: segfault at 0 ip 00007f302de29b94 sp 00007fffca943cc0 error 4 in libsss_util.so[7f302de15000+6c000]
Apr 19 16:40:05 dcmilphlum128 kernel: [ 6211.036205] sssd_be[12256]: segfault at 0 ip 00007fd196169b94 sp 00007ffd624249f0 error 4 in libsss_util.so[7fd196155000+6c000]
Apr 19 16:40:07 dcmilphlum128 kernel: [ 6225.081902] sssd_be[12257]: segfault at 0 ip 00007fd1f669bb94 sp 00007ffdd8e5bf80 error 4 in libsss_util.so[7fd1f6687000+6c000]

*******************
My sssd package are at 1.13.4:
sssd 1.13.4-1ubuntu1.1 amd64
sssd-ad 1.13.4-1ubuntu1.1 amd64
sssd-ad-common 1.13.4-1ubuntu1.1 amd64
sssd-common 1.13.4-1ubuntu1.1 amd64
sssd-ipa 1.13.4-1ubuntu1.1 amd64
sssd-krb5 1.13.4-1ubuntu1.1 amd64
sssd-krb5-common 1.13.4-1ubuntu1.1 amd64
sssd-ldap 1.13.4-1ubuntu1.1 amd64
sssd-proxy 1.13.4-1ubuntu1.1 amd64
sssd-tools 1.13.4-1ubuntu1.1 amd64

***************
I upgraded all the sssd packages to 1.13.4-1ubuntu1.4 and had the same problem.

I downgraded them to 1.12.5-2 and was NOT able to reproduce the problem.

I attached my sssd.conf file.

** Affects: sssd (Ubuntu)
Importance: Undecided
Status: New

** Attachment added: "sssd.conf file"
https://bugs.launchpad.net/bugs/1684295/+attachment/4865180/+files/sssd.conf.txt
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

Andreas,
thank you for looking into this. It has been a while since I looked at this problem, since our temporary fix of holding the sssd packages at version 1.12.5-2 was working. I see that in the meantime, new versions of the sssd packages have been released. I have tested with the 1.13.4-1ubuntu1.5 and 1.13.4-1ubuntu1.6 and I am not able to reproduce the problem with those versions. It would be nice to know what fixed the issue but I do not see anything obvious in the package changelogs.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Launchpad Bug Tracker]

[Expired for sssd (Ubuntu) because there has been no activity for 60
days.]

** Changed in: sssd (Ubuntu)
Status: Incomplete => Expired
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

Sorry, finally got back to working on this. Can I continue to post here
or do I need to reopen?

I have set up a test server configured with only the one domain, tpad.
I also updated the sssd packages to 1.13.4-1ubuntu1.8, the highest I had.
It still crashes as soon as I login with any id (local or tpad).

***@dcmilphlum127:~# systemctl status sssd
â sssd.service - System Security Services Daemon
Loaded: loaded (/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2017-10-04 15:52:08 EDT; 24min ago
Process: 13480 ExecStart=/usr/sbin/sssd -i -f (code=exited, status=1/FAILURE)
Main PID: 13480 (code=exited, status=1/FAILURE)

Oct 04 15:50:48 dcmilphlum127.edc.nam.gm.com systemd[1]: Started System Security Services Daemon.
Oct 04 15:51:49 dcmilphlum127.edc.nam.gm.com sssd[be[13528]: Starting up
Oct 04 15:51:52 dcmilphlum127.edc.nam.gm.com sssd[be[13530]: Starting up
Oct 04 15:51:58 dcmilphlum127.edc.nam.gm.com sssd[be[13531]: Starting up
Oct 04 15:52:08 dcmilphlum127.edc.nam.gm.com sssd[13480]: Exiting the SSSD. Could not restart critical service [tpad].
Oct 04 15:52:08 dcmilphlum127.edc.nam.gm.com sssd[13484]: Shutting down
Oct 04 15:52:08 dcmilphlum127.edc.nam.gm.com sssd[13483]: Shutting down
Oct 04 15:52:08 dcmilphlum127.edc.nam.gm.com systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
Oct 04 15:52:08 dcmilphlum127.edc.nam.gm.com systemd[1]: sssd.service: Unit entered failed state.
Oct 04 15:52:08 dcmilphlum127.edc.nam.gm.com systemd[1]: sssd.service: Failed with result 'exit-code'.

***@dcmilphlum127:~# tail /var/log/all/kern.log

Oct 4 15:51:50 dcmilphlum127 kernel: [ 6814.060738] sssd_be[13528]: segfault at 0 ip 00007f302c49db94 sp 00007fff0b589cb0 error 4 in libsss_util.so[7f302c489000+6c000]
Oct 4 15:51:54 dcmilphlum127 kernel: [ 6818.117855] sssd_be[13530]: segfault at 0 ip 00007f6e98ac0b94 sp 00007fffc35cae40 error 4 in libsss_util.so[7f6e98aac000+6c000]
Oct 4 15:52:08 dcmilphlum127 kernel: [ 6832.177545] sssd_be[13531]: segfault at 0 ip 00007fb2b6ccdb94 sp 00007fff392e1e00 error 4 in libsss_util.so[7fb2b6cb9000+6c000]

I attached my modified sssd.conf file.




** Attachment added: "sssd.conf"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/4962331/+files/sssd.conf
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

Continuing on this bug is fine, thanks for getting back to us with a
simplified configuration file. I reopened the bug so that it will be
picked up again.

** Changed in: sssd (Ubuntu)
Status: Expired => New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

I turned on debugging and collected the sssd logs at the time of the
failure.

** Attachment added: "sssd_tpad.log"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/4995131/+files/sssd_tpad.log
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

** Attachment added: "sssd.log"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/4995132/+files/sssd.log
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

Thanks for these
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

Looking at this again.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

I think the best way forward here is to get a core dump, so we can have
a better idea of where the crash is happening.

I induced a crash in my test sssd container, and since I have apport installed, a crash file was produced in /var/crash:
# ll /var/crash/
total 644
drwxrwxrwt 2 root root 4 Nov 1 20:34 ./
drwxr-xr-x 13 root root 15 Sep 19 19:18 ../
-rwxr-xr-x 1 root root 0 Nov 1 20:34 .lock*
-rw-r----- 1 root root 593417 Nov 1 20:34 _usr_lib_x86_64-linux-gnu_sssd_sssd_be.0.crash

Could you please check if you have a recent crash file related to sssd
in that directory.


If not, do this:
sudo apt install apport

# check the kernel core_pattern:

# sysctl kernel.core_pattern
kernel.core_pattern = |/usr/share/apport/apport %p %s %c %P

And then restart sssd and induce the crash again, and then hopefully you
will have a crash file and we can go from there.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

Crash file attached.

** Attachment added: "_usr_lib_x86_64-linux-gnu_sssd_sssd_be.0.crash"
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+attachment/5002554/+files/_usr_lib_x86_64-linux-gnu_sssd_sssd_be.0.crash
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

Thanks, I can track this down more easily now.

(gdb) bt
#0 sysdb_attrs_get_el_ext (attrs=***@entry=0x0, name=***@entry=0x7f1e14bb504c "stamp", alloc=***@entry=true, el=***@entry=0x7ffc041e48a8) at ../src/db/sysdb.c:326
#1 0x00007f1e2d283d0d in sysdb_attrs_get_el (attrs=***@entry=0x0, name=***@entry=0x7f1e14bb504c "stamp", el=***@entry=0x7ffc041e48a8) at ../src/db/sysdb.c:360
#2 0x00007f1e14b6dda6 in sdap_attrs_get_sid_str (mem_ctx=***@entry=0x1664b40, idmap_ctx=0x1682ba0, sysdb_attrs=***@entry=0x0, sid_attr=0x7f1e14bb504c "stamp",
_sid_str=***@entry=0x7ffc041e4998) at ../src/providers/ldap/ldap_common.c:897
#3 0x00007f1e14b7a878 in sdap_save_user (memctx=***@entry=0x1bc3c20, opts=0x1679b20, dom=0x167aa80, attrs=0x0, _usn_value=***@entry=0x0, now=***@entry=0)
at ../src/providers/ldap/sdap_async_users.c:160
#4 0x00007f1e14b8be07 in sdap_get_initgr_user (subreq=0x0) at ../src/providers/ldap/sdap_async_initgroups.c:2896
#5 0x00007f1e14b75428 in generic_ext_search_handler (subreq=0x0, opts=<optimized out>) at ../src/providers/ldap/sdap_async.c:1668
#6 0x00007f1e14b77908 in sdap_get_generic_op_finished (op=<optimized out>, reply=<optimized out>, error=<optimized out>, pvt=<optimized out>) at ../src/providers/ldap/sdap_async.c:1561
#7 0x00007f1e14b7638d in sdap_process_message (ev=<optimized out>, sh=<optimized out>, msg=0x1664ae0) at ../src/providers/ldap/sdap_async.c:352
#8 sdap_process_result (ev=<optimized out>, pvt=<optimized out>) at ../src/providers/ldap/sdap_async.c:196
#9 0x00007f1e2df90613 in ?? () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#10 0x00007f1e2df8eb57 in ?? () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#11 0x00007f1e2df8ad3d in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#12 0x00007f1e2df8aedb in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#13 0x00007f1e2df8eaf7 in ?? () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#14 0x00007f1e2d2aff83 in server_loop (main_ctx=0x15ec060) at ../src/util/server.c:692
#15 0x0000000000406412 in main (argc=8, argv=<optimized out>) at ../src/providers/data_provider_be.c:2994
(gdb) frame 0
#0 sysdb_attrs_get_el_ext (attrs=***@entry=0x0, name=***@entry=0x7f1e14bb504c "stamp", alloc=***@entry=true, el=***@entry=0x7ffc041e48a8) at ../src/db/sysdb.c:326
326 for (i = 0; i < attrs->num; i++) {
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

I believe this is the upstream issue:
https://pagure.io/SSSD/sssd/issue/3045

The patch is
https://pagure.io/SSSD/sssd/c/5a0fb268e836e600d864ded7de5d935946ae6c61


** Changed in: sssd (Ubuntu)
Status: New => Triaged

** Changed in: sssd (Ubuntu)
Importance: Undecided => High

** Tags added: bitesize server-next
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

My coworker has recompiled the 1.13.4-1ubuntu1.8 sssd packages with that
patch and we are testing.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

I got a small reproducer case. With a simple "id <user>" command I get
sssd_be to segfault, and with the above patch applied it no longer
segfaults and also produces the correct result. I'll use that for the
SRU test plan.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

This PPA has my test packages:
https://launchpad.net/~ahasenack/+archive/ubuntu/sssd-bad-initgroups-
results-1684295/
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

** Changed in: sssd (Ubuntu)
Assignee: (unassigned) => Andreas Hasenack (ahasenack)

** Changed in: sssd (Ubuntu)
Status: Triaged => In Progress
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

It's only xenial that is affected, that is, version 1.13.4 and perhaps
earlier. Trusty, zesty and higher are OK.

** Description changed:

- This is Ubuntu 16.04.2 LTS
+ [Impact]

- sssd is configured to connect to two domains, our TPAD directory and
- Active Directory. sssd starts up at boot time. As soon as I ssh login
- (with any id, AD, TPAD or local), sssd fails with the error message in
- the title. After that, we can only login with local ids, not TPAD or AD
- ids.
+ * An explanation of the effects of the bug on users and

- ****************
- Here is the output from systemctl status sssd after the failure:
- ***@dcmilphlum128:~# systemctl status sssd
- â sssd.service - System Security Services Daemon
- Loaded: loaded (/lib/systemd/system/sssd.service; enabled; vendor preset: enabled)
- Active: failed (Result: exit-code) since Wed 2017-04-19 16:40:08 EDT; 7min ago
- Process: 119143 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS)
- Main PID: 119145 (code=exited, status=1/FAILURE)
+ * justification for backporting the fix to the stable release.

- Apr 19 16:39:47 dcmilphlum128.edc.nam.gm.com sssd[be[119187]: Starting up
- Apr 19 16:39:51 dcmilphlum128.edc.nam.gm.com sssd[be[119191]: Starting up
- Apr 19 16:39:57 dcmilphlum128.edc.nam.gm.com sssd[be[119206]: Starting up
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119145]: Exiting the SSSD. Could not restart critical service [tpad].
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119149]: Shutting down
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[119148]: Shutting down
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com sssd[be[119146]: Shutting down
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Main process exited, code=exited, status=1/FAILURE
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Unit entered failed state.
- Apr 19 16:40:08 dcmilphlum128.edc.nam.gm.com systemd[1]: sssd.service: Failed with result 'exit-code'.
+ * In addition, it is helpful, but not required, to include an
+ explanation of how the upload fixes this bug.

- ******************
- Also, in kern.log I have four of these (I have retries set to 3):
- Apr 19 16:39:59 dcmilphlum128 kernel: [ 6205.937807] sssd_be[12218]: segfault at 0 ip 00007fecb32b6b94 sp 00007ffce49a2230 error 4 in libsss_util.so[7fecb32a2000+6c000]
- Apr 19 16:40:02 dcmilphlum128 kernel: [ 6206.980725] sssd_be[12253]: segfault at 0 ip 00007f302de29b94 sp 00007fffca943cc0 error 4 in libsss_util.so[7f302de15000+6c000]
- Apr 19 16:40:05 dcmilphlum128 kernel: [ 6211.036205] sssd_be[12256]: segfault at 0 ip 00007fd196169b94 sp 00007ffd624249f0 error 4 in libsss_util.so[7fd196155000+6c000]
- Apr 19 16:40:07 dcmilphlum128 kernel: [ 6225.081902] sssd_be[12257]: segfault at 0 ip 00007fd1f669bb94 sp 00007ffdd8e5bf80 error 4 in libsss_util.so[7fd1f6687000+6c000]
+ [Test Case]

- *******************
- My sssd package are at 1.13.4:
- sssd 1.13.4-1ubuntu1.1 amd64
- sssd-ad 1.13.4-1ubuntu1.1 amd64
- sssd-ad-common 1.13.4-1ubuntu1.1 amd64
- sssd-common 1.13.4-1ubuntu1.1 amd64
- sssd-ipa 1.13.4-1ubuntu1.1 amd64
- sssd-krb5 1.13.4-1ubuntu1.1 amd64
- sssd-krb5-common 1.13.4-1ubuntu1.1 amd64
- sssd-ldap 1.13.4-1ubuntu1.1 amd64
- sssd-proxy 1.13.4-1ubuntu1.1 amd64
- sssd-tools 1.13.4-1ubuntu1.1 amd64
+ * detailed instructions how to reproduce the bug

- ***************
- I upgraded all the sssd packages to 1.13.4-1ubuntu1.4 and had the same problem.
+ * these should allow someone who is not familiar with the affected
+ package to reproduce the bug and verify that the updated package fixes
+ the problem.

- I downgraded them to 1.12.5-2 and was NOT able to reproduce the problem.
+ [Regression Potential]

- I attached my sssd.conf file.
+ * discussion of how regressions are most likely to manifest as a result
+ of this change.
+
+ * It is assumed that any SRU candidate patch is well-tested before
+ upload and has a low overall risk of regression, but it's important
+ to make the effort to think about what ''could'' happen in the
+ event of a regression.
+
+ * This both shows the SRU team that the risks have been considered,
+ and provides guidance to testers in regression-testing the SRU.
+
+ [Other Info]
+
+ * Anything else you think is useful to include
+ * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
+ * and address these questions in advance
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

** Description changed:

[Impact]
+ In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented.

- * An explanation of the effects of the bug on users and
+ The original scenario is a bit more complex and involves setting up an
+ Active Directory server, but with the help from the bug reporter (thanks
+ @pam-s!) we managed to narrow it down to this simple test case.

- * justification for backporting the fix to the stable release.
-
- * In addition, it is helpful, but not required, to include an
- explanation of how the upload fixes this bug.

[Test Case]

- * detailed instructions how to reproduce the bug
+  * detailed instructions how to reproduce the bug

- * these should allow someone who is not familiar with the affected
- package to reproduce the bug and verify that the updated package fixes
- the problem.
+  * these should allow someone who is not familiar with the affected
+    package to reproduce the bug and verify that the updated package fixes
+    the problem.

[Regression Potential]

- * discussion of how regressions are most likely to manifest as a result
+  * discussion of how regressions are most likely to manifest as a result
of this change.

- * It is assumed that any SRU candidate patch is well-tested before
- upload and has a low overall risk of regression, but it's important
- to make the effort to think about what ''could'' happen in the
- event of a regression.
+  * It is assumed that any SRU candidate patch is well-tested before
+    upload and has a low overall risk of regression, but it's important
+    to make the effort to think about what ''could'' happen in the
+    event of a regression.

- * This both shows the SRU team that the risks have been considered,
- and provides guidance to testers in regression-testing the SRU.
+  * This both shows the SRU team that the risks have been considered,
+    and provides guidance to testers in regression-testing the SRU.

[Other Info]
-
- * Anything else you think is useful to include
- * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
- * and address these questions in advance
+
+  * Anything else you think is useful to include
+  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
+  * and address these questions in advance

** Description changed:

[Impact]
In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented.

The original scenario is a bit more complex and involves setting up an
Active Directory server, but with the help from the bug reporter (thanks
@pam-s!) we managed to narrow it down to this simple test case.

-
[Test Case]

-  * detailed instructions how to reproduce the bug
+ # Install the packages. When prompted, choose any password for the ldap admin
+ $ sudo apt update; sudo apt install sssd slapd

-  * these should allow someone who is not familiar with the affected
-    package to reproduce the bug and verify that the updated package fixes
-    the problem.
+ # create the sssd config
+ $ sudo tee /etc/sssd/sssd.conf <<EOF
+ [sssd]
+ config_file_version = 2
+ services = nss, pam
+ domains = LDAP
+
+ [domain/LDAP]
+ id_provider = ldap
+ ldap_uri = ldap://localhost
+ ldap_search_base = dc=example,dc=com
+ ldap_rfc2307_fallback_to_local_users = True
+ EOF
+
+ $ sudo chmod 0600 /etc/sssd/sssd.conf
+ # reconfigure slapd for domain example.com, organization example. For the rest, accept defaults
+ $ sudo dpkg-reconfigure slapd
+
+ # add the base ldif. When prompted, use the password you chose when reconfiguring slapd earlier
+ $ ldapadd -x -D cn=admin,dc=example,dc=com -W <<EOF
+ dn: ou=People,dc=example,dc=com
+ ou: People
+ objectClass: organizationalUnit
+
+ dn: ou=Group,dc=example,dc=com
+ ou: Group
+ objectClass: organizationalUnit
+
+ dn: cn=ldapusers,ou=Group,dc=example,dc=com
+ cn: ldapusers
+ objectClass: posixGroup
+ gidNumber: 10000
+ memberUid: localuser
+ EOF
+
+ adding new entry "ou=People,dc=example,dc=com"
+
+ adding new entry "ou=Group,dc=example,dc=com"
+
+ adding new entry "cn=ldapusers,ou=Group,dc=example,dc=com"
+
+ # create a localuser with that name
+ $ sudo useradd -M localuser
+
+ # restart sssd
+ $ sudo service sssd restart
+
+ # take note of the sssd_be process id:
+ $ pidof sssd_be
+ 15474
+
+ # in one terminal, keep tailing /var/log/syslog
+ $ sudo tail -f /var/log/syslog
+
+ # in another terminal, run this id command. It will possibly hang for a bit, and won't show the "ldapusers" group membership
+ $ id localuser
+ (hangs a bit)
+ uid=1001(localuser) gid=1001(localuser) groups=1001(localuser)
+
+
+ # /var/log/syslog will emit messages like these, about a crash and sssd_be restarting (if you don't have apport installed, you will just see the "starting up" bit about sssd_be):
+ Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Starting Apport crash forwarding receiver...
+ Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 sssd[be[LDAP]]: Starting up
+ Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Started Apport crash forwarding receiver.
+
+ # verify that the sssd_be process id changed, confirming that it crashed and was restarted:
+ $ pidof sssd_be
+ 15485
+
+ # install the fixed packages from proposed
+ $ apt install/dist-upgrade ....
+
+ # repeat the id command. Now it finishes quickly, shows the "ldapusers" group membership, and there won't be any sign of an sssd_be restart in /var/log/syslog:
+ $ id localuser
+ uid=1001(localuser) gid=1001(localuser) groups=1001(localuser),10000(ldapusers)
+

[Regression Potential]

 * discussion of how regressions are most likely to manifest as a result
of this change.

 * It is assumed that any SRU candidate patch is well-tested before
   upload and has a low overall risk of regression, but it's important
   to make the effort to think about what ''could'' happen in the
   event of a regression.

 * This both shows the SRU team that the risks have been considered,
   and provides guidance to testers in regression-testing the SRU.

[Other Info]

 * Anything else you think is useful to include
 * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
 * and address these questions in advance
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

I only have xenial and trusty and I can confirm that trusty is not affected.
For Xenial, both the amd64 and s390x architectures are affected.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

@pam-s, as soon as you can confirm this patch fixes your problem (feel
free to use my PPA packages), please let us know so we can proceed with
the SRU.

My test case reproduces the segfault, but I would like to be sure it
also fixes it in your environment before continuing.

Thanks again


** Description changed:

[Impact]
In this particular configuration, when ldap_rfc2307_fallback_to_local_users is set to true in /etc/sss/sssd.conf and a local user is a member of an ldap group and does not exist in the directory (other scenarios are possible), the sssd_be process segfaults and logins might be prevented.

The original scenario is a bit more complex and involves setting up an
Active Directory server, but with the help from the bug reporter (thanks
@pam-s!) we managed to narrow it down to this simple test case.

[Test Case]

# Install the packages. When prompted, choose any password for the ldap admin
$ sudo apt update; sudo apt install sssd slapd

# create the sssd config
$ sudo tee /etc/sssd/sssd.conf <<EOF
[sssd]
config_file_version = 2
services = nss, pam
domains = LDAP

[domain/LDAP]
id_provider = ldap
ldap_uri = ldap://localhost
ldap_search_base = dc=example,dc=com
ldap_rfc2307_fallback_to_local_users = True
EOF

$ sudo chmod 0600 /etc/sssd/sssd.conf
# reconfigure slapd for domain example.com, organization example. For the rest, accept defaults
$ sudo dpkg-reconfigure slapd

# add the base ldif. When prompted, use the password you chose when reconfiguring slapd earlier
$ ldapadd -x -D cn=admin,dc=example,dc=com -W <<EOF
dn: ou=People,dc=example,dc=com
ou: People
objectClass: organizationalUnit

dn: ou=Group,dc=example,dc=com
ou: Group
objectClass: organizationalUnit

dn: cn=ldapusers,ou=Group,dc=example,dc=com
cn: ldapusers
objectClass: posixGroup
gidNumber: 10000
memberUid: localuser
EOF

adding new entry "ou=People,dc=example,dc=com"

adding new entry "ou=Group,dc=example,dc=com"

adding new entry "cn=ldapusers,ou=Group,dc=example,dc=com"

# create a localuser with that name
$ sudo useradd -M localuser

# restart sssd
$ sudo service sssd restart

# take note of the sssd_be process id:
$ pidof sssd_be
15474

# in one terminal, keep tailing /var/log/syslog
$ sudo tail -f /var/log/syslog

# in another terminal, run this id command. It will possibly hang for a bit, and won't show the "ldapusers" group membership
$ id localuser
(hangs a bit)
uid=1001(localuser) gid=1001(localuser) groups=1001(localuser)

-
# /var/log/syslog will emit messages like these, about a crash and sssd_be restarting (if you don't have apport installed, you will just see the "starting up" bit about sssd_be):
Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Starting Apport crash forwarding receiver...
Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 sssd[be[LDAP]]: Starting up
Nov 6 17:17:08 xenial-sssd-bad-initgroups-result-1684295 systemd[1]: Started Apport crash forwarding receiver.

# verify that the sssd_be process id changed, confirming that it crashed and was restarted:
$ pidof sssd_be
15485

# install the fixed packages from proposed
$ apt install/dist-upgrade ....

# repeat the id command. Now it finishes quickly, shows the "ldapusers" group membership, and there won't be any sign of an sssd_be restart in /var/log/syslog:
$ id localuser
uid=1001(localuser) gid=1001(localuser) groups=1001(localuser),10000(ldapusers)

+ [Regression Potential]
+ The patch is very specific, but given in how many different ways sssd can be configured, it would really help if users actually tested the package from proposed in their deployments. Specially considering it's a login service.

- [Regression Potential]
-
-  * discussion of how regressions are most likely to manifest as a result
- of this change.
-
-  * It is assumed that any SRU candidate patch is well-tested before
-    upload and has a low overall risk of regression, but it's important
-    to make the effort to think about what ''could'' happen in the
-    event of a regression.
-
-  * This both shows the SRU team that the risks have been considered,
-    and provides guidance to testers in regression-testing the SRU.
+ That being said, the patch is applied in the 1.13, 1,14 and current 1.15
+ series upstream and is more than a year old by now. It could rely on
+ other changes that I missed, though, but at least one I chose to ignore
+ (see [other info]).

[Other Info]
-
-  * Anything else you think is useful to include
-  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
-  * and address these questions in advance
+ The exact upstream patch wasn't applied (https://pagure.io/SSSD/sssd/c/5a0fb268e836e600d864ded7de5d935946ae6c61), because it relied on dropping an unused parameter from sdap_fallback_local_user(), namely the *opts struct pointer (https://pagure.io/SSSD/sssd/c/77f960ab32c2d2245fed55671f24af287ea0ba50). It is indeed not used, but I rather not drop it for an SRU because I don't know if some library could be using it, and also because a new upstream version for this series (1.13.5) wasn't released yet with this change.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

I tested with your PPA packages and it does fix my problem. Please
proceed and thank you!
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Launchpad Bug Tracker]

** Merge proposal linked:
https://code.launchpad.net/~ahasenack/ubuntu/+source/sssd/+git/sssd/+merge/333333
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[ChristianEhrhardt]

** Also affects: sssd (Ubuntu Xenial)
Importance: Undecided
Status: New

** Changed in: sssd (Ubuntu Xenial)
Status: New => In Progress

** Changed in: sssd (Ubuntu Xenial)
Assignee: (unassigned) => Andreas Hasenack (ahasenack)

** Changed in: sssd (Ubuntu)
Status: In Progress => Fix Released

** Changed in: sssd (Ubuntu Xenial)
Importance: Undecided => High
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Łukasz Zemczak]

Hello Pamela, or anyone else affected,

Accepted sssd into xenial-proposed. The package will build now and be
available at https://launchpad.net/ubuntu/+source/sssd/1.13.4-1ubuntu1.9
in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-xenial to verification-done-xenial. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-xenial. In either case, details of your
testing will help us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance!

** Changed in: sssd (Ubuntu Xenial)
Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-xenial
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Andreas Hasenack]

xenial verification:

Reproducing the crash:
***@xenial-sssd-1684295:~$ pidof sssd_be
3516

***@xenial-sssd-1684295:~$ id localuser
(stuck for 30s or more)
uid=1001(localuser) gid=1001(localuser) groups=1001(localuser)


syslog shows the crash:
Nov 13 13:05:55 xenial-sssd-1684295 systemd[1]: Starting Apport crash forwarding receiver...
Nov 13 13:06:40 xenial-sssd-1684295 sssd: Killing service [LDAP], not responding to pings!
Nov 13 13:06:54 xenial-sssd-1684295 sssd[be[LDAP]]: Starting up


pid changed:
***@xenial-sssd-1684295:~$ pidof sssd_be
4639


With the new package it works:
Version table:
*** 1.13.4-1ubuntu1.9 500
500 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages

***@xenial-sssd-1684295:~$ id localuser
uid=1001(localuser) gid=1001(localuser) groups=1001(localuser),10000(ldapusers)


xenial verification succeeded.

** Tags removed: verification-needed-xenial
** Tags added: verification-done-xenial
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Pamela Skutnik]

Yes, the sssd-1.13.4-1ubuntu1.9 fixed the problem. I did not see any
other issues as a result.

thanks,
Pam
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Launchpad Bug Tracker]

This bug was fixed in the package sssd - 1.13.4-1ubuntu1.9

---------------
sssd (1.13.4-1ubuntu1.9) xenial; urgency=medium

* debian/patches/bad-initgroups-results-3045.patch: sdap: Fix
ldap_rfc_2307_fallback_to_local_users. Thanks to Michal Židek
<***@redhat.com>. Closes LP: #1684295.

-- Andreas Hasenack <***@canonical.com> Mon, 06 Nov 2017 12:15:20
-0200

** Changed in: sssd (Ubuntu Xenial)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions

[Brian Murray]

The verification of the Stable Release Update for sssd has completed
successfully and the package has now been released to -updates.
Subsequently, the Ubuntu Stable Release Updates Team is being
unsubscribed and will not receive messages about this bug report. In
the event that you encounter a regression using the package from
-updates please report a new bug using ubuntu-bug and tag the bug report
regression-update so we can easily find any regressions.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1684295

Title:
sssd fails with 'Exiting the SSSD. Could not restart critical service
[tpad].

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1684295/+subscriptions