bugMail Notification, A mail notification for freedesktop.org-compliant system trays - Bugs: bug #19643, openssl assertions about multiple...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #19643: openssl assertions about multiple threads in a context are triggered

Submitter:  None
Submitted:  Fri 20 Apr 2007 01:56:38 PM UTC
   
 
Severity:  5 - Blocker Status:  Fixed
Privacy:  Public Assigned to:  None
Originator Name:  Originator Email:  -email is unavailable-
Open/Closed:  Closed

Jump to the original submission

Tue 26 Jun 2007 12:40:30 PM UTC, comment #13: 

Fixed in 4.1 (locking added, and a multi-threaded OpenSSL is now required by the configure script).

Jean-Yves Lefort <jylefort>
Group administrator
Fri 20 Apr 2007 06:10:33 PM UTC, comment #12: 

I have added your patch for now.

I could depend on the pkgsrc openssl, but if openssl in base is broken, then I'd rather get that fixed.  Using pkgsrc versions of things in base is really intended for when the base versions are inadequate.

-Deleted Account- <lexort>
Fri 20 Apr 2007 06:07:22 PM UTC, comment #11: 

You're right, deadlock is not the appropriate word. I mean that it would cause mailboxes to block on each other for a very long time. For instance, when the IMAP IDLE extension is used, a perfectly functioning mailbox can wait in SSL_read() for as much as 29 minutes.

I see that pkgsrc has security/openssl. Maybe you could enable thread support there and make the MN package depend on it (and add my initial patch while waiting for the next MN version).

Jean-Yves Lefort <jylefort>
Group administrator
Fri 20 Apr 2007 05:50:47 PM UTC, comment #10: 

I don't see that locking SSL_{read,write} would cause 'deadlocks' (which is a cycle), but it would allow one broken network connection to block another, so I agree that it's undesirable, but perhaps better than not locking.

openssl is part of the base NetBSD system, so it's a bit awkward to build it with threads.  I've posted asking about thread support in openssl and will see what people say.

Your first patch seems fairly clearly needed for correctness.

Thanks for spending time on this problem.

-Deleted Account- <lexort>
Fri 20 Apr 2007 05:46:55 PM UTC, comment #9: 

Sorry, from reading the code in  mn_ssl_init, it looked like the first call stored the context in a static variable and returned it for subsequent calls.  But I see that then session->ssl is initialized from ctx, and that's the "context" passed into the calls.

With the global lock patch, mail-notification starts up correctly every time.  Perhaps the global locking should be conditional on lack of thread support.  Perhaps all ssl operations need the lock, not just initialization.



-Deleted Account- <lexort>
Fri 20 Apr 2007 05:45:35 PM UTC, comment #8: 

Actually, no need to test the patch. I'd probably also need to lock SSL_read() and SSL_write(), and that is not feasible since it would cause deadlocks between mailboxes.

Instead I'm going to require a thread-enabled OpenSSL from the configure script. Could you please rebuild OpenSSL with thread support and check if it fixes the problem (along with my initial mn-ssl-locking.diff patch)?

Jean-Yves Lefort <jylefort>
Group administrator
Fri 20 Apr 2007 05:23:47 PM UTC, comment #7: 

MN does not use the same SSL connection from multiple threads. Each mailbox check is performed in a separate thread, and the SSL* handle is only used from that thread.

The problem is that since thread support is disabled, the assertion is compiled in and fails if multiple threads are using ssleay_rand_add() concurrently.

Please test if the attached patch solves the problem.

(file #12537)

Jean-Yves Lefort <jylefort>
Group administrator
Fri 20 Apr 2007 04:32:44 PM UTC, comment #6: 

The openssl docs say that a connection may not be used simultaneously by multiple threads:

http://www.openssl.org/support/faq.html#PROG1

-Deleted Account- <lexort>
Fri 20 Apr 2007 04:29:26 PM UTC, comment #5: 

That's correct; there is no THREAD defined.  I will inquire about this on NetBSD lists.   Does openssl with thread support protect against multiple operations on the same context?

-Deleted Account- <lexort>
Fri 20 Apr 2007 04:25:41 PM UTC, comment #4: 

It appears that your OpenSSL was not compiled with thread support, otherwise you would not get that assertion failure:

#if !defined(OPENSSL_THREADS) && !defined(OPENSSL_SYS_WIN32)
        assert(md_c[1] == md_count[1]);
#endif

Please confirm that your opensslconf.h does not define THREADS.

Jean-Yves Lefort <jylefort>
Group administrator
Fri 20 Apr 2007 03:46:27 PM UTC, comment #3: 

Thanks for the quick reply.  The patch does not fix the problem, although it seems to make it less frequent.  I got a backtrace that shows the problem is not in openssl's calling of non-threadsafe functions without a lock, but mn calling openssl with the same context at the same time.  The openssl documentation was unclear about this, but it seems reasonable to avoid doing this.  So I think you need an additional lock for the context, or to create a new context for each use.

(gdb) thread 1
[Switching to thread 1 (process 136270)]#0  0xbac48767 in __sigaction_sigtramp
    () from /usr/lib/libc.so.12
(gdb) bt
#0  0xbac48767 in __sigaction_sigtramp () from /usr/lib/libc.so.12
#1  0xbac4860f in __sigaction14 () from /usr/lib/libc.so.12
#2  0xbacae721 in signal () from /usr/lib/libc.so.12
#3  0xbacae0d7 in abort () from /usr/

#0  0xbacaed7b in sbrk () from /usr/lib/libc.so.12
#1  0xbacacb81 in tcgetattr () from /usr/lib/libc.so.12
#2  0xbacace9c in tcgetattr () from /usr/lib/libc.so.12
#3  0xbacaddb3 in malloc () from /usr/lib/libc.so.12
#4  0xbadcac8c in lh_new () from /usr/lib/libcrypto.so.3
#5  0xbadcb1db in CRYPTO_malloc () from /usr/lib/libcrypto.so.3
#6  0xbadca5ba in BUF_MEM_grow () from /usr/lib/libcrypto.so.3
#7  0xbae1e1bc in ssl23_connect () from /usr/lib/libssl.so.4
#8  0xbae38b35 in SSL_connect () from /usr/lib/libssl.so.4
#9  0x08070f2f in mn_client_session_enable_ssl (session=0xba7ffe84)
    at mn-client-session.c:397
#10 0x080716ba in mn_client_session_run (states=0x807c6a0,
    callbacks=0x807c684, use_ssl=1, server=0x812d0e0 "[ip address redacted]", port=993,
    private=0xba7fff18, err=0xba7fff70) at mn-client-session.c:197
#11 0x0806e28f in ___1e_mn_imap_mailbox_authenticated_check (mailbox=0x812b0c8)
    at mn-imap-mailbox.gob:1064
#12 0x080732c5 in mn_authenticated_mailbox_check_thread_cb (self=0x812b0c8)
    at mn-authenticated-mailbox.c:324
#13 0xbae91282 in g_thread_create_full () from /usr/pkg/lib/libglib-2.0.so.0
#14 0xbaf7a40d in pthread_setcancelstate () from /usr/lib/libpthread.so.0
#15 0xbac337b8 in swapcontext () from /usr/lib/libc.so.12

(gdb) thread 2
[Switching to thread 2 (process 201806)]#0  0xbacaed7b in sbrk ()
   from /usr/lib/libc.so.12
(gdb) bt
#0  0xbacaed7b in sbrk () from /usr/lib/libc.so.12
#1  0xbacacb81 in tcgetattr () from /usr/lib/libc.so.12
#2  0xbacace9c in tcgetattr () from /usr/lib/libc.so.12
#3  0xbacaddb3 in malloc () from /usr/lib/libc.so.12
#4  0xbadcac8c in lh_new () from /usr/lib/libcrypto.so.3
#5  0xbadcb1db in CRYPTO_malloc () from /usr/lib/libcrypto.so.3
#6  0xbadca5ba in BUF_MEM_grow () from /usr/lib/libcrypto.so.3
#7  0xbae1e1bc in ssl23_connect () from /usr/lib/libssl.so.4
#8  0xbae38b35 in SSL_connect () from /usr/lib/libssl.so.4
#9  0x08070f2f in mn_client_session_enable_ssl (session=0xba7ffe84)
    at mn-client-session.c:397
#10 0x080716ba in mn_client_session_run (states=0x807c6a0,
    callbacks=0x807c684, use_ssl=1, server=0x812d0e0 "[ip address redacted]", port=993,
    private=0xba7fff18, err=0xba7fff70) at mn-client-session.c:197
#11 0x0806e28f in ___1e_mn_imap_mailbox_authenticated_check (mailbox=0x812b0c8)
    at mn-imap-mailbox.gob:1064
#12 0x080732c5 in mn_authenticated_mailbox_check_thread_cb (self=0x812b0c8)
    at mn-authenticated-mailbox.c:324
#13 0xbae91282 in g_thread_create_full () from /usr/pkg/lib/libglib-2.0.so.0
#14 0xbaf7a40d in pthread_setcancelstate () from /usr/lib/libpthread.so.0
#15 0xbac337b8 in swapcontext () from /usr/lib/libc.so.12
(gdb)


-Deleted Account- <lexort>
Fri 20 Apr 2007 02:58:27 PM UTC, comment #2: 

Please test if the attached patch fixes the problem.

(file #12536)

Jean-Yves Lefort <jylefort>
Group administrator
Fri 20 Apr 2007 01:58:02 PM UTC, comment #1: 

Sorry, forgot to sign this.   Bug is entered by Greg Troxel <gdt@ir.bbn.com>, and I'm the maintainer of the pkgsrc entry for mail-notification.

Anonymous
Fri 20 Apr 2007 01:56:38 PM UTC, original submission:  

On NetBSD 4.99.17 with a dual-processor machine, monitoring two IMAP/ssl mailboxes, mail-notification 4.0 release crashes on startup with:


assertion "md_c[1] == md_count[1]" failed: file
"/usr/home/gdt/NetBSD-current/src/crypto/dist/openssl/crypto/rand/md_rand.c",
line 312, function "ssleay_rand_add"

3.0 used to run, but started getting this error after NetBSD's thread implementation was changed to permit multiple threads to run at once by default.   I updated to 4.0 and got the same assertion.

I believe that the problem is that multiple operations are being invoked simultaneously on a single openssl context, and that locking is required.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #12537:  mn-ssl-global-lock.diff added by jylefort (746B - text/x-patch - locks SSL_connect())
file #12536:  mn-ssl-locking.diff added by jylefort (2KiB - text/x-patch - locking fix)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by lexort (Posted a comment)
  • -email is unavailable- added by jylefort (Updated the item)
  •  

    Follow 7 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-06-26 jylefort StatusConfirmed Fixed
        Open/ClosedOpen Closed
    2007-04-20 jylefort StatusNeed Info Confirmed
    2007-04-20 jylefort Attached File- Added mn-ssl-global-lock.diff, #12537
    2007-04-20 jylefort Attached File- Added mn-ssl-locking.diff, #12536
        StatusNone Need Info
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code