Issue84

Title crash when database connection fails in presence of fallback mirrors
Priority bug Status resolved
Superseder Nosy List poeml
Assigned To poeml Keywords

Created on 2011-02-23.20:50:35 by poeml, last changed by poeml.

Messages
msg347 (view) Author: poeml Date: 2012-02-20.23:59:15
In 2.16.0.
msg324 (view) Author: poeml Date: 2011-02-23.21:36:10
I commmitted an untested fix to svn trunk:

http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=8238
http://svn.mirrorbrain.org/viewvc/mirrorbrain?view=revision&revision=8239

ToDo: test it; release it.
msg323 (view) Author: poeml Date: 2011-02-23.20:50:32
I saw a crash that seems to occur when the database connection fails (for any reason) and 
fallback mirrors are configured:



(gdb) thread apply all bt
....

(gdb) thread 24
[Switching to thread 24 (Thread 29704)]#0  0xffffe422 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe422 in __kernel_vsyscall ()
#1  0xb738ccb6 in kill () from /lib/libc.so.6
#2  0xb7795e06 in sig_coredump (sig=11) at mpm_common.c:1231
#3  <signal handler called>
#4  0xb70f594c in mb_handler (r=0xb798f540) at mod_mirrorbrain.c:1943
#5  0xb778c03d in ap_run_handler (r=0xb798f540) at config.c:158
#6  0xb778fbb4 in ap_invoke_handler (r=0xb798f540) at config.c:376
#7  0xb779c641 in ap_process_request (r=0xb798f540) at http_request.c:282
#8  0xb77990d8 in ap_process_http_connection (c=0xb798b6e0) at http_core.c:190
#9  0xb779449d in ap_run_process_connection (c=0xb798b6e0) at connection.c:43
#10 0xb77a2c2c in worker_thread (thd=0xb77fd538, dummy=0xb78235d0) at worker.c:544
#11 0xb7542b56 in ?? () from /usr/lib/libapr-1.so.0
#12 0xb74ca1b5 in start_thread () from /lib/libpthread.so.0
#13 0xb743362e in clone () from /lib/libc.so.6

(gdb) up
...


(gdb) l
1938            return HTTP_NOT_FOUND;
1939            }
1940        }
1941
1942
1943        statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);
1944
1945        if (statement == NULL) {
1946            ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
1947                          "[mod_mirrorbrain] Could not get prepared statement labelled 
'%s'",


(gdb) i lo
cfg = (mb_dir_conf *) 0xb7995950
ptr = <value optimized out>
uri = <value optimized out>
filename = 0xb79964d9 "stable/3.3.0/OOo_3.3.0_MacOS_x86_install_en-US.dmg"
basename = 0xb79964e6 "OOo_3.3.0_MacOS_x86_install_en-US.dmg"
mirror_base = 0xb7996480 "/srv/ooo"
realfile = <value optimized out>
yum = (yumdir_t *) 0x0
clientip = 0xb798bb60 "128.232.241.11"
clientaddr = <value optimized out>
query_country = 0x0
query_asn = 0x0
fakefile = 0 '\0'
only_hash = 0 '\0'
rep = 0
rep_ext = 0x0
meta_negotiated = 0 '\0'
continent_code = 0xb70b92c7 "EU"
country_code = 0xb70b8bc7 "GB"
country_name = 0xb70b8078 "United Kingdom"
slat = <value optimized out>
slng = <value optimized out>
lat = 30.4797993
lng = -87.865799
state_id = <value optimized out>
state_name = <value optimized out>
as = 0xb71003dd "--"
prefix = 0xb71003dd "--"
i = -1215364960
mirror_cnt = <value optimized out>
len = 20
nr = 3056701544
new = <value optimized out>
mirror = <value optimized out>
mirrorp = <value optimized out>
chosen = <value optimized out>
hashbag = (hashbag_t *) 0x0
rv = <value optimized out>
res = (apr_dbd_results_t *) 0x0
row = (apr_dbd_row_t *) 0x0
mirrors = <value optimized out>
mirrors_same_prefix = <value optimized out>
mirrors_same_as = <value optimized out>
mirrors_same_country = <value optimized out>
mirrors_fallback_country = <value optimized out>
mirrors_same_region = <value optimized out>
mirrors_elsewhere = <value optimized out>
form_lookup = (const char *(*)(request_rec *, const char *)) 0xb710ad30 
<__cxa_finalize@plt+416>
cmp_mirror_best = <value optimized out>
find_best = <value optimized out>
dbd = (ap_dbd_t *) 0x0
magnet = <value optimized out>
found_in = <value optimized out>
(gdb)


(gdb) l
1938            return HTTP_NOT_FOUND;
1939            }
1940        }
1941
1942
1943        statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);
1944
1945        if (statement == NULL) {
1946            ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, 
1947                          "[mod_mirrorbrain] Could not get prepared statement labelled 
'%s'",



(gdb) p dbd
$1 = (ap_dbd_t *) 0x0

              ^^^^^^^^^^



Earlier in the code, a failed connection is handled like this:


   ap_dbd_t *dbd = mb_dbd_acquire_fn(r);
    if (dbd == NULL) {
        ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
                "[mod_mirrorbrain] Error acquiring database connection");
        if (apr_is_empty_array(cfg->fallbacks)) {
            setenv_give(r, "file");
            return DECLINED; /* fail gracefully */
        }
    }
    debugLog(r, cfg, "Successfully acquired database connection.");


This means that in case of fallback configuration the mirrorbrain handler continues running, 
and will later try to use the database connection (*dbd), which is NULL and causes the crash 
in the line

statement = apr_hash_get(dbd->prepared, scfg->query_label, APR_HASH_KEY_STRING);


This suspicion is backed up by the fact that Apache's error log shows "[mod_mirrorbrain] 
Error acquiring database connection" at the same times where the crashes happened.
History
Date User Action Args
2012-02-20 23:59:15poemlsetstatus: testing -> resolved
messages: + msg347
2011-02-23 21:36:18poemlsetstatus: in-progress -> testing
2011-02-23 21:36:10poemlsetmessages: + msg324
2011-02-23 20:50:35poemlcreate