From ucbvax!MathStat:carl Sun Sep 20 16:31:30 1981 remote from purdue To: purdue!cincy!chris Subject: fp fixes paper UNIX* Problems With Floating-Point Processors Bob Campbell Ed Gould Vance Vaughan Computer Facilities and Operations University of California Berkeley, California Jim Reeds Department of Statistics University of California Berkeley, California A number of problems have been discovered relating to UNIX systems with floating-point pro- cessors. They fall in two areas. First, there were some errors in the way the UNIX kernel dealt with floating-point traps, and also with signals in general. Second, there were two cases in which the floating-point registers might not be saved when a process was being swapped out of memory. This is an internal document describing our analyses and fixes to the trap and signal prob- lems. At the end are some excerpts from the UNIX kernel, showing the code as it is now, including the fixes. Code fixes for the swap problems are also included; these are from J. W. Stevenson at Vrije Universiteit in Amsterdam, via Dennis Ritchie. January 20, 1981 UNIX* Problems With Floating-Point Processors Bob Campbell Ed Gould Vance Vaughan Computer Facilities and Operations University of California Berkeley, California Jim Reeds Department of Statistics University of California Berkeley, California A number of problems have recently been discovered in the UNIX kernel and some of the important UNIX utilities. The problems are quite serious and could lead to incorrect results in any calculation which generated floating-point exceptions, especially calculations which do their own floating-point exception handling. Conspicuous products which do their own floating-point exception handling include _b_a_s_i_c+ and _f_4_p. (Note that these are not standard UNIX utilities. Licenses from organizations other than Western Electric must be obtained before using them.) The discussion below is divided into two areas, the UNIX system kernel proper, and process codes (which includes not only codes prepared by the customer, but also such products as _b_a_s_i_c+ and _f_4_p). _U_N_I_X _K_e_r_n_e_l Four separate problems in the UNIX kernel proper have been identified. All relate only to machines which have floating-point hardware. Contents of Floating-Point Status Registers Not Reliably Available _P_r_o_b_l_e_m. There was no way for a process to get the correct floating-point status registers reli- ably after a floating-point exception. The system did not provide a way to get the contents. If a process read the registers itself, an interrupt from some other __________________________ *UNIX is a trademark of Bell Laboratories. - 2 - system activity could occur after the calculation and disturb the registers before they were read. _E_f_f_e_c_t. In a sense, this is not a bug, but a lack of facility. However, it would imply that no customer process could use the floating-point status in floating-point interrupt handling, and is certainly a serious short-coming. Several products (including _b_a_s_i_c+ and _f_4_p) went ahead and used the hardware regis- ters. Note that for this error to occur, two different processes must be generating floating-point exceptions and one which is doing its own handling must be inter- rupted by the other at an inopportune moment. A test with two different _f_4_p programs in tight loops generat- ing exceptions failed about once every 500 exceptions. _S_o_l_u_t_i_o_n. The solution is to have the system save the floating-point status registers whenever a process is interrupted and to provide a system call to retrieve the contents of the registers. Process codes which formerly read the registers must be changed to utilize the new system call. This problem existed on Version 6 as well as on Version 7. Scheduling Error _P_r_o_b_l_e_m. The floating-point processor and the cpu run asynchronously. Consequently, interrupts from the floating-point processor will normally occur after the cpu has already executed some additional instructions. In particular, the cpu may have taken an interrupt from some other activity and be executing system code when the floating-point interrupt arrives. The UNIX kernel noted the interrupt on behalf of the process, but resumed execution of the process without checking the arrival of the interrupt. The process would not see the interrupt at the appropriate time, but rather at some later time when a system call was executed or the process was rescheduled. _E_f_f_e_c_t. The most serious effect was on a process which was doing its own floating-point interrupt processing and trying to fix up a result depending on the floating-point status. Such a process would occasion- ally get the interrupt much later than it should have. It would typically do a calculation assuming that the registers contained what they did at the time the interrupt should have occurred. As they had typically been changed, the fix-up would be wrong. This was cer- tainly the most frequent cause of the _b_a_s_i_c+ problem with subtracting zero (see below). A _b_a_s_i_c+ loop which - 3 - just did _a=_b-0 and then checked _a=_b failed once every few thousand iterations. For processes not doing their own floating-point excep- tion handling, exceptions cause fatal errors, so a delay in noticing an exception would merely cause it to die later than it should have. The worst case we could concoct is a code like this: divide by zero write something on a file This code shouldn't write on the file, but if the bug hits it can terminate after the write instead of before. It still bombs, but this might not be noticed or the data in the file might be misleading. _S_o_l_u_t_i_o_n. The solution is to have the system interrupt processing ``reschedule'' the interrupted process before returning to it so that any pending interrupt would be handled at the appropriate time. No change to process code is necessary. This problem existed on Version 6 as well as on Version 7. Incorrect Handling of Simultaneous Interrupts _P_r_o_b_l_e_m. If more than one interrupt had arrived for a process, only one was posted for the process when it was scheduled; others would wait for a subsequent scheduling. This again resulted in floating-point interrupts not being processed at the appropriate time. _E_f_f_e_c_t. For a process handling its own interrupts, the effect is the same as for the previous bug. This prob- lem could have been the cause of some _b_a_s_i_c+ failures handling floating-point exceptions when the customer was introducing interrupts from the console. A _b_a_s_i_c+ program constructed to test this case failed once every few console interrupts but a problem in _b_a_s_i_c+ itself contributes to this failure rate. A process not handling its own interrupts would not be affected as it would be bombed by whichever interrupt happened to arrive or get handled first. _S_o_l_u_t_i_o_n. The solution is to have the system stack all waiting interrupts before resuming execution of a pro- cess. No change to process code is necessary. (How- ever, at least one process code (_b_a_s_i_c+) has a similar problem not caused or affected by the system problem.) - 4 - The problem existed on Version 6 as well as on Version 7. Interrupt Handling and Process Termination _P_r_o_b_l_e_m. Say a code executes the following sequence: divide by zero (DON'T store the result!) terminate the process The programmer may expect his program to terminate abnormally, but it won't: the interrupt from the divide by zero will not arrive until the system is already destroying the process and termination will appear nor- mal. _E_f_f_e_c_t. No language processors that we know of would produce such code except for machine-code assemblers. _S_o_l_u_t_i_o_n. We feel this is pathological code and is really a customer bug. If the customer were to store the result, abnormal termination would result. We do not intend to change the system to fix this ``prob- lem.'' _P_r_o_c_e_s_s _C_o_d_e_s As mentioned above, there is no way for a process to reli- ably read the contents of the floating-point status regis- ters in the multi-user UNIX environment. Unfortunately, at least two standard utilities, _b_a_s_i_c+ and _f_4_p, did read the registers and then make decisions based on them. _I_s_p uses _f_4_p and hence is implicated; we are still checking to ascer- tain whether other products have the same problem. Pascal does not, nor does f77. Basic+ _B_a_s_i_c+ needs to get the floating-point status from the system instead of reading the registers directly. _B_a_s_i_c+ also has some internal inconsistency when it gets a simultaneous floating-point exception and a con- sole interrupt: the floating-point exception does not get processed properly. This is still being investi- gated. It is difficult to characterize some set of _b_a_s_i_c+ floating-point calculations which could not have failed without notification. In addition to the ``normal'' floating-point exceptions (overflow, underflow, divide by zero, use of undefined operand, integer conversion error), _b_a_s_i_c+ generates an exception on the operation of _s_u_b_t_r_a_c_t_i_n_g floating-point zero. This exception is handled without customer notification of any sort and could have been done incorrectly. Also, a divide by - 5 - zero or an overflow could have been incorrectly handled as though it were a subtract of zero, giving incorrect results with no notification. F4p _F_4_p also reads the floating-point registers directly. We are looking to see if this can fixed (we do not have complete source for _f_4_p). As with _b_a_s_i_c+, any calculation which generated floating-point exceptions could produce incorrect results without notification. _F_4_p is not _k_n_o_w_n to gen- erate exceptions on ``normal'' calculations such as the _b_a_s_i_c+ subtract of zero. Customer Codes Which Handle Floating-Point Interrupts Any customers who have written code which does floating-point interrupt handling explicitly should review it. If it uses the hardware floating-point status registers, it is incorrect and should be changed to use the new system call. - 6 - Following are excerpts from the Unix kernel showing the changes made to correct the floating-point errors. New or changed lines of code are shown in boldface type. If no lines are in boldface, then the whole section of code is changed. _u_s_e_r._h #_i_n_c_l_u_d_e <_s_y_s/_f_p_e_r_r._h> /* * The user structure. * One allocated per process. * Contains all per process data * that doesn't need to be referenced * while the process is swapped. * The user block is USIZE*64 bytes * long; resides at virtual kernel * loc 140000; contains the system * stack per user; is cross referenced * with the proc structure for the * same process. */ struct user { label_t u_rsav; /* save info when exchanging stacks */ _i_n_t _u__d_u_m_m_y; /* _f_o_r_m_e_r_l_y _u__f_p_e_r - _s_e_e _u__f_p_e_r_r _b_e_l_o_w */ int u_fpsaved; /* FP regs saved for this proc */ struct { int u_fpsr; /* FP status register */ double u_fpregs[6]; /* FP registers */ } u_fps; char u_segflg; /* IO flag: 0:user D; 1:system; 2:user I */ char u_error; /* return error code */ short u_uid; /* effective user id */ short u_gid; /* effective group id */ short u_ruid; /* real user id */ short u_rgid; /* real group id */ struct proc *u_procp; /* pointer to proc structure */ int *u_ap; /* pointer to arglist */ union { /* syscall return values */ struct { int r_val1; int r_val2; }; off_t r_off; time_t r_time; } u_r; caddr_t u_base; /* base address for IO */ unsigned int u_count; /* bytes remaining for IO */ off_t u_offset; /* offset in file for IO */ struct inode *u_cdir; /* pointer to inode of current directory */ struct inode *u_rdir; /* root directory of current process */ - 7 - char u_dbuf[DIRSIZ]; /* current pathname component */ caddr_t u_dirp; /* pathname pointer */ struct direct u_dent; /* current directory entry */ struct inode *u_pdir; /* inode of parent directory of dirp */ int u_uisa[16]; /* prototype of segmentation addresses */ int u_uisd[16]; /* prototype of segmentation descriptors */ struct file *u_ofile[NOFILE]; /* pointers to file structures of open files */ char u_pofile[NOFILE]; /* per-process flags of open files */ int u_arg[5]; /* arguments to current system call */ unsigned u_tsize; /* text size (clicks) */ unsigned u_dsize; /* data size (clicks) */ unsigned u_ssize; /* stack size (clicks) */ label_t u_qsav; /* label variable for quits and interrupts */ label_t u_ssav; /* label variable for swapping */ int u_signal[NSIG]; /* disposition of signals */ time_t u_utime; /* this process user time */ time_t u_stime; /* this process system time */ time_t u_cutime; /* sum of childs' utimes */ time_t u_cstime; /* sum of childs' stimes */ int *u_ar0; /* address of users saved R0 */ struct { /* profile arguments */ short *pr_base; /* buffer base */ unsigned pr_size; /* buffer size */ unsigned pr_off; /* pc offset */ unsigned pr_scale; /* pc scaling */ } u_prof; char u_intflg; /* catch intr from sys */ char u_sep; /* flag for I and D separation */ struct tty *u_ttyp; /* controlling tty pointer */ dev_t u_ttyd; /* controlling tty dev */ struct { /* header of executable file */ int ux_mag; /* magic number */ unsigned ux_tsize; /* text size */ unsigned ux_dsize; /* data size */ unsigned ux_bsize; /* bss size */ unsigned ux_ssize; /* symbol table size */ unsigned ux_entloc; /* entry location */ unsigned ux_unused; unsigned ux_relflg; } u_exdata; char u_comm[DIRSIZ]; time_t u_start; char u_acflag; short u_fpflag; /* unused now, will be later */ short u_cmask; /* mask for file creation */ #ifdef UCB_LOGIN int u_login; /* login flag: 0 or ttyslot */ char u_crn[4]; /* charge record number */ #endif #ifdef MENLO_OVLY struct u_ovd { /* automatic overlay data */ int uo_curov; /* current overlay */ caddr_t uo_ovbase; /* base of overlay area */ caddr_t uo_dbase; /* start of data */ - 8 - unsigned uo_ov_offst[8];/* overlay offsets in text */ int uo_nseg; /* number of overlay seg. regs. */ } u_ovdata; #endif _s_t_r_u_c_t _f_p_e_r_r _u__f_p_e_r_r; /* _f_l_o_a_t_i_n_g _p_o_i_n_t _e_r_r_o_r _s_a_v_e */ int u_stack[1]; /* kernel stack per user * extends from u + USIZE*64 * backward not to reach here */ }; - 9 - _N_e_w _f_i_l_e: _f_p_e_r_r._h /* * structure of the floating-point error register save/return */ struct fperr { int f_fec; /* floating error code */ caddr_t f_fea; /* floating error address */ }; _m_c_h._s / save floating poing error registers / argument is a pointer to a two-word / structure _stst: tst fpp beq 9f stst *2(sp) 9: rts pc _t_r_a_p._c /* * Since the floating exception is an * imprecise trap, a user generated * trap may actually come from kernel * mode. In this case, a signal is sent * to the current process to be picked * up later. */ case 8: /* floating exception */ stst(&u.u_fperr); /* save error code and address */ psignal(u.u_procp, SIGFPT); _r_u_n_r_u_n++; return; case 8+USER: i = SIGFPT; stst(&u.u_fperr); break; - 10 - _n_e_w _s_y_s_t_e_m _c_a_l_l _f_p_e_r_r() - _s_y_s_l_o_c_a_l._c /* * fperr - return floating point error registers */ fperr() { u.u_r.r_val1 = u.u_fperr.f_fec; u.u_r.r_val2 = u.u_fperr.f_fea; } _f_r_o_m _n_e_w_p_r_o_c()/_s_l_p._c /* * When the resume is executed for the new process, * here's where it will resume. */ if (save(u.u_ssav)) { sureg(); return(1); } _i_f(_u._u__f_p_s_a_v_e_d == _0) { _s_a_v_f_p(&_u._u__f_p_s); _u._u__f_p_s_a_v_e_d = _1; } #ifndef UCB_FRCSWAP a2 = malloc(coremap, n); #else if(idleflg) a2 = malloc(coremap, n); else a2 = NULL; #endif - 11 - _f_r_o_m _e_x_p_a_n_d()/_s_l_p._c: /* * If there is not enough core for the * new process, swap out the current process to generate the * copy. */ if (save(u.u_ssav)) { sureg(); return; } #ifdef UCB_FRCSWAP if(idleflg) a2 = malloc(coremap, newsize); else a2 = NULL; #else a2 = malloc(coremap, newsize); #endif _i_f(_u._u__f_p_s_a_v_e_d == _0) { _s_a_v_f_p(&_u._u__f_p_s); _u._u__f_p_s_a_v_e_d = _1; } if(a2 == NULL) { - 12 - _f_r_o_m _p_s_i_g()/_s_i_g._c if (rp->p_flag&STRC) stop(); _w_h_i_l_e(_n = _f_s_i_g(_r_p)) { _r_p->_p__s_i_g &= ~(_1<<(_n-_1)); _i_f((_p=_u._u__s_i_g_n_a_l[_n]) != _0) { _u._u__e_r_r_o_r = _0; _i_f(_n != _S_I_G_I_N_S && _n != _S_I_G_T_R_C) _u._u__s_i_g_n_a_l[_n] = _0; _s_e_n_d_s_i_g((_c_a_d_d_r__t)_p, _n); } _e_l_s_e { _s_w_i_t_c_h(_n) { _c_a_s_e _S_I_G_Q_U_I_T: _c_a_s_e _S_I_G_I_N_S: _c_a_s_e _S_I_G_T_R_C: _c_a_s_e _S_I_G_I_O_T: _c_a_s_e _S_I_G_E_M_T: _c_a_s_e _S_I_G_F_P_T: _c_a_s_e _S_I_G_B_U_S: _c_a_s_e _S_I_G_S_E_G: _c_a_s_e _S_I_G_S_Y_S: _i_f(_c_o_r_e()) _n += _0_2_0_0; } _e_x_i_t(_n); } } - 13 - _f_s_i_g()/_s_i_g._c /* * This table defines the order in which signals * will be reflected to the user process. Signals * later in the list will be stacked after (processed * before) signals earlier in the list. The value * contained in the list is the signal number. */ char siglist[] = { 9, /* kill */ 15, /* terminate */ 4, /* illegal instruction */ 5, /* trace trap */ 6, /* iot */ 7, /* emt */ 10, /* bus error */ 11, /* segmentation error */ 12, /* invalid system call */ 3, /* quit */ 2, /* interrupt */ 1, /* hangup */ 13, /* broken pipe */ 14, /* alarm clock */ 8, /* floating exception */ 0, /* end of list */ }; /* * find the highest-priority signal in * bit-position representation in p_sig. */ fsig(p) struct proc *p; { register char *cp; register psig; psig = p->p_sig; for(cp=siglist; *cp; cp++) if(psig & (1 << (*cp-1))) return(*cp); return(0); }