A p × p bit fraction model of binary floating point division and extremal rounding cases

  • Authors:

  • David W. Matula;Lee D. McFearin

  • Affiliations:

  • Department of Computer Science and Engineering, Southern Methodist University, Dallas, TX;Department of Computer Science and Engineering, Southern Methodist University, Dallas, TX

  • Venue:
  • Theoretical Computer Science - Real numbers and computers
  • Year:
  • 2003

Quantified Score

Hi-index 0.00

Visualization

Abstract

We introduce the ordered series Fp×p of irreducible p × p bit fractions as a model of p-bit precision binary floating point division. We employ and extend results from the number theoretic literature on Farey fractions and continued fractions to provide a foundation for generation and analysis of the series Fp×p. An algorithm for ordered on-the-fly enumeration of a consecutive subsequence of Fp×p over a selected interval is introduced which requires only a couple of integer additions and/or subtractions per p × p bit fraction enumerated.We characterize two extremal rounding boundary sets, RNp, respectively RDp, of irreducible p × p bit fractions over the standard binade [1,2) whose 2p+O(1) member fractions have rational values that are each comparably close to a boundary for rounding to a normalized p-bit floating point number by round-to-nearest, respectively, by a directed rounding. A transformation is shown allowing either set RNp, RDp, to be simply computed from the other. We determine properties of these extremal rounding boundary sets RNp, RDp, and describe their use in the testing of floating point division implementations.