Midpoint sets contained in the unit sphere of a normed space

The midpoint set M(S) of a set S of points is the set of all midpoints of pairs of points in S. We study the largest cardinality of a midpoint set M(S) in a finite-dimensional normed space, such that M(S) is contained in the unit sphere, and S is outside the closed unit ball. We show in three dimensions that this maximum (if it exists) is determined by the facial structure of the unit ball. In higher dimensions no such relationship exists. We also determine the maximum for euclidean and sup norm spaces.