The hot dense environment of the early universe is known to have produced large numbers of baryons, photons, and neutrinos. These extreme conditions may have also produced other long-lived species, including new light particles (such as axions or sterile neutrinos) or gravitational waves. The gravitational effects of any such light relics can be observed through their unique imprint in the cosmic microwave background (CMB), the large-scale structure, and the primordial light element abundances, and are important in determining the initial conditions of the universe. We argue that future cosmological observations, in particular improved maps of the CMB on small angular scales, can be orders of magnitude more sensitive for probing the thermal history of the early universe than current experiments. These observations offer a unique and broad discovery space for new physics in the dark sector and beyond, even when its effects would not be visible in terrestrial experiments or in astrophysical environments. A detection of an excess light relic abundance would be a clear indication of new physics and would provide the first direct information about the universe between the times of reheating and neutrino decoupling one second later.