Constructing models of inflation and/or baryogenesis in the context of $N=1$ supergravity is known to be difficult as the finite energy density during inflation typically generates large (order the Hubble scale) mass terms. This is the well-known eta problem in inflation. The same effect gives masses along low energy flat directions of the scalar potential thus potentially preventing Affleck-Dine baryogenesis to occur. It has been shown that adding a chiral multiplet $S$ coupled to the inflaton (with a shift symmetry) can serve to stabilize the inflationary potential and allows one to derive simple inflationary potentials without an eta problem. Here, we show that by coupling the same stabilizing field $S$ to the flat direction, may naturally lead to a negative mass-squared contribution to the flat direction thus generating the necessarily large vacuum expectation value needed to realize Affleck-Dine baryogenesis. We trace the evolution of the inflaton, stabilizer, and flat direction field, as well as a Polonyi-like modulus responsible for soft supersymmetry breaking.