Deke Slayton gave Armstrong an option to replace Aldrin with what other Apollo astronaut (who shared Gemini 12 with Aldrin) because some thought Aldrin was difficult to work with. Armstrong had no issues working with Aldrin and declined. The astronaut in question became famous in his own right a few years later.
Jim Lovell (Armstrong wanted Lovell to have his own command and the rest is history.)