Baker had an impressive amateur career. He only lost one amateur fight and won the 1949 Intercity Golden Gloves championship in the heavyweight division.He started his professional career with twenty-six consecutive wins. However, his winning streak came to an end when he was defeated by another leading contender, Clarence Henry, in 1954. In his previous fight, only days before, he drew with bulky Kid Riviera.