Niekro pitched for the Chicago Cubs, San Diego Padres and Detroit Tigers with occasional success before joining Phil with the Atlanta Braves in 1973. Joe used a fastball and a slider early in his career, with mixed results. He spent two seasons in Atlanta with Phil and got re-acquainted with the knuckleball that their father taught them. The knuckleball became an essential part of his arsenal though never his sole pitch. Joe threw harder than Phil and could set up batters nearly as effectively with his fastball in combination with his excellent changeup.

The Houston Astros purchased Niekro's contract from the Braves for $35,000 in 1975. He blossomed into a dominant pitcher as he perfected his knuckleball in Houston, going 21-11 in 1979 and 20-12 in 1980, to became the first Astros pitcher to win 20 games in consecutive seasons. He also made the National League All-Star team in 1979, a season in which he led the league with his 21 wins and five shutouts, won the TSN Pitcher of the Year Award, and ended second in voting for the Cy Young Award behind Bruce Sutter.

In 1980, Houston had a three-game lead over the Los Angeles Dodgers in the NL West, then lost their last three games of the regular season in Los Angeles, to force a one-game playoff. Niekro allowed six hits in a 7–1 Houston victory that propelled the Astros to their first postseason. He then pitched 10 shutout innings in Game 3 of the NLCS and the Astros won 1–0, though they lost the series to the Philadelphia Phillies, 3–2.

In 1985, Houston traded the 40-year-old Niekro to the New York Yankees, where he briefly reunited again with Phil. Niekro finished his career with the Minnesota Twins, where he pitched in the World Series for the only time, in 1987. Earlier that season, Niekro was suspended for ten games when umpire Steve Palermo discovered a nail file in his pocket. The video clip of Niekro reaching into his pockets, pulling out his hands, and throwing them in the air while the nail file fluttered to the ground made a lot of sports-highlight shows and is a common "blooper" clip today. Niekro said he was filing his nails in the dugout, but American League president Dr. Bobby Brown did not believe him, and ordered the suspension. Niekro played his final game on May 4, 1988, ending his 22-year majors career.

On October 26, 2006, Niekro suffered a brain aneurysm and was taken to South Florida Baptist Hospital in Plant City, Florida. He was later transferred to St. Joseph's Hospital in Tampa, Florida, where he died the following day at age 61.

The Joe Niekro Foundation, created by his daughter Natalie, donates money towards aneurysm treatment and research. The Joe Niekro Foundation hosts several events to boost donations including its annual gala in Houston, The Knuckle Ball.