Joe was one of the greatest Los Angeles City stars of all-time at Fremont High School even though he didn’t start playing on the varsity until his junior year. Known as “Jumpin Joe” and “Pogo Joe”, Caldwell was the Los Angeles City Basketball Player of the Year in 1960, averaging 24.8 points per game and leading Fremont to the City title. Also, a fine high jumper, he was the Los Angeles City champion in 1959 and 1960. He then became a Collegiate All-American at Arizona State after turning down UCLA. The 6-5 Caldwell finished ASU as the all-time leading scorer and second leading rebounder after sparking the Sun Devils to a record three straight NCAA appearances by averaging 19 points and 11 rebounds per game.

He was the 1964 NBA draft second overall selection by Detroit and is a historical rarity in being a double all-star in both the NBA where he played six seasons, and in the ABA where he played five seasons. He was also a key member of the 1964 USA Olympic Basketball team that won the Tokyo Gold Medal. He was a charter member of the ASU Athletics Hall of Fame, inducted into the Pac-10 Hall of Fame and a charter member of the LA City Athletics HOF in 2013. He is the grandfather of NBA player Marvin Bagley III of Detroit.