Boeheim was born in 1944 in Lyons, New York, a small town about 57 miles west of Syracuse. He graduated from Lyons Central High School.

Boeheim enrolled in Syracuse University as a student in 1962, and graduated with a bachelor’s degree in social science. During his freshman year, Boeheim was a walk-on with the freshman basketball team. By his senior year, he was the varsity team captain and a teammate of All-American Dave Bing, his freshman roommate. The pair led coach Fred Lewis’s Orange to a 22–6 overall win-loss record that earned the team’s second-ever NCAA Tournament tournament berth. While at Syracuse, he joined the Delta Upsilon fraternity.

In 1969, Boeheim decided to coach basketball and was hired as a graduate assistant at Syracuse under Roy Danforth. Soon thereafter he was promoted to a full-time assistant coach and was a member of the coaching staff that helped guide the Orange to the 1975 NCAA Tournament, where Syracuse University made its first Final Four appearance.

– From 1975 to 1982, the Eastern College Athletic Conference (ECAC) organized annual regional end-of-season men’s basketball tournaments for independent Division I ECAC member colleges and universities in the Northeastern United States. The winner of each regional tournament was declared the ECAC regional champion for the season and received an automatic bid in the NCAA Men’s Division I Basketball Tournament.

In 1976, Danforth left to become the head basketball coach and athletic director at Tulane University. A coaching search then led to nought, and Boeheim was promoted to head coach. Apart from his brief stint in the pros, Boeheim has spent his entire adult life at Syracuse, as either a student, player, assistant coach or head coach, a rarity in modern-day major collegiate athletics. In 2018, CBSSports.com writer Matt Norlander emphasized this in a piece where he speculated on potential successors for Boeheim, stating:

In 1986 Boeheim was offered the head coaching job at Ohio State but turned it down to stay at Syracuse.

In 42 years as head coach at Syracuse, Boeheim has guided the Orange to postseason berths, either in the NCAA or NIT tournaments, in every year in which the Orange have been eligible. The only times the Orange missed the postseason was in 1993 when NCAA sanctions barred them from postseason play despite a 20–9 record and in 2015 when Syracuse University self-imposed a one-year postseason ban related to the 2015 NCAA sanctions against the University’s sports programs. During his tenure, the Orange have never had a losing season, have appeared in three NCAA national championship games (1987, 1996, and 2003) and won the national title in 2003.

Boeheim has also been a coach for USA national teams. In 2001, during his seventh year as a USA basketball coach, Boeheim helped lead the Young Men’s Team to a gold medal at the World Championship in Japan. During the fall of that year he was named USA Basketball 2001 National Coach of the Year. He was an assistant coach under Mike Krzyzewski for the US national team in the 1990 FIBA World Championship and 2006 FIBA World Championship, winning the bronze medal both times. He returned as an assistant coach under Mike Krzyzewski for the 2008 Summer Olympics in Beijing, China, and again at the 2012 Summer Olympics in London, England, where the United States won the gold medal both times.

Boeheim had prostate cancer in 2001 and subsequently became a major fund-raiser for Coaches vs. Cancer, a non-profit collaboration between the NABC and the American Cancer Society, through which he has helped raise more than US$4.5 million for ACS’s Central New York chapter since 2000. In 2009, Boeheim and his wife, Juli, founded the Jim and Juli Boeheim Foundation to expand their charitable mission to organizations around Central New York concerned with child welfare, as well as cancer treatment and prevention.

Boeheim has been named Big East coach of the year four times, and has been named as District II Coach of the Year by the National Association of Basketball Coaches ten times. In 2004, Boeheim received two additional awards. The first was during the spring when he was awarded the Clair Bee Award in recognition of his contributions to the sport of basketball. During the fall of the same year Boeheim was presented with Syracuse University’s Arents Award, the University’s highest alumni honor.

In an exhibition game on November 7, 2005 against Division II school Saint Rose from Albany, New York, Boeheim was ejected for the first time in his career after arguing a call late in the first half in the Orange’s 86–73 victory. He was also ejected from Cameron Indoor Stadium on February 22, 2014 against Duke after arguing a player control foul call.

On March 6, 2015, the NCAA suspended Boeheim for the first nine games of 2015–16 ACC conference play and took away 12 scholarships over a four-year period, as a result of a multi-year investigation into the university’s athletic programs. The program was forced to vacate a total of 101 wins from the 2004–2005, 2005–2006, 2006–2007, 2010–2011, and 2011–2012 seasons, which included any game during those years where one or more players deemed to have been ineligible played. This constitutes the third-most wins ever permanently vacated by one program, behind the 113 wins vacated by Michigan and 123 wins by Louisville. However, the NCAA confirmed that sanctions did not include the removal of any trophies or banners. Therefore, Syracuse displays banners for all of its NCAA appearances and conference titles from those years.

Boeheim had previously stated that he would retire in April 2018. However, in 2017 when his son committed to play at Syracuse starting in 2018 he extended his contract to beyond the 2017–18 season.

Around 11:22 p.m. on Feb 20, 2019, Boeheim was driving his car on Interstate 690 in Syracuse when he struck and killed a 51-year-old man who was standing near the side of the road. The accident was reportedly a result of a disabled car being in the middle of the road in bad weather. When Boeheim maneuvered around the other car, he struck the man, who had been a passenger in the disabled car. Field, speed, and sobriety tests were administered by police. It was additionally found that the disabled vehicle did not have lights on and had non-operational rear marker lights, and that the passengers of the disabled car, including the man who was struck and killed, were wearing dark clothing. Boeheim was cleared of any wrongdoing in the incident.