In 1959, at The 2i’s Coffee Bar, he met Marty Wilde and was invited to become a member of his backing group, the Wildcats, who were the opening act in the television series, Oh Boy!, produced by Jack Good. The Wildcats backed Eddie Cochran and Gene Vincent on their tour of Britain in 1960, during which Cochran died. Wilde bought Sullivan a Gibson Les Paul guitar, reputedly the first to be played in Britain, which he had bought from Sister Rosetta Tharpe. He later played a cherry-red Gibson ES-345 guitar.

In the early 1960s he also played on hits by Billy Fury, Frank Ifield, Adam Faith, Frankie Vaughan, Helen Shapiro, Johnny Hallyday, Freddie and the Dreamers, Cilla Black, Tom Jones, Shirley Bassey, Dusty Springfield and others. He played guitar on the Alexis Korner’s and Blues Incorporated’s album R&B from the Marquee in 1962 and Georgie Fame’s first album Rhythm & Blues at the Flamingo in 1964. In addition to playing on many UK albums Sullivan played on the Everly Brothers’s 1963 live album at Olympia, Bobby Darin’s 1966 live album Something Special, Little Richard’s 1966 album Get Down With It: The OKeh Sessions and Del Shannon’s 1967 album, Home and Away. He appeared regularly on several British television and radio programmes, including the Tom Jones show, Bay City Rollers show “Shang-A-Lang”,Top of the Pops, Ready Steady Go! and Saturday Club.

Jack Good introduced Sullivan to studio work. Sullivan became one of the most sought-after guitarists throughout the 1960s and the 1970s, in part because of his flexibility in playing different styles of music. He was often referred to as “Big Jim” both for his physical appearance and as he was usually first choice to play guitar on sessions for major musicians and bands. Another session musician at the time, and at some of the same sessions, Jimmy Page, was referred to as “Little Jim.” Sullivan played on around 750 UK chart entries, and averaged three recording sessions a day. He played on the first records in the UK to use a wah-wah effect – Michael Cox’s 1961 “Sweet Little Sixteen” and Dave Berry’s 1964 hit “The Crying Game” used a DeArmond Tone and Volume pedal. He played on the first record in the UK to use a fuzzbox, which he had borrowed from session guitarist Eric Ford, on P.J. Proby’s 1964 hit “Hold Me”.

Later in the 1960s and 1970s, Sullivan continued to play on a succession of hit records including those by The Walker Brothers, Jonathan King, Donovan, David Bowie (he played banjo, guitar and sitar on Bowie’s first album which was published in 1967), Benny Hill, The New Seekers, Thunderclap Newman, Love Affair, Long John Baldry, Marmalade, Small Faces, and Rolf Harris. In 1968 he played on George Harrison’s Wonderwall. He directed and played on Amazing Blondel’s first album in 1969, and in the same year played on the album Sound of Sunforest, the overture from which was used in the film A Clockwork Orange. In 1971, he played in the Jean-Claude Vannier Orchestra for Serge Gainsbourg’s Histoire de Melody Nelson, and also played on Frank Zappa’s 200 Motels. In 1972, he did arrangements for the orchestral version of The Who’s Tommy.

In 1969, Sullivan joined Tom Jones’ band. During his time with Jones in Las Vegas, he met and formed a friendship with Elvis Presley. Sullivan was an innovator of the talk box, which he demonstrated on Jones’ TV show. He released an instrumental album Sullivan Plays O’Sullivan (1971) and was also featured giving guitar lessons on the Bay City Rollers’ TV series Shang A Lang. In the 1970s he composed the score for an episode of the science fiction series, Space: 1999 (“The Troubled Spirit”), in which he also appeared and performed part of the score on screen, as a crew member giving a Coral sitar concert.

In 1974, Sullivan teamed up with the record producer, Derek Lawrence, to form the record label, Retreat Records. One album release was Big Jim’s Back (1975). He fronted a band called Tiger, alongside vocalist Nicky Moore, releasing three albums under this name before the group split up in 1976. Retreat Records also produced various artists. Amongst them were Labi Siffre, Chas & Dave and McGuinness Flint. Sullivan produced and arranged Siffre’s “I Got The …”, sampled by Eminem. Lawrence and Sullivan went to the United States during this period, to produce the glam metal band, Angel.

In 1978, he became part of the James Last Orchestra for nine years, also touring with Olivia Newton-John after her success with Grease. In 1987, he began composing music for films and jingles.

Sullivan played in the Big Jim Sullivan Band with Duncan McKenzie, Malcolm Mortimore and Pete Shaw. He also worked with Doug Prudenin BJS duo. In 2006 he was featured in the Guitar Maestros DVD series with Doug Pruden.

Sullivan died on 2 October 2012, aged 71 due to complications from heart disease and diabetes.