Olsson began his playing career in Sweden by playing for Högaborgs BK, where former national star Henrik Larsson began his career. Olsson was working his way through the youth setup at the club in 2005–06. His talent attracted the attention of football clubs in Europe and in January 2006 Premier League club Blackburn Rovers signed him for their youth academy.

He made his Premier League debut against Derby County in a 2–1 victory on 30 December 2007, coming on as an 84th-minute substitute for Morten Gamst Pedersen at Pride Park, in front of a crowd of 30,048 spectators and the referee was Peter Walton. He was voted Blackburn’s Young Player of the Year for the season 2007–08.

In May 2008, Olsson signed a new four-year contract, keeping him at Blackburn until June 2012.

On 24 September 2008, Olsson scored his first Blackburn goal, scoring the winning goal in a League Cup tie against Everton. The goal turned out to be the only goal of the match. On 20 January 2010, he scored a spectacular overhead kick in a 6–4 League Cup semi-final defeat against Aston Villa in the 63rd minute at Villa Park.

On 23 November 2009, Olsson was sent off for two yellow card offences on Tottenham Hotspur winger Aaron Lennon. He scored his first Premier League goal on 10 February against Hull City at Ewood Park in a 1–0 victory. Olsson made his 25th league appearance at home to Birmingham City on 24 March 2010 in a 2–1 win. On 28 March 2010, when he made his 40th appearance for Blackburn against Burnley in the East Lancashire derby at Turf Moor in the 1–0 victory, he caused controversy when he looked like he had dived inside Burnley’s penalty area, consequently by winning a penalty after taking the ball round Burnley goalkeeper Brian Jensen. Subsequently, the penalty was scored by David Dunn. On 29 March 2010, Olsson admitted to the dive, Rovers manager, Sam Allardyce defended Olsson by saying he was his man of the match and created havoc and chaos in and around the eighteen yard box all match.

In June 2010, Olsson signed a new five-year-deal at Blackburn Rovers that would keep him at the club until summer 2015. At the start of the 2010–11 season, Olsson acquired the number three jersey by manager Sam Allardyce, which he had for most of pre-season. He featured in Rovers’ opening 2010–11 Premier League match against Everton at Ewood Park on 14 August 2010, playing the full 90 minutes on the left-wing in a 1–0 victory. He scored his first league goal for Blackburn against Liverpool inside 32 minutes on 5 January 2011 in a 3–1 victory at Ewood Park. On 12 February 2011, Olsson made his 50th Premier League appearance for the club when he started at left-back against Newcastle United playing the full 90 minutes in a 0–0 draw. On 30 April 2011, he scored with a left-foot shot on 20 minutes against local rivals Bolton Wanderers at home. On 22 May, Olsson started and played the full 90 minutes against Wolverhampton Wanderers in a 3–2 win at Molineux which kept both teams up for another season in the Premier League. At the end of the season, he made 32 appearances for Rovers in total and scored two goals in all competitions.

He was selected to be part of the senior 23-man squad that would face Bosnia and Herzegovina on 29 May and Belarus on 2 June 2010. Against Bosnia and Herzegovina, Olsson put Sweden back in front from close range in the 68th minute, two minutes after coming on as a substitute, and added his second in the 82nd from Marcus Berg’s center. On 6 September 2011, he made his first start for the national team, playing at left back, and scored Sweden’s third goal in the 81st minute of the 5–0 victory over San Marino after a clever reverse pass by Zlatan Ibrahimović in a Group E Euro 2012 qualifying fixture. In May 2012, Olsson was named in the 23-man squad to compete for Sweden at UEFA Euro 2012 and was given the number 5 shirt.

On 10 July 2013, Olsson was signed by Premier League side Norwich City for an undisclosed fee. He signed a four-year deal at Carrow Road. He scored his first goal for the club against Bury in the League Cup on 27 August 2013. Olsson made 37 appearances in all competitions but could not prevent his new team from relegation in his return to Premier League football.

Olsson would however make fewer appearances in the 2015–16 Premier League season, only appearing in 24 of the 38 league matches, including four appearances as a substitute, with Robbie Brady often being preferred at left-back. He would however score a dramatic winner on 2 April 2016 against Newcastle United. With the match 2–2 going into stoppage time, Olsson scored with a long distance shot to give Norwich a 3–2 win. The win was ultimately in vain however, as Norwich were relegated to the Championship again at the end of the season. The relegation was confirmed after Sunderland won 3–0 against Everton where the Canaries were four points off safety.

On 17 January 2017, it was announced that Olsson had signed for Premier League club Swansea City for around £4 million. Olsson scored his first goal on 12 February 2017 in a match against reigning champions Leicester City in a 2–0 victory for the Welsh club.

In May 2018 he was named in Sweden’s 23 man squad for the 2018 World Cup in Russia.

The club confirmed on 18 May 2019 that he will be released upon expiry of his contract.