Speed, 38, was appointed first-team coach by Lee in May when he took over from Sam Allardyce as boss.
But Bolton - who lie 19th in the Premier League - have had a poor start to the season and Lee said: "I want him to focus on his game at this moment.
"I don't want anything to distract him. I want everybody - coaches and players - to be fully focused on their game."
Speed himself went onto say, "It was my decision, I decided it wasn't working out. I want to concentrate totally on my football."