The robins flew out of the traps with goals from Andy Gallinagh and Josh Low inside the first seven minutes.
But after Bury hit back to level things, it was down to Low to bag the winner.
"We had an open and frank discussion on Sunday morning - a few things were said, a few things were taken on board. We've been in a poor run of form and we said, hopefully that's the lowest of the low," Yates said.
"We came with a game plan, we defended superbly and I'm disappointed we conceded two goals, to be honest, against a really good team.
"I came to see them on Tuesday and they were like Brazil first half, so we knew it was a tough place to come. They have a front four that can rip teams apart, but that's the first time this season a Cheltenham side have literally defended in their own half and its paid off for us."
Source: PA
Source: PA