He said: "Just because of what happened last Saturday doesn't mean that all of a sudden Ryan Williams has become a world-beater and would walk straight back into our team.
"If Bristol want him, then I won't stand in their way. They know exactly what the situation is."
Meanwhile, Taylor has also played down the rave reviews Williams got following the match.
"There was no way he scored their first goal, even though the reports said he did. For my money, Junior Agogo definitely got a touch.
"In the second half, I didn't think he did that much. I know it was his corner which set up the winning goal. To be honest, it was a poor strike. The goal came from our mistake."