Conservation experts have raised concerns about Richard Hammond's plans for his country home in Herefordshire.
Historic England says it has problems with some of the changes at Bollitree Castle in Weston Under Penyard on "heritage grounds".
The TV presenter wants to add five bedrooms to the listed building, which dates back to the 15th Century.
He also wants to build a glazed extension to link a dining room to a showroom for his motorbike collection.
Neighbours and conservation chiefs have made no objections to the plan.
The Hamster and his family moved into the estate in 2008.