To be called a castle (officially, and REALLY getting into pedantry ) you need to have an 'license to crenelate'. It was a private, fortified house and you could not fortify a house unless the king allowed you. While people would, they couldn't call their residence a castle without the license. Bodiam Castle in Sussex was one of the last houses to be given a license to crenelate in the mid 15th century. This was English law for many hundreds of years.
Of course, there was a later resurgence of "fake" castles and fortified houses which we see around the country. 'Modern' castles (including the Pugin designed Alton Castle) were often built on the sites of older castles. While they were called castles, they often barely fit into the common description of a castle as used when referencing "true" castles. The fortifications are decorative, and not practical. It was still acceptable to call a new building a castle though.
It's true that the original build on Alverton lodge was known as Alton Abbey (nobody knows the reason why, as it was clearly never designed as an abbey or ecclesiastic building of any kind) the name was rescinded as the build became more extensive.
The correct term is "The Towers". I would happily accept castle as a description of the collection of the buildings though in the looser modern term (though I would snigger silently behind my hand
).
I would also, maybe, at a push, accept Abbey if referring to the actual original Abbey part of the Towers complex. I wouldn't expect anyone but somebody who has studied the Towers history to know the location though and refer to it as such. Likewise, I wouldn't expect them to use it unless they were in the presence of somebody with equal knowledge. Essentially, people using the term to be smug gits
I would then also expect them to refer to the original lodge building as "the lodge" and to be able to refer to each room correctly throughout the entire building
I can't do that, so The Towers is fine, castle is fine-ish, pedantry over-rated, expectations of pedantry from somebody who has spent less than three days in the country out of order