The town takes its name from the castle built by Bernard de Balliot and extended by his son of the same name. Between them they erected a powerful stone castle in the second half of the twelfth century, strongly situated on a rock above the River Tees.
Today the castle is an extensive but very ruinous pile. It possesses an exceptional four baileys, all walled in stone during the period of the two Bernards. From the town of Norman arch - once part of a gatehouse - leads into the northern outer bailey, known as the Town Ward. Much of its curtain still stands as well as the vaulted undercroft of the Brackenbury Tower. The southern outer bailey doubles the size of the castle but its defenses are now fragmentary.
West of the Town Ward are the ditch and curtain of the inner bailey, with two flanking towers added by the Beauchamps. To reach the inner bailey it is necessary to pass through a middle ward, then turn sharp right over a deep ditch hewn out of solid rock. This succession of defenses is quite advanced for the twelfth century. Once inside the inner bailey the dominant is the Balliol Tower or keep which projects from the curtain.
This cylindrical tower of ashlar is actually an early addition to the castle, though it could still be the second Bernard's work as he survived until 1199.
As keeps go, it is a bit of a fraud, because it was not isolated from the rest of the castle. It was entered directly from the vanished solar at first floor level, and the triangular spur projecting from the keep is not a defensive feature but merely a wedge between the two. All the same, the keep is the only part of the castle to survive more or less complete and an unusual domed vault covers its ground floor.