You're forgetting an important fact.

Having the blood of your brother will change you, no matter how hard you fight it. Once you turn twenty, there's nothing you can do about it.

Even aside from Yaiba and Kiyotaka, the same change has been observed in the Blade Children. As if suddenly becoming another person to fulfill a predetermined fate...

[It's a fear Hizumi had never thought he would have to face, personally. But Hinoto-Ri had changed that.]