Eventually your (for example) great*N paternal grandfather is going to show up on your mother's side, forming a loop. That's just due to bounds on the human population and how the number of labels for previous generations grows exponentially backwards in time.