Nice test. I tested it with llama.cpp and the 20b model. He got the answer correct at first try on my side. The internal/thinking dialogue is fun to read - it could be my own inner dialogue ;)

``` > "Bobs dad has five daughters, Lala, Lele, Lili, Lolo and ??? <|channel|>analysis<|message|>The question: "Bobs dad has five daughters, Lala, Lele, Lili, Lolo and ???". It's a common riddle. The answer is Bob? Actually "Bob's dad has five daughters: Lala, Lele, Lili, Lolo and..." The riddle is to trick: it's "Bob" himself is a daughter? Wait Bob's dad has five daughters: Lala, Lele, Lili, Lolo, and Bob? But Bob is a name often male. But the riddle says Bob's dad. So Bob is a person who has a dad, and that dad has five daughters. But Bob could be the fifth daughter? That would be weird: Bob is likely the name of the fifth daughter. So answer: Bob. But sometimes the answer is "Lulu" or "Bob's sister"? Let's think. The riddle is a trick: It's "Bobs dad has five d aughters: Lala, Lele, Lili, Lolo, and ... Bob." Because the riddle begins with "Bobs dad has five daughters" implying Bob is one of the daughters. So the missing daughter is Bob. So answer: "Bob".

Thus answer: Bob. But sometimes the riddle goes "Bob's dad has five daughters, Lala, Lele, Lili, Lolo, and ... Who is the fifth?" The answer: Bob (the name of the person speaking, "Bobs"). But in riddle it's "Bobs dad has five daughters , Lala, Lele, Lili, Lolo and ???". The missing name is "Bob." So answer: Bob.

Thus the answer: Bob. Or maybe it's "Boba"? But no.

Hence answer: Bob.

We'll answer accordingly.<|start|>assistant<|channel|>final<|message|>The missing daughter is *Bob*. The riddle starts with “Bob’s dad …” so Bob himself is the fifth daughter. ```

Your riddle assumes the dad doesn't have any sons.

A Daughter Named Bob, what a great name for AI documentary.

We call her Bobby Tables. https://xkcd.com/327/

I don’t get it. Wouldn’t it be Lulu in both cases?

The original riddle has a girls name. It's a common riddle so the LLM would recognize the pattern of the riddle and assumes the name is the answer.

But if it's good at reasoning it should understand that if the name is a Boys name then it is not the 5th daughter.

Now the last daughter can be called anything, but because there is a vowel pattern it should pick Lulu to solve the puzzle, but that is the wrong answer in the original puzzle.

Presumably Jane is a girl and therefore the fifth daughter in the original riddle.

It’s Bob or Jane.

The dad of <Person> has 5 daughters. Four are listed off. So the answer for the fifth is <Person>.

Except having five daughters doesn't prevent them also having 20 sons one of whom is called Bob.

That’s why it’s a riddle.

But that makes no sense, if you ask me that John's dad has two daughters, Mary and ???, there's no way I will say "John" as an answer. I'll simply assume he also has at least one son.

Original riddle the answer is Jane.