I don't know why anyone would expect to trust a model smaller than about the size of qwen 3.6 27B (or 3.8 27B, or 3.6 35B-A3B) for coding. There just isn't enough baked-in knowledge of existing correct code syntax from having vacuumed up various open source projects.

That further extends to concepts like knowing if an API exists as a real thing it has code examples of in its training data set vs. just hallucinating the name of something in an attempt to satisfy the person issuing it a prompt.

Other models of this size have done well in the past.

Qwen2.5 coder, for example, can correctly answer the question at 7B.

Deepseek R1 was also capable of giving a correct response.

It's obviously a doable. Such a model locally is useful in autocomplete while programming.

Not sure a model that small is really supposed to be used for any real coding. At that size you're usually using the model to do simple tasks like summarization.

To be clear, the question wasn't a complex one. It was more on the level of "could I use this for a fast inline coder" IE, single somewhat simple function question.

I wouldn't have dreamed to use this as an agent model.

7B models of the past have been able to pass this question. I've not tested it on a 4B model until now.

I'd you have some tips for coming up with such tests, I would love to hear them. My Gmail username is the same as my HN username. Thank you!

It's actually just a coding interview test that I liked to ask in the past. You can find it and others on leetcode.

The reason I personally like my question is because it's pretty close to some of the real world work we do. It's mostly mundane and easy to bang out, but really easy for someone to do a n log n solution where an n solution exists.

A good example (but not my question) would be something like

"I have a list of People objects with a `first` and `last` name. Write a function which groups together all the People with the same last name in `your language of choice`"

LLMs have a problem with that type of question? I might try it later at home.

Now a days? No. It's actually getting to be a bad question because they all push out about the exact same answer.

But much earlier they did and, apparently, these really small models still do. At this point it serves as more of a smoke test for me. Success means little, failure means a lot.

Terrific, thank you.

why is your actual benchmark question so secret?

Probably so it does not become a target to meet.

Bingo. I know that people that work on LLMs read sites like HN. Already my question is losing it's usefulness as most models pass it now a days, but not every model does. The "car wash" question is a good example of this happening. Pretty much every model now correctly answers that question because it gained enough notoriety that the LLM authors now train to avoid looking silly on it.

7B produced 2 answers, 1 was correct though more expensive and the second was incorrect.

The first attempt with 7B the model got stuck in an infinite loop.