Its Doomed.
Machine learning research is doomed. After LLMs and transformers, few things in the field are interesting anymore. If you weren’t in NLP before, you have probably pivoted. And if you were, your papers have probably converged to these three topics: prompt engineering, fine-tuning, and scaling with more data. In industry, almost nothing exists outside of these anymore. In academia, researchers and professors are scrambling to keep up with the LLM hype.
To be fair there is quite a lot of low hanging fruit. There are lots of minimal effort things to play around with and you can crank out several experiments in parallel quickly. It's also much easier to publish having keywords like LLMs, transformers, foundation models, scaling laws, etc.
NLP research is fast and furious, and don't get me wrong, it’s all very exciting in terms of practical use cases and applications in business, technology, increasing shareholder value, things of that sort. But IMHO, it's made research very boring. Whatever happened to the evolutionary algorithms, and the spiking neural networks, and the bayesian models in ML? Maybe these fell off for a reason – they were hard to scale, or didn't have any real-world deployment purposes. Or maybe, we didn’t explore them deep enough and abandoned them prematurely when transformers came out. Perhaps we’ve just settled into a local maximum with transformers. Shit. Is attention really all you need, or is it just what we've been pressured into and incentivized to do?
What I hope to see re-emerge in the field are cognitive-inspired architectures. Even though I have been swayed into basic NLP research coming from a more cogsci oriented background, I still refuse to be bitterlessonpilled by these compute fanatics that I’m surrounded by here in the Bay. I don’t think it's practical or sustainable to just keep scaling and scaling until we hit some upper bound (but who knows where that will be). And I do believe that thinking about intelligence from first principles will serve helpful in the future.