ACL2026
This year, I got the honor of being selected to go to ACL conference. Previously, I did CVPR, Neurips, EMNLP and Interspeech. This is my first time coming to ACL. I did 1 year of computational linguistics in Saarbrucken during erasmus and it left a lasting respect for linguistics, language and cognitive science, so I was fairly intrigued and hopeful when I went. This years focus (or, my personal interest to justify corporate sponsorship) was information retrieval and multilingual ACL2026 is in San Diego, 1 hour flight away from home! The hotel I got was 20 minute walk every day so I definitely have been getting my steps in.
Day 1 - Future of work with AI Agents:blog/conference/ACL2026: :journal/2026-07-05:
On day 1, I took a tutorial out of curiosity. I went to see [[https://future-of-work-llm-tutorial.github.io/static/slides/acl-2026-tutorial_future-of-work.pdf][Future of work with AI agents]]tutorial. It was mostly based on the [[https://www.anthropic.com/research/labor-market-impacts][blog post]] by Anthropic. The two most intersting tidbits I learned was the collaboration gap, where multiple agents working on serial tasks perform worse than a single agent and the gap between benchmarks and real life work performance, because real life is more than just coding. I also remember they talked about a general desire to work together with AI, with AI taking over more menial tasks and humans doing the more creative things (that has not been the case so far!), and agents not acting like humans when collaborating. If I had to be a CLI servant of another person, I would definitely ask more clarification questions and check in more often, instead of blurbing out thousands of lines of code on a 2 sentence prompt.
-
To work with humans, AI should work like humans? It seems the tutorial makers prefer more finding common grounds (something.. humans would do), whereas developers prefer agents to just do their own thing and maximum velocity. I feel the latter also pushes us to unsustainable work reviewing all of this alien code, compared to having more confidence and trust in a single output when doing former. Time will tell! I liked that they bundled in some linguistic tasks like clarification, follow-up and acknowledgement and said AI behaves very differently on these axis compared to human. This is something ML people don't measure. After listening to this I realized the way we post train for maximum helpfulness, the agents definitely behave differently than a colleague would.
-
Human agency scale My favorite diagram was the human agency scale from their survey paper, where H1 is agent doing everything with no involvement, and H5 being human doing the task themselves. And in this spectrum H3 is equal partnership which is my personal goal. I felt I should categorize my various projects between H2 and H4. Sometimes we do want agents to do most of the work with human being easy verifier, other times we want the opposite.
-
US Job market is not just maths Finally, they stated while AI companies focus on maths and programming (antrophic/openai poorly branching into finance and health), it is not where majority of the GDP is. Most work is administrative, sales and management if we consider the capital and man hours. These tasks might not be best served as a coding task, but as a human communication task, which the AI agents are not that good at, closing the loop.
-
What do people want? Finally, the survey asked workers how they want AI to help them, and most wanted reusable, modifiable workflows they can verify. There were some talks in how current swe benchmarks are made (take a repository, decompose it into commits, ask model to turn it into tasks, then use that as a benchmark) and how to analyze human and agent traces.
Day 1 - Towards Effective and Efficient Multi-Agent Language Model Systems :blog/conference/ACL2026: :journal/2026-07-05:
As second tutorial, I went to multi-agent systems. It was more about how other industries hopped on the "loop engineering" schtick (which I have not personally gotten to work on coding) in biomedical and quantum research. They showed benchmarks, AI scientists and a lot of fancy systems and diagrams I was a bit skeptical about.
Seems after finetuning on domain data with benchmarks that include environments with tools, they can get work done with 30B parameters outperforming GPT 4. But I wasn't sure if this applies for GPT 5.5 today which would presumably subsume all of it (especially if they put the benchmark online). I do think it makes sense to turn a lot of domain knowledge into RL environments and benchmarks to see if frontier models can capture them, but I was also skeptical about the quality of the environments, since it is mostly code and a docker image instead of more realistic tools like the quantum measurements device they showed. It also seems too code-heavy where the science is just writing python code analyzing raw data, instead of working with devices that produce measurements and analyzing those.
Finally, why use VLLM to analyze a plot, if you have the table data underneath and can do any statistical tests instead? Are we going backwards?
Day 3 - 21st Workshop on Innovative Use of NLP for Building Educational Applications :blog/conference/ACL2026: :journal/2026-07-05:
On day 3, I went to a workshop that was more practical and hands on about education. The first speaker, Lucy Li, left the biggst impression on me. ** The aftermath of DrawEduMath Her task was The aftermath of DrawEduMath, which introduced a dataset of math problems, with student drawn solutions and the proper solution. The findings were:
- VLMs are worse at describing contents of student work, if it has math errors, compared to student work without errors.
- This means VLMS just assumed the work is correct and would not catch errors! The data the models are trained on is of high quality and low mistakes, which is opposite of what students are!
- students with math errors also had worse handwriting. Fixing handwriting improved the model performance a bit, but the main learning is: If you are a poor student in need of help, AI will fail you, which is a representation problem.
- Models today are optimized for experts, not novices. A expert user has a prescribed way to do a set of tasks, but a novice does not, and current models absolutely focus on expert users instead of novices. ** Knowledge tracing modeling from dialogue The following two talks were about knowledge tracing. This is modeling the state of the student learning. They extract it from dialogue data with various granularities. I thought knowledge tracing is incredible if it can be done automatically, since we can evaluate various teaching paradigms and quantify them! I was interested because I am in process of learning languages. The approaches were.. interesting. One of them measured LLM logits for GOOD/BAD/HARD/EASY tokens to get probabilities, before merging the scores into a logistic regression model to predict state, the other talk tried to map dialogue turns to a state such as "student gave direct answer, teacher gave hint, student struggling" and then tried to formalize a knowledge learnign state from this, showing the best learning outcomes are when student slightly struggles and gets a hint. There were some statistical modeling tests proving the consecutive learning states were best predictor of success, but it majorly went over my head.
** Vocabulary difficulty prediction for english learners This was a fun regression task, where they give english words and try to predict the difficulty. The annotation was produced by taking british council test taker data and some postprocessing. Most winners were bert-backbone swaps with ensembles.
MeLLM was the main reason I went to the conference and it was inspiring. I also noticed a bigger concentration of deepmind and meta folks hanging around which was a clear signal we are all in this mess together.
- Code switching in dialogue and dialogue systems I learned a fair bit about code switching in humans and reason we do it (empathy, linguistic alignment between 2 people) and how code switching changes the prosody (pitch, intensity and speaking rate) I also learned LLMs should learn to process these better, and maybe they could also code switch to mimic humans. In my culture (Czech), code switching and using non-czech is frowned upon, so I wonder if it is something that differs from culture to culture. If I met a Czech person, I would exclusively speak czech. Prof Julia Hirschberg did tate some neat open research questions:
- How to identify information load (diffculty of producing particular words)
- Code switching in low resource languages?
- How do we generate realistic, natural sounding code switching?
- How is LLM generated code switched text compared to human produced text
-
Multilingual disparities in LLM-Based Safety Judgments - Evidence from brand safety applications Brand safety was a fairly pragmatic task, but the findings were in my opinion universal. They took euronews articles in 13 languages, which should be semantically identical, and ran the same LLM with same prompt to detect undesirable content on them. There was major cross-lingual disagreement between the judgements, where 1 language was fine but another one had a ton of red flags. There is a bias we can't control for which is surfaced after seeing document in a certain language. No solution, but super neat finding!
-
When English isnt the best teacher - source language effects in cross-lingual in-context learning This paper went for best paper award. It analyzed exhaustivelly what happens if you add in context example that is same or different from the rest of the prompt, to work on multilingual tasks.
Key findings
- Target language is not always best source (ICL example)! - 24%. English is worst source language in 16%. Best target languages are worst source languages.
- no correlation between transfer performance and linguistic similarity
- low resource, non-latin script languages are best sources
- Lookin at task performance we can't predict language confusion. Non-latin script cause least confusion when used at source, but see most confusion when being target languages. This is something that occurds on 3B sized models!
-
On the limits of model merging for multilinguality in Pre-Training This paper showed a failure (compelte metric collapse) when they trained 10 mono lingual LLMs and merged them with various strategies. I appreciated the candid results a lot with no "magic solution that fixes it". They did extensive ablation study merging various pairs of models together and evaluating on a simple benchmark to verify how the model works. (poorly) Seems representational alignment is critical during training and that different languages elicit different weight / attention patterns that might not be compatible. I wondered what kind of tokenizer they used for 10 distinct languages before merging the models. I did not have courage to ask. I regret it.
-
Lost in execution - on the multilingual robusness of tool calling in LLMs It seems models return tool calls with argument in target language instead of english. My only thought was, why not make the tools i18n friendly? To me, it seemed a LLM calling a maps tool with chinese address in mandarin behaves correctly. But good to see another example of language confusion. They showed various solutions, like adding prompt explicitly begging to return in english, translating the query before asking the agent, or translating the model output. None of them fully fixed the issue.
-
From multilingual to multicultural modeling This talk was fantastic, and also rather alarming. Cohere trained a 3B model specifically to be multilingual with a proper mixture, but they discovered the cultures are not equally represented even when we balance languages. They tagged data (using their own model) with the geographical location and cultural content tags and did some data analysis -> there is a double long tail distribution where low resource languages get even compounded to low-low resource regions. And even a certain region has no representation at all because the dominant culture subsumes it in the location. The paper opens a lot of research questions I was really excited about, and no solution.
There are many small, focused benchmarks for a given culture or region, there is no global "cultural awareness" benchmark. Multicultural benchmark does not have to be multilingual.
** Culture as a data problem Is cultural knowledge in the data today? -> No! Current Datapoints have locale assigned as metadata, but not culture as a taxonomy of labels. One solution is to tag datapoints with cultural labels, location metadata. Maybe cultural dimension (specifying the rubric) and measuring the training data allows us to balance data better? 5.6M datapoints
** Culture funnel Cultural content decreases from pre to post training datasets. In pretraining we have cultural data, in post-training we see it go down from 60% to 12%. In reasoning datasets and task/use-case datasets there is 0 cultural grounding, they are increasingly technical. ** Cultural geolocation diversity Adding more languages doesn't make data more cultural, but more geographically diverse. Combining the long tail of languages, cultures and regions, we see a even bigger skew. A double long tail ** Do tasks need culture? There is alignment in cultural (tagged) data with some tasks - translation, but less so for medical QA. ** Cultural finetuning
- use culturally marked data only by filtering data with the tags
- marker-augmented finetuning -> Keep the tags in the data, which makes the model predict the tags during regression. Keep all data! Cultural data is rare, but the tags help it focus; does that leak during inference? ... if we use locale-specific prompt suffixes, is that sufficient to incorporate cultural preference and sensitivity?
On day 4, I sat through some oral talks and walked the poster session. The welcome talk had some banger quotes. "We have a habit of making theories that keep getting proven wrong" or "Some people are concerned ACL will just become a B-tier ML conference" The speaker talked really candidly about the current state of ACL, computational linguistics as a field, LLMs, jobs, frontier labs hogging all the capital and only making themselves richer and where the industry R&D is headed.
One thing that resonated to me was the desire to distinguish and diversify the study of linguistics as a cognitive science to not become a H-Index booster for ML papers. That resonated with me a lot, because I in fact thought of it as a B-tier ML conference. There were a lot of AI/Agent research papers of dubious quality and, if I might be a bit harsh, some of them looked like EMNLP rejects. My favorite talks were definitely about culture, code switching, and measuring phenomena, not the LLM agent stuff. A lot of the research was painstakingly done on GPT3.5 and older models that are not even publicly available anymore while I casually prompted Opus 4.8 do to a problem ticket triaging tool. I felt the gap between "industry" and academia to be bigger than ever before. At least in 2019, we could use a old GPU cluster at university and get within 70% of a typical ML research paper in compute, now colleges don't even have sufficient access to entire automatic research harnesses that can vibe generate entire experiments, while meta engineers can tokenmaxx themselves to death.