NLI: once more around the track

Writing blog posts can be very useful. Sometimes. It helps you sort out your own ideas and it helps you communicate these ideas to someone who hasn't been part of the process of developing them. It can also help the group of people developing the ideas to get a better understanding of how pieces of a project fit together. Finally it can be very useful as a way of tidying up several thoughts, usually represented by papers, that you know are connected, but whose exact connection escapes you.

I have been writing in this blog on and off since 2011. This is an awful long time now. I hear younger friends dismissing the idea of blogs as something from a bygone era, something antiquated and abandoned like Orkut or Geocities. But for myself I do find the blogs of friends, or of  people I admire (like Terry Tao) extremely helpful. They tell you what you need to know, trying to be as easily  and friendly as they can be. My blog also reminds of things I want/need to know, so it's a shame that I haven't written much recently.

 I have been using my blog recently though. I have been reminding myself of Natural Language Inference (NLI) and its artifacts.  Because now I want to do it for mathematical texts, using LLMs and I sure had forgotten some of the lessons learned in the past. 

I started working on NLI in 2016, when Google produced its "we solved parsing" blog post. It's odd to read, 8 years later, the triumphalism of "The World’s Most Accurate Parser Goes Open Source". But hype can be very convincing and learning that they had open-sourced their parser I decided to try to see how far simply bag-of-concepts could get us when reasoning with sentences of the SICK corpus. That work with Alexandre and Fabricio never got written, but a series of papers with Katerina and Livy did get written and presented. 

The blog post mentioned above was written about some of the issues with that line of work and a journal paper (with Katerina, Hai Hu, Larry Moss and Alex Webb) was written about the line of work itself. A great presentation of the work by Katerina (only 6 minutes) is available here.

So now I want to make a list of the new work that I should be paying attention to:

1. Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, Dieuwke Hupkes

 

2. InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

 

Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, Dahua Lin 

 

3. Conditional and Modal Reasoning in Large Language Models 

Wesley H. Holliday and Matthew Mandelkern and Cedegao E. Zhang

4. To Know or Not To Know? Analyzing Self-Consistency
of Large Language Models under Ambiguity
Anastasiia Sedova Robert Litschko Diego Frassinelli Benjamin Roth Barbara Plank

 

5. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond,

 

Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, Erik Cambria

6. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers

Authors: Theo X. Olausson, Alex Gu, Benjamin Lipkin, Cedegao E. Zhang, Armando Solar-Lezama, Joshua B. Tenenbaum, Roger Levy