NLI: once more around the track

Writing blog posts can be very useful. Sometimes. It helps you sort out your own ideas and it helps you communicate these ideas to someone who hasn't been part of the process of developing them. It can also help the group of people developing the ideas to get a better understanding of how pieces of a project fit together. Finally it can be very useful as a way of tidying up several thoughts, usually represented by papers, that you know are connected, but whose exact connection escapes you.

I have been writing in this blog on and off since 2011. This is an awful long time now. I hear younger friends dismissing the idea of blogs as something from a bygone era, something antiquated and abandoned like Orkut or Geocities. But for myself I do find the blogs of friends, or of  people I admire (like Terry Tao) extremely helpful. They tell you what you need to know, trying to be as easily  and friendly as they can be. My blog also reminds of things I want/need to know, so it's a shame that I haven't written much recently.

 I have been using my blog recently though. I have been reminding myself of Natural Language Inference (NLI) and its artifacts.  Because now I want to do it for mathematical texts, using LLMs and I sure had forgotten some of the lessons learned in the past. 

I started working on NLI in 2016, when Google produced its "we solved parsing" blog post. It's odd to read, 8 years later, the triumphalism of "The World’s Most Accurate Parser Goes Open Source". But hype can be very convincing and learning that they had open-sourced their parser I decided to try to see how far simply bag-of-concepts could get us when reasoning with sentences of the SICK corpus. That work with Alexandre and Fabricio never got written, but a series of papers with Katerina and Livy did get written and presented. 

The blog post mentioned above was written about some of the issues with that line of work and a journal paper (with Katerina, Hai Hu, Larry Moss and Alex Webb) was written about the line of work itself. A great presentation of the work by Katerina (only 6 minutes) is available here.

So now I want to make a list of the new work that I should be paying attention to:

1. Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

Lovish Madaan, David Esiobu, Pontus Stenetorp, Barbara Plank, Dieuwke Hupkes

 

2. InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning

 

Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, Dahua Lin 

 

3. Conditional and Modal Reasoning in Large Language Models 

Wesley H. Holliday and Matthew Mandelkern and Cedegao E. Zhang

4. To Know or Not To Know? Analyzing Self-Consistency
of Large Language Models under Ambiguity
Anastasiia Sedova Robert Litschko Diego Frassinelli Benjamin Roth Barbara Plank

 

5. Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond,

 

Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, Erik Cambria

6. LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers

Authors: Theo X. Olausson, Alex Gu, Benjamin Lipkin, Cedegao E. Zhang, Armando Solar-Lezama, Joshua B. Tenenbaum, Roger Levy

The Importance of summer

 Some time ago I've decided to copy some of my blog posts in the Topos Institute Blog to here, where it's easier for me to find what I am looking for.  There I have written (with others) the following posts:

1. Introducing the MathFoldr Project

https://topos.site/blog/2021-07-11-introducing-mathfoldr/

2. The many facets of Networked Mathematics

https://topos.site/blog/2022-04-18-facets-of-networked-mathematics/

3. Mathematical concepts: how do you recognize them?

https://topos.site/blog/2022-11-16-mathematical-concepts/

4. Preparing for Networked Mathematics

https://topos.site/blog/2023-01-05-preparing-for-networked-mathematics/ 

but recently I have been very bad about writing blog posts. Not only here, but especially there. It is the usual running around, trying to do things and not getting anything much done. I exaggerate a little, after all 2024 was the year of the `Prospects of Formal Mathematics" at the Hausdorff Institute of Mathematics in Bonn! This was excellent, but quite a bit of work. 

 

(I was going to post one of the many lovely pictures of our program, but reading Hausdorff's history there, I think I prefer to have him instead. He was really impressive!)
 

There was also the great week in Oxford for 40th MFPS and ACT 2024. Working as Alex Simpson's co-chair was great fun!

 

 Being pampered by Steve and Nicola in Oxford, then going to Cambridge to see lifelong friends, everything was simply wonderful!

 

 

Marta Bunge's Thesis Republished

 

On Monday 4th November 2024 Professor Marta Bunge's 1966 PhD thesis Categories of set valued functors was published  in  Reprints in Theory and Applications of Categories (TAC), No. 30 (2024) pp. 1-8 http://www.tac.mta.ca/tac/reprints/articles/30/tr30.pdf. Category Theory is traditionally a small branch of Mathematics and a reprint in TAC is an important accomplishment in our community.

Together with TAC's Volume 40 - Bunge Festschrift* and the donation of Marta's Library to USP, SP in Brazil, this completes a multi-year project that hoped to showcase the achievements of Prof. Bunge in a very male-oriented environment. 

Marta was a good friend and, like many women I know, one that did not believe very much in the impact of her own work. When I first told her that I wanted to get a special volume of TAC dedicated to the whole of her oeuvre, the first thing she said was that she was not sure she deserved it. But of course she did and many members of our category theory community wanted to celebrate her work. Thus it was nice to be able to see the volume 40 of TAC, celebrating her work. Especially nice to see the remembrances of Marta from Prof Andrée C. Ehresmann and from Prof Silvia Bunge, her daughter.

Maria Manuel Clementino, Jonathon Funk (one of her earlier PhD students) and I had the pleasure of editing the volume. Many thanks Maria Manuel and Jonathon!

It was very sad that a few months after we started thinking about the festschrift, Marta fell ill and did not live to see the volume in her honour. She died on 25 October 2022. I've asked mathematicians about other ways of celebrating her work and Nathanael Arkor suggested producing an online version of her original PhD thesis from 1966. Prof Michael Barr and Pam Freyd, as well as Silvia and Eric Bunge worked  hard to get a copy of the thesis from which Nathanael could work, and very soon Nathanael had a first version of the reprint done. This needed some support from editors of TAC to be accepted and Maria Manuel and Matias Menni were happy to lend their support to the project. Geoff Crutwell (as managing editor of TAC) was always very supportive, so the thesis has just been reissued as a TAC reprint.

This kind of completes a cycle, started  in 2013 when Marta was my Ada Lovelace hero. Then I think I called Marta 'Doyenne' of the women category theorists. But Andrée C. Ehresmann corrected me, as she started before Marta (in any case Andrée was Ada Lovelace hero 2023, a bit late, I'm afraid). I wanted to thank again all the collaborators mentioned here, as well as all the authors and reviewers of the special volume for their help in this project. But I especially want to thank Silvia and Eric Bunge for their donation of Marta's books to the Dept of Mathematics of USP in Brazil. This will make a huge difference to many people!

.

(I really need to write more in this blog, but somehow it seems harder and harder.)

Ada Lovelace Day 2024: Annie Zaenen

 

This Ada Lovelace Day (ALD) blog post is long overdue. ALD was only yesterday, but I meant to write this blog post a long time ago. The tradition of ALD is that women should do something to commemorate the fact that women are still considered second class citizens in this day and age on Ada Lovelace Day.

It can be a very small thing (like this overdue post) or it can be something  that requires more effort  like organizing a meeting and/or talking to many people about the issues. I have produced ALD blog posts celebrating heroes of mine since 2011. A recap of the first ten years was in Ada Lovelace Day 2021. Another overdue ALD post was Andree Ehresmann in 2023, but  I should have had one for Annie a while back. Maybe the issue was that Annie is foremost a linguist and people tend of think of linguistics as not being mathematical enough. Well, they're wrong.

Making Annie my Ada Lovelace hero for 2024 is overdue, but the world at large has been paying attention to Annie's work. Next week she's getting an Honorary Doctorate from the University of Konstanz, Germany. Here's the website with the program for the workshop associated to it. 

Miriam Butt and Tracy Halloway King are organizing the party!

 

 I just have to make sure that I manage to get there, not a triviality with my track record.

Hausdorff Institute of Mathematics: the beginning of our trimester

 

Summer 2024 was the summer of the Hausdorff Institute of Mathematics programme on formalized mathematics. The programme was a whole trimester 6th May til 9th August, but I was a bit late as Eric graduated from Law School in St Louis, MO and I didn't know when that was going to happen exactly. So I could only promised to leave the US in June and I arrived in Bonn as soon as I could, on the 2nd June.

I hope to produce several blog posts about the programme, but indeed I am starting late. The arrival in Bonn was not without problems. Thank goodness Davide Trotta was already there and he able to get my keys for me and be up until midnight or so, when I finally managed to arrive. By contrast, arriving at the Institute was easy and it's a very pleasant place.

My office was very much the first thing you see when entering the the building. I've posted the picture before, but here it goes again. It's nice to have an office of your own, with your name on the door. Very rare in industry these days: it's all open plan settings, so I've enjoyed it.

Shame the huge electronic "whiteboard" didn't work very well with Macs, and I didn't feel like harassing IT to get to work for me. I was very surprised that macs weren't considered the default option, but there you go.  Also shame I forgot to take pictures of the flat (that was nice and modern) and of the next door building (some sort of offices for the University of Bonn) that I mistook for my accommodation when I arrived. I'm glad I didn't persevere in trying to call the porter/superintendent as there wasn't one and I was on the wrong building, number 57 instead of the intended 59. I blame Grothendieck and the lateness of the hour. But of course the keys didn't work in the wrong building, but when I realized my mistake all was fine.

Topos-Chapman Collaboration

In May 2024  Topos Institute and the University of Chapman came together to launch an official partnership. Our main aim is to explore joint research projects and expand our role mentoring and teaching the next generation of technologists working in the public interest, as Brendan Fong wrote in the Topos blog post. The official announcement from Chapman can be found in the Doctor of Science in Mathematics, Physics and Philosophy (MPP) program  and here.

We celebrated our partnership with a workshop where talks were given both by Topos and Chapman scientists. The schedule is attached, and the recordings are available in YouTube in this playlist. This collaboration came about, after several months of articulation, after I gave a talk in Chapman's OCIE seminar in April 2023 on Dialectica constructions.

Pasadena Last December

 

 

 In December 2023 I went to the the Workshop on 

Open-source cyberinfrastructure supporting mathematics research 

organized by Robert Beezer, Steven Clontz, and David Lowry-Duda at the American Instittue of Mathematics (AIM), CalTech, Pasadena.  There was a report on the workshop activities and some Workshop videos were recorded.

For me it was very good to meet a whole community that I wasn't aware of. This community is committed to thinking about the accesssibility of learning mathematics, and everyone is trying to think about how to use digital tools for that. But of course the experiences, the tools and the approaches are all very different. 

 

There are people interested in computer algebra systems and in theorem provers, just like in the Dagstuhl Seminar  23401 "Automated mathematics: integrating proofs, algorithms and data" that I went to in October 2023. But there were also people interested in visualizations, in publishing textbooks and on all sorts of teaching systems. The emphasis was definitely on the teaching aspects, more than on the research side that I tend to concentrate on.

However, there was a momentous little story that I should write down now, or I will forget it. Kim Morrison was giving a tutorial on Lean for Mathematics on Tuesday morning at 9 am. When they went to  explain Lean's blueprint and show an example of its action, Kim stopped and said: oh well, when I was writing my slides last night after dinner, this (the "blueprint" for the proof of the theorem Gowers, Tao, Green and Manners had proved recently) was still blue. But now there's nothing to show, it's green! 

 

The theorem had been proved by someone in Lean while we were sleeping! It felt somewhat magical, the dawn of a different kind of mathematics. It was a bit like when, on a tip of a number theorist friend in Cambridge, I went to hear Wyles talk about the proof of Fermat's last theorem. His formal announcement in the Isaac Newton Institute, that many years ago. I didn't understand anything of Wyles proof and I still hardly understand what a Lean blueprint is, but the magical side of maths just happens!

If you want to understand the history behind the theorem the Quanta magazine article ‘A-Team’ of Math Proves a Critical Link Between Addition and Sets by Leila Sloman explains it very well!

I confess that instead of enjoying the magic, I went directly to the Lean Zulip to check that there were no reports of vandalism or hacking on mathlib, no triggering of the green, by mistake. Only when everything seemed normal over there, I managed to enjoy the moment! Proofs more than 20 years in the making are like fine wine.

Dagstuhl in the Fall (2023)

 

In the spring of 2022 I got sick, not as sick as I thought then, but sick enough to end up in the emergency room of the hospital in Stanford twice in the same week in May. So the whole year of 2022 got pear-shaped and many of my plans had be cancelled. I had a small procedure at the end of the year--on the day before Thanksgiving exactly-- and this made the year of 2023 a year of recovery.

The year of recovery meant I was going to try to do  three trips (since I had to cancel many in 2022): one simply to Chapman, South California, one to Bloomington, Indiana and one to  Pisa and Padova, in Italy.  All of them went well, mostly: I got covid when returning from Indiana. So I  decided to do another three trips: to St. Louis, Missouri, to Dagstuhl, Germany and to Pasadena, CA.

The one I was most worried about was the one to Dagstuhl, as I do not speak German and Dagstuhl can be fairly inaccessible. But I partnered with Jacques Carrette and Stefania Dumbrova and all went well getting there and getting back home. And being at Dagstuhl is always a pleasure! I have met several people who, like me, are concerned about using digital tools to make mathematics more accessible.

The schedule of the talks and discussions is in https://www.dagstuhl.de/23401/schedule.pdf. I really need to read a bit more about some of the work that I did not manage to grasp while there.

Preparing for Network Mathematics 2023

 

 

(source: Twitter/X account Stonehenge UK)

This post is a continuation of Preparing for Networked Mathematics.

During 2023 we had a few more talks on theorem provers and automated deduction. And on how the work in math provers can impact the world.

On the 27th April we heard Georges Gonthier on

Foothills and cathedrals: organising the libraries behind big proofs

 

Not exactly like the picture below, but close:

On the 4th May we had Michael Barany on

How Categories Come to Matter: On the history and sociology of categories in modern mathematics

 

 

 

 

which wasn't so much about math provers, but about why category theory matters so much right now. And, as usual with Michael, how to study the `relationship between abstract knowledge and the modern world'.

On 18th May we had Clark Barrett on

Proof Certificates in Satisfiability Modulo Theories

 

and on the 31st August we had Nat Shankar on

Abstraction Engineering with the Prototype Verification System (PVS)

 

Then we had Leonardo de Moura on the 7th September talking about

Lean 4: Empowering the Formal Mathematics Revolution and Beyond

Yes, we already had Jeremy Avigad, Kevin Buzzard and Johan Commelin talking about LEAN in 2022, but it was good to hear from the main creator of the Lean  prover too!

 

(He even talked on the 7th September, a Brazilian National Holiday, and neither of us noticed it!) 

In 12th October we had Dominic Orchard talking on           Programming for the Planet

 Tom Leinster from Edinburgh University on

Entropy and diversity: the axiomatic approach

and then in November we had Minhyong Kim with a beautiful talk on

Who Owns Mathematics: A Question of Identity

 

 

 

 

 

 

Of course these last three talks say more about how I feel about the role of mathematics in the world than about how we can use provers to do more  mathematics. But I think this is essential too!

Formal Math: are we there yet?

 I have not touched my blog much this last year. A shame. I need to write more and I need to try to correct pictures and links that stopped working. Now the programme in Bonn, on Prospects of Formal Mathematics is coming closer, so writing should help me decide to organize my time in Germany. But first things first. I need to email a few more people about our arrangements.