Friday, March 14, 2025

What about Topos?


 This post is simply a condensed version of an interview that Brendan gave to Eric Gilliam

I wanted to reproduce it here as I want to think about what my answers would be. (the full text of the interview is found in the link above)


  1. What is Topos from a very high level?


Topos is a new structure for doing research for the public benefit. 

 In pursuit of this goal, we combine a focus on fundamental, long-term research

 similar to a university with a theory of dynamic change similar to a tech startup.

The technical area we work in is mathematics and computer science — 

particularly in collective modeling, collective inquiry, and cooperating in situations

 that are incredibly complex and multidisciplinary. 

We do this as a nonprofit whose work is supported by grants and contracts from 

groups like DARPA, ARIA, philanthropic foundations, technical firms, and 

gifts from high-net-worth individuals.


  1. Can you talk more about collective modeling  and how that 

    drives Topos’ grand technical vision?

Topos works on projects in areas like climate modeling, the systems engineering 

of large airplanes, ensuring AI systems are resistant to catastrophic risks, etc. 

The communities contributing to each of these technical systems are made up of a 

variety of technical experts navigating different aspects of the system with their 

own overlapping models — which often have their own merits and their own 

limitations. Topos works on projects with specific user communities. 

That keeps us grounded in  the practicalities, ensuring we don’t get too caught up

solving problems we find beautiful but are only modestly connected to practical 

needs. We want to build a tool  that helps a variety of users reason across 

situations with the richness and  complexity that comes with combining

and interpreting a mix of overlapping  models of the world. 

One difference between this general modeling environment we hope to build [is] 

that we want to empower our users to represent the world in the conceptual language

they use in their work, not force them to translate those representations into numbers 

and formal logic. 


  1. Which distinct advantages Topos has in tackling technical 

    problems — when compared to universities, startups, etc.? 

For groundbreaking, public good technologies, we need tight integration 

between research — the source of deep new ideas — and products — the 

source of deep new questions. Our core advantage over a university is that we can 

be user- and impact-driven. Our core advantage over a start-up is that we can tackle 

deep technical risk. Let me add a bit of color to both points though.

First, academia. One thing that cannot be done in a university setting is, I think,  

build and maintain technologies for the long term. And that is a core part of 

our theory of change and how we assess our success in basic research. We want to do 

basic inquiry around foundational questions — and we win enough grants that allow 

us to do that — but we don’t want to be measured by the publications we produce: 

the publications are instrumental. This contrasts with the known incentives against 

maintaining software and a user base in academia.What separates us from start-ups? 

We’re driven by producing public goods — as long as we can find the funding. We 

don’t orient ourselves around what’s best for investors. Since I know you want to move 

on, the last thing I’d like to add is that Topos is a lot of fun! We’re a collaborative, 

creative, and compassionate place with a great sense of collective purpose. 

I’m inspired by our team every day.

  1. Topos’ technical agenda is driven, in no small part, by pursuing a 

    General Theory of the Specific. Could you talk about some specific

    projects you’ve 1) already worked on or 2) would love to pursue?

One very illustrative example is a systems engineering example. Imagine you’re 

Boeing or Airbus and you’re constructing a passenger jet. There are thousands of 

people involved in the process of this construction. How do you ensure it all comes 

together into a vehicle that people can trust with their travel and lives for decades to 

come?Some of that is scientific or engineering modeling. “What is the weight of the 

engine?” “What are the properties of the materials going into the airplane wing?” 

“How do the material properties relate to the aerodynamic properties?” Etc. 

But in bringing the parts together, very significant social processes are also going on. 

Initially, this group is responsible for this piece of the puzzle, specified in this way; 

that group is responsible for another piece specified in that way. There are also a lot 

of initial guesses that get refined over time, through new insights and changes in 

responsibility. This is the evolution of a process, a highly dynamic process of 

knowledge discovery.  It’s really important to keep track of that history and understand

how different decisions have been made in response to different sorts of problems, 

insights, and constraints as the work of various groups converges into an interoperable 

design. Passenger jets like the Boeing 737 have been flying for 50 years or more…

And, at this point, these firms often need to keep long-retired engineers 

on retainer, sometimes until they die, in case they’re needed to answer 

a question. This need often derives from this issue of organizational 

memory — not remembering why you did what all those years ago, how that 

learning should apply to the present situation, etc.

Right. Exactly. That sort of knowledge, that sort of understanding of the product, 

is very difficult to institutionalize with existing technologies. But if you do study the

processes of collective sense-making carefully, there is reason to suspect — and it’s 

Topos’ ambition to build — knowledge management software, structures, techniques, 

and cultures so you don’t have to keep someone on retainer…unless they want to be

 on retainer and you want them  to be on retainer.

  1. I’d love to hear about your current AI Safety project with ARIA.

What we’re doing with ARIA is a project for Davidad’s Safeguarded AI program

There is this idea that AI systems will increasingly drive actions in the world. This will 

sometimes extend to critical or complex situations. For example, it would be great if we 

could use AI systems to optimize the power grid of the UK — payload distribution, 

inform the construction of new power generation sites, and so on. But there are huge 

possibilities for catastrophic risks to a country. 

The power grid is literally the backbone of getting work, as in physical force, done 

in society. Power grid failures, whether accidental, due to natural disasters, or from 

cyber and physical attacks, could cause widespread economic and physical harm, 

and even deaths.How do you negotiate over the constraints of what you want — 

in an accessible, high-level way — so your team can use AI tools to generate 

solutions that you know to be safe and respect those constraints? One thing you 

need in these situations, where you have all these social and physical factors, is 

an appropriately precise model of the world. 

This model should be transparent, interpretable, and verifiable in certain ways. 

By providing a way for people — in a collaborative and interdisciplinary way — 

to construct subtle, formal models of the world, you can enable them to run these

AI systems inside sandboxes. In these sandboxes, you might have tools helping 

you to combine techniques that allow you to validate your model and constraints 

—e.g. automated reasoning, formal verification, theorem proving, etc. 

Doing this, a team can generate very strong pieces of evidence 

(almost guarantees) that the system will be safe enough for public deployment.


  6. Do you have any examples of ARIA projects you’d jump at the 

opportunity  to work on? 

 A program that has conceptual similarities to the Safeguarded AI program is 

 Sarah and  Gemma’s program on Forecasting Climate Tipping Points. Their 

program deals with an extremely complex system — understood only through 

integrating perspectives from multiple disciplines, including both the physical 

and social sciences — with deep impacts on society. Their goal is to produce 

an early warning system for climate catastrophes that is precise, trustworthy, 

and actionable.Similarly to our ARIA project with Davidad, we could work with

ARIA creators (contractors) to figure out things like, “What are the bounds 

of the system?” and “Under what circumstances will it go into a catastrophic

 region of behavior?” We’d try to develop tools that enabled domain 

scientists to construct a shared, interdisciplinary understanding of the climate 

system that is responsive to new information from sensing systems and research.

Of course, what that hypothetical project would look like specifically depends 

on many things — the needs of the program and other creators, etc. I’d be 

interested in exploring what you’ve mentioned, which is the idea of Topos 

being a sort of consulting AI for science shop that comes in and assists with

 tooling to solve problems in and between particular domain areas. 

Academia is not set up to support cross-disciplinary tooling and workflows, 

and we’d love to complement the academic system in this way. But…

I’d like to be very clear and say that domains are very important. 

When talking about providing AI tools for other scientific disciplines, 

I want to do it from a place of humility…

Climate science is a great example of where our work can add value. 

We build computational tools for cooperating across the different conceptual 

frameworks and disciplines required to navigate complex systems. One basic, 

almost too-obvious-to-state idea underlying our work is that a good way of 

making concrete progress on difficult topics is writing things down. 

You want to write down models; you want these written models to precisely 

reflect your understanding of the world.

To adequately describe complex systems, we must use models from many 

different disciplines simultaneously. The problem, however, is that  

disciplines often create silos and it becomes difficult to collaborate across 

these different perspectives. So, we need a way of allowing conceptual-specific 

language that also enables interoperability between the various languages. 

A major focus of our tools is this process of translation, mapping between

perspectives, identifying overlap, and identifying difference.

In tackling this work, you really reveal the complexity of the situation. It’s a challenge. 

Things become very messy, very fast. Having to translate what water means in one field vs.

 another can be incredibly complex. A single concept here can turn into a web of relationships

 over there. When dealing with problems like how water is witnessed through its relationships 

to animal life in the oceans, you might even have to translate between webs of relationships.

 In trying to create interoperability between frameworks, there are a whole lot of 

interdependenciesthat are incredibly difficult to track. You want sophisticated, formal

 computational tools to help you track that. That’s what Topos tries to build..

Our work relies on deep, respectful relationships with domain experts, our collaborators. 

To establish such relationships, it’s critical we acknowledge that it’s very rare for an 

outside group to come in and do things that are groundbreaking in the minds of people

 who have been working in that area…

 

7. Topos pitch  sounds something like: “This is a really complex system 

with a lot of disciplines that have something useful to say about it. 

We’d like to come in and build you tools that can be something like 

a synthesized  reasoning engine.” Your proposal attempts to take in 

all the existing models and output something very interpretable, while

 overtly attempting to preserve what  specialists would consider useful 

about their fields’ models. You’re not looking to supplant them with 

any kind  of neural net or random forest. Is this a fair assessment?

Yes, exactly. We’re not the kind of AI for science shop that seeks to combine clever 

algorithms, a wealth of data, and compute to yield breakthrough insights. 

That certainly has its place, and we’ve seen some stunning successes from this 

approach, including the Nobel Prize-winning work on AlphaFold. But that’s 

not our comparative advantage. What we want  to offer is a much more bespoke 

product. We hope that through our applied projects we can assist users in 

eliciting the structures of and clarify the thinking in their domain. Over time, 

we hope to build up a software stack that helps reify some of that knowledge 

that has been clarified. If all goes well, our tools will enable them to more 

easily build up or utilize that knowledge.

 

 8. Can you say how you obtain applied contracts?

Our most productive applied projects are driven by shared visions and strong 

relationships with program managers, whether at DARPA, ARIA, or in private 

industry. They’ll have a particular concrete problem space they’re exploring. 

Over the years, through our publications and through conversations, they 

develop an understanding of what we do and how we can help their programs.

 That often turns into them saying something like, “Okay, we want a proof 

of concept. My program has a need in [simulated multiphysics, systems biology, 

epidemiology, or something like that]. Can you propose a way to demonstrate 

your ideas in this particular domain?” We then go do a  bunch of research on the

 problem area, often collaborating with domain experts, and construct a 

bunch of software around this problem — built on top of some basic core 

libraries we build and maintain.

 

9. How about undirected funds?

Undirected funds and, just as important, a strong network of visionary supporters, have been 

critical in getting Topos off the ground. We are an organizational experiment. Experimenting,

 exploring, and learning what does and doesn’t work for us: this takes real resources — it’s been

 around a third of our total funding so far.

Our founding board — particularly our founding chair, Ilyas Khan — provided not just resources

 but also mentorship, courage, and connections. I’ll be forever grateful to early major donors like

 Jaan Tallinn and Jed McCaleb for their trust, even despite our naivety and inexperience. 

Berkeley and Oxford have both been great communities for finding collaboration with 

open-minded funders who both appreciate our technical ambition and that breakthroughs in

 some of our approaches can have exciting spillovers for the wider science and technology

 ecosystem.

 

10. Any final words?

 

We’re something like a BBN-style org, right now. We have a clear technical vision that we 

pursue with a mix of contracts with carefully selected partners and grants. These bespoke 

projects are key to what we do and create deep insights into the nature of what the general

 problem actually is. Balancing vision and contracts enables us to build technologies that 

enable cooperation across different perspectives. Parts of our fundraising and governance

 approaches may change over time — there are many upsides to the FRO governance 

model, raising a basic research endowment, consulting using internal tools, etc. But our 

general goals of enabling cooperation in areas of complexity through a balance of basic 

research and applied projects will not.




A picture that I decolorized from 2022, before my health issues.

 


No comments:

Post a Comment