This post is simply a condensed version of an interview that Brendan gave to Eric Gilliam.
I wanted to reproduce it here as I want to think about what my answers would be. (the full text of the interview is found in the link above)
What is Topos from a very high level?
Topos is a new structure for doing research for the public benefit.
In pursuit of this goal, we combine a focus on fundamental, long-term research
similar to a university with a theory of dynamic change similar to a tech startup.
The technical area we work in is mathematics and computer science —
particularly in collective modeling, collective inquiry, and cooperating in situations
that are incredibly complex and multidisciplinary.
We do this as a nonprofit whose work is supported by grants and contracts from
groups like DARPA, ARIA, philanthropic foundations, technical firms, and
gifts from high-net-worth individuals.
Can you talk more about collective modeling and how that
drives Topos’ grand technical vision?
Topos works on projects in areas like climate modeling, the systems engineering
of large airplanes, ensuring AI systems are resistant to catastrophic risks, etc.
The communities contributing to each of these technical systems are made up of a
variety of technical experts navigating different aspects of the system with their
own overlapping models — which often have their own merits and their own limitations.
Topos works on projects with specific user communities. That keeps us grounded in
the practicalities, ensuring we don’t get too caught up solving problems we find
beautiful but are only modestly connected to practical needs. We want to build a tool
that helps a variety of users reason across situations with the richness and
complexity that comes with combining and interpreting a mix of overlapping
models of the world.
One difference between this general modeling environment we hope to build that we
want to empower our users to represent the world in the conceptual language they use
in their work, not force them to translate those representations into numbers and
formal logic.
Which distinct advantages Topos has in tackling technical problems —
when compared to universities, startups, etc.?
For groundbreaking, public good technologies, we need tight integration
between research — the source of deep new ideas — and product — the
source of deep new questions. Our core advantage over a university is that we can
be user- and impact-driven. Our core advantage over a start-up is that we can tackle
deep technical risk. Let me add a bit of color to both points though.
First, academia. One thing that cannot be done in a university setting is, I think,
build and maintain technologies for the long term. And that is a core part of our
theory of change and how we assess our success in basic research. We want to do basic inquiry
around foundational questions — and we win enough grants that allow us to do that —
but we don’t want to be measured by the publications we produce: the publications are instrumental.
This contrasts with the known incentives against maintaining software and a user base in academia.
What separates us from start-ups? We’re driven by producing public goods — as long as we
can find the funding. We don’t orient ourselves around what’s best for investors.
Since I know you want to move on, the last thing I’d like to add is that Topos is a lot of fun!
We’re a collaborative, creative, and compassionate place with a great sense of collective purpose.
I’m inspired by our team every day.
Topos’ technical agenda is driven, in no small part, by pursuing a
General Theory of the Specific. Could you talk about some specific
projects you’ve 1) already worked on or 2) would love to pursue?
One very illustrative example is a systems engineering example. Imagine you’re
Boeing or Airbus and you’re constructing a passenger jet. There are thousands of people
involved in the process of this construction. How do you ensure it all comes together into a
vehicle that people can trust with their travel and lives for decades to come?
Some of that is scientific or engineering modeling. “What is the weight of the engine?”
“What are the properties of the materials going into the airplane wing?” “How do the material
properties relate to the aerodynamic properties?” Etc. But in bringing the parts together,
very significant social processes are also going on. Initially, this group is responsible for this
piece of the puzzle, specified in this way; that group is responsible for another piece specified
in that way. There are also a lot of initial guesses that get refined over time, through new
insights and changes in responsibility. This is the evolution of a process, a highly dynamic
process of knowledge discovery. It’s really important to keep track of that history and
understand how different decisions have been made in response to different sorts of
problems, insights, and constraints as the work of various groups converges into an
interoperable design. Passenger jets like the Boeing 737 have been flying for 50 years or more…
And, at this point, these firms often need to keep long-retired engineers
on retainer, sometimes until they die, in case they’re needed to answer a question.
This need often derives from this issue of organizational memory — not
remembering why you did what all those years ago, how that learning should
apply to the present situation, etc.
Right. Exactly. That sort of knowledge, that sort of understanding of the product, is very
difficult to institutionalize with existing technologies. But if you do study the processes of
collective sense-making carefully, there is reason to suspect — and it’s Topos’ ambition to
build — knowledge management software, structures, techniques, and cultures so you don’t
have to keep someone on retainer…unless they want to be on retainer and you want them
to be on retainer.
I’d love to hear about your current AI Safety project with ARIA.
What we’re doing with ARIA is a project for Davidad’s Safeguarded AI program. There is this
idea that AI systems will increasingly drive actions in the world. This will sometimes extend to
critical or complex situations. For example, it would be great if we could use AI systems to
optimize the power grid of the UK — payload distribution, inform the construction of new power generation sites, and so on. But there are huge possibilities for catastrophic risks to a country.
The power grid is literally the backbone of getting work, as in physical force, done in society.
Power grid failures, whether accidental, due to natural disasters, or from cyber and physical attacks, could cause widespread economic and physical harm, and even deaths.
How do you negotiate over the constraints of what you want — in an accessible, high-level
way — so your team can use AI tools to generate solutions that you know to be safe and respect
those constraints? One thing you need in these situations, where you have all these social
and physical factors, is an appropriately precise model of the world. This model should be
transparent, interpretable, and verifiable in certain ways. By providing a way for people — in a collaborative and interdisciplinary way — to construct subtle, formal models of the world, you
can enable them to run these AI systems inside sandboxes. In these sandboxes, you might have
tools helping you to combine techniques that allow you to validate your model and constraints —
e.g. automated reasoning, formal verification, theorem proving, etc. Doing this, a team can generate
very strong pieces of evidence (almost guarantees) that the system will be safe enough for public deployment.
6. Do you have any examples of ARIA projects you’d jump at the opportunity
to work on?
A program that has conceptual similarities to the Safeguarded AI program is Sarah and
Gemma’s program on Forecasting Climate Tipping Points. Their program deals with an extremely complex system — understood only through integrating perspectives from multiple disciplines,
including both the physical and social sciences — with deep impacts on society. Their goal is to
produce an early warning system for climate catastrophes that is precise, trustworthy, and actionable.
Similarly to our ARIA project with Davidad, we could work with ARIA creators (contractors) to
figure out things like, “What are the bounds of the system?” and “Under what circumstances will
it go into a catastrophic region of behavior?” We’d try to develop tools that enabled domain
scientists to construct a shared, interdisciplinary understanding of the climate system that is
responsive to new information from sensing systems and research.1
Of course, what that hypothetical project would look like specifically depends on many things —
the needs of the program and other creators, etc. I’d be interested in exploring what you’ve
mentioned, which is the idea of Topos being a sort of consulting AI for science shop that comes
in and assists with tooling to solve problems in and between particular domain areas.
Academia is not set up to support cross-disciplinary tooling and workflows, and we’d love to
complement the academic system in this way. But…I’d like to be very clear and say that domains
are very important. When talking about providing AI tools for other scientific disciplines,
I want to do it from a place of humility…
Climate science is a great example of where our work can add value. We build computational
tools for cooperating across the different conceptual frameworks and disciplines required to
navigate complex systems. One basic, almost too-obvious-to-state idea underlying our work
is that a good way of making concrete progress on difficult topics is writing things down.
You want to write down models; you want these written models to precisely reflect your
understanding of the world.
To adequately describe complex systems, we must use models from many different disciplines simultaneously. The problem, however, is that disciplines often create silos and it
becomes difficult to collaborate across these different perspectives. So, we need a way
of allowing conceptual-specific language that also enables interoperability between
the various languages. A major focus of our tools is this process of translation, mapping between perspectives, identifying overlap, and identifying difference.
In tackling this work, you really reveal the complexity of the situation. It’s a challenge.
Things become very messy, very fast. Having to translate what water means in one field vs.
another can be incredibly complex. A single concept here can turn into a web of relationships
over there. When dealing with problems like how water is witnessed through its relationships
to animal life in the oceans, you might even have to translate between webs of relationships.
In trying to create interoperability between frameworks, there are a whole lot of interdependencies
that are incredibly difficult to track. You want sophisticated, formal computational tools to
help you track that. That’s what Topos tries to build..
Our work relies on deep, respectful relationships with domain experts, our collaborators.
To establish such relationships, it’s critical we acknowledge that it’s very rare for an outside group to come in and do things that are groundbreaking in the minds of people who have been working in that area…
7. Topos pitch sounds something like: “This is a really complex system with
a lot of disciplines that have something useful to say about it. We’d like to come
in and build you tools that can be something like a synthesized reasoning engine.”
Your proposal attempts to take in all the existing models and output something very
interpretable, while overtly attempting to preserve what specialists would consider
useful about their fields’ models. You’re not looking to supplant them with any kind
of neural net or random forest. Is this a fair assessment?
Yes, exactly. We’re not the kind of AI for science shop that seeks to combine clever
algorithms, a wealth of data, and compute to yield breakthrough insights. That certainly has
its place, and we’ve seen some stunning successes from this approach, including the Nobel
Prize-winning work on AlphaFold. But that’s not our comparative advantage. What we want
to offer is a much more bespoke product. We hope that through our applied projects we can
assist users in eliciting the structures of and clarify the thinking in their domain. Over time,
we hope to build up a software stack that helps reify some of that knowledge that has been
clarified. If all goes well, our tools will enable them to more easily build up or utilize that knowledge.
8. Can you say how you obtain applied contracts?
Our most productive applied projects are driven by shared visions and strong relationships
with program managers, whether at DARPA, ARIA, or in private industry. They’ll have a particular concrete problem space they’re exploring. Over the years, through our publications and through conversations, they develop an understanding of what we do and how we can help their programs.
That often turns into them saying something like, “Okay, we want a proof of concept. My program
has a need in [simulated multiphysics, systems biology, epidemiology, or something like that].
Can you propose a way to demonstrate your ideas in this particular domain?” We then go do a
bunch of research on the problem area, often collaborating with domain experts, and construct a
bunch of software around this problem — built on top of some basic core libraries we build and
maintain.
9. How about undirected funds?
Undirected funds and, just as important, a strong network of visionary supporters, have been
critical in getting Topos off the ground. We are an organizational experiment. Experimenting,
exploring, and learning what does and doesn’t work for us: this takes real resources — it’s been
around a third of our total funding so far.
Our founding board — particularly our founding chair, Ilyas Khan — provided not just resources
but also mentorship, courage, and connections. I’ll be forever grateful to early major donors like
Jaan Tallinn and Jed McCaleb for their trust, even despite our naivety and inexperience.
Berkeley and Oxford have both been great communities for finding collaboration with
open-minded funders who both appreciate our technical ambition and that breakthroughs in
some of our approaches can have exciting spillovers for the wider science and technology ecosystem.
10. Any final words?
We’re something like a BBN-style org, right now. We have a clear technical vision that we
pursue with a mix of contracts with carefully selected partners and grants. These bespoke
projects are key to what we do and create deep insights into the nature of what the general
problem actually is. Balancing vision and contracts enables us to build technologies that
enable cooperation across different perspectives. Parts of our fundraising and governance
approaches may change over time — there are many upsides to the FRO governance
model, raising a basic research endowment, consulting using internal tools, etc. But our
general goals of enabling cooperation in areas of complexity through a balance of basic
research and applied projects will not.
Finally a picture that I decolorized from 2022, before my health issues.
