This post is simply a condensed version of an interview that Brendan gave to Eric Gilliam.
I wanted to reproduce it here as I want to think about what my answers would be. (the full text of the interview is found in the link above)
What is Topos from a very high level?
Topos is a new structure for doing research for the public benefit.
In pursuit of this goal, we combine a focus on fundamental, long-term research
similar to a university with a theory of dynamic change similar to a tech startup.
The technical area we work in is mathematics and computer science —
particularly in collective modeling, collective inquiry, and cooperating in situations
that are incredibly complex and multidisciplinary.
We do this as a nonprofit whose work is supported by grants and contracts from
groups like DARPA, ARIA, philanthropic foundations, technical firms, and
gifts from high-net-worth individuals.
Can you talk more about collective modeling and how that
drives Topos’ grand technical vision?
Topos works on projects in areas like climate modeling, the systems engineering
of large airplanes, ensuring AI systems are resistant to catastrophic risks, etc.
The communities contributing to each of these technical systems are made up of a
variety of technical experts navigating different aspects of the system with their
own overlapping models — which often have their own merits and their own
limitations. Topos works on projects with specific user communities.
That keeps us grounded in the practicalities, ensuring we don’t get too caught up
solving problems we find beautiful but are only modestly connected to practical
needs. We want to build a tool that helps a variety of users reason across
situations with the richness and complexity that comes with combining
and interpreting a mix of overlapping models of the world.
One difference between this general modeling environment we hope to build [is]
that we want to empower our users to represent the world in the conceptual language
they use in their work, not force them to translate those representations into numbers
and formal logic.
Which distinct advantages Topos has in tackling technical
problems — when compared to universities, startups, etc.?
For groundbreaking, public good technologies, we need tight integration
between research — the source of deep new ideas — and products — the
source of deep new questions. Our core advantage over a university is that we can
be user- and impact-driven. Our core advantage over a start-up is that we can tackle
deep technical risk. Let me add a bit of color to both points though.
First, academia. One thing that cannot be done in a university setting is, I think,
build and maintain technologies for the long term. And that is a core part of
our theory of change and how we assess our success in basic research. We want to do
basic inquiry around foundational questions — and we win enough grants that allow
us to do that — but we don’t want to be measured by the publications we produce:
the publications are instrumental. This contrasts with the known incentives against
maintaining software and a user base in academia.What separates us from start-ups?
We’re driven by producing public goods — as long as we can find the funding. We
don’t orient ourselves around what’s best for investors. Since I know you want to move
on, the last thing I’d like to add is that Topos is a lot of fun! We’re a collaborative,
creative, and compassionate place with a great sense of collective purpose.
I’m inspired by our team every day.
Topos’ technical agenda is driven, in no small part, by pursuing a
General Theory of the Specific. Could you talk about some specific
projects you’ve 1) already worked on or 2) would love to pursue?
One very illustrative example is a systems engineering example. Imagine you’re
Boeing or Airbus and you’re constructing a passenger jet. There are thousands of
people involved in the process of this construction. How do you ensure it all comes
together into a vehicle that people can trust with their travel and lives for decades to
come?Some of that is scientific or engineering modeling. “What is the weight of the
engine?” “What are the properties of the materials going into the airplane wing?”
“How do the material properties relate to the aerodynamic properties?” Etc.
But in bringing the parts together, very significant social processes are also going on.
Initially, this group is responsible for this piece of the puzzle, specified in this way;
that group is responsible for another piece specified in that way. There are also a lot
of initial guesses that get refined over time, through new insights and changes in
responsibility. This is the evolution of a process, a highly dynamic process of
knowledge discovery. It’s really important to keep track of that history and understand
how different decisions have been made in response to different sorts of problems,
insights, and constraints as the work of various groups converges into an interoperable
design. Passenger jets like the Boeing 737 have been flying for 50 years or more…
And, at this point, these firms often need to keep long-retired engineers
on retainer, sometimes until they die, in case they’re needed to answer
a question. This need often derives from this issue of organizational
memory — not remembering why you did what all those years ago, how that
learning should apply to the present situation, etc.
Right. Exactly. That sort of knowledge, that sort of understanding of the product,
is very difficult to institutionalize with existing technologies. But if you do study the
processes of collective sense-making carefully, there is reason to suspect — and it’s
Topos’ ambition to build — knowledge management software, structures, techniques,
and cultures so you don’t have to keep someone on retainer…unless they want to be
on retainer and you want them to be on retainer.
I’d love to hear about your current AI Safety project with ARIA.
What we’re doing with ARIA is a project for Davidad’s Safeguarded AI program.
There is this idea that AI systems will increasingly drive actions in the world. This will
sometimes extend to critical or complex situations. For example, it would be great if we
could use AI systems to optimize the power grid of the UK — payload distribution,
inform the construction of new power generation sites, and so on. But there are huge
possibilities for catastrophic risks to a country.
The power grid is literally the backbone of getting work, as in physical force, done
in society. Power grid failures, whether accidental, due to natural disasters, or from
cyber and physical attacks, could cause widespread economic and physical harm,
and even deaths.How do you negotiate over the constraints of what you want —
in an accessible, high-level way — so your team can use AI tools to generate
solutions that you know to be safe and respect those constraints? One thing you
need in these situations, where you have all these social and physical factors, is
an appropriately precise model of the world.
This model should be transparent, interpretable, and verifiable in certain ways.
By providing a way for people — in a collaborative and interdisciplinary way —
to construct subtle, formal models of the world, you can enable them to run these
AI systems inside sandboxes. In these sandboxes, you might have tools helping
you to combine techniques that allow you to validate your model and constraints
—e.g. automated reasoning, formal verification, theorem proving, etc.
Doing this, a team can generate very strong pieces of evidence
(almost guarantees) that the system will be safe enough for public deployment.
6. Do you have any examples of ARIA projects you’d jump at the
opportunity to work on?
A program that has conceptual similarities to the Safeguarded AI program is
Sarah and Gemma’s program on Forecasting Climate Tipping Points. Their
program deals with an extremely complex system — understood only through
integrating perspectives from multiple disciplines, including both the physical
and social sciences — with deep impacts on society. Their goal is to produce
an early warning system for climate catastrophes that is precise, trustworthy,
and actionable.Similarly to our ARIA project with Davidad, we could work with
ARIA creators (contractors) to figure out things like, “What are the bounds
of the system?” and “Under what circumstances will it go into a catastrophic
region of behavior?” We’d try to develop tools that enabled domain
scientists to construct a shared, interdisciplinary understanding of the climate
system that is responsive to new information from sensing systems and research.
Of course, what that hypothetical project would look like specifically depends
on many things — the needs of the program and other creators, etc. I’d be
interested in exploring what you’ve mentioned, which is the idea of Topos
being a sort of consulting AI for science shop that comes in and assists with
tooling to solve problems in and between particular domain areas.
Academia is not set up to support cross-disciplinary tooling and workflows,
and we’d love to complement the academic system in this way. But…
I’d like to be very clear and say that domains are very important.
When talking about providing AI tools for other scientific disciplines,
I want to do it from a place of humility…
Climate science is a great example of where our work can add value.
We build computational tools for cooperating across the different conceptual
frameworks and disciplines required to navigate complex systems. One basic,
almost too-obvious-to-state idea underlying our work is that a good way of
making concrete progress on difficult topics is writing things down.
You want to write down models; you want these written models to precisely
reflect your understanding of the world.
To adequately describe complex systems, we must use models from many
different disciplines simultaneously. The problem, however, is that
disciplines often create silos and it becomes difficult to collaborate across
these different perspectives. So, we need a way of allowing conceptual-specific
language that also enables interoperability between the various languages.
A major focus of our tools is this process of translation, mapping between
perspectives, identifying overlap, and identifying difference.
In tackling this work, you really reveal the complexity of the situation. It’s a challenge.
Things become very messy, very fast. Having to translate what water means in one field vs.
another can be incredibly complex. A single concept here can turn into a web of relationships
over there. When dealing with problems like how water is witnessed through its relationships
to animal life in the oceans, you might even have to translate between webs of relationships.
In trying to create interoperability between frameworks, there are a whole lot of
interdependenciesthat are incredibly difficult to track. You want sophisticated, formal
computational tools to help you track that. That’s what Topos tries to build..
Our work relies on deep, respectful relationships with domain experts, our collaborators.
To establish such relationships, it’s critical we acknowledge that it’s very rare for an
outside group to come in and do things that are groundbreaking in the minds of people
who have been working in that area…
7. Topos pitch sounds something like: “This is a really complex system
with a lot of disciplines that have something useful to say about it.
We’d like to come in and build you tools that can be something like
a synthesized reasoning engine.” Your proposal attempts to take in
all the existing models and output something very interpretable, while
overtly attempting to preserve what specialists would consider useful
about their fields’ models. You’re not looking to supplant them with
any kind of neural net or random forest. Is this a fair assessment?
Yes, exactly. We’re not the kind of AI for science shop that seeks to combine clever
algorithms, a wealth of data, and compute to yield breakthrough insights.
That certainly has its place, and we’ve seen some stunning successes from this
approach, including the Nobel Prize-winning work on AlphaFold. But that’s
not our comparative advantage. What we want to offer is a much more bespoke
product. We hope that through our applied projects we can assist users in
eliciting the structures of and clarify the thinking in their domain. Over time,
we hope to build up a software stack that helps reify some of that knowledge
that has been clarified. If all goes well, our tools will enable them to more
easily build up or utilize that knowledge.
8. Can you say how you obtain applied contracts?
Our most productive applied projects are driven by shared visions and strong
relationships with program managers, whether at DARPA, ARIA, or in private
industry. They’ll have a particular concrete problem space they’re exploring.
Over the years, through our publications and through conversations, they
develop an understanding of what we do and how we can help their programs.
That often turns into them saying something like, “Okay, we want a proof
of concept. My program has a need in [simulated multiphysics, systems biology,
epidemiology, or something like that]. Can you propose a way to demonstrate
your ideas in this particular domain?” We then go do a bunch of research on the
problem area, often collaborating with domain experts, and construct a
bunch of software around this problem — built on top of some basic core
libraries we build and maintain.
9. How about undirected funds?
Undirected funds and, just as important, a strong network of visionary supporters, have been
critical in getting Topos off the ground. We are an organizational experiment. Experimenting,
exploring, and learning what does and doesn’t work for us: this takes real resources — it’s been
around a third of our total funding so far.
Our founding board — particularly our founding chair, Ilyas Khan — provided not just resources
but also mentorship, courage, and connections. I’ll be forever grateful to early major donors like
Jaan Tallinn and Jed McCaleb for their trust, even despite our naivety and inexperience.
Berkeley and Oxford have both been great communities for finding collaboration with
open-minded funders who both appreciate our technical ambition and that breakthroughs in
some of our approaches can have exciting spillovers for the wider science and technology
ecosystem.
10. Any final words?
We’re something like a BBN-style org, right now. We have a clear technical vision that we
pursue with a mix of contracts with carefully selected partners and grants. These bespoke
projects are key to what we do and create deep insights into the nature of what the general
problem actually is. Balancing vision and contracts enables us to build technologies that
enable cooperation across different perspectives. Parts of our fundraising and governance
approaches may change over time — there are many upsides to the FRO governance
model, raising a basic research endowment, consulting using internal tools, etc. But our
general goals of enabling cooperation in areas of complexity through a balance of basic
research and applied projects will not.
A picture that I decolorized from 2022, before my health issues.
No comments:
Post a Comment