← Back to home

Where Objectives Come From — and Why Solutions Become Strategic Assets

A pedagogical position note on embedded agency, reward, and theory as a game transformer

Position Note · Macheng Shen · Collaborative drafting: GPT-5.4 Pro · 2026-03-17

Abstract

This note argues for two linked reframings.

First, an intelligent system's objective should not be treated as a primitive reward function that is simply handed to it from nowhere. Reward is often a useful representation language for behavior, but that does not mean reward is the first causal source of goals. For embedded agents, objectives are better understood as effective preferences over trajectories that emerge from joint agent–world dynamics. In this view, objectives are not purely endogenous and not purely exogenous. They are co-constituted by embodiment, viability constraints, interaction, social feedback, and explicit instruction.

Second, a theory about intelligence, control, alignment, or objective formation is not merely an answer to a static technical question. Once discovered, a theory can change what future actions are available, what can be predicted, what can be verified, and what kinds of commitments become possible. In that sense, some solutions behave like strategic assets: they alter the structure of later games. This is why safety theory can never be treated as purely neutral knowledge. Some theories stabilize the future; others accelerate races.

The note is intentionally pedagogical. Its aim is not to close the matter, but to make a clear conceptual map: what we think we have found, what remains conjectural, what this implies for current action, and which theoretical directions now seem most worth pursuing.

1. Why start with the question of objective at all?

One of the deepest questions in a theory of intelligence is not simply how an agent optimizes, but what it is optimizing in the first place.

Standard reinforcement learning gives a clean picture:

This picture is powerful. It gives us the modern language of policies, value functions, exploration, credit assignment, and planning. It is also often a good engineering approximation.

But it quietly leaves one major issue unresolved:

Where does the objective itself come from?

The reward function is typically treated as given. It is part of the problem statement. But for a deeper theory of intelligence, this is not enough. A representation of an objective is not yet an explanation of its origin.

This note begins from that gap.

2. Reward is strong as a representation, weak as an ontology

The first distinction we need is simple but important:

A reward function may be a very good way to represent a goal
without being the ontological source of that goal.

This distinction matters because two very different claims are often run together:

  1. Representation claim: behavior or preference can be expressed as the maximization of some reward-like quantity.
  2. Origin claim: the system fundamentally has that reward as its primitive objective.

The first claim may often be true, approximately or exactly. The second is much stronger.

This is why I want to say:

MDP reward is often a useful normal form, but not a first theory of objective.

Put differently:

That is the real problem.

3. From endogenous vs. exogenous objectives to coupled emergence

A natural response is to say: "Fine. Some objectives are internal, others are externally imposed."

That is a useful start, but I think it is still too coarse.

For biological agents, many goals appear deeply internal:

For current AI systems, many objectives seem more external:

But once we look carefully, the distinction starts to blur.

A system may begin with external instruction, yet internalize it into habits, heuristics, proxies, or self-maintaining loops. A biological objective may feel internal, yet still be shaped by environment, ecology, other agents, language, and culture.

So a more promising picture is this:

objective is neither purely endogenous nor purely exogenous; it emerges from the long-run coupling between a system and the world it inhabits.

Sometimes that coupling is best described as a game. Sometimes it is better described as coupled stochastic dynamics. Either way, the main point is the same:

4. A minimal formal skeleton

A clean place to start is not with r(s,a), but with the joint system.

Let the joint state be

x_t = (i_t, e_t)

where

The important caveat is that this decomposition is itself a coarse-graining. The agent/world boundary may be practical, not metaphysically fundamental.

Now write the joint dynamics as

x_(t+1) ~ F(x_t, ξ_t)

where ξ_t captures noise, hidden variation, or unmodeled disturbance.

At this level, the basic object is not a reward, but a trajectory:

τ = (x_0, x_1, x_2, ...)

and an effective preference over trajectories.

A schematic trajectory-level objective might look like

J(τ) =
  Φ_viability(τ)
+ Φ_control(τ)
+ Φ_epistemic(τ)
+ Φ_social(τ)

where:

The point is not that this exact decomposition is final. The point is that the objective appears here as an effective trajectory functional, not yet as a primitive Markov reward.

Only under additional conditions — for example, time consistency, additive decomposability, stable state space, and a good enough coarse-graining — might we reduce the problem to something like

max_π  E[ Σ_t γ^t r(s_t, a_t) ].

This suggests a very useful research question:

Under what conditions can an objective that emerges from coupled dynamics be faithfully reduced to reward maximization, and under what conditions can it not?

5. Three layers of objective

I find it helpful to distinguish three layers.

5.1 Constitutive objective

This is the deepest layer: what must hold for the system to continue being this system at all?

Examples:

5.2 Interactive objective

This layer emerges from repeated coupling with world and others.

Examples:

5.3 Explicit objective

This is the surface layer:

A lot of current AI engineering works almost entirely at this third layer. That is often useful. But a theory of intelligence that stops there risks mistaking the most legible layer for the deepest one.

One way to summarize the thesis is:

Reward usually captures the explicit layer, sometimes approximates the interactive layer, and often misses the constitutive layer.

6. When MDP-style reduction is likely to fail

If the objective really emerges from joint dynamics, then an MDP reduction can fail in identifiable ways.

Here are five important failure modes.

6.1 Boundary instability

The agent changes its own body, memory, sensors, or internal architecture, so the state space itself is not stable.

6.2 Viability-first regimes

The system cares first about remaining inside viability constraints, not about maximizing a single task score.

6.3 Path dependence

Two final outcomes may look identical in the terminal state, while still being non-equivalent because the paths taken to get there mattered.

6.4 Epistemic-control coupling

The system does not only want payoffs. It also wants to maintain model quality, controllability, or option value.

6.5 Social constitution of goals

Other agents, institutions, norms, language, and expectation loops do not merely constrain action. They help constitute what counts as success.

These failure modes do not show that reward-based modeling is useless. They show that it is often a reduction whose limits should be studied rather than assumed away.

7. A second thesis: theories are not just answers

Now comes the second half of the note.

Suppose we discover a theory about:

What have we gained?

A naïve answer is: we gained a better answer to a technical problem.

But that is usually too weak.

A theory can change:

So the right meta-level picture is:

A theory is not only an answer inside a game. A theory can transform the game itself.

8. The theory-as-game-transformer view

Write a base game as

G = (N, X, A, I, C, P, J)

where:

Now let a theory T induce a transformation

Γ_T : G ↦ G^T

This means T can change:

This is why a discovered solution can feel "weapon-like."

It is not merely a move. It is a game transformer.

9. Why some solutions behave like strategic assets

Once we see theory this way, another distinction becomes important.

Not all theories are equal. Some are mostly explanatory. Others are strategically powerful because they generate asymmetric leverage.

A rough signature for a theory might be:

Σ(T) = (ΔK, ΔS, ΔV, ΔR, τ_diff)

where:

This lets us say more than "good" or "bad."

Some theories are mostly stabilizers:

Some are mostly accelerants:

A very important special case is what I would call:

the capability externality of safety.

A safety theory may reduce local failure probability and still increase systemic race pressure, because it makes powerful deployment feel more justified, scalable, or governable to the actor who holds it.

So the question is not only:

Does this theory make one system safer?

It is also:

What equilibrium does this theory induce once many actors respond to it?

10. Private value versus social value

This naturally leads to one of the most practical questions.

Let:

Then define

Ω(T) = PV(T) - SV(T).

If Ω(T) is large and positive, then the theory will be strongly pursued privately even if society would prefer it to arrive later, arrive under stronger safeguards, or arrive in a more diffused and auditable form.

That gap is one way to formalize the intuition that:

some solutions are not just useful; they are strategically scarce in the short run, and therefore destabilizing.

This is part of why the nuclear analogy is suggestive but incomplete. The stakes may be nuclear-scale. But theories and algorithms are closer to information goods than to bombs: once found, they may diffuse like software rather than like fissile material.

So the sharper analogy is:

nuclear-scale stakes with software-style diffusion.

11. What I think we have actually found

This is a position note, not a theorem paper, so it is useful to separate findings from conjectures.

11.1 Findings

Finding 1. Reward is often a representation language for objective, not its first source.

Finding 2. For embedded agents, objectives are better modeled at the level of joint trajectories than at the level of primitive scalar rewards.

Finding 3. A theory about intelligence or safety can change the structure of later games, not only the quality of one action inside them.

Finding 4. The strategic value of a theory depends not only on truth, but on leverage, deployability, diffusibility, and equilibrium effects.

11.2 Conjectures

Conjecture 1. Objective Emergence Thesis
For embedded agents, objective is an emergent property of joint agent–world dynamics; reward is a compressed representation that applies only under further conditions.

Conjecture 2. Theory-as-Game-Transformer Thesis
Some theories about intelligence, control, or alignment are best understood as transformations of action sets, information structures, commitment structures, and effective objectives.

Conjecture 3. Goal-channel primacy
Theories that shape how objectives are formed may be strategically deeper than theories that only improve optimization under a fixed objective.

Conjecture 4. Capability externality of safety
A safety result can be locally stabilizing while systemically accelerating competition and deployment.

12. What this suggests we should do now

If the framing above is even partly right, then it suggests a different set of practical priorities.

12.1 Study objective emergence, not only reward design

We should ask not just how to specify a reward, but how objectives are constituted, internalized, and stabilized in coupled systems.

12.2 Build reductions carefully and diagnose their failure modes

Rather than assuming the MDP picture, we should characterize when it is a good reduction and when it throws away the very structure we care about.

12.3 Evaluate safety results by equilibrium signature, not only local effect

A method that improves local control may still raise systemic race pressure. We should evaluate safety proposals using both direct and strategic effects.

12.4 Treat key theories as governance-relevant assets

Some insights should be shared widely and quickly. Others may require staged release, structured review, red-teaming, or institutional safeguards before broad publication.

12.5 Build institutions that can absorb theory without immediately weaponizing it

The problem is not only "discovering the truth." It is discovering the truth in forms and under conditions that do not automatically turn every gain in understanding into a race multiplier.

13. Where this theory could go next

I see four strong future directions.

13.1 Representation theorem direction

Under what assumptions can trajectory-level objectives be represented by Markov rewards?

13.2 Impossibility / approximation direction

Under what assumptions must any MDP reduction lose crucial structure?

13.3 Meta-game direction

How do theories change action sets, information structures, commitment power, auditability, and diffusion speed?

13.4 Governance direction

How should societies govern theories whose private value, social value, and diffusion profile sharply diverge?

These directions would connect reinforcement learning, viability theory, active inference, game theory, information design, AI governance, and the economics of ideas into a more unified research program.

14. Closing

The deepest problem is not only how to optimize well once an objective is given.

It is also:

That is why this note argues for two reframings.

First:

objective should be studied as an emergent property of embedded dynamics.

Second:

some solutions should be studied as strategic assets that transform later games.

If these claims are even partly correct, then AI safety is not merely a matter of better reward design or bigger oversight benchmarks. It is also a matter of understanding how goals arise, how theories reshape power, and how to discover stabilizing forms of knowledge faster than destabilizing ones spread.

I think that is a research program worth making public early.

Selected pointers

This is a position note, not a literature review. But readers who want nearby foundations may look at:

← Back to home