>[...] or alternatively that they ARE working toward the same goal, but they don’t know that everyone else is also working toward that goal.
This is actually a very interesting and true idea. The issue here is that sharing the same goal can be detrimental to agents. Alice and Bob want to eat the same apple, but are unaware of each other. Here you could propose some game-theoretic example of how their knowledge of each other could make them split the apple and avoid conflict, but not necessarily so. If Alice has a far bigger capacity to operate in the world, then "alignment" could result in even worse outcomes for Bob as Alice could make sure that Bob fails. Maybe I am being nitpicky with the word goal, but I do not think it implies cooperation in itself.
>[...] or alternatively that they ARE working toward the same goal, but they don’t know that everyone else is also working toward that goal.
This is actually a very interesting and true idea. The issue here is that sharing the same goal can be detrimental to agents. Alice and Bob want to eat the same apple, but are unaware of each other. Here you could propose some game-theoretic example of how their knowledge of each other could make them split the apple and avoid conflict, but not necessarily so. If Alice has a far bigger capacity to operate in the world, then "alignment" could result in even worse outcomes for Bob as Alice could make sure that Bob fails. Maybe I am being nitpicky with the word goal, but I do not think it implies cooperation in itself.