Argument Formalizer

Source

The Instrumental Convergence Thesis: Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by many intelligent agents. [...] Where there are convergent instrumental values, we may be able to predict some aspects of a superintelligence's behavior even if we know virtually nothing about that superintelligence's final goals.

As written, the conclusion doesn't follow from these premises.

Steps marked ‘assumed’ state what you'd have to assume to reach the conclusion.

Nick Bostrom, "The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents", Minds and Machines 22(2), 2012

  1. Several instrumental values can be identified that are convergent, meaning their attainment would increase the chances of an agent's goal being realized across a wide range of final goals and a wide range of situations.

    #
  2. Convergent instrumental values are likely to be pursued by many intelligent agents.

    #
  3. assumed

    If [1] and [2], then where there are convergent instrumental values, it may be possible to predict some aspects of a superintelligence's behavior even if virtually nothing is known about that superintelligence's final goals.

    Assumed so the listed premises lead to the conclusion.

    #
  4. Therefore, where there are convergent instrumental values, it may be possible to predict some aspects of a superintelligence's behavior even if virtually nothing is known about that superintelligence's final goals. [1,2,3]

    #
As written, before the assumed premise
  1. Several instrumental values can be identified that are convergent, meaning their attainment would increase the chances of an agent's goal being realized across a wide range of final goals and a wide range of situations.

  2. Therefore, convergent instrumental values are likely to be pursued by many intelligent agents. [1]

  3. Therefore, where there are convergent instrumental values, it may be possible to predict some aspects of a superintelligence's behavior even if virtually nothing is known about that superintelligence's final goals. [2]

What's missing

The conclusion introduces the concept 'intelligent agents' and the evaluative characterization 'likely to be pursued by many,' neither of which appears in premise 1. Premise 1 speaks of 'an agent's goal being realized across a wide range of final goals and a wide range of situations,' but does not mention intelligence or likelihood of pursuit by many agents. A bridge premise linking convergent instrumental values to pursuit by intelligent agents would be needed, making this valid with unstated reasoning rather than valid.

The conclusion introduces the concept of 'superintelligence' and the epistemic claim that 'it may be possible to predict some aspects of behavior even if virtually nothing is known about final goals.' Premise 1 only states that convergent instrumental values are likely to be pursued by many intelligent agents. The conclusion applies this to superintelligence specifically and draws a predictability inference that requires an unstated bridge premise connecting 'likely to be pursued by many intelligent agents' to 'predictability of behavior when final goals are unknown.' These concepts are not present in premise 1. However, a true bridge premise (e.g., 'if a value is likely pursued by many intelligent agents including superintelligences, then its pursuit can be predicted even without knowing final goals') would make the conclusion follow, so this is valid with unstated reasoning rather than invalid.

Formalize another