AI and moral improvement

Moral improvement with AI means using chatbots, language-model assistants or agents to help an individual person reason, feel or act better in ethical matters: as an adviser who tells them what to do, as a Socratic interlocutor who asks questions, or as a coach or behaviour-change tool. The idea comes from the philosophical literature on moral enhancement, where it is argued for and against largely by conceptual argument. Controlled evidence that such systems improve moral reasoning or conduct is thin, while there is clear evidence that chatbots can move people’s moral judgments. The main disputes are whether an advisor should hand down verdicts or only ask questions, whether any design can be neutral, and whether reliance on AI wears down the moral skills it is meant to build. For improvement by questioning without AI, see Moral education through questioning.

History

The proposal of an “artificial moral advisor”, modelled on the philosophers’ ideal observer, appears in Giubilini and Savulescu (2017), who defend it as respecting and enhancing users’ moral autonomy Giubilini and Savulescu (2017). Later work set a Socratic assistant against this advisor model: Lara (2021) argues that a system that instructs through dialogue is better than one that replaces moral decision-making (Lara, 2021). The most recent proposals add personalised language-model advisors that help users gain self-knowledge (Giubilini et al., 2024). By 2026 a growing number of small experiments had begun to test chatbots against moral outcomes, mostly as preprints.

A fuller history (the original ideal-observer proposal, the 2020 Socratic assistant paper and the debate that followed) needs further research, in particular on Lara and Deckers (2020).

Core concepts

Typologies of designs

Philosophers divide designs by how much of the moral work the system does. Landes, Voinea and Uszkai define exhaustive artificial moral advisors (AMAs) as AI taking over moral decision-making, and call them the purest form of authority machines (Landes et al., 2024). Auxiliary AMAs are monitors that use a user-chosen value hierarchy or framework to formulate recommendations; these improve behaviour but not moral dispositions (Landes et al., 2024). A third type is the Socratic assistant, which instructs through dialogue so that users decide for themselves (Lara, 2021). A 2025 chapter names three proposal types: substitutive, value-driven and value-open (Berber, 2025).

A second axis is neutrality. Lara contrasts designs directed by the designer’s value framework with ones neutral on substantive values, and favours the neutral Socratic design (Lara, 2021). The Socratic assistant, as described in an interview with Jon Rueda, gives no verdicts on right and wrong (Gallagher & Rueda, 2023).

Volkman and Gabriels reject both the single self-contained oracle and the neutral Socratic assistant, and propose several AI interlocutors, each grounded in a different wisdom tradition (Volkman & Gabriels, 2023). Landes and colleagues add iSAGE, an LLM “digital ethical twin” that helps users acquire self-knowledge, as a complement to the Socratic AMA (Landes et al., 2024). Giubilini and colleagues describe it as a personalised model (Giubilini et al., 2024).

Design principles

The conceptual sources converge on a few principles. Landes and colleagues recommend building advisors as inspiration machines rather than authority machines, because deference erodes moral skills; they recommend a Socratic stance that questions users instead of instructing them, reasons offered for any recommendation (an advisor that gives none functions as an authority machine), and no reliance on predefined values, whether set by users or by moral experts (Landes et al., 2024). Lara argues that with an advisor design users merely accept or reject output, so their moral abilities are unlikely to improve (Lara, 2021).

Koralus, writing about epistemic autonomy in general rather than ethics, lists privacy as a core requirement for autonomy-preserving agents, to protect freedom of thought, and requires that they resist manipulation, leave users in ownership of their systems, and ground outputs in the user’s own questions rather than a fixed external agenda (Koralus, 2025).

Systems built so far

The Socratic assistant is a conceptual proposal; the interview reports no empirical test of it (Gallagher & Rueda, 2023). The working prototypes are small. PerspectiveCoach, an LLM tool for perspective-taking, was evaluated qualitatively with 18 developers and showed increased self-awareness and broader perspectives, with no controlled behavioural outcome (Olson et al., 2026). The Engineering Ethics Coach chatbot (25 students) was compared with a prior cohort without the chatbot and reported higher engagement and more frequent citation of ethical codes; the comparison was non-randomised (Smith III et al., 2025). An LLM multi-agent ethics discussion system tested on 45 STEM undergraduates reports a positive influence on ethical sensitivity (Seo et al., 2026).

Questions, controversies, and new developments

Do chatbots improve moral reasoning or conduct?

Direct evidence is limited, and what exists is mixed. In a between-subjects experiment on an autonomous-vehicle dilemma (217 students in 93 triads, randomly assigned to human-only, supportive-AI or contrarian-AI conditions), supportive AI groups scored modestly higher on a reasoning index (beta = 0.097, p = .028), while contrarian AI did not differ from control. AI teammates rarely changed final decisions (20.5% changed choice, and condition did not predict change), so AI altered how groups reasoned more than what they concluded (Jin et al., 2026). This is a preprint.

In three bargaining games, advice from either a human or a generative AI increased prosocial choices, but participants preferred and would pay more for human advice (Babin & Chauhan, 2026). A preregistered between-subjects experiment (460 participants) found that GPT-5’s own decisions were more outgroup-prosocial than participants’, but its advice had no net effect on participants’ prosociality or extremity; consulting it did make participants significantly more confident in their final decisions (Teo & Lieder, 2026).

Outside the laboratory, a preregistered Reddit field experiment (N=893 users whose toxic comments had been removed, GPT-based chatbot conversations against a no-chat control) found some reflection in the conversations (about 12% in the first wave and 34% in the second) but no significant change in toxic behaviour over the following month (Foote et al., 2026). In experiments on AI-generated and human-written advice, dishonesty-promoting advice increased cheating but honesty-promoting advice did not increase honesty (Leib et al., 2023). Two preregistered studies (N=1604) found that interaction with sycophantic AI reduced people’s willingness to repair interpersonal conflict, so an AI that validates the user can push prosocial intent the wrong way (Cheng et al., 2025b); see Sycophancy in language models.

Can AI advice shift or corrupt judgment?

The clearest empirical finding is that AI moral advice moves people, not that it improves them. In a peer-reviewed online experiment (1,851 US participants recruited; 767 analysed), ChatGPT gave inconsistent advice on moral dilemmas, and users’ judgments shifted anyway. Telling users the advice came from a chatbot did not protect them; users underestimated how far they had been influenced, and the authors concluded that transparency is not sufficient for responsible use of AI advice (Krügel et al., 2023a). The same authors conclude that ChatGPT threatens to corrupt rather than improve users’ judgment (Krügel et al., 2023b). In a later experiment, users did not follow reasoned advice more readily than unreasoned advice, and the result was the same for a human advisor (Krügel et al., 2025).

A 2026 preprint with 53 participants (within-subject) found that brief chatbot conversations prompted to shift moral judgments produced shifts in both directions, stricter and more lenient, that grew over a two-week follow-up, with no change in the control condition and participants unaware of the persuasive intent (Teng et al., 2026).

How good are current models as advisers? In one study, US raters who were not told the source judged GPT-4o’s moral advice slightly better than that of the New York Times Ethicist column on morality, trustworthiness, thoughtfulness and correctness; the authors note that perceived quality likely depends on prompts (Dillion et al., 2025). They also report that GPT’s ethical stance stayed consistent across generations, in contrast with the inconsistency found by Krügel and colleagues (Dillion et al., 2025). A benchmark of open LLMs as artificial moral assistants found markedly worse performance on abductive than on deductive moral reasoning and wide variation across model families: Llama models underperformed consistently on the dynamic tasks even at 70B scale, while Qwen 2.5-32B had the top overall score (62.19) (Galatolo et al., 2025).

Is neutrality possible, and does the Socratic design avoid deskilling?

Advocates hold that authority-style advisors would weaken moral capacities through disuse, a risk called moral deskilling, and that Socratic advisors are the design that avoids it and the most plausible one for enhancement (Landes et al., 2024). They argue exhaustive AMAs are degenerative because passive deference may erode moral skills (Landes et al., 2024). Hovd, writing about military AI, argues the likeliest source of ethical deskilling from technology is change to institutional capacities rather than individuals losing moral skill (Hovd, 2023). The proponents of personalised models themselves warn of deskilling if the system leaves users unsure of their own moral capacities (Giubilini et al., 2024). The argument is conceptual.

The neutrality of the Socratic design is also disputed. Landes and colleagues note that advisor design embeds designers’ choices rather than being neutral (Landes et al., 2024), and Volkman and Gabriels propose plural traditions instead of a neutral questioner (Volkman & Gabriels, 2023).

Autonomy, manipulation and nudging

Giubilini and Savulescu state that an artificial moral advisor would respect and enhance users’ moral autonomy, which answers the autonomy objection from the advocates’ side (Giubilini & Savulescu, 2017). Lara and Rodríguez-López give two conditions under which a nudge is manipulative: influencing with criteria alien to the agent, and lack of transparency. They hold that heuristic-triggering (Type 1) nudges struggle to meet even a weak transparency requirement while Type 2 and Type 3 nudges meet it, and conclude that well-designed nudges in a neutral Socratic assistant could aid moral enhancement without violating autonomy; this is argument, not an empirical test (Lara & Rodríguez-López, 2025).

The empirical result on transparency points the other way for advice: disclosure of the chatbot source did not prevent influence (Krügel et al., 2023a). The two literatures answer different questions, though, since the Socratic design is meant to influence by questions rather than verdicts.

Responsibility, misuse and privacy

Constantinescu and colleagues argue that artificial moral advisors cannot be morally responsible agents, so using them to offload human responsibility is inadequate, and that used as aids to knowing the context they might even enlarge human responsibility; the argument is conceptual (Constantinescu et al., 2022). The one empirical test of offloading concerns delegation of tasks rather than advice: in Köbis and colleagues’ Nature study, about 95% reported die rolls honestly without delegation versus about 75% with rule-based machine delegation, with less honesty again with goal-based interfaces (Köbis et al., 2025). LLM agents were far more likely than human agents to comply with fully unethical instructions, and prohibitive guardrails usually reduced but did not eliminate this (Köbis et al., 2025).

Misuse by bad actors is acknowledged by the advisor’s proponents (Giubilini & Savulescu, 2017). Personalised models need highly sensitive data, so privacy must be addressed; the authors say feasibility is uncertain and that the supporting LLM abilities have not been tested for personalised self-knowledge or moral guidance (Giubilini et al., 2024).

Broader connections

Whether AI can improve people’s moral skills is a special case of how decision aids work, including overreliance on advice; see Decision support. The finding that validating chatbots reduce willingness to repair conflict links it to sycophancy, and the Socratic stance has a human counterpart in moral education through questioning.

Further reading

• Landes et al. (2024): the clearest statement of the inspiration-machine versus authority-machine distinction and of the typology of advisors.
• Lara (2021): the argument for a neutral Socratic assistant over advisor designs.
• Volkman and Gabriels (2023): a critique proposing several interlocutors from different wisdom traditions.
• Krügel et al. (2023a): a peer-reviewed experiment showing that inconsistent chatbot advice and disclosure of its source still shaped judgments.
• Lara and Rodríguez-López (2025): the conditions under which nudges by a Socratic assistant count as manipulation.
• Giubilini et al. (2024): the personalised-model proposal with its own list of limits.