The take I'm gonna make is overly reductionist but I suspect the point is not to generate truth per-se but rather to provide the impression of an external validation. "Our models and analysis are excellent because of our AI toolbar; This scientific paper is good because we used AI to provide insight. and so on. It's used as a way to create new truths through an appeal to authority, though as you pointed out, people are starting to route around it mentally and general public trust of the AI secret sauce is low though it still carries the weight of truth in political, business and a academic circles.
The take I'm gonna make is overly reductionist but I suspect the point is not to generate truth per-se but rather to provide the impression of an external validation. "Our models and analysis are excellent because of our AI toolbar; This scientific paper is good because we used AI to provide insight. and so on. It's used as a way to create new truths through an appeal to authority, though as you pointed out, people are starting to route around it mentally and general public trust of the AI secret sauce is low though it still carries the weight of truth in political, business and a academic circles.