AI: What's it good for?

[touchingcloth]

I work for a large public entity, so typically requests are garbled or have been through a few rounds of consultations before they get to me.

Example: today, somebody in Ottawa wanted the total increase of [thing] in [regions] to show how it's unaffordable these days (or affordable, or whatever point they were trying to make).

Ok well does an increase show its affordability? Not really, what if a big increase in one place is really just price harmonization? What if something decreasing in price is actually just a 1% reduction while it remains laughably unaffordable in that part of the country? So really, you want the actual cost of [thing], maybe with contextual data showing increases? What CPI raises? Or maybe increases as compared to CPI? Etc etc.

Basically - what point are you trying to make, mate, because it's not as simple as "write me a paper that proves we were right about [thing]".

Try getting AI to sort that.

To play devil's advocate, they call that sort of thing p-hacking or HARKing depending on what field you're working in, and AI would really simplify the work of hacking some p-values.

[Ferris]

To play devil's advocate, they call that sort of thing p-hacking or HARKing depending on what field you're working in, and AI would really simplify the work of hacking some p-values.

But you'd still want to know the outcome you're P-hacking towards, and translate the request from normal person into numbers to make it possible.

I'm quite ethical about it, so I suppose if you bridged that gap then replaced me with a t-1000 then it may be an improvement.

[touchingcloth]

But you'd still want to know the outcome you're P-hacking towards, and translate the request from normal person into numbers to make it possible.

I'm quite ethical about it, so I suppose if you bridged that gap then replaced me with a t-1000 then it may be an improvement.

I don't think LLMs are a million miles away from being able to take a request like "find things that can graph money as a line, and find me things that correlate with line-go-up where the correlation has a p-value <0.05, and sort them lowest-p and most appealing to public body decision makers in Ottawa and framed around the concept of affordability".

I'm on the same page as you re. ethics, and if someone presented those things to me I'd be asking why they decided to look for a correlation between the specific things that they did, but if you want to dazzle people with statistically significant numbers then AI could help you out.

[Endicott]

This is why I can never quite anticipate what my dog is going to want at any given moment, because he runs on a different platform than I do and has different goals.

Sausages.

[Ferris]

Yeah it's obviously sausages at all times.

[BlodwynPig]

I don't think LLMs are a million miles away from being able to take a request like "find things that can graph money as a line, and find me things that correlate with line-go-up where the correlation has a p-value <0.05, and sort them lowest-p and most appealing to public body decision makers in Ottawa and framed around the concept of affordability".

I'm on the same page as you re. ethics, and if someone presented those things to me I'd be asking why they decided to look for a correlation between the specific things that they did, but if you want to dazzle people with statistically significant numbers then AI could help you out.

I call it p-hucking

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5738950/#:~:text=Indeed%2C%20a%202016%20joint%20statement,or%20range%20of%20an%20effect.

[touchingcloth]

I call it p-hucking

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5738950/#:~:text=Indeed%2C%20a%202016%20joint%20statement,or%20range%20of%20an%20effect.

https://www.tandfonline.com/doi/full/10.1080/01973533.2015.1012991

[BlodwynPig]

https://www.tandfonline.com/doi/full/10.1080/01973533.2015.1012991

That's really informative and a bit intimidating given the majority of papers i review fall back on these methods.

[touchingcloth]

That's really informative and a bit intimidating given the majority of papers i review fall back on these methods.

It's an interesting approach. I can see why the tests have become the standard because they're easily comparable, but they're so easy to game unless reviewers are blind or protocols are pre-registered. It's a good example of Goodhart's Law in practice, especially when academics' careers depend on being published, and being published depends on statistical significance.