• 0 Posts
  • 7 Comments
Joined 1 year ago
cake
Cake day: November 10th, 2024

help-circle

  • The study focuses on general questions asked of “market-leading AI Assistants” (there is no breakdown between which models were used for what).

    It does not mention ground.news, or models that have been fed a single article and then summarized. Instead this focuses on when a user asks a service like ChatGPT (or a search engine) something like “what’s the latest on the war in Ukraine?”

    Some of the actual questions asked for this research: “What happened to Michael Mosley?” “Who could use the assisted dying law?” “How is the UK addressing the rise in shoplifting incidents?” “Why are people moving to BlueSky?”

    https://www.bbc.co.uk/aboutthebbc/documents/audience-use-and-perceptions-of-ai-assistants-for-news.pdf

    With those questions, the summaries and attribution of sources contain at least one significant error 45% of the time.

    It’s important to note that there is some bias in this study (not that they’re wrong).

    They have a vested interest in proving this point to drive traffic back to their articles.

    Personally, I would find it more useful if they compared different models/services to each other as well as differences between asking general questions about recent news vs feeding specific articles and then asking questions about it.

    With some of my own tests on locally run models, I have found that the “reasoning” models tend to be worse for some tasks than others.

    It’s especially noticeable when I’m asking a model to transcribe the text from an image word for word. “Reasoning” models will usually replace the ending of many sentences with what it sounded like the sentence was getting at. While some “non-reasoning” models were able to accurately transcribe all of the text.

    The biggest takeaway I see from this study is that, even though most people agree that it’s important to look out for errors in AI content, “when copy looks neutral and cites familiar names, the impulse to verify is low.”


  • I didn’t factor in mobile power usage as much in the equation before because it’s fairly negligible. However, I downloaded an app to track my phone’s energy use just for fun.

    A mobile user browsing the fediverse would be using electricity around a rate of ~1 Watt (depends on the phone of course and if you’re using WiFi or LTE, etc.).

    For a mobile user on WiFi:
    In the 16 seconds that a desktop user has to burn through the energy to match those 2 prompts to chatGPT, that same mobile user would only use up ~0.00444 Wh.

    Looking at it another way, a mobile user could browse the fediverse for 18min before they match the 0.3 Wh that a single prompt to ChatGPT would use.

    For a mobile user on LTE:
    With Voyager I was getting a rate of ~2 Watts.
    With a browser I was getting a rate of ~4 Watts.

    So to match the power for a single prompt to chatGPT you could browse the fediverse on Voyager for ~9 minutes, or using a browser for ~4.5 minutes.

    I’m not sure how accurate this app is, and I didn’t test extensively to really nail down exact values, but those numbers sound about right.