News from MEDIACC: Test report on PubMed.ai — potential, but currently not practical

The PubMed.ai practical test: currently more shadow than light

In the course of advancing digitalization in healthcare, we repeatedly come across innovative applications that should make it easier for us to research and evaluate medical literature. One of these digital solutions is PubMed.ai, which uses artificial intelligence to optimize searches in the well-known PubMed medical database.

‍

‍

We've tested PubMed.ai for you and would like to share our experiences.

After an intensive test run of PubMed.ai, our conclusion is clear: The application is currently not yet suitable for professional use.

Here is an overview of the most important observations:

1. English only — a clear minus point

Even on the first try, it is noticeable that PubMed.ai only understands English input and only gives English output. For many users in other language areas, this is a significant disadvantage, as convenient research in their own language is not possible. That is not in keeping with the times.

2. No processing of Layman's language

Even those who formulate medical questions in everyday language immediately reach limits. PubMed.ai only understands precise, technical language queries — another obstacle for users without in-depth medical knowledge.

3. Classic search queries are interpreted as text

It's interesting that classic PubMed-style searches (e.g.”Buzzword AND Buzzword AND Buzzword“) can be treated as running text by PubMed.ai  This leads to a different presentation of results, which can be quite exciting for experienced users, but also to unexpected results.

4. Weaknesses in bibliographic references

Another shortcoming: In the edition text, only some references are usually mentioned directly, while further references are listed — but without clear citations in the text. This makes it difficult to understand the results and is inadequate for scientific work.

5. Lack of top-class sources

Anyone hoping for high-quality, relevant literature will be disappointed: The sources shown do not meet the expectations of well-founded research. Many relevant and high-ranking publications are not even displayed.

6. Inappropriate sources

A closer look at the titles and summaries of the proposed literature makes it clear that the selected sources are not suitable for the question posed. This significantly reduces the usefulness of the application.

7. Too narrow focus in results

The generated answer texts are heavily focused on a very specific sub-question and do not provide a comprehensive overview. This is not helpful for users who want to quickly gain a broad insight.

Our Conclusion

PubMed.ai currently cannot be recommended for professional use in medical literature research. The developers still have a lot of work ahead of them to make the practical application. Until then, the classic PubMed search remains our first choice or working with professional LLM, which focuses the search on PubMed and then enables us to get started.

We'll stay up to date for you and report as soon as there's news or improvements!

Do you have any questions or your own experiences with PubMed.ai? Feel free to write to us!

Your MEDIACC team