You are here:

AI chatbots – what not to do?

AI-based tools can significantly boost work efficiency and speed up research, content creation and data analysis. They are increasingly becoming an everyday aid both in business and in our private lives.

The problem arises when users start to treat AI chatbots as an infallible source of truth or thoughtlessly pass on data to them that should never leave the organisation or their personal devices.

Below, we outline the most common mistakes associated with using AI chatbots – and explain why they can be dangerous.

Digital padlock symbolising data security

1. Sharing sensitive data

The biggest mistake is pasting confidential information – both business and personal – into AI chatbots. Many people assume that the conversation ‘stays on the screen’, whereas in practice the data may be stored, analysed or processed by the service provider.

Why is this risky?

Models may use the data for further training.

The consumer versions of many chatbots use the text entered by users by default to improve the quality of the models. This means that confidential information may be used in the training process or stored in system logs.

Lack of control over the infrastructure

Data ends up on the servers of third-party providers, such as OpenAI, Google or Anthropic. The user usually has no real control over:

  • where the data is stored,
  • who has access to it,
  • how long it is stored,
  • or what the security procedures are.

Selected conversations may be analysed by humans

AI providers use moderators and quality analysts. Excerpts from conversations may be reviewed to improve system performance or detect abuse.

System errors and cyberattacks do happen

AI systems are also vulnerable to security flaws. One example was the 2023 vulnerability in ChatGPT, which allowed some users to view other people’s conversation histories. Furthermore, if a cybercriminal takes over an employee’s account, they may gain access to the entire history of prompts containing company data.

Risk of a GDPR breach

Pasting the personal data of customers, employees or business partners into public chatbots may constitute a breach of data protection regulations.

Loss of intellectual property protection

Inclusion in a consumer AI tool of:

  • source code,
  • business strategies,
  • technical documentation,
  • product designs,
  • the company’s know-how

may be treated as a disclosure of information to a third party. In practice, this may result in the loss of legal protection or a weakening of the ability to pursue claims against competitors.

What should you never paste into AI chatbots?

  • customer data,
  • PESEL numbers,
  • login details,
  • company documents,
  • source code,
  • financial reports,
  • contracts,
  • medical data,
  • photographs of identity documents.
Magnifying glass with a check mark symbolising verification

2. Lack of fact-checking

AI models do not ‘know’ that something is true. They generate responses based on statistical predictions of subsequent words. In practice, this means that a chatbot may state with complete conviction:

  • non-existent sources,
  • incorrect statistics,
  • false quotations,
  • out-of-date information,
  • or completely fabricated facts.

This phenomenon is often referred to as AI hallucinations.

‘Internet access’ does not solve the problem

Some tools – such as Gemini or Perplexity – are capable of searching the internet before generating a response. However, this does not mean that the information is automatically verified.

AI still:

  • misinterpret sources,
  • use out-of-date websites,
  • rely on unverified content,
  • or combine several pieces of information to reach a false conclusion.

Therefore, any significant information should be verified against a reliable source – particularly if it concerns:

  • the law,
  • finance,
  • medicine,
  • security,
  • business data,
  • or strategic decisions.

An AI chatbot should be treated as a support tool, rather than the definitive source of facts.

Judge’s gavel on a circuit board symbolising legal rules for AI

3. Treating AI as a legal or regulatory adviser

Using AI to make legal, financial or regulatory decisions is a highly risky practice.

Language models do not analyse legislation in the same way as a lawyer. They do not understand legal consequences and do not take responsibility for the implications of their responses. They generate text that sounds credible but may be incorrect or incomplete.

Why is this a problem?

The law depends on context:

  • country,
  • industry,
  • the type of contract,
  • the status of the parties,
  • terms,
  • local regulations,
  • and current legislative changes.

Omitting a single detail can completely alter the interpretation of a situation.

In practice, an incorrect AI response may lead to:

  • a poorly drafted contract,
  • an incorrect response to a complaint,
  • a breach of obligations towards an employee,
  • errors in documentation,
  • or costly legal disputes.

AI can be useful:

  • in preparing a draft,
  • organising information,
  • carrying out preliminary research,
  • or summarising documents.

However, it is no substitute for a lawyer, compliance specialist or tax adviser.

Handshake between a human and a digital hand

4. The thoughtless use of AI in private life

The risks associated with AI do not apply solely to businesses. More and more people are using chatbots in their everyday lives – and often start to treat them as a universal advisor.

This, too, can lead to problems.

Treating AI like a search engine

A chatbot does not always provide up-to-date information. It may generate:

  • non-existent links,
  • incorrect prices,
  • out-of-date opening hours,
  • or incorrect information about products or services.

Sharing personal data

Many people enter the following into chatbots:

  • addresses,
  • phone numbers,
  • details from documents,
  • health information,
  • family problems,
  • work-related conflicts.

This is a very dangerous practice. This data may be stored, analysed or leaked as a result of a security breach.

In particular, you should avoid sending:

  • your PESEL number,
  • photographs of your identity card or passport,
  • test results,
  • medical records,
  • bank details.

Blind trust in health and fitness advice

AI can generate convincing diet and exercise plans, but it does not know:

  • medical history,
  • medicines taken,
  • allergies,
  • test results,
  • health restrictions,
  • or the body’s actual condition.

Advice generated by AI may be incorrect or even dangerous.

Emotional and psychological issues

Increasingly, users treat chatbots as therapists or emotional counsellors. It is worth remembering that AI:

  • does not understand emotions,
  • does not take responsibility for the consequences of its advice,
  • does not diagnose mental health issues,
  • and does not replace a specialist.

The models are designed to be helpful, polite and conflict-averse. This may give the impression of ‘understanding people’, but in reality it is the result of algorithm optimisation, not consciousness or empathy.

 

Summary

AI is a tool – not an oracle

Artificial intelligence can genuinely boost productivity and support day-to-day work. However, this depends on the conscious and responsible use of these tools.

The key principles are simple:

  • do not share confidential data,
  • verify information,
  • do not make critical decisions based solely on AI,
  • and remember that a chatbot generates responses probabilistically – it does not ‘know’ that something is true.

AI works best as a support for humans, not as a replacement for them.

 

Author: Szymon Kwaśniewski, IT Support Specialist, Support and Service, akquinet consulting

 

This text is part of the series: AKQUINET Tech Zone