- Artificial intelligence
- •
When AI develops software: who remains in control?
A glimpse into the year 2038
The year is 2038. AI agents develop software largely independently, designing architectures, writing code and testing their results. What used to take weeks is now completed within a few hours. Productivity has increased enormously.
But then problems arise.
In a bank, automated changes lead to errors in payment transactions. In an industrial group, production lines suddenly come to a standstill. Although AI systems continue to suggest solutions, hardly anyone is able to assess independently whether these actually work.
The problem is not that AI fails. The problem is that people have increasingly lost the ability to verify its results.
Prof. Dr Peter Mandl describes this scenario in his essay on the future, published on evaltech.de, entitled ‘2038 – When software belonged to AI’.
This article is deliberately fictional. Nevertheless, it raises a question that is already relevant to software development today:
How can we make the most of the opportunities offered by artificial intelligence without losing control of our own systems?
AI is changing software development – and our responsibilities
Generative AI can already support developers with a wide range of tasks. It helps them write and analyse code, create tests and document complex systems.
However, as AI becomes more capable, it is not only the tools that are changing, but also the demands placed on professional software development.
After all, a result that works is not necessarily a result that can be trusted. Particularly in complex enterprise environments, software changes must remain traceable, verifiable and maintainable in the long term.
The more tasks AI systems take on, the more important it becomes to consider how their outputs can be checked and integrated into existing processes.
At iSYS, one thing is clear to us:
AI expertise is not just about being able to use powerful models. It also means understanding their limitations, critically evaluating their outputs and retaining responsibility for technical decisions.
When research meets practical challenges
How this oversight can be implemented technically is also the subject of ongoing academic research.
In the paper “LLM Non-Determinism in Deterministic Processes: A Controlled Variation Space for Constraining the Propagation of Non-Determinism”, published in October 2026, Prof. Dr Peter Mandl, together with Paul Mandl (Head of Development @ Findustrial) and Dr Martin Häusl (Head of AI & Data @ iSYS), examines a fundamental challenge in the use of generative AI.
Language models do not necessarily produce the same results every time they are run. Business processes, by contrast, often require clear rules, verifiable states and traceable decisions.
The research paper introduces the concept of Controlled Non-Determinism to address this challenge. Put simply:
AI may generate different approaches and outcomes within a defined framework. However, a separate, rules-based validation process determines whether an outcome is actually accepted.
This makes it possible to harness the flexibility of generative AI without handing over full control of binding process states to the model.
Our approach at iSYS: making AI useful and manageable
What the research paper describes for individual outputs, we pursue at platform level with the iSYS AI-Hub: it is not the model that decides what happens to data, but explicit rules governing data protection requirements, access permissions and the model capabilities needed.
The AI-Hub combines self-hosted models with external AI services and selects an appropriate processing route for each request. Sensitive information stays in-house, while other requests can benefit from the capabilities of external models without company data being shared indiscriminately.
Our research focuses include permission-aware Retrieval-Augmented Generation (RAG), dynamic model selection, and methods for quality assessment and monitoring.
The German Research Allowance Certification Body (Bescheinigungsstelle Forschungszulage, BSFZ) confirmed in September 2026 that this constitutes genuine research and development work: it certified our research project centred on the iSYS AI-Hub as eligible for funding and awarded us the BSFZ seal.
Technological sovereignty starts with our own expertise
The essay describes a future in which companies have highly automated software development capabilities but can barely assess their own systems independently. Technological sovereignty therefore requires more than powerful tools. It depends on people who understand how those tools work, question their outputs and can intervene themselves when necessary.
The essay draws a clear conclusion: universities and companies must continue to train people – and not merely in how to write effective prompts. This is a very practical concern for us. We deliberately designed this year’s hackathon as a learning opportunity: two teams independently tackled the same task drawn from our client work. The underlying question was what changes when AI makes implementation fast, while fully understanding the task becomes the real bottleneck.
Incidentally, the essay itself provides an excellent test of precisely this ability. At the end, it reveals that it was written with the help of AI. Anyone reading it is therefore already confronted with the very question at its heart: the text sounds plausible – but do its assumptions hold up? Being able to answer that question is the skill we want to preserve and pass on.
For us, the future of software development is not about handing over as much responsibility as possible to AI, but about using AI in a way that keeps people in control.
Literature and sources:
evaltech (2026): 2038 – When Software Belonged to AI. Published online in November 2026. Available at: https://evaltech.de/insights/2038-als-die-software-der-ki-gehoerte (accessed 13 November 2026).
Mandl, P. D., Mandl, P. and Häusl, M. (2026): LLM Non-Determinism in Deterministic Processes: A Controlled Variation Space for Constraining the Propagation of Non-Determinism. arXiv preprint, arXiv:2610.06255, published 5 October 2026. DOI: https://doi.org/10.48550/arXiv.2610.06255 (accessed 13 November 2026).
iSYS Software GmbH (2026): iSYS receives BSFZ seal for research and development. Published 16 September 2026. Available at: https://www.isys.de/en/isys-receives-bsfz-seal-of-approval-for-research-and-development/ (accessed 13 November 2026).