Six Choices Every AI Engineer Has to Make (and Nobody Teaches)

teach you how to make a model accurate. They rarely teach you the decisionsthat come right after.

How do you know when to fully automatesomething versus keeping a human in the loop?

When does promptingstop being enough and fine-tuning become worth the cost? What does it actually mean to pick real-time inference over batch when the bill arrives?

These questions don’t show up in coursework. They show up your first week in production!

This article walks through 6 trade-offs that show up in productionAI work. All backed by the latest research, so you get a glimpse into how people are dealing with these common trade-offs.

There are no right answers here. There are useful frames, real numbers, and the kind of context that makes the next decision faster.

Index

1. Build vs. Buy in the LLM Era (When calling an API stops making sense)
2. Model Complexity vs. Maintainability (Who debugs this in 6 months?)
3. Data Quantity vs. Data Quality (More data isn’t always the answer)
4. Throughput vs. Latency (Batch or real-time)
5. Prompt Engineering vs. Fine-Tuning (Two very different investment curves)
6. Automation vs. Human Oversight (How much do you trust the model to act alone?)

• * *

Hey there! My name is Sara Nóbrega and I teach you how to become an AI power user on**Learn AI**. Free to subscribe!

• * *

1. Build vs. Buy in the LLM Era

#### When calling an API stops making sense

The oldversion of this question was: do we train our own model? That one is mostly settled. Almost nobody trains from scratch anymore.

The 2026version is harder.

You have 3 optionsnow: call an API, fine-tune an open-source model, or build and host your own stack. Each one has very different cost curves and very different failure modes.

Image created with DALL-E.

A 2025 Omdia survey of 376 technical and business stakeholders found that 95% agreed building gives more customization and control

The same survey found 91% agreed prebuilt platforms ship faster. Both numbers are true at the same time, which is the problem.

Where it gets concrete is at scale. Below 100k daily requests, calling an API like GPT-4o Mini is usually the right call. Low overhead. Fast iteration. Above 1M daily requests, per-token costs start eating margin [2].

Here is the part teams undervalue. A 2024 analysis found that hardware and electricity make up only 20 to 30% of self-hosting cost. Staff is the other 70 to 80% [2]. These means that most build-vs-buy spreadsheets account for the GPUs and forget the engineers.

Another study found teams exceeded their LLM cost budgetsby 340% on average. In most cases the cause was missing per-tenant usage tracking and missing query-level cost attribution, not the per-token rate itself [3].

Teams couldn’t see which feature or prompt was burning the budget, so they couldn’t fix it.

Framework lock-in shows up later and shows up hard. Hugging Face’s Text Generation Inference went into maintenance mode in late 2025, and teams who built on it had to migrate. Teams who used an API didn’t have to do anything.

The practical frame I use:

• Start with the API.
• Instrument every call with cost, latency, and feature attribution from day 1.
• Switch when the math forces you to.

2. Model Complexity vs. Maintainability

#### Who debugs this in 6 months?

A famous Google paper introduced the CACE principle: Changing Anything Changes Everything [4].

In MLsystems, a small tweak in one part of the pipeline can trigger surprising changes elsewhere. This rarely happens with a linear regression. It happens often with ensembles and neural nets.

Research on ML technical debt shows that data dependencyis more expensive than code dependency [4].

Image created with DALL-E.

Why? Because data is harder to track, harder to version, and harder to explain to whoever inherits the system 6 months from now.

The original paper estimated that the actual model code is a small fraction of a real-world ML system. The bulkis feature stores, pipeline logic, monitoring, retraining triggers, and the glue between all of them [5].

In practice, teams pick a more complexmodel for a 2% accuracy gain and pay for that choice for 18 months in debugging time, retraining overhead, and the “nobody remembers why we did this” tax.

The question to ask beforeshipping a complex model is: who owns this in a year? If the honest answer is “unclear,” that is the decision point.

• * *

Learn how to give your fav AIunlimited updated context: Give Your AI Unlimited Updated Context | Towards Data Science

• * *

3. Data Quantity vs. Data Quality

#### More data isn’t always the answer

Moredata wins for foundation models trained on internet-scale corpora. In applied ML, the relationship breaks down much sooner.

Research shows that beyond a noisethreshold, adding more low-quality data flattens or degrades model performance [6].

This means that the relationship between sample size and accuracy breaks down once noise crosses a certain level!

Image created with DALL-E.

The “data swamp” problem is what this looks like at companies. Teams collecteverything because storage is cheap and they assume it will be useful one day.

Without governance, you get a pool that takes weeks to clean, raises storage and pipeline costs, and slows experimentation without improving outcomes [7].

Medical AI is the clearest case. Smalldatasets with expert-verified labels have repeatedly outperformed larger datasets with unreliable annotations. The model learned the right patterns from less data because the signal was clean.

The question I find more useful in practice:

how noisy is what we have, and what does 1 more hour of cleaning buy us versus 1 more day of collection?

4. Throughput vs. Latency: Batch or Real-Time

#### Batch or real-time

Batch and real-time inference are 2 different system architectures. Picking the wrong one cascades into infrastructure, cost, and user experience choices that are hard to reverse later.

Batch inference: predictions generated on a schedule (hourly, daily), stored in a database, served from there. Lower cost. Simpler infrastructure and easier to debug. Predictions can be stale.

Real-time inference: predictions on demand, in milliseconds to seconds. Always current and more expensive (24/7 uptime). More moving parts and harder to monitor [8].

Image created with DALL-E.

The tensionat the system level is the fact that bigger batch sizes give higher throughput but higher latency per request. Real-time systems use batch size 1, which gives speed but can lose efficiency.

The mistakeI see most is teams defaulting to real-time because it sounds more impressive.

But most business problems do not need sub-second predictions!

Nightly churn scores, weekly recommendation refreshes, daily fraud-model updates. These are batch problems being over-engineered as real-time ones, and the cost difference at scale is significant.

Practical signal: if your users won’t notice whether the prediction is 5 minutes old or 5 milliseconds old, use batch inference instead of real-time.

5. Prompt Engineering vs. Fine-Tuning

#### Two very different investment curves

Image created with DALL-E.

The decision logic here got cleaner over the last months.

Prompt engineering is fast, cheap, and flexible. It can take hours to days to iterate and it works well for most tasks, especially with capable frontier models.

The downside is fragility because small input changes produce inconsistent outputs, and long prompts with complex formatting rules tend to break under edge cases.

Fine-tuning is expensive upfront in compute, data preparation, and engineering time. It is reliable and consistent at scale once the work is done.

A real example I’ve seen quoted: fine-tuning GPT-4o for a customer support chatbot ran roughly $10k in compute and 6 weeks of data prep [9]. The RAG alternative shipped in 2 weeks.

My opinion on current practitioner guidance: start with prompts.

Escalate to fine-tuningonly when you hit failure modes that prompting can’t fix. Below 100k queries, prompting is almost always the right call. It has been shown that fine-tuning pays off at high volumewhen the task is stable and well-defined [10].

A 2025 analysis found that prompt optimizationwith tools like DSPy beat fine-tuning by 6 to 19 points on some benchmarks, using 35x fewer rollouts [10].

It seems that the gap is closingyear over year. Fine-tuning has become a last step in most stacks I see, used after prompting has clearly hit its ceiling.

The hybrid pattern is increasingly common in production: a model fine-tuned on domain style and tone, combined with RAG for factual grounding. The two techniques solve different problems.

6. Automation vs. Human Oversight

#### How much do you trust the model to act alone?

Image created with DALL-E.

The useful question in production is: what is the costof a wrong decision, and whoabsorbs it?

Human-in-the-loop (HITL) sits on a spectrum.

At one end, humans review every AI output before it acts. At the other, full automationwith humans only watching for anomalies.

Most production systems sit somewhere between, routing low-confidence predictions to humans and letting high-confidence ones through [11].

But the operational cost of HITL is real: reviewing every model decision does not scale!

The truth is that real-time human intervention slows the system and reviewer inconsistency degrades label quality.

The working pattern is selectiveHITL: human review is triggered only for edgecases, low-confidence outputs, and high-stakes decisions.

In healthcare, finance, and legal, HITL is often a compliance requirement. A radiologist reviewing AI-flagged tumors or a lawyer reviewing AI-flagged contract clauses. These are the cases where the cost of an error is too high to fully automate.

A way to think about the split:

• AI handles volume, speed, and pattern recognition.
• Humans handle irreversibility.

The designquestion is where exactly that line sits in your specific workflow, and whether the humans in the loop have clear authority to override the model when they disagree.

What to Take Away

If I had to compress the 6 trade-offs into one principle, it would be this: in production, the cost of a decision is rarely paid where the decision is made.

A more complex model costs you in maintenance 6 months later. A real-time system costs you in 24/7 infra forever.

Dirty data at scale costs you in retraining cycles. A clever prompt costs you in fragility under edge cases. And full automation costs you when something irreversible goes wrong!

The hard part is knowing where the cost actually lands, and asking the right question early enough to act on it.

Thanks for reading!

References

[1] Omdia, _Navigating Build-Vs.-Buy Dynamics for Enterprise-Ready AI_ (2025).

Source: https://www.techtarget.com/searchenterpriseai/tip/LLM-build-vs-buy-A-decision-framework-for-LLM-adoption

[2] Ptolemay, _LLM Total Cost of Ownership 2025: Build vs Buy Math_ (2025).

Source: https://www.ptolemay.com/post/llm-total-cost-of-ownership

[3] TianPan, _The Build-vs-Buy LLM Infrastructure Decision Most Teams Get Wrong_ (2026).

Source: https://tianpan.co/blog/2026-04-15-build-vs-buy-llm-infrastructure

[4] D. Sculley et al., _Hidden Technical Debt in Machine Learning Systems_ (2015), NeurIPS.

Source: https://lathashreeh.medium.com/hidden-technical-debt-in-machine-learning-systems-27fa1b13040c

[5] CMU MLIP, _Technical Debt — Machine Learning in Production_ (2024).

Source: https://mlip-cmu.github.io/book/22-technical-debt.html

[6] Z. Qi et al., _Impacts of Dirty Data: an Experimental Evaluation_ (2018).

Source: https://arxiv.org/pdf/1803.06071

[7] S. Sigari, _Striking the Balance Between Data Quality and Quantity in Machine Learning_ (2023).

Source: https://medium.com/@sigari.salman/striking-the-balance-between-data-quality-and-quantity-in-machine-learning-1f935a89f59b

[8] C. Zhou, _Batch Inference vs. Real-Time Inference: What, When, and Why_ (2025).

Source: https://medium.com/@conniezhou678/be-a-better-machine-learning-engineer-part-1-batch-inference-vs-0857587bf39a

[9] S. Jolfaei, _Fine-Tuning vs RAG vs Prompt Engineering: When to Use What_ (2025).

Source: https://medium.com/@sa.aghadavood/fine-tuning-vs-rag-vs-prompt-engineering-when-to-use-what-b288340e33aa

[10] LLM Stats, _Is Fine-Tuning Better Than Prompt Engineering in 2026?_ (2026).

Source: https://llm-stats.com/blog/research/fine-tuning-vs-prompt-engineering-2026

[11] A. Masood, _Operationalizing Trust: Human-in-the-Loop AI at Enterprise Scale_ (2025).

Source: https://medium.com/@adnanmasood/operationalizing-trust-human-in-the-loop-ai-at-enterprise-scale-a0f2f9e0b26e