07 - Cloud and Artificial Intelligence

Cloud and Artificial Intelligence: Where Data and Models Live, and What It Costs to Keep Them There

Machine Learning Engineer course · Module 00 · Data-driven mindset · Lesson 7

Goal. Give the reader a precise definition of the cloud (the normative one, not the marketing one), the three service families with their split of responsibility, and two calculations for reasoning about costs: elasticity versus fixed capacity, and the break-even point between owning and renting hardware (typically GPUs). By the end the reader can decide where to run a model and why.

False myth. "The cloud costs less." Sometimes yes, sometimes no: it depends on utilisation. The code of this lesson shows it with numbers.

Introduction

Machine learning models have three needs: data (to be stored), computation (often GPUs, for training) and availability (to serve predictions to whoever asks). The cloud is a way of meeting these needs by paying per use instead of buying in advance. It is not the only way, and not always the best.

Roadmap. (1) What it is, with the NIST definition. (2) IaaS, PaaS, SaaS: who manages what. (3) Elasticity: the first calculation. (4) Buy or rent: the second calculation. (5) Cloud and personal data. (6) Business intelligence and analytical data. (7) Where, for an ML engineer, the cloud really helps.

What it is, why it exists, when NOT to use it

The definition that counts

The NIST document SP 800-145 (2011) defines the cloud as a model for enabling network access, on demand, to a shared pool of configurable computing resources (networks, servers, storage, applications, services) that can be rapidly provisioned and released with minimal management effort. It characterises it with five essential characteristics:

1. On-demand self-service: you obtain resources by yourself, without phoning anybody.
2. Broad network access: you reach it over the network, from different devices.
3. Resource pooling: the provider serves many customers from the same pool of resources (multi-tenancy).
4. Rapid elasticity: resources expand and shrink quickly, often automatically.
5. Measured service: usage is measured, and you pay for that.

The five characteristics describe the relationship between customer and provider more than the technology: the first two say how you obtain the resources, the last three how the provider manages them and bills you for them. If metering or self-service is missing, the NIST document does not consider it cloud, and the practical test in the next paragraph is its translation.

Then three service models (SaaS, PaaS, IaaS) and four deployment models (private, community, public, hybrid).

Practical test: if a service has no usage metering and cannot be obtained by yourself on demand, it is probably "hosting" with a cloud label.

Analogy: the house

The higher you go, the less responsibility you have and the less freedom. It is not a quality scale: it is a shift of the boundary.

When NOT to use the public cloud

Preparation

pip install numpy

The prices in the code are invented: they serve to show the method. Before deciding, insert the real prices of the provider you are evaluating, which change over time.

IaaS, PaaS, SaaS and shared responsibility

The shared responsibility model formalises the split: the provider is responsible for the security of the cloud (buildings, hardware, virtualisation); you are responsible for the security in the cloud (configuration, access, data). Where the line falls depends on the service model: in IaaS you also manage the operating system; in SaaS almost nothing. A storage bucket left public by mistake is, by construction, a customer responsibility. Specific details are in the provider's documentation, which must be read.

For machine learning

First calculation: elasticity

The idea

The demand for a service varies over time (evening peaks, working days, campaigns). With fixed capacity you must size for the peak, and for the rest of the time much capacity stays unused. With elastic capacity you pay at each instant roughly what you need. Armbrust et al. (2010) describe it as the transfer of the risk of over- and under-provisioning from the company to the provider.

Analogy: the restaurant. A restaurant with 80 seats available for the Sunday evening peak and 20 customers on Tuesday lunch pays the same rent every day. If it could make tables appear and disappear, it would pay only for the occupied ones.

Code

We generate the hourly demand for a week (a base, an evening peak, noise) and compare the costs. The price per unit of capacity and hour (0.10) is invented.

import numpy as np

rng = np.random.default_rng(5)
ore = np.arange(24 * 7)
ora_del_giorno = ore % 24
domanda = 20 + 60 * np.exp(-((ora_del_giorno - 20) ** 2) / 8) + rng.normal(0, 3, ore.size)
domanda = np.clip(domanda, 5, None)

capacita_picco = np.ceil(domanda.max())
prezzo_ora_unita = 0.10                     # INVENTED: replace with the real price

costo_fisso = capacita_picco * len(ore) * prezzo_ora_unita
costo_elastico = np.ceil(domanda).sum() * prezzo_ora_unita
utilizzo_fisso = domanda.sum() / (capacita_picco * len(ore))

print("peak", capacita_picco, "| mean", round(float(domanda.mean()), 1))
print("fixed cost", round(float(costo_fisso), 1), "| elastic", round(float(costo_elastico), 1))
print("utilisation with fixed capacity", round(float(utilizzo_fisso), 3))

# Output (Python 3.14.4, NumPy 2.3.5, seed 5, two identical runs):
# peak 82.0 | mean 31.7
# fixed cost 1377.6 | elastic 541.3
# utilisation with fixed capacity 0.387

How to read the code. ore counts the 168 hours of a week and ora_del_giorno = ore % 24 brings them back to the hour of the day (0-23). domanda is the sum of three ingredients: a base of 20 units, a bell-shaped evening peak centred at 20 (exp(-((ora - 20)**2)/8) is 1 at 20 and fades in a few hours), and random noise with standard deviation 3; np.clip(..., 5, None) avoids values that are too low. capacita_picco is the rounding up of the maximum: with fixed capacity you must buy the peak, always. costo_fisso multiplies that capacity by the 168 hours and by the price; costo_elastico instead pays, hour by hour, only what is needed (np.ceil(domanda).sum()). The utilizzo (utilisation) is the ratio between the capacity really used and the capacity paid for: if it is 0.39, 61% of what you pay stays idle.

Output obtained (seed 5): peak 82, mean demand 31.7; fixed cost 1377.6, elastic 541.3; utilisation with fixed capacity 0.387.

How to read it. With fixed capacity you use 39% of what you pay for. Elasticity would reduce the cost by 61% in this scenario. The result depends entirely on the shape of the demand: the more the peak exceeds the mean, the more elasticity pays off. A flat demand would give an almost null advantage.

What the model simplifies: it assumes that capacity adapts instantly and that the elastic unit price is equal to the fixed one. In reality elastic capacity often has a higher unit price and minimum billing granularity; and adaptation has latency. The calculation serves to understand the order of magnitude, not to predict the invoice.

A second, contrasting example: when elasticity does not pay off

The advantage of elasticity depends on the shape of the demand and on the unit price. We compare two demands, one with an evening peak and one flat, and two price hypotheses: equal or with a 40% surcharge for elastic capacity (more realistic: flexibility usually costs).

import numpy as np

def domanda(forma, seme=5):
    rng = np.random.default_rng(seme)
    h = np.arange(24 * 7) % 24
    if forma == "with evening peak":
        d = 20 + 60 * np.exp(-((h - 20) ** 2) / 8) + rng.normal(0, 3, h.size)
    else:
        d = 20 + rng.normal(0, 3, h.size)
    return np.clip(d, 5, None)

prezzo = 0.10
print(f"{'demand':<18} {'peak/mean':>11} {'fixed':>8} {'elastic at same price':>24} {'elastic +40%':>15}")
for forma in ("with evening peak", "flat"):
    d = domanda(forma)
    fisso = np.ceil(d.max()) * d.size * prezzo
    elastico = np.ceil(d).sum() * prezzo
    print(f"{forma:<18} {d.max() / d.mean():>11.2f} {fisso:>8.1f} {elastico:>24.1f} {elastico * 1.4:>15.1f}")

Output obtained:

demand               peak/mean    fixed    elastic at same price    elastic +40%
with evening peak         2.58   1377.6                    541.3           757.8
flat                      1.39    470.4                    339.2           474.9

How to read the code. domanda generates two profiles with the same seed: one with the evening peak, the other flat (20 plus noise). For each we compute the peak/mean ratio, the cost with fixed capacity, the elastic cost at the same price and the elastic cost with a 40% surcharge. The reasoning is simple and can be derived: with fixed capacity you pay peak × hours × price; with elastic capacity you pay mean × hours × price × (1 + s), where s is the surcharge. Elasticity pays off when mean × (1 + s) < peak, that is when

peak / mean > 1 + s.

With the evening-peak demand the ratio is about 2.6: it pays off even with a 40% surcharge (1.4). With the flat demand the ratio is about 1.4 (1.39): at the same price elasticity saves (339 versus 470), but with +40% the elastic cost (475) exceeds the fixed one (470): it no longer pays off. The conclusion for the engineer is operational: before choosing an elastic architecture, measure the peak/mean ratio of your real demand.

Second calculation: buy or rent

The problem

You need a GPU to train models. You can rent it by the hour (rate r per hour) or own it, with a monthly cost O that includes depreciation (purchase price divided by the months of useful life), energy and maintenance. For how long per month must you use it for owning to pay off?

Derivation

Monthly cost renting, with h hours of use: C_rent(h) = r · h. Monthly cost owning: C_own = O (constant: you pay it even if you do not use it).

Break-even: r · h* = O, so

h* = O / r

If you use the machine for more than h* hours a month, owning costs less; if less, renting costs less. The fraction of the month at break-even is h* / 730 (an average month has about 730 hours, 8760/12).

costo_mensile_proprio = 900.0      # INVENTED: depreciation + energy + maintenance
tariffa_nolo_ora = 2.0             # INVENTED

h_pareggio = costo_mensile_proprio / tariffa_nolo_ora
print("break-even hours per month:", h_pareggio)
print("fraction of the month:", round(h_pareggio / 730, 3))

# Output:
# break-even hours per month: 450.0
# fraction of the month: 0.616

Output: break-even hours per month: 450.0, fraction 0.616. With these invented numbers, if the GPU works less than 62% of the time, renting pays off.

The number that counts is called utilisation. Hardly anyone trains models 24 hours a day. A training is a peak: intense hours or days, then nothing. For this reason, at low utilisation, renting almost always wins. At high and constant utilisation (an online prediction service that always runs) the calculation reverses.

The cost per hour actually used

For the break-even between buying and renting there is a more intuitive way to read the result: the cost per hour of actual use. If you own the machine and use it for h hours a month, each hour costs you O/h (the fixed cost divided by the hours used); renting it costs you always r. Owning wins when O/h < r.

import math

print(f"{'O (€/month)':>11} {'r (€/hour)':>10} {'break-even hours':>16} {'fraction of month':>18}")
for O in (300, 600, 900, 1500):
    for r in (1.0, 2.0, 4.0):
        h_star = O / r
        print(f"{O:>11} {r:>10.1f} {h_star:>16.0f} {h_star / 730:>18.1%}")

print()
O, r = 900, 2.0
for ore_uso in (50, 150, 300, 450, 600, 730):
    print(f"usage {ore_uso:>3} hours/month: own {O / ore_uso:6.2f} €/hour used | rental {r:.2f} €/hour")

Output obtained:

O (€/month) r (€/hour) break-even hours  fraction of month
        300        1.0              300              41.1%
        300        2.0              150              20.5%
        300        4.0               75              10.3%
        600        1.0              600              82.2%
        600        2.0              300              41.1%
        600        4.0              150              20.5%
        900        1.0              900             123.3%
        900        2.0              450              61.6%
        900        4.0              225              30.8%
       1500        1.0             1500             205.5%
       1500        2.0              750             102.7%
       1500        4.0              375              51.4%

usage  50 hours/month: own  18.00 €/hour used | rental 2.00 €/hour
usage 150 hours/month: own   6.00 €/hour used | rental 2.00 €/hour
usage 300 hours/month: own   3.00 €/hour used | rental 2.00 €/hour
usage 450 hours/month: own   2.00 €/hour used | rental 2.00 €/hour
usage 600 hours/month: own   1.50 €/hour used | rental 2.00 €/hour
usage 730 hours/month: own   1.23 €/hour used | rental 2.00 €/hour

How to read the code. The first table shows how the break-even h* = O/r changes as the two parameters vary: if the machine costs double (at the same rate) you need double the hours to amortise it; if the rental costs double, half are enough. With the invented numbers O = 900 and r = 2 the break-even is at 450 hours, 61.6% of the month. The second part shows the same fact from another angle: at 50 hours of use the owned machine costs 18 euros for every hour used, nine times the rental; at 730 hours (always on) it costs 1.23 euros per hour, less than the rental. The fixed cost is "amortised" over the hours of use: the more you use, the less each hour costs you. It is the same logic for which an annual subscription pays off for those who use the service a lot.

What the model ignores (and you must add): the cost of whoever manages the hardware; the risk of obsolescence; data transfer costs; discounts for long-term commitments; the actual availability of GPUs at the moment you need them.

Overall trade-off.

Putting the pieces together: a small decision matrix

The two calculations are not enough on their own: the choice of where to make an ML system live combines cost, constraints on data and management capacity. An orderly way to decide is to write a matrix with the options on the rows and the criteria on the columns, and fill it in writing:

There is no "winning" column: the answer changes with your case. The usefulness of the matrix is that it forces you to write down the criteria and the weight they have for you, so that the decision can be discussed and revised (if utilisation changes in two years, you reopen the matrix, you do not start from zero).

What is in a cloud bill

Whoever sets up an ML project should be able to distinguish the four chapters of cost, because they are controlled in different ways. Compute: hours of virtual machines or GPUs; governed with automatic shutdown and right-sizing. Storage: bytes kept over time; governed with expiry rules (deleting temporary data) and with cheaper storage classes for what is rarely read. Egress transfer: data leaving the provider, often paid; governed by computing close to the data. Managed services: databases, queues, functions; governed by reading how they are billed (per request, per hour, per volume). A good habit is to assign an owner and an alarm to each chapter: the surprise in the bill usually comes from the chapter nobody felt responsible for.

Cloud and personal data

Putting data in a provider's cloud does not remove responsibility: under the GDPR the data controller remains you. Points to verify, documenting the answers:

• Where the data physically reside and where they can be processed (chapter V of the GDPR on transfers to third countries).
• Who is the processor and whether a written agreement with the provider exists.
• Encryption at rest and in transit, and who holds the keys.
• Deletion: how it is truly obtained, including from backup copies.
• Access logging: who read what.

For models there is one more choice: sending the data to a service hosted by third parties (convenient, but the data leave your perimeter) or running the model in your own infrastructure (more work, total control). For sensitive data the second option is often the one that makes the compliance assessment simpler. It is not a rule: it depends on the contract, the type of data and your resources.

Business intelligence and data analysis

Two families of systems, with opposite tasks:

A frequent mistake is to run heavy analyses on the transactional database in production: it slows down users. The standard solution is to copy the data to a separate analytical system. Business intelligence (dashboards, reports) lives on the latter. Machine learning usually draws from there, not from the operational database.

A minimal vocabulary

• Cloud: on-demand, measured and shared access to computing resources.
• IaaS / PaaS / SaaS: infrastructure, platform, software as a service.
• Shared responsibility: the provider protects the cloud, you protect what you put in it.
• Elasticity: capacity that follows demand.
• Utilisation: the share of paid capacity actually used.
• Break-even: the point at which two options cost the same (h* = O/r).
• Lock-in: difficulty of changing provider.
• OLTP / OLAP: systems for recording operations / for analysing lots of historical data.

The eight entries fall into three themes: what the cloud is (cloud, IaaS/PaaS/SaaS, shared responsibility), how to reason about costs (elasticity, utilisation, break-even, lock-in) and where the data are drawn from (OLTP/OLAP). In an architecture discussion, the most useful word to demand is "utilisation": without that number no cost comparison is valid.

Typical mistakes

Best practices

• Tag every resource (project, owner, expiry).
• Set spending limits and alarms before launching the first workload.
• Infrastructure as code: describe the environment in versioned files (module 11).
• Principle of least privilege in access.
• Separate environments (development, test, production).
• Do the utilisation calculation before committing to long-term contracts.

The first three rules prevent billing and configuration surprises, the fourth and fifth those of security, the last those of contracts. If you start from zero, set the spending alarms: they are the control that costs least and that has prevented the most incidents in the cases the lesson cites.

Mini-project

1. Modify the elasticity code: replace the evening peak with a flat demand (20 + noise). Measure the advantage of elasticity. What does it show?
2. Add a 40% surcharge for the elastic unit (prezzo_elastico = 1.4 * prezzo_ora_unita). With what peak/mean ratio does elasticity stop paying off?
3. For the break-even h*, produce a table with O ∈ {300, 600, 900, 1500} and r ∈ {1, 2, 4}.
4. Write five lines of recommendation for a case of your choice (monthly training of 40 hours; always-on service; irregular experimentation).

The four points use the lesson's code as a laboratory: the first two repeat the elasticity with a flat demand and with the surcharge, the third the break-even, the fourth forces you to turn the numbers into a written recommendation. The fourth is the real product of the work.

Challenge

• Easy. Classify into IaaS / PaaS / SaaS five services you use.
• Medium. Add to the break-even the cost of a half-time person who manages the owned hardware. How does h* shift?
• Hard. Model the demand with weekly and seasonal peaks; compute the advantage of elasticity with hourly versus per-minute billing granularity.
• Expert. Add the commitment discount (you pay 80% of the rate if you commit for the whole month) and find the new break-even between hourly rental, committed rental and ownership.
• Enterprise. Write a design record for an ML system of your organisation: where data, training and serving live, with economic and compliance reasons.

Easy and Medium take ten minutes with the code provided, Hard and Expert extend the models with granularity and discounts, Enterprise brings them into your context. Expert is the most useful: it adds the commitment discount, which in real price lists shifts the break-even more than any other parameter.

Recap

Mind map. Cloud = five NIST characteristics. Three services (IaaS/PaaS/SaaS) = three responsibility boundaries. Elasticity wins when peak ≫ mean. Break-even h* = O/r. Personal data: where, who, encryption, deletion.

3 concepts: shared responsibility; elasticity; utilisation. 3 mistakes: forgotten resources; data public by mistake; comparison without utilisation. 3 best practices: tags and spending alarms; least privilege; infrastructure as code. 1 exercise: the mini-project of this lesson.

End of module 00. You have the way of thinking: what AI is, which risks to measure, how to translate a problem into a decision, how to experiment, where to run. Next module (01): the tools, that is Python for data.

The same series also includes: 06 - Test and Learn: Controlled Experiments and A/B Tests (the previous lesson), 08 - Python for Data 1: Environment and Fundamentals (the next lesson).

Topic left out in this lesson: blockchain, present in the syllabi of many introductory courses. It is not a prerequisite of machine learning and will not be used in the following modules; I do not cover it so as not to dilute the path.