Monitoring and Logging: Keeping Apps Healthy

10:00 AM, Tatkal Booking Opens

Every day at 10 AM, the IRCTC website opens Tatkal bookings — emergency train tickets released a day before travel. In the space of a few minutes, lakhs of people across India hit "refresh" and submit their travel details at almost the same instant. Somewhere in a data centre, servers that were comfortably handling a few hundred requests per second suddenly have to handle tens of thousands.

Now imagine you are one of the engineers responsible for keeping that website running. You are not sitting at a train station watching a departure board. You cannot see users, and you cannot see the servers either — they are humming inside racks in a building you may never visit. Yet within seconds of something going wrong — a payment gateway timing out, a database running out of connections, a page taking eight seconds instead of half a second — you need to know. Not after users start tweeting angrily. Not after the app crashes completely. While it is still recoverable.

This is the problem that monitoring and logging solve. They are two of the most important habits in DevOps — the discipline of running software reliably in the real world, not just writing code that works once on a laptop. This chapter builds both ideas from scratch, with real numbers and real code, so that by the end you can explain not just what they are, but exactly how they let engineers "see" a system they cannot physically observe.

Two Different Questions About the Same System

Before writing a single line of code, it helps to separate two questions that people often blur together:

• "What exactly happened, and when?" — this is what logging answers. A log is a detailed, timestamped written record of individual events: a user logged in, a payment failed, a function threw an error. Think of it as a ship's logbook — a captain doesn't just note "the ship is fine"; the logbook records specific events: "14:32 — course changed to 090°," "15:10 — engine room reports vibration in turbine 2." When something goes wrong later, you reread the logbook to reconstruct exactly what happened.
• "Is the system healthy right now?" — this is what monitoring answers. Monitoring continuously measures numeric signals — how many requests per second, how long each one takes, how much memory is being used, how many requests are failing — and tracks them over time, the way a hospital's bedside monitor continuously displays a patient's heart rate and blood oxygen level rather than writing a paragraph about every heartbeat.

Neither one replaces the other. Monitoring is what tells you, at 10:03 AM, "response times just jumped from 200 ms to 2.5 seconds — something is wrong right now." Logging is what you then dig into to find out why — which specific requests were slow, which users were affected, what error message the database returned. Monitoring finds the fire; logs tell you where it started.

Anatomy of a Log Line

Let's write actual logging code and trace exactly what it produces. Most languages have a built-in logging tool rather than relying on scattered print statements, because a proper logging library automatically stamps every line with a timestamp and a severity level. Here is a small Python example simulating a UPI-style payment function:

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s"
)

def process_payment(amount):
    logging.info(f"Payment started: amount=Rs.{amount}")
    if amount <= 0:
        logging.error(f"Invalid amount: {amount}")
        return False
    logging.info("Payment successful")
    return True

process_payment(500)
process_payment(-50)

Trace this line by line. logging.basicConfig sets two things: the minimum level of message to record (INFO and above), and the format each line will be printed in — a timestamp, the level name, then the message. When process_payment(500) runs, amount is 500, so the if condition is false, and we get two INFO lines. When process_payment(-50) runs, the condition is true, so we get one INFO line followed by one ERROR line, and the function returns False before the "Payment successful" line ever executes. The actual output looks like this (timestamps will differ on your machine):

2026-08-13 10:15:32,101 INFO Payment started: amount=Rs.500
2026-08-13 10:15:32,101 INFO Payment successful
2026-08-13 10:15:32,102 INFO Payment started: amount=Rs.-50
2026-08-13 10:15:32,102 ERROR Invalid amount: -50

Notice what this buys you. Six months from now, if a user complains their payment of ₹-50 (a bug in some other part of the app that let a negative amount through) silently failed, an engineer can search the logs for "ERROR" and find this exact line, with the exact timestamp and the exact amount that caused it — without needing to reproduce the bug live.

Log Levels: Not Every Event Deserves the Same Attention

If every single thing a program did were logged at the same importance, the useful lines would be buried under noise. That is why logging libraries define levels, ordered from least to most severe:

• DEBUG — fine-grained detail useful only while actively developing or hunting a bug (e.g., "cache lookup returned key=user_4521").
• INFO — routine events confirming things are working as expected (e.g., "user logged in," "order #8827 created").
• WARNING — something unexpected happened, but the app recovered on its own (e.g., "retrying database connection, attempt 2 of 3").
• ERROR — an operation failed and could not complete (e.g., "payment gateway returned timeout").
• CRITICAL — the whole application or a major part of it is in danger of failing (e.g., "database connection pool exhausted, rejecting new requests").

Each level has an internal severity number (DEBUG=10, INFO=20, WARNING=30, ERROR=40, CRITICAL=50 in Python's logging module), and when you set a minimum level like logging.INFO, only events at that severity or higher get recorded. This is why the code above never printed a DEBUG line — none were written — but if we had added logging.debug("checking amount validity") before the if, it still would not appear, because DEBUG (10) is below the configured minimum of INFO (20). In a production system handling millions of Tatkal requests, engineers usually run at INFO or WARNING level day-to-day, and temporarily drop to DEBUG only while actively chasing a specific bug — otherwise the sheer volume of debug detail would make finding the important lines like searching for a single specific transaction in a warehouse of receipts.

Structured Logs: Making Logs Searchable by Machines, Not Just Eyes

The log lines above are plain text, meant to be read by a human scrolling through a file. That works for small projects, but IRCTC-scale systems generate millions of log lines per hour, spread across hundreds of servers. No human scrolls through that. Instead, modern systems write structured logs — the same information, but as a machine-readable record (commonly JSON) with named fields:

{"timestamp": "2026-08-13T10:15:32", "level": "ERROR", "event": "payment_failed", "user_id": 45231, "amount": -50}

Compare this to the plain-text line "2026-08-13 10:15:32 ERROR Invalid amount: -50". Both carry the same fact, but the structured version lets a log-analysis tool run a precise query like "show me every payment_failed event for user_id=45231 in the last 24 hours" instantly, across every server, without any human reading a single line by eye. This is the difference between a paragraph description and a spreadsheet row — the moment you have thousands of rows, the spreadsheet becomes searchable and the paragraph does not.

From Individual Events to Numbers: What Monitoring Actually Measures

Logs record discrete events. Monitoring instead tracks continuously changing numbers called metrics. The four metrics engineers watch most closely for any web application are:

• Latency — how long a request takes to get a response, usually in milliseconds.
• Error rate — what fraction of requests fail, usually as a percentage.
• Throughput — how many requests the system is handling per second.
• Resource usage — how much CPU, memory, or disk the servers are consuming, usually as a percentage of capacity.

Let's work with real numbers. Suppose a monitoring system records the response time, in milliseconds, of ten consecutive requests to a ticket-booking server during the Tatkal rush:

Request:   1    2    3    4    5    6     7    8    9    10
Time(ms): 180  210  195  220  205  190  2450  215  200  195

The sum of all ten values is 180+210+195+220+205+190+2450+215+200+195 = 4260 ms. Dividing by 10 gives an average response time of 426 ms. On its own, that number might look mildly concerning but not alarming — most booking systems aim to stay under 500 ms. A dashboard showing "average latency: 426 ms" might not trigger any alarm bells at all.

Misconception: "The Average Looks Fine, So the System Is Fine"

This is exactly the trap. Look again at request #7: 2450 ms — more than ten times slower than every other request in that batch. If we remove that single outlier and average the remaining nine values (180+210+195+220+205+190+215+200+195 = 1810 ms), we get 1810 ÷ 9 ≈ 201 ms. One single bad request pulled the overall average from 201 ms up to 426 ms — more than double — while telling you almost nothing about the fact that 90% of users had a perfectly normal experience and one specific user (or one specific type of request, like a request that hit a slow, overloaded part of the database) had a genuinely broken experience.

This is why engineers never rely on the average alone. They also track the maximum response time and how many requests crossed a specific threshold, because a single number like "average latency" can hide a real, painful failure happening to a real user. A useful mental habit: averages answer "how is the typical request doing," but they actively hide rare, severe failures — which is often exactly what you most need to catch.

Thresholds and Alerts: Turning a Number Into an Action

Watching a dashboard 24 hours a day is not realistic, so monitoring systems are configured with thresholds — a limit that, when crossed, automatically triggers an alert (an SMS, an email, a message in a team's chat channel). Here is the logic, written as code, applied to the same ten requests:

def check_and_alert(response_time_ms, threshold_ms=2000):
    if response_time_ms > threshold_ms:
        send_alert(f"Response time {response_time_ms}ms exceeds threshold {threshold_ms}ms")
        return "ALERT"
    return "OK"

response_times = [180, 210, 195, 220, 205, 190, 2450, 215, 200, 195]
for rt in response_times:
    print(rt, check_and_alert(rt))

Tracing this loop: for each of the nine normal values, response_time_ms > threshold_ms evaluates to False (since none exceed 2000), so the function returns "OK" and no alert fires. Only when rt equals 2450 does the condition become True: send_alert(...) runs, and "ALERT" is returned. Out of ten requests, exactly one line of output reads 2450 ALERT — everything else reads OK. This is precisely how an engineer's phone can buzz within seconds of request #7 happening, even though the average across all ten requests looked unremarkable.

The diagram below shows this same data plotted as a graph, which is how monitoring dashboards actually display it — a moving line with a threshold marked, so a spike is visible at a glance rather than requiring anyone to read through raw numbers.

Uptime: The Metric Behind Every "99.9% Guarantee"

Besides latency and errors, one more metric matters enormously in DevOps: uptime — the percentage of time a service was actually available and working, out of all the time it was supposed to be available. Suppose a booking website was down for maintenance and an unexpected crash for a total of 45 minutes during a 30-day month. A 30-day month has 30 × 24 × 60 = 43,200 minutes. Uptime is calculated as:

Uptime % = (Total minutes - Downtime minutes) / Total minutes * 100
         = (43200 - 45) / 43200 * 100
         = 43155 / 43200 * 100
         ≈ 99.90%

That 99.90% sounds impressively close to perfect, but it is worth translating percentages into minutes, because the gap between "good" and "excellent" uptime is much larger than it first appears. A service advertising "three nines" (99.9%) is allowed roughly 43 minutes of downtime per month. A service advertising "five nines" (99.999%) — the standard demanded of systems like payment networks or emergency services — is allowed only about 5 minutes of downtime per year (0.001% of the 525,600 minutes in a year is about 5.26 minutes). Each additional "nine" is not a small improvement; it is roughly a tenfold tightening of how little failure is tolerated, and reaching it usually requires monitoring sensitive enough to detect and react to problems in seconds, not minutes.

Misconception: "If the Server Process Is Running, the App Is Healthy"

A second common misconception is confusing "the server is up" with "the application is working." Imagine the web server process is running perfectly — it is accepting connections and responding to requests — but its connection to the database has silently frozen. The server will happily reply to every request with an error page, or simply hang forever waiting for a database response that never comes. If your monitoring only checks "is the process alive?" (called a liveness check), it will report everything as green while real users see nothing but errors.

This is why production systems use a more thorough health check: a special endpoint, often at a URL like /health, that the application itself runs whenever it's asked, and which actively tests the things that could be broken — "can I reach the database? can I reach the payment gateway? is my internal queue backed up?" — before replying "healthy" or "unhealthy." Monitoring calls this endpoint every few seconds. A process that is technically running but cannot actually serve users correctly will fail this check and get flagged, which is very different from just checking whether the operating system still lists the process.

Where Do All These Logs Go?

A single busy server can generate gigabytes of log data per day. Left unmanaged, log files would eventually fill up the entire disk and crash the very server they were meant to help debug — an ironic failure logging itself would have to record. Production systems handle this with log rotation: instead of one endlessly growing file, the system automatically closes the current log file at a fixed interval (say, once a day), compresses it, and starts a fresh one. Older compressed logs are kept for a defined retention period — for example, 30 days for routine logs, potentially much longer for logs that have compliance or security value — after which they are automatically deleted. This keeps storage costs and search time under control while still preserving enough history to investigate most real problems, which are almost always noticed and reported within days, not months.

Putting the Pipeline Together

Step back and look at the full picture. An application, as it runs, produces two parallel streams of information: structured log events for every significant thing that happens, and numeric metrics sampled continuously (response time, error count, CPU usage). Both streams are shipped off the server itself to a separate monitoring and logging system, so that even if the original server crashes completely, the evidence of what led up to the crash survives on a different machine. That system plots metrics on dashboards, evaluates them against configured thresholds, and — the moment a threshold is crossed — fires an alert to the on-call engineer, who then searches the structured logs for the exact events that explain what went wrong. None of this requires a human to be staring at a screen at the exact moment of the Tatkal rush; the system watches itself and only interrupts a human when something has actually crossed a line worth acting on.

Practice: Test Your Understanding

1. A monitoring system records these five response times, in milliseconds, for a UPI payment API: 120, 135, 4200, 128, 140. Calculate the average with all five values, then recalculate it excluding the outlier. By how many milliseconds does removing the outlier change the average?
2. Using an alert threshold of 1000 ms, write out, request by request, which of the five values above would trigger "ALERT" and which would return "OK", following the same logic as the check_and_alert function in this chapter.
3. A logging system is configured with level=logging.WARNING. If the code calls logging.info("cache refreshed") and then logging.error("cache refresh failed"), which of these two lines will actually appear in the log output, and why?
4. A ticketing service was down for 12 minutes during a single 7-day week. A week has 7 × 24 × 60 = 10,080 minutes. Calculate its uptime percentage for that week to two decimal places.
5. Explain, in your own words, why a monitoring system that only checks "is the server process running" can still report a service as healthy while every real user is seeing errors. What would you check instead?
6. A teammate says, "Our average latency dropped from 300 ms to 250 ms this week, so nothing needs our attention." Using an idea from this chapter, explain why this conclusion could be wrong even if the average genuinely improved.

Summary

Monitoring and logging exist because engineers cannot physically watch every server during an event like the Tatkal rush, so systems are built to watch themselves. Logging records discrete, timestamped events — what happened and when — at graded severity levels (DEBUG through CRITICAL), increasingly written in structured, machine-searchable formats like JSON rather than plain text. Monitoring instead tracks continuous numeric metrics — latency, error rate, throughput, resource usage, and uptime — and compares them against configured thresholds, automatically firing alerts the instant a limit is crossed, exactly as traced through the check_and_alert function in this chapter. The most important habit this chapter builds is distrust of single summary numbers: an average response time of 426 ms sounds mild, but hid a single request that took over ten times longer than normal; a running server process sounds healthy, but can be silently failing every real request behind the scenes. Good monitoring is built specifically to catch what an average, or a simple "is it on?" check, would otherwise let slip past unnoticed.