Logging and Monitoring: System Health Visibility

Logging and Monitoring: System Health Visibility

Logging records application events (errors, warnings, user actions). Monitoring tracks system metrics (CPU, memory, response time). Together they provide visibility: what happened (logging) and how system performs (monitoring). Without them, production issues are blind guesses. With them, every issue has clear cause and context.

Logging Levels

DEBUG: Detailed info for development. Variable values, function calls. Only in development. INFO: General informational messages. User login, request started. Confirms normal operation. WARNING: Unexpected but non-critical issues. High response time, deprecated API usage. Requires investigation. ERROR: Recoverable errors. Failed database query with retry, invalid user input. System continues. CRITICAL: System failure, immediate action needed. Cannot connect to database, out of memory. System likely down.

Python Logging

import logging; logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'). Use in code: logger = logging.getLogger(__name__); logger.info('User logged in'); logger.error('Database error', exc_info=True). exc_info=True includes traceback. Output to file: handler = logging.FileHandler('app.log'); logger.addHandler(handler). Rotate logs: from logging.handlers import RotatingFileHandler; handler = RotatingFileHandler('app.log', maxBytes=10MB, backupCount=5). Keeps only last 5 files, preventing disk space issues.

Structured Logging

Traditional: "User 123 logged in at 10:30". Unstructured, hard to parse. Structured: {"timestamp": "2026-02-11T10:30:00Z", "user_id": 123, "action": "login"}. JSON format, easily searchable. import structlog; structlog.configure(processors=[structlog.json_renderer]); log = structlog.get_logger(); log.info('user_login', user_id=123, ip='192.168.1.1'). Tools (ELK stack, Splunk) parse JSON logs, enable searching and aggregation. Query: Find all login errors in last hour. Structured logging makes this trivial.

Centralized Logging

Distributed systems have many services each with own logs. Single server crash: check its logs. Service A fails, caused by service B? Check both logs simultaneously. Centralized logging: All services send logs to central server (ELK: Elasticsearch, Logstash, Kibana). Logstash collects logs, Elasticsearch indexes, Kibana visualizes. Query: Find all errors across all services in last hour. Sort by frequency. Alerting: If error rate spikes, send Slack notification. Dashboards: Real-time error rates, response times, user activity.

Metrics and Monitoring

Metrics: Quantifiable measurements. Request count, response time, error rate, CPU %, memory %, disk usage. Store in time-series database: Prometheus, InfluxDB. Collect via instrumentation: every request tracked, response time recorded. export gauge = Gauge('response_time_ms', 'Request response time'); gauge.set(response_time). Histograms: Distribution of values. Response time histogram: 10% of requests 10ms, 50% 50ms, 90% 200ms. Helps identify percentile issues.

Prometheus and Grafana

Prometheus: Time-series database for metrics. Scrapes metrics from applications at regular intervals. pip install prometheus_client. Expose metrics endpoint: from prometheus_client import Counter, Histogram; request_count = Counter('requests_total', 'Total requests'); response_time = Histogram('response_time_seconds', 'Response time'). In API: request_count.inc(); response_time.observe(elapsed_seconds). Prometheus scrapes: curl http://localhost:8000/metrics returns Prometheus format. Grafana: Visualization for Prometheus. Create dashboards: graphs showing request count, error rate, response time trends. Set alerts: if error rate > 5%, send alert. Daily reports: email dashboard showing metrics summary.

Distributed Tracing

Request flows through multiple services: API → Database → Cache → Message Queue. Single slow request: where's bottleneck? Distributed tracing follows request through all services, measuring time in each. Jaeger, Zipkin: tracing systems. Trace ID: unique identifier for request. Every service logs trace ID. Request starts: trace_id = uuid.uuid4(); context.set_trace_id(trace_id). Database service: log(f'trace_id={trace_id}, query_time=100ms'). Cache service: log(f'trace_id={trace_id}, cache_time=50ms'). Trace UI shows: API 50ms → Database 100ms → Cache miss 50ms → Cache fetch 500ms. Immediately see bottleneck.

Alerting

Metrics alone insufficient; too much data. Alerting: Rules trigger when thresholds exceeded. Error rate > 5%: ALERT (critical, needs investigation). Response time > 1 second: WARNING (monitor). CPU > 90%: ALERT. Set up alerts: if error_rate > 0.05: send_slack_message('Error rate critical'). Escalation: Page on-call engineer if critical. Sleep app crashes, developer gets woken up. Avoid alert fatigue: tune thresholds, only alert on important issues. 100 false alerts a day = ignored alerts (boy who cried wolf).

Debugging with Logs

Production issue: "Users can't login". Check logs: grep ERROR app.log (all errors). grep login app.log | grep ERROR. Database connection error. Check database logs. Connection refused, port 5432. Database container down. Restart. Logs provide context, speed debugging. Structured logs enable: grep user_id=123 app.log (all events for user 123). grep timestamp=[2026-02-11T10:00 2026-02-11T11:00] app.log (logs in time range). Combine with log parsing tools (awk, jq).

Best Practices

Log meaningful information. Not "something happened" but "user 123 login failed: invalid password". Include context: request_id, user_id, operation. Sensitive data: Don't log passwords, tokens, credit cards. Hash sensitive data before logging if needed. Log errors with full traceback: logger.exception() captures stack trace. Performance: Excessive logging slows app. Lazy logging: logger.debug(f'User {expensive_function()}') calls function always. Use: logger.debug('User %s', expensive_function) (lazy).

From Concept to Reality: Logging and Monitoring: System Health Visibility

In the professional world, the difference between a good engineer and a great one often comes down to understanding fundamentals deeply. Anyone can copy code from Stack Overflow. But when that code breaks at 2 AM and your application is down — affecting millions of users — only someone who truly understands the underlying concepts can diagnose and fix the problem.

Logging and Monitoring: System Health Visibility is one of those fundamentals. Whether you end up working at Google, building your own startup, or applying CS to solve problems in agriculture, healthcare, or education, these concepts will be the foundation everything else is built on. Indian engineers are known globally for their strong fundamentals — this is why companies worldwide recruit from IITs, NITs, IIIT Hyderabad, and BITS Pilani. Let us make sure you have that same strong foundation.

Object-Oriented Programming: Modelling the Real World

OOP lets you model real-world entities as code "objects." Each object has properties (data) and methods (behaviour). Here is a practical example:

class BankAccount:
    """A simple bank account — like what SBI or HDFC uses internally"""

    def __init__(self, holder_name, initial_balance=0):
        self.holder = holder_name
        self.balance = initial_balance    # Private in practice
        self.transactions = []            # History log

    def deposit(self, amount):
        if amount <= 0:
            raise ValueError("Deposit must be positive")
        self.balance += amount
        self.transactions.append(f"+₹{amount}")
        return self.balance

    def withdraw(self, amount):
        if amount > self.balance:
            raise ValueError("Insufficient funds!")
        self.balance -= amount
        self.transactions.append(f"-₹{amount}")
        return self.balance

    def statement(self):
        print(f"
--- Account Statement: {self.holder} ---")
        for t in self.transactions:
            print(f"  {t}")
        print(f"  Balance: ₹{self.balance}")

# Usage
acc = BankAccount("Rahul Sharma", 5000)
acc.deposit(15000)      # Salary credited
acc.withdraw(2000)      # UPI payment to Swiggy
acc.withdraw(500)       # Metro card recharge
acc.statement()

This is encapsulation — bundling data and behaviour together. The user of BankAccount does not need to know HOW deposit works internally; they just call it. Inheritance lets you extend this: a SavingsAccount could inherit from BankAccount and add interest calculation. Polymorphism means different account types can respond to the same .withdraw() method differently (savings accounts might check minimum balance, current accounts might allow overdraft).

Real-World System Design: Swiggy's Architecture

When you order food on Swiggy, here is what happens behind the scenes in about 2 seconds: your location is geocoded (algorithms), nearby restaurants are queried from a spatial index (data structures), menu prices are pulled from a database (SQL), delivery time is estimated using ML models trained on historical data (AI), the order is placed in a distributed message queue (Kafka), a delivery partner is assigned using a matching algorithm (optimization), and real-time tracking begins using WebSocket connections (networking). EVERY concept in your CS curriculum is being used simultaneously to deliver your biryani.

How the Web Request Cycle Works

Every time you visit a website, a precise sequence of events occurs. Here is the flow:

    You (Browser)          DNS Server          Web Server
        |                      |                    |
        |---[1] bharath.ai --->|                    |
        |                      |                    |
        |<--[2] IP: 76.76.21.9|                    |
        |                      |                    |
        |---[3] GET /index.html ----------------->  |
        |                      |                    |
        |                      |    [4] Server finds file,
        |                      |        runs server code,
        |                      |        prepares response
        |                      |                    |
        |<---[5] HTTP 200 OK + HTML + CSS + JS --- |
        |                      |                    |
   [6] Browser parses HTML                          |
       Loads CSS (styling)                          |
       Executes JS (interactivity)                  |
       Renders final page                           |

Step 1-2 is DNS resolution — converting a human-readable domain name to a machine-readable IP address. Step 3 is the HTTP request. Step 4 is server-side processing (this is where frameworks like Node.js, Django, or Flask operate). Step 5 is the HTTP response. Step 6 is client-side rendering (this is where React, Angular, or Vue operate).

In a real-world scenario, this cycle also involves CDNs (Content Delivery Networks), load balancers, caching layers, and potentially microservices. Indian companies like Jio use this exact architecture to serve 400+ million subscribers.

Production Engineering: Logging and Monitoring: System Health Visibility at Scale

Understanding logging and monitoring: system health visibility at an academic level is necessary but not sufficient. Let us examine how these concepts manifest in production environments where failure has real consequences.

Consider India's UPI system processing 10+ billion transactions monthly. The architecture must guarantee: atomicity (a transfer either completes fully or not at all — no half-transfers), consistency (balances always add up correctly across all banks), isolation (concurrent transactions on the same account do not interfere), and durability (once confirmed, a transaction survives any failure). These are the ACID properties, and violating any one of them in a payment system would cause financial chaos for millions of people.

At scale, you also face the thundering herd problem: what happens when a million users check their exam results at the same time? (CBSE result day, anyone?) Without rate limiting, connection pooling, caching, and graceful degradation, the system crashes. Good engineering means designing for the worst case while optimising for the common case. Companies like NPCI (the organisation behind UPI) invest heavily in load testing — simulating peak traffic to identify bottlenecks before they affect real users.

Monitoring and observability become critical at scale. You need metrics (how many requests per second? what is the 99th percentile latency?), logs (what happened when something went wrong?), and traces (how did a single request flow through 15 different microservices?). Tools like Prometheus, Grafana, ELK Stack, and Jaeger are standard in Indian tech companies. When Hotstar streams IPL to 50 million concurrent users, their engineering team watches these dashboards in real-time, ready to intervene if any metric goes anomalous.

The career implications are clear: engineers who understand both the theory (from chapters like this one) AND the practice (from building real systems) command the highest salaries and most interesting roles. India's top engineering talent earns ₹50-100+ LPA at companies like Google, Microsoft, and Goldman Sachs, or builds their own startups. The foundation starts here.

Key Vocabulary

Here are important terms from this chapter that you should know:

💡 Interview-Style Problem

Here is a problem that frequently appears in technical interviews at companies like Google, Amazon, and Flipkart: "Design a URL shortener like bit.ly. How would you generate unique short codes? How would you handle millions of redirects per second? What database would you use and why? How would you track click analytics?"

Think about: hash functions for generating short codes, read-heavy workload (99% redirects, 1% creates) suggesting caching, database choice (Redis for cache, PostgreSQL for persistence), and horizontal scaling with consistent hashing. Try sketching the system architecture on paper before looking up solutions. The ability to think through system design problems is the single most valuable skill for senior engineering roles.

Where This Takes You

The knowledge you have gained about logging and monitoring: system health visibility is directly applicable to: competitive programming (Codeforces, CodeChef — India has the 2nd largest competitive programming community globally), open-source contribution (India is the 2nd largest contributor on GitHub), placement preparation (these concepts form 60% of technical interview questions), and building real products (every startup needs engineers who understand these fundamentals).

India's tech ecosystem offers incredible opportunities. Freshers at top companies earn ₹15-50 LPA; experienced engineers at FAANG companies in India earn ₹50-1 Cr+. But more importantly, the problems being solved in India — digital payments for 1.4 billion people, healthcare AI for rural areas, agricultural tech for 150 million farmers — are some of the most impactful engineering challenges in the world. The fundamentals you are building will be the tools you use to tackle them.

Crafted for Class 7–9 • Programming & Coding • Aligned with NEP 2020 & CBSE Curriculum