Building an Indian Food Classifier: From Data to Deployment

Building an Indian Food Classifier: From Data to Deployment

The Problem: Too Many Delicious Options

Imagine you're building a recommendation system for Flipkart or JioMart's food delivery section. A user takes a photo of their dinner—maybe a dosa, maybe biryani, maybe chole bhature. Your app needs to identify the food in 0.5 seconds so it can recommend similar recipes or suggest pairing drinks. This is an image classification problem, and it's trickier than it sounds.

Why is it hard? Because dosa from different regions looks different. South Indian dosa is thin and crispy. North Indian dosa is thicker. Some have sambar, some don't. A CNN needs to learn that all these variations are still "dosa." Meanwhile, it needs to not confuse dosa with uttapam, which are similarly shaped but different foods.

In this chapter, we'll build a production-ready food classifier from scratch. We'll handle real challenges: limited labeled data, class imbalance, and the need for speed. Most importantly, we'll learn how transfer learning lets us build sophisticated models without needing millions of labeled images.

Step 1: Data Collection and Organization

The foundation of any ML project is data. We need to collect or download images of Indian foods. Here's what we'll collect:

Foods to classify:
1. Dosa (South Indian crepe)
2. Biryani (rice dish)
3. Chole Bhature (fried bread with chickpeas)
4. Samosa (fried triangular pastry)
5. Idli (steamed rice cake)
6. Tandoori Chicken (roasted chicken)
7. Paneer Tikka (marinated cottage cheese)
8. Rogan Josh (meat curry)

Target: 200-300 images per class = 1600-2400 total images

In real projects, data collection is tedious. You'd either download images from food websites, scrape them (legally!), or hire photographers. For this project, we can use open-source datasets like Food-101 or Indian Food Images dataset.

The directory structure matters:

food_classifier/
├── data/
│   ├── train/
│   │   ├── dosa/
│   │   │   ├── dosa_001.jpg
│   │   │   ├── dosa_002.jpg
│   │   │   └── ...
│   │   ├── biryani/
│   │   │   └── ...
│   │   └── ... (other classes)
│   ├── val/
│   │   ├── dosa/
│   │   ├── biryani/
│   │   └── ...
│   └── test/
│       ├── dosa/
│       ├── biryani/
│       └── ...
├── model.py
├── train.py
├── evaluate.py
└── app.py

Why separate train, val, and test? This is crucial for honest evaluation. When we train, the model sees training images and learns. It overfits if given too much capacity. Validation data lets us monitor if the model is really learning or just memorizing. Test data is held completely secret until the end—it's our final exam.

A typical split is 70% train, 15% val, 15% test. With 2400 images and 8 classes, that's 210 training images per class, 45 validation per class, and 45 test per class.

Step 2: Data Augmentation—Creating More Data From What We Have

Here's a problem: we probably don't have 2400 perfect images. Maybe we have 500. Can we build a good classifier with just 500 images? Maybe, if we're clever.

Data augmentation is a trick: we artificially create new training images by transforming existing images. A dosa is still a dosa if it's rotated slightly, shifted, or if the brightness is adjusted slightly. Here are common augmentations:

Original Image (dosa.jpg)
        ↓
Augmentation Pipeline:
  1. Random rotation (-15° to +15°)
  2. Random horizontal flip
  3. Random zoom (0.8 to 1.2)
  4. Random brightness adjustment (-0.2 to +0.2)
  5. Random crop
        ↓
8 New Images from 1 Original

Now 500 images becomes 4000!

In Python with TensorFlow, this is straightforward:

from tensorflow.keras.preprocessing.image import ImageDataGenerator

train_augmentation = ImageDataGenerator(
    rotation_range=15,
    horizontal_flip=True,
    zoom_range=0.2,
    brightness_range=[0.8, 1.2],
    width_shift_range=0.2,
    height_shift_range=0.2,
    fill_mode='nearest'
)

train_generator = train_augmentation.flow_from_directory(
    'data/train/',
    target_size=(224, 224),
    batch_size=32,
    class_mode='categorical'
)

Notice target_size=(224, 224). We're resizing all images to 224×224 pixels because that's the input size the pre-trained MobileNet model expects. More on this soon.

Step 3: Understanding Transfer Learning—Standing on Giants' Shoulders

Here's a profound insight: a CNN trained on ImageNet (1.2 million images, 1000 classes) has already learned to recognize edges, textures, colors, simple shapes, and complex objects. If we train a fresh CNN from scratch on just 500 food images, it'll be terrible—insufficient data. But what if we take that pre-trained CNN and adapt it to food classification? This is transfer learning.

The idea: the early layers of the ImageNet-trained CNN detect general features (edges, textures) that are useful for any image classification task. We keep these layers frozen (don't update their weights). We replace the last layer—which was trained for ImageNet's 1000 classes—with new layers trained for our 8 food classes. We train only this new part on our food images.

Why does this work? Because learning to classify food uses the same edge-detection and texture-detection skills that ImageNet already mastered. We're not starting from scratch; we're fine-tuning.

ImageNet-Trained MobileNet:
Input (224×224×3)
  ↓
[Pre-trained Conv Blocks] ← FROZEN (don't update weights)
  ↓
[Feature Maps: 1280 values]
  ↓
[Pre-trained FC Layer: 1000 classes] ← REMOVE THIS
  ↓
Output: probability of ImageNet classes

Our Modification:
Input (224×224×3)
  ↓
[Pre-trained Conv Blocks] ← FROZEN
  ↓
[Feature Maps: 1280 values]
  ↓
[NEW Global Average Pooling]
  ↓
[NEW Dense Layer: 256 neurons] ← TRAIN THESE
  ↓
[NEW Dense Layer: 8 classes] ← TRAIN THESE
  ↓
Output: probability of our 8 foods

MobileNet is perfect for this. While larger CNNs like ResNet or VGG have millions of parameters, MobileNet achieves similar accuracy with far fewer parameters. This makes it fast and perfect for deployment on phones or servers with limited resources.

Step 4: Building and Training the Model

from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras import layers, models

# Load pre-trained MobileNetV2
base_model = MobileNetV2(
    input_shape=(224, 224, 3),
    include_top=False,  # Remove the top classification layer
    weights='imagenet'
)

# Freeze the base model
base_model.trainable = False

# Create our model on top
model = models.Sequential([
    base_model,
    layers.GlobalAveragePooling2D(),
    layers.Dense(256, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(8, activation='softmax')  # 8 food classes
])

# Compile
model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

# Train
history = model.fit(
    train_generator,
    epochs=20,
    validation_data=val_generator,
    steps_per_epoch=100,  # ~3200 images / 32 batch size
    validation_steps=50
)

The training process: we feed batches of 32 images through the network, calculate loss (how wrong our predictions are), backpropagate gradients, and update weights in the new layers. The pre-trained layers don't update because we set trainable=False.

After 20 epochs on our food data, we expect validation accuracy around 92-96%. That's powerful! We went from training on 500 images to achieving high accuracy because we leveraged knowledge from ImageNet.

Step 5: Evaluation—Understanding What Our Model Gets Wrong

After training, we evaluate on the test set (images the model never saw during training or validation). Here's what we measure:

Accuracy: percentage of images correctly classified. If our model correctly classifies 450 out of 500 test images, accuracy is 90%.

Confusion Matrix: a table showing which foods get confused. Here's an example:

              Predicted
           Dosa Biryani Samosa
Actual Dosa  [95    3      2]
       Biryani[2    98     0]
       Samosa [1    1     98]

This is excellent! Our model rarely confuses different foods. But if we saw:

              Predicted
           Dosa Biryani Samosa
Actual Dosa  [85    10     5]
       Biryani[8    80    12]
       Samosa [6    15    79]

This tells us the model is confused—maybe our training data has issues, or maybe these foods genuinely look similar.

Per-class Metrics:

Precision: Of all images we predicted as dosa, how many were actually dosa?

Recall: Of all actual dosa images, how many did we correctly identify?

F1-Score: The harmonic mean of precision and recall.

from sklearn.metrics import classification_report, confusion_matrix

y_pred = model.predict(test_images)
y_pred_classes = np.argmax(y_pred, axis=1)

print(classification_report(y_test, y_pred_classes,
                           target_names=food_classes))
print(confusion_matrix(y_test, y_pred_classes))

Looking at these metrics tells us: Is the model uniformly good? Or does it struggle with specific foods? If samosa recall is 0.75 but dosa recall is 0.95, samosas are harder to classify. Maybe we need more samosa training images, or maybe samosa images are taken from bad angles in our dataset.

Step 6: Deployment as a Web App

A trained model sitting on a researcher's laptop is useless. We need to deploy it. Here's a Flask web app that serves our classifier:

from flask import Flask, request, jsonify
from PIL import Image
import numpy as np
import tensorflow as tf

app = Flask(__name__)
model = tf.keras.models.load_model('food_classifier.h5')
class_names = ['Dosa', 'Biryani', 'Chole Bhature', 'Samosa',
               'Idli', 'Tandoori Chicken', 'Paneer Tikka', 'Rogan Josh']

@app.route('/classify', methods=['POST'])
def classify():
    # Get image from request
    image_file = request.files['image']
    image = Image.open(image_file).convert('RGB')
    image = image.resize((224, 224))

    # Preprocess
    image_array = np.array(image) / 255.0
    image_array = np.expand_dims(image_array, axis=0)

    # Predict
    predictions = model.predict(image_array)
    predicted_class = np.argmax(predictions[0])
    confidence = predictions[0][predicted_class]

    return jsonify({
        'food': class_names[predicted_class],
        'confidence': float(confidence),
        'all_predictions': {
            class_names[i]: float(predictions[0][i])
            for i in range(len(class_names))
        }
    })

if __name__ == '__main__':
    app.run(debug=False, host='0.0.0.0', port=5000)

Now anyone with internet access can upload a food image and get instant classification. A JioMart recommendation engine could use this to suggest recipes or shopping items based on what you're cooking.

Step 7: Addressing Real-World Challenges

Class Imbalance: What if you have 500 dosa images but only 50 samosa images? The model sees more dosa examples and becomes biased toward predicting dosa. Solutions: oversample the minority class, undersample the majority class, or use class weights during training:

class_weights = {
    0: 1.0,      # Dosa (common)
    1: 1.0,      # Biryani (common)
    2: 10.0      # Samosa (rare)
}

model.fit(train_generator, class_weight=class_weights, ...)

Poor Image Quality: Real user photos from phones might be blurry, poorly lit, or partially occluded. Your model was trained on clean images. Solutions: augment your training data with blur, noise, and lighting changes. Test on phone photos before deployment.

Ambiguous Cases: Some images might genuinely be hard—samosa and empanada look similar; different biryani preparations vary wildly. Instead of forcing a prediction, output confidence scores. If confidence < 0.7, ask the user to confirm: "Is this biryani?"

Step 8: Monitoring in Production

Your model works great in the lab but what about after deployment? Here's what changes:

1. Users photograph food in real conditions: bad lighting, weird angles, small portions, multiple foods in frame.

2. Food trends change. New restaurant chains introduce new dishes. Cooking styles evolve.

3. Model drift: performance gradually degrades because the real-world data distribution doesn't match training data distribution.

Best practices:

- Log all predictions and true labels (with user confirmation).

- Monitor accuracy weekly. If it drops below 85%, investigate.

- Retrain monthly with new user data.

- A/B test: serve two model versions to different user subsets and compare performance.

Why Transfer Learning Matters

Transfer learning is why AI is accessible today. Without it, you'd need millions of labeled images and months of GPU training to build a good food classifier. With it, you need hundreds of images and a few hours. This democratization of AI is transforming India's startup ecosystem—Flipkart, JioMart, and food delivery apps rely heavily on transfer learning for computer vision tasks.

Understanding image classification, data augmentation, confusion matrices, and deployment takes you from "AI hobbyist" to "AI engineer." These skills are in huge demand in India's tech industry.