{"id":119,"date":"2024-07-02T15:52:00","date_gmt":"2024-07-02T15:52:00","guid":{"rendered":"https:\/\/amirhooshang.com\/blog\/?p=119"},"modified":"2025-05-19T16:22:47","modified_gmt":"2025-05-19T16:22:47","slug":"machine-learning-metrics-in-online-learning-how-to-know-if-your-model-is-doing-well","status":"publish","type":"post","link":"https:\/\/amirhooshang.com\/blog\/2024\/07\/02\/machine-learning-metrics-in-online-learning-how-to-know-if-your-model-is-doing-well\/","title":{"rendered":"Machine Learning Metrics in Online Learning: How to Know If Your Model is Doing Well?"},"content":{"rendered":"\n

Introduction: Online Learning is Like Driving in Fog!<\/strong><\/h2>\n\n\n\n

Imagine driving through a foggy road where new signs appear every few seconds, and you must react immediately. Online Learning<\/strong> works the same way! Your model continuously learns from new data, just like a driver adapting to changing road conditions.<\/p>\n\n\n\n

But how do we know if the model is actually<\/em> performing well? This is where evaluation metrics<\/strong> come in. In this article, I\u2019ll break down these metrics in plain English with relatable examples.<\/p>\n\n\n\n

Evaluation Metrics: Measuring Sticks for Online Learning Models<\/strong><\/h2>\n\n\n\n

1. Accuracy \u2013 \u201cWhat\u2019s My Hit Rate?\u201d<\/strong><\/h3>\n\n\n\n

Definition:<\/strong> If your model aces 8 out of 10 questions on a quiz, its accuracy is 80%.
Real-World Example:<\/strong> A spam filter that correctly labels 90 out of 100 emails has 90% accuracy.
The Catch:<\/strong> If 95% of emails are legit and only 5% are spam, a model that always<\/em> says \u201cnot spam\u201d still gets 95% accuracy! Misleading for imbalanced data<\/strong>.<\/p>\n\n\n\n

2. Error Rate \u2013 \u201cHow Often Am I Messing Up?\u201d<\/strong><\/h3>\n\n\n\n

Definition:<\/strong> The flip side of accuracy. If accuracy is 90%, error rate is 10%.
Use Case:<\/strong> Tells you how often<\/em> the model is wrong but doesn\u2019t explain the type of errors<\/strong> (e.g., false alarms vs. missed threats).<\/p>\n\n\n\n

3. Precision & Recall \u2013 \u201cQuality vs. Completeness\u201d<\/strong><\/h3>\n\n\n\n

Precision:<\/strong><\/h4>\n\n\n\n
    \n
  • \u201cOut of all the positives I predicted, how many were actually<\/em> positive?\u201d<\/strong><\/li>\n\n\n\n
  • Example: A COVID test with high precision means most people flagged as \u201cpositive\u201d are<\/em> truly sick (few false alarms).<\/li>\n<\/ul>\n\n\n\n

    Recall:<\/strong><\/h4>\n\n\n\n
      \n
    • \u201cOut of all real<\/em> positives, how many did I catch?\u201d<\/strong><\/li>\n\n\n\n
    • Example: A test with high recall won\u2019t miss sick people<\/strong>, even if it sometimes flags healthy folks as positive.<\/li>\n<\/ul>\n\n\n\n

      Fun Example:<\/strong><\/p>\n\n\n\n

        \n
      • Low-Precision Car Alarm:<\/strong> Keeps blaring for no reason (too many false positives).<\/li>\n\n\n\n
      • Low-Recall Car Alarm:<\/strong> Fails to go off when a thief breaks in (dangerous false negatives!).<\/li>\n<\/ul>\n\n\n\n

        4. F1-Score \u2013 \u201cThe Best of Both Worlds\u201d<\/strong><\/h3>\n\n\n\n

        Definition:<\/strong> A balanced score combining precision and recall.
        Example:<\/strong> If precision = 80% and recall = 60%, F1 \u2248 68%.
        When to Use:<\/strong> When you need to balance<\/strong> false positives and false negatives.<\/p>\n\n\n\n

        5. MSE (Mean Squared Error) \u2013 \u201cHow Wild Are My Guesses?\u201d<\/strong><\/h3>\n\n\n\n

        Definition:<\/strong> The average squared<\/em> difference between predictions and actual values.
        Example:<\/strong> Predicting stock prices: If the real price is $1000 and your model predicts $1100, the error is $100. Squaring it (10,000) penalizes big mistakes harder.
        Why Squared?<\/strong> To make large errors hurt more<\/em>!<\/p>\n\n\n\n

        6. Regret \u2013 \u201cHow Much Am I Missing Out?\u201d<\/strong><\/h3>\n\n\n\n

        Definition:<\/strong> The gap between your model\u2019s performance and the best possible outcome<\/em>.
        Example:<\/strong> Think of an online chess player. Regret answers: \u201cIf I\u2019d played the perfect moves, how many more points would I have?\u201d<\/em><\/p>\n\n\n\n

        Combining Metrics: Why One Metric Isn\u2019t Enough<\/strong><\/h2>\n\n\n\n

        Real-World Example: A Movie Recommender (Like Netflix)<\/strong><\/h3>\n\n\n\n
          \n
        • Accuracy:<\/strong> Tells you what % of recommendations users liked.<\/li>\n\n\n\n
        • Precision:<\/strong> Ensures recommendations are truly relevant<\/em>.<\/li>\n\n\n\n
        • Recall:<\/strong> Ensures you\u2019re covering all<\/em> user interests.<\/li>\n\n\n\n
        • Regret:<\/strong> Measures how far you are from the \u201cperfect\u201d recommendations.<\/li>\n<\/ul>\n\n\n\n

          Why Mix Metrics?<\/strong><\/p>\n\n\n\n

            \n
          • Focusing only on precision might make the model too cautious<\/strong> (fewer recommendations).<\/li>\n\n\n\n
          • Focusing only on recall might overwhelm users<\/strong> with irrelevant suggestions.<\/li>\n\n\n\n
          • F1-Score + Regret<\/strong> can strike the right balance.<\/li>\n<\/ul>\n\n\n\n

            Conclusion: Be Your Model\u2019s Doctor!<\/strong><\/h2>\n\n\n\n

            Online learning is like a patient whose condition keeps changing. As the \u201cmodel doctor,\u201d you need to:
            Treat accuracy<\/strong> as a general checkup.
            Use precision & recall<\/strong> as specialized lab tests.
            Prescribe F1-Score<\/strong> to maintain balance.
            Ask regret<\/strong> to see if you\u2019re using the \u201cbest possible treatment.\u201d<\/p>\n\n\n\n

            No single metric is enough!<\/strong> Depending on the problem (disease diagnosis, movie recommendations, stock predictions), combine metrics<\/strong> to get the full picture.<\/p>\n\n\n\n

            <\/h2>\n\n\n\n

            <\/p>\n","protected":false},"excerpt":{"rendered":"

            Introduction: Online Learning is Like Driving in Fog! Imagine driving through a foggy road where new signs appear every few seconds, and you must react immediately. Online Learning works the…<\/p>\n","protected":false},"author":1,"featured_media":121,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[61,62],"tags":[73,56,137,98,112,134,133,135,136],"class_list":["post-119","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","category-model-evaluation","tag-accuracy-2","tag-f1-score","tag-imbalanced-data","tag-machine-learning-metrics","tag-model-evaluation","tag-mse","tag-online-learning","tag-precision-vs-recall","tag-regret"],"_links":{"self":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts\/119","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/comments?post=119"}],"version-history":[{"count":2,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts\/119\/revisions"}],"predecessor-version":[{"id":122,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts\/119\/revisions\/122"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/media\/121"}],"wp:attachment":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/media?parent=119"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/categories?post=119"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/tags?post=119"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}