{"id":65,"date":"2023-09-01T08:51:00","date_gmt":"2023-09-01T08:51:00","guid":{"rendered":"https:\/\/amirhooshang.com\/blog\/?p=65"},"modified":"2025-05-16T20:57:22","modified_gmt":"2025-05-16T20:57:22","slug":"evaluation-metrics-for-machine-learning-classification-models-from-accuracy-to-roc-auc","status":"publish","type":"post","link":"https:\/\/amirhooshang.com\/blog\/2023\/09\/01\/evaluation-metrics-for-machine-learning-classification-models-from-accuracy-to-roc-auc\/","title":{"rendered":"Evaluation Metrics for Machine Learning Classification Models: From Accuracy to ROC-AUC"},"content":{"rendered":"\n

<\/h3>\n\n\n\n

Imagine you\u2019ve built a machine learning model to detect cancer from medical scans or filter spam emails. How do you know if it\u2019s actually working well? Evaluation metrics<\/strong> act like a report card for your model\u2014they tell you where it\u2019s crushing it and where it\u2019s falling short. In this article, we\u2019ll break down the key metrics for classification models and explain when to use them.<\/p>\n\n\n\n

\n\n\n\n

1. Accuracy \u2013 The Basic “Passing Grade”<\/strong><\/h3>\n\n\n\n
    \n
  • What is it?<\/strong> The percentage of correct predictions. For example, if your model correctly labels 95 out of 100 cat\/dog images, its accuracy is 95%.<\/li>\n\n\n\n
  • Formula:<\/strong>

      <\/span>   <\/span>\"\[\text{Accuracy} = \frac{\text{Correct Predictions}}{\text{Total Data}}\]\"<\/p><\/li>\n\n\n\n

  • When to use it?<\/strong> Great for balanced datasets<\/strong> (where classes are roughly equal).<\/li>\n\n\n\n
  • The catch?<\/strong> Misleading for imbalanced data<\/strong>. For example, if 95% of emails are not<\/em> spam, a model that labels everything<\/em> as “not spam” still gets 95% accuracy\u2014but fails to detect spam!<\/li>\n<\/ul>\n\n\n\n
    \n\n\n\n

    2. Precision \u2013 The “Trustworthy Yes” Metric<\/strong><\/h4>\n\n\n\n
      \n
    • What is it?<\/strong> Out of all the positive<\/em> predictions your model makes (e.g., “this is spam” or “this person is sick”), how many are actually<\/em> correct?<\/li>\n\n\n\n
    • Formula:<\/strong>

        <\/span>   <\/span>\"\[\text{Precision} = \frac{\text{True Positives}}{\text{True Positives + False Positives}}\]\"<\/p><\/li>\n\n\n\n

    • Example:<\/strong> If your model flags 80 patients as “sick” but 20 of those are actually healthy, Precision = 80%.<\/li>\n\n\n\n
    • When does it matter?<\/strong> When False Positives (FP)<\/strong> are costly. For example, wrongly diagnosing a healthy person with a disease wastes resources and causes stress.<\/li>\n<\/ul>\n\n\n\n
      \n\n\n\n

      3. Recall \u2013 The “Don\u2019t Miss the Positives” Metric<\/strong><\/h4>\n\n\n\n
        \n
      • What is it?<\/strong> Out of all the actual<\/em> positive cases (e.g., real cancer patients), how many did your model correctly identify?<\/li>\n\n\n\n
      • Formula:<\/strong>

          <\/span>   <\/span>\"\[\text{Recall} = \frac{\text{True Positives}}{\text{True Positives + False Negatives}}\]\"<\/p><\/li>\n\n\n\n

      • Example:<\/strong> If there are 10 cancer patients and your model catches 9, Recall = 90%.<\/li>\n\n\n\n
      • When does it matter?<\/strong> When False Negatives (FN)<\/strong> are dangerous. For example, missing a cancer diagnosis could be life-threatening!<\/li>\n<\/ul>\n\n\n\n
        \n\n\n\n

        4. F1-Score \u2013 The Balanced “Best of Both Worlds”<\/strong><\/h4>\n\n\n\n
          \n
        • What is it?<\/strong> A single score that balances Precision and Recall. High F1 means your model is good at both<\/em> avoiding false alarms and<\/em> catching true positives.<\/li>\n\n\n\n
        • Formula:<\/strong>

            <\/span>   <\/span>\"\[F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision + Recall}}\]\"<\/p><\/li>\n\n\n\n

        • Example:<\/strong> If Precision = 80% and Recall = 90%, F1 \u2248 85%.<\/li>\n\n\n\n
        • When to use it?<\/strong> Ideal for imbalanced data<\/strong> or when both FP and FN matter. For example, fraud detection: you don\u2019t want to miss fraud (high Recall) or<\/em> falsely accuse users (high Precision).<\/li>\n<\/ul>\n\n\n\n
          \n\n\n\n

          5. ROC-AUC \u2013 The “How Well Can It Tell Apart?” Score<\/strong><\/h4>\n\n\n\n
            \n
          • What is it?<\/strong> A score between 0 and 1 that measures how well your model distinguishes between classes (e.g., sick vs. healthy). Closer to 1 = better.<\/li>\n\n\n\n
          • Example:<\/strong> A model with AUC = 0.95 is way stronger than one with AUC = 0.70.<\/li>\n\n\n\n
          • How does it work?<\/strong> The ROC curve plots the trade-off between True Positive Rate (Recall) and False Positive Rate. AUC = area under this curve.<\/li>\n<\/ul>\n\n\n\n
            \n\n\n\n

            6. Confusion Matrix \u2013 The Model\u2019s “Performance Report”<\/strong><\/h4>\n\n\n\n
              \n
            • What is it?<\/strong> A 2×2 table summarizing your model\u2019s predictions:<\/li>\n\n\n\n
            • True Positive (TP):<\/strong> Correctly predicted “positive” (e.g., correctly identified spam).<\/li>\n\n\n\n
            • False Positive (FP):<\/strong> Wrongly predicted “positive” (e.g., flagged a normal email as spam).<\/li>\n\n\n\n
            • True Negative (TN):<\/strong> Correctly predicted “negative” (e.g., correctly identified non-spam).<\/li>\n\n\n\n
            • False Negative (FN):<\/strong> Wrongly predicted “negative” (e.g., missed a spam email).<\/li>\n\n\n\n
            • Example:<\/strong> Predicted Sick Predicted Healthy Actual Sick<\/strong> 50 (TP) 5 (FN) Actual Healthy<\/strong> 10 (FP) 100 (TN)<\/li>\n\n\n\n
            • Why use it?<\/strong> It shows where<\/em> your model struggles. For instance, high FN means it\u2019s missing sick patients!<\/li>\n<\/ul>\n\n\n\n
              \n\n\n\n

              Why You Can\u2019t Rely on Just One Metric<\/strong><\/h3>\n\n\n\n

              Choosing metrics depends on your problem\u2019s context<\/strong> and error costs<\/strong>:<\/p>\n\n\n\n

                \n
              • Imbalanced data (e.g., 90% healthy vs. 10% sick):<\/strong><\/li>\n\n\n\n
              • Accuracy is misleading. Use F1-Score<\/strong> or AUC<\/strong> instead.<\/li>\n\n\n\n
              • Costly errors:<\/strong><\/li>\n\n\n\n
              • High FP cost?<\/strong> Prioritize Precision<\/strong> (e.g., avoiding false fraud alerts).<\/li>\n\n\n\n
              • High FN cost?<\/strong> Prioritize Recall<\/strong> (e.g., cancer screening).<\/li>\n\n\n\n
              • Combining metrics:<\/strong><\/li>\n\n\n\n
              • Bank fraud detection:<\/strong> Balance Precision (avoid false accusations) and Recall (catch all fraud). Use F1-Score<\/strong>.<\/li>\n\n\n\n
              • Email marketing:<\/strong> Maximize Recall to reach all potential customers, even if some emails go to uninterested users.<\/li>\n<\/ul>\n\n\n\n
                \n\n\n\n

                Conclusion: Which Metric Should You Pick?<\/strong><\/h3>\n\n\n\n

                No single metric tells the whole story!<\/p>\n\n\n\n

                  \n
                • Quick check:<\/strong> Start with Accuracy, but always<\/em> look at the Confusion Matrix.<\/li>\n\n\n\n
                • Imbalanced data?<\/strong> Use F1-Score<\/strong> or AUC<\/strong>.<\/li>\n\n\n\n
                • High-stakes errors?<\/strong> Focus on Recall<\/strong> (e.g., medical diagnoses) or Precision<\/strong> (e.g., drug testing).<\/li>\n<\/ul>\n\n\n\n

                  Ultimately, your choice should align with business goals<\/strong> and real-world impact<\/strong>. For example, in rare disease detection, you need high Recall (don\u2019t miss patients) and<\/em> decent Precision (avoid overwhelming the system with false alarms).<\/p>\n","protected":false},"excerpt":{"rendered":"

                  Imagine you\u2019ve built a machine learning model to detect cancer from medical scans or filter spam emails. How do you know if it\u2019s actually working well? Evaluation metrics act like…<\/p>\n","protected":false},"author":1,"featured_media":93,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[61,62],"tags":[73,78,76,80,56,79,74,75,77],"class_list":["post-65","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","category-model-evaluation","tag-accuracy-2","tag-classification-2","tag-confusion-matrix","tag-evaluation-metrics","tag-f1-score","tag-machine-learning","tag-precision-2","tag-recall","tag-roc-auc"],"_links":{"self":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts\/65","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/comments?post=65"}],"version-history":[{"count":4,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts\/65\/revisions"}],"predecessor-version":[{"id":74,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/posts\/65\/revisions\/74"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/media\/93"}],"wp:attachment":[{"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/media?parent=65"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/categories?post=65"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/amirhooshang.com\/blog\/wp-json\/wp\/v2\/tags?post=65"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}