Home/Resilience Patterns/Graceful Degradation

Graceful Degradation

Fallbacks and partial functionality

advanced
degradationfallbackresilience
🎮 Interactive Playground

What is Graceful Degradation?

Graceful degradation maintains partial functionality when components fail. Instead of complete failure, the system provides reduced but still useful service, prioritizing critical features over optional ones.

The Problem

Systems with hard dependencies fail catastrophically:

  • All-or-nothing: One service down means everything down
  • No fallbacks: Users get errors instead of alternatives
  • Rigid architecture: Can't adapt to partial failures
  • Poor user experience: Complete outages frustrate users

Example Code

use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::RwLock;

/// Feature flags for degradation levels
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Feature {
    Search,
    Recommendations,
    UserReviews,
    RealTimePricing,
    Personalization,
    Analytics,
    ExternalIntegrations,
}

/// System degradation levels
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub enum DegradationLevel {
    /// Full functionality
    Normal,
    /// Non-essential features disabled
    Reduced,
    /// Only core features available
    Minimal,
    /// Read-only mode
    ReadOnly,
    /// Complete failure
    Offline,
}

/// Degradation manager
pub struct DegradationManager {
    level: RwLock<DegradationLevel>,
    feature_status: RwLock<HashMap<Feature, bool>>,
    fallbacks: HashMap<Feature, Arc<dyn Fallback + Send + Sync>>,
}

impl DegradationManager {
    pub fn new() -> Self {
        let mut feature_status = HashMap::new();
        // All features enabled by default
        feature_status.insert(Feature::Search, true);
        feature_status.insert(Feature::Recommendations, true);
        feature_status.insert(Feature::UserReviews, true);
        feature_status.insert(Feature::RealTimePricing, true);
        feature_status.insert(Feature::Personalization, true);
        feature_status.insert(Feature::Analytics, true);
        feature_status.insert(Feature::ExternalIntegrations, true);

        DegradationManager {
            level: RwLock::new(DegradationLevel::Normal),
            feature_status: RwLock::new(feature_status),
            fallbacks: HashMap::new(),
        }
    }

    pub fn with_fallback(
        mut self,
        feature: Feature,
        fallback: impl Fallback + Send + Sync + 'static,
    ) -> Self {
        self.fallbacks.insert(feature, Arc::new(fallback));
        self
    }

    pub async fn current_level(&self) -> DegradationLevel {
        *self.level.read().await
    }

    pub async fn set_level(&self, level: DegradationLevel) {
        *self.level.write().await = level;
        self.apply_degradation(level).await;
    }

    pub async fn is_feature_enabled(&self, feature: Feature) -> bool {
        *self.feature_status.read().await.get(&feature).unwrap_or(&false)
    }

    pub async fn disable_feature(&self, feature: Feature) {
        self.feature_status.write().await.insert(feature, false);
    }

    pub async fn enable_feature(&self, feature: Feature) {
        self.feature_status.write().await.insert(feature, true);
    }

    async fn apply_degradation(&self, level: DegradationLevel) {
        let mut status = self.feature_status.write().await;

        match level {
            DegradationLevel::Normal => {
                // Enable all features
                for (_, enabled) in status.iter_mut() {
                    *enabled = true;
                }
            }
            DegradationLevel::Reduced => {
                // Disable non-essential features
                status.insert(Feature::Recommendations, false);
                status.insert(Feature::Analytics, false);
                status.insert(Feature::Personalization, false);
            }
            DegradationLevel::Minimal => {
                // Only core features
                status.insert(Feature::Search, true);
                status.insert(Feature::RealTimePricing, true);
                status.insert(Feature::Recommendations, false);
                status.insert(Feature::UserReviews, false);
                status.insert(Feature::Personalization, false);
                status.insert(Feature::Analytics, false);
                status.insert(Feature::ExternalIntegrations, false);
            }
            DegradationLevel::ReadOnly => {
                // Read-only mode - no writes
                for (_, enabled) in status.iter_mut() {
                    *enabled = false;
                }
                status.insert(Feature::Search, true); // Read operations only
            }
            DegradationLevel::Offline => {
                for (_, enabled) in status.iter_mut() {
                    *enabled = false;
                }
            }
        }
    }

    pub fn get_fallback(&self, feature: Feature) -> Option<Arc<dyn Fallback + Send + Sync>> {
        self.fallbacks.get(&feature).cloned()
    }
}

impl Default for DegradationManager {
    fn default() -> Self {
        Self::new()
    }
}

/// Fallback behavior trait
pub trait Fallback {
    fn execute(&self) -> FallbackResult;
}

pub enum FallbackResult {
    CachedData(String),
    DefaultValue(String),
    EmptyResponse,
    ErrorMessage(String),
}

/// Cached data fallback
pub struct CacheFallback {
    cache: Arc<RwLock<HashMap<String, CachedItem>>>,
}

struct CachedItem {
    data: String,
    cached_at: std::time::Instant,
}

impl CacheFallback {
    pub fn new() -> Self {
        CacheFallback {
            cache: Arc::new(RwLock::new(HashMap::new())),
        }
    }
}

impl Default for CacheFallback {
    fn default() -> Self {
        Self::new()
    }
}

impl Fallback for CacheFallback {
    fn execute(&self) -> FallbackResult {
        // Return cached data
        FallbackResult::CachedData("Cached result".to_string())
    }
}

/// Default value fallback
pub struct DefaultFallback {
    default: String,
}

impl DefaultFallback {
    pub fn new(default: impl Into<String>) -> Self {
        DefaultFallback {
            default: default.into(),
        }
    }
}

impl Fallback for DefaultFallback {
    fn execute(&self) -> FallbackResult {
        FallbackResult::DefaultValue(self.default.clone())
    }
}

/// Service with graceful degradation
pub struct ProductService {
    degradation: Arc<DegradationManager>,
    cache: Arc<RwLock<HashMap<String, Product>>>,
}

#[derive(Debug, Clone)]
pub struct Product {
    pub id: String,
    pub name: String,
    pub price: f64,
    pub reviews: Vec<Review>,
    pub recommendations: Vec<String>,
}

#[derive(Debug, Clone)]
pub struct Review {
    pub rating: u8,
    pub comment: String,
}

impl ProductService {
    pub fn new(degradation: Arc<DegradationManager>) -> Self {
        ProductService {
            degradation,
            cache: Arc::new(RwLock::new(HashMap::new())),
        }
    }

    pub async fn get_product(&self, id: &str) -> Result<ProductResponse, ServiceError> {
        let level = self.degradation.current_level().await;

        // Check if we're completely offline
        if level == DegradationLevel::Offline {
            return Err(ServiceError::ServiceUnavailable);
        }

        // Try to get fresh data
        let product = match self.fetch_product(id).await {
            Ok(p) => p,
            Err(_) => {
                // Fall back to cache
                self.get_from_cache(id).await
                    .ok_or(ServiceError::NotFound)?
            }
        };

        // Build response based on enabled features
        let mut response = ProductResponse {
            id: product.id.clone(),
            name: product.name.clone(),
            price: None,
            reviews: None,
            recommendations: None,
        };

        // Include real-time pricing if enabled
        if self.degradation.is_feature_enabled(Feature::RealTimePricing).await {
            response.price = Some(self.get_realtime_price(&product.id).await
                .unwrap_or(product.price)); // Fall back to cached price
        } else {
            response.price = Some(product.price); // Use cached price
        }

        // Include reviews if enabled
        if self.degradation.is_feature_enabled(Feature::UserReviews).await {
            response.reviews = Some(product.reviews.clone());
        }

        // Include recommendations if enabled
        if self.degradation.is_feature_enabled(Feature::Recommendations).await {
            response.recommendations = self.get_recommendations(&product.id).await.ok();
        }

        Ok(response)
    }

    async fn fetch_product(&self, id: &str) -> Result<Product, ServiceError> {
        // Simulate fetching from primary source
        // In real implementation, this would call the database
        Ok(Product {
            id: id.to_string(),
            name: format!("Product {}", id),
            price: 29.99,
            reviews: vec![
                Review { rating: 5, comment: "Great!".to_string() },
            ],
            recommendations: vec!["rec1".to_string(), "rec2".to_string()],
        })
    }

    async fn get_from_cache(&self, id: &str) -> Option<Product> {
        self.cache.read().await.get(id).cloned()
    }

    async fn get_realtime_price(&self, _id: &str) -> Result<f64, ServiceError> {
        // Would call pricing service
        Ok(29.99)
    }

    async fn get_recommendations(&self, _id: &str) -> Result<Vec<String>, ServiceError> {
        // Would call recommendation service
        Ok(vec!["rec1".to_string(), "rec2".to_string()])
    }
}

#[derive(Debug)]
pub struct ProductResponse {
    pub id: String,
    pub name: String,
    pub price: Option<f64>,
    pub reviews: Option<Vec<Review>>,
    pub recommendations: Option<Vec<String>>,
}

#[derive(Debug)]
pub enum ServiceError {
    NotFound,
    ServiceUnavailable,
    Timeout,
}

/// Health-based automatic degradation
pub struct AutoDegrader {
    degradation: Arc<DegradationManager>,
    thresholds: DegradationThresholds,
}

#[derive(Debug, Clone)]
pub struct DegradationThresholds {
    /// Error rate to trigger reduced mode
    pub reduced_error_rate: f64,
    /// Error rate to trigger minimal mode
    pub minimal_error_rate: f64,
    /// Latency to trigger reduced mode (ms)
    pub reduced_latency_ms: u64,
    /// Latency to trigger minimal mode (ms)
    pub minimal_latency_ms: u64,
}

impl Default for DegradationThresholds {
    fn default() -> Self {
        DegradationThresholds {
            reduced_error_rate: 0.05,  // 5% errors
            minimal_error_rate: 0.20,  // 20% errors
            reduced_latency_ms: 500,   // 500ms p99
            minimal_latency_ms: 2000,  // 2s p99
        }
    }
}

impl AutoDegrader {
    pub fn new(
        degradation: Arc<DegradationManager>,
        thresholds: DegradationThresholds,
    ) -> Self {
        AutoDegrader {
            degradation,
            thresholds,
        }
    }

    pub async fn evaluate(&self, metrics: &SystemMetrics) {
        let new_level = self.calculate_level(metrics);
        let current = self.degradation.current_level().await;

        if new_level != current {
            self.degradation.set_level(new_level).await;
        }
    }

    fn calculate_level(&self, metrics: &SystemMetrics) -> DegradationLevel {
        if metrics.error_rate >= self.thresholds.minimal_error_rate
            || metrics.p99_latency_ms >= self.thresholds.minimal_latency_ms
        {
            DegradationLevel::Minimal
        } else if metrics.error_rate >= self.thresholds.reduced_error_rate
            || metrics.p99_latency_ms >= self.thresholds.reduced_latency_ms
        {
            DegradationLevel::Reduced
        } else {
            DegradationLevel::Normal
        }
    }
}

pub struct SystemMetrics {
    pub error_rate: f64,
    pub p99_latency_ms: u64,
    pub cpu_usage: f64,
    pub memory_usage: f64,
}

fn main() {
    println!("Graceful Degradation Pattern");

    println!("\nDegradation Levels:");
    println!("  Normal  - All features enabled");
    println!("  Reduced - Non-essential features disabled");
    println!("  Minimal - Core features only");
    println!("  ReadOnly - No write operations");
    println!("  Offline - Service unavailable");

    println!("\nFeatures that can degrade:");
    println!("  - Search (core)");
    println!("  - Real-time Pricing (core)");
    println!("  - Recommendations (optional)");
    println!("  - User Reviews (optional)");
    println!("  - Personalization (optional)");
    println!("  - Analytics (optional)");
    println!("  - External Integrations (optional)");
}

Why This Works

  1. Partial availability: Users get something instead of nothing
  2. Priority features: Core functionality preserved
  3. Automatic adaptation: System responds to failures
  4. Clear boundaries: Features explicitly categorized

Degradation Strategies

| Strategy | Description | Example |

|----------|-------------|---------|

| Feature Flags | Disable non-essential features | Turn off recommendations |

| Caching | Serve stale data | Show cached prices |

| Default Values | Return safe defaults | Empty suggestions list |

| Read-only | Disable writes | Browse but can't order |

| Static Content | Serve pre-generated pages | Static product catalog |

⚠️ Anti-patterns

// DON'T: Binary availability
async fn get_product(id: &str) -> Result<Product, Error> {
    let product = db.get(id).await?;           // Fails
    let price = pricing.get(id).await?;        // Never reached
    let reviews = reviews.get(id).await?;      // Never reached
    // All or nothing - if any fails, everything fails
}

// DON'T: Silent degradation
async fn get_recommendations() -> Vec<String> {
    recommendations.get().await.unwrap_or_default()
    // User doesn't know recommendations are unavailable
}

// DO: Explicit degradation with user feedback
async fn get_recommendations() -> RecommendationsResponse {
    match recommendations.get().await {
        Ok(recs) => RecommendationsResponse::Available(recs),
        Err(_) => RecommendationsResponse::Degraded {
            message: "Personalized recommendations temporarily unavailable",
            fallback: get_popular_items().await,
        }
    }
}

// DO: Cascading fallbacks
async fn get_price(id: &str) -> f64 {
    // Try real-time pricing
    if let Ok(price) = realtime_pricing.get(id).await {
        return price;
    }
    // Fall back to cached pricing
    if let Some(price) = cache.get_price(id).await {
        return price;
    }
    // Fall back to base price from database
    db.get_base_price(id).await.unwrap_or(0.0)
}

Exercises

  1. Implement circuit breaker integration with degradation manager
  2. Add degradation notifications (webhooks, alerts)
  3. Create dashboard showing current degradation state
  4. Implement graceful degradation for a distributed system

🎮 Try it Yourself

🎮

Graceful Degradation - Playground

Run this code in the official Rust Playground