Simplify Venue Image Logic For Efficiency

Streamlining Venue Image Enrichment is crucial for maintaining a high-quality user experience. The current system, however, has become overly complex, leading to inefficiencies and wasted resources. This article delves into the intricacies of the existing logic, identifies key problems, and proposes solutions to create a more streamlined and effective process.

The Complexity Conundrum: Timestamp Troubles and Cooldown Confusion

The existing system suffers from a few critical issues. The primary problem lies in the unnecessary complexity of the timestamp fields. The system uses multiple timestamp fields and inconsistent cooldown periods. Additionally, it contains redundant checks that often clash with each other. This creates a situation where jobs are created only to be immediately skipped, which is a waste of valuable resources.

Timestamp Fields: Redundancy and Inefficiency

Currently, both last_enriched_at and last_attempt_at timestamps are set to the same value every time an enrichment attempt is made. This behavior occurs regardless of whether the attempt succeeds or fails. This redundancy adds complexity without providing any real benefits. It makes the code harder to understand, maintain, and debug. The presence of two identical timestamps for every attempt needlessly complicates the logic and increases the potential for errors.

Multiple Staleness Checks: A Recipe for Confusion

The system employs several staleness checks, each with its own specific logic. This creates confusion and makes it difficult to understand the overall process. This creates a fragmented and inconsistent approach to determining when an image enrichment is needed. The different parts of the code use different thresholds. This results in unpredictable behavior and makes it difficult to manage the image enrichment process effectively.

Check #1: Orchestrator.needs_enrichment?/1: This check uses last_enriched_at to determine if a venue needs enrichment. Venues with images have a 90-day cooldown period. Venues without images have a shorter, 7-day cooldown. The core logic here is based on the time since the last enrichment.

Check #2: BackfillOrchestrator.find_venues_without_images: This check focuses on venues without images, using last_attempt_at and last_attempt_result. It skips venues if the last attempt resulted in "no_images" and was within the 7-day cooldown. However, it immediately retries if the result was "error" or "success".

Check #3: EnrichmentJob.perform/1: This check duplicates the logic of Orchestrator.needs_enrichment?. This double-checking approach creates significant inefficiency.

Problems Unpacked: Deep Dive into the Core Issues

Let's break down the core problems:

Problem 1: Redundant Timestamp Fields

As previously mentioned, the use of both last_enriched_at and last_attempt_at is redundant. Both fields are updated with the same timestamp. This redundancy adds unnecessary complexity.

Problem 2: Inconsistent Cooldown Periods

The system uses different cooldown periods depending on the scenario. Venues with images have a 90-day cooldown. Venues without images have a 7-day cooldown. The code uses different thresholds in different places, leading to inconsistent behavior. The lack of a unified cooldown period makes it difficult to predict when an image enrichment will be attempted.

Problem 3: Complex Conditional Logic in BackfillOrchestrator

The SQL in BackfillOrchestrator uses complex conditional logic, which is hard to understand and debug. The logic determines whether a venue should be enqueued for image enrichment. The conditional statements make it difficult to reason about the system's behavior. This complexity increases the risk of errors and makes it difficult to maintain the code.

Problem 4: Double-Checking and Wasted Jobs

The double-checking approach leads to wasted resources. The BackfillOrchestrator filters venues based on last_attempt_at and last_attempt_result. This creates jobs that are then immediately skipped because EnrichmentJob.perform performs a second check using last_enriched_at. Since both timestamps are usually the same, the logic should align, but it often does not.

The Path Forward: Desired Behavior and Streamlined Solutions

The goal is to prioritize venues that have never been checked, or those that haven't been checked in a long time. We want to avoid re-checking venues that have been recently processed, regardless of the outcome.

Priority Order

1. Never checked before: Highest priority (no last_attempt_at).
2. Not checked in a long time: Stale based on a single cooldown period.
3. SKIP: Recently checked: Regardless of success, failure, or error.

Simple Logic

• Use ONE timestamp: When was the venue last checked?
• Use ONE cooldown period: How long to wait before re-checking?
• Don't care about success/failure: If checked recently, skip it.

Option A: Single Timestamp + Single Cooldown

Use a single timestamp field, last_checked_at, to record when a venue was last checked. This simplifies the logic and makes it easier to understand.

# Metadata structure
%{
  "last_checked_at" => DateTime.to_iso8601(now),  # Unified timestamp
  "last_check_result" => "success" | "no_images" | "error",
  "image_count" => count,
  # ... other fields
}

# Staleness check
def needs_enrichment?(venue) do
  metadata = venue.image_enrichment_metadata || %{}
  last_checked = metadata["last_checked_at"]

  if is_nil(last_checked) do
    true  # Never checked
  else
    days_since_check = DateTime.diff(DateTime.utc_now(), last_checked, :day)
    cooldown_days = Application.get_env(:eventasaurus, :venue_images)[:enrichment_cooldown_days] || 30
    days_since_check > cooldown_days
  end
end

Benefits:

• Single source of truth for when a venue was last checked.
• Simplified, understandable logic.
• No double-checking.
• No conflicts between different parts of the system.

Option B: Keep Separate Fields but Unify Logic

If it's necessary to track both "last successful enrichment" and "last attempt", then use last_attempt_at consistently for all staleness checks. last_enriched_at can be kept for reporting or analytics.

# Use last_attempt_at for ALL staleness checks
# last_enriched_at is just metadata for reporting/analytics

def needs_enrichment?(venue) do
  metadata = venue.image_enrichment_metadata || %{}
  last_attempt = metadata["last_attempt_at"]  # Use this everywhere

  # Same simple cooldown logic
  # ...
end

Benefits:

• Keeps historical data.
• Consistent checking logic.
• Single cooldown period.

Key Questions and Actionable Steps

To move forward effectively, we need to address these questions and take the following steps.

Questions to Answer

1. What is the intended cooldown period? Is it 7 days, 30 days, or 90 days? Or does it vary based on the scenario?
2. Should we immediately retry "error" results? The current logic in BackfillOrchestrator allows this. Is this the intended behavior?
3. Should we immediately retry "success" results with no images? Does this align with the desired behavior?
4. Do we truly need separate last_enriched_at and last_attempt_at fields? If they are consistently set to the same value, why maintain both?
5. What defines "recently checked"? What is the time threshold for skipping venues that have already been checked?

Action Items

• Decide on a single cooldown period, or clearly document different periods for various scenarios.
• Determine whether to consolidate the timestamp fields or keep them separate.
• Update needs_enrichment? to use consistent, unified logic.
• Update BackfillOrchestrator.find_venues_without_images to align with the new logic.
• Ensure that there is no double-checking between the orchestrator and job execution.
• Clearly document the staleness policy in code comments.
• Add tests to cover various staleness edge cases.

By simplifying the logic, standardizing cooldown periods, and eliminating redundant checks, we can create a more efficient and reliable system for managing venue image enrichment. This will result in a better user experience and a more efficient use of resources. This will improve the overall performance of the system.

For additional information, consider exploring resources like Elixir's DateTime documentation to deepen your understanding of time-based operations within the context of this system.