Fixing Multi-Byte Characters In URL Facet Expansion

Alex Johnson

-Oct 29, 2025

Fixing Multi-Byte Characters In URL Facet Expansion

In this article, we delve into the crucial task of fixing multi-byte character handling within URL facet expansion. This is a significant issue that can impact how URLs are processed and displayed, especially in applications dealing with diverse character sets. We'll explore the problem, its impact, and a detailed solution, complete with code examples and tests.

Understanding the Issue: Multi-Byte Characters and URL Facet Expansion

When dealing with URL facet expansion, it's essential to correctly handle multi-byte characters. The core of the issue lies in how character positions are calculated within a string. Traditional methods often assume that each character occupies a single byte, which is true for basic ASCII characters. However, languages like Chinese, Japanese, and Korean (CJK), as well as symbols like emojis, use multi-byte characters, meaning they require more than one byte to represent a single character. This discrepancy can lead to incorrect calculations of string positions, resulting in misaligned or truncated URLs.

The current implementation in the ContentProcessor._expand_urls_from_facets() function has a limitation in handling these multi-byte characters. As highlighted in the original code:

# Note: This assumes UTF-8 encoding where most characters are 1 byte
# For more accurate conversion, we'd need to properly handle multi-byte chars

This acknowledgement points to the need for a more robust solution that accurately accounts for multi-byte characters. The impact of this issue is substantial. Facets, which use byte positions (byteStart, byteEnd), can become misaligned when encountering emojis or CJK characters. This misalignment can lead to URLs being incorrectly expanded, potentially corrupting links in posts and affecting the user experience. The consequences range from broken links to the display of malformed text, making it crucial to address this problem effectively.

To illustrate the issue, consider this scenario:

Post text: "🎉 Check out https://example.com..."
Facet byteStart: 7, byteEnd: 35
Problem: "🎉" is 4 bytes, not 1 character
Result: URL expansion may extract wrong substring

In this example, the emoji "🎉" occupies four bytes. If the code naively assumes one byte per character, the calculated positions for URL expansion will be off, leading to an incorrect substring extraction. This example underscores the importance of handling multi-byte characters correctly to ensure accurate and reliable URL facet expansion.

The Solution: Implementing Proper UTF-8 Byte-Level Handling

To address the multi-byte character issue, the solution involves implementing proper UTF-8 byte-level handling in the _expand_urls_from_facets() function. This approach ensures that character positions are calculated accurately, regardless of whether they are single-byte or multi-byte characters. The key is to work with the byte representation of the string rather than the character representation when determining the start and end positions for URL expansion.

The updated implementation of _expand_urls_from_facets() in src/content_processor.py is as follows:

@staticmethod
def _expand_urls_from_facets(text: str, facets: List[Dict[str, Any]]) -> str:
    """Expand truncated URLs using facets data from Bluesky
    
    Properly handles multi-byte UTF-8 characters in byte position calculations.
    """
    if not facets:
        return text
    
    # Convert text to bytes for accurate indexing
    text_bytes = text.encode('utf-8')
    
    # Process facets in reverse order to avoid index shifting
    sorted_facets = sorted(
        facets, 
        key=lambda f: f.get("index", {}).get("byteStart", 0), 
        reverse=True
    )
    
    for facet in sorted_facets:
        try:
            facet_index = facet.get("index", {})
            byte_start = facet_index.get("byteStart")
            byte_end = facet_index.get("byteEnd")
            
            if byte_start is None or byte_end is None:
                continue
            
            features = facet.get("features", [])
            for feature in features:
                feature_type = feature.get("$type", "") or feature.get("py_type", "")
                
                if "link" in feature_type.lower():
                    full_url = feature.get("uri")
                    if full_url:
                        # Replace at byte positions
                        text_bytes = (
                            text_bytes[:byte_start] +
                            full_url.encode('utf-8') +
                            text_bytes[byte_end:]
                        )
                        break
        except Exception as e:
            logger.warning(f"Error processing facet: {e}")
            continue
    
    # Decode back to string
    return text_bytes.decode('utf-8', errors='replace')

This updated code addresses the issue by first converting the input text to its byte representation using text.encode('utf-8'). This ensures that all subsequent operations work with byte positions, accurately accounting for multi-byte characters. The facets are processed in reverse order to avoid index shifting as URLs are expanded. For each facet, the code extracts the byte start and end positions, and if a link feature is found, it replaces the corresponding bytes in the text with the full URL. Finally, the modified byte string is decoded back into a string using text_bytes.decode('utf-8', errors='replace'), with error handling to gracefully manage any decoding issues.

Comprehensive Testing: Ensuring Correctness and Preventing Regressions

To ensure that the solution works correctly and to prevent regressions, comprehensive testing is essential. This involves creating test cases that specifically target multi-byte character scenarios, such as those involving emojis and CJK characters. These tests should verify that URLs are expanded correctly in the presence of these characters and that no existing functionality is broken.

The following tests were added to tests/test_content_processor.py:

def test_expand_urls_with_emoji():
    """Test URL expansion with emoji before URL"""
    text = "🎉 Check this out: example.co..."
    facets = [{
        "index": {"byteStart": 24, "byteEnd": 35},
        "features": [{
            "$type": "app.bsky.richtext.facet#link",
            "uri": "https://example.com/full-url"
        }]
    }]
    result = ContentProcessor._expand_urls_from_facets(text, facets)
    assert "https://example.com/full-url" in result

def test_expand_urls_with_cjk_characters():
    """Test URL expansion with Chinese/Japanese/Korean characters"""
    text = "こんにちは example.co..."
    facets = [{
        "index": {"byteStart": 18, "byteEnd": 29},
        "features": [{
            "$type": "app.bsky.richtext.facet#link",
            "uri": "https://example.com"
        }]
    }]
    result = ContentProcessor._expand_urls_from_facets(text, facets)
    assert "https://example.com" in result

The test_expand_urls_with_emoji() function tests URL expansion when an emoji precedes the URL. It checks if the URL is correctly expanded in this scenario. Similarly, the test_expand_urls_with_cjk_characters() function tests URL expansion with CJK characters, ensuring that the function correctly handles these characters as well. These tests are crucial for verifying the correctness of the multi-byte character handling implementation.

Validation and Acceptance Criteria

Validation is a critical step in ensuring the effectiveness of the solution. This involves running the full test suite to check for regressions and testing with real-world Bluesky posts that contain emojis and international characters. The goal is to verify URL expansion accuracy across different character sets and ensure that no existing functionality is compromised.

The acceptance criteria for this fix are:

[x] Byte-level string slicing implemented
[x] Tests pass with emoji before/after URLs
[x] Tests pass with CJK characters
[x] No regressions in existing URL expansion
[x] Code properly handles UTF-8 decoding errors

These criteria ensure that the solution is robust and addresses the multi-byte character handling issue effectively. Each criterion serves as a checkpoint to confirm that the implementation meets the required standards of accuracy and reliability.

Conclusion

Fixing multi-byte character handling in URL facet expansion is crucial for ensuring the accuracy and reliability of URL processing in applications dealing with diverse character sets. By implementing proper UTF-8 byte-level handling, we can correctly calculate character positions and expand URLs without issues. Comprehensive testing and validation are essential to prevent regressions and ensure that the solution works effectively in real-world scenarios. This fix enhances the user experience by preventing broken links and malformed text, making it a significant improvement.

For further information on text encoding and character handling, you can refer to the Unicode Consortium website.