Capture Network Requests With Ichrome And Regex

Alex Johnson

-Oct 26, 2025

Capture Network Requests With Ichrome And Regex

How to Use ichrome to Capture Requests That Match a Specified Regular Expression and Obtain Their Packets

Introduction to ichrome and Network Request Capture

In the realm of web automation and data extraction, ichrome stands out as a powerful tool, especially when combined with Python's asynchronous capabilities. ichrome simplifies interacting with Chrome's DevTools Protocol, enabling you to automate browser actions, inspect network traffic, and extract valuable data. This article delves into how you can leverage ichrome to capture specific network requests based on a regular expression (regex) and retrieve their associated packets. This is particularly useful when you need to monitor or analyze specific types of data being transferred between a web page and a server.

When delving into web automation and data extraction, ichrome emerges as a robust and efficient tool, particularly when synergized with Python's asynchronous features. At its core, ichrome streamlines the process of interacting with Chrome's DevTools Protocol, allowing you to automate browser actions, meticulously inspect network traffic, and extract invaluable data with ease. In essence, this article serves as a comprehensive guide, elucidating how you can harness the capabilities of ichrome to capture network requests that precisely match a predefined regular expression (regex), and subsequently retrieve their corresponding packets. This functionality proves exceptionally beneficial in scenarios where there is a need to monitor or analyze specific data exchanges between a web page and a server, offering insights into the intricacies of web communication. With ichrome, developers and researchers alike can gain unprecedented visibility into network interactions, empowering them to optimize web applications and extract meaningful intelligence from web traffic. Understanding how to capture network requests using ichrome is essential for tasks such as debugging web applications, analyzing API calls, and monitoring data streams. This involves setting up ichrome, enabling network monitoring, and defining the regular expression to filter the desired requests. Furthermore, once a matching request is found, you can retrieve the detailed packet information for further analysis. This article provides a step-by-step guide to accomplish this, complete with code examples and explanations.

Setting Up Asynchronous Chrome with ichrome

Before diving into the code, ensure you have ichrome installed. You can install it using pip:

pip install ichrome

Next, you need to set up an asynchronous Chrome instance using AsyncChromeDaemon. This class manages the Chrome process and allows you to connect to it asynchronously. The following code snippet demonstrates how to initialize and connect to a Chrome tab:

import asyncio
from ichrome import AsyncChromeDaemon
import json
import re


async def main():
    async with AsyncChromeDaemon(headless=False) as cd:
        async with cd.connect_tab() as tab:
            # Your code will go here
            pass

if __name__ == "__main__":
    asyncio.run(main())

In this setup, AsyncChromeDaemon(headless=False) starts Chrome in a non-headless mode, meaning you'll see the browser window. The connect_tab() method establishes a connection to a new tab in Chrome, providing an interface to send commands and receive events. Setting up the asynchronous Chrome environment involves several crucial steps to ensure seamless interaction and efficient automation. Firstly, it's imperative to have ichrome properly installed, which can be easily accomplished using pip, the Python package installer. This installation process equips your system with the necessary libraries and dependencies to interface with Chrome's DevTools Protocol. Following the installation, the next step is to initialize an asynchronous Chrome instance using the AsyncChromeDaemon class. This class serves as the backbone for managing the Chrome process and facilitates asynchronous connections, enabling you to perform tasks concurrently without blocking the main thread. Within the code snippet, the AsyncChromeDaemon(headless=False) configuration starts Chrome in a non-headless mode, allowing you to visually observe the browser window and monitor its behavior. The connect_tab() method then establishes a connection to a new tab in Chrome, providing a conduit for sending commands and receiving events from the browser. This setup forms the foundation for automating browser actions, inspecting network traffic, and extracting valuable data, all while maintaining responsiveness and efficiency. With the asynchronous Chrome environment configured, you're ready to implement the logic for capturing network requests and analyzing their contents, as outlined in subsequent sections.

Enabling Network Monitoring

To capture network requests, you need to enable network monitoring in the Chrome DevTools. This is done using the Network.enable command. Once enabled, Chrome will start sending network-related events to your script. Here’s how to enable network monitoring:

await tab.send("Network.enable")

This line sends the Network.enable command to the Chrome tab, instructing it to start monitoring network activity. After enabling network monitoring, Chrome diligently begins transmitting network-related events to your script, providing real-time insights into the communication between the browser and remote servers. Enabling network monitoring is a fundamental step in the process of capturing network requests with ichrome, as it establishes the foundation for intercepting and analyzing web traffic. By sending the Network.enable command to the Chrome tab, you effectively activate the browser's network interception capabilities, allowing you to observe and scrutinize the data being exchanged between the browser and remote servers. This includes details such as HTTP requests, responses, headers, and payloads, providing a comprehensive view of the network interactions. With network monitoring enabled, your script gains access to a wealth of information that can be used for debugging, performance analysis, security auditing, and various other purposes. This foundational step sets the stage for more advanced operations, such as filtering network requests based on specific criteria and extracting relevant data for further processing.

Capturing Requests with Regular Expressions

To capture requests that match a specific regular expression, you need to listen for network events and filter them based on your regex. The primary event for network requests is Network.requestWillBeSent. You can listen for this event and apply your regex to the request URL. Below is an example of how to do this:

regex = re.compile(r".*your_regex_here.*")

async def capture_requests(tab, regex):
    while True:
        event = await tab.wait_event("Network.requestWillBeSent")
        request_url = event["request"]["url"]
        if regex.match(request_url):
            print(f"Matching request found: {request_url}")
            # Process the request here
            pass

# Start capturing requests in a separate task
asyncio.create_task(capture_requests(tab, regex))

In this code, re.compile(r".*your_regex_here.*") compiles your regular expression. The capture_requests function continuously listens for Network.requestWillBeSent events. When an event is received, it extracts the request URL and checks if it matches the regex. If it matches, it prints the URL and allows you to add further processing logic. To effectively capture requests with regular expressions, you need to implement a robust mechanism for listening to network events and filtering them based on your defined criteria. The Network.requestWillBeSent event serves as the primary source of information for network requests, providing details about the URL, headers, and other relevant metadata. By listening for this event and applying your regular expression to the request URL, you can selectively capture requests that match your specified pattern. The code example demonstrates how to compile a regular expression using re.compile() and then define an asynchronous function, capture_requests, to continuously monitor network events. Within this function, the await tab.wait_event("Network.requestWillBeSent") call suspends execution until a new network request event is received. Once an event is received, the code extracts the request URL and applies the regular expression using the regex.match() method. If the URL matches the regular expression, the code executes the desired processing logic, which could include printing the URL, logging the request details, or performing further analysis. To ensure that the request capturing process runs concurrently without blocking the main thread, the asyncio.create_task() function is used to launch the capture_requests function as a separate task. This allows the main program to continue executing while the request capturing process operates in the background, providing a seamless and responsive user experience.

Obtaining Packet Information

To obtain the packet information for a captured request, you can use the Network.getResponseBody command. This command requires the requestId of the request, which is available in the Network.requestWillBeSent event. Here’s how to retrieve the response body:

async def capture_requests(tab, regex):
    while True:
        event = await tab.wait_event("Network.requestWillBeSent")
        request_id = event["requestId"]
        request_url = event["request"]["url"]
        if regex.match(request_url):
            print(f"Matching request found: {request_url}")
            try:
                response = await tab.send("Network.getResponseBody", {"requestId": request_id})
                body = response["body"]
                print(f"Response body: {body[:200]}...")  # Print first 200 characters
            except Exception as e:
                print(f"Error getting response body: {e}")

In this enhanced code, we extract the requestId from the Network.requestWillBeSent event. Then, we use tab.send to call the Network.getResponseBody method, passing the requestId. The response contains the body of the response, which you can then process as needed. A try-except block is used to handle potential errors when retrieving the response body. Obtaining packet information for captured requests involves leveraging the Network.getResponseBody command, which enables you to retrieve the contents of the response associated with a specific network request. This process requires the requestId of the request, which is conveniently available in the Network.requestWillBeSent event. By extracting the requestId and passing it to the Network.getResponseBody command, you can retrieve the response body and gain insights into the data being transmitted. The enhanced code demonstrates how to extract the requestId from the Network.requestWillBeSent event and then use tab.send to invoke the Network.getResponseBody method, passing the requestId as a parameter. The response from this command contains the body of the response, which represents the actual data being returned by the server. Once you have obtained the response body, you can process it as needed, such as parsing it, analyzing its contents, or extracting specific information. To ensure robustness and error handling, the code includes a try-except block to catch potential exceptions that may occur during the retrieval of the response body. This allows the program to gracefully handle errors and prevent unexpected crashes. Overall, obtaining packet information is a crucial step in the process of capturing and analyzing network requests, as it provides access to the actual data being transmitted between the browser and the server, enabling you to gain deeper insights into the communication process.

Complete Example

Here is the complete code example:

import asyncio
from ichrome import AsyncChromeDaemon
import json
import re


async def main():
    async with AsyncChromeDaemon(headless=False) as cd:
        async with cd.connect_tab() as tab:
            await tab.send("Network.enable")
            regex = re.compile(r".*tiktok.*")

            async def capture_requests(tab, regex):
                while True:
                    event = await tab.wait_event("Network.requestWillBeSent")
                    request_id = event["requestId"]
                    request_url = event["request"]["url"]
                    if regex.match(request_url):
                        print(f"Matching request found: {request_url}")
                        try:
                            response = await tab.send("Network.getResponseBody", {"requestId": request_id})
                            body = response["body"]
                            print(f"Response body: {body[:200]}...")  # Print first 200 characters
                        except Exception as e:
                            print(f"Error getting response body: {e}")

            asyncio.create_task(capture_requests(tab, regex))
            await tab.goto("https://www.tiktok.com/@samsungindonesia/photo/7561265486105906488?is_from_webapp=1", timeout=10)
            await asyncio.sleep(5)  # Keep the script running for a while to capture requests

if __name__ == "__main__":
    asyncio.run(main())

This complete example combines all the steps discussed above. It initializes ichrome, enables network monitoring, defines a regular expression to match TikTok-related requests, captures the requests, retrieves their response bodies, and prints the first 200 characters of each response. The script navigates to a TikTok page and keeps running for a few seconds to capture network requests. This consolidated example serves as a practical demonstration of how to effectively utilize ichrome for capturing network requests that match a specified regular expression and obtaining their associated packets. By combining all the steps discussed previously, this example provides a comprehensive solution for monitoring and analyzing network traffic in a targeted manner. The code initializes ichrome, enables network monitoring, defines a regular expression to specifically match TikTok-related requests, captures the requests, retrieves their response bodies, and prints the first 200 characters of each response for quick inspection. Furthermore, the script navigates to a TikTok page and remains active for a few seconds to ensure that network requests are captured during the page's loading process. This complete example offers a solid foundation for building more sophisticated network analysis tools and applications, allowing you to gain deeper insights into the communication between web browsers and remote servers.

Conclusion

Using ichrome to capture network requests that match a specified regular expression and obtain their packets involves setting up an asynchronous Chrome instance, enabling network monitoring, listening for network events, filtering requests based on a regex, and retrieving the response bodies. This approach is invaluable for debugging, analyzing, and monitoring web applications. By following the steps and code examples provided in this article, you can effectively leverage ichrome to gain deeper insights into network traffic and extract valuable data. With the ability to capture and analyze specific network requests, you can enhance your web development workflow, optimize application performance, and ensure the security and integrity of your web applications. Understanding how to use ichrome in this way opens up a wide range of possibilities for web automation and data extraction, empowering you to build more sophisticated and efficient solutions. By mastering these techniques, you can unlock the full potential of ichrome and leverage its capabilities to address a variety of real-world challenges.

For more information on Chrome DevTools Protocol, visit the official documentation.