How we captured AJAX requests from a website tab with a Chrome Extension
Background
I assumed there would be a lot of Chrome extension to monitor HTTP requests from AJAX calls, but the few that we found such as the Postman Intercept Chrome Extension seems to only capture the request only and not the response. Even though Chrome DevTools has the network tab, it hard to share those captured HTTP traces with teammates.
Our engineers divide responsibilities between the front-end and back-end so we constantly have to copy/paste the HTTP headers and JSON payload from the DevTools into email or Slack when debugging API issues. Sending multi-megabyte *.har files through Slack is just as cumbersome. In addition to not being easy to share, inspecting HTTP traffic in Chrome DevTools is not ideal because there is no easy way to filter the API Calls or inspect JSON payloads. The network tab is designed for traditional HTML websites rather than modern Single-Page Apps and APIs that return JSON.
Due to this time sink, we decided to build a Chrome Extension to make capturing and debugging these AJAX requests (and responses) from any website real easy. The extension is designed for REST APIs in mind such as those powering Single-Page Apps. However, some monkey business was needed to do the capturing that went around the Chrome Extension APIs due to limitations in Chrome’s APIs.
Issues with WebRequest approach
Chrome WebRequest API is part of the Chrome Extension set of APIs.
Initially, I looked at the Chrome’s WebRequest APIs to capture API calls from an open browser tab. We leveraged those APIs before for our CORS and Origin Changer. It seemed to be the most natural place to capture the data.
However, we ran into a bug where the WebRequest API doesn’t expose an interface to read the response body. See the issue here on Chromium.
While we were surprised by the lack of a response body getters, we now understand a possible reason why most other extensions do not capture the responses. Our requirement was to capture the response so that they could be shared with team members. Without responses, debug would be painful and resolving issues would be slow.
XmlHttpRequest event listeners approach
Since XmlHttpRequest is used by all browsers for AJAX calls, I leveraged this to monitor the API calls through monkey patching.
There are several XmlHttpRequest listeners for hooking into:
- loadstart
- progress
- abort
- error
- load
- timeout
- loadend
- readystatechange
From the listener callbacks, I can get the event target, which is the XmlHttpRequest
object itself. Looking at the XmlHttpRequest api documentation, I realized it has exact the opposite
problem as WebRequest API. The object has interfaces to get the HTTP response but not the original
request. There is a setRequestHeaders(), but no getter method to pair with it to get the request
HTTP headers. In fact, it doesn’t even have a public method or property to get the original URL or path.
XmlHttpRequest Monkey patch.
To truly monitor all the data (i.e. request headers/body and response headers/body), I’d have to monkey patch it. The monkey patch allows us to log the data.
The key methods to patch are
open()This method is called whenever a new AJAX call is initiated which I can then capture the method and url.setRequestHeader()This method is called when the request header is set. I can patch it so that every time the method is called, I save the headers as hash.send() This method is used to send data as part of the request. I can capture the request body from this patch.
Now, I can simply add addEventListener('load', event) to capture the response data.
There are a few resources on the web such as this article explaining how to monkey patch.
Execution Environment
With code ready for monkey patching, we have to execute it. Chrome Extension Tabs API has a method to execute the code: executeScript(). However, the code executed by executeScript() is executed in a different context than the code on the
webpage, which is where the AJAX calls are made. Meaning the monkey patch won’t have any effect on the open website.
To learn more about how execution environment works, see this video on content scripts.
The solution: Injected script
Due to the isolated context, I had to take a different approach to successfully monkey patch the website.
I had to use chrome.tabs.executeScript() to create a <script> tag injected on the website which would load the monkey patch
code inside the website’s context.
const actualCode = `
var s = document.createElement('script');
s.src = ${resourceUrl};
s.onload = function() {
this.remove();
};
(document.head || document.documentElement).appendChild(s);
`;
chrome.tabs.executeScript(tabId, {code: actualCode, runAt: 'document_end'}, cb);
There are several ways to add the code into the script tag. Since we also wanted to add some nice UI
elements, we decided to put the monkey patch code and the UI related code into a script file, which the resourceUrl links to.
If you do this, be sure to add the resource url to the web_accessible_resource in
the manifest. More information on Web Accessible Resources
Known issues we punted on
CSS leaking
Since the code is running as part of the web page, the UI we generated can be impacted by the CSS that is already on the page. We are using React and Material-UI library, which uses inline styling to minimize the CSS leaking. However, the CSS leaking still can happen on some websites.
Content Security Policy: content-src
Content Security Policies is
another issue. If content-src is set to be very strict such as on GitHub.com, then it can blocks AJAX calls to servers that are not in a whitelist. Since we wanted to log and store the API calls to be retrieved from shareable workspace links on ApiRequest.io, we need to make POSTS to our own API. This allows the HTTP trace to be saved for 30-days as a reference during later debug.
The Chrome Extension execution environment is not fully subject to that restriction. There is a solution to modify the Content Security Policy which will be posted as a separate article.
At the moment, many website are not using such stringent Content Security Policies outside a few big ones like GitHub and Twitter. We are assuming that if you are using our Chrome Extension, then it probably is used on your own website. If so, then you can turn that restriction off temporarily. Email us if you have issues or feature requests for our Chrome extension.
Update: Content Security Policy Issue Solved
Now the Apirequest.io Chome Extension works on website with strict Content Security Policy. We created another article on how to work around the Content Security Policy, it also have a more in-depth explaination of different execution environments of Chrome Extensions.
Closing Thoughts
I had fun creating this Chrome Extension, so I hope you enjoy using it. As always, for any feature requests or issues when using the extension, just let us know by emailing us.
Do you spend a lot of time debugging customer issues?
Moesif makes debugging easier for RESTful APIs and integrated apps
Learn More
Related Articles
API Analytics and Monitoring
Introducing AI Explain
We are thrilled to introduce our newest feature to Moesif - AI Explain - a powerful conversational tool that lets you interact with your API analytics data l...
API Analytics and Monitoring
Monitoring Cost and Consumption of AI APIs and Apps
Discover how Moesif can help manage and optimize your AI-driven APIs with powerful observability tools that track costs and usage efficiently.
API Development
Unlock Deep API Insights with Moesif’s HTTP Body Analytics
Learn how to use Moesif to analyze HTTP bodies.
API Analytics and Monitoring
How Moesif API Observability Compares to Treblle
Learn how Moesif’s API observability platform compares to Treblle.