

But with the rise of AI, the dynamic is changing: We are observing a significant increase in request volume, with most of this traffic being driven by scraping bots collecting training data for large language models (LLMs) and other use cases. Automated requests for our content have grown exponentially, alongside the broader technology economy, via mechanisms including scraping, APIs, and bulk downloads. This expansion happened largely without sufficient attribution, which is key to drive new users to participate in the movement, and is causing a significant load on the underlying infrastructure that keeps our sites available for everyone.
- https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/
If you’re building a website, you’ll probably want to stick to Javascript over Rust.
This MDN article does a pretty good job at introducing the concept of making network requests in Javascript: https://developer.mozilla.org/en-US/docs/Learn_web_development/Core/Scripting/Network_requests. It focuses on the “fetch” API as the tool for making requests, which is the standard way to make network requests in Javascript. There are other tools like Axios that may make things easier, but “fetch” should be fine for your use case.
Another concept that will be relevant here is asynchronous programming: https://developer.mozilla.org/en-US/docs/Learn_web_development/Extensions/Async_JS/Introducing. Basically, there will be some delay between when you make the request and when you get a response. So you’ll need to write your logic in a way that does the “waiting” part correctly.
One important detail is that most APIs use some form of authentication. So when you’re “grabbing the data” from an external site, the site knows who you are and that you are allowed to access that data. Getting authentication right might be a little tricky, but here is an entry point: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Authentication. Basically, you’ll need to figure out what authentication strategy your headless CMS is using, and then make sure to safely pass those credentials when making your network requests. If the API(s) you are using are public, you won’t need to worry about this.
If your goal is primarily to get data from an external source, this should be a good starting point. You don’t necessarily need to get too deep into the backend or even the technical details of things like HTTP or REST. However, if you’re interesting in getting a deeper understanding of Web APIs, the other comment talking about building a skeleton API would be a good exercise.