How I Built APIs That Respond in Under 80ms for 10K+ Colleges

Hello guys, Avinash this side. Today I want to talk about how I managed to make our college APIs fast and bring the response time down to **under 80ms**, even though we were dealing with data for **10,000+ colleges**, with multiple departments and a huge amount of related information. We had built a website called **SikshaHelpline.com**, which helps students after 12th find and compare colleges based on their programs, eligibility criteria, fees, departments, and other important information. The main challenge was the **amount of data**. We had 10K+ colleges, and each college could have multiple departments, courses, fees, eligibility criteria, and many other details. On the homepage alone, we had multiple sections that required different types of college data. Initially, if we tried to call multiple APIs for every section and then process, calculate, and extract the required data on the client side, the homepage could become unresponsive — even on high-end devices. The server also had a problem. The amount of data being queried and processed was huge, so putting everything on a single server instance and querying the database for every request could eventually make the instance unresponsive. So, what did I do? First, I worked heavily on **database indexing**. I identified the fields that were frequently used for filtering, sorting, and searching and created appropriate indexes to reduce the amount of data the database had to scan. Then, instead of fetching huge datasets and processing everything on the application side, I created multiple **aggregation pipelines at the DB level**. This allowed the database to perform filtering, grouping, calculations, and data extraction before sending the response back to the application. The biggest improvement came from **Redis caching**. Instead of querying the database every time a user requested the same data, we stored the processed results in Redis and served subsequent requests directly from the cache. I also implemented a **fresh-cache/background refresh approach**. Even if no user requested a particular piece of data, a background process would refresh the Redis cache periodically — in our case, every minute. It would check whether the underlying data had changed, and if there was an update, Redis would refresh the cached data. So when a user requested the data, we didn't have to perform the entire database query and aggregation again. We could simply serve the latest processed data from Redis. The overall flow became something like: **Database → Aggregation → Redis → API → User** instead of: **User → API → Database → Huge Query → Aggregation → API Response** This significantly reduced the load on the database and application server and made the APIs much faster. The result? We were able to bring the API response time down to **under 80ms** for these use cases, while still dealing with data from **10K+ colleges and their multiple departments/courses**. For me, the biggest learning from this was that when an application becomes slow, the solution isn't always “use a bigger server.” Sometimes it's about **how you query the database, where you process the data, what you cache, and when you refresh that cache.**