Getting a Higher Rate Limit - UWFlow Importer

[jerryzhou196]

Request: Elevated rate limit for UW Flow importer

Who we are

UW Flow is a student-built course discovery and review platform serving 30,000+ daily active users across the
University of Waterloo undergraduate population. Course catalog, schedule, and term data are sourced directly from the UW Open Data API v3.

What we're requesting

A dedicated API key (or rate-limit uplift on our existing UW_API_KEY_V3) that allows our importer to complete a full sync without being throttled. Today we routinely hit HTTP 429 on the ClassSchedules endpoint and rely on exponential backoff to recover, which extends sync time and risks stale data for end users.

Our usage profile

Schedule

The importer is a Go service running in a single container, driven by crond. It executes two jobs (times in UTC):

Cron	Command	Purpose
20 */2 * * *	/app/uw hourly	Refresh terms, courses, class schedules
30 05 * * *	/app/uw vacuum	Daily DB cleanup (no API calls)

So the API is called once every 2 hours, on the :20 mark — 12 runs per day total.

Endpoints hit per hourly run

All against https://openapi.data.uwaterloo.ca/v3:

1. GET /ImportantDates — 1 request, fetches term metadata.
2. GET /Courses/{termId} — 4 requests, one per term in our window (previous-2, previous-1, current, next).
3. GET /ClassSchedules/{termId}/{subject}/{catalogNumber} — the bulk of the load: roughly unique_courses × 4 terms requests,
   currently several thousand per run. These are issued sequentially with a 150 ms delay between requests; 404s are tolerated, 429s are retried
   with exponential backoff (up to 5 attempts).

Volume estimate

• ~12 runs/day × ~(1 + 4 + N_classSchedules) requests
• ClassSchedules dominates: typically several thousand calls per run

Why an elevated limit helps

• Faster, more reliable syncs → fresher data for 30k+ students checking sections during peak enrollment windows.
• Fewer retries → less wasted load on your side from our backoff loops.
• Predictable behavior we can document for our open-source contributors.

Contact

Happy to share our source (the importer lives at [flow/importer/uw (https://github.com/UWFlow/uwflow/tree/main/flow/importer/uw) in our public repo) or adjust our request pattern (batching, off-peak windows, etc.) to fit your constraints. Please let us know what you need from us.

[sbobkin]

Hi,

Thanks for reaching out. I'll start with a bit of context; Open Data doesn't currently have a support or governance model to live under. Effectively, there is no value or leadership drive to spend significant resources. This is compounded by some of the budget problems that UW has been communicating.

With that out of the way, there are some improvements that can be made on our end and your end.

1. From your description it looks like you query class schedules for every course entry you get back from the courses endpoint. Not every course will have a schedule in a term. We've brought this up before with previous maintainers. We provide an endpoint on class schedules https://openapi.data.uwaterloo.ca/v3/ClassSchedules/{termID} that gives an array of CourseIDs that do have a schedule. If you query just those it will reduce the amount of requests you need to do.
2. There is no good reason to query the current-1, and current-2 terms on your hourly schedule. The backdating of data happens, but it's very rare, and even more rare on the course and schedule data. If you run it weekly, or even monthly, it would functionally be the same.
3. I've increased the request limiter on or about April 24th. The current limiter resets on a per minute basis. We may increase it in the future as there are some backend changes that could support this, but I can't make promises.

Lets start there.