goscraper

Goscraper is a backend that fetches and scrapes data from SRM Academia and parses it dynamically. The API is designed for performance, speed, and accuracy, accommodating changes in the HTML structure.

Important

This is part of a monorepo containing both frontend and backend components. Please fork the entire monorepo instead of cloning this backend separately.

Prerequisites

Before starting the application, you need to set up Supabase and configure the required environment variables.

1. Supabase Setup

First, create a new Supabase project and set up the required database tables:

Create the goscrape table:

create table public.goscrape (
  "regNumber" text not null,
  "user" text null,
  timetable text null,
  courses text null,
  attendance text null,
  marks text null,
  "lastUpdated" numeric null,
  token text not null,
  ophour text null default ''::text,
  constraint goscrape_pkey primary key ("regNumber", token),
  constraint goscrape_regNumber_key unique ("regNumber")
);

Create the gocal table:

create table public.gocal (
  id bigint generated by default as identity not null,
  date text null,
  month text null,
  day text null,
  "order" text null,
  event text null,
  created_at numeric null
);

Set up CRON Jobs

Warning

Install CRON Integration in Supabase first

Deleting old user data:

UPDATE goscrape
SET "user" = NULL, timetable = NULL, attendance = NULL, marks = NULL, courses = NULL
WHERE "lastUpdated" < (EXTRACT(EPOCH FROM NOW()) * 1000) - (12 * 60 * 60 * 1000);

Deleting old calendar:

CREATE OR REPLACE FUNCTION delete_from_gocal()
RETURNS void AS $$
BEGIN
    DELETE FROM gocal;
END;
$$ LANGUAGE plpgsql SECURITY INVOKER;

select
  cron.schedule (
    '0 0 * * *',
    'SELECT delete_old_calendar_events()'
  );

2. Environment Variables

Environment variables are handled by the monorepo. Please refer to the main repository's README.md for complete setup instructions.

---

WHEN DEPLOYING

Go to globals/DevMode and set the variable to false

---

Setup Instructions

1. Fork the monorepo:

   git clone --recurse-submodules https://github.com/rahuletto/classpro
   cd classpro/backend
   

2. Install dependencies:

   go mod tidy
   

3. Development Run the application: (DEV SERVER)

   go run main.go
   

4. Build and Run the application: (BUILD SERVER)

   go build main.go
   ./main
   

Docker Setup

docker compose up --build

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any enhancements or bug fixes.