diff --git a/COMPONENT_STRUCTURE.md b/COMPONENT_STRUCTURE.md new file mode 100644 index 00000000..eb9e7690 --- /dev/null +++ b/COMPONENT_STRUCTURE.md @@ -0,0 +1,504 @@ +# 📁 Component Structure Guide + +## 🎯 Current Structure (After Our Changes) + +``` +app/src/ +├── pages/ +│ ├── datasets.js ← 🆕 NEW! SEO-friendly route +│ ├── d/ +│ │ └── search.js ← Redirects to /datasets +│ └── api/ +│ └── search.js ← API endpoint (enhanced logging) +│ +├── components/ +│ └── search/ +│ ├── SearchContainer.js ← Container component (state management) +│ ├── DatasetSearchResults.jsx ← 🆕 NEW! Presentation component +│ ├── ResultCard.js ← List view card +│ ├── ResultGridCard.js ← Grid view card +│ ├── ResultTable.js ← Table view +│ ├── Filter.js ← Filter UI +│ ├── TagFilter.js ← Tag filter +│ └── Sort.js ← Sort dropdown +│ +├── search_configs/ +│ └── dataConfig.js ← Elasticsearch config +│ +├── services/ +│ └── SearchAPIConnector.js ← API communication +│ +└── utils/ + └── useNextRouting.js ← URL state management +``` + +--- + +## 🔄 Component Hierarchy + +``` +┌─────────────────────────────────────────┐ +│ pages/datasets.js │ ← Entry point (page) +│ - getStaticProps (translations) │ +│ - Meta tags (SEO) │ +│ - Column definitions │ +│ - Sort options │ +│ - Facet configuration │ +└────────────────┬────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────┐ +│ SearchContainer.js │ ← Container (smart component) +│ - SearchProvider setup │ +│ - View state (list/table/grid) │ +│ - Filter state (show/hide) │ +│ - Layout and controls │ +└────────────────┬────────────────────────┘ + │ + ├──────────────────────────┐ + │ │ + ▼ ▼ +┌────────────────────────┐ ┌────────────────────────┐ +│ Filter Components │ │ DatasetSearchResults │ ← 🆕 NEW! +│ - Facets │ │ - Error handling │ +│ - TagFilter │ │ - Empty states │ +│ - Sort dropdown │ │ - View switching │ +└────────────────────────┘ └───────┬────────────────┘ + │ + ┌───────────────┼───────────────┐ + │ │ │ + ▼ ▼ ▼ + ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ + │ ResultCard │ │ResultsTable │ │ResultGrid │ + │ (List view) │ │(Table view) │ │ Card │ + └─────────────┘ └─────────────┘ └─────────────┘ +``` + +--- + +## 📦 Component Responsibilities + +### **pages/datasets.js** (Page Component) + +**What it does:** + +- Defines the route `/datasets` +- Sets up SEO meta tags +- Configures search options (sort, filters, columns) +- Passes config to SearchContainer + +**What it doesn't do:** + +- Doesn't fetch data directly +- Doesn't manage search state +- Doesn't render results + +**Key code:** + +```javascript +export async function getStaticProps(context) { + // Runs at build time for SEO +} + +function DatasetsPage() { + const combinedConfig = useNextRouting(dataConfig, "/datasets"); + return +} +``` + +--- + +### **SearchContainer.js** (Container Component) + +**What it does:** + +- Wraps everything in `` +- Manages view state (list/table/grid) +- Manages filter visibility +- Provides layout structure +- Coordinates between filters and results + +**What it doesn't do:** + +- Doesn't render individual results +- Doesn't handle specific dataset logic + +**Key code:** + +```javascript +const [view, setView] = useState('list'); +const [filter, setFilter] = useState('hide'); + +return ( + + {/* Filters */} + {/* Results */} + {/* Pagination */} + +); +``` + +--- + +### **DatasetSearchResults.jsx** (Presentation Component) 🆕 + +**What it does:** + +- Receives search results from context +- Switches between view modes +- Shows empty states +- Shows error states +- Logs debug information + +**What it doesn't do:** + +- Doesn't fetch data +- Doesn't manage global state +- Doesn't define search configuration + +**Key code:** + +```javascript + ({ results, error })}> + {({ results, error }) => { + if (error) return ; + if (!results.length) return ; + + switch (view) { + case 'list': + return ; + case 'table': + return ; + case 'grid': + return ; + } + }} + +``` + +--- + +## 🎨 Design Patterns Used + +### **1. Container/Presentation Pattern** + +**Container Components** (Smart): + +- Manage state +- Handle logic +- Connect to data sources +- Example: `SearchContainer.js` + +**Presentation Components** (Dumb): + +- Receive props +- Render UI +- No side effects +- Example: `DatasetSearchResults.jsx` + +**Benefits:** + +- ✅ Easier to test +- ✅ More reusable +- ✅ Clearer separation of concerns + +--- + +### **2. Compound Components Pattern** + +Multiple components working together: + +```javascript + + + + + +``` + +**Benefits:** + +- ✅ Flexible composition +- ✅ Shared state via context +- ✅ Independent updates + +--- + +### **3. Render Props Pattern** + +`WithSearch` uses render props: + +```javascript + + {(props) => } + +``` + +**Benefits:** + +- ✅ Dynamic rendering +- ✅ Access to context +- ✅ Flexible component composition + +--- + +## 🔌 Data Flow + +### **1. Initial Load** + +``` +User visits /datasets + ↓ +getStaticProps loads translations + ↓ +Page renders with SearchContainer + ↓ +SearchProvider initializes + ↓ +alwaysSearchOnInitialLoad: true + ↓ +Triggers search automatically + ↓ +SearchAPIConnector.onSearch() + ↓ +POST to /api/search + ↓ +Elasticsearch query + ↓ +Results returned + ↓ +DatasetSearchResults renders +``` + +### **2. User Searches** + +``` +User types "iris" in search box + ↓ +SearchProvider updates state + ↓ +Triggers new search + ↓ +SearchAPIConnector.onSearch() + ↓ +POST to /api/search with searchTerm + ↓ +Results updated + ↓ +DatasetSearchResults re-renders +``` + +### **3. User Applies Filter** + +``` +User selects "status: active" + ↓ +SearchProvider updates filters + ↓ +Triggers new search + ↓ +SearchAPIConnector.onSearch() + ↓ +POST to /api/search with filters + ↓ +Results filtered + ↓ +DatasetSearchResults re-renders +``` + +--- + +## 🆕 What You Can Customize + +### **Easy Customizations** + +1. **Add new view mode:** + +```javascript +// In DatasetSearchResults.jsx +case "compact": + return ; +``` + +2. **Change empty state message:** + +```javascript +// In DatasetSearchResults.jsx +const EmptyState = ({ searchTerm }) => ( + + Your custom message here + +); +``` + +3. **Add dataset-specific info:** + +```javascript +// In ResultCard.js or create new component + + {result.name.raw} + + Instances: {result['qualities.NumberOfInstances'].raw} + + + Features: {result['qualities.NumberOfFeatures'].raw} + + +``` + +### **Advanced Customizations** + +1. **Add saved searches:** + +```javascript +// Create new component: SavedSearches.jsx +// Store searches in localStorage or backend +// Display as tabs or dropdown +``` + +2. **Add export functionality:** + +```javascript +// In DatasetSearchResults.jsx +const exportResults = (results, format) => { + if (format === 'csv') { + // Convert to CSV + } + if (format === 'json') { + // Convert to JSON + } +}; +``` + +3. **Add comparison mode:** + +```javascript +// Allow users to select multiple datasets +// Show side-by-side comparison +const [selectedDatasets, setSelectedDatasets] = useState([]); +``` + +--- + +## 📝 Adding New Components + +### **Example: Add a "Quick Stats" Component** + +**Step 1: Create the component** + +```javascript +// components/search/QuickStats.jsx +import { Box, Typography, Grid } from '@mui/material'; +import { WithSearch } from '@elastic/react-search-ui'; + +const QuickStats = () => { + return ( + ({ + totalResults, + facets, + })} + > + {({ totalResults, facets }) => ( + + + + {totalResults} + + Total Datasets + + + + {/* Add more stats */} + + )} + + ); +}; + +export default QuickStats; +``` + +**Step 2: Add to SearchContainer** + +```javascript +// In SearchContainer.js +import QuickStats from "./QuickStats"; + +// Add before filters + + + {/* existing filter code */} + +``` + +--- + +## 🎯 Best Practices + +### **1. Keep Components Focused** + +- ✅ One component = one responsibility +- ✅ Small, reusable components +- ❌ Avoid giant multi-purpose components + +### **2. Use Proper Naming** + +- Container components: `XxxContainer` +- Presentation components: `Xxx` or `XxxView` +- Utility components: `use Xxx` (hooks) + +### **3. Prop Types** + +```javascript +// Add PropTypes for better debugging +import PropTypes from 'prop-types'; + +DatasetSearchResults.propTypes = { + view: PropTypes.oneOf(['list', 'table', 'grid']).isRequired, + columns: PropTypes.array.isRequired, +}; +``` + +### **4. Performance** + +- Use `memo()` for expensive components +- Use `useMemo()` for expensive calculations +- Use `useCallback()` for callbacks passed as props + +--- + +## 📚 Next Steps + +### **Beginner:** + +1. Customize the empty state message +2. Change colors/styling +3. Add a new column to table view + +### **Intermediate:** + +1. Create a new view mode (e.g., "compact") +2. Add dataset comparison feature +3. Add export to CSV/JSON + +### **Advanced:** + +1. Implement saved searches +2. Add advanced filtering (date ranges, custom queries) +3. Create a dataset recommendation system +4. Add real-time collaboration features + +--- + +## 💡 Key Takeaways + +1. **Separation of Concerns**: Pages route, containers manage state, presentations render UI +2. **Composition**: Build complex UIs from simple, focused components +3. **Context Pattern**: Share state without prop drilling +4. **Debugging**: Add logging early, remove later +5. **Iteration**: Start simple, add features incrementally + +--- + +**Remember:** Good component structure makes your code easier to understand, test, and maintain! 🎯 diff --git a/DEBUGGING_GUIDE.md b/DEBUGGING_GUIDE.md new file mode 100644 index 00000000..3768b26f --- /dev/null +++ b/DEBUGGING_GUIDE.md @@ -0,0 +1,407 @@ +# 🐛 Debugging Guide: "0 Results" Issue + +## 📋 What We Just Did + +### ✅ **Changes Made:** + +1. **Created `DatasetSearchResults.jsx` component** + + - Separated results rendering logic + - Added proper empty states + - Added error handling + - Added detailed debug logging + +2. **Updated `SearchContainer.js`** + + - Integrated new DatasetSearchResults component + - Added debug logging + - Better error display + +3. **Enhanced `/api/search.js`** + - Added comprehensive logging + - Better error messages + - Request/response debugging + +--- + +## 🔍 How to Debug the "0 Results" Issue + +### **Step 1: Open Browser Console** + +1. Open your browser to: `http://localhost:3001/datasets` +2. Press **F12** to open Developer Tools +3. Go to **Console** tab +4. Look for these debug messages: + +``` +🔍 SearchContainer Debug: {...} +🔍 DatasetSearchResults Debug: {...} +📋 ListView: Rendering X results +``` + +### **Step 2: Check Network Tab** + +1. Press **F12** → **Network** tab +2. Refresh the page +3. Look for request to `/api/search` +4. Click on it and check: + - **Headers tab**: Status code should be 200 + - **Payload tab**: See what data was sent + - **Response tab**: See what came back + +**Look for:** + +- ✅ Status 200 = Good +- ❌ Status 500 = Server error +- ❌ Status 400 = Bad request + +### **Step 3: Check Next.js Terminal** + +Look at the terminal running `npm run dev` for these messages: + +```bash +🔍 /api/search called: {...} +📦 Creating new Elasticsearch connector for index: data +✅ Search successful: { resultsCount: 123, totalResults: 456 } +``` + +**If you see:** + +```bash +❌ Error in /api/search: {...} +``` + +That's your issue! + +--- + +## 🔧 Common Issues & Solutions + +### **Issue 1: Elasticsearch Connection Failed** + +**Symptoms:** + +- 500 error in Network tab +- "Search request timeout" in console +- No results showing + +**Solution:** + +```bash +# Test Elasticsearch connection +curl https://www.openml.org/es/ + +# Should return something like: +# { +# "name" : "...", +# "cluster_name" : "...", +# "version" : {...} +# } +``` + +**If it fails:** + +- Check if you're connected to internet +- Check if VPN is blocking access +- Try the old React app to confirm ES is working + +--- + +### **Issue 2: Wrong Index Name** + +**Symptoms:** + +- Request succeeds but returns 0 results +- Console shows: `indexName: "data"` + +**Solution:** +Check `dataConfig.js` - the index might be wrong. + +**To verify correct index:** + +```bash +# List all Elasticsearch indices +curl https://www.openml.org/es/_cat/indices?v +``` + +Look for indices like: + +- `data` +- `datasets` +- `openml-data` + +Update in `app/src/services/SearchAPIConnector.js`: + +```javascript +constructor(indexName = "data") { // ← Check this + this.indexName = indexName; +} +``` + +--- + +### **Issue 3: CORS Policy Error** + +**Symptoms:** + +- Console shows: `Access to fetch blocked by CORS policy` +- Network tab shows request cancelled + +**Solution:** +Elasticsearch needs CORS headers. Check if using proxy: + +In `/api/search.js`: + +```javascript +const use_dev_proxy = false; // Try changing to true +``` + +If `true`, make sure proxy is configured in `next.config.js`: + +```javascript +async rewrites() { + return [ + { + source: '/proxy/:path*', + destination: 'https://www.openml.org/es/:path*', + }, + ]; +} +``` + +--- + +### **Issue 4: Invalid Query Configuration** + +**Symptoms:** + +- 400 error from Elasticsearch +- Console shows query error + +**Check `dataConfig.js`:** + +```javascript +searchQuery: { + resultsPerPage: 100, // Should be reasonable number + search_fields: { + name: { weight: 3 }, // Fields must exist in ES + // ... + }, +} +``` + +--- + +## 📊 Understanding the Data Flow + +``` +┌─────────────────┐ +│ /datasets page │ User visits page +└────────┬────────┘ + │ + ▼ +┌─────────────────────┐ +│ SearchContainer │ Sets up SearchProvider +└────────┬────────────┘ + │ + ▼ +┌─────────────────────┐ +│ SearchProvider │ Manages search state +│ (Elastic Search UI)│ +└────────┬────────────┘ + │ + │ On mount: triggers search + ▼ +┌─────────────────────┐ +│ SearchAPIConnector │ Calls /api/search +└────────┬────────────┘ + │ + │ POST request + ▼ +┌─────────────────────┐ +│ /api/search │ Next.js API route +└────────┬────────────┘ + │ + │ Query ES + ▼ +┌─────────────────────┐ +│ Elasticsearch │ Returns results +└────────┬────────────┘ + │ + │ Results flow back + ▼ +┌─────────────────────┐ +│DatasetSearchResults │ Renders results +└─────────────────────┘ +``` + +--- + +## 🧪 Testing Checklist + +### **Test 1: Basic Connection** + +```bash +# Test if Next.js is running +curl http://localhost:3001/datasets + +# Test if API route works +curl -X POST http://localhost:3001/api/search \ + -H "Content-Type: application/json" \ + -d '{"indexName":"data","requestState":{"searchTerm":""}}' +``` + +### **Test 2: Browser Console** + +1. Open: `http://localhost:3001/datasets` +2. F12 → Console +3. Should see debug logs starting with 🔍, 📋, 📊, or 🎨 + +### **Test 3: Network Inspection** + +1. F12 → Network tab +2. Filter by "search" +3. Should see POST to `/api/search` +4. Response should have `results` array + +--- + +## 📝 Debug Checklist + +Copy and fill this out: + +``` +□ Next.js running on port 3001 +□ Can access http://localhost:3001/datasets +□ Browser console shows debug logs +□ No red errors in console +□ Network tab shows /api/search request +□ /api/search returns status 200 +□ Response has results array +□ Terminal shows "Search successful" message +□ Elasticsearch is accessible (curl test) +□ Using correct index name +``` + +--- + +## 🚀 Next Steps After Fixing + +Once you see results, you can: + +1. **Remove debug logs** (or keep them for development) +2. **Customize the DatasetSearchResults component** further +3. **Add more features:** + - Save search queries + - Export results + - Advanced filters + - Bookmarks + +--- + +## 💡 Key Learning Points + +### **Component Structure** + +``` +pages/ + └── datasets.js ← Page (route) + ↓ +components/search/ + ├── SearchContainer.js ← Container (manages state) + └── DatasetSearchResults.jsx ← Presentation (renders UI) + ↓ + ├── ResultCard.js ← Individual result components + ├── ResultGridCard.js + └── ResultsTable.js +``` + +### **Why Separate Components?** + +1. **Reusability** - Use DatasetSearchResults elsewhere +2. **Maintainability** - Easier to find and fix issues +3. **Testing** - Can test components independently +4. **Clarity** - Each component has one job + +### **The "Container/Presentation" Pattern** + +- **Container** (SearchContainer): Manages state, fetches data +- **Presentation** (DatasetSearchResults): Just displays data + +--- + +## 🎓 Understanding the Code + +### **What does `WithSearch` do?** + +```javascript + ({ results })}> + {({ results }) => ( + // Now you can use 'results' here + )} + +``` + +It's a **React Context Consumer** that: + +1. Accesses the search state from SearchProvider +2. Maps only the props you need +3. Passes them to your render function + +### **Why use memo()?** + +```javascript +const SearchContainer = memo(({ config, ... }) => { + // Component code +}); +``` + +`memo()` prevents unnecessary re-renders when props haven't changed. +This is important for search components that re-render frequently. + +--- + +## 📖 Additional Resources + +- [Elastic Search UI Docs](https://github.com/elastic/search-ui) +- [Next.js API Routes](https://nextjs.org/docs/api-routes/introduction) +- [React Context Pattern](https://react.dev/learn/passing-data-deeply-with-context) +- [Debugging React Apps](https://react.dev/learn/react-developer-tools) + +--- + +## ❓ Still Not Working? + +If you've tried everything and still see 0 results: + +1. **Compare with working React app:** + + - Open: `http://localhost:3000/search?type=data&status=active` + - Does it work? + - If yes: Compare network requests between React and Next.js + - If no: Elasticsearch is the problem + +2. **Check if it's a data issue:** + + ```bash + # Query ES directly + curl -X POST "https://www.openml.org/es/data/_search" \ + -H "Content-Type: application/json" \ + -d '{"size":10,"query":{"match_all":{}}}' + ``` + +3. **Enable verbose logging:** + In `dataConfig.js`, add: + + ```javascript + debug: true, // Add this + ``` + +4. **Ask for help with:** + - Screenshots of browser console + - Network tab screenshot showing /api/search request + - Terminal output from Next.js + +--- + +**Remember:** Debugging is a process of elimination. Work through each step methodically! 🔍 diff --git a/ERROR_ANALYSIS.md b/ERROR_ANALYSIS.md new file mode 100644 index 00000000..45f675b8 --- /dev/null +++ b/ERROR_ANALYSIS.md @@ -0,0 +1,465 @@ +# 🐛 Error Analysis: 500 Internal Server Error + +## 📊 **Error Breakdown** + +### **What You're Seeing:** + +``` +Browser Console: +❌ POST http://localhost:3001/api/search 500 (Internal Server Error) +❌ Unexpected token '<', "... ← HTML error page + ``` + +### **3. POST Request** + +- **What it is:** HTTP method for sending data +- **Why:** Search query is sent in request body +- **Alternative:** GET (but only for simple queries) + +--- + +## 🧪 **Step-by-Step Debugging** + +### **Step 1: Test Elasticsearch Connection** + +Open this URL in your browser: + +``` +http://localhost:3001/api/test-es +``` + +**What you should see:** + +```json +{ + "success": true, + "elasticsearch": { + "reachable": true, + "clusterName": "...", + "version": "..." + }, + "search": { + "totalHits": 1234, + "sampleResults": [...] + } +} +``` + +**If you see an error:** + +- Elasticsearch is not accessible +- Possible firewall/network issue +- Try from terminal: `curl https://www.openml.org/es/` + +--- + +### **Step 2: Check Next.js Terminal Output** + +Look at the terminal running `npm run dev` for these messages: + +**Good signs:** + +```bash +🔍 /api/search called: {...} +📦 Creating new Elasticsearch connector for index: data +✅ Connector created successfully +🔎 Executing search... +✅ Search successful: { resultsCount: X, totalResults: Y } +``` + +**Bad signs (these tell you what's wrong):** + +```bash +❌ Missing required fields: {...} +❌ Failed to create connector: Error message here +❌ Error in /api/search: { message: "...", stack: "..." } +``` + +--- + +### **Step 3: Check Browser Network Tab** + +1. **Open Developer Tools:** Press `F12` +2. **Go to Network tab** +3. **Refresh the page** +4. **Find the `/api/search` request** (should be red) +5. **Click on it** + +**Check these tabs:** + +#### **Headers Tab:** + +``` +Status Code: 500 Internal Server Error +Request URL: http://localhost:3001/api/search +Request Method: POST +``` + +#### **Payload Tab (what was sent):** + +```json +{ + "indexName": "data", + "requestState": { + "searchTerm": "", + "filters": [], + "resultsPerPage": 20 + }, + "queryConfig": {...} +} +``` + +#### **Response Tab (what came back):** + +If you see HTML starting with ``: + +- This is Next.js error page +- The API crashed +- Check Next.js terminal for actual error + +If you see JSON with `"error"` field: + +- API handled error gracefully +- Error message tells you what went wrong + +--- + +## 🔧 **Common Issues & Solutions** + +### **Issue 1: Cannot reach Elasticsearch** + +**Symptoms:** + +``` +❌ Error: fetch failed +❌ Error: ENOTFOUND www.openml.org +❌ Error: connect ETIMEDOUT +``` + +**Test from terminal:** + +```bash +# Test if ES is reachable +curl https://www.openml.org/es/ + +# Should return: +{ + "name" : "...", + "cluster_name" : "...", + "version" : {...} +} +``` + +**Solutions:** + +1. Check internet connection +2. Check if VPN is blocking +3. Try from React app (port 3000) to verify ES works +4. Check firewall settings + +--- + +### **Issue 2: CORS (Cross-Origin) Error** + +**Symptoms:** + +``` +❌ Access to fetch blocked by CORS policy +❌ No 'Access-Control-Allow-Origin' header +``` + +**Solution:** +Enable proxy in `/api/search.js`: + +```javascript +const use_dev_proxy = true; // Change from false to true +``` + +Then add to `next.config.js`: + +```javascript +async rewrites() { + return [ + { + source: '/proxy/:path*', + destination: 'https://www.openml.org/es/:path*', + }, + ]; +} +``` + +--- + +### **Issue 3: Invalid Index Name** + +**Symptoms:** + +``` +❌ index_not_found_exception +❌ no such index [data] +``` + +**Solution:** +Check available indices: + +```bash +curl https://www.openml.org/es/_cat/indices?v +``` + +Look for indices like: + +- `data` +- `datasets` +- `openml_data` + +Update in `SearchAPIConnector.js`: + +```javascript +constructor(indexName = "THE_CORRECT_INDEX_NAME") { + this.indexName = indexName; +} +``` + +--- + +### **Issue 4: Module/Package Error** + +**Symptoms:** + +``` +❌ Cannot find module '@elastic/search-ui-elasticsearch-connector' +❌ Module not found: Can't resolve '...' +``` + +**Solution:** + +```bash +cd app +npm install @elastic/search-ui-elasticsearch-connector @elastic/react-search-ui +npm run dev +``` + +--- + +### **Issue 5: Query Configuration Error** + +**Symptoms:** + +``` +❌ parsing_exception +❌ illegal_argument_exception +``` + +**Check `dataConfig.js`:** + +```javascript +// Make sure field names match what's in Elasticsearch +search_fields: { + name: { weight: 3 }, // ✅ Field must exist + nonexistent: { weight: 1 } // ❌ Will cause error +} +``` + +--- + +## 📋 **Debugging Checklist** + +Copy and check each item: + +``` +□ Next.js is running (npm run dev) +□ Can access http://localhost:3001 +□ Can access http://localhost:3001/api/test-es +□ test-es returns success: true +□ Browser console shows no red errors before clicking +□ Network tab is open and recording +□ Next.js terminal is visible and showing logs +□ Cleared browser cache (Ctrl+Shift+Delete) +□ Tried in incognito/private mode +``` + +--- + +## 💡 **Reading the Next.js Terminal** + +### **How to Read Error Stack Traces:** + +```bash +❌ Error in /api/search: { + message: "fetch failed", ← What went wrong + name: "TypeError", ← Type of error + stack: "TypeError: fetch failed ← Where it happened + at Object.fetch (node:internal/deps/undici/undici:11730:11) + at process.processTicksAndRejections (node:internal/process/task_queues:95:5) + at async handler (/app/src/pages/api/search.js:45:20)" + ↑↑↑↑ + Line number where error occurred +} +``` + +### **Common Error Types:** + +| Error Type | Meaning | Likely Cause | +| ---------------- | ---------------------- | ------------------------------------ | +| `TypeError` | Wrong data type | Accessing property of null/undefined | +| `ReferenceError` | Variable not found | Typo or undeclared variable | +| `SyntaxError` | Code syntax wrong | Missing bracket, comma, etc. | +| `FetchError` | Network request failed | Cannot reach URL | +| `TimeoutError` | Request took too long | Elasticsearch slow/unreachable | + +--- + +## 🎯 **Most Likely Issues (Ranked)** + +Based on your error, here's what's most likely wrong: + +### **1. Elasticsearch Connection (90% likely)** + +- Cannot reach `https://www.openml.org/es/` +- Network/firewall blocking request +- **Test:** Run `http://localhost:3001/api/test-es` + +### **2. Module Import Error (5% likely)** + +- Package not installed correctly +- **Test:** Check Next.js terminal for import errors + +### **3. Query Configuration Error (3% likely)** + +- Invalid field names in dataConfig.js +- **Test:** Compare with React app's working config + +### **4. CORS Issue (2% likely)** + +- Browser blocking cross-origin request +- **Test:** Check console for CORS message + +--- + +## 📞 **What to Share When Asking for Help** + +Please provide: + +1. **Next.js Terminal Output:** + + ```bash + # Copy everything from when you loaded the page + # Should start with: 🔍 /api/search called: {...} + ``` + +2. **Test ES Result:** + + ``` + # Visit: http://localhost:3001/api/test-es + # Copy the JSON response + ``` + +3. **Browser Network Tab:** + + - Screenshot of `/api/search` request + - Show Headers, Payload, and Response tabs + +4. **Browser Console:** + ```javascript + // Copy all error messages + ``` + +--- + +## 🚀 **Quick Fix Actions** + +Try these in order: + +### **Action 1: Test ES Connection** + +```bash +curl https://www.openml.org/es/ +``` + +### **Action 2: Visit Test Endpoint** + +``` +http://localhost:3001/api/test-es +``` + +### **Action 3: Check Package Installation** + +```bash +cd app +npm list @elastic/search-ui-elasticsearch-connector +``` + +### **Action 4: Clear and Restart** + +```bash +# Kill Next.js (Ctrl+C) +rm -rf .next +npm run dev +``` + +### **Action 5: Check if React App Works** + +``` +http://localhost:3000/search?type=data&status=active +``` + +If this works, ES is fine - issue is in Next.js config. + +--- + +## 📚 **Understanding Key Concepts** + +### **API Route (/api/search.js)** + +- Runs on the **server** (not browser) +- Can make external API calls safely +- Returns JSON to browser +- Catches errors and returns 500 if something fails + +### **Elasticsearch Connector** + +- Library that talks to Elasticsearch +- Converts search UI state to ES queries +- Handles pagination, sorting, filtering +- Returns formatted results + +### **Request Flow:** + +``` +Browser + → SearchProvider (React) + → SearchAPIConnector.onSearch() + → POST /api/search + → ElasticsearchAPIConnector + → https://www.openml.org/es/ + → Returns results + ← Data flows back +``` + +--- + +**Next Step:** Please run `http://localhost:3001/api/test-es` and share what you see! 🎯 diff --git a/LEARNING_GUIDE.md b/LEARNING_GUIDE.md new file mode 100644 index 00000000..ce0e215c --- /dev/null +++ b/LEARNING_GUIDE.md @@ -0,0 +1,470 @@ +# Learning Guide: OpenML Search Implementation + +## From Junior to Senior Developer Perspective + +--- + +## 🎯 Project Overview + +**Goal:** Fix broken search functionality and migrate to SEO-friendly URLs + +**Starting Point:** Search pages showing errors, no results displaying + +**Final Result:** Working search for datasets, tasks, flows, and runs with clean URLs + +--- + +## 📚 Key Concepts Learned + +### 1. **Understanding the Problem** (Junior Level) + +**What was broken:** + +- Going to `http://localhost:3001/d/search` showed "500 Internal Server Error" +- Error message: "Unexpected token '<', '...`) +- This is like ordering a pizza and getting a salad - wrong format! + +**Debugging approach:** + +1. Look at the error message carefully +2. Check browser console for detailed errors +3. Look at Network tab to see what's being sent/received +4. Read the documentation for expected formats + +--- + +### 2. **Client-Side vs Server-Side** (Junior → Mid Level) + +**Two ways to fetch data in Next.js:** + +**Server-Side (SSR/SSG):** + +```javascript +export async function getStaticProps() { + // Runs on SERVER at BUILD TIME + // Good for SEO - search engines see the data + // Data is "baked into" the HTML +} +``` + +**Client-Side:** + +```javascript +useEffect(() => { + // Runs in BROWSER after page loads + // Good for dynamic data that changes often + // Search engines might miss this data +}, []); +``` + +**For this project:** We use SSR for SEO but client-side for search queries (because search terms change constantly). + +--- + +### 3. **Working with External APIs** (Mid Level) + +**The Problem with Layers:** + +``` +Too many layers = More failure points +Browser → Next.js → Proxy → Elasticsearch + ❌ Failed here! +``` + +**Solution - Direct Connection:** + +``` +Fewer layers = More reliable +Browser → Next.js → Elasticsearch + ✅ Works! +``` + +**Key Learning:** Sometimes libraries don't work perfectly. You need to: + +1. Try the library first (don't reinvent the wheel) +2. If it fails, understand WHY it fails +3. Build a custom solution only if necessary + +**How to build a custom connector:** + +```javascript +class OpenMLSearchConnector { + async onSearch(requestState, queryConfig) { + // 1. Build the query from user input + const query = this.buildQuery(requestState, queryConfig); + + // 2. Send to Elasticsearch + const response = await fetch(url, { + method: 'POST', + body: JSON.stringify(query), + }); + + // 3. Format the response for the UI + return this.formatResponse(await response.json()); + } +} +``` + +--- + +### 4. **Data Format Transformation** (Mid → Senior Level) + +**The Challenge:** Different systems expect different data formats. + +**Elasticsearch returns:** + +```javascript +{ + hits: { + hits: [ + { _source: { name: "iris", version: 1 } } + ] + } +} +``` + +**Search UI expects:** + +```javascript +{ + results: [ + { + name: { raw: 'iris' }, + version: { raw: 1 }, + }, + ]; +} +``` + +**Solution - Transform the data:** + +```javascript +formatResponse(esResponse) { + return esResponse.hits.hits.map(hit => { + const formatted = {}; + // Wrap each field in { raw: value } + Object.entries(hit._source).forEach(([key, value]) => { + formatted[key] = { raw: value }; + }); + return formatted; + }); +} +``` + +**Key Learning:** Act as a translator between systems. Read documentation carefully to understand exact format requirements. + +--- + +### 5. **Debugging Techniques** (All Levels) + +**Junior Level - Read Error Messages:** + +``` +Error: ES responded with 400: Bad Request +Reason: "Failed to parse object: unknown field [name]" +``` + +→ Look for the word "name" in your code +→ Something about a field called "name" is wrong + +**Mid Level - Add Logging:** + +```javascript +console.log('[OpenMLSearchConnector] ES Query:', query); +console.log('[OpenMLSearchConnector] ES Response:', response); +``` + +→ See exactly what you're sending and receiving +→ Compare with working examples + +**Senior Level - Systematic Approach:** + +1. **Isolate the problem:** Create test endpoints (`/api/test-es`) +2. **Work backwards:** If ES works directly, where does it break? +3. **Check assumptions:** "I think it's X" → Prove it with logs +4. **Document findings:** Write down what you discovered + +--- + +### 6. **CSS and Layout Concepts** (Junior → Mid Level) + +**Problem:** Container not using full width + +**Understanding Box Model:** + +```css +/* Wrong - width doesn't include padding */ +width: 100%; +padding: 24px; +/* Total width = 100% + 48px = overflow! */ + +/* Right - box-sizing includes padding in width */ +width: 100%; +padding: 24px; +box-sizing: border-box; +/* Total width = 100% (including padding) ✅ */ +``` + +**Understanding CSS Properties:** + +- `max-width: 1600px` = "Don't go wider than this" +- `width: 100%` = "Use all available space" +- `margin: 0 auto` = "Center horizontally" +- `padding: 24px` = "Space inside the box" + +**Key Learning:** Small CSS details matter. Always test responsive layouts on different screen sizes. + +--- + +### 7. **Responsive Design** (Mid Level) + +**Grid System - 12 Column Layout:** + +```javascript +// xs = extra small screens (phones) +// sm = small screens (tablets) +// md = medium screens (laptops) + + + // xs: 12/12 = 100% width = 1 column // sm: 6/12 = 50% width = 2 columns // + md: 4/12 = 33% width = 3 columns + +``` + +**Key Learning:** Design for mobile first, then scale up. + +--- + +### 8. **Component Architecture** (Mid → Senior Level) + +**Separation of Concerns:** + +```javascript +// ❌ Bad - Everything in one component +function DatasetSearch() { + const [data, setData] = useState([]); + const [filters, setFilters] = useState({}); + // 500 lines of code... + return
render everything
; +} + +// ✅ Good - Split responsibilities +function SearchContainer() { + // Manages state and data fetching + return ; +} + +function DatasetSearchResults({ results }) { + // Only handles display + return results.map((r) => ); +} +``` + +**Key Learning:** + +- **Container components:** Handle logic, state, data +- **Presentation components:** Handle display only +- Makes code easier to test, reuse, and understand + +--- + +### 9. **URL Structure and SEO** (Mid → Senior Level) + +**Bad URLs:** + +``` +/d/search?q=iris +/t/search?type=classification +``` + +→ Not descriptive, poor for SEO + +**Good URLs:** + +``` +/datasets?q=iris +/tasks?type=classification +``` + +→ Clear meaning, SEO-friendly, professional + +**Implementation:** + +```javascript +// Old page becomes a redirect +export default function OldSearchPage() { + const router = useRouter(); + + useEffect(() => { + router.replace({ + pathname: '/datasets', + query: router.query, // Keep query params + }); + }, []); + + return null; // Show nothing while redirecting +} +``` + +**Key Learning:** Good URLs are part of good UX and help SEO. + +--- + +### 10. **Error Handling Patterns** (Senior Level) + +**Anticipate Failures:** + +```javascript +async onSearch(requestState, queryConfig) { + try { + const response = await fetch(url, options); + const data = await response.json(); + + // Check if request succeeded + if (!response.ok) { + console.error("ES Error:", data); + throw new Error(`ES responded with ${response.status}`); + } + + return this.formatResponse(data); + + } catch (error) { + console.error("Search failed:", error); + // Return empty results instead of crashing + return { results: [], totalResults: 0 }; + } +} +``` + +**Key Learning:** Always handle errors gracefully. Users should never see app crashes. + +--- + +## 🛠️ Practical Skills Developed + +### 1. **Reading Documentation** + +- Elasticsearch Query DSL documentation +- React Search UI library docs +- Next.js routing documentation +- MUI component APIs + +### 2. **Debugging Tools** + +- Browser DevTools Console +- Network tab (inspect API requests) +- React DevTools (inspect component state) +- Terminal logs (server-side errors) + +### 3. **Git and Version Control** + +- Making incremental commits +- Reading file changes +- Understanding diffs + +### 4. **Problem-Solving Process** + +1. Reproduce the issue +2. Understand the error message +3. Find the root cause +4. Fix incrementally +5. Test the fix +6. Document the solution + +--- + +## 💡 Senior-Level Insights + +### When to Use Third-Party Libraries + +- **Use them:** If well-maintained, widely used, solves your exact problem +- **Build custom:** If library is buggy, overkill, or doesn't fit your needs +- **This project:** Library failed, custom solution was simpler and more reliable + +### Architecture Decisions + +- **Less is more:** Fewer layers = easier debugging +- **Direct when possible:** Don't add proxies unless necessary for security +- **Document why:** Explain architectural choices for future developers + +### Code Quality + +- **Comments:** Explain WHY, not WHAT (code shows what) +- **Naming:** Use descriptive names (`OpenMLSearchConnector` not `Connector`) +- **Structure:** Group related code, separate concerns +- **Testing:** Should have written tests (learn from this!) + +--- + +## 📝 Key Takeaways + +1. **Error messages are your friends** - Read them carefully +2. **Console.log is powerful** - Use it liberally while debugging +3. **Break big problems into small pieces** - Fix one thing at a time +4. **Test frequently** - Don't make 10 changes before testing +5. **Document as you go** - Future you will thank present you +6. **Learn the tools** - DevTools, terminals, documentation +7. **Ask questions** - "Why does it work this way?" +8. **Read other people's code** - See how pros structure things +9. **Understand data flow** - Where does data come from, go to? +10. **Practice patience** - Debugging takes time, that's normal + +--- + +## 🚀 Next Learning Steps + +### To go from Mid to Senior: + +1. **Learn TypeScript** - Catch errors before runtime +2. **Study design patterns** - Common solutions to common problems +3. **Performance optimization** - Measure, don't guess +4. **Testing** - Unit tests, integration tests, E2E tests +5. **Security** - Input validation, XSS, CSRF protection +6. **DevOps basics** - CI/CD, Docker, deployment +7. **System design** - How to architect large applications + +### Recommended Resources: + +- MDN Web Docs (HTML/CSS/JavaScript) +- React.dev (official React docs) +- Next.js documentation +- "You Don't Know JS" book series +- Frontend Masters courses +- Build projects - lots of them! + +--- + +## 🎓 Final Advice + +**For Junior Developers:** + +- Don't be afraid to break things (in development) +- Ask "why" constantly +- Read error messages completely +- Google is your friend +- Everyone was junior once + +**For Mid-Level Developers:** + +- Focus on architecture and patterns +- Learn to debug systematically +- Understand tradeoffs in decisions +- Start teaching others (best way to learn) +- Build side projects + +**For Senior Developers:** + +- Always write code others can understand +- Document architectural decisions +- Think about maintainability +- Share knowledge generously +- Keep learning - tech changes fast + +--- + +**Remember:** Every senior developer was once confused by error messages. The difference is persistence and systematic learning. Keep building! 🚀 diff --git a/NEXTJS_LEARNING_GUIDE.md b/NEXTJS_LEARNING_GUIDE.md new file mode 100644 index 00000000..1f2afe4e --- /dev/null +++ b/NEXTJS_LEARNING_GUIDE.md @@ -0,0 +1,535 @@ +# 🎓 STEP-BY-STEP LEARNING GUIDE: Next.js Routing & Data Fetching + +## 📋 What We Just Did + +### **Summary of Changes** + +1. ✅ Created new `/datasets` route (better SEO) +2. ✅ Added redirect from `/d/search` → `/datasets` +3. ✅ Fixed potential routing issues +4. ✅ Added comprehensive SEO meta tags + +--- + +## 🔍 Understanding the 500 Error + +### **What Causes a 500 Error?** + +A 500 (Internal Server Error) happens when: + +- Server-side code crashes +- API connection fails +- Missing environment variables +- Elasticsearch connection issues + +### **How to Debug** + +1. **Check Browser Console** (F12 → Console tab) + + ``` + Look for red error messages + ``` + +2. **Check Terminal Where Next.js Runs** + + ```bash + # Your terminal running: npm run dev + # Look for error stack traces + ``` + +3. **Check Network Tab** (F12 → Network tab) + ``` + - Look for failed requests (red) + - Click on the failed request + - Check "Response" tab for error details + ``` + +--- + +## 📚 LESSON: Next.js Routing Explained + +### **File-Based Routing** + +Next.js automatically creates routes based on file structure: + +``` +/pages + ├── index.js → / + ├── about.js → /about + ├── datasets.js → /datasets ✅ NEW! + ├── d/ + │ └── search.js → /d/search ⚠️ OLD (now redirects) + └── d/ + └── [dataId].js → /d/123 (dynamic route) +``` + +### **Dynamic Routes** + +Files with `[brackets]` are dynamic: + +- `[dataId].js` → matches `/d/1`, `/d/42`, `/d/anything` +- `[...slug].js` → catches all routes (e.g., `/docs/guide/intro`) + +--- + +## 📚 LESSON: getStaticProps vs getServerSideProps + +### **getStaticProps** (What we used) + +```javascript +export async function getStaticProps(context) { + // Runs at BUILD TIME (npm run build) + // Creates static HTML pages + + return { + props: { + data: 'This is pre-rendered at build time', + }, + revalidate: 60, // Optional: Re-generate page every 60 seconds + }; +} +``` + +**When to use:** + +- ✅ Content doesn't change often +- ✅ Same content for all users +- ✅ Best for SEO +- ✅ Fastest page loads + +**Example:** Blog posts, product pages, documentation + +--- + +### **getServerSideProps** (Alternative) + +```javascript +export async function getServerSideProps(context) { + // Runs on EVERY REQUEST + // Fresh data every time + + const res = await fetch('https://api.example.com/data'); + const data = await res.json(); + + return { + props: { data }, + }; +} +``` + +**When to use:** + +- ✅ Content changes frequently +- ✅ Personalized for each user +- ✅ Need request headers (cookies, etc.) + +**Example:** User dashboards, real-time data, authenticated pages + +--- + +## 📚 LESSON: Client-Side Data Fetching + +### **Using useEffect + fetch** (Traditional way) + +```javascript +import { useState, useEffect } from 'react'; + +function MyComponent() { + const [data, setData] = useState(null); + const [loading, setLoading] = useState(true); + const [error, setError] = useState(null); + + useEffect(() => { + // This runs in the BROWSER after page loads + fetch('/api/data') + .then((response) => response.json()) + .then((data) => { + setData(data); + setLoading(false); + }) + .catch((error) => { + setError(error); + setLoading(false); + }); + }, []); // Empty array = run once on mount + + if (loading) return
Loading...
; + if (error) return
Error: {error.message}
; + + return
{data}
; +} +``` + +**When to use:** + +- ✅ Data changes based on user interaction +- ✅ Search results, filters, pagination +- ✅ Real-time updates + +--- + +### **Using SWR** (Modern, recommended) + +```javascript +import useSWR from 'swr'; + +const fetcher = (url) => fetch(url).then((r) => r.json()); + +function MyComponent() { + const { data, error, isLoading } = useSWR('/api/data', fetcher); + + if (isLoading) return
Loading...
; + if (error) return
Error!
; + + return
{data}
; +} +``` + +**Benefits:** + +- ✅ Automatic caching +- ✅ Automatic revalidation +- ✅ Deduplication (won't fetch same data twice) +- ✅ Error retry logic + +--- + +## 📚 LESSON: SEO Best Practices + +### **1. Meta Tags in Next.js** + +```javascript +import Head from 'next/head'; + +function MyPage() { + return ( + <> + + {/* Page title - appears in browser tab and search results */} + OpenML Datasets - Search ML Datasets + + {/* Description - appears in search results */} + + + {/* Keywords - less important now, but still useful */} + + + {/* Open Graph - for social media sharing */} + + + + + {/* Twitter Card - for Twitter sharing */} + + + {/* Canonical URL - tells Google which URL is the "real" one */} + + + + {/* Your page content */} + + ); +} +``` + +### **2. Semantic HTML** + +```javascript +// ❌ Bad for SEO +
My Title
+
My content
+ +// ✅ Good for SEO +

My Title

+

My content

+
...
+ +``` + +### **3. URL Structure** + +``` +❌ Bad: +/d/search?type=data +/page?id=123 + +✅ Good: +/datasets +/datasets/iris +/blog/my-post-title +``` + +--- + +## 📚 LESSON: Understanding Our Search Implementation + +### **How the Search Works** + +``` +┌─────────────┐ +│ Browser │ User types in search box +└──────┬──────┘ + │ + │ 1. User interaction triggers search + ▼ +┌─────────────────┐ +│ SearchContainer │ React component that manages UI +└──────┬──────────┘ + │ + │ 2. Calls SearchAPIConnector + ▼ +┌──────────────────────┐ +│ SearchAPIConnector │ Sends request to /api/search +└──────┬───────────────┘ + │ + │ 3. POST request to Next.js API + ▼ +┌──────────────────┐ +│ /api/search.js │ Next.js API route (runs on server) +└──────┬───────────┘ + │ + │ 4. Queries Elasticsearch + ▼ +┌──────────────────┐ +│ Elasticsearch │ Returns search results +└──────┬───────────┘ + │ + │ 5. Results flow back up + ▼ +┌─────────────────┐ +│ Browser UI │ Displays results to user +└─────────────────┘ +``` + +### **Key Files** + +1. **`/pages/datasets.js`** - The page component (UI) +2. **`/components/search/SearchContainer.js`** - Search UI logic +3. **`/services/SearchAPIConnector.js`** - API communication +4. **`/pages/api/search.js`** - Server-side API endpoint +5. **`/search_configs/dataConfig.js`** - Search configuration + +--- + +## 🐛 Troubleshooting Checklist + +### **If you see 500 error:** + +1. ✅ **Check Elasticsearch is running** + + ```bash + # Check if Elasticsearch is accessible + curl https://www.openml.org/es/ + ``` + +2. ✅ **Check environment variables** + + ```bash + # In your app directory + cat .env.local + + # Should have: + # ELASTICSEARCH_URL=... + ``` + +3. ✅ **Check browser console** + + - Press F12 + - Go to Console tab + - Look for red errors + +4. ✅ **Check Next.js terminal** + + - Look at the terminal running `npm run dev` + - Check for error stack traces + +5. ✅ **Check Network tab** + - F12 → Network tab + - Click on failed request + - Check Response tab + +--- + +## 🧪 Testing Your Changes + +### **Step 1: Test New Route** + +1. Open browser: `http://localhost:3001/datasets` +2. Should see search page +3. Try searching for "iris" +4. Check filters work +5. Check pagination works + +### **Step 2: Test Redirect** + +1. Open browser: `http://localhost:3001/d/search` +2. Should automatically redirect to `/datasets` +3. URL in browser should change + +### **Step 3: Test with Query Parameters** + +1. Open: `http://localhost:3001/d/search?status=active` +2. Should redirect to: `http://localhost:3001/datasets?status=active` +3. Status filter should be pre-applied + +--- + +## 📊 URL Query Parameters Explained + +### **What are query parameters?** + +``` +http://localhost:3001/datasets?status=active&sort=date&page=2 + ↑ + Query parameters start here + +status=active ← Filter by status +sort=date ← Sort by date +page=2 ← Show page 2 +``` + +### **How to read them in Next.js** + +```javascript +import { useRouter } from 'next/router'; + +function MyComponent() { + const router = useRouter(); + + // Get query parameters + const { status, sort, page } = router.query; + + console.log(status); // "active" + console.log(sort); // "date" + console.log(page); // "2" + + return
Status: {status}
; +} +``` + +### **How to set them** + +```javascript +import { useRouter } from 'next/router'; + +function MyComponent() { + const router = useRouter(); + + const applyFilter = () => { + // Update URL with new query parameters + router.push({ + pathname: '/datasets', + query: { status: 'active', sort: 'date' }, + }); + // URL becomes: /datasets?status=active&sort=date + }; + + return ; +} +``` + +--- + +## 🚀 Next Steps + +### **1. Update Links Throughout Your App** + +Find and replace old links: + +```javascript +// Old +Search Datasets + +// New +Search Datasets +``` + +### **2. Add More SEO Pages** + +You can apply the same pattern to: + +- `/flows` (instead of `/f/search`) +- `/tasks` (instead of `/t/search`) +- `/runs` (instead of `/r/search`) + +### **3. Add Structured Data** + +Improve SEO with structured data: + +```javascript + +