This document provides comprehensive guidance on configuring database connections in fib-flow. It covers various connection methods, supported database types, and transaction management features. Whether you're using SQLite for simple deployments or MySQL for production environments, you'll find detailed information about setup options and best practices.
fib-flow supports both SQLite and MySQL databases. You can specify the database connection in three ways:
// SQLite
const taskManager = new TaskManager({
dbConnection: 'sqlite:tasks.db'
});const dbConn = db.open('sqlite:tasks.db');
const taskManager = new TaskManager({
dbConnection: dbConn
});// When using a connection pool, you must specify the database type
const pool = Pool({
create: () => db.open('sqlite:tasks.db'),
destroy: conn => conn.close(),
timeout: 30000,
retry: 1,
maxsize: 5
});
const taskManager = new TaskManager({
dbConnection: pool,
dbType: 'sqlite' // Required when using connection pool
});- Supports file-based and in-memory databases
- Automatic schema initialization
- Ideal for single-instance deployments
- No configuration required for in-memory mode
- Supports both relative and absolute paths for database files
- Thread-safe operations for concurrent access
- Built-in WAL (Write-Ahead Logging) support
- In-memory database example:
const taskManager = new TaskManager({
dbConnection: 'sqlite::memory:'
});- Supports connection pooling for better performance
- Automatic reconnection handling
- Built-in connection timeout management
- Compatible with MySQL 5.7 and later versions
- Handles connection string parsing automatically
const mysqlPool = Pool({
create: () => db.open('mysql://user:password@localhost:3306/dbname'),
destroy: conn => conn.close(),
timeout: 30000,
retry: 1,
maxsize: 5
});
const taskManager = new TaskManager({
dbConnection: mysqlPool,
dbType: 'mysql' // Required when using connection pool
});- Full support for PostgreSQL 12 and later
- Connection pooling support
- Native JSON data type support
- Automatic schema migration
- SSL connection support
- Connection string and pool configuration:
// Using connection string
const taskManager = new TaskManager({
dbConnection: 'psql://user:password@localhost:5432/dbname'
});
// Using connection pool
const pgPool = Pool({
create: () => db.open('psql://user:password@localhost:5432/dbname'),
destroy: conn => conn.close(),
timeout: 30000,
retry: 1,
maxsize: 5
});
const taskManager = new TaskManager({
dbConnection: pgPool,
dbType: 'postgres' // Required when using connection pool
});Note: The dbType parameter is only required when using a connection pool. When using a connection string, the database type is automatically inferred from the connection string prefix ('sqlite:' or 'mysql:').
You can now create a TaskManager without explicitly providing a database connection. In such cases, an in-memory SQLite database will be automatically created:
const taskManager = new TaskManager(); // No database connection specified
taskManager.db.setup(); // Initialize database schemaThis feature is particularly beneficial for single-instance, in-process scenarios where:
- Distributed task management is not required
- High fault tolerance is not critical
- Simple, lightweight task orchestration is needed
- Tasks are executed within a single process or application
Benefits in single-instance scenarios:
- Zero configuration overhead
- Minimal performance impact
- Simplified task management for local, non-distributed workloads
- Ideal for microservices, background processing, and event-driven architectures
Note: For production environments with high reliability requirements, it's recommended to use a persistent database configuration.
fib-flow automatically handles transactions for task state changes and workflow operations. All database operations related to task status updates, child task creation, and result storage are wrapped in transactions to ensure data consistency.
The transaction system provides the following guarantees:
- Atomic task state transitions
- Consistent parent-child task relationships
- Isolated concurrent task operations
- Durable task status updates
- Automatic rollback on errors
- Nested transaction support for complex workflows
// Automatic transaction handling for workflow operations
taskManager.use('parent_task', async (task, next) => {
// All child task creation and state changes are atomic
return next([
{ name: 'child1' },
{ name: 'child2' }
]);
});When working with transactions:
- Avoid long-running operations within transactions
- Use connection pools for better performance
- Set appropriate timeout values
- Monitor transaction duration
- Handle transaction failures gracefully
Note: The transaction system is designed to be transparent to the user while ensuring data consistency across all database operations.
The core table structure used by fib-flow includes the following fields:
-- Task identification and basic info
id -- Unique task identifier (AUTO_INCREMENT)
name -- Task type name
type -- Task type (async/cron)
priority -- Task priority (-20 to 20)
payload -- JSON encoded task data
created_at -- Creation timestamp
tag -- Optional task categorization tag
-- Task execution status
status -- Current task state
next_run_time -- Next scheduled execution time
last_active_time -- Last activity timestamp
result -- JSON encoded execution result
error -- Error message if failed
-- Task settings
stage -- Current execution stage
timeout -- Execution timeout in seconds
retry_count -- Current retry attempts
max_retries -- Maximum retry attempts
retry_interval -- Delay between retries
cron_expr -- Cron expression (for cron tasks)
-- Workflow relationships
root_id -- Root task ID in workflow
parent_id -- Parent task ID
total_children -- Number of child tasks
completed_children -- Completed child tasks
-- Worker information
worker_id -- Worker instance ID
start_time -- Task start timestampEach database adapter includes optimized indexes for common operations:
-- Task scheduling
idx_task_scheduling (status, next_run_time, priority)
-- Timeout detection
idx_task_timeout (status, last_active_time)
-- Workflow management
idx_task_workflow (parent_id, status, completed_children)
-- Task statistics
idx_task_stats (tag, name, status)When initializing TaskManager, you can configure the following global options that affect task execution and database operations:
const taskManager = new TaskManager({
// Database connection options
dbConnection: 'sqlite:tasks.db',
dbType: 'sqlite',
// Task execution defaults
poll_interval: 1000, // How often to check for new tasks (ms)
max_retries: 3, // Default retry attempts
retry_interval: 0, // No delay between retries (seconds)
timeout: 60, // Default task timeout (seconds)
// Worker configuration
max_concurrent_tasks: 10,
active_update_interval: 1000,
worker_id: 'worker-123' // Optional custom worker ID
});These settings provide the baseline configuration for all tasks. Individual tasks can override certain options (timeout, retries, etc.) during registration or execution.
Note: When using a connection pool, you must explicitly specify the database type using the dbType option.