Nosana Indexer

A standalone service for indexing Nosana data including jobs, markets, and runs from the Solana blockchain.

🌟 Features

• Real-time WebSocket monitoring of Nosana job, market, and run accounts
• Historical data synchronization via getProgramAccounts (GPA) calls
• Job processing pipeline for IPFS data retrieval and USD pricing
• Health monitoring API with status endpoints
• PostgreSQL database with Drizzle ORM for data persistence
• Cron-based scheduling for periodic data synchronization
• Docker support for easy deployment

🏗️ Architecture

The indexer consists of several key components:

• Indexer Core (src/indexer.ts): Main indexing logic and WebSocket monitoring
• API Server (src/api-server.ts): Health check and monitoring endpoints
• Database Layer (src/database/): PostgreSQL schemas and table definitions
• Utilities (src/utils/): Helper functions for data processing and price fetching
• Configuration (src/config/): Environment variable management

🚀 Quick Start

Prerequisites

• Node.js 18+
• PostgreSQL 12+
• Docker & Docker Compose (optional)

Using Docker Compose (Recommended)

1. Clone and configure:

   git clone <repository-url>
   cd nosana-indexer
   cp .env.example .env
   # Edit .env with your configuration

2. Start services:

   docker-compose up -d

3. Check health:

   curl http://localhost:3001/health

Manual Installation

1. Install dependencies:

   npm install

2. Set up database:

   # Create PostgreSQL database
   createdb nosana
   
   # Run migrations
   npm run db:push

3. Configure environment:

   cp .env.example .env
   # Edit .env with your settings

4. Start the indexer:

   npm run dev  # Development mode
   npm start    # Production mode

⚙️ Configuration

Environment Variables

Create a .env file with the following variables:

# Database Configuration
POSTGRES_HOST=postgres
POSTGRES_USER=nosana
POSTGRES_PASSWORD=your_password
POSTGRES_DB=nosana

# Solana Network Configuration
SOLANA_NETWORK=mainnet  # or devnet, testnet
SOLANA_RPC=https://api.mainnet-beta.solana.com

# API Configuration
API_PORT=3001

# Optional: Advanced Configuration
# PAYER_PRIVATE_KEY=your_wallet_private_key_base58

RPC Requirements

• You must configure an RPC endpoint that supports getProgramAccounts (GPA).
• The example value shown for SOLANA_RPC (https://api.mainnet-beta.solana.com) does not support GPA and will typically return HTTP 410 (Gone) for GPA requests.
• If GPA is unsupported, the indexer will emit a warning and skip GPA-based syncs (Jobs GPA and Markets GPA). To enable full synchronization, switch to a full-featured RPC provider that allows getProgramAccounts.

Network Options

• mainnet: Solana mainnet (production)
• devnet: Solana devnet (development)

📊 API Endpoints

Health Check

GET /health or /

Returns the current status of the indexer:

{
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00.000Z",
  "indexer": {
    "isRunning": true,
    "lastActivity": "2024-01-15T10:29:45.000Z",
    "startTime": "2024-01-15T10:00:00.000Z",
    "uptime": 1800,
    "timeSinceLastActivity": 15000
  }
}

🔄 How It Works

1. WebSocket Monitoring

• Connects to Solana WebSocket for real-time account updates
• Monitors job accounts, market accounts, and run accounts
• Processes updates immediately and stores in database

2. Periodic Synchronization

• Jobs GPA: Runs every 5 minutes to sync all job accounts
• Markets GPA: Runs every 5 minutes to sync market data
• Job Processing: Runs every 2 minutes to process IPFS data and calculate USD values

3. Data Processing

• Retrieves job definitions and results from IPFS [COMING-SOON]
• Calculates USD reward values using historical NOS prices [COMING-SOON]
• Handles job state transitions and validation

🗃️ Database Schema

The indexer uses PostgreSQL with the following main tables:

• job_accounts: Core job data and metadata
• markets: Market configuration and queue data
• stats: Price data and historical statistics [COMING-SOON]

🐳 Docker Deployment

Production Deployment

# Build and deploy
docker-compose up -d --build

# View logs
docker-compose logs -f indexer

## 🔧 Development

### Running Dev mode

```bash
npm run dev

Code Quality

# Linting
npm run lint
npm run lint:fix

# Type checking
npx tsc --noEmit

Database Management

# Generate new migration
npm run db:generate

# Open database studio
npm run db:studio

Database UI (Adminer)

When running via Docker Compose, a lightweight database UI (Adminer) is exposed.

• URL: http://localhost:8081
• Default connection settings:
  • System: PostgreSQL
  • Server/Host: podman
  • Username: nosana
  • Password: nosana
  • Database: nosana

Use these settings to log in and inspect the PostgreSQL database.

📈 Monitoring

Metrics and Observability

The indexer provides several monitoring capabilities:

• Health endpoint: Real-time status information
• Database metrics: Job counts, processing rates
• Error logging: Comprehensive error tracking
• Performance monitoring: Processing times and throughput

Logging

Logs are structured and include:

• WebSocket events and processing
• Database operations and errors
• IPFS retrieval status
• Price fetching and caching
• Cron job execution

🚨 Troubleshooting

Common Issues

WebSocket connection failures:

# Check RPC endpoint connectivity
curl -X POST -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"getHealth"}' \
  $SOLANA_RPC

Database connection issues:

# Test database connectivity
psql -h $POSTGRES_HOST -U $POSTGRES_USER -d $POSTGRES_DB -c "SELECT 1"

IPFS retrieval failures:

• Check network connectivity to IPFS gateways
• Monitor rate limiting from Pinata or other providers
• Verify IPFS hashes in job data

Performance Tuning

• Database indexing: Ensure proper indexes on frequently queried columns
• Connection pooling: Adjust PostgreSQL connection limits
• Batch processing: Tune batch sizes for GPA operations
• Rate limiting: Implement backoff strategies for API calls

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📞 Support

For issues and questions:

• Create an issue on GitHub
• Join the Nosana Discord community
• Check the documentation at docs.nosana.io

🔗 Related Projects

• @nosana/kit - Nosana TypeScript SDK
• Nosana Dashboard - Web interface for Nosana
• Nosana CLI - Nosana TypeScript CLI