ng-vui-grid Performance Guide

Overview

The ng-vui-grid has been completely redesigned to handle massive datasets (10+ million records) with minimal memory usage and consistent performance. This guide covers all performance features and optimization strategies.

Performance Architecture

Component Rendering Modes

The grid automatically selects the optimal rendering strategy:

// Traditional Mode (< 1000 records)
*ngFor="let row of paginatedData(); trackBy: trackByRowData"

// Virtual Scrolling Mode (1000+ records, no pagination)
virtualScrolling: true + custom viewport management

// Streaming Mode (10M+ records)
streaming: { enabled: true } + chunk management

Data Flow Architecture

Virtual Scrolling

Virtual scrolling renders only visible rows in the DOM, dramatically reducing memory usage and improving performance.

Configuration

const gridOptions: GridOptions = {
  // Enable virtual scrolling
  virtualScrolling: true,

  // Performance tuning
  rowHeight: 48,                    // Fixed row height (px) or function
  maxRowsInDom: 20,                // Maximum DOM elements
  rowBuffer: 5,                    // Extra rows for smooth scrolling

  // Disable pagination when using virtual scrolling
  pagination: false
};

Dynamic Row Heights

const gridOptions: GridOptions = {
  virtualScrolling: true,
  rowHeight: (row: any, index: number) => {
    // Dynamic heights based on content
    if (row.isExpanded) return 120;
    if (row.hasImage) return 80;
    return 48;
  }
};

Virtual Scrolling CSS Optimization

/* Automatic hardware acceleration */
.grid-virtual-container .grid-row {
  position: absolute;
  will-change: transform;
  contain: layout style paint;
  transform: translate3d(0, 0, 0); /* Force GPU acceleration */
}

Server-Side Operations

Server-side operations move filtering, sorting, and pagination to the server, handling datasets too large for client-side processing.

Basic Setup

// 1. Implement ServerSideDataSource
class MyDataSource implements ServerSideDataSource {
  constructor(private http: HttpClient) {}

  getRows(request: ServerSideRequest): Observable {
    const params = {
      startRow: request.startRow.toString(),
      endRow: request.endRow.toString(),
      sortModel: JSON.stringify(request.sortModel),
      filterModel: JSON.stringify(request.filterModel),
      search: request.searchTerm || ''
    };

    return this.http.get('/api/data', { params });
  }
}

// 2. Configure grid
const gridOptions: GridOptions = {
  serverSideOperations: {
    enabled: true,
    dataSource: new MyDataSource(httpClient),
    blockSize: 100,              // Records per request
    maxBlocksInCache: 50,        // Cache limit
    debounceMs: 300             // Request debouncing
  }
};

Server API Contract

// Request format
interface ServerSideRequest {
  startRow: number;              // Start index (0-based)
  endRow: number;               // End index (exclusive)
  sortModel: SortModel[];       // Sort configuration
  filterModel: FilterModel;     // Filter configuration
  searchTerm?: string;          // Global search
}

// Response format
interface ServerSideResponse {
  rowData: RowData[];           // Actual data rows
  rowCount: number;             // Rows in this response
  totalRecords: number;         // Total available records
  lastRow?: number;             // For infinite scroll
}

Advanced Filtering

// Filter model structure
interface FilterModel {
  [field: string]: {
    type: 'text' | 'number' | 'date' | 'select';
    filter: any;                // Filter value
    filterType: 'contains' | 'equals' | 'startsWith' | 'lessThan' | 'greaterThan';
    condition?: 'AND' | 'OR';   // For multiple conditions
  };
}

// Example server-side filter processing (Node.js)
function buildWhereClause(filterModel: FilterModel): string {
  const conditions: string[] = [];

  Object.entries(filterModel).forEach(([field, filter]) => {
    switch (filter.filterType) {
      case 'contains':
        conditions.push(`${field} ILIKE '%${filter.filter}%'`);
        break;
      case 'equals':
        conditions.push(`${field} = '${filter.filter}'`);
        break;
      case 'greaterThan':
        conditions.push(`${field} > ${filter.filter}`);
        break;
      // ... more filter types
    }
  });

  return conditions.join(' AND ');
}

Data Streaming

Data streaming loads data in chunks, enabling smooth interaction with massive datasets while maintaining memory efficiency.

Configuration

// 1. Implement DataProvider
class MyDataProvider implements DataProvider {
  constructor(private http: HttpClient, private baseUrl: string) {}

  getChunk(startIndex: number, endIndex: number): Observable {
    const params = { start: startIndex, end: endIndex };
    return this.http.get(`${this.baseUrl}/chunk`, { params });
  }

  getTotalCount(): Observable {
    return this.http.get(`${this.baseUrl}/count`);
  }
}

// 2. Configure streaming
const gridOptions: GridOptions = {
  streaming: {
    enabled: true,
    dataProvider: new MyDataProvider(httpClient, '/api/users'),
    chunkSize: 100,              // Records per chunk
    maxChunksInMemory: 50,       // Memory limit
    preloadChunks: 2,            // Chunks to prefetch
    enablePrefetching: true      // Background loading
  }
};

Streaming Performance Tuning

// Advanced streaming configuration
const streamingConfig: StreamingOptions = {
  enabled: true,
  dataProvider: myProvider,

  // Chunk management
  chunkSize: 100,                    // Optimal: 50-200 records
  maxChunksInMemory: 50,             // Adjust based on available memory
  preloadChunks: 3,                  // More = smoother, but more memory

  // Performance
  enablePrefetching: true,           // Background loading
  maxConcurrentRequests: 3,          // Parallel chunk loading
  retryAttempts: 3,                  // Network resilience
  retryDelay: 1000,                  // Exponential backoff

  // Memory management
  memoryThreshold: 100,              // MB before cleanup
  lazyLoadThreshold: 10              // Trigger distance
};

Streaming with Virtual Scrolling

// Optimal configuration for 10M+ records
const gridOptions: GridOptions = {
  // Combine virtual scrolling + streaming
  virtualScrolling: true,
  rowHeight: 48,
  maxRowsInDom: 20,

  streaming: {
    enabled: true,
    dataProvider: myProvider,
    chunkSize: 100,
    enablePrefetching: true
  },

  // Disable client-side pagination
  pagination: false
};

Memory Management

Automatic Memory Management

The grid includes sophisticated memory management:

class DataStreamingService {
  // Automatic cleanup triggers
  private cleanupMemory(): void {
    // 1. Remove chunks far from viewport
    // 2. Keep recently accessed chunks
    // 3. Respect memory thresholds
    // 4. Prioritize by usage patterns
  }

  // Memory monitoring
  memoryStats = computed(() => ({
    chunksInMemory: this.chunks().size,
    totalRows: this.calculateTotalRows(),
    memoryUsage: this.formatBytes(this.memoryUsage),
    isNearMemoryLimit: this.isNearLimit()
  }));
}

Manual Memory Control

// Get memory statistics
const stats = grid.getStreamingStats();
console.log('Memory usage:', stats.memoryStats.memoryUsage);
console.log('Chunks in memory:', stats.memoryStats.chunksInMemory);

// Force cleanup if needed
if (stats.memoryStats.isNearMemoryLimit) {
  streamingService.clearCache();
}

// Configure memory limits
const config: StreamingConfig = {
  memoryThreshold: 150,        // 150MB limit
  maxChunksInMemory: 30,       // Reduce cache size
  preloadChunks: 1            // Minimal prefetching
};

Browser Memory Optimization

// Component lifecycle management
export class MyGridComponent implements OnDestroy {
  ngOnDestroy(): void {
    // Automatic cleanup on component destruction
    this.streamingService.destroy();
    this.serverSideService.clearCache();
  }
}

// Browser memory pressure handling
if (typeof window !== 'undefined' && 'memory' in performance) {
  // Monitor browser memory usage
  const memoryInfo = (performance as any).memory;
  if (memoryInfo.usedJSHeapSize > memoryInfo.jsHeapSizeLimit * 0.8) {
    // Trigger aggressive cleanup
    this.streamingService.clearCache();
  }
}

Configuration Examples

const gridOptions: GridOptions = {
  // Traditional client-side mode
  pagination: true,
  paginationPageSize: 50,
  enableSorting: true,
  enableFiltering: true,
  enableSelection: true
};

// Simple usage

const gridOptions: GridOptions = {
  // Client-side with virtual scrolling
  virtualScrolling: true,
  rowHeight: 48,
  maxRowsInDom: 25,
  rowBuffer: 5,

  // Client-side operations
  enableSorting: true,
  enableFiltering: true,
  pagination: false  // Disabled for virtual scrolling
};

const gridOptions: GridOptions = {
  // Server-side operations
  serverSideOperations: {
    enabled: true,
    dataSource: new MyServerSideDataSource(),
    blockSize: 100,
    maxBlocksInCache: 30,
    debounceMs: 300
  },

  // Keep pagination for UX
  pagination: true,
  paginationPageSize: 100
};

const gridOptions: GridOptions = {
  // Full performance stack
  virtualScrolling: true,
  rowHeight: 48,
  maxRowsInDom: 20,

  serverSideOperations: {
    enabled: true,
    dataSource: myDataSource,
    blockSize: 200,
    maxBlocksInCache: 50
  },

  streaming: {
    enabled: true,
    dataProvider: myStreamingProvider,
    chunkSize: 200,
    maxChunksInMemory: 30,
    enablePrefetching: true
  },

  // Disable client-side pagination
  pagination: false
};

const gridOptions: GridOptions = {
  // Maximum performance configuration
  virtualScrolling: true,
  rowHeight: 40,                    // Smaller rows for density
  maxRowsInDom: 15,                 // Minimal DOM
  rowBuffer: 3,

  serverSideOperations: {
    enabled: true,
    dataSource: enterpriseDataSource,
    blockSize: 500,                 // Larger blocks for efficiency
    maxBlocksInCache: 20,           // Conservative caching
    debounceMs: 500,                // Less frequent requests
    maxConcurrentRequests: 2
  },

  streaming: {
    enabled: true,
    dataProvider: enterpriseStreamingProvider,
    chunkSize: 500,
    maxChunksInMemory: 20,
    preloadChunks: 1,               // Minimal prefetching
    enablePrefetching: false,       // Disable for maximum control
    memoryThreshold: 50             // Strict memory limit
  },

  // Enterprise features
  header: {
    showTitle: true,
    title: "Enterprise Data Grid (10M+ Records)",
    showControls: true
  },

  footer: {
    showPagination: false,          // Virtual scrolling handles navigation
    showResultsInfo: true
  }
};

Best Practices

1. Dataset Size Guidelines

2. Server API Optimization

// Optimized server endpoint
app.get('/api/data', async (req, res) => {
  const { startRow, endRow, sortModel, filterModel } = req.query;

  // 1. Parse parameters safely
  const start = Math.max(0, parseInt(startRow) || 0);
  const end = Math.min(parseInt(endRow) || 100, start + 1000); // Limit chunk size

  // 2. Build efficient database query
  let query = db.select('*').from('large_table');

  // 3. Apply filters (use indexes!)
  if (filterModel) {
    const filters = JSON.parse(filterModel);
    Object.entries(filters).forEach(([field, filter]) => {
      // Use parameterized queries to prevent SQL injection
      query = query.where(field, filter.filterType, filter.filter);
    });
  }

  // 4. Apply sorting (use indexes!)
  if (sortModel) {
    const sorts = JSON.parse(sortModel);
    sorts.forEach(sort => {
      query = query.orderBy(sort.field, sort.direction);
    });
  }

  // 5. Apply pagination with LIMIT/OFFSET
  query = query.limit(end - start).offset(start);

  // 6. Execute with connection pooling
  const [data, totalCount] = await Promise.all([
    query.execute(),
    db.count('*').from('large_table').first()
  ]);

  res.json({
    rowData: data,
    rowCount: data.length,
    totalRecords: totalCount.count
  });
});

3. Column Configuration

// Optimized column definitions
const columnDefs: ColumnDefinition[] = [
  {
    field: 'id',
    headerName: 'ID',
    width: 80,                    // Fixed width for performance
    sortable: true,
    filterable: false,            // Don't filter ID columns
    pinned: 'left'               // Pin important columns
  },
  {
    field: 'name',
    headerName: 'Name',
    width: 200,
    sortable: true,
    filterable: true,
    cellRenderer: 'textRenderer', // Use simple renderers
    valueFormatter: (params) => {
      // Lightweight formatting only
      return params.value?.substring(0, 50) + '...';
    }
  },
  {
    field: 'status',
    headerName: 'Status',
    width: 120,
    cellTemplate: statusTemplate,  // Cached template
    sortable: true,
    filterable: true
  },
  {
    field: 'actions',
    headerName: 'Actions',
    width: 150,
    sortable: false,
    filterable: false,
    cellTemplate: actionsTemplate,
    // Disable for virtual scrolling performance
    visible: !isVirtualScrollingEnabled
  }
];

Troubleshooting

Common Issues and Solutions

// Optimize initial chunk size
streaming: {
  chunkSize: 50,                 // Smaller initial chunks
  preloadChunks: 1,             // Less prefetching
  enablePrefetching: false      // Disable until loaded
}

// Use server-side pagination
serverSideOperations: {
  enabled: true,
  blockSize: 25                 // Smaller initial block
}

// Aggressive cleanup
streaming: {
  maxChunksInMemory: 10,        // Strict limit
  memoryThreshold: 30           // Lower threshold
}

// Manual cleanup
ngOnDestroy(): void {
  this.streamingService.destroy();
  this.serverSideService.clearCache();
}

// Optimize virtual scrolling
virtualScrolling: true,
rowHeight: 40,                  // Smaller, fixed height
maxRowsInDom: 15,              // Fewer DOM elements
rowBuffer: 2                    // Minimal buffer

// Disable expensive features
enableSelection: false,         // Disable if not needed
cellTemplate: null              // Use simple text rendering

// Server-side operations
serverSideOperations: {
  enabled: true,
  debounceMs: 500               // Reduce request frequency
}

// Database optimization
// - Add indexes on filterable/sortable columns
// - Use database views for complex queries
// - Implement query caching

// Robust networking
streaming: {
  retryAttempts: 5,
  retryDelay: 2000,
  maxConcurrentRequests: 1      // Reduce concurrent load
}

// Implement offline support
const networkHandler = {
  handleOffline: () => {
    // Switch to cached data
    this.useOfflineMode();
  },
  handleOnline: () => {
    // Resume streaming
    this.resumeStreaming();
  }
};

Performance Benchmarks

Memory Usage by Dataset Size

*Traditional mode crashes browser

Scroll Performance (FPS)

Initial Load Time

const enterpriseConfig: GridOptions = {
  virtualScrolling: true,
  rowHeight: 44,
  maxRowsInDom: 18,

  serverSideOperations: {
    enabled: true,
    blockSize: 300,
    maxBlocksInCache: 25
  },

  streaming: {
    enabled: true,
    chunkSize: 300,
    maxChunksInMemory: 25,
    preloadChunks: 1
  }
};

Security Considerations

Data Protection

// Secure data handling
class SecureDataProvider implements DataProvider {
  getChunk(startIndex: number, endIndex: number): Observable {
    // 1. Validate parameters
    if (startIndex < 0 || endIndex < startIndex || endIndex - startIndex > 1000) {
      throw new Error('Invalid chunk parameters');
    }

    // 2. Use HTTPS endpoints
    return this.http.get(`https://secure-api.com/data`, {
      params: { start: startIndex, end: endIndex },
      headers: { 'Authorization': `Bearer ${this.authToken}` }
    }).pipe(
      // 3. Sanitize response data
      map(data => data.map(row => this.sanitizeRow(row)))
    );
  }

  private sanitizeRow(row: RowData): RowData {
    // Remove or escape potentially dangerous content
    const sanitized = { ...row };
    Object.keys(sanitized).forEach(key => {
      if (typeof sanitized[key] === 'string') {
        sanitized[key] = this.sanitizeHtml(sanitized[key]);
      }
    });
    return sanitized;
  }
}

State Storage Security

// Secure state persistence
class SecureStateService extends StateService {
  protected encrypt(data: string): string {
    // Use proper encryption (not just base64)
    return CryptoJS.AES.encrypt(data, this.getEncryptionKey()).toString();
  }

  protected decrypt(data: string): string {
    const bytes = CryptoJS.AES.decrypt(data, this.getEncryptionKey());
    return bytes.toString(CryptoJS.enc.Utf8);
  }

  private getEncryptionKey(): string {
    // Use environment-specific keys
    return environment.gridStateEncryptionKey;
  }
}

Conclusion

The ng-vui-grid now provides enterprise-grade performance for datasets of any size. By combining virtual scrolling, server-side operations, intelligent streaming, and memory management, the grid maintains consistent performance from thousands to millions of records.

// For 10M+ records - copy this configuration
const largeDatasetConfig: GridOptions = {
  virtualScrolling: true,
  rowHeight: 48,
  maxRowsInDom: 20,

  serverSideOperations: {
    enabled: true,
    dataSource: myDataSource,
    blockSize: 200,
    maxBlocksInCache: 30
  },

  streaming: {
    enabled: true,
    dataProvider: myProvider,
    chunkSize: 200,
    maxChunksInMemory: 30,
    enablePrefetching: true
  },

  pagination: false
};