Audit Logging

For Data Analysts

Olytix Core's audit logging captures security-related events for compliance, forensic analysis, and monitoring. Every data access, policy evaluation, and security decision is logged with full context.

Overview

Audit logging in Olytix Core:

• Records all security-relevant events
• Captures full request context (user, IP, timestamp)
• Tracks policy evaluations and decisions
• Supports multiple output handlers
• Enables compliance reporting and forensic analysis

Audit Event Types

Event Type	Description	When Logged
data_access	User accessed data	Every successful query
access_denied	Access was denied	RLS or auth rejection
policy_evaluated	Policies were evaluated	Every query with policies
masking_applied	Column masking was applied	Query with masked columns
policy_created	New policy created	Policy management
policy_updated	Policy was modified	Policy management
policy_deleted	Policy was removed	Policy management
authentication	User authenticated	Login events
authorization	Authorization check	Permission checks

Audit Entry Structure

Each audit entry contains comprehensive context:

{
  "request_id": "req-abc123-def456",
  "user_id": "user123",
  "user_email": "analyst@company.com",
  "user_roles": ["analyst", "finance_team"],
  "client_ip": "10.0.1.50",
  "cube_name": "orders",
  "measures_requested": ["total_revenue", "order_count"],
  "dimensions_requested": ["region", "order_date"],
  "rls_policies_applied": ["regional_access"],
  "masking_policies_applied": ["email_masking"],
  "rls_predicates": ["(region = 'north_america')"],
  "columns_masked": ["customer_email"],
  "policy_evaluation_ms": 1.5,
  "rows_returned": 150,
  "access_granted": true,
  "denial_reason": null,
  "event_type": "data_access",
  "timestamp": "2025-01-20T14:30:00.000Z",
  "additional_data": {
    "query_duration_ms": 45,
    "cache_hit": false
  }
}

Configuration

Environment Variables

# Enable audit logging
OLYTIX_SECURITY__AUDIT_ENABLED=true

# Log level for audit events
OLYTIX_SECURITY__AUDIT_LEVEL=INFO

# File handler configuration
OLYTIX_SECURITY__AUDIT_FILE_PATH=/var/log/olytix-core/audit.log
OLYTIX_SECURITY__AUDIT_FILE_ROTATION=daily
OLYTIX_SECURITY__AUDIT_FILE_RETENTION=90

# Database handler configuration
OLYTIX_SECURITY__AUDIT_DB_URL=postgresql://user:pass@host/audit_db
OLYTIX_SECURITY__AUDIT_DB_TABLE=security_audit_log

Project Configuration

# olytix-core_project.yml
security:
  audit:
    enabled: true

    # Event types to log
    events:
      - data_access
      - access_denied
      - policy_evaluated
      - masking_applied
      - policy_created
      - policy_updated
      - policy_deleted
      - authentication

    # Handlers (multiple supported)
    handlers:
      # File handler
      - type: file
        path: /var/log/olytix-core/audit.log
        format: json
        rotation: daily
        retention_days: 90
        max_size_mb: 100

      # Database handler
      - type: database
        connection: ${AUDIT_DB_URL}
        table: security_audit_log

      # Logging handler (Python logging)
      - type: logging
        logger_name: security.audit
        level: INFO

      # Webhook handler
      - type: webhook
        url: https://siem.company.com/api/events
        headers:
          Authorization: Bearer ${SIEM_API_KEY}

    # Filtering
    filters:
      # Only log denied access for specific cubes
      denied_access_cubes:
        - sensitive_data
        - financial_reports

      # Exclude health checks
      exclude_paths:
        - /health
        - /metrics

Audit Handlers

File Handler

Write audit logs to files with rotation:

handlers:
  - type: file
    path: /var/log/olytix-core/audit.log
    format: json  # json or text
    rotation: daily  # daily, weekly, or size-based
    retention_days: 90
    max_size_mb: 100
    compress: true

Output (JSON format):

{"timestamp":"2025-01-20T14:30:00.000Z","event_type":"data_access","user_id":"user123","cube_name":"orders","access_granted":true}

Output (text format):

2025-01-20 14:30:00 | DATA_ACCESS | user123 | orders | GRANTED | 150 rows

Database Handler

Store audit logs in a database table:

handlers:
  - type: database
    connection: postgresql://user:pass@host/audit_db
    table: security_audit_log
    batch_size: 100
    flush_interval_seconds: 5

Table Schema:

CREATE TABLE security_audit_log (
    id BIGSERIAL PRIMARY KEY,
    request_id VARCHAR(64) NOT NULL,
    user_id VARCHAR(255) NOT NULL,
    user_email VARCHAR(255),
    user_roles TEXT[],
    client_ip INET,
    cube_name VARCHAR(255),
    measures_requested TEXT[],
    dimensions_requested TEXT[],
    rls_policies_applied TEXT[],
    masking_policies_applied TEXT[],
    rls_predicates TEXT[],
    columns_masked TEXT[],
    policy_evaluation_ms DECIMAL(10,2),
    rows_returned INTEGER,
    access_granted BOOLEAN NOT NULL,
    denial_reason TEXT,
    event_type VARCHAR(50) NOT NULL,
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    additional_data JSONB,

    INDEX idx_audit_timestamp (timestamp),
    INDEX idx_audit_user (user_id),
    INDEX idx_audit_cube (cube_name),
    INDEX idx_audit_event_type (event_type)
);

Logging Handler

Integrate with Python's logging system:

handlers:
  - type: logging
    logger_name: security.audit
    level: INFO

Python logging configuration:

import logging

# Configure audit logger
audit_logger = logging.getLogger('security.audit')
audit_logger.setLevel(logging.INFO)

# Add handler (e.g., to send to external SIEM)
handler = logging.handlers.SysLogHandler(address='/dev/log')
audit_logger.addHandler(handler)

Webhook Handler

Send audit events to external systems:

handlers:
  - type: webhook
    url: https://siem.company.com/api/events
    method: POST
    headers:
      Authorization: Bearer ${SIEM_API_KEY}
      Content-Type: application/json
    batch_size: 50
    retry_count: 3
    timeout_seconds: 10

In-Memory Handler (Testing)

For development and testing:

handlers:
  - type: memory
    max_entries: 10000

Querying Audit Logs

Via API

# Get recent audit events
curl http://localhost:8000/api/v1/security/audit \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Filter by user
curl "http://localhost:8000/api/v1/security/audit?user_id=user123" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Filter by date range
curl "http://localhost:8000/api/v1/security/audit?start_date=2025-01-01&end_date=2025-01-31" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Filter by event type
curl "http://localhost:8000/api/v1/security/audit?event_type=access_denied" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Filter by cube
curl "http://localhost:8000/api/v1/security/audit?cube_name=orders" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Response:

{
  "entries": [
    {
      "request_id": "req-abc123",
      "user_id": "user123",
      "user_email": "analyst@company.com",
      "cube_name": "orders",
      "event_type": "data_access",
      "access_granted": true,
      "timestamp": "2025-01-20T14:30:00.000Z"
    }
  ],
  "pagination": {
    "total": 1542,
    "limit": 100,
    "offset": 0
  }
}

Via CLI

# Recent events
olytix-core security audit-logs --limit 100

# Filter by user
olytix-core security audit-logs --user-id user123

# Filter by date
olytix-core security audit-logs --start-date 2025-01-01 --end-date 2025-01-31

# Export to file
olytix-core security audit-logs --format json --output audit-report.json

Via SQL (Database Handler)

-- Recent access denials
SELECT
    timestamp,
    user_id,
    user_email,
    cube_name,
    denial_reason
FROM security_audit_log
WHERE event_type = 'access_denied'
  AND timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY timestamp DESC;

-- User access patterns
SELECT
    user_id,
    cube_name,
    COUNT(*) as access_count,
    MAX(timestamp) as last_access
FROM security_audit_log
WHERE event_type = 'data_access'
  AND timestamp >= NOW() - INTERVAL '7 days'
GROUP BY user_id, cube_name
ORDER BY access_count DESC;

-- Policy evaluation performance
SELECT
    cube_name,
    AVG(policy_evaluation_ms) as avg_eval_ms,
    MAX(policy_evaluation_ms) as max_eval_ms,
    COUNT(*) as query_count
FROM security_audit_log
WHERE event_type = 'policy_evaluated'
  AND timestamp >= NOW() - INTERVAL '24 hours'
GROUP BY cube_name
ORDER BY avg_eval_ms DESC;

Audit Logging Examples

Data Access Event

# Automatically logged when user queries data
await audit_logger.log_access(
    AuditEntry(
        request_id="req-abc123",
        user_id="user123",
        user_email="analyst@company.com",
        user_roles=["analyst"],
        client_ip="10.0.1.50",
        cube_name="orders",
        measures_requested=["total_revenue"],
        dimensions_requested=["region"],
        rls_policies_applied=["regional_access"],
        rls_predicates=["(region = 'north_america')"],
        rows_returned=150,
        access_granted=True,
        event_type=AuditEventType.DATA_ACCESS
    )
)

Access Denial Event

# Logged when RLS denies access
await audit_logger.log_denial(
    context=security_context,
    cube_name="sensitive_data",
    reason="No matching RLS policies for user"
)

Policy Change Event

# Logged when policies are modified
await audit_logger.log_policy_change(
    context=admin_context,
    policy_name="regional_access",
    action="updated",
    policy_type="access"
)

Compliance Reporting

Data Access Report

Generate reports for compliance audits:

olytix-core security audit-report \
  --type data-access \
  --start-date 2025-01-01 \
  --end-date 2025-01-31 \
  --format pdf \
  --output january-access-report.pdf

User Activity Summary

curl -X POST http://localhost:8000/api/v1/security/audit/report \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "report_type": "user_activity",
    "start_date": "2025-01-01",
    "end_date": "2025-01-31",
    "group_by": ["user_id", "cube_name"]
  }'

Response:

{
  "report": {
    "period": "2025-01-01 to 2025-01-31",
    "summary": {
      "total_queries": 15420,
      "unique_users": 45,
      "cubes_accessed": 12,
      "access_denials": 23
    },
    "by_user": [
      {
        "user_id": "user123",
        "user_email": "analyst@company.com",
        "query_count": 542,
        "cubes_accessed": ["orders", "customers"],
        "last_access": "2025-01-31T16:45:00Z"
      }
    ]
  }
}

Policy Evaluation Report

curl -X POST http://localhost:8000/api/v1/security/audit/report \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "report_type": "policy_evaluation",
    "start_date": "2025-01-01",
    "end_date": "2025-01-31"
  }'

Alerting

Configure alerts for security events:

# olytix-core_project.yml
security:
  audit:
    alerts:
      # Alert on access denials
      - name: access_denied_alert
        event_type: access_denied
        threshold: 10
        window_minutes: 5
        action:
          type: webhook
          url: https://alerts.company.com/security

      # Alert on unusual access patterns
      - name: unusual_access_alert
        event_type: data_access
        condition: "rows_returned > 100000"
        action:
          type: email
          recipients:
            - security@company.com

      # Alert on policy changes
      - name: policy_change_alert
        event_types:
          - policy_created
          - policy_updated
          - policy_deleted
        action:
          type: slack
          webhook: ${SLACK_SECURITY_WEBHOOK}

Best Practices

Storage and Retention

1. Set appropriate retention - Balance compliance needs with storage costs
2. Archive old logs - Move to cold storage after active period
3. Encrypt sensitive data - Protect audit logs at rest
4. Backup regularly - Audit logs are critical for compliance

Performance

1. Use batch writes - Reduce I/O overhead
2. Index key fields - Enable efficient querying
3. Monitor log volume - Plan capacity accordingly
4. Use async handlers - Don't block queries for logging

Compliance

1. Log all access - Don't filter too aggressively
2. Include full context - User, IP, timestamp, query details
3. Protect log integrity - Use write-once storage or signing
4. Document retention policies - Meet regulatory requirements

Monitoring

1. Alert on anomalies - Unusual access patterns
2. Monitor denied access - Potential security issues
3. Track policy changes - Audit administrative actions
4. Review regularly - Periodic security audits

Troubleshooting

Logs Not Appearing

1. Verify audit is enabled in configuration
2. Check handler configuration
3. Verify file permissions (for file handler)
4. Check database connectivity (for database handler)

Performance Impact

1. Increase batch size
2. Use async handlers
3. Add database indexes
4. Consider log sampling for high-volume environments

Missing Events

1. Check event type filter
2. Verify path exclusions
3. Check handler error logs
4. Ensure all handlers are healthy

Next Steps

• Configure row-level security
• Configure column masking
• Security overview