Project Structure

For Everyone

Understanding how a Olytix Core project is organized helps you work efficiently and maintain a clean codebase. This guide covers the standard project structure and how all the pieces fit together.

Standard Project Layout

When you run olytix-core init, Olytix Core creates this structure:

my-analytics/
├── olytix-core_project.yml          # Project configuration
├── profiles.yml              # Connection profiles (optional, for local dev)
├── .env                      # Environment variables (gitignored)
├── .gitignore                # Git ignore patterns
│
├── sources/                  # Data source definitions
│   ├── raw.yml              # Raw data sources
│   └── external.yml         # External API sources
│
├── models/                   # SQL transformation models
│   ├── staging/             # Staging models (stg_*)
│   │   ├── stg_orders.sql
│   │   └── stg_customers.sql
│   ├── intermediate/        # Intermediate models (int_*)
│   │   └── int_order_items.sql
│   └── marts/               # Mart models (fct_*, dim_*)
│       ├── fct_orders.sql
│       ├── dim_customers.sql
│       └── schema.yml       # Model documentation & tests
│
├── cubes/                    # Semantic cube definitions
│   ├── orders.yml
│   ├── customers.yml
│   └── products.yml
│
├── metrics/                  # Business metric definitions
│   ├── revenue.yml
│   ├── customers.yml
│   └── operations.yml
│
├── macros/                   # Reusable SQL/Jinja macros
│   ├── date_utils.sql
│   └── currency.sql
│
├── seeds/                    # Static CSV data files
│   ├── country_codes.csv
│   └── exchange_rates.csv
│
├── tests/                    # Custom data tests
│   └── assert_positive_revenue.sql
│
├── snapshots/                # SCD (slowly changing dimension) definitions
│   └── customers_snapshot.sql
│
└── target/                   # Compiled output (gitignored)
    ├── manifest.json        # Full project manifest
    ├── compiled/            # Compiled SQL
    └── run_results.json     # Last run results

Configuration Files

olytix-core_project.yml

The main project configuration file:

# Project metadata
name: my_analytics
version: 1.0.0
description: "E-commerce analytics platform"

# Required Olytix Core version
require-olytix-core-version: ">=1.0.0"

# Warehouse connection
warehouse:
  type: postgresql
  host: ${OLYTIX_DB_HOST}
  port: ${OLYTIX_DB_PORT:-5432}
  database: ${OLYTIX_DB_NAME}
  user: ${OLYTIX_DB_USER}
  password: ${OLYTIX_DB_PASSWORD}

# Model configurations
models:
  my_analytics:
    # Default materialization
    +materialized: view

    # Staging models
    staging:
      +schema: staging
      +materialized: view

    # Mart models
    marts:
      +schema: marts
      +materialized: table

# Cube configurations
cubes:
  +meta:
    owner: analytics-team

# Seed configurations
seeds:
  my_analytics:
    +schema: seeds

# Variable definitions
vars:
  start_date: '2020-01-01'
  default_currency: 'USD'

# Query limits for safety
query-timeout: 300  # seconds
row-limit: 1000000

# Logging
log-level: INFO

profiles.yml (Optional)

For local development with multiple environments:

my_analytics:
  target: dev
  outputs:
    dev:
      type: postgresql
      host: localhost
      port: 5432
      database: analytics_dev
      user: developer
      password: ${DEV_PASSWORD}
      schema: dev_{{ env_var('USER') }}

    staging:
      type: postgresql
      host: staging.db.example.com
      port: 5432
      database: analytics
      user: staging_user
      password: ${STAGING_PASSWORD}
      schema: staging

    prod:
      type: snowflake
      account: ${SNOWFLAKE_ACCOUNT}
      user: ${SNOWFLAKE_USER}
      password: ${SNOWFLAKE_PASSWORD}
      warehouse: ANALYTICS_WH
      database: ANALYTICS
      schema: PROD

.env File

Store sensitive values (never commit this file):

# Database credentials
OLYTIX_DB_HOST=localhost
OLYTIX_DB_PORT=5432
OLYTIX_DB_NAME=analytics
OLYTIX_DB_USER=admin
OLYTIX_DB_PASSWORD=secret123

# API keys
OLYTIX_API_KEY=your-api-key

# Feature flags
OLYTIX_ENABLE_CACHE=true
OLYTIX_LOG_LEVEL=DEBUG

Directory Details

sources/

Define your raw data sources:

# sources/raw.yml
version: 2

sources:
  - name: raw
    database: production
    schema: public

    # Freshness checking
    freshness:
      warn_after: { count: 12, period: hour }
      error_after: { count: 24, period: hour }

    tables:
      - name: orders
        description: "Raw order data from e-commerce platform"
        loaded_at_field: _loaded_at

        columns:
          - name: id
            description: "Primary key"
            tests:
              - unique
              - not_null

models/

Organized in layers:

models/
├── staging/      # 1:1 with sources, cleaning & renaming
├── intermediate/ # Complex transformations
└── marts/        # Business-facing tables

Each folder can have a schema.yml for documentation and tests:

# models/marts/schema.yml
version: 2

models:
  - name: fct_orders
    description: "Order fact table"
    config:
      materialized: table
    columns:
      - name: order_id
        tests: [unique, not_null]

cubes/

Semantic layer definitions:

# cubes/orders.yml
cubes:
  - name: orders
    sql: "SELECT * FROM {{ ref('fct_orders') }}"
    description: "Order analytics cube"

    measures:
      - name: total_revenue
        type: sum
        sql: amount

    dimensions:
      - name: order_date
        type: time
        sql: order_date

metrics/

Business metrics built on cubes:

# metrics/revenue.yml
metrics:
  - name: monthly_revenue
    type: simple
    expression: orders.total_revenue
    time_grain: month
    description: "Total monthly revenue"

  - name: revenue_growth
    type: derived
    expression: "(current.monthly_revenue - previous.monthly_revenue) / previous.monthly_revenue"
    description: "Month-over-month revenue growth"

macros/

Reusable SQL functions:

-- macros/date_utils.sql
{% macro fiscal_quarter(date_column) %}
CASE
  WHEN EXTRACT(MONTH FROM {{ date_column }}) IN (7,8,9) THEN 'Q1'
  WHEN EXTRACT(MONTH FROM {{ date_column }}) IN (10,11,12) THEN 'Q2'
  WHEN EXTRACT(MONTH FROM {{ date_column }}) IN (1,2,3) THEN 'Q3'
  ELSE 'Q4'
END
{% endmacro %}

seeds/

Static reference data (CSV files):

# seeds/country_codes.csv
code,name,region
US,United States,North America
CA,Canada,North America
GB,United Kingdom,Europe
DE,Germany,Europe

Load with olytix-core seed.

tests/

Custom data quality tests:

-- tests/assert_positive_revenue.sql
-- This test fails if any orders have negative revenue

SELECT order_id, total_amount
FROM {{ ref('fct_orders') }}
WHERE total_amount < 0

target/

Compiled output (auto-generated, gitignored):

target/
├── manifest.json       # Complete project metadata
├── compiled/           # Rendered SQL files
│   └── models/
│       └── marts/
│           └── fct_orders.sql
├── run_results.json    # Results of last run
└── graph.json          # Dependency graph

Dependency Flow

┌─────────────────────────────────────────────────────────────────────┐
│                         Data Flow                                    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐      │
│  │ Sources  │───►│ Staging  │───►│  Marts   │───►│  Cubes   │      │
│  │ (raw)    │    │ Models   │    │  Models  │    │          │      │
│  └──────────┘    └──────────┘    └──────────┘    └────┬─────┘      │
│       │                                                │            │
│       │              ┌─────────────┐                   │            │
│       └──────────────│   Seeds     │───────────────────┘            │
│                      └─────────────┘                   │            │
│                                                        ▼            │
│                                                  ┌──────────┐       │
│                                                  │ Metrics  │       │
│                                                  └──────────┘       │
│                                                        │            │
│                                                        ▼            │
│                                                  ┌──────────┐       │
│                                                  │   API    │       │
│                                                  └──────────┘       │
└─────────────────────────────────────────────────────────────────────┘

Naming Conventions

Type	Convention	Example
Staging models	stg_<source>_<table>	stg_raw_orders
Intermediate models	int_<description>	int_order_items_joined
Fact models	fct_<entity>	fct_orders
Dimension models	dim_<entity>	dim_customers
Cubes	<entity> (singular)	orders, customer
Metrics	<descriptive_name>	monthly_revenue

Configuration Hierarchy

Settings cascade from project to folder to file:

# olytix-core_project.yml (project level)
models:
  my_project:
    +materialized: view        # Default for all models

    marts:
      +materialized: table     # Override for marts folder

-- models/marts/special.sql (file level)
{{ config(materialized='incremental') }}  -- Override for this file

Priority: File config > Folder config > Project config

Environment Variables

Olytix Core uses ${VAR} syntax for environment variables:

warehouse:
  host: ${OLYTIX_DB_HOST}
  port: ${OLYTIX_DB_PORT:-5432}  # With default value
  password: ${OLYTIX_DB_PASSWORD}

Variable resolution order:

1. Environment variables
2. .env file
3. Default values (if specified)

Working with Git

Recommended .gitignore:

# Olytix Core
target/
logs/
.user.yml

# Environment
.env
.env.local
*.local

# Python
__pycache__/
*.pyc
.venv/

# IDE
.idea/
.vscode/
*.swp

Commands for Project Management

# Initialize new project
olytix-core init

# Compile project (validate without running)
olytix-core compile

# Run all models
olytix-core run

# Run specific models
olytix-core run --select fct_orders
olytix-core run --select +fct_orders  # With dependencies
olytix-core run --select marts.*      # All in folder

# Test data quality
olytix-core test

# Generate documentation
olytix-core docs generate

# View dependency graph
olytix-core graph

Next Steps

Now that you understand the project structure:

1. Explore the Analyst Manual →
2. Learn about data modeling →
3. Set up CI/CD →
4. Deploy to production →