This directory contains the refactored GitLab integration test system that replaces the monolithic bash scripts with modular, maintainable Go-based tools.

Overview

The refactored system consists of:

Configuration-based Seeding: JSON configuration files define the seed data
Go-based Seeder: A Go tool that reads configuration and creates GitLab resources
Test Framework: A Go-based test runner that executes configurable test scenarios
Modular Scripts: Clean shell scripts that orchestrate the components

Directory Structure

scripts/local-gitlab/
├── configs/
│   ├── seed-data.json          # Defines GitLab resources to create
│   └── test-scenarios.json     # Defines integration test scenarios
├── seeder/
│   ├── main.go                 # Go-based seeder implementation
│   └── go.mod                  # Seeder dependencies
├── test-runner/
│   ├── main.go                 # Go-based test runner implementation
│   └── go.mod                  # Test runner dependencies
├── start-ee.sh                # Refactored main entry point
├── seed.sh                    # New seeding script using Go seeder
├── integration-tests.sh       # New test script using Go test runner
├── add-test-scenario.sh       # Utility to add new test scenarios
└── README-refactored.md       # This file

Quick Start

Running All Tests (Refactored Version)

# Run the refactored integration tests
./start-ee.sh

# Or with custom parameters
./start-ee.sh true false latest

Script Arguments

Quick Reference

Script	Arguments	Purpose
`start-ee.sh`	7 optional args	Main entry point - runs entire test suite
`seed.sh`	3 optional args	Seeds GitLab with test data
`integration-tests.sh`	3 optional args	Runs integration tests only
`run-ee.sh`	4 optional args	Starts GitLab container (internal)

`start-ee.sh` Arguments

The main entry point script accepts up to 7 optional arguments. All arguments have sensible defaults if not provided.

Usage:

./start-ee.sh [STOP_GITLAB_WHEN_FINISHED] [PERSIST_GITLAB_LOCALLY] [GITLAB_IMAGE_TAG] [GITLAB_HOME] [GITLAB_HOST] [GITLAB_URL] [LOCAL_GITLAB_GHORG_DIR]

Argument	Default	Description
`STOP_GITLAB_WHEN_FINISHED`	`'true'`	Whether to stop and remove the GitLab container after tests complete. Set to `'false'` to keep GitLab running for debugging.
`PERSIST_GITLAB_LOCALLY`	`'false'`	Whether to persist GitLab data locally across container restarts. Set to `'true'` to keep data between runs.
`GITLAB_IMAGE_TAG`	`'latest'`	GitLab Docker image tag to use. Can be specific version like `'16.4.0-ce.0'` or `'latest'`.
`GITLAB_HOME`	`"$HOME/ghorg/local-gitlab-ee-data-${GITLAB_IMAGE_TAG}"`	Directory where GitLab stores persistent data on the host machine.
`GITLAB_HOST`	`'gitlab.example.com'`	Hostname for the GitLab instance. Used for container networking and /etc/hosts entries.
`GITLAB_URL`	`'http://gitlab.example.com'`	Full URL to access the GitLab instance. Used by ghorg and the test tools.
`LOCAL_GITLAB_GHORG_DIR`	`"${HOME}/ghorg"`	Local directory where ghorg will clone repositories and store its working files.

Examples:

# Default behavior - run tests and clean up
./start-ee.sh

# Keep GitLab running after tests for debugging
./start-ee.sh false

# Use specific GitLab version and keep it running
./start-ee.sh false false 16.4.0-ce.0

# Full custom configuration
./start-ee.sh true true latest /tmp/gitlab-data gitlab.local http://gitlab.local /tmp/ghorg

Common Scenarios:

# Development - keep GitLab running for multiple test iterations
./start-ee.sh false false latest

# CI/CD - use clean environment and cleanup afterwards (default)
./start-ee.sh true false latest

# Testing specific GitLab version
./start-ee.sh true false 16.3.0-ce.0

# Custom data persistence for repeated testing
./start-ee.sh false true latest /data/gitlab-persistent

Individual Component Arguments

`seed.sh` Arguments

Seeds the GitLab instance with test data using the Go-based seeder.

Usage:

./seed.sh [API_TOKEN] [GITLAB_URL] [LOCAL_GITLAB_GHORG_DIR]

Argument	Default	Description
`API_TOKEN`	`"password"`	GitLab API token for authentication (default root password)
`GITLAB_URL`	`"http://gitlab.example.com"`	Full URL to the GitLab instance
`LOCAL_GITLAB_GHORG_DIR`	`"${HOME}/ghorg"`	Directory where ghorg stores its configuration and temp files

Example:

# Use defaults
./seed.sh

# Custom parameters
./seed.sh "my-token" "http://gitlab.local:8080" "/tmp/ghorg"

`integration-tests.sh` Arguments

Runs the integration tests using the Go-based test runner.

Usage:

./integration-tests.sh [LOCAL_GITLAB_GHORG_DIR] [API_TOKEN] [GITLAB_URL]

Argument	Default	Description
`LOCAL_GITLAB_GHORG_DIR`	`"${HOME}/ghorg"`	Directory where ghorg will clone repositories for testing
`API_TOKEN`	`"password"`	GitLab API token for authentication
`GITLAB_URL`	`"http://gitlab.example.com"`	Full URL to the GitLab instance

Example:

# Use defaults
./integration-tests.sh

# Custom parameters
./integration-tests.sh "/tmp/ghorg" "my-token" "http://gitlab.local:8080"

`run-ee.sh` Arguments (Internal)

Starts the GitLab Docker container. Called internally by start-ee.sh.

Usage:

./run-ee.sh [GITLAB_IMAGE_TAG] [GITLAB_HOME] [GITLAB_HOST] [PERSIST_GITLAB_LOCALLY]

Argument	Default	Description
`GITLAB_IMAGE_TAG`	`"latest"`	GitLab Docker image tag
`GITLAB_HOME`	Dynamic	Host directory for GitLab data persistence
`GITLAB_HOST`	`"gitlab.example.com"`	Container hostname
`PERSIST_GITLAB_LOCALLY`	`"false"`	Whether to persist data between container restarts

Go Tool Arguments (Direct Usage)

For advanced usage, you can run the Go tools directly:

Seeder (seeder/gitlab-seeder):

./gitlab-seeder [flags]
  -config string
        Path to seed data configuration file (default "configs/seed-data.json")
  -token string
        GitLab API token (required)
  -base-url string
        GitLab base URL (required)

Test Runner (test-runner/gitlab-test-runner):

./gitlab-test-runner [flags]
  -config string
        Path to test scenarios configuration file (default "configs/test-scenarios.json")
  -token string
        GitLab API token (required)
  -base-url string
        GitLab base URL (required)
  -ghorg-dir string
        Ghorg directory path (default "${HOME}/ghorg")
  -test string
        Run specific test by name (optional)
  -list
        List all available tests and exit

Examples:

# List all available test scenarios
./test-runner/gitlab-test-runner -list -token="password"

# Run specific test
./test-runner/gitlab-test-runner -test="all-groups-preserve-dir-output-dir" -token="password" -base-url="http://gitlab.example.com"

# Seed with custom config
./seeder/gitlab-seeder -config="my-seed-data.json" -token="password" -base-url="http://gitlab.example.com"

Running Individual Components

# Seed GitLab instance only
./seed.sh "password" "http://gitlab.example.com" "${HOME}/ghorg"

# Run integration tests only (assumes seeded instance)
./integration-tests.sh "${HOME}/ghorg" "password" "http://gitlab.example.com"

Configuration

Seed Data Configuration (`configs/seed-data.json`)

Defines the GitLab resources to create during seeding:

{
  "groups": [
    {
      "name": "my-group",
      "path": "my-group",
      "description": "My test group",
      "repositories": [
        {
          "name": "my-repo",
          "initialize_with_readme": true,
          "snippets": [
            {
              "title": "My Snippet",
              "file_name": "test.txt",
              "content": "Test content",
              "visibility": "public"
            }
          ]
        }
      ],
      "subgroups": [...]
    }
  ],
  "users": [...],
  "root_user": {...},
  "root_snippets": [...]
}

Test Scenarios Configuration (`configs/test-scenarios.json`)

Defines the integration test scenarios:

{
  "test_scenarios": [
    {
      "name": "my-test-scenario",
      "description": "Test description",
      "command": "ghorg clone all-groups --scm=gitlab --base-url={{.BaseURL}} --token={{.Token}} --output-dir=test-output",
      "run_twice": true,
      "setup_commands": ["git init {{.GhorgDir}}/test-setup"],
      "verify_commands": ["test -d '{{.GhorgDir}}/test-output'"],
      "expected_structure": [
        "test-output/group1/repo1",
        "test-output/group2/repo2"
      ]
    }
  ]
}

Adding New Seed Data

Edit the configuration: Modify configs/seed-data.json to add new groups, repositories, users, or snippets
Test the changes: Run ./seed.sh to verify the new seed data is created correctly

Example: Adding a New Group

{
  "name": "new-group",
  "path": "new-group",
  "description": "Description of the new group",
  "repositories": [
    {
      "name": "new-repo",
      "initialize_with_readme": true
    }
  ]
}

Adding New Test Scenarios

Method 1: Use the Helper Script

./add-test-scenario.sh

This interactive script will guide you through creating a new test scenario.

Method 2: Manual Configuration

Edit configs/test-scenarios.json
Add a new test scenario object to the test_scenarios array
Test with: ./test-runner/gitlab-test-runner -test="your-test-name"

Method 3: Programmatically

# Build the test runner
cd test-runner && go build -o gitlab-test-runner main.go

# List available tests
./gitlab-test-runner -list

# Run a specific test
./gitlab-test-runner -test="specific-test-name" -token="password" -base-url="http://gitlab.example.com"

Template Variables

Both seeder and test runner support template variables:

{{.BaseURL}} - GitLab base URL
{{.Token}} - GitLab API token
{{.GhorgDir}} - Ghorg directory path

Development

Building the Components

# Build seeder
cd seeder && go build -o gitlab-seeder main.go

# Build test runner
cd test-runner && go build -o gitlab-test-runner main.go

Running Tests in Development

# Run specific test scenario
cd test-runner
go run main.go -test="all-groups-preserve-dir-output-dir" -token="password" -base-url="http://gitlab.example.com"

# List all available test scenarios
go run main.go -list -token="password"

Advantages of Refactored System

Maintainability: Configuration-driven approach makes it easy to modify tests and seed data
Modularity: Separate components for seeding and testing
Reusability: Test scenarios can be easily copied and modified
Better Error Handling: Go-based tools provide clearer error messages
Extensibility: Easy to add new test scenarios or seed data configurations
Documentation: Clear separation of concerns and self-documenting configuration

Migration from Old System

The refactored system is designed to be fully backward-compatible. The original scripts (seed.sh, integration-tests.sh, start-ee.sh) remain unchanged and continue to work.

To migrate to the refactored system:

Use start-ee.sh for the refactored system
All existing test scenarios have been converted to the new configuration format
The test results should be identical between old and new systems

Troubleshooting

Build Errors

# Ensure Go modules are downloaded
cd seeder && go mod download
cd test-runner && go mod download

Test Failures

# Check GitLab is accessible
curl -I http://gitlab.example.com

# Verify seeding completed
./seeder/gitlab-seeder -token="password" -base-url="http://gitlab.example.com"

# Run specific failing test
./test-runner/gitlab-test-runner -test="failing-test-name" -token="password"

Configuration Issues

# Validate JSON configuration
python3 -m json.tool configs/seed-data.json
python3 -m json.tool configs/test-scenarios.json

README.md

Refactored GitLab Integration Tests

Overview

Directory Structure

Quick Start

Running All Tests (Refactored Version)

Script Arguments

Quick Reference

start-ee.sh Arguments

Individual Component Arguments

seed.sh Arguments

integration-tests.sh Arguments

run-ee.sh Arguments (Internal)

Go Tool Arguments (Direct Usage)

Running Individual Components

Configuration

Seed Data Configuration (configs/seed-data.json)

Test Scenarios Configuration (configs/test-scenarios.json)

Adding New Seed Data

Example: Adding a New Group

Adding New Test Scenarios

Method 1: Use the Helper Script

Method 2: Manual Configuration

Method 3: Programmatically

Template Variables

Development

Building the Components

Running Tests in Development

Advantages of Refactored System

Migration from Old System

Troubleshooting

Build Errors

Test Failures

Configuration Issues

`start-ee.sh` Arguments

`seed.sh` Arguments

`integration-tests.sh` Arguments

`run-ee.sh` Arguments (Internal)

Seed Data Configuration (`configs/seed-data.json`)

Test Scenarios Configuration (`configs/test-scenarios.json`)