ghorg/scripts/local-gitlab/README.md

GitLab Integration Tests

This directory contains the refactored GitLab integration test system.

Directory Structure

scripts/local-gitlab/
├── configs/
│   ├── seed-data.json          # Defines GitLab resources to create
│   └── test-scenarios.json     # Defines integration test scenarios
├── seeder/
│   ├── main.go                 # Go-based seeder implementation
│   └── go.mod                  # Seeder dependencies
├── test-runner/
│   ├── main.go                 # Go-based test runner implementation
│   └── go.mod                  # Test runner dependencies
├── start-ee.sh                # Refactored main entry point
├── seed.sh                    # New seeding script using Go seeder
├── integration-tests.sh       # New test script using Go test runner
├── add-test-scenario.sh       # Utility to add new test scenarios
└── README-refactored.md       # This file

Quick Start

Running All Tests

# Run the refactored integration tests
./start-ee.sh

# Or with custom parameters
./start-ee.sh true false latest

Script Arguments

Quick Reference

Script	Arguments	Purpose
start-ee.sh	7 optional args	Main entry point - runs entire test suite
seed.sh	3 optional args	Seeds GitLab with test data
integration-tests.sh	3 optional args	Runs integration tests only
run-ee.sh	4 optional args	Starts GitLab container (internal)

start-ee.sh Arguments

The main entry point script accepts up to 7 optional arguments. All arguments have sensible defaults if not provided.

Usage:

./start-ee.sh [STOP_GITLAB_WHEN_FINISHED] [PERSIST_GITLAB_LOCALLY] [GITLAB_IMAGE_TAG] [GITLAB_HOME] [GITLAB_HOST] [GITLAB_URL] [LOCAL_GITLAB_GHORG_DIR]

Argument	Default	Description
STOP_GITLAB_WHEN_FINISHED	'true'	Whether to stop and remove the GitLab container after tests complete. Set to 'false' to keep GitLab running for debugging.
PERSIST_GITLAB_LOCALLY	'false'	Whether to persist GitLab data locally across container restarts. Set to 'true' to keep data between runs.
GITLAB_IMAGE_TAG	'latest'	GitLab Docker image tag to use. Can be specific version like '16.4.0-ce.0' or 'latest'.
GITLAB_HOME	"$HOME/ghorg/local-gitlab-ee-data-${GITLAB_IMAGE_TAG}"	Directory where GitLab stores persistent data on the host machine.
GITLAB_HOST	'gitlab.example.com'	Hostname for the GitLab instance. Used for container networking and /etc/hosts entries.
GITLAB_URL	'http://gitlab.example.com'	Full URL to access the GitLab instance. Used by ghorg and the test tools.
LOCAL_GITLAB_GHORG_DIR	"${HOME}/ghorg"	Local directory where ghorg will clone repositories and store its working files.

Examples:

# Default behavior - run tests and clean up
./start-ee.sh

# Keep GitLab running after tests for debugging
./start-ee.sh false

# Use specific GitLab version and keep it running
./start-ee.sh false false 16.4.0-ce.0

# Full custom configuration
./start-ee.sh true true latest /tmp/gitlab-data gitlab.local http://gitlab.local /tmp/ghorg

Common Scenarios:

# Development - keep GitLab running for multiple test iterations
./start-ee.sh false false latest

# CI/CD - use clean environment and cleanup afterwards (default)
./start-ee.sh true false latest

# Testing specific GitLab version
./start-ee.sh true false 16.3.0-ce.0

# Custom data persistence for repeated testing
./start-ee.sh false true latest /data/gitlab-persistent

Individual Component Arguments

seed.sh Arguments

Seeds the GitLab instance with test data using the Go-based seeder.

Usage:

./seed.sh [API_TOKEN] [GITLAB_URL] [LOCAL_GITLAB_GHORG_DIR]

Argument	Default	Description
API_TOKEN	"password"	GitLab API token for authentication (default root password)
GITLAB_URL	"http://gitlab.example.com"	Full URL to the GitLab instance
LOCAL_GITLAB_GHORG_DIR	"${HOME}/ghorg"	Directory where ghorg stores its configuration and temp files

Example:

# Use defaults
./seed.sh

# Custom parameters
./seed.sh "my-token" "http://gitlab.local:8080" "/tmp/ghorg"

integration-tests.sh Arguments

Runs the integration tests using the Go-based test runner.

Usage:

./integration-tests.sh [LOCAL_GITLAB_GHORG_DIR] [API_TOKEN] [GITLAB_URL]

Argument	Default	Description
LOCAL_GITLAB_GHORG_DIR	"${HOME}/ghorg"	Directory where ghorg will clone repositories for testing
API_TOKEN	"password"	GitLab API token for authentication
GITLAB_URL	"http://gitlab.example.com"	Full URL to the GitLab instance

Example:

# Use defaults
./integration-tests.sh

# Custom parameters
./integration-tests.sh "/tmp/ghorg" "my-token" "http://gitlab.local:8080"

run-ee.sh Arguments (Internal)

Starts the GitLab Docker container. Called internally by start-ee.sh.

Usage:

./run-ee.sh [GITLAB_IMAGE_TAG] [GITLAB_HOME] [GITLAB_HOST] [PERSIST_GITLAB_LOCALLY]

Argument	Default	Description
GITLAB_IMAGE_TAG	"latest"	GitLab Docker image tag
GITLAB_HOME	Dynamic	Host directory for GitLab data persistence
GITLAB_HOST	"gitlab.example.com"	Container hostname
PERSIST_GITLAB_LOCALLY	"false"	Whether to persist data between container restarts

Go Tool Arguments (Direct Usage)

For advanced usage, you can run the Go tools directly:

Seeder (seeder/gitlab-seeder):

./gitlab-seeder [flags]
  -config string
        Path to seed data configuration file (default "configs/seed-data.json")
  -token string
        GitLab API token (required)
  -base-url string
        GitLab base URL (required)

Test Runner (test-runner/gitlab-test-runner):

./gitlab-test-runner [flags]
  -config string
        Path to test scenarios configuration file (default "configs/test-scenarios.json")
  -token string
        GitLab API token (required)
  -base-url string
        GitLab base URL (required)
  -ghorg-dir string
        Ghorg directory path (default "${HOME}/ghorg")
  -test string
        Run specific test by name (optional)
  -list
        List all available tests and exit

Examples:

# List all available test scenarios
./test-runner/gitlab-test-runner -list -token="password"

# Run specific test
./test-runner/gitlab-test-runner -test="all-groups-preserve-dir-output-dir" -token="password" -base-url="http://gitlab.example.com"

# Seed with custom config
./seeder/gitlab-seeder -config="my-seed-data.json" -token="password" -base-url="http://gitlab.example.com"

Running Individual Components

# Seed GitLab instance only
./seed.sh "password" "http://gitlab.example.com" "${HOME}/ghorg"

# Run integration tests only (assumes seeded instance)
./integration-tests.sh "${HOME}/ghorg" "password" "http://gitlab.example.com"

Configuration

Seed Data Configuration (configs/seed-data.json)

Defines the GitLab resources to create during seeding:

{
  "groups": [
    {
      "name": "my-group",
      "path": "my-group",
      "description": "My test group",
      "repositories": [
        {
          "name": "my-repo",
          "initialize_with_readme": true,
          "snippets": [
            {
              "title": "My Snippet",
              "file_name": "test.txt",
              "content": "Test content",
              "visibility": "public"
            }
          ]
        }
      ],
      "subgroups": [...]
    }
  ],
  "users": [...],
  "root_user": {...},
  "root_snippets": [...]
}

Test Scenarios Configuration (configs/test-scenarios.json)

Defines the integration test scenarios:

{
  "test_scenarios": [
    {
      "name": "my-test-scenario",
      "description": "Test description",
      "command": "ghorg clone all-groups --scm=gitlab --base-url={{.BaseURL}} --token={{.Token}} --output-dir=test-output",
      "run_twice": true,
      "setup_commands": ["git init {{.GhorgDir}}/test-setup"],
      "verify_commands": ["test -d '{{.GhorgDir}}/test-output'"],
      "expected_structure": [
        "test-output/group1/repo1",
        "test-output/group2/repo2"
      ]
    }
  ]
}

Adding New Seed Data

1. Edit the configuration: Modify configs/seed-data.json to add new groups, repositories, users, or snippets
2. Test the changes: Run ./seed.sh to verify the new seed data is created correctly

Example: Adding a New Group

{
  "name": "new-group",
  "path": "new-group",
  "description": "Description of the new group",
  "repositories": [
    {
      "name": "new-repo",
      "initialize_with_readme": true
    }
  ]
}

Adding New Test Scenarios

Method 1: Use the Helper Script

./add-test-scenario.sh

This interactive script will guide you through creating a new test scenario.

Method 2: Manual Configuration

1. Edit configs/test-scenarios.json
2. Add a new test scenario object to the test_scenarios array
3. Test with: ./test-runner/gitlab-test-runner -test="your-test-name"

Method 3: Programmatically

# Build the test runner
cd test-runner && go build -o gitlab-test-runner main.go

# List available tests
./gitlab-test-runner -list

# Run a specific test
./gitlab-test-runner -test="specific-test-name" -token="password" -base-url="http://gitlab.example.com"

Template Variables

Both seeder and test runner support template variables:

• {{.BaseURL}} - GitLab base URL
• {{.Token}} - GitLab API token
• {{.GhorgDir}} - Ghorg directory path

Development

Building the Components

# Build seeder
cd seeder && go build -o gitlab-seeder main.go

# Build test runner
cd test-runner && go build -o gitlab-test-runner main.go

Running Tests in Development

# Run specific test scenario
cd test-runner
go run main.go -test="all-groups-preserve-dir-output-dir" -token="password" -base-url="http://gitlab.example.com"

# List all available test scenarios
go run main.go -list -token="password"

Troubleshooting

Build Errors

# Ensure Go modules are downloaded
cd seeder && go mod download
cd test-runner && go mod download

Test Failures

# Check GitLab is accessible
curl -I http://gitlab.example.com

# Verify seeding completed
./seeder/gitlab-seeder -token="password" -base-url="http://gitlab.example.com"

# Run specific failing test
./test-runner/gitlab-test-runner -test="failing-test-name" -token="password"

Configuration Issues

# Validate JSON configuration
python3 -m json.tool configs/seed-data.json
python3 -m json.tool configs/test-scenarios.json