diff --git a/.gitignore b/.gitignore index b60fdf59..dbc044dc 100644 --- a/.gitignore +++ b/.gitignore @@ -32,3 +32,5 @@ target .#* service-account.json + +config/local.toml diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 55824753..26284347 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -33,6 +33,7 @@ When submitting a PR: - Ensure your changes do not reduce code coverage of the test suite. - Please do not include merge commits in pull requests; include only commits with the new relevant code. +- PR naming conventions - begins with type (fix, feature, doc, chore, etc) and a short description with no period. See the main [README.md](/README.md) for information on prerequisites, installing, running and testing. diff --git a/Makefile b/Makefile new file mode 100644 index 00000000..4ddf03cb --- /dev/null +++ b/Makefile @@ -0,0 +1,2 @@ +run-local: + RUST_LOG=debug RUST_BACKTRACE=1 cargo run -- --config config/local.toml diff --git a/PULL_REQUEST_TEMPLATE.md b/PULL_REQUEST_TEMPLATE.md index e3ad1743..1413835a 100644 --- a/PULL_REQUEST_TEMPLATE.md +++ b/PULL_REQUEST_TEMPLATE.md @@ -1,6 +1,11 @@ -Please remember to consult our [contributing guidelines](https://github.com/mozilla-services/syncstorage-rs/blob/master/CONTRIBUTING.md#sending-pull-requests) before opening your PR. +## Description -- [ ] **Title** begins with _type_ (fix, feature, doc, chore, etc) and a short description with no period -- [ ] **Description** outlines the change -- [ ] **Test cases** included in the change (if appropriate) -- [ ] **Closes** or **Issue** link to associated issue(s) \ No newline at end of file +Describe these changes. + +## Testing + +How should reviewers test? + +## Issue(s) + +Closes [link](link). diff --git a/README.md b/README.md index 08eee669..f98ee191 100644 --- a/README.md +++ b/README.md @@ -1,73 +1,152 @@ + + + + +- [Syncstorage-rs](#syncstorage-rs) + - [System Requirements](#system-requirements) + - [Local Setup](#local-setup) + - [MySQL](#mysql) + - [Spanner](#spanner) + - [Logging](#logging) + - [Tests](#tests) + - [Unit tests](#unit-tests) + - [End-to-End tests](#end-to-end-tests) + - [Troubleshooting](#troubleshooting) + - [Related Documentation](#related-documentation) + + + [![License: MPL 2.0][mpl-svg]][mpl] [![Test Status][travis-badge]][travis] [![Build Status][circleci-badge]][circleci] # Syncstorage-rs -Mozilla Sync Storage node built with [Rust](https://rust-lang.org). +Mozilla Sync Storage built with [Rust](https://rust-lang.org). -API docs: https://mozilla-services.readthedocs.io/en/latest/storage/apis-1.5.html +## System Requirements -Code docs: https://mozilla-services.github.io/syncstorage-rs/syncstorage/ +- [Rust stable](https://rustup.rs) +- MySQL 5.7 (or compatible) + -\* libmysqlclient (`brew install mysql` on macOS, `apt-get install libmysqlclient-dev` on Ubuntu) +- [Go](https://golang.org/doc/install) -Functional tests live in https://github.com/mozilla-services/server-syncstorage/ -and can be run against a local server, e.g.: +Depending on your OS, you may also need to install `libgrpcdev`, and `protobuf-compiler-grpc`. -## Requirements +## Local Setup - * Rust stable - * MySQL 5.7 (or compatible) - * libmysqlclient (`brew install mysql` on macOS, `apt-get install libmysqlclient-dev` on Ubuntu) +1. Follow the instructions below to use either MySQL or Spanner as your DB. +2. Now `cp config/local.example.toml config/local.toml`. Open `config/local.toml` and make sure you have the desired settings configured. For a complete list of available configuration options, check out [docs/config.md](docs/config.md). +3. `make run-local` starts the server in debug mode, using your new `local.toml` file for config options. Or, simply `cargo run` with your own config options provided as env vars. +4. Visit `http://localhost:8000/__heartbeat__` to make sure the server is running. -## Setting Up +### MySQL -1) [Install Rust](https://rustup.rs) and [Install Go](https://golang.org/doc/install). +Durable sync needs only a valid mysql DSN in order to set up connections to a MySQL database. The database can be local and is usually specified with a DSN like: + +`mysql://_user_:_password_@_host_/_database_` + +To setup a fresh MySQL DB and user: (`mysql -u root`): -2) Create a `syncstorage` user/database. In the mysql shell: ``` - CREATE USER "syncstorage"@"localhost" IDENTIFIED BY "password"; - CREATE DATABASE syncstorage;` - ``` -3) Run: + CREATE USER "sample_user"@"localhost" IDENTIFIED BY "sample_password"; + CREATE DATABASE syncstorage_rs; - $ export SYNC_MASTER_SECRET= - $ export SYNC_DATABASE_URL=mysql://:@localhost/syncstorage - $ cargo run + GRANT ALL PRIVILEGES on syncstorage_rs.* to sample_user@localhost; + ``` + +### Spanner + +Spanner requires a key in order to access the database. It's important that you know which keys have access to the spanner database. Contact your administrator +to find out. One you know the key, log into the [Google Cloud Console Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) page. Be sure to +select the correct project. + +- Locate the email identifier of the access key and pick the vertical dot menu at the far right of the row. +- Select "_Create Key_" from the pop-up menu. +- Select "JSON" from the Dialog Box. + +A proper Key file will be downloaded to your local directory. It's important to safeguard that key file. For this example, we're going to name the file +`sync-spanner.json` and store it in a subdirectory called `./keys` + +The proper key file is in JSON format. An example file is provided below, with private information replaced by "`...`" + +```json +{ + "type": "service_account", + "project_id": "...", + "private_key_id": "...", + "private_key": "...", + "client_email": "...", + "client_id": "...", + "auth_uri": "https://accounts.google.com/o/oauth2/auth", + "token_uri": "https://oauth2.googleapis.com/token", + "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", + "client_x509_cert_url": "..." +} +``` + +You can then specify the path to the key file using the environment variable `GOOGLE_APPLICATION_CREDENTIALS` when running the application. + +e.g. + +```bash +RUST_LOG=warn GOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/sync-spanner.json` cargo run -- --config sync.ini +``` + +Note, that unlike MySQL, there is no automatic migrations facility. Currently Spanner schema must be hand edited and modified. ## Logging -If you want to connect to the existing [Sentry project](https://sentry.prod.mozaws.net/operations/syncstorage-dev/) for local development, login to Sentry, and go to the page with [api keys](https://sentry.prod.mozaws.net/settings/operations/syncstorage-dev/keys/). Copy the `DSN` value, and `export SENTRY_DSN=DSN_VALUE_GOES_HERE` to the environment when running this project. +- If you want to connect to the existing [Sentry project](https://sentry.prod.mozaws.net/operations/syncstorage-dev/) for local development, login to Sentry, and go to the page with [api keys](https://sentry.prod.mozaws.net/settings/operations/syncstorage-dev/keys/). Copy the `DSN` value, and `export SENTRY_DSN=DSN_VALUE_GOES_HERE` to the environment when running this project. +- Using [env_logger](https://crates.io/crates/env_logger): set the `RUST_LOG` env var. -## Running the Unit tests +## Tests -1) Run: +### Unit tests - $ export RUST_TEST_THREADS=1 - $ cargo test +1. `cd db-tests`. +2. Pass along your `SYNC_DATABASE_URL` to the test runner. Ie: -## Running the End-to-End tests +``` +SYNC_DATABASE_URL="mysql://sample_user:sample_password@localhost/syncstorage_rs" && / +RUST_TEST_THREADS=1 && / +cargo test +``` -1) If you haven't already followed the instructions [here](https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html) to get all the dependencies for the [server-syncstorage](https://github.com/mozilla-services/server-syncstorage/) repo, you should start there. +### End-to-End tests -2) Install (Python) server-syncstorage: +Functional tests live in [server-syncstorage](https://github.com/mozilla-services/server-syncstorage/) and can be run against a local server, e.g.: - $ git clone https://github.com/mozilla-services/server-syncstorage/ - $ cd server-syncstorage - $ make build +1. If you haven't already followed the instructions [here](https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html) to get all the dependencies for the [server-syncstorage](https://github.com/mozilla-services/server-syncstorage/) repo, you should start there. -3) Run an instance of syncstorage-rs (see step 3 above). +2. Install (Python) server-syncstorage: -4) Run: + $ git clone https://github.com/mozilla-services/server-syncstorage/ + $ cd server-syncstorage + $ make build - $ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000# +3. Run an instance of syncstorage-rs (`cargo run` in this repo). -Individual tests can be specified via the `SYNC_TEST_PREFIX` env var: +4. To run all tests: + + $ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000# + +5. Individual tests can be specified via the `SYNC_TEST_PREFIX` env var: $ SYNC_TEST_PREFIX=test_get_collection \ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000# +## Troubleshooting + +- `rm Cargo.lock; cargo clean;` - Try this if you're having problems compiling. + +## Related Documentation + +- [API docs](https://mozilla-services.readthedocs.io/en/latest/storage/apis-1.5.html) + +- [Code docs](https://mozilla-services.github.io/syncstorage-rs/syncstorage/) [mpl-svg]: https://img.shields.io/badge/License-MPL%202.0-blue.svg [mpl]: https://opensource.org/licenses/MPL-2.0 [travis-badge]: https://travis-ci.org/mozilla-services/syncstorage-rs.svg?branch=master [travis]: https://travis-ci.org/mozilla-services/syncstorage-rs [circleci-badge]: https://circleci.com/gh/mozilla-services/syncstorage-rs.svg?style=shield -[circleci]: https://circleci.com/gh/mozilla-services/syncstorage-rs \ No newline at end of file +[circleci]: https://circleci.com/gh/mozilla-services/syncstorage-rs diff --git a/config/local.example.toml b/config/local.example.toml new file mode 100644 index 00000000..60c43344 --- /dev/null +++ b/config/local.example.toml @@ -0,0 +1,2 @@ +database_url = "mysql://sample_user:sample_password@localhost/syncstorage_rs" +master_secret = "some_secret_here" \ No newline at end of file diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 00000000..9bd449fb --- /dev/null +++ b/docs/config.md @@ -0,0 +1,43 @@ +# Configuration +Rust uses environment variables for a number of configuration options. Some of these include: + +| variable | value | description | +| --- | --- | --- | +| **RUST_LOG** | *debug*, *info*, *warn*, *error* | minimum Rust error logging level | +| **RUST_TEST_THREADS** | 1 | maximum number of concurrent threads for testing. | + +In addition, durable sync configuration options can either be specified as environment variables (prefixed with **SYNC_***) or in a configuration file using the `--config` option. + +For example the following are equivalent: +```bash +$ SYNC_HOST=0.0.0.0 SYNC_MASTER_SECRET="SuperSikkr3t" SYNC_DATABASE_URL=mysql://scott:tiger@localhost/syncstorage cargo run +``` + +```bash +$ cat sync.ini +HOST=0.0.0.0 +MASTER_SECRET=SuperSikkr3t +DATABASE_URL=mysql://scott:tiger@localhost/syncstorage +$ cargo run -- --config sync.ini +``` + +Options can be mixed between environment and configuration. + +## Options +The following configuration options are avaialble. + +| Option | Default value |Description | +| --- | --- | --- | +| debug | false | _unused_ | +| port | 8000 | connection port | +| host | 127.0.0.1 | host to listen for connections | +| database_url | mysql://root@127.0.0.1/syncstorage | database DSN | +| database_pool_max_size | _None_ | Max pool of database connections | +| master_secret| _None_ | Sync master encryption secret | +| limits.max_post_bytes | 2,097,152‬ | Largest record post size | +| limits.max_post_records | 100 | Largest number of records per post | +| limits.max_records_payload_bytes | 2,097,152‬ | Largest ... | +| limits.max_request_bytes | 2,101,248 | Largest ... | +| limits.max_total_bytes | 209,715,200 | Largest ... | +| limits.max_total_records | 100,000 | Largest ... | + diff --git a/docs/setup.md b/docs/setup.md deleted file mode 100644 index 291a3923..00000000 --- a/docs/setup.md +++ /dev/null @@ -1,94 +0,0 @@ -# Durable Sync Setup Guide - -## General setup - 1) [Install Rust](https://rustup.rs) - Durable sync uses the *stable* branch, so no additional elements should be required. - 2) Install `libmysqlclient`, the mysql client development library - * macOS: `brew install mysql` - * ubuntu: `apt install libmysqlclient-dev` - 3) You may need to also install `libgrpcdev`, `protobuf-compiler-grpc`, and `golang` - -## Configuration -Rust uses environment variables for a number of configuration options. Some of these include: - -| variable | value | description | -| --- | --- | --- | -| **RUST_LOG** | *debug*, *info*, *warn*, *error* | minimum Rust error logging level | -| **RUST_TEST_THREADS** | 1 | maximum number of concurrent threads for testing. | - -In addition, durable sync configuration options can either be specified as environment variables (prefixed with **SYNC_***) or in a configuration file using the `--config` option. - -For example the following are equivalent: -```bash -$ SYNC_HOST=0.0.0.0 SYNC_MASTER_SECRET="SuperSikkr3t" SYNC_DATABASE_URL=mysql://scott:tiger@localhost/syncstorage cargo run -``` - -```bash -$ cat sync.ini -HOST=0.0.0.0 -MASTER_SECRET=SuperSikkr3t -DATABASE_URL=mysql://scott:tiger@localhost/syncstorage -$ cargo run -- --config sync.ini -``` - -Options can be mixed between environment and configuration. - -### Options -The following configuration options are avaialble. - -| Option | Default value |Description | -| --- | --- | --- | -| debug | false | _unused_ | -| port | 8000 | connection port | -| host | 127.0.0.1 | host to listen for connections | -| database_url | mysql://root@127.0.0.1/syncstorage | database DSN | -| database_pool_max_size | _None_ | Max pool of database connections | -| master_secret| _None_ | Sync master encryption secret | -| limits.max_post_bytes | 2,097,152‬ | Largest record post size | -| limits.max_post_records | 100 | Largest number of records per post | -| limits.max_records_payload_bytes | 2,097,152‬ | Largest ... | -| limits.max_request_bytes | 2,101,248 | Largest ... | -| limits.max_total_bytes | 209,715,200 | Largest ... | -| limits.max_total_records | 100,000 | Largest ... | - -## Mysql Integration - -Durable sync needs only a valid mysql DSN in order to set up connections to a MySQL database. The database can be local and is usually specified with a DSN like: - -mysql://_user_:_password_@_host_/_database_ - -## Setting up Spanner integration - -Spanner requires a key in order to access the database. It's important that you know which keys have access to the spanner database. Contact your administrator -to find out. One you know the key, log into the [Google Cloud Console Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) page. Be sure to -select the correct project. - -* Locate the email identifier of the access key and pick the vertical dot menu at the far right of the row. -* Select "*Create Key*" from the pop-up menu. -* Select "JSON" from the Dialog Box. - -A proper Key file will be downloaded to your local directroy. It's important to safeguard that key file. For this example, we're going to name the file -`sync-spanner.json` and store it in a subdirectory called `./keys` - -The proper key file is in JSON format. An example file is provided below, with private information replaced by "`...`" -```json -{ - "type": "service_account", - "project_id": "...", - "private_key_id": "...", - "private_key": "...", - "client_email": "...", - "client_id": "...", - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", - "client_x509_cert_url": "..." -} -``` -You can then specify the path to the key file using the environment variable `GOOGLE_APPLICATION_CREDENTIALS` when running the application. - -e.g. -```bash -RUST_LOG=warn GOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/sync-spanner.json` cargo run -- --config sync.ini -``` - -Note, that unlike MySQL, there is no automatic migrations facility. Currently Spanner schema must be hand edited and modified. \ No newline at end of file