mirrors/syncstorage-rs
1
0
Fork 0
mirror of https://github.com/mozilla-services/syncstorage-rs.git synced 2026-05-12 08:06:47 +02:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity

Merge pull request #349 from mozilla-services/docs/combine-readmes

Browse Source 
docs: combine setup instructions into one master doc
This commit is contained in:
Rachel Tublitz 2019-11-22 13:11:40 -05:00 committed by GitHub
commit 15f3d4cdf6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 174 additions and 134 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -32,3 +32,5 @@ target
.#*
service-account.json
config/local.toml

1
CONTRIBUTING.md
View File

@ -33,6 +33,7 @@ When submitting a PR:
- Ensure your changes do not reduce code coverage of the test suite.
- Please do not include merge commits in pull requests; include only commits
with the new relevant code.
- PR naming conventions - begins with type (fix, feature, doc, chore, etc) and a short description with no period.
See the main [README.md](/README.md) for information on prerequisites,
installing, running and testing.

2
Makefile Normal file
View File

@ -0,0 +1,2 @@
run-local:
RUST_LOG=debug RUST_BACKTRACE=1 cargo run -- --config config/local.toml

15
PULL_REQUEST_TEMPLATE.md
View File

@ -1,6 +1,11 @@
Please remember to consult our [contributing guidelines](https://github.com/mozilla-services/syncstorage-rs/blob/master/CONTRIBUTING.md#sending-pull-requests) before opening your PR.
## Description
- [ ] **Title** begins with _type_ (fix, feature, doc, chore, etc) and a short description with no period
- [ ] **Description** outlines the change
- [ ] **Test cases** included in the change (if appropriate)
- [ ] **Closes** or **Issue** link to associated issue(s)
Describe these changes.
## Testing
How should reviewers test?
## Issue(s)
Closes [link](link).

149
README.md
View File

@ -1,73 +1,152 @@
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
- [Syncstorage-rs](#syncstorage-rs)
- [System Requirements](#system-requirements)
- [Local Setup](#local-setup)
- [MySQL](#mysql)
- [Spanner](#spanner)
- [Logging](#logging)
- [Tests](#tests)
- [Unit tests](#unit-tests)
- [End-to-End tests](#end-to-end-tests)
- [Troubleshooting](#troubleshooting)
- [Related Documentation](#related-documentation)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
[![License: MPL 2.0][mpl-svg]][mpl] [![Test Status][travis-badge]][travis] [![Build Status][circleci-badge]][circleci]
# Syncstorage-rs
Mozilla Sync Storage node built with [Rust](https://rust-lang.org).
Mozilla Sync Storage built with [Rust](https://rust-lang.org).
API docs: https://mozilla-services.readthedocs.io/en/latest/storage/apis-1.5.html
## System Requirements
Code docs: https://mozilla-services.github.io/syncstorage-rs/syncstorage/
- [Rust stable](https://rustup.rs)
- MySQL 5.7 (or compatible)
-\* libmysqlclient (`brew install mysql` on macOS, `apt-get install libmysqlclient-dev` on Ubuntu)
- [Go](https://golang.org/doc/install)
Functional tests live in https://github.com/mozilla-services/server-syncstorage/
and can be run against a local server, e.g.:
Depending on your OS, you may also need to install `libgrpcdev`, and `protobuf-compiler-grpc`.
## Requirements
## Local Setup
* Rust stable
* MySQL 5.7 (or compatible)
* libmysqlclient (`brew install mysql` on macOS, `apt-get install libmysqlclient-dev` on Ubuntu)
1. Follow the instructions below to use either MySQL or Spanner as your DB.
2. Now `cp config/local.example.toml config/local.toml`. Open `config/local.toml` and make sure you have the desired settings configured. For a complete list of available configuration options, check out [docs/config.md](docs/config.md).
3. `make run-local` starts the server in debug mode, using your new `local.toml` file for config options. Or, simply `cargo run` with your own config options provided as env vars.
4. Visit `http://localhost:8000/__heartbeat__` to make sure the server is running.
## Setting Up
### MySQL
1) [Install Rust](https://rustup.rs) and [Install Go](https://golang.org/doc/install).
Durable sync needs only a valid mysql DSN in order to set up connections to a MySQL database. The database can be local and is usually specified with a DSN like:
`mysql://_user_:_password_@_host_/_database_`
To setup a fresh MySQL DB and user: (`mysql -u root`):
2) Create a `syncstorage` user/database. In the mysql shell:
```
CREATE USER "syncstorage"@"localhost" IDENTIFIED BY "password";
CREATE DATABASE syncstorage;`
```
3) Run:
CREATE USER "sample_user"@"localhost" IDENTIFIED BY "sample_password";
CREATE DATABASE syncstorage_rs;
$ export SYNC_MASTER_SECRET=<SOMESECRET>
$ export SYNC_DATABASE_URL=mysql://<YOUR_MYSQL_USERNAME>:<YOUR_MYSQL_PASSWORD>@localhost/syncstorage
$ cargo run
GRANT ALL PRIVILEGES on syncstorage_rs.* to sample_user@localhost;
```
### Spanner
Spanner requires a key in order to access the database. It's important that you know which keys have access to the spanner database. Contact your administrator
to find out. One you know the key, log into the [Google Cloud Console Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) page. Be sure to
select the correct project.
- Locate the email identifier of the access key and pick the vertical dot menu at the far right of the row.
- Select "_Create Key_" from the pop-up menu.
- Select "JSON" from the Dialog Box.
A proper Key file will be downloaded to your local directory. It's important to safeguard that key file. For this example, we're going to name the file
`sync-spanner.json` and store it in a subdirectory called `./keys`
The proper key file is in JSON format. An example file is provided below, with private information replaced by "`...`"
```json
{
"type": "service_account",
"project_id": "...",
"private_key_id": "...",
"private_key": "...",
"client_email": "...",
"client_id": "...",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "..."
}
```
You can then specify the path to the key file using the environment variable `GOOGLE_APPLICATION_CREDENTIALS` when running the application.
e.g.
```bash
RUST_LOG=warn GOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/sync-spanner.json` cargo run -- --config sync.ini
```
Note, that unlike MySQL, there is no automatic migrations facility. Currently Spanner schema must be hand edited and modified.
## Logging
If you want to connect to the existing [Sentry project](https://sentry.prod.mozaws.net/operations/syncstorage-dev/) for local development, login to Sentry, and go to the page with [api keys](https://sentry.prod.mozaws.net/settings/operations/syncstorage-dev/keys/). Copy the `DSN` value, and `export SENTRY_DSN=DSN_VALUE_GOES_HERE` to the environment when running this project.
- If you want to connect to the existing [Sentry project](https://sentry.prod.mozaws.net/operations/syncstorage-dev/) for local development, login to Sentry, and go to the page with [api keys](https://sentry.prod.mozaws.net/settings/operations/syncstorage-dev/keys/). Copy the `DSN` value, and `export SENTRY_DSN=DSN_VALUE_GOES_HERE` to the environment when running this project.
- Using [env_logger](https://crates.io/crates/env_logger): set the `RUST_LOG` env var.
## Running the Unit tests
## Tests
1) Run:
### Unit tests
$ export RUST_TEST_THREADS=1
$ cargo test
1. `cd db-tests`.
2. Pass along your `SYNC_DATABASE_URL` to the test runner. Ie:
## Running the End-to-End tests
```
SYNC_DATABASE_URL="mysql://sample_user:sample_password@localhost/syncstorage_rs" && /
RUST_TEST_THREADS=1 && /
cargo test
```
1) If you haven't already followed the instructions [here](https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html) to get all the dependencies for the [server-syncstorage](https://github.com/mozilla-services/server-syncstorage/) repo, you should start there.
### End-to-End tests
2) Install (Python) server-syncstorage:
Functional tests live in [server-syncstorage](https://github.com/mozilla-services/server-syncstorage/) and can be run against a local server, e.g.:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
1. If you haven't already followed the instructions [here](https://mozilla-services.readthedocs.io/en/latest/howtos/run-sync-1.5.html) to get all the dependencies for the [server-syncstorage](https://github.com/mozilla-services/server-syncstorage/) repo, you should start there.
3) Run an instance of syncstorage-rs (see step 3 above).
2. Install (Python) server-syncstorage:
4) Run:
$ git clone https://github.com/mozilla-services/server-syncstorage/
$ cd server-syncstorage
$ make build
$ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000#<SOMESECRET>
3. Run an instance of syncstorage-rs (`cargo run` in this repo).
Individual tests can be specified via the `SYNC_TEST_PREFIX` env var:
4. To run all tests:
$ ./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000#<SOMESECRET>
5. Individual tests can be specified via the `SYNC_TEST_PREFIX` env var:
$ SYNC_TEST_PREFIX=test_get_collection \
./local/bin/python syncstorage/tests/functional/test_storage.py http://localhost:8000#<SOMESECRET>
## Troubleshooting
- `rm Cargo.lock; cargo clean;` - Try this if you're having problems compiling.
## Related Documentation
- [API docs](https://mozilla-services.readthedocs.io/en/latest/storage/apis-1.5.html)
- [Code docs](https://mozilla-services.github.io/syncstorage-rs/syncstorage/)
[mpl-svg]: https://img.shields.io/badge/License-MPL%202.0-blue.svg
[mpl]: https://opensource.org/licenses/MPL-2.0
[travis-badge]: https://travis-ci.org/mozilla-services/syncstorage-rs.svg?branch=master
[travis]: https://travis-ci.org/mozilla-services/syncstorage-rs
[circleci-badge]: https://circleci.com/gh/mozilla-services/syncstorage-rs.svg?style=shield
[circleci]: https://circleci.com/gh/mozilla-services/syncstorage-rs
[circleci]: https://circleci.com/gh/mozilla-services/syncstorage-rs

2
config/local.example.toml Normal file
View File

@ -0,0 +1,2 @@
database_url = "mysql://sample_user:sample_password@localhost/syncstorage_rs"
master_secret = "some_secret_here"

43
docs/config.md Normal file
View File

@ -0,0 +1,43 @@
# Configuration
Rust uses environment variables for a number of configuration options. Some of these include:
| variable | value | description |
| --- | --- | --- |
| **RUST_LOG** | *debug*, *info*, *warn*, *error* | minimum Rust error logging level |
| **RUST_TEST_THREADS** | 1 | maximum number of concurrent threads for testing. |
In addition, durable sync configuration options can either be specified as environment variables (prefixed with **SYNC_***) or in a configuration file using the `--config` option.
For example the following are equivalent:
```bash
$ SYNC_HOST=0.0.0.0 SYNC_MASTER_SECRET="SuperSikkr3t" SYNC_DATABASE_URL=mysql://scott:tiger@localhost/syncstorage cargo run
```
```bash
$ cat sync.ini
HOST=0.0.0.0
MASTER_SECRET=SuperSikkr3t
DATABASE_URL=mysql://scott:tiger@localhost/syncstorage
$ cargo run -- --config sync.ini
```
Options can be mixed between environment and configuration.
## Options
The following configuration options are avaialble.
| Option | Default value |Description |
| --- | --- | --- |
| debug | false | _unused_ |
| port | 8000 | connection port |
| host | 127.0.0.1 | host to listen for connections |
| database_url | mysql://root@127.0.0.1/syncstorage | database DSN |
| database_pool_max_size | _None_ | Max pool of database connections |
| master_secret| _None_ | Sync master encryption secret |
| limits.max_post_bytes | 2,097,152‬ | Largest record post size |
| limits.max_post_records | 100 | Largest number of records per post |
| limits.max_records_payload_bytes | 2,097,152‬ | Largest ... |
| limits.max_request_bytes | 2,101,248 | Largest ... |
| limits.max_total_bytes | 209,715,200 | Largest ... |
| limits.max_total_records | 100,000 | Largest ... |

94
docs/setup.md
View File

@ -1,94 +0,0 @@
# Durable Sync Setup Guide
## General setup
1) [Install Rust](https://rustup.rs) - Durable sync uses the *stable* branch, so no additional elements should be required.
2) Install `libmysqlclient`, the mysql client development library
* macOS: `brew install mysql`
* ubuntu: `apt install libmysqlclient-dev`
3) You may need to also install `libgrpcdev`, `protobuf-compiler-grpc`, and `golang`
## Configuration
Rust uses environment variables for a number of configuration options. Some of these include:
| variable | value | description |
| --- | --- | --- |
| **RUST_LOG** | *debug*, *info*, *warn*, *error* | minimum Rust error logging level |
| **RUST_TEST_THREADS** | 1 | maximum number of concurrent threads for testing. |
In addition, durable sync configuration options can either be specified as environment variables (prefixed with **SYNC_***) or in a configuration file using the `--config` option.
For example the following are equivalent:
```bash
$ SYNC_HOST=0.0.0.0 SYNC_MASTER_SECRET="SuperSikkr3t" SYNC_DATABASE_URL=mysql://scott:tiger@localhost/syncstorage cargo run
```
```bash
$ cat sync.ini
HOST=0.0.0.0
MASTER_SECRET=SuperSikkr3t
DATABASE_URL=mysql://scott:tiger@localhost/syncstorage
$ cargo run -- --config sync.ini
```
Options can be mixed between environment and configuration.
### Options
The following configuration options are avaialble.
| Option | Default value |Description |
| --- | --- | --- |
| debug | false | _unused_ |
| port | 8000 | connection port |
| host | 127.0.0.1 | host to listen for connections |
| database_url | mysql://root@127.0.0.1/syncstorage | database DSN |
| database_pool_max_size | _None_ | Max pool of database connections |
| master_secret| _None_ | Sync master encryption secret |
| limits.max_post_bytes | 2,097,152‬ | Largest record post size |
| limits.max_post_records | 100 | Largest number of records per post |
| limits.max_records_payload_bytes | 2,097,152‬ | Largest ... |
| limits.max_request_bytes | 2,101,248 | Largest ... |
| limits.max_total_bytes | 209,715,200 | Largest ... |
| limits.max_total_records | 100,000 | Largest ... |
## Mysql Integration
Durable sync needs only a valid mysql DSN in order to set up connections to a MySQL database. The database can be local and is usually specified with a DSN like:
mysql://_user_:_password_@_host_/_database_
## Setting up Spanner integration
Spanner requires a key in order to access the database. It's important that you know which keys have access to the spanner database. Contact your administrator
to find out. One you know the key, log into the [Google Cloud Console Service Accounts](https://console.cloud.google.com/iam-admin/serviceaccounts) page. Be sure to
select the correct project.
* Locate the email identifier of the access key and pick the vertical dot menu at the far right of the row.
* Select "*Create Key*" from the pop-up menu.
* Select "JSON" from the Dialog Box.
A proper Key file will be downloaded to your local directroy. It's important to safeguard that key file. For this example, we're going to name the file
`sync-spanner.json` and store it in a subdirectory called `./keys`
The proper key file is in JSON format. An example file is provided below, with private information replaced by "`...`"
```json
{
"type": "service_account",
"project_id": "...",
"private_key_id": "...",
"private_key": "...",
"client_email": "...",
"client_id": "...",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "..."
}
```
You can then specify the path to the key file using the environment variable `GOOGLE_APPLICATION_CREDENTIALS` when running the application.
e.g.
```bash
RUST_LOG=warn GOOGLE_APPLICATION_CREDENTIALS=`pwd`/keys/sync-spanner.json` cargo run -- --config sync.ini
```
Note, that unlike MySQL, there is no automatic migrations facility. Currently Spanner schema must be hand edited and modified.