This commit enables Postgres builds and tests in CircleCI. Only
Tokenserver supports Postgres at the moment, and the Python based
integration tests have an implicit dependency on MySQL, so the CI jobs only
use Postgres where possible.
The jobs have been split up and named more explicitly. The hope is to
simply delete the mysql jobs in the future.
- Update docker compose steps for mysql and spanner to use pytest
- Add infra and configuration for pytest to run tests
- Remove old "run.py" test setup
Closes STOR-235
* chore: upload test results to gcs
* chore: use specific ete variables for gcp upload
* chore: override google_project_number to blank
* chore: use actual env var
* style: fixes flake8 E302 error
* docs: add documentation for new circleci vars
Closes SYNC-4584
* add pytest and script to make for running tests with pytest
* add conftest for pytest path handling
* move store test results to capture integration results
* remove unneeded run_tests and fix warnings about assert_()
Closes SYNC-4612
This reverts commit dc0d571c055741297a77dd47c70b7ef55b552530.
(minus the added RUSTSEC ignore)
utilize the match statement now that we're back on debian
bookworm/python 3.10
Issue SYNC-4259
* feat: Remove support for BrowserID
* mark test only functions as such. I think we can drop MapAndThenTrait for more recent implementations of rust.
Closes: SYNC-3684
* feat: Add metrics, gcp logging to tokenserver scripts
This adds GCP logging JSON output formatting for logs as well as DataDog
style metric reporting.
New Environs:
* `METRIC_API_KEY` - Optional Datadog API key
* `METRIC_APP_KEY` - Optional Datadog APP key
* `METRIC_HOST` - Collector Host address
* `METRIC_PORT` - Collector Host port
This also adds the `--human` arg if you want to use the older, human
readable logs instead.
---------
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
* bug: validate val names
* not sure why we look for two args when the secret should always be last
* fix names of the range vals to match opts
* normalize `range` to `uid_range` everywhere to prevent possible
problems.
* feat: Allow uid range for purge function
In an attempt to parallelize this script after a very long delay,
specify a range so that multiple scripts can try to process
different ranges of the database.
Closes#1548
* feat: special case purging of users previously migrated to Spanner
don't issue deletes to their current data on syncstorage if their old
"migration records" point to it
and update per RUSTSEC-2024-0336
Closes SYNC-4225
* bug: Add try/except handler for force
This will wrap the force `delete_service_data` with a try/except handler
so as not to prevent the tokenserver record from being removed if the
delete_service_data fails.
* Try/except will capture any errors from trying to call the delete on a
given node, this could fail if the auth or node aren't correct.
* Override will override the attempted node. This will allow us to force
the delete message, even if the original data was copied over and the
node was incorrect.
* Removes pyo3 and derives tokens directly in Rust
* Adds tests for JWT verifying
* Adds tests for token generation
* Adds metrics for oauth verify error cases
* Updates jsonwebtoken to not include default features (including pem loading)
* Adds context and logs errors during oauth verify
* Uses ring for cryptographic rng
* Adds back python impl under feature flag
* Uses one cached http client for reqwest
This is a breaking change. This commit separates syncstorage and tokenserver settings into separate structs that are contained by a parent `Settings` struct. This means that any env vars that hold settings specific to syncstorage (e.g. `SYNC_DATABASE_URL`) have been renamed to `SYNC_SYNCSTORAGE__DATABASE_URL`. Any settings that were moved from the top-level `Settings` struct to the lower level, syncstorage-specific struct will now have a `SYNC_SYNCSTORAGE__` prefix instead of a `SYNC_` prefix.
Closes#1276
