41 Commits

Author SHA1 Message Date
Philip Jenvey
f2c7f94440
chore: allow overriding RUST_LOG (#1813)
Some checks failed
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
and bump the integration test timeout allowance
2025-09-22 13:20:44 -07:00
Taddes
6100529da7
test: add max total records e2e test (#1796)
Some checks failed
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
test: add max total records e2e test
2025-09-09 14:59:39 -04:00
Taddes
8995db6268
feat: ruff for python lint and format (#1742)
Some checks are pending
Glean probe-scraper / glean-probe-scraper (push) Waiting to run
feat: ruff for python lint and format
2025-08-12 20:26:55 -04:00
Taddes
8055e7429d
test: resolve test deprecations (#1732)
test: resolve test deprecations
2025-08-12 17:47:55 -04:00
Taddes
77254b4a6e
refactor: python imports (#1730)
Some checks failed
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
refactor: unified python imports for consistency in local, production, and test environments
2025-08-04 15:37:46 -04:00
Taddes
f8715d4e91
feat: use poetry for dependency management (#1706)
Some checks are pending
Glean probe-scraper / glean-probe-scraper (push) Waiting to run
feat: use poetry for dependency management
2025-07-28 21:48:46 -04:00
Nick Shirley
6f15ad546d
test(e2e): run integration and e2e tests with pytest (#1697)
Some checks are pending
Glean probe-scraper / glean-probe-scraper (push) Waiting to run
- Update docker compose steps for mysql and spanner to use pytest
- Add infra and configuration for pytest to run tests
- Remove old "run.py" test setup

Closes STOR-235
2025-05-09 14:50:18 -06:00
Philip Jenvey
d0336c8869
fix: re-enable tokensever e2e tests
w/ the pyfxa 0.8.1 fix

Closes STOR-204
2025-05-02 13:43:41 -07:00
Philip Jenvey
e93bb8821c
chore: update python cryptography (#1690)
Some checks failed
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
2025-04-18 13:33:10 -07:00
dependabot[bot]
be23e39135
chore(deps): bump cryptography in /tools/integration_tests (#1594)
chore(deps): bump cryptography in /tools/integration_tests
2024-09-20 14:58:06 -04:00
Philip Jenvey
6c9b771bc5
chore: bump crytography/pyramid to quiet a number of security alerts (#1574)
Closes SYNC-4301
2024-06-14 17:05:43 -07:00
JR Conlin
dbbdd1dfc3
feat: Remove support for BrowserID (#1531)
* feat: Remove support for BrowserID
* mark test only functions as such. I think we can drop MapAndThenTrait for more recent implementations of rust.

Closes: SYNC-3684
2024-06-14 12:51:02 -07:00
JR Conlin
cc16082241
feat: Allow uid range for purge function (SYNC-4246) (#1547)
* feat: Allow uid range for purge function

In an attempt to parallelize this script after a very long delay,
specify a range so that multiple scripts can try to process
different ranges of the database.

Closes #1548
2024-04-30 08:53:27 -07:00
JR Conlin
b777fa0d96
bug: Add try/except handler for force (#1535)
* bug: Add try/except handler for force

This will wrap the force `delete_service_data` with a try/except handler
so as not to prevent the tokenserver record from being removed if the
delete_service_data fails.

* Try/except will capture any errors from trying to call the delete on a
given node, this could fail if the auth or node aren't correct.
* Override will override the attempted node. This will allow us to force
the delete message, even if the original data was copied over and the
node was incorrect.
2024-04-16 10:07:01 -07:00
JR Conlin
06ecb78e24
bug: Copy modified purge script from old tokenserver repo (#1512)
* bug: Copy modified purge script from old tokenserver  repo
2024-02-27 12:46:47 -08:00
Tarik Eshaq
1b11684648
feat: Puts pyo3 behind feature flag and derives tokens directly in Rust (#1513)
* Removes pyo3 and derives tokens directly in Rust

* Adds tests for JWT verifying

* Adds tests for token generation

* Adds metrics for oauth verify error cases

* Updates jsonwebtoken to not include default features (including pem loading)

* Adds context and logs errors during oauth verify

* Uses ring for cryptographic rng

* Adds back python impl under feature flag

* Uses one cached http client for reqwest
2024-02-12 11:14:15 -05:00
JR Conlin
1b9ebbf40f
Chore/contributor rollup (#1479)
* These needed to be rolled in due to various dependency changes.

 * Fix the URL people should set to use their Custom Server (https://github.com/mozilla-services/syncstorage-rs/pull/1453) -
[Ekleog](https://github.com/Ekleog)
 * Make docker files more podman friendly (https://github.com/mozilla-services/syncstorage-rs/pull/1431) -
[mb](https://github.com/mb)

* This PR also updates a number of dependencies
2023-10-18 14:24:41 -07:00
Ethan Donowitz
3f836b1e98
bug: fix CORS issue (#1447) 2022-12-16 13:12:25 -05:00
Ethan Donowitz
3a18de01bb
test: run the Tokenserver E2E tests without a cached JWK (#1390) 2022-10-25 14:04:17 -04:00
Ethan Donowitz
0ae5fd2059
refactor: add settings crates (#1306)
This is a breaking change. This commit separates syncstorage and tokenserver settings into separate structs that are contained by a parent `Settings` struct. This means that any env vars that hold settings specific to syncstorage (e.g. `SYNC_DATABASE_URL`) have been renamed to `SYNC_SYNCSTORAGE__DATABASE_URL`. Any settings that were moved from the top-level `Settings` struct to the lower level, syncstorage-specific struct will now have a `SYNC_SYNCSTORAGE__` prefix instead of a `SYNC_` prefix.

Closes #1276
2022-10-19 17:15:56 -04:00
Ethan Donowitz
613f71ed99
feat: Add X-Content-Type-Options: nosniff to Tokenserver responses (#1403) 2022-09-23 17:17:54 -04:00
Ethan Donowitz
914e375b2b
bug: fix Tokenserver generation and keys_changed_at handling (#1397) 2022-09-15 10:58:26 -04:00
Ethan Donowitz
9bea32803c
bug: convert DbErrors to TokenserverErrors (#1327)
Closes #1316
2022-06-10 17:30:45 -04:00
Ethan Donowitz
a4c340e194
feat: Pass Tokenserver origin field through token payload (#1264)
Closes #1245
2022-04-04 10:08:31 -04:00
Ethan Donowitz
38d6a27b02
feat: add BrowserID support for Tokenserver (#1216)
Closes #1215
2022-03-31 15:13:04 -04:00
Ethan Donowitz
38de8332a5
bug: add missing Tokenserver headers (#1243)
Closes #1242
2022-03-16 17:26:24 -04:00
Ethan Donowitz
eac6b55889
refactor: Remove Tokenserver support for per-node secrets (#1211)
Closes #1208
2022-02-04 15:44:16 -05:00
Ethan Donowitz
ae6597022c
refactor: remove static service IDs (#1199)
Note: this commit contains a breaking change. Previously, there was a migration that included two services with static IDs by default, but this commit adds a new migration that removes those two services and adjusts the code to account for the change.

Closes #1144
2022-01-07 10:41:28 -05:00
Ethan Donowitz
aa18c1a01d
test: add Tokenserver integration tests to CI (#1180)
Closes #1174
2021-12-08 08:56:16 -05:00
Ethan Donowitz
a3d4f094cd
feat: add missing Tokenserver response fields (#1176)
Closes #1173
2021-12-07 11:44:07 -05:00
Tif Tran
022d235e87
changes to allow cors allowed origin and max age to be configurable (#1177)
* allowed cors origin and max_age to be configurable

* e2e test edit

* moved comment

* feedback changes

* e2e test edit

* remove extra line at end of file

Co-authored-by: JR Conlin <jconlin+git@mozilla.com>
2021-12-06 14:46:11 -08:00
Ethan Donowitz
bed59e2cb7
feat: Tokenserver: add per-node secrets (#1169)
Closes #1104
2021-11-23 11:30:27 -05:00
Ethan Donowitz
0996cb154f
feat: Add client state validation (#1160)
Closes #1091
2021-11-11 10:38:16 -05:00
Ethan Donowitz
db739defbe
feat: Tokenserver: Add node assignment logic (#1158)
Closes #1051
2021-11-05 09:51:30 -04:00
Ethan Donowitz
337ab8f406
feat: Tokenserver: Add validations and user updating for generation, keys_changed_at, and client_state (#1145)
Closes #866
2021-10-28 14:36:17 -04:00
Ethan Donowitz
7209ccf551
test: Add Tokenserver integration tests (#1152)
Closes #1048
2021-10-05 16:59:23 -04:00
Tif Tran
dee69dd33d
chore: enable flake8 in circleci config for tools/integration_tests (#1121)
closes #944

Enable flake8 in circleci config for tools/integration_tests
Format contents in tool/integration_tests to be flake8 compliant
2021-07-26 11:24:34 -07:00
Ethan Donowitz
e6ec1acd87
feat: Integrate Spanner emulator with CI (#1079)
Closes #566
2021-05-19 18:20:25 -04:00
JR Conlin
ba6e5f4b94
bug: write bsos contained within a commit after the batch has been commited. (#980)
* bug: fold commit message bsos into pending batch

The client may sometimes include bsos in the batch
commit message. The problem is that due to the way
that data is written to spanner, mutations do not
retain the ability to see data previously written
in the same transaction. This causes collisions.

To solve this, treat the bsos included in the commit
as another batch update, then commit all the data.
This does run the risk of bumping up against the
mutation limit, but it ensures the best chance of
data consistency.

Writing the commit bsos prior to batch commit will
result in the "newer" records being overwritten by
"older" ones in the batch.

Writing the commit bsos after the batch commit runs
the same mutation index conflict problem.

Closes #882

Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2021-02-09 18:42:27 -08:00
JR Conlin
8c280ccda0
bug: ensure "extra" data added to Hawk header in e2e tests (#1003)
* bug: ensure "extra" data added to Hawk header in e2e tests

Closes: #1002
2021-02-09 17:26:23 -08:00
Donovan Preston
d6e6c2c0a1
Py3 integration tests (#917)
* feat: Port integration tests to py3, and run them in the same docker container as the rust server.

Fix #911
2020-12-01 21:20:59 -05:00