36 Commits

Author SHA1 Message Date
Philip Jenvey
e8d89d7ab6
refactor: move syncstorage-mysql's error -> db-common
so syncstorage-postgres can share it

Issue STOR-330
2025-10-17 16:13:05 -07:00
Philip Jenvey
0156613781
refactor: separate batch calls into a BatchDb trait
Issue STOR-330
2025-10-17 14:03:19 -07:00
Philip Jenvey
21bcc42feb
refactor: String -> &str
Issue STOR-330
2025-10-17 14:03:19 -07:00
Philip Jenvey
d32c669b7a
refactor: move syncstorage's Db to async-trait (#1830)
Some checks failed
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
Closes STOR-144
2025-10-03 14:45:18 -07:00
Barry Chen
730d6b8173
refactor(db): return only a SyncTimestamp from post_bsos db fns
Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
2025-09-29 14:36:09 -05:00
Philip Jenvey
bcf425c635
refactor: kill r2d2 references, share From<PoolError> 2025-09-23 18:32:54 -07:00
Philip Jenvey
192d64c7c3
feat: switch syncstorage to diesel-async
Closes STOR-364
2025-09-23 18:32:51 -07:00
Philip Jenvey
b89b06d24f
feat: kill Db's impl of Clone (#1789)
Some checks failed
Glean probe-scraper / glean-probe-scraper (push) Has been cancelled
by utilizing async closures' improved borrowing semantics

Closes STOR-350
2025-09-04 14:58:11 -07:00
Philip Jenvey
38b4db401f
refactor: apply &mut self to syncstorage-mysql
Closes STOR-327
2025-09-02 13:50:20 -07:00
Philip Jenvey
38cb38fbd9
refactor: switch syncstorage Db methods to &mut self
Closes STOR-327
2025-09-02 13:50:20 -07:00
Philip Jenvey
9381bc68b8
fix: kill unnecessary transactions (savepoints) (#1782)
Issue STOR-327
2025-09-02 13:50:08 -07:00
Philip Jenvey
e71980c242
feat: workaround batch_id conflicts w/ a simple retry
and don't specify patch versions of diesel in Cargo.toml

Issue STOR-77
2025-08-28 17:26:39 -07:00
Philip Jenvey
dfe216466d
fix: Revert "fix: mysql: Call set_timestamp in lock_for_write_sync."
This reverts commit d6c0b9efc68111d6a433e7b618dde789ba5377d6.
2025-08-28 17:26:18 -07:00
Philip Jenvey
44148da7c8
merge master 2025-08-28 17:25:26 -07:00
Tommie Gannert
2a1268ed04
fix: mysql: Replace the user_id%10 in batch_id with a global counter.
Since the batch_uploads primary key already contains user_id, it does
nothing to increase time resolution. The counter may still see
conflicts with other processes, but the probably is much lower
now. (Spanner uses UUIDv4 for batch IDs.)

The user_id hack was added in e9b455fd "to match the Python
implementation". E2E tests are failing because we can now
generate more than one batch within 10 ms.

Signed-off-by: Tommie Gannert <tommie@gannert.se>
2025-08-28 17:07:16 -07:00
Tommie Gannert
d4511cc746
fix: mysql: Call set_timestamp in lock_for_write_sync.
There is no other way for the timestamp to increase in production.

Signed-off-by: Tommie Gannert <tommie@gannert.se>
2025-08-28 17:07:16 -07:00
Tommie Gannert
b9507e36c3
chore: Upgrade Diesel to 2.x.
See https://diesel.rs/guides/migration_guide.html.

- This pulls in mysqlclient-sys >=0.4.0, which allows using libmariadb instead of libmysqlclient.
  That allows using Debian's bundled client library.
  MySQL's APT repository only has packages for AMD64.
- Connections are now passed as mut refs, which requires RefCell/RwLock.
- Embedded migrations had a slight API change.
- put_bso_sync was calling update_collection without its transaction, which now causes a deadlock.
  Updated to write inside its transaction.

Signed-off-by: Tommie Gannert <tommie@gannert.se>
2025-08-28 17:07:15 -07:00
Taddes
f7197fefb8
chore: update syncstorage rust (#1749)
chore: update syncstorage rust
2025-08-13 15:14:34 -04:00
Philip Jenvey
69005091a5
chore: require minimum mysql 8 (#1717)
Some checks are pending
Glean probe-scraper / glean-probe-scraper (push) Waiting to run
BREAKING CHANGE: now depending on >= mysql 8 features

Closes STOR-160
2025-07-17 13:06:13 -07:00
Philip Jenvey
0148e04dd2
chore: bump to latest rust
for cargo nextest/backtrace
2025-05-07 16:49:09 -07:00
Philip Jenvey
0ca435fb1a
refactor: simplify metric_label to return a &str
Issue STOR-64
2025-04-23 16:01:07 -07:00
Taddes
c01021b87d
Feat: add task to release unused db conns (#1640)
Feat: add task to release unused db conns
2024-12-05 17:10:14 -05:00
Eragon
2ae1ef46df
Rename MySQL error to use a more generic name (#1619)
* style: Rename MysqlError to SqlError as they are generic
* refactor: Don't use legacy numerics contant and methods
* style: Cargo fmt
* style: Keep Mysql specific error in Mysql code

---------

Co-authored-by: JR Conlin <jconlin+git@mozilla.com>
2024-12-04 15:47:12 -08:00
Philip Jenvey
1f0e28d7af
feat: track the pool's queued vs actually active tasks (#1605)
and consider idle threads as a "max" value as the underlying tokio threadpool
will exit threads idle for too long

Closes SYNC-4424
2024-09-30 16:13:51 -07:00
JR Conlin
1713962c6a
Chore/update 2406 (#1576)
chore: Updates for Jun-2024

* f normalize diesel to 1.4

We're currenlty locked to diesel 1.4 due to some significant changes
in the `diesel_logger` crate and how it expects `Connections` to be
defined.
2024-06-25 13:29:42 -07:00
JR Conlin
5cdfd03498
docs: Remove commented code, unneeded TODO, unneeded collision tracking (#1563) 2024-05-20 12:52:36 -07:00
JR Conlin
771813087c
feat: Add normalized ReportableError to errors (#1559)
* feat: Add normalized ReportableError to errors

We want to do things like add tags and other features to sync errors the
way that we do in other packages. To do so, we're backporting
ReportableError from Autopush to Syncstorage.

This also addresses some clippy fixes required by 1.78

This continues to use the `Taggable` trait, which we may want to
port to autopush.

Closes SYNC-4262
2024-05-20 08:51:58 -07:00
JR Conlin
ab7b4221fd
bug: Allow threadpool size to be set. (#1560)
* bug: Allow threadpool size to be set.

It appears that the replacement for setting ACTIX_THREADPOOL would be to
call `ServiceBuilder.worker_max_blocking_threads()` This PR introduces
the ability to set this value by using the
`worker_max_blocking_threads`configuration variable.

Closes #SYNC-4271
2024-05-16 16:36:41 -07:00
Tarik Eshaq
1b11684648
feat: Puts pyo3 behind feature flag and derives tokens directly in Rust (#1513)
* Removes pyo3 and derives tokens directly in Rust

* Adds tests for JWT verifying

* Adds tests for token generation

* Adds metrics for oauth verify error cases

* Updates jsonwebtoken to not include default features (including pem loading)

* Adds context and logs errors during oauth verify

* Uses ring for cryptographic rng

* Adds back python impl under feature flag

* Uses one cached http client for reqwest
2024-02-12 11:14:15 -05:00
JR Conlin
cc8d5847dc
Chore/update 2312 (#1510)
* chore: Update for 12-2023

Closes: #1503, #1502

switch to fetching mysql_pubkey.asc

Closes #1509 SYNC-4055
2024-01-12 15:44:44 -08:00
Philip Jenvey
18f4d59490
fix: restore emitting backend specific db errors to sentry (#1500)
SYNC-4023
2023-11-29 18:34:09 -08:00
Philip Jenvey
fb70128824
fix: switch more test flags to cfg(debug_assertions) (#1488)
switch mutation usage to an explicit flag

fix the db tests' uid handling

SYNC-3955
2023-10-10 17:41:42 -07:00
Philip Jenvey
dc98e95ff3
refactor: quiet latest clippy warnings 2023-09-25 12:24:16 -07:00
JR Conlin
1f9323b7b3
feat: convert dependencies to use workspaces
Update package description and some dependencies to use `workspace`
inheritance.

* Also updates circleci to produce dockerhub image.

Closes #1461
Issue #1362

* f touch to retry circleci

* f try to pre-emptively lock protobuf to the older 2.25.2 version

* f force Cargo.lock to use protobuf 2.25.2, dammit

* f move protobuf pin to syncstorage-spanner

* f revert cargo.lock

* f revert #448

* f update base64 / tickle circleci

* f update cadence (and kick circleci)

* f clippy

* f switch to latest google-cloud-rust

* f update to use google-cloud-rust
2023-08-29 13:57:53 -07:00
Philip Jenvey
80d21e7143
merge master 2023-03-07 12:02:16 -08:00
Ethan Donowitz
b5b7e57f93
refactor: add database crates (#1407)
Closes #1277
2023-01-10 16:06:03 -05:00