Compare commits

..
9 Commits
121 changed files with 6262 additions and 508 deletions

No files matched your search

+17
View File
@@ -0,0 +1,17 @@
version: 2
updates:
- package-ecosystem: maven
directory: /control-plane-java
schedule:
interval: weekly
open-pull-requests-limit: 5
- package-ecosystem: npm
directory: /prototype
schedule:
interval: weekly
open-pull-requests-limit: 5
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 5
+66
View File
@@ -0,0 +1,66 @@
name: Repository verification
on:
pull_request:
push:
branches: [master, main]
permissions:
contents: read
concurrency:
group: verify-${{ github.ref }}
cancel-in-progress: true
jobs:
backend:
runs-on: ubuntu-latest
timeout-minutes: 20
services:
postgres:
image: postgres:15.18
env:
POSTGRES_USER: moyi_role_test
POSTGRES_DB: moyi_ci_test
POSTGRES_HOST_AUTH_METHOD: trust
ports: ['5432:5432']
options: >-
--health-cmd "pg_isready -U moyi_role_test -d moyi_ci_test"
--health-interval 5s --health-timeout 5s --health-retries 10
redis:
image: redis:7.4-alpine
ports: ['6380:6379']
options: >-
--health-cmd "redis-cli ping"
--health-interval 5s --health-timeout 5s --health-retries 10
env:
MOYI_AUTH_TEST_JDBC_URL: jdbc:postgresql://127.0.0.1:5432/moyi_ci_test
MOYI_GATEWAY_TEST_REDIS_PORT: '6380'
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: actions/setup-java@cf277c60eb25467037889841efdb72551f06f6c3 # v4
with:
distribution: temurin
java-version: '21'
cache: maven
- run: mvn -B -ntp -f control-plane-java/pom.xml clean verify
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
if: failure()
with:
name: backend-test-results
path: control-plane-java/target/surefire-reports/
frontend:
runs-on: ubuntu-latest
timeout-minutes: 10
defaults:
run:
working-directory: prototype
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- uses: pnpm/action-setup@b906affcce14559ad1aafd4ab0e942779e9f58b1 # v4
with:
version: '9'
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: '22'
cache: pnpm
cache-dependency-path: prototype/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- run: pnpm run build
- run: pnpm test
+8
View File
@@ -6,6 +6,14 @@
>
> One Gateway for Every Model You Use.
## 当前工程状态
2026-10-22:当前产品为 Java/Vue 自研控制面与多模型网关,已补请求模块化、账务恢复、异步幂等和审计核账,并继续加固认证与 HTTP 边界。服务端参考图下载已补 DNS 地址绑定;注册/重置已补事务原子性与加密异步验证码投递,重置请求不再查询账号存在性或等待外部发送结果。生产网络策略仍需验收。下文包含产品演进目标,不等于功能均已实现。**尚未完成完整 Codex WebSocket/续接、长请求续租、公平排队、账号全生命周期和生产安全验收。**
逐项完成状态与剩余验收见 [安全与 CPR 任务追踪](docs/24-security-and-cpr-task-tracker.md),认证边界见 [认证及有界 HTTP 增量](docs/25-auth-and-bounded-transport.md),出站加固与验收阻塞见 [媒体 DNS 绑定与验收状态](docs/26-media-dns-binding-and-acceptance.md),注册/重置原子性见 [验证码与账号事务](docs/27-verified-account-transactions.md)。本地验证通过不代表已部署或全仓库安全审计完成。
最新 [验证码投递隔离](docs/28-verification-delivery-isolation.md) 需要先迁移 V50,并配置 `CREDENTIAL_ENCRYPTION_KEY` 及邮件/短信服务。2026-10-22 用户要求的每半小时本地提交检查已取消;产品内部的任务与投递 worker 不属于该聊天自动化。
## 一、产品简介
模亦灵枢是由模灵科技与亦城财富等合作方共同推动的多模型智能服务平台。
+24 -4
View File
@@ -1,14 +1,34 @@
# MOYI Java Control Plane
这是 MOYI Control Plane 的 Java 版本,目标运行时为 Java 21,框架为 Spring Boot。当前已提供健康检查、Flyway 数据库迁移、邀请码认证、用户会话、供应商/模型目录、项目、API Key、模型权限和最小 OpenAI 兼容接口;真实供应商适配、异步任务队列和计费仍按后续节点迁移。
这是 MOYI Control Plane 的 Java 版本,目标运行时为 Java 21,框架为 Spring Boot。当前包括健康检查、Flyway 数据库迁移、邀请码认证、用户会话、供应商/模型目录、项目、API Key、模型权限、同步与原生协议网关、异步任务、计费和用量审计。
目标依赖边界:
- PostgreSQL:用户、工作区、供应商、模型目录、Key、计费和审计等持久化数据。
- Redis:会话、限流、并发计数、幂等键和异步任务队列。
- PostgreSQL:用户、会话、供应商、模型目录、Key、任务队列、幂等、上游账号健康、计费和审计等持久化数据。
- Redis:请求频率、并发准入租约;当前会话、调用幂等和异步任务队列以 PostgreSQL 持久化为准。
- OSS:3D、图片、视频等任务结果和用户上传文件;只保存对象元数据,应用不保存大文件本体。
- Provider Adapter:服务端调用 DeepSeek、Kimi、GLM、Seedance、Tripo/VAST 和后续中转渠道。
## API 请求模块边界
`gateway.request` 是统一的模型 API 请求应用边界。HTTP/Playground 控制器负责认证、协议解析和规范化;请求模块接收不含凭据、提示词和原始请求体的 `ApiRequestContext`,先执行风险策略,再通过准入租约包住钱包预留、Provider 调用和最终结算。Chat、原生协议和异步任务提交共用该风险策略。
协议字段映射留在各自 Meter/Adapter;`ApiRequestRiskRule` 只能读取最小化上下文并返回决策,不能调用 Provider 或改账。新增协议时应复用准入、风控和账务边界,而不是在 Controller 中另写一套钱包/重试流程。默认 `GATEWAY_RISK_MODE=shadow` 仅记录规则 ID,不改变调用;只有显式部署规则并设置 `enforce` 才会拒绝或限速。请求来源由服务端路由确定,不能信任 `X-Moyi-Source` 作为风控分类。
账务不变量:风险拒绝必须先于并发准入和钱包预留;预留使用价格快照;已确认失败的释放和成功结算沿用原子事务;不完整流或无效用量不得估算扣费。
异步创建支持用户级 `Idempotency-Key`(`V45`),同键同请求返回原任务,同键不同请求返回 409。异步上游受理未知、回执丢失、worker 中断或轮询期限到期时进入 `reconciling`(`V46`),保留冻结与预算占用,不自动重发/退款。已观察的生成结果先持久化再结算,账本失败只重试本地结算。发布时须先停旧 worker,不能混跑旧的未知结果退款逻辑。详见 `../docs/22-async-task-safety.md`。
`V47` 冻结异步 provider/model 绑定并约束已证明的上游任务身份;`V48` 增加仅追加核账收据。管理端“价格版本与成本对账”提供人工核账,可根据证据恢复只读轮询、确认失败释放原冻结、确认成功并按原价和实际用量结算。操作使用管理员会话、版本检查、UUID 幂等收据和与钱包同事务的审计;worker 和人工操作共用 `TaskSettlementService`,不另写扣费逻辑。历史无法证明的已受理绑定不能猜测补齐。详见 `../docs/23-audited-task-reconciliation.md`;尚无双人审批,不等同于完整财务流程。
同步和原生调用在结算前单独提交精简用量快照(Flyway `V44`)。快照已持久化而账本事务失败时,接口返回 `SETTLEMENT_PENDING`,冻结额度保留,由后台恢复任务重试结算。该快照不保存提示词、生成内容或密钥;不能将其当作响应重放存储。若数据库在快照写入前已不可用,则尚不能保证结果的持久恢复,必须结合上游记录核对。
`provider.credentials.UpstreamCredentialResolver` 共享密钥池解析与轮转;`provider.transport.SseEventReader` 专门负责 SSE 解帧、UTF-8 校验和有界读取,不承担风控、路由或计费。完整模块边界、验证命令和后续风险清单见 `../docs/21-api-request-modularization.md`。
`V49` 增加数据库共享登录窗口;所有新密码统一 8–256 位,改密撤销全部浏览器会话,登录/改密/重置通过用户行锁协调。初次创建管理员必须显式配置非默认密码;已存在管理员不会被启动配置重置。API Key 不随改密自动撤销。
`provider.transport.BoundedHttpTransport` 已供异步 JSON/multipart/音频/媒体读取复用:接收中限量、总期限覆盖响应体、取消订阅、不自动跟随重定向。它不等于 SSRF 已解决,DNS 预检与实际解析的绑定仍需补齐;同步/SSE 生命周期和 Codex 长请求亦未统一完成。详见 `../docs/25-auth-and-bounded-transport.md` 与 `../docs/24-security-and-cpr-task-tracker.md`。
Flyway 脚本位于 `src/main/resources/db/migration/`,与现有 TypeScript 服务使用同一套基础表命名,后续迁移数据时可以按表逐步切换。
本目录的镜像由 `infra/moyi/docker-compose.yml` 构建。切换生产流量前仍需完成真实供应商适配、Redis 限流/队列和完整集成测试;TypeScript Control Plane 继续作为迁移期间的参考实现。
本目录的镜像由 `infra/moyi/docker-compose.yml` 构建。生产发布前应验证目标渠道能力、价格快照、Redis 准入和 PostgreSQL 集成测试;本地通过不代表线上部署验收。TypeScript Control Plane 保留为历史参考实现,不与 Java 后端同时承担同一钱包的写入。
+7
View File
@@ -17,6 +17,9 @@
<properties>
<java.version>21</java.version>
<!-- Keep the new DNS-bound media transport on the matching stable Client/Core pair. -->
<httpclient5.version>5.6.5</httpclient5.version>
<httpcore5.version>5.4.4</httpcore5.version>
</properties>
<dependencies>
@@ -53,6 +56,10 @@
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-crypto</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
@@ -28,17 +28,20 @@ public class AuthController {
private static final String SESSION_COOKIE = "moyi_session";
private final AuthService authService;
private final VerificationService verification;
private final VerifiedAccountService verifiedAccounts;
private final int sessionTtlHours;
private final boolean secureCookie;
public AuthController(
AuthService authService,
VerificationService verification,
VerifiedAccountService verifiedAccounts,
@Value("${moyi.auth.session-ttl-hours}") int sessionTtlHours,
@Value("${moyi.auth.secure-cookie}") boolean secureCookie
) {
this.authService = authService;
this.verification = verification;
this.verifiedAccounts = verifiedAccounts;
this.sessionTtlHours = sessionTtlHours;
this.secureCookie = secureCookie;
}
@@ -67,9 +70,8 @@ public class AuthController {
@PostMapping("/auth/password-reset")
public ResponseEntity<Void> resetPassword(@RequestBody PasswordResetRequest input,HttpServletRequest request) {
String account = verification.consumeReset(input.account(), input.verificationCode());
authService.resetPassword(account, input.newPassword(), meta(request));
return ResponseEntity.noContent().build();
verifiedAccounts.resetPassword(input.account(), input.verificationCode(), input.newPassword(), meta(request));
return ResponseEntity.noContent().header(HttpHeaders.SET_COOKIE, sessionCookie("", Duration.ZERO).toString()).build();
}
public record InvitationRequest(String label, Integer maxUses, OffsetDateTime expiresAt) {}
@@ -102,9 +104,8 @@ public class AuthController {
HttpServletRequest request,
HttpServletResponse response
) {
verification.consume(input.account(),input.verificationCode());
AuthService.AuthResult result = authService.register(
input.invitationCode(), input.account(), input.password(), input.displayName(), meta(request)
AuthService.AuthResult result = verifiedAccounts.register(
input.invitationCode(), input.account(), input.password(), input.displayName(), input.verificationCode(), meta(request)
);
response.addHeader(HttpHeaders.SET_COOKIE, sessionCookie(result.token(), null).toString());
return ResponseEntity.status(HttpStatus.CREATED).body(Map.of("user", result.user()));
@@ -123,10 +124,12 @@ public class AuthController {
}
@PatchMapping("/auth/account")
public Map<String, Object> updateAccount(@RequestBody AccountSettingsRequest input, HttpServletRequest request) {
return Map.of("user", authService.updateAccount(
public Map<String, Object> updateAccount(@RequestBody AccountSettingsRequest input, HttpServletRequest request, HttpServletResponse response) {
var user = authService.updateAccount(
sessionToken(request), input.displayName(), input.currentPassword(), input.newPassword(), meta(request)
));
);
if (input.newPassword() != null) response.addHeader(HttpHeaders.SET_COOKIE, sessionCookie("", Duration.ZERO).toString());
return Map.of("user", user);
}
@GetMapping("/admin/users")
@@ -24,6 +24,8 @@ import org.springframework.dao.EmptyResultDataAccessException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
@Service
public class AuthService {
@@ -34,6 +36,10 @@ public class AuthService {
private final ObjectMapper objectMapper;
private final int sessionTtlHours;
private final SecretCipher secretCipher;
private final LoginAttemptLimiter loginAttempts;
private final TransactionTemplate loginTransactions;
// Valid PBKDF2 shape for comparable unknown-account work; this is not a usable user credential.
private static final String UNKNOWN_PASSWORD_HASH = "pbkdf2$210000$AAECAwQFBgcICQoLDA0ODw$_ZW4zVJDiheXQ51WuP_SSr_f8xjYJXBHvjkzLlPtFHo";
@Autowired private TotpService totp;
/** Kept for isolated unit tests that do not exercise encrypted invitation display. */
@@ -59,6 +65,8 @@ public class AuthService {
this.objectMapper = objectMapper;
this.sessionTtlHours = sessionTtlHours;
this.secretCipher = secretCipher;
this.loginAttempts = new LoginAttemptLimiter(jdbc);
this.loginTransactions = jdbc.getDataSource() == null ? null : new TransactionTemplate(new DataSourceTransactionManager(jdbc.getDataSource()));
}
public record UserView(
@@ -91,7 +99,7 @@ public class AuthService {
@Transactional
public void ensureAdmin(String accountInput, String password, String displayName) {
String account = normalizeAccount(accountInput);
if (account.isBlank() || password == null || password.isBlank()) {
if (account.isBlank()) {
throw new ApiException("INVALID_ADMIN_CONFIG", "管理员初始配置无效", 500);
}
// Serialize bootstrap across replicas, including the first creation.
@@ -112,6 +120,10 @@ public class AuthService {
return;
}
OffsetDateTime now = now();
PasswordPolicy.validate(password);
if (java.util.Set.of("admin123", "generate_admin_password", "replace-with-a-strong-initial-password").contains(password.strip().toLowerCase(Locale.ROOT))) {
throw new ApiException("INVALID_ADMIN_CONFIG", "首次启动必须配置非默认、非示例占位的管理员密码", 500);
}
jdbc.update(
"INSERT INTO users (id, account, account_kind, password_hash, display_name, role, status, created_at, updated_at) VALUES (?, ?, ?, ?, ?, 'super_admin', 'active', ?, ?)",
UUID.randomUUID(), account, accountKind(account), passwordHasher.hash(password),
@@ -123,18 +135,36 @@ public class AuthService {
return login(accountInput,password,null,meta);
}
public AuthResult login(String accountInput,String password,String otp,RequestMeta meta) {
UserRow user = findUserByAccount(normalizeAccount(accountInput));
if (user == null || !passwordHasher.matches(password, user.passwordHash())) {
String account = normalizeAccount(accountInput);
if (account.length() > 320 || password == null || password.length() > 4096) throw new ApiException("INVALID_CREDENTIALS", "账号或密码错误", 401);
loginAttempts.check(account, meta == null ? null : meta.ipAddress());
// Acquire the row lock only AFTER the independently committed limiter. Do not hold a pool
// connection while asking REQUIRES_NEW for another connection under concurrent login load.
if (loginTransactions == null) throw new IllegalStateException("Login requires a transactional data source");
LoginOutcome outcome = loginTransactions.execute(status -> authenticateLocked(account, password, otp, meta));
if (outcome.error() != null) throw outcome.error();
return outcome.result();
}
private record LoginOutcome(AuthResult result, ApiException error) {}
private LoginOutcome authenticateLocked(String account, String password, String otp, RequestMeta meta) {
UserRow user = findUserByAccountLocked(account);
boolean matches = passwordHasher.matches(password, user == null ? UNKNOWN_PASSWORD_HASH : user.passwordHash());
if (user == null || !matches) {
throw new ApiException("INVALID_CREDENTIALS", "账号或密码错误", 401);
}
if (!"active".equals(user.status())) {
throw new ApiException("ACCOUNT_DISABLED", "账号已停用", 403);
}
if(totp!=null) totp.verifyLogin(user.id(),otp);
if(totp!=null) {
try { totp.verifyLogin(user.id(),otp); }
catch (ApiException exception) { return new LoginOutcome(null, exception); } // Persist MFA failure/replay counters.
}
OffsetDateTime loginAt = now();
jdbc.update("UPDATE users SET last_login_at = ?, updated_at = ? WHERE id = ?", loginAt, loginAt, user.id());
audit("user.logged_in", user.id(), meta, Map.of());
return new AuthResult(createSession(user.id(), meta), toView(user, loginAt));
return new LoginOutcome(new AuthResult(createSession(user.id(), meta), toView(user, loginAt)), null);
}
public void verifyPassword(UUID userId,String password) {
String hash=jdbc.queryForObject("SELECT password_hash FROM users WHERE id=? AND status='active'",String.class,userId);
@@ -151,9 +181,7 @@ public class AuthService {
) {
String account = normalizeAccount(accountInput);
validatePublicAccount(account);
if (password == null || password.length() < 8) {
throw new ApiException("WEAK_PASSWORD", "密码至少需要 8 位");
}
PasswordPolicy.validate(password);
if (invitationCode == null || invitationCode.isBlank()) {
throw new ApiException("INVITATION_REQUIRED", "第一版注册需要邀请码");
}
@@ -177,6 +205,7 @@ public class AuthService {
}
UUID userId = UUID.randomUUID();
String name = displayName == null || displayName.isBlank() ? defaultDisplayName(account) : displayName.trim();
if (name.length() > 80) throw new ApiException("INVALID_NAME", "显示名称最多 80 字符", 400);
jdbc.update(
"INSERT INTO users (id, account, account_kind, password_hash, display_name, role, status, created_at, updated_at) VALUES (?, ?, ?, ?, ?, 'user', 'active', ?, ?)",
userId, account, accountKind(account), passwordHasher.hash(password), name, now, now
@@ -242,22 +271,23 @@ public class AuthService {
String newPassword,
RequestMeta meta
) {
UserRow user = requireUserRow(token);
UserRow authenticated = requireUserRow(token);
jdbc.queryForList("SELECT id FROM users WHERE id=? FOR UPDATE", authenticated.id());
UserRow user = requireUserRow(token); // Recheck after any concurrent password change revoked this session.
String displayName = displayNameInput == null ? user.displayName() : displayNameInput.trim();
if (displayName.isBlank() || displayName.length() > 80) {
throw new ApiException("INVALID_NAME", "显示名称不能为空且最多 80 字符", 400);
}
boolean changingPassword = newPassword != null && !newPassword.isBlank();
boolean changingPassword = newPassword != null;
if (changingPassword) {
if (currentPassword == null || currentPassword.isBlank() || !passwordHasher.matches(currentPassword, user.passwordHash())) {
throw new ApiException("INVALID_CURRENT_PASSWORD", "当前密码错误", 403);
}
if (newPassword.length() > 256 || (!"super_admin".equals(user.role()) && newPassword.length() < 8)) {
throw new ApiException("WEAK_PASSWORD", "普通用户新密码长度为 8 至 256 位", 400);
}
PasswordPolicy.validate(newPassword);
}
if (changingPassword) {
jdbc.update("UPDATE users SET display_name = ?, password_hash = ?, updated_at = ? WHERE id = ?", displayName, passwordHasher.hash(newPassword), now(), user.id());
jdbc.update("UPDATE auth_sessions SET revoked_at=? WHERE user_id=? AND revoked_at IS NULL", now(), user.id());
} else {
jdbc.update("UPDATE users SET display_name = ?, updated_at = ? WHERE id = ?", displayName, now(), user.id());
}
@@ -268,14 +298,11 @@ public class AuthService {
@Transactional
public void resetPassword(String accountInput, String newPassword, RequestMeta meta) {
String account = normalizeAccount(accountInput);
UserRow user = findUserByAccount(account);
UserRow user = findUserByAccountLocked(account);
if (user == null || !"active".equals(user.status())) {
throw new ApiException("ACCOUNT_NOT_FOUND", "账号不存在或已停用", 404);
}
if (newPassword == null || newPassword.isBlank() || newPassword.length() > 256
|| (!"super_admin".equals(user.role()) && newPassword.length() < 8)) {
throw new ApiException("WEAK_PASSWORD", "普通用户新密码长度为 8 至 256 位", 400);
}
PasswordPolicy.validate(newPassword);
OffsetDateTime changedAt = now();
jdbc.update("UPDATE users SET password_hash=?, updated_at=? WHERE id=?", passwordHasher.hash(newPassword), changedAt, user.id());
jdbc.update("UPDATE auth_sessions SET revoked_at=? WHERE user_id=? AND revoked_at IS NULL", changedAt, user.id());
@@ -312,7 +339,7 @@ public class AuthService {
UserView actor=requireAdmin(token);
String account=normalizeAccount(accountInput);
if(account.isBlank() || account.length()>254 || (!EMAIL.matcher(account).matches() && !PHONE.matcher(account).matches() && !account.matches("[a-z0-9_]{3,64}"))) throw new ApiException("INVALID_ACCOUNT","请输入有效邮箱、手机号或 3 至 64 位字母数字下划线用户名");
if(password==null || password.length()<8 || password.length()>256) throw new ApiException("WEAK_PASSWORD","普通用户密码长度为 8 至 256 位");
PasswordPolicy.validate(password);
if(findUserByAccount(account)!=null) throw new ApiException("ACCOUNT_EXISTS","账号已存在",409);
String name=displayName==null || displayName.isBlank()?defaultDisplayName(account):displayName.trim();
if(name.length()>80) throw new ApiException("INVALID_NAME","显示名称最多 80 字符");
@@ -446,6 +473,11 @@ public class AuthService {
}
}
private UserRow findUserByAccountLocked(String account) {
try { return jdbc.queryForObject("SELECT * FROM users WHERE account=? FOR UPDATE", this::mapUser, account); }
catch (EmptyResultDataAccessException exception) { return null; }
}
private UserRow findUserById(UUID id) {
return jdbc.queryForObject("SELECT * FROM users WHERE id = ?", this::mapUser, id);
}
@@ -511,13 +543,13 @@ public class AuthService {
}
private void validatePublicAccount(String account) {
if (!EMAIL.matcher(account).matches() && !PHONE.matcher(account).matches()) {
if (account.length() > 254 || (!EMAIL.matcher(account).matches() && !PHONE.matcher(account).matches())) {
throw new ApiException("INVALID_ACCOUNT", "请输入有效的邮箱或手机号");
}
}
private String normalizeAccount(String account) {
String normalized = account == null ? "" : account.trim().toLowerCase();
String normalized = account == null ? "" : account.trim().toLowerCase(Locale.ROOT);
return normalized.contains("@") ? normalized : normalized.replaceAll("[\\s-]", "");
}
@@ -26,6 +26,11 @@ public class CloudVerificationSender implements VerificationSender {
@Override
public void send(String channel,String account,String code) {
send(channel,account,code,VerificationPurpose.REGISTRATION);
}
@Override
public void send(String channel,String account,String code,VerificationPurpose purpose) {
if(!available(channel)) throw new ApiException("VERIFICATION_UNAVAILABLE","该验证码发送服务尚未配置,请联系管理员",503);
try {
if("email".equals(channel)) {
@@ -47,8 +52,8 @@ public class CloudVerificationSender implements VerificationSender {
SimpleMailMessage message=new SimpleMailMessage();
message.setFrom(value("MOYI_SMTP_FROM"));
message.setTo(account);
message.setSubject("MOYI 注册验证码");
message.setText("您的注册验证码为:"+code+",5 分钟内有效。请勿向他人泄露。若非本人操作,请忽略此邮件。");
message.setSubject("MOYI "+purpose.label+"验证码");
message.setText("您的"+purpose.label+"验证码为:"+code+",5 分钟内有效。请勿向他人泄露。若非本人操作,请忽略此邮件。");
mail.send(message);
} else {
Config config=new Config().setAccessKeyId(value("MOYI_SMS_ACCESS_KEY_ID")).setAccessKeySecret(value("MOYI_SMS_ACCESS_KEY_SECRET")).setEndpoint("dysmsapi.aliyuncs.com").setProtocol("https");
@@ -57,8 +62,7 @@ public class CloudVerificationSender implements VerificationSender {
.setSignName(value("MOYI_SMS_SIGN_NAME")).setTemplateCode(value("MOYI_SMS_TEMPLATE_CODE")).setTemplateParam("{\"code\":\""+code+"\"}");
var response=client.sendSmsWithOptions(request,new RuntimeOptions().setAutoretry(false).setConnectTimeout(5000).setReadTimeout(10000));
if(response.getBody()==null || !"OK".equals(response.getBody().getCode())) {
String resultCode = response.getBody() == null ? "EMPTY_RESPONSE" : String.valueOf(response.getBody().getCode());
log.warn("Aliyun SMS verification rejected: resultCode={}", resultCode);
log.warn("Aliyun SMS verification rejected");
throw new IllegalStateException("SMS rejected");
}
}
@@ -0,0 +1,40 @@
package com.moyi.controlplane.auth;
import com.moyi.controlplane.gateway.GatewayLimitException;
import java.util.List;
import java.util.Map;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.support.TransactionTemplate;
/** Bounds expensive password work for known AND unknown users; counters survive failed-login rollbacks. */
final class LoginAttemptLimiter {
private final JdbcTemplate jdbc;
private final TransactionTemplate transactions;
LoginAttemptLimiter(JdbcTemplate jdbc) {
this.jdbc = jdbc;
if (jdbc.getDataSource() == null) { transactions = null; return; } // Isolated mock-only constructors.
transactions = new TransactionTemplate(new DataSourceTransactionManager(jdbc.getDataSource()));
transactions.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
}
void check(String normalizedAccount, String sourceIp) {
if (transactions == null) throw new IllegalStateException("Login limiter requires a transactional data source");
int retry = transactions.execute(status -> {
var limits = Map.of(AuthService.hashSecret("login:account:" + normalizedAccount), 10,
AuthService.hashSecret("login:source:" + (sourceIp == null || sourceIp.isBlank() ? "unknown" : sourceIp)), 50);
int remaining = 0;
// Stable ordering prevents cross-account/source deadlocks on concurrent replicas.
for (String bucket : limits.keySet().stream().sorted().toList()) {
int limit = limits.get(bucket);
var row = jdbc.queryForMap("INSERT INTO auth_login_windows(bucket_hash,attempts) VALUES (?,1) ON CONFLICT(bucket_hash) DO UPDATE SET attempts=CASE WHEN auth_login_windows.expires_at<=clock_timestamp() THEN 1 ELSE LEAST(auth_login_windows.attempts+1,?) END,window_started_at=CASE WHEN auth_login_windows.expires_at<=clock_timestamp() THEN clock_timestamp() ELSE auth_login_windows.window_started_at END,expires_at=CASE WHEN auth_login_windows.expires_at<=clock_timestamp() THEN clock_timestamp()+interval '60 seconds' ELSE auth_login_windows.expires_at END RETURNING attempts,GREATEST(1,CEIL(EXTRACT(EPOCH FROM (expires_at-clock_timestamp()))))::integer AS retry", bucket, limit + 1);
if (((Number) row.get("attempts")).intValue() > limit) remaining = Math.max(remaining, ((Number) row.get("retry")).intValue());
}
jdbc.update("DELETE FROM auth_login_windows WHERE bucket_hash IN (SELECT bucket_hash FROM auth_login_windows WHERE expires_at<clock_timestamp()-interval '5 minutes' LIMIT 500)");
return remaining;
});
if (retry > 0) throw new GatewayLimitException("AUTH_RATE_LIMITED", "登录尝试过于频繁,请稍后重试", 429, retry);
}
}
@@ -30,13 +30,17 @@ public class PasswordHasher {
public boolean matches(String password, String encoded) {
try {
// Accept existing bounded legacy passwords, but never let malformed hashes choose unbounded KDF work.
if (password == null || password.length() > 4096 || encoded == null || encoded.length() > 1024) return false;
String[] parts = encoded.split("\\$");
if (parts.length != 4 || !"pbkdf2".equals(parts[0])) {
return false;
}
int iterations = Integer.parseInt(parts[1]);
if (iterations < 10_000 || iterations > 1_000_000) return false;
byte[] salt = Base64.getUrlDecoder().decode(parts[2]);
byte[] expected = Base64.getUrlDecoder().decode(parts[3]);
if (salt.length < 16 || salt.length > 64 || expected.length != KEY_LENGTH / 8) return false;
byte[] actual = derive(password, salt, iterations, expected.length * 8);
return MessageDigest.isEqual(expected, actual);
} catch (RuntimeException exception) {
@@ -0,0 +1,13 @@
package com.moyi.controlplane.auth;
import com.moyi.controlplane.common.ApiException;
/** One issuance policy for every role; do not silently trim or truncate passwords. */
public final class PasswordPolicy {
private PasswordPolicy() {}
public static void validate(String password) {
if (password == null || password.isBlank() || password.length() < 8 || password.length() > 256) {
throw new ApiException("WEAK_PASSWORD", "所有账号的新密码长度须为 8 至 256 位,不能仅含空白", 400);
}
}
}
@@ -0,0 +1,134 @@
package com.moyi.controlplane.auth;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.security.SecretCipher;
import jakarta.annotation.PreDestroy;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.annotation.Propagation;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionTemplate;
/** Durable, encrypted single-attempt delivery. Never hold a DB transaction over a mail/SMS call. */
@Service
public class VerificationDispatchService {
private static final Logger log = LoggerFactory.getLogger(VerificationDispatchService.class);
private final JdbcTemplate jdbc;
private final VerificationSender sender;
private final SecretCipher cipher;
private final ObjectMapper json;
private final TransactionTemplate transactions;
private final ExecutorService workers = new ThreadPoolExecutor(0, 2, 30, TimeUnit.SECONDS, new SynchronousQueue<>(),
task -> { var thread=new Thread(task, "verification-delivery"); thread.setDaemon(true); return thread; },
new ThreadPoolExecutor.AbortPolicy());
public VerificationDispatchService(JdbcTemplate jdbc, VerificationSender sender, SecretCipher cipher,
ObjectMapper json, PlatformTransactionManager manager) {
this.jdbc=jdbc; this.sender=sender; this.cipher=cipher; this.json=json;
transactions=new TransactionTemplate(manager);
transactions.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
}
public boolean available(String channel) { return cipher.isConfigured() && sender.available(channel); }
/** Called only after common quotas; code generation and queue insert commit or roll back together. */
@Transactional(propagation=Propagation.MANDATORY)
public void enqueue(VerificationPurpose purpose, String account, String code, String codeHash) {
UUID id=UUID.randomUUID();
String payload;
try { payload=cipher.encrypt(json.writeValueAsString(Map.of("id",id.toString(),"purpose",purpose.key,"account",account,"code",code))); }
catch (ApiException error) { throw error; }
catch (Exception error) { throw new ApiException("VERIFICATION_QUEUE_FAILED", "验证码服务暂时不可用", 503); }
jdbc.update("INSERT INTO "+purpose.table+"(account,code_hash,expires_at,delivery_id) VALUES (?,?,clock_timestamp()+interval '5 minutes',?) ON CONFLICT(account) DO UPDATE SET code_hash=excluded.code_hash,expires_at=excluded.expires_at,attempts=0,consumed=false,created_at=clock_timestamp(),delivery_id=excluded.delivery_id", account,codeHash,id);
jdbc.update("INSERT INTO verification_delivery_outbox(id,purpose,payload_ciphertext) VALUES (?,?,?)",id,purpose.key,payload);
}
@Scheduled(fixedDelayString="${moyi.auth.verification-dispatch-interval-ms:1000}")
public void dispatchPending() {
// No queue behind these two workers: a slow provider cannot consume unbounded threads/tasks.
for (int i=0;i<2;i++) {
try { workers.submit(() -> {
try { dispatchOne(); }
catch (RuntimeException error) { log.warn("Verification delivery persistence failed; no automatic replay"); }
}); } catch (RejectedExecutionException busyOrStopping) { break; }
}
}
private record Attempt(UUID id, VerificationPurpose purpose, String ciphertext) {
@Override public String toString() { return "VerificationAttempt[id="+id+"]"; }
}
/** Package-visible for isolated deterministic database tests; not an HTTP operation. */
boolean dispatchOne() {
Attempt attempt=transactions.execute(status -> {
var rows=jdbc.queryForList("SELECT id,purpose,payload_ciphertext FROM verification_delivery_outbox WHERE status='queued' AND expires_at>clock_timestamp() ORDER BY created_at,id LIMIT 1 FOR UPDATE SKIP LOCKED");
if (rows.isEmpty()) return null;
var row=rows.getFirst(); UUID id=(UUID)row.get("id");
jdbc.update("UPDATE verification_delivery_outbox SET status='sending',started_at=clock_timestamp() WHERE id=?",id);
return new Attempt(id,VerificationPurpose.fromKey(String.valueOf(row.get("purpose"))),String.valueOf(row.get("payload_ciphertext")));
});
if (attempt==null) { cleanupExpired(); return false; }
try {
var payload=json.readTree(cipher.decrypt(attempt.ciphertext()));
if (!attempt.id().toString().equals(payload.path("id").asText())
|| !attempt.purpose().key.equals(payload.path("purpose").asText())
|| !payload.path("code").asText().matches("[0-9]{6}")) throw new IllegalArgumentException("Invalid delivery envelope");
String account=payload.path("account").asText(), code=payload.path("code").asText();
if (!isCurrent(attempt,account)) { finish(attempt,"discarded","SUPERSEDED_OR_INACTIVE"); return true; }
if (Thread.currentThread().isInterrupted()) { finish(attempt,"failed","INTERRUPTED"); return true; }
String channel=account.contains("@")?"email":"phone";
// At this point the claim is committed and its connection is released.
sender.send(channel,account,code,attempt.purpose());
} catch (Exception error) {
// Do not attach/log exceptions: providers can echo recipients, OTPs or credentials.
finish(attempt,"failed","DELIVERY_FAILED");
return true;
}
// A storage failure here leaves 'sending'. Never re-send an ambiguously accepted message.
finish(attempt,"sent",null);
return true;
}
private boolean isCurrent(Attempt attempt, String account) {
String activeUser=attempt.purpose()==VerificationPurpose.PASSWORD_RESET
? " AND EXISTS(SELECT 1 FROM users WHERE account=c.account AND status='active')" : "";
return Boolean.TRUE.equals(jdbc.queryForObject("SELECT EXISTS(SELECT 1 FROM "+attempt.purpose().table+" c WHERE account=? AND delivery_id=? AND consumed=false AND attempts<5 AND expires_at>clock_timestamp()"+activeUser+")",Boolean.class,account,attempt.id()));
}
private void finish(Attempt attempt, String state, String failureCode) {
transactions.executeWithoutResult(status -> {
int changed=jdbc.update("UPDATE verification_delivery_outbox SET status=?,payload_ciphertext=NULL,finished_at=clock_timestamp(),failure_code=? WHERE id=? AND status='sending'",state,failureCode,attempt.id());
if (changed>0 && !"sent".equals(state)) {
jdbc.update("UPDATE "+attempt.purpose().table+" SET consumed=true WHERE delivery_id=?",attempt.id());
}
});
}
private void cleanupExpired() {
transactions.executeWithoutResult(status -> {
// 'sending' abandoned on restart is never retried. Erase the encrypted payload after expiry.
jdbc.update("UPDATE verification_delivery_outbox SET status='discarded',payload_ciphertext=NULL,finished_at=clock_timestamp(),failure_code='EXPIRED' WHERE id IN (SELECT id FROM verification_delivery_outbox WHERE status IN ('queued','sending') AND expires_at<=clock_timestamp() LIMIT 100 FOR UPDATE SKIP LOCKED)");
jdbc.update("DELETE FROM verification_delivery_outbox WHERE id IN (SELECT id FROM verification_delivery_outbox WHERE finished_at<clock_timestamp()-interval '2 days' LIMIT 100)");
for (var purpose : VerificationPurpose.values()) jdbc.update("DELETE FROM "+purpose.table+" WHERE account IN (SELECT account FROM "+purpose.table+" WHERE expires_at<clock_timestamp()-interval '1 day' LIMIT 100 FOR UPDATE SKIP LOCKED)");
});
}
@PreDestroy public void close() {
workers.shutdownNow();
try { workers.awaitTermination(3,TimeUnit.SECONDS); }
catch (InterruptedException error) { Thread.currentThread().interrupt(); }
}
}
@@ -0,0 +1,14 @@
package com.moyi.controlplane.auth;
/** Closed purpose set: SQL identifiers and message intent must never come from a request. */
public enum VerificationPurpose {
REGISTRATION("registration", "registration_codes", "注册"),
PASSWORD_RESET("password-reset", "password_reset_codes", "密码重置");
final String key, table, label;
VerificationPurpose(String key, String table, String label) { this.key=key; this.table=table; this.label=label; }
static VerificationPurpose fromKey(String key) {
for (var purpose : values()) if (purpose.key.equals(key)) return purpose;
throw new IllegalArgumentException("Invalid verification purpose");
}
}
@@ -3,4 +3,5 @@ package com.moyi.controlplane.auth;
public interface VerificationSender {
boolean available(String channel);
void send(String channel, String account, String code);
default void send(String channel, String account, String code, VerificationPurpose purpose) { send(channel,account,code); }
}
@@ -4,7 +4,6 @@ import com.moyi.controlplane.common.ApiException;
import java.security.MessageDigest;
import java.security.SecureRandom;
import java.nio.charset.StandardCharsets;
import java.time.OffsetDateTime;
import java.util.Locale;
import java.util.Map;
import java.util.UUID;
@@ -16,13 +15,13 @@ import org.springframework.transaction.annotation.Transactional;
@Service
public class VerificationService {
private final JdbcTemplate jdbc;
private final VerificationSender sender;
private final VerificationDispatchService dispatch;
private final String pepper;
private final SecureRandom random=new SecureRandom();
public VerificationService(JdbcTemplate jdbc,VerificationSender sender,@Value("${moyi.auth.password-pepper}") String pepper) {
this.jdbc=jdbc; this.sender=sender; this.pepper=pepper;
public VerificationService(JdbcTemplate jdbc,VerificationDispatchService dispatch,@Value("${moyi.auth.password-pepper}") String pepper) {
this.jdbc=jdbc; this.dispatch=dispatch; this.pepper=pepper;
}
public Map<String,Boolean> capabilities() { return Map.of("email",sender.available("email"),"phone",sender.available("phone")); }
public Map<String,Boolean> capabilities() { return Map.of("email",dispatch.available("email"),"phone",dispatch.available("phone")); }
public String normalize(String raw) {
String account=raw==null?"":raw.trim().toLowerCase(Locale.ROOT);
if(!account.contains("@")) account=account.replaceAll("[\\s-]","");
@@ -33,29 +32,33 @@ public class VerificationService {
private String registrationDigest(String account,String code) { return digest("registration",account,code); }
private String resetDigest(String account,String code) { return digest("password-reset",account,code); }
@Transactional(noRollbackFor=ApiException.class)
@Transactional
public Map<String,Object> send(String raw,String invitation,String ip) {
String account=normalize(raw), channel=account.contains("@")?"email":"phone";
if(!sender.available(channel)) throw new ApiException("VERIFICATION_UNAVAILABLE","该验证码发送服务尚未配置,请联系管理员",503);
if(!dispatch.available(channel)) throw new ApiException("VERIFICATION_UNAVAILABLE","该验证码发送服务尚未配置,请联系管理员",503);
Integer invites=jdbc.queryForObject("SELECT count(*) FROM invitation_codes WHERE code_hash=? AND disabled_at IS NULL AND used_count<max_uses AND (expires_at IS NULL OR expires_at>now())",Integer.class,AuthService.hashSecret(invitation==null?"":invitation.trim()));
if(invites==null || invites==0) throw new ApiException("INVITATION_INVALID","请先填写有效邀请码",403);
String source=ip==null?"unknown":ip;
// One global lock bounds aggregate sending and serializes account/IP limits across replicas.
jdbc.queryForList("SELECT pg_advisory_xact_lock(734092615)");
Integer limited=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account=? AND created_at>now()-interval '60 seconds'",Integer.class,account);
Integer daily=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account=? AND created_at>now()-interval '24 hours'",Integer.class,account);
Integer hourly=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE ip_address=? AND created_at>now()-interval '1 hour'",Integer.class,source);
Integer total=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE created_at>now()-interval '24 hours'",Integer.class);
if(limited>0 || daily>=10 || hourly>=30 || total>=1000) throw new ApiException("VERIFICATION_RATE_LIMIT","发送过于频繁,请稍后重试",429);
admitSend(account,ip);
String code=String.format(Locale.ROOT,"%06d",random.nextInt(1000000));
jdbc.update("INSERT INTO verification_send_events(id,account,ip_address) VALUES (?,?,?)",UUID.randomUUID(),account,source);
jdbc.update("INSERT INTO registration_codes(account,code_hash,expires_at) VALUES (?,?,now()+interval '5 minutes') ON CONFLICT(account) DO UPDATE SET code_hash=excluded.code_hash, expires_at=excluded.expires_at, attempts=0, consumed=false, created_at=now()",account,registrationDigest(account,code));
try { sender.send(channel,account,code); }
catch(ApiException exception) { jdbc.update("UPDATE registration_codes SET consumed=true WHERE account=?",account); throw exception; }
jdbc.update("DELETE FROM verification_send_events WHERE created_at<now()-interval '2 days'");
return Map.of("sent",true,"retryAfterSeconds",60,"expiresInSeconds",300);
dispatch.enqueue(VerificationPurpose.REGISTRATION,account,code,registrationDigest(account,code));
return accepted();
}
private void admitSend(String account,String ip) {
String source=ip==null || ip.isBlank()?"unknown":ip;
// This lock covers only bounded DB work, NEVER the mail/SMS provider.
jdbc.queryForObject("SELECT set_config('lock_timeout','2s',true)",String.class);
jdbc.queryForList("SELECT pg_advisory_xact_lock(734092615)");
jdbc.update("DELETE FROM verification_send_events WHERE id IN (SELECT id FROM verification_send_events WHERE created_at<clock_timestamp()-interval '2 days' LIMIT 500)");
Integer limited=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account=? AND created_at>clock_timestamp()-interval '60 seconds'",Integer.class,account);
Integer daily=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account=? AND created_at>clock_timestamp()-interval '24 hours'",Integer.class,account);
Integer hourly=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE ip_address=? AND created_at>clock_timestamp()-interval '1 hour'",Integer.class,source);
Integer total=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE created_at>clock_timestamp()-interval '24 hours'",Integer.class);
if(limited>0 || daily>=10 || hourly>=30 || total>=1000) throw new ApiException("VERIFICATION_RATE_LIMIT","发送过于频繁,请稍后重试",429);
jdbc.update("INSERT INTO verification_send_events(id,account,ip_address,created_at) VALUES (?,?,?,clock_timestamp())",UUID.randomUUID(),account,source);
}
private Map<String,Object> accepted() { return Map.of("sent",true,"retryAfterSeconds",60,"expiresInSeconds",300); }
@Transactional(noRollbackFor=ApiException.class)
public void consume(String raw,String code) {
String account=normalize(raw);
@@ -69,39 +72,30 @@ public class VerificationService {
jdbc.update("UPDATE registration_codes SET consumed=true WHERE account=?",account);
}
@Transactional(noRollbackFor=ApiException.class)
@Transactional
public Map<String,Object> sendReset(String raw,String ip) {
String account=normalize(raw), channel=account.contains("@")?"email":"phone";
if(!sender.available(channel)) throw new ApiException("VERIFICATION_UNAVAILABLE","该验证码发送服务尚未配置,请联系管理员",503);
Integer exists=jdbc.queryForObject("SELECT count(*) FROM users WHERE account=? AND status='active'",Integer.class,account);
// Do not reveal whether an account exists. The client receives the same response either way.
if(exists==null || exists==0) return Map.of("sent",true,"retryAfterSeconds",60,"expiresInSeconds",300);
String source=ip==null?"unknown":ip;
jdbc.queryForList("SELECT pg_advisory_xact_lock(734092615)");
Integer limited=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account=? AND created_at>now()-interval '60 seconds'",Integer.class,account);
Integer daily=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account=? AND created_at>now()-interval '24 hours'",Integer.class,account);
Integer hourly=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE ip_address=? AND created_at>now()-interval '1 hour'",Integer.class,source);
Integer total=jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE created_at>now()-interval '24 hours'",Integer.class);
if(limited>0 || daily>=10 || hourly>=30 || total>=1000) throw new ApiException("VERIFICATION_RATE_LIMIT","发送过于频繁,请稍后重试",429);
if(!dispatch.available(channel)) throw new ApiException("VERIFICATION_UNAVAILABLE","该验证码发送服务尚未配置,请联系管理员",503);
// Exact same request-side DB/crypto path for active, unknown and disabled accounts.
// Existence is checked ONLY in the worker; no network result reaches this receipt.
admitSend(account,ip);
String code=String.format(Locale.ROOT,"%06d",random.nextInt(1000000));
jdbc.update("INSERT INTO verification_send_events(id,account,ip_address) VALUES (?,?,?)",UUID.randomUUID(),account,source);
jdbc.update("INSERT INTO password_reset_codes(account,code_hash,expires_at) VALUES (?,?,now()+interval '5 minutes') ON CONFLICT(account) DO UPDATE SET code_hash=excluded.code_hash, expires_at=excluded.expires_at, attempts=0, consumed=false, created_at=now()",account,resetDigest(account,code));
try { sender.send(channel,account,code); }
catch(ApiException exception) { jdbc.update("UPDATE password_reset_codes SET consumed=true WHERE account=?",account); throw exception; }
return Map.of("sent",true,"retryAfterSeconds",60,"expiresInSeconds",300);
dispatch.enqueue(VerificationPurpose.PASSWORD_RESET,account,code,resetDigest(account,code));
return accepted();
}
@Transactional(noRollbackFor=ApiException.class)
public String consumeReset(String raw,String code) {
String account=normalize(raw);
var rows=jdbc.queryForList("SELECT * FROM password_reset_codes WHERE account=? FOR UPDATE",account);
if(rows.isEmpty()) throw new ApiException("VERIFICATION_INVALID","请先获取验证码");
if(rows.isEmpty()) throw invalidResetCode();
var row=rows.getFirst();
java.time.Instant expiry=((java.sql.Timestamp)row.get("expires_at")).toInstant();
if(Boolean.TRUE.equals(row.get("consumed")) || expiry.isBefore(java.time.Instant.now()) || ((Number)row.get("attempts")).intValue()>=5) throw new ApiException("VERIFICATION_EXPIRED","验证码已失效,请重新获取");
if(Boolean.TRUE.equals(row.get("consumed")) || !expiry.isAfter(java.time.Instant.now()) || ((Number)row.get("attempts")).intValue()>=5) throw invalidResetCode();
jdbc.update("UPDATE password_reset_codes SET attempts=attempts+1 WHERE account=?",account);
if(code==null || !code.matches("[0-9]{6}") || !MessageDigest.isEqual(String.valueOf(row.get("code_hash")).getBytes(StandardCharsets.UTF_8),resetDigest(account,code).getBytes(StandardCharsets.UTF_8))) throw new ApiException("VERIFICATION_INVALID","验证码错误");
if(code==null || !code.matches("[0-9]{6}") || !MessageDigest.isEqual(String.valueOf(row.get("code_hash")).getBytes(StandardCharsets.UTF_8),resetDigest(account,code).getBytes(StandardCharsets.UTF_8))) throw invalidResetCode();
jdbc.update("UPDATE password_reset_codes SET consumed=true WHERE account=?",account);
return account;
}
private ApiException invalidResetCode() { return new ApiException("VERIFICATION_INVALID","验证码无效或已失效,请重新获取"); }
}
@@ -0,0 +1,60 @@
package com.moyi.controlplane.auth;
import com.moyi.controlplane.common.ApiException;
import java.util.function.Function;
import java.util.function.Supplier;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionDefinition;
import org.springframework.transaction.support.TransactionTemplate;
/** Public registration/recovery transaction boundary. Never call these flows inside a business transaction. */
@Service
public class VerifiedAccountService {
private final AuthService auth;
private final VerificationService verification;
private final TransactionTemplate transactions;
public VerifiedAccountService(AuthService auth, VerificationService verification, PlatformTransactionManager manager) {
this.auth = auth;
this.verification = verification;
transactions = new TransactionTemplate(manager);
// Own the public operation; a caller rollback cannot erase committed incorrect-code counters.
transactions.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRES_NEW);
}
public AuthService.AuthResult register(String invitation, String account, String password, String displayName,
String code, AuthService.RequestMeta meta) {
PasswordPolicy.validate(password);
return complete(() -> { verification.consume(account, code); return account; },
verified -> auth.register(invitation, verified, password, displayName, meta));
}
public void resetPassword(String account, String code, String password, AuthService.RequestMeta meta) {
PasswordPolicy.validate(password);
complete(() -> verification.consumeReset(account, code), verified -> {
auth.resetPassword(verified, password, meta);
return Boolean.TRUE;
});
}
private record Completion<T>(T result, ApiException rejectedCode) {}
private <T> T complete(Supplier<String> consume, Function<String, T> changeAccount) {
Completion<T> outcome = transactions.execute(status -> {
String account;
try { account = consume.get(); }
catch (ApiException rejected) {
// VerificationService intentionally commits wrong-code attempts. Return the rejection,
// then throw AFTER commit. No account mutation has run at this point.
return new Completion<T>(null, rejected);
}
// Do not catch business/audit/storage failures here: roll back successful code consumption,
// invitation redemption, password/session changes and audit as one operation.
return new Completion<>(changeAccount.apply(account), null);
});
if (outcome == null) throw new IllegalStateException("Verified account transaction returned no outcome");
if (outcome.rejectedCode() != null) throw outcome.rejectedCode();
return outcome.result();
}
}
@@ -383,6 +383,7 @@ public class CatalogService {
result.put("tasks", Map.of(
"queued", taskCounts.getOrDefault("queued", 0L),
"running", taskCounts.getOrDefault("running", 0L),
"reconciling", taskCounts.getOrDefault("reconciling", 0L),
"succeeded", taskCounts.getOrDefault("succeeded", 0L),
"failed", taskCounts.getOrDefault("failed", 0L)
));
@@ -4,7 +4,12 @@ import com.moyi.controlplane.catalog.CatalogService;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.provider.OpenAiCompatibleProvider;
import com.moyi.controlplane.wallet.ChatBillingService;
import com.moyi.controlplane.gateway.request.ApiRequestContext;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
import com.moyi.controlplane.gateway.request.ApiRequestProtocol;
import com.moyi.controlplane.gateway.request.GatewayRequestExecutionService;
import com.fasterxml.jackson.databind.ObjectMapper;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.time.Instant;
@@ -26,15 +31,16 @@ public class GatewayController {
private final CatalogService catalogService;
private final OpenAiCompatibleProvider provider;
private final GatewayGuard guard;
private final ChatBillingService billing;
private final GatewayRequestExecutionService requests;
private final ObjectMapper json;
@org.springframework.beans.factory.annotation.Autowired private GatewayFailover failover;
public GatewayController(CatalogService catalogService, OpenAiCompatibleProvider provider, GatewayGuard guard, ChatBillingService billing, ObjectMapper json) {
public GatewayController(CatalogService catalogService, OpenAiCompatibleProvider provider, GatewayGuard guard,
GatewayRequestExecutionService requests, ObjectMapper json) {
this.catalogService = catalogService;
this.provider = provider;
this.guard = guard;
this.billing = billing;
this.requests = requests;
this.json = json;
}
@@ -73,15 +79,31 @@ public class GatewayController {
@RequestHeader(value = "Authorization", required = false) String authorization,
@RequestBody Map<String, Object> input,
@RequestHeader(value = "Idempotency-Key", required = false) String idempotencyKey,
HttpServletRequest request,
HttpServletResponse servletResponse
) {
return executeChat(requireKey(authorization), input, idempotencyKey, servletResponse);
return executeChat(requireKey(authorization), input, idempotencyKey, servletResponse,
ApiRequestMetadata.fromHttp(request.getRemoteAddr()));
}
// Both the public gateway and the authenticated playground share admission and billing.
/** Direct-call compatibility for application tests and non-HTTP adapters. */
public ResponseEntity<Map<String, Object>> chatCompletions(String authorization,
Map<String, Object> input, String idempotencyKey, HttpServletResponse servletResponse) {
return executeChat(requireKey(authorization), input, idempotencyKey, servletResponse, ApiRequestMetadata.internal());
}
// Public API and Playground share the same request execution and risk boundary.
public ResponseEntity<Map<String, Object>> executeChat(Map<String, Object> key,
Map<String, Object> input, String idempotencyKey, HttpServletResponse servletResponse) {
return executeChat(key, input, idempotencyKey, servletResponse, ApiRequestMetadata.internal());
}
public ResponseEntity<Map<String, Object>> executeChat(Map<String, Object> key,
Map<String, Object> input, String idempotencyKey, HttpServletResponse servletResponse, ApiRequestMetadata metadata) {
long startedAt = System.currentTimeMillis();
UUID keyId = UUID.fromString(String.valueOf(key.get("id")));
// Reject request floods before catalog reads and payload normalization, including malformed requests.
guard.checkRate(keyId, false);
String modelId = String.valueOf(input.getOrDefault("model", ""));
Map<String, Object> model = resolveModel(key, modelId);
if (model == null) throw new ApiException("MODEL_NOT_FOUND", "模型不存在或已下线;请使用当前 API Key 的 /v1/models 返回的模型 ID", 404);
@@ -92,13 +114,15 @@ public class GatewayController {
boolean mock="provider-mock".equals(providerId);
if(!mock && !provider.isConfigured(providerId, routeId, String.valueOf(model.get("upstreamModel")))) throw new ApiException("PROVIDER_NOT_CONFIGURED","该供应商尚未配置密钥,不能执行真实调用",503);
ChatMeter.Request request=ChatMeter.request(input,json);
guard.checkRate(UUID.fromString(String.valueOf(key.get("id"))), false);
// Reserve both limits atomically. A queued request never consumes a
// model slot while waiting for the API-Key slot (or vice versa).
try (GatewayGuard.Admission admission = guard.acquire(model, UUID.fromString(String.valueOf(key.get("id"))), guard.queueWaitMs())) {
var ticket=billing.reserve(key,model,request,idempotencyKey);
ApiRequestContext context = new ApiRequestContext(
UUID.fromString(String.valueOf(key.get("userId"))), keyId, ApiRequestProtocol.CHAT_COMPLETIONS,
modelId, routeId, Boolean.TRUE.equals(request.body().get("stream")), request.reservedInputTokens(),
request.maxOutputTokens(), metadata, System.nanoTime());
return requests.execute(context, key, model, request, idempotencyKey, () -> {
// A queued request never consumes one scarce slot while waiting for another.
return guard.acquire(model, keyId, guard.queueWaitMs());
}, ticket -> {
servletResponse.setHeader("X-Request-Id",ticket.id().toString());
try {
catalogService.touchKey(key);
if(Boolean.TRUE.equals(request.body().get("stream"))) {
stream(key,model,modelId,routeId,request.body(),ticket,servletResponse,startedAt,mock);
@@ -106,13 +130,12 @@ public class GatewayController {
}
Map<String,Object> body=mock?mockResponse(modelId,input):failover!=null?failover.invoke(key,model,"chat",request.body(),ticket):provider.chat(providerId,String.valueOf(model.get("upstreamModel")),routeId,request.body());
body.put("model",modelId);
billing.observedUsage(ticket,"chat",body.get("usage"));
requests.observedUsage(ticket,"chat",body.get("usage"));
requireCompletion(body);
var cost=billing.settle(ticket,body,elapsed(startedAt),null);
var cost=requests.settle(ticket,body,elapsed(startedAt),null);
return ResponseEntity.ok().header("X-Request-Id",ticket.id().toString()).header("X-Moyi-Cost-CNY",cost.toPlainString()).body(body);
} catch(Exception exception) {
ApiException error=exception instanceof ApiException e?e:new ApiException("CALL_INTERRUPTED","调用中断,请查看使用记录确认结算状态",502);
billing.fail(ticket,error.getCode(),error.getMessage(),elapsed(startedAt));
}, exception -> exception instanceof ApiException api ? api
: new ApiException("CALL_INTERRUPTED","调用中断,请查看使用记录确认结算状态",502), error -> {
if(servletResponse.isCommitted()) {
try { event(servletResponse,Map.of("error",Map.of("code",error.getCode(),"message",error.getMessage()))); done(servletResponse); } catch(IOException ignored) { }
return null;
@@ -120,8 +143,7 @@ public class GatewayController {
servletResponse.resetBuffer();
servletResponse.setContentType("application/json");
throw error;
}
}
});
}
private void stream(Map<String,Object> key,Map<String,Object> model, String modelId, String routeId, Map<String,Object> input, ChatBillingService.Ticket ticket, HttpServletResponse response, long started, boolean mock) throws IOException {
@@ -131,7 +153,8 @@ public class GatewayController {
var choices=(List<?>)body.get("choices");
var message=((Map<?,?>)choices.getFirst()).get("message");
event(response,Map.of("id",body.get("id"),"object","chat.completion.chunk","model",modelId,"choices",List.of(Map.of("index",0,"delta",message))));
billing.settle(ticket,body,elapsed(started),elapsed(started));
requests.observedUsage(ticket,"chat",body.get("usage"));
requests.settle(ticket,body,elapsed(started),elapsed(started));
event(response,Map.of("id",body.get("id"),"object","chat.completion.chunk","model",modelId,"choices",List.of(Map.of("index",0,"delta",Map.of(),"finish_reason","stop"))));
event(response,Map.of("choices",List.of(),"usage",body.get("usage"))); done(response); return;
}
@@ -147,9 +170,9 @@ public class GatewayController {
// A closed browser must not turn an otherwise completed provider call into free usage.
// Continue within the same hard deadline to obtain the final meter and settle once.
if(!disconnected[0]) try { event(response,chunk); } catch(IOException clientClosed) { disconnected[0]=true; }
}); } finally {billing.observedUsage(ticket,"chat",collector.observedUsage());}
}); } finally {requests.observedUsage(ticket,"chat",collector.observedUsage());}
var body=collector.result();
billing.settle(ticket,body,elapsed(started),collector.firstTokenMs());
requests.settle(ticket,body,elapsed(started),collector.firstTokenMs());
if(!disconnected[0]) done(response);
}
}
@@ -6,6 +6,7 @@ import java.util.LinkedHashMap;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicBoolean;
import com.moyi.controlplane.gateway.request.ApiRequestAdmission;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
@@ -22,6 +23,7 @@ public class GatewayGuard {
private static final long SLOT_TTL_MS = 180_000;
private static final DefaultRedisScript<Long> RATE = script("redis/rate-limit.lua");
private static final DefaultRedisScript<Long> ACQUIRE = script("redis/acquire-slot.lua");
private static final DefaultRedisScript<Long> ACQUIRE_TASK = script("redis/acquire-task-slot.lua");
private static final DefaultRedisScript<Long> ACQUIRE_PAIR = script("redis/acquire-pair.lua");
private static final DefaultRedisScript<List> CAPACITY = new DefaultRedisScript<>("""
local time = redis.call('TIME')
@@ -162,9 +164,11 @@ public class GatewayGuard {
if(policies==null) return true;
var policy=policies.taskLimit(keyId);
String bucket=redisPrefix+"gateway:slots:user:"+policy.userId();
boolean acquired=execute(ACQUIRE,bucket,Integer.toString(policy.concurrency()),Long.toString(Math.max(180000,ttlMs)),taskId)==1;
long result=execute(ACQUIRE_TASK,bucket,Integer.toString(policy.concurrency()),Long.toString(Math.max(180000,ttlMs)),taskId);
boolean acquired=result>0;
// Redis is outside the database transaction: roll back the lease if claiming the task fails.
if(acquired&&org.springframework.transaction.support.TransactionSynchronizationManager.isSynchronizationActive())
// Rolling back renewal must not remove a lease that existed before this transaction.
if(result==1&&org.springframework.transaction.support.TransactionSynchronizationManager.isSynchronizationActive())
org.springframework.transaction.support.TransactionSynchronizationManager.registerSynchronization(new org.springframework.transaction.support.TransactionSynchronization() {@Override public void afterCompletion(int status) {if(status!=STATUS_COMMITTED) releaseQuietly(bucket,taskId);}});
return acquired;
}
@@ -315,7 +319,7 @@ public class GatewayGuard {
}
}
public final class Admission implements AutoCloseable {
public final class Admission implements ApiRequestAdmission {
private Slot userSlot;
private final Slot modelSlot;
private final Slot keySlot;
@@ -4,7 +4,10 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.catalog.CatalogService;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.provider.OpenAiCompatibleProvider;
import com.moyi.controlplane.wallet.ChatBillingService;
import com.moyi.controlplane.gateway.request.ApiRequestContext;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
import com.moyi.controlplane.gateway.request.ApiRequestProtocol;
import com.moyi.controlplane.gateway.request.GatewayRequestExecutionService;
import jakarta.servlet.http.*;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
@@ -15,9 +18,9 @@ import org.springframework.web.bind.annotation.*;
@RestController
public class NativeGatewayController {
private final CatalogService catalog; private final GatewayGuard guard;
private final ChatBillingService billing; private final OpenAiCompatibleProvider provider; private final ObjectMapper json;
private final GatewayRequestExecutionService requests; private final OpenAiCompatibleProvider provider; private final ObjectMapper json;
@org.springframework.beans.factory.annotation.Autowired private GatewayFailover failover;
public NativeGatewayController(CatalogService catalog,GatewayGuard guard,ChatBillingService billing,OpenAiCompatibleProvider provider,ObjectMapper json) {this.catalog=catalog;this.guard=guard;this.billing=billing;this.provider=provider;this.json=json;}
public NativeGatewayController(CatalogService catalog,GatewayGuard guard,GatewayRequestExecutionService requests,OpenAiCompatibleProvider provider,ObjectMapper json) {this.catalog=catalog;this.guard=guard;this.requests=requests;this.provider=provider;this.json=json;}
@PostMapping({"/v1/embeddings","/v1/messages","/v1/responses"})
public ResponseEntity<Map<String,Object>> invoke(@RequestBody Map<String,Object> input,HttpServletRequest request,HttpServletResponse response) {
String protocol=request.getRequestURI().substring(request.getRequestURI().lastIndexOf('/')+1);
@@ -41,7 +44,11 @@ public class NativeGatewayController {
return executeForKey(protocol,input,streaming,key,request,response);
}
public ResponseEntity<Map<String,Object>> executeForKey(String protocol,Map<String,Object> input,boolean streaming,Map<String,Object> key,HttpServletRequest request,HttpServletResponse response) {
UUID keyId=UUID.fromString(key.get("id").toString());guard.checkRate(keyId,false);
return executeForKey(protocol,input,streaming,key,request,response,ApiRequestMetadata.fromHttp(request.getRemoteAddr()));
}
public ResponseEntity<Map<String,Object>> executeForKey(String protocol,Map<String,Object> input,boolean streaming,Map<String,Object> key,HttpServletRequest request,HttpServletResponse response,ApiRequestMetadata metadata) {
UUID keyId=UUID.fromString(key.get("id").toString());
guard.checkRate(keyId,false);
String requested=String.valueOf(input.getOrDefault("model",""));
var model=GatewayModelIds.resolve(catalog,key,requested);
if(model==null) throw new ApiException("MODEL_NOT_FOUND","模型不存在或未授权",404);
@@ -50,17 +57,21 @@ public class NativeGatewayController {
if(!provider.isConfigured(providerId,route,upstream)) throw new ApiException("PROVIDER_NOT_CONFIGURED","渠道凭据未配置",503);
var normalized=NativeProtocolMeter.request(protocol,input,json);
long start=System.currentTimeMillis();
try(var admission=guard.acquire(model,keyId,guard.queueWaitMs())) {
var ticket=billing.reserve(key,model,normalized,request.getHeader("Idempotency-Key"));
billing.endpoint(ticket.id(),"gemini".equals(protocol)?"/v1beta/models:generateContent":"/v1/"+protocol);
ApiRequestProtocol requestProtocol = ApiRequestProtocol.from(protocol);
ApiRequestContext context = new ApiRequestContext(UUID.fromString(key.get("userId").toString()), keyId,
requestProtocol, requested, route, streaming, normalized.reservedInputTokens(), normalized.maxOutputTokens(),
metadata, System.nanoTime());
return requests.execute(context, key, model, normalized, request.getHeader("Idempotency-Key"), () -> {
return guard.acquire(model,keyId,guard.queueWaitMs());
}, ticket -> {
requests.endpoint(ticket,"gemini".equals(protocol)?"/v1beta/models:generateContent":"/v1/"+protocol);
response.setHeader("X-Request-Id",ticket.id().toString());
try {
catalog.touchKey(key);
if(!streaming) {
var result=failover!=null?failover.invoke(key,model,protocol,normalized.body(),ticket):provider.invoke(providerId,upstream,route,normalized.body(),protocol);
if(result.containsKey("model")) result.put("model",requested);
billing.observedUsage(ticket,protocol,result.get("gemini".equals(protocol)?"usageMetadata":"usage"));
var cost=billing.settle(ticket,NativeProtocolMeter.normalize(protocol,result),elapsed(start),null);
requests.observedUsage(ticket,protocol,result.get("gemini".equals(protocol)?"usageMetadata":"usage"));
var cost=requests.settle(ticket,NativeProtocolMeter.normalize(protocol,result),elapsed(start),null);
return ResponseEntity.ok().header("X-Request-Id",ticket.id().toString()).header("X-Moyi-Cost-CNY",cost.toPlainString()).body(result);
}
try(var stream=failover!=null?failover.stream(key,model,protocol,normalized.body(),ticket):provider.nativeStream(providerId,upstream,route,normalized.body(),protocol)) {
@@ -71,17 +82,15 @@ public class NativeGatewayController {
if(first[0]==null) first[0]=elapsed(start);
collector.accept(chunk);
if(!disconnected[0]) try {event(response,chunk);} catch(IOException e) {disconnected[0]=true;}
},false);} finally {billing.observedUsage(ticket,protocol,collector.observedUsage());}
billing.settle(ticket,collector.result(),elapsed(start),first[0]);
},false);} finally {requests.observedUsage(ticket,protocol,collector.observedUsage());}
requests.settle(ticket,collector.result(),elapsed(start),first[0]);
}
return null;
} catch(Exception failure) {
var error=failure instanceof ApiException api?api:new ApiException("CALL_INTERRUPTED","调用中断,渠道成本待核对",502);
billing.fail(ticket,error.getCode(),error.getMessage(),elapsed(start));
}, failure -> failure instanceof ApiException api ? api
: new ApiException("CALL_INTERRUPTED","调用中断,渠道成本待核对",502), error -> {
if(response.isCommitted()) {try {event(response,Map.of("type","error","error",Map.of("type",error.getCode(),"message",error.getMessage())));} catch(IOException ignored) {}return null;}
response.resetBuffer();response.setContentType("application/json");throw error;
}
}
});
}
private int elapsed(long start) {return (int)Math.min(Integer.MAX_VALUE,System.currentTimeMillis()-start);}
private void event(HttpServletResponse response,Map<String,Object> chunk) throws IOException {
@@ -21,7 +21,8 @@ public class PlaygroundController {
@org.springframework.beans.factory.annotation.Autowired private NativeGatewayController nativeGateway;
@PostMapping(value="/embeddings",consumes="application/json")
public ResponseEntity<Map<String,Object>> embeddings(@PathVariable String keyId,@RequestBody Map<String,Object> input,HttpServletRequest request,HttpServletResponse response) {
return nativeGateway.executeForKey("embeddings",input,false,requireKey(request,keyId),request,response);
return nativeGateway.executeForKey("embeddings",input,false,requireKey(request,keyId),request,response,
com.moyi.controlplane.gateway.request.ApiRequestMetadata.playground(request.getRemoteAddr()));
}
public PlaygroundController(AuthService auth, CatalogService catalog, GatewayController gateway, AsyncTaskService tasks) {
@@ -36,13 +37,16 @@ public class PlaygroundController {
@RequestBody Map<String, Object> input,
@RequestHeader(value="Idempotency-Key", required=false) String idempotencyKey,
HttpServletRequest request, HttpServletResponse response) {
return gateway.executeChat(requireKey(request, keyId), input, idempotencyKey, response);
return gateway.executeChat(requireKey(request, keyId), input, idempotencyKey, response,
com.moyi.controlplane.gateway.request.ApiRequestMetadata.playground(request.getRemoteAddr()));
}
@PostMapping(value="/{type:3d|video|image|audio}/tasks", consumes="application/json")
public ResponseEntity<Map<String, Object>> createTask(@PathVariable String keyId, @PathVariable String type,
@RequestBody Map<String, Object> input, HttpServletRequest request) {
return ResponseEntity.accepted().body(Map.of("task", tasks.createForKey(requireKey(request, keyId), type, input)));
@RequestBody Map<String, Object> input,
@RequestHeader(value="Idempotency-Key", required=false) String idempotencyKey, HttpServletRequest request) {
return ResponseEntity.accepted().body(Map.of("task", tasks.createForKey(requireKey(request, keyId), type, input, idempotencyKey,
com.moyi.controlplane.gateway.request.ApiRequestMetadata.playground(request.getRemoteAddr()))));
}
@GetMapping("/tasks/{taskId}")
@@ -0,0 +1,7 @@
package com.moyi.controlplane.gateway.request;
/** One request's releasable admission lease. */
public interface ApiRequestAdmission extends AutoCloseable {
@Override
void close();
}
@@ -0,0 +1,42 @@
package com.moyi.controlplane.gateway.request;
import java.util.Objects;
import java.util.UUID;
/** Immutable, payload-free facts supplied to admission and risk policies. */
public record ApiRequestContext(
UUID userId,
UUID apiKeyId,
ApiRequestProtocol protocol,
String requestedModelId,
String routeId,
boolean streaming,
long estimatedInputUnits,
int maxOutputUnits,
ApiRequestMetadata metadata,
long startedAtNanos
) {
public ApiRequestContext {
Objects.requireNonNull(userId, "userId");
Objects.requireNonNull(apiKeyId, "apiKeyId");
Objects.requireNonNull(protocol, "protocol");
requestedModelId = Objects.requireNonNullElse(requestedModelId, "");
routeId = Objects.requireNonNullElse(routeId, "");
metadata = metadata == null ? ApiRequestMetadata.internal() : metadata;
if (estimatedInputUnits < 0 || maxOutputUnits < 0) {
throw new IllegalArgumentException("Request estimates cannot be negative");
}
}
public int elapsedMillis() {
return (int) Math.min(Integer.MAX_VALUE,
Math.max(0, java.util.concurrent.TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startedAtNanos)));
}
/** Deliberately omit client metadata so incidental logging cannot expose it. */
@Override
public String toString() {
return "ApiRequestContext[protocol=" + protocol + ", userId=" + userId + ", apiKeyId=" + apiKeyId
+ ", requestedModelId=" + requestedModelId + ", routeId=" + routeId + ", streaming=" + streaming + "]";
}
}
@@ -0,0 +1,34 @@
package com.moyi.controlplane.gateway.request;
/** Trusted transport metadata captured by the HTTP adapter; never includes credentials or payloads. */
public record ApiRequestMetadata(String clientIp, Source source) {
public enum Source {
API,
PLAYGROUND,
INTERNAL
}
public ApiRequestMetadata {
source = source == null ? Source.INTERNAL : source;
if (clientIp != null && clientIp.length() > 64) clientIp = clientIp.substring(0, 64);
}
/** Public HTTP source is determined by the mapped server endpoint, never a caller-controlled header. */
public static ApiRequestMetadata fromHttp(String clientIp) {
return new ApiRequestMetadata(clientIp, Source.API);
}
/** Use only after the session-backed Playground controller has authorized its key. */
public static ApiRequestMetadata playground(String clientIp) {
return new ApiRequestMetadata(clientIp, Source.PLAYGROUND);
}
public static ApiRequestMetadata internal() {
return new ApiRequestMetadata(null, Source.INTERNAL);
}
@Override
public String toString() {
return "ApiRequestMetadata[source=" + source + "]";
}
}
@@ -0,0 +1,19 @@
package com.moyi.controlplane.gateway.request;
import java.util.List;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration(proxyBeanMethods = false)
public class ApiRequestModuleConfiguration {
@Bean
@ConditionalOnMissingBean(ApiRequestRiskEngine.class)
ApiRequestRiskEngine apiRequestRiskEngine(
List<ApiRequestRiskRule> rules,
@Value("${moyi.gateway.risk-mode:shadow}") String mode
) {
return new CompositeApiRequestRiskEngine(rules, CompositeApiRequestRiskEngine.Mode.parse(mode));
}
}
@@ -0,0 +1,24 @@
package com.moyi.controlplane.gateway.request;
import com.moyi.controlplane.common.ApiException;
public enum ApiRequestProtocol {
CHAT_COMPLETIONS,
MESSAGES,
RESPONSES,
GEMINI,
EMBEDDINGS,
ASYNC_TASK;
public static ApiRequestProtocol from(String value) {
return switch (value) {
case "chat", "chat.completions" -> CHAT_COMPLETIONS;
case "messages" -> MESSAGES;
case "responses" -> RESPONSES;
case "gemini" -> GEMINI;
case "embeddings" -> EMBEDDINGS;
case "async", "3d", "image", "audio", "video" -> ASYNC_TASK;
default -> throw new ApiException("UNSUPPORTED_PROTOCOL", "请求协议未接入", 400);
};
}
}
@@ -0,0 +1,63 @@
package com.moyi.controlplane.gateway.request;
import java.util.List;
import java.util.Objects;
public record ApiRequestRiskDecision(
Action action,
List<String> ruleIds,
String code,
String publicMessage,
int retryAfterSeconds
) {
public enum Action {
ALLOW,
SHADOW,
DENY,
THROTTLE,
UNAVAILABLE
}
public ApiRequestRiskDecision {
Objects.requireNonNull(action, "action");
ruleIds = ruleIds == null ? List.of() : List.copyOf(ruleIds);
for (String ruleId : ruleIds) {
if (ruleId == null || !ruleId.matches("[a-z][a-z0-9._-]{0,63}")) {
throw new IllegalArgumentException("Risk decision contains an invalid rule ID");
}
}
if (action == Action.DENY || action == Action.THROTTLE) {
if (code == null || !code.matches("[A-Z][A-Z0-9_]{1,63}")) {
throw new IllegalArgumentException("Blocking risk decision requires a safe error code");
}
if (publicMessage == null || publicMessage.isBlank() || publicMessage.length() > 240) {
throw new IllegalArgumentException("Blocking risk decision requires a short public message");
}
if (action == Action.THROTTLE ? retryAfterSeconds < 1 : retryAfterSeconds != 0) {
throw new IllegalArgumentException("Risk retry delay does not match its action");
}
} else if (retryAfterSeconds != 0) {
throw new IllegalArgumentException("Non-blocking risk decisions cannot include a retry delay");
}
}
public static ApiRequestRiskDecision allow() {
return new ApiRequestRiskDecision(Action.ALLOW, List.of(), null, null, 0);
}
public static ApiRequestRiskDecision shadow(List<String> ruleIds) {
return new ApiRequestRiskDecision(Action.SHADOW, ruleIds, null, null, 0);
}
public static ApiRequestRiskDecision deny(ApiRequestRiskSignal signal) {
return new ApiRequestRiskDecision(Action.DENY, List.of(signal.ruleId()), signal.code(), signal.publicMessage(), 0);
}
public static ApiRequestRiskDecision throttle(ApiRequestRiskSignal signal) {
return new ApiRequestRiskDecision(Action.THROTTLE, List.of(signal.ruleId()), signal.code(), signal.publicMessage(), signal.retryAfterSeconds());
}
public static ApiRequestRiskDecision unavailable(String ruleId) {
return new ApiRequestRiskDecision(Action.UNAVAILABLE, List.of(ruleId), null, null, 0);
}
}
@@ -0,0 +1,10 @@
package com.moyi.controlplane.gateway.request;
@FunctionalInterface
public interface ApiRequestRiskEngine {
ApiRequestRiskDecision evaluate(ApiRequestContext context);
static ApiRequestRiskEngine allowAll() {
return ignored -> ApiRequestRiskDecision.allow();
}
}
@@ -0,0 +1,50 @@
package com.moyi.controlplane.gateway.request;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.GatewayLimitException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;
/** Applies risk decisions before admission or wallet reservation. */
@Service
public class ApiRequestRiskPolicy {
private static final Logger log = LoggerFactory.getLogger(ApiRequestRiskPolicy.class);
private final ApiRequestRiskEngine engine;
public ApiRequestRiskPolicy(ApiRequestRiskEngine engine) {
this.engine = java.util.Objects.requireNonNull(engine, "engine");
}
public void enforce(ApiRequestContext context) {
final ApiRequestRiskDecision decision;
try {
decision = engine.evaluate(context);
} catch (RuntimeException exception) {
log.warn("API request risk engine failed ({})", exception.getClass().getSimpleName());
throw unavailable();
}
if (decision == null) throw unavailable();
switch (decision.action()) {
case ALLOW -> { }
case SHADOW -> log.info("API request risk shadow signal: rules={} protocol={} user={} key={} route={}",
decision.ruleIds(), context.protocol(), context.userId(), context.apiKeyId(), context.routeId());
case DENY -> {
log.warn("API request blocked by risk policy: rules={} protocol={} user={} key={} route={}",
decision.ruleIds(), context.protocol(), context.userId(), context.apiKeyId(), context.routeId());
throw new ApiException(decision.code(), decision.publicMessage(), 403);
}
case THROTTLE -> {
log.warn("API request throttled by risk policy: rules={} protocol={} user={} key={} route={}",
decision.ruleIds(), context.protocol(), context.userId(), context.apiKeyId(), context.routeId());
throw new GatewayLimitException(decision.code(), decision.publicMessage(), 429, decision.retryAfterSeconds());
}
case UNAVAILABLE -> throw unavailable();
}
}
private ApiException unavailable() {
return new ApiException("RISK_ENGINE_UNAVAILABLE", "请求安全检查暂不可用,请稍后重试", 503);
}
}
@@ -0,0 +1,10 @@
package com.moyi.controlplane.gateway.request;
import java.util.Optional;
/** A deterministic policy over request metadata only. Rules never call providers or mutate billing state. */
public interface ApiRequestRiskRule {
String id();
Optional<ApiRequestRiskSignal> evaluate(ApiRequestContext context);
}
@@ -0,0 +1,30 @@
package com.moyi.controlplane.gateway.request;
import java.util.Objects;
/** A rule's stable, public-safe decision; rule implementations must not put prompts or secrets in the message. */
public record ApiRequestRiskSignal(Action action, String ruleId, String code, String publicMessage, int retryAfterSeconds) {
public enum Action {
DENY,
THROTTLE
}
public ApiRequestRiskSignal {
Objects.requireNonNull(action, "action");
if (ruleId == null || !ruleId.matches("[a-z][a-z0-9._-]{0,63}")) {
throw new IllegalArgumentException("Risk rule ID is invalid");
}
if (code == null || !code.matches("[A-Z][A-Z0-9_]{1,63}")) {
throw new IllegalArgumentException("Risk error code is invalid");
}
if (publicMessage == null || publicMessage.isBlank() || publicMessage.length() > 240) {
throw new IllegalArgumentException("Risk public message must contain 1 to 240 characters");
}
if (action == Action.THROTTLE && retryAfterSeconds < 1) {
throw new IllegalArgumentException("A throttle decision must include a positive retry delay");
}
if (action == Action.DENY && retryAfterSeconds != 0) {
throw new IllegalArgumentException("A deny decision cannot include a retry delay");
}
}
}
@@ -0,0 +1,69 @@
package com.moyi.controlplane.gateway.request;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
import java.util.Optional;
/** Composes independently testable rules without giving them access to billing or provider ports. */
public final class CompositeApiRequestRiskEngine implements ApiRequestRiskEngine {
public enum Mode {
SHADOW,
ENFORCE;
public static Mode parse(String value) {
if (value == null) return SHADOW;
return switch (value.trim().toLowerCase(java.util.Locale.ROOT)) {
case "shadow" -> SHADOW;
case "enforce" -> ENFORCE;
default -> throw new IllegalArgumentException("GATEWAY_RISK_MODE must be 'shadow' or 'enforce'");
};
}
}
private final List<ApiRequestRiskRule> rules;
private final Mode mode;
public CompositeApiRequestRiskEngine(List<ApiRequestRiskRule> rules, Mode mode) {
this.rules = rules == null ? List.of() : rules.stream()
.sorted(Comparator.comparing(ApiRequestRiskRule::id))
.toList();
this.mode = mode == null ? Mode.SHADOW : mode;
var ids = new HashSet<String>();
for (ApiRequestRiskRule rule : this.rules) {
if (rule.id() == null || !rule.id().matches("[a-z][a-z0-9._-]{0,63}") || !ids.add(rule.id())) {
throw new IllegalArgumentException("Risk rule IDs must be unique lowercase identifiers");
}
}
}
@Override
public ApiRequestRiskDecision evaluate(ApiRequestContext context) {
var signals = new java.util.ArrayList<ApiRequestRiskSignal>();
for (ApiRequestRiskRule rule : rules) {
try {
Optional<ApiRequestRiskSignal> signal = rule.evaluate(context);
if (signal.isPresent()) {
if (!rule.id().equals(signal.get().ruleId())) {
throw new IllegalStateException("Risk rule returned another rule's ID");
}
signals.add(signal.get());
}
} catch (RuntimeException exception) {
if (mode == Mode.ENFORCE) return ApiRequestRiskDecision.unavailable(rule.id());
signals.add(new ApiRequestRiskSignal(ApiRequestRiskSignal.Action.DENY,
rule.id(), "RISK_RULE_UNAVAILABLE", "Risk rule evaluation is unavailable", 0));
}
}
if (signals.isEmpty()) return ApiRequestRiskDecision.allow();
if (mode == Mode.SHADOW) return ApiRequestRiskDecision.shadow(signals.stream().map(ApiRequestRiskSignal::ruleId).distinct().toList());
return signals.stream()
.max(Comparator.comparingInt((ApiRequestRiskSignal signal) -> signal.action() == ApiRequestRiskSignal.Action.DENY ? 2 : 1)
.thenComparingInt(ApiRequestRiskSignal::retryAfterSeconds))
.map(signal -> signal.action() == ApiRequestRiskSignal.Action.DENY
? ApiRequestRiskDecision.deny(signal)
: ApiRequestRiskDecision.throttle(signal))
.orElseGet(ApiRequestRiskDecision::allow);
}
}
@@ -0,0 +1,95 @@
package com.moyi.controlplane.gateway.request;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.ChatMeter;
import com.moyi.controlplane.wallet.ChatBillingService;
import java.math.BigDecimal;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.function.Function;
import java.util.function.Supplier;
import org.springframework.stereotype.Service;
/** Shared application boundary for synchronous and native-protocol model requests. */
@Service
public class GatewayRequestExecutionService {
private final ChatBillingService billing;
private final ApiRequestRiskPolicy risk;
public GatewayRequestExecutionService(ChatBillingService billing, ApiRequestRiskPolicy risk) {
this.billing = java.util.Objects.requireNonNull(billing, "billing");
this.risk = java.util.Objects.requireNonNull(risk, "risk");
}
public <T> T execute(
ApiRequestContext context,
Map<String, Object> key,
Map<String, Object> model,
ChatMeter.Request normalizedRequest,
String idempotencyKey,
Supplier<? extends ApiRequestAdmission> admission,
RequestWork<T> work,
Function<Exception, ApiException> classifyFailure,
Function<ApiException, T> deliverFailure
) {
risk.enforce(context);
try (ApiRequestAdmission ignored = admission.get()) {
ChatBillingService.Ticket ticket = billing.reserve(key, model, normalizedRequest, idempotencyKey, context.metadata());
try {
return work.execute(ticket);
} catch (Exception failure) {
if (failure instanceof SettlementPendingException pending) {
// The provider completed but persistence failed. Do not classify that as a provider failure.
// Recovery can retry automatically if prepareSettlement committed its normalized meter.
return deliverFailure.apply(pending);
}
ApiException error = classifyFailure.apply(failure);
billing.fail(ticket, error.getCode(), error.getMessage(), context.elapsedMillis());
return deliverFailure.apply(error);
}
}
}
public BigDecimal settle(ChatBillingService.Ticket ticket, Map<String, Object> result, int latencyMs, Integer firstTokenMs) {
// Validate and reduce to the only data required to price the call. The response body may contain
// prompts, generated content, tool arguments or other sensitive data and is never recovery state.
ChatMeter.Tokens tokens = ChatMeter.tokens(result);
Map<String,Object> usage = new LinkedHashMap<>();
usage.put("prompt_tokens", tokens.prompt());
usage.put("completion_tokens", tokens.completion());
usage.put("prompt_cache_hit_tokens", tokens.cached());
long cacheWrite = tokens.cacheWrite5m() + tokens.cacheWrite1h();
if (cacheWrite > 0) {
usage.put("cache_creation_input_tokens", cacheWrite);
usage.put("cache_creation", Map.of(
"ephemeral_5m_input_tokens", tokens.cacheWrite5m(),
"ephemeral_1h_input_tokens", tokens.cacheWrite1h()));
}
Map<String,Object> meter = Map.of("usage", usage);
try {
billing.prepareSettlement(ticket, meter, latencyMs, firstTokenMs);
return billing.settle(ticket, result, latencyMs, firstTokenMs);
} catch (org.springframework.dao.DataAccessException | org.springframework.transaction.TransactionException failure) {
throw new SettlementPendingException();
}
}
public void endpoint(ChatBillingService.Ticket ticket, String endpoint) {
billing.endpoint(ticket.id(), endpoint);
}
public void observedUsage(ChatBillingService.Ticket ticket, String protocol, Object usage) {
billing.observedUsage(ticket, protocol, usage);
}
@FunctionalInterface
public interface RequestWork<T> {
T execute(ChatBillingService.Ticket ticket) throws Exception;
}
public static final class SettlementPendingException extends ApiException {
private SettlementPendingException() {
super("SETTLEMENT_PENDING", "调用已完成,账务正在核对;请稍后查看使用记录", 503);
}
}
}
@@ -4,6 +4,8 @@ import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.security.SecretCipher;
import com.moyi.controlplane.provider.credentials.UpstreamCredentialResolver;
import com.moyi.controlplane.provider.transport.SseEventReader;
import java.net.URI;
import java.net.SocketTimeoutException;
import java.net.http.HttpClient;
@@ -16,7 +18,6 @@ import java.time.Duration;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.ArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.*;
import java.util.concurrent.ConcurrentHashMap;
@@ -242,35 +243,10 @@ public class OpenAiCompatibleProvider {
events(consumer,true);
}
public void events(EventConsumer consumer,boolean requireDone) throws IOException {
var reader=new BufferedReader(new InputStreamReader(body,StandardCharsets.UTF_8));
StringBuilder event=new StringBuilder(), line=new StringBuilder();
long total=0;
int c;
try {
// Bound both individual frames and the whole response before allocating unbounded strings.
while((c=reader.read())!=-1) {
if(expired.get()) throw new ApiException("UPSTREAM_PROVIDER_TIMEOUT","上游响应超时,本次未扣费",504);
if(++total>4*1024*1024 || line.length()>262144 || event.length()>262144) throw new ApiException("UPSTREAM_RESPONSE_TOO_LARGE","上游响应超过限制",502);
if(c=='\n') {
String text=line.toString(); line.setLength(0);
if(text.endsWith("\r")) text=text.substring(0,text.length()-1);
if(text.isEmpty()) {
if(!event.isEmpty()) {
String data=event.substring(0,event.length()-1); event.setLength(0);
if("[DONE]".equals(data)) return;
Map<String,Object> chunk=objectMapper.readValue(data,new com.fasterxml.jackson.core.type.TypeReference<>(){});
new SseEventReader(objectMapper).read(body, expired::get, requireDone, chunk -> {
if (chunk.containsKey("error")) throw classifyUpstreamError(502, writeJson(chunk));
consumer.accept(chunk);
}
} else if(text.startsWith("data:")) event.append(text.substring(5).stripLeading()).append('\n');
} else line.append((char)c);
}
} catch(IOException exception) {
if(expired.get()) throw new ApiException("UPSTREAM_PROVIDER_TIMEOUT","上游响应超时,本次未扣费",504);
throw exception;
}
if(!requireDone && !expired.get() && event.isEmpty() && line.isEmpty()) return;
throw new ApiException(expired.get()?"UPSTREAM_PROVIDER_TIMEOUT":"UPSTREAM_STREAM_INCOMPLETE","上游流式响应中断,本次未扣费",502);
});
}
@Override public void close() { deadline.cancel(false); try { body.close(); } catch(IOException ignored) {} finally {afterClose.run();afterClose=()->{};} }
}
@@ -351,25 +327,17 @@ public class OpenAiCompatibleProvider {
}
private ProviderConfig configured(String providerId, String providerCode, String fallbackKey, String baseUrl, org.springframework.core.env.Environment environment, AtomicInteger cursor) {
List<String> keys = new ArrayList<>();
if (fallbackKey != null && !fallbackKey.isBlank()) keys.add(fallbackKey.trim());
addEnvironmentKeys(keys, providerId, environment);
if (providerCode != null && !providerCode.isBlank() && !providerCode.equals(providerId)) addEnvironmentKeys(keys, providerCode, environment);
return new ProviderConfig(providerId, providerCode, keys.stream().distinct().toList(), baseUrl, cursor);
}
private void addEnvironmentKeys(List<String> keys, String alias, org.springframework.core.env.Environment environment) {
String envName = "MOYI_UPSTREAM_API_KEYS_" + alias.toUpperCase(java.util.Locale.ROOT).replace('-', '_');
String configured = environment == null ? null : environment.getProperty(envName);
if (configured != null && !configured.isBlank()) for (String key : configured.split("\\|")) if (!key.isBlank()) keys.add(key.trim());
List<String> keys = new UpstreamCredentialResolver(environment).providerKeys(fallbackKey, providerId, providerCode);
return new ProviderConfig(providerId, providerCode, keys, baseUrl, cursor);
}
private record ProviderConfig(String alias, String providerCode, List<String> apiKeys, String baseUrl, AtomicInteger cursor) {
@Override public String toString() { return "ProviderConfig[alias=" + alias + ", credentialCount=" + apiKeys.size() + "]"; }
private boolean hasKey() { return !apiKeys.isEmpty(); }
private boolean hasKey(String routeId, String upstreamModel, org.springframework.core.env.Environment environment) {
return !apiKeys.isEmpty() || !routeKeys(routeId, upstreamModel, environment).isEmpty();
}
private String nextApiKey() { return apiKeys.get(Math.floorMod(cursor.getAndIncrement(), apiKeys.size())); }
private String nextApiKey() { return UpstreamCredentialResolver.nextKey(apiKeys, cursor); }
private String nextApiKey(String routeId, String upstreamModel, org.springframework.core.env.Environment environment,
ConcurrentMap<String, AtomicInteger> cursors) {
@@ -377,32 +345,11 @@ public class OpenAiCompatibleProvider {
if (routeKeys.isEmpty()) return nextApiKey();
String cursorKey = "route:" + (routeId == null ? "" : routeId);
AtomicInteger routeCursor = cursors.computeIfAbsent(cursorKey, ignored -> new AtomicInteger());
return routeKeys.get(Math.floorMod(routeCursor.getAndIncrement(), routeKeys.size()));
return UpstreamCredentialResolver.nextKey(routeKeys, routeCursor);
}
private List<String> routeKeys(String routeId, String upstreamModel, org.springframework.core.env.Environment environment) {
if (environment == null) return List.of();
if (routeId != null && !routeId.isBlank()) {
String value = environment.getProperty("MOYI_UPSTREAM_API_KEYS_ROUTE_" + normalize(routeId));
if (value != null && !value.isBlank()) return parseKeys(value);
}
if (upstreamModel != null && !upstreamModel.isBlank()) {
String scopedName = "MOYI_UPSTREAM_API_KEYS_MODEL_" + normalize(providerCode) + "_" + normalize(upstreamModel);
String value = environment.getProperty(scopedName);
if (value == null || value.isBlank()) return List.of();
return parseKeys(value);
}
return List.of();
}
private static List<String> parseKeys(String value) {
List<String> keys = new ArrayList<>();
for (String key : value.split("\\|")) if (!key.isBlank()) keys.add(key.trim());
return keys.stream().distinct().toList();
}
private static String normalize(String value) {
return String.valueOf(value == null ? "" : value).toUpperCase(java.util.Locale.ROOT).replaceAll("[^A-Z0-9]", "_");
return new UpstreamCredentialResolver(environment).scopedKeys(providerCode, routeId, upstreamModel);
}
}
}
@@ -0,0 +1,68 @@
package com.moyi.controlplane.provider.credentials;
import com.moyi.controlplane.common.ApiException;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.concurrent.atomic.AtomicInteger;
import org.springframework.core.env.Environment;
/**
* Shared, side-effect-free credential pool resolution for sync and async providers.
* Selection state belongs to the caller so resolving a dynamic DB channel never resets rotation.
* This component neither logs credentials nor decides retries, health cooldowns or task affinity.
*/
public final class UpstreamCredentialResolver {
private final Environment environment;
public UpstreamCredentialResolver(Environment environment) {
this.environment = environment;
}
/** Preserve the supplied alias ordering; the encrypted/single configured key remains first. */
public List<String> providerKeys(String fallbackKey, String... aliases) {
var keys = new LinkedHashSet<String>();
if (fallbackKey != null && !fallbackKey.isBlank()) keys.add(fallbackKey.trim());
for (String alias : aliases) {
if (alias == null || alias.isBlank()) continue;
// Keep the existing provider alias naming convention for deployed environment variables.
keys.addAll(environmentKeys("MOYI_UPSTREAM_API_KEYS_" + alias.toUpperCase(Locale.ROOT).replace('-', '_')));
}
return List.copyOf(keys);
}
/** Explicit property names are used only for provider-specific official-key fallbacks. */
public List<String> environmentKeys(String propertyName) {
return parseKeys(environment == null ? null : environment.getProperty(propertyName));
}
/** A route override wins over a provider+model override. Neither is combined with channel keys. */
public List<String> scopedKeys(String providerCode, String routeId, String upstreamModel) {
if (environment == null) return List.of();
if (routeId != null && !routeId.isBlank()) {
String value = environment.getProperty("MOYI_UPSTREAM_API_KEYS_ROUTE_" + normalize(routeId));
if (value != null && !value.isBlank()) return parseKeys(value);
}
if (upstreamModel != null && !upstreamModel.isBlank()) {
return environmentKeys("MOYI_UPSTREAM_API_KEYS_MODEL_" + normalize(providerCode) + "_" + normalize(upstreamModel));
}
return List.of();
}
/** Stable order, no blank entries or duplicates; immutable so callers cannot corrupt a pool. */
public static List<String> parseKeys(String value) {
if (value == null || value.isBlank()) return List.of();
var keys = new LinkedHashSet<String>();
for (String key : value.split("\\|")) if (!key.isBlank()) keys.add(key.trim());
return List.copyOf(keys);
}
public static String nextKey(List<String> keys, AtomicInteger cursor) {
if (keys.isEmpty()) throw new ApiException("UPSTREAM_KEY_NOT_AVAILABLE", "渠道没有可用的上游凭据", 503);
return keys.get(Math.floorMod(cursor.getAndIncrement(), keys.size()));
}
private static String normalize(String value) {
return (value == null ? "" : value).toUpperCase(Locale.ROOT).replaceAll("[^A-Z0-9]", "_");
}
}
@@ -0,0 +1,59 @@
package com.moyi.controlplane.provider.media;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.provider.transport.BoundedHttpTransport;
import com.moyi.controlplane.provider.transport.PinnedHttpsTransport;
import java.time.Duration;
import java.util.Locale;
/** Public reference-image input only. No provider credentials, routing, retries or billing. */
public final class MediaDownloader {
private static final int MAX_BYTES = 8 * 1024 * 1024;
private final PublicMediaAddressPolicy policy = new PublicMediaAddressPolicy();
private final PinnedHttpsTransport transport;
private final PinnedHttpsTransport.AddressResolver resolver;
public MediaDownloader() { this(new PinnedHttpsTransport(), new PublicMediaAddressPolicy()::resolve); }
MediaDownloader(PinnedHttpsTransport transport, PinnedHttpsTransport.AddressResolver resolver) {
this.transport = transport;
this.resolver = resolver;
}
public record MediaBytes(byte[] bytes, String contentType, String filename) {
@Override public String toString() { return "MediaBytes[type=" + contentType + ", bytes=" + bytes.length + "]"; }
}
public MediaBytes download(String value) {
java.net.URI uri;
try { uri = policy.requireUri(value); }
catch (IllegalArgumentException error) { throw invalidAddress(); }
BoundedHttpTransport.Response response;
try { response = transport.get(uri, resolver, Duration.ofSeconds(15), MAX_BYTES); }
catch (BoundedHttpTransport.Failure failure) {
if (failure.reason() == BoundedHttpTransport.Reason.INVALID_ADDRESS) throw invalidAddress();
if (failure.reason() == BoundedHttpTransport.Reason.BODY_TOO_LARGE) throw new ApiException("MEDIA_TOO_LARGE", "媒体输入超过 8 MiB", 400);
throw new ApiException("MEDIA_FETCH_FAILED", "无法在期限内读取媒体输入", 400);
}
if (response.statusCode() < 200 || response.statusCode() >= 300) {
throw new ApiException("MEDIA_FETCH_FAILED", "无法读取媒体输入(HTTP " + response.statusCode() + ")", 400);
}
if (response.body().length == 0) throw new ApiException("MEDIA_TOO_LARGE", "媒体输入为空或超过 8 MiB", 400);
String contentType = response.headers().firstValue("Content-Type").orElse("application/octet-stream")
.split(";", 2)[0].trim().toLowerCase(Locale.ROOT);
if (!contentType.matches("image/[a-z0-9.+-]{1,64}")) {
throw new ApiException("MEDIA_TYPE_UNSUPPORTED", "图像编辑和视频参考图必须是 image/* 文件", 400);
}
String extension = switch (contentType) {
case "image/jpeg" -> ".jpg";
case "image/png" -> ".png";
case "image/webp" -> ".webp";
case "image/gif" -> ".gif";
default -> ".bin";
};
return new MediaBytes(response.body(), contentType, "input" + extension);
}
private ApiException invalidAddress() {
return new ApiException("INVALID_MEDIA_URL", "媒体地址必须是公开 HTTPS 地址,不能指向本机、内网或特殊用途地址", 400);
}
}
@@ -0,0 +1,53 @@
package com.moyi.controlplane.provider.media;
import com.moyi.controlplane.provider.transport.PinnedHttpsTransport;
import java.io.IOException;
import java.net.InetAddress;
import java.net.URI;
/** Fail closed for non-global and special-purpose targets, including mixed DNS answer sets. */
final class PublicMediaAddressPolicy {
URI requireUri(String value) {
try {
URI uri = URI.create(value);
if (!"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null || uri.getHost().contains("%")
|| uri.getUserInfo() != null || uri.getRawFragment() != null || uri.getPort() == 0 || uri.getPort() > 65535) {
throw new IllegalArgumentException();
}
return uri;
} catch (RuntimeException error) { throw new IllegalArgumentException("Invalid public HTTPS media URL"); }
}
InetAddress[] resolve(String host) throws IOException {
return requirePublic(InetAddress.getAllByName(host));
}
InetAddress[] requirePublic(InetAddress[] addresses) throws PinnedHttpsTransport.AddressRejectedException {
if (addresses == null || addresses.length == 0 || addresses.length > 16) throw new PinnedHttpsTransport.AddressRejectedException();
for (var address : addresses) if (address == null || !isPublic(address)) throw new PinnedHttpsTransport.AddressRejectedException();
return addresses.clone();
}
boolean isPublic(InetAddress address) {
byte[] raw = address.getAddress();
int a = Byte.toUnsignedInt(raw[0]), b = Byte.toUnsignedInt(raw[1]);
if (raw.length == 4) {
int c = Byte.toUnsignedInt(raw[2]), d = Byte.toUnsignedInt(raw[3]);
return !(a == 0 || a == 10 || a == 127 || a >= 224
|| a == 100 && b >= 64 && b <= 127
|| a == 169 && b == 254 || a == 172 && b >= 16 && b <= 31
|| a == 192 && (b == 168 || b == 0 && (c == 0 || c == 2) || b == 88 && c == 99)
|| a == 198 && (b == 18 || b == 19 || b == 51 && c == 100)
|| a == 203 && b == 0 && c == 113
|| a == 168 && b == 63 && c == 129 && d == 16); // Azure platform virtual address
}
// Only native global unicast 2000::/3, excluding special/tunnel/documentation prefixes.
// This also rejects ULA, link/site local, multicast, compatible/mapped IPv4 and well-known NAT64.
if (raw.length != 16 || (a & 0xe0) != 0x20) return false;
int c = Byte.toUnsignedInt(raw[2]), d = Byte.toUnsignedInt(raw[3]);
return !(a == 0x20 && b == 0x01 && (c <= 1 || c == 0x0d && d == 0xb8)
|| a == 0x20 && b == 0x02 // 6to4 can encode a private IPv4 target
|| a == 0x3f && b == 0xfe // retired 6bone allocation
|| a == 0x3f && b == 0xff && (c & 0xf0) == 0); // 3fff::/20 documentation
}
}
@@ -0,0 +1,114 @@
package com.moyi.controlplane.provider.transport;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.http.HttpClient;
import java.net.http.HttpHeaders;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.ByteBuffer;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Flow;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
/** One bounded HTTP attempt. No credentials, route selection, protocol mapping, retry, risk or billing policy. */
public final class BoundedHttpTransport {
private static final int ERROR_BODY_LIMIT = 64 * 1024;
private final HttpClient client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10))
.followRedirects(HttpClient.Redirect.NEVER).build();
public record Response(int statusCode, HttpHeaders headers, byte[] body) {
@Override public String toString() { return "TransportResponse[status=" + statusCode + ", bytes=" + body.length + "]"; }
}
public enum Reason { TIMEOUT, INTERRUPTED, BODY_TOO_LARGE, IO, INVALID_ADDRESS, BUSY }
public static final class Failure extends IOException {
private final Reason reason;
private final Integer observedStatus;
Failure(Reason reason, Integer status) {
super("Bounded HTTP attempt failed: " + reason);
this.reason = reason; this.observedStatus = status;
}
public Reason reason() { return reason; }
public Integer observedStatus() { return observedStatus; }
}
public Response send(HttpRequest request, int maxResponseBytes) throws Failure {
if (maxResponseBytes < 1 || maxResponseBytes > 32 * 1024 * 1024) throw new IllegalArgumentException("Invalid response limit");
Duration timeout = request.timeout().orElseThrow(() -> new IllegalArgumentException("An explicit total HTTP deadline is required"));
var status = new AtomicInteger();
var subscriber = new AtomicReference<LimitedBody>();
CompletableFuture<HttpResponse<byte[]>> attempt = client.sendAsync(request, info -> {
status.set(info.statusCode());
int limit = info.statusCode() >= 200 && info.statusCode() < 300 ? maxResponseBytes : Math.min(maxResponseBytes, ERROR_BODY_LIMIT);
var body = new LimitedBody(limit, info.statusCode(), info.headers().firstValueAsLong("Content-Length").orElse(-1));
subscriber.set(body);
return body;
});
try {
// Unlike a header-only timeout, this also bounds a server that drips or never finishes the body.
HttpResponse<byte[]> response = attempt.get(Math.max(1, timeout.toMillis()), TimeUnit.MILLISECONDS);
return new Response(response.statusCode(), response.headers(), response.body());
} catch (TimeoutException exception) {
throw cancel(attempt, subscriber.get(), Reason.TIMEOUT, status.get());
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw cancel(attempt, subscriber.get(), Reason.INTERRUPTED, status.get());
} catch (ExecutionException exception) {
Throwable cause = exception.getCause();
while (cause instanceof java.util.concurrent.CompletionException && cause.getCause() != null) cause = cause.getCause();
if (cause instanceof Failure failure) throw failure;
throw cancel(attempt, subscriber.get(), cause instanceof java.net.http.HttpTimeoutException ? Reason.TIMEOUT : Reason.IO, status.get());
}
}
private Failure cancel(CompletableFuture<?> attempt, LimitedBody body, Reason reason, int status) {
Failure failure = new Failure(reason, status == 0 ? null : status);
if (body != null) body.cancel(failure);
attempt.cancel(true);
return failure;
}
private static final class LimitedBody implements HttpResponse.BodySubscriber<byte[]> {
private final CompletableFuture<byte[]> completed = new CompletableFuture<>();
private final ByteArrayOutputStream bytes = new ByteArrayOutputStream();
private final int limit, status;
private final long declaredLength;
private Flow.Subscription subscription;
private int received;
LimitedBody(int limit, int status, long declaredLength) { this.limit = limit; this.status = status; this.declaredLength = declaredLength; }
@Override public CompletionStage<byte[]> getBody() { return completed; }
@Override public void onSubscribe(Flow.Subscription subscription) {
this.subscription = subscription;
if (completed.isDone()) { subscription.cancel(); return; }
if (declaredLength > limit) { cancel(new Failure(Reason.BODY_TOO_LARGE, status)); return; }
subscription.request(1);
}
@Override public void onNext(List<ByteBuffer> buffers) {
if (completed.isDone()) { if (subscription != null) subscription.cancel(); return; }
long chunkBytes = 0;
for (ByteBuffer buffer : buffers) chunkBytes += buffer.remaining();
if (chunkBytes > limit - received) { cancel(new Failure(Reason.BODY_TOO_LARGE, status)); return; }
received += (int) chunkBytes;
byte[] part = new byte[8192];
for (ByteBuffer buffer : buffers) while (buffer.hasRemaining()) {
int count = Math.min(part.length, buffer.remaining());
buffer.get(part, 0, count); bytes.write(part, 0, count);
}
subscription.request(1);
}
@Override public void onError(Throwable error) { completed.completeExceptionally(new Failure(Reason.IO, status)); }
@Override public void onComplete() { completed.complete(bytes.toByteArray()); }
void cancel(Failure failure) {
completed.completeExceptionally(failure);
if (subscription != null) subscription.cancel();
}
}
}
@@ -0,0 +1,187 @@
package com.moyi.controlplane.provider.transport;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.net.InetAddress;
import java.net.Proxy;
import java.net.Socket;
import java.net.URI;
import java.net.UnknownHostException;
import java.net.http.HttpHeaders;
import java.time.Duration;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
import java.util.concurrent.RejectedExecutionException;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicInteger;
import javax.net.ssl.SSLContext;
import org.apache.hc.client5.http.DnsResolver;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.config.ConnectionConfig;
import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.client5.http.config.TlsConfig;
import org.apache.hc.client5.http.impl.DefaultSchemePortResolver;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.BasicHttpClientConnectionManager;
import org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator;
import org.apache.hc.client5.http.impl.io.ManagedHttpClientConnectionFactory;
import org.apache.hc.client5.http.impl.routing.DefaultRoutePlanner;
import org.apache.hc.client5.http.ssl.DefaultClientTlsStrategy;
import org.apache.hc.client5.http.ssl.DefaultHostnameVerifier;
import org.apache.hc.client5.http.ssl.HostnameVerificationPolicy;
import org.apache.hc.core5.http.config.Http1Config;
import org.apache.hc.core5.io.CloseMode;
import org.apache.hc.core5.ssl.SSLContexts;
import org.apache.hc.core5.util.Timeout;
import static com.moyi.controlplane.provider.transport.BoundedHttpTransport.Reason;
/**
* A credential-free HTTPS GET with per-attempt DNS binding. The caller supplies address policy;
* this module owns only bounded transport. Never rewrite the URI to an IP: TLS/SNI and Host use
* the original authority. No proxy, cookies, redirects, retry or connection reuse across attempts.
*/
public final class PinnedHttpsTransport {
private static final ExecutorService WORKERS = new ThreadPoolExecutor(0, 8, 60, TimeUnit.SECONDS,
new SynchronousQueue<>(), task -> {
var thread = new Thread(task, "moyi-media-fetch");
thread.setDaemon(true);
return thread;
}, new ThreadPoolExecutor.AbortPolicy());
private final SSLContext sslContext;
private final ExecutorService workers;
public PinnedHttpsTransport() { this(SSLContexts.createDefault(), WORKERS); }
PinnedHttpsTransport(SSLContext sslContext, ExecutorService workers) {
this.sslContext = sslContext;
this.workers = workers;
}
@FunctionalInterface
public interface AddressResolver { InetAddress[] resolve(String host) throws IOException; }
public static final class AddressRejectedException extends IOException {
public AddressRejectedException() { super("Target address rejected"); }
}
public BoundedHttpTransport.Response get(URI uri, AddressResolver resolver, Duration deadline, int maxBytes)
throws BoundedHttpTransport.Failure {
if (uri == null || !"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null
|| uri.getUserInfo() != null || uri.getRawFragment() != null || uri.getHost().contains("%")
|| uri.getPort() == 0 || uri.getPort() > 65535) throw new IllegalArgumentException("Invalid HTTPS target");
if (maxBytes < 1 || maxBytes > 32 * 1024 * 1024 || deadline == null || deadline.isNegative()
|| deadline.isZero() || deadline.compareTo(Duration.ofMinutes(1)) > 0) throw new IllegalArgumentException("Invalid transport limits");
var request = new HttpGet(uri);
var observed = new AtomicInteger();
Future<BoundedHttpTransport.Response> attempt;
try {
attempt = workers.submit(() -> execute(uri, resolver, request, deadline, maxBytes, observed));
} catch (RejectedExecutionException error) {
throw failure(Reason.BUSY, observed);
}
try {
return attempt.get(Math.max(1, deadline.toMillis()), TimeUnit.MILLISECONDS);
} catch (TimeoutException error) {
cancel(request, attempt);
throw failure(Reason.TIMEOUT, observed);
} catch (InterruptedException error) {
cancel(request, attempt);
Thread.currentThread().interrupt();
throw failure(Reason.INTERRUPTED, observed);
} catch (ExecutionException error) {
cancel(request, attempt);
if (error.getCause() instanceof BoundedHttpTransport.Failure safe) throw safe;
if (error.getCause() instanceof AddressRejectedException) throw failure(Reason.INVALID_ADDRESS, observed);
if (error.getCause() instanceof java.net.SocketTimeoutException) throw failure(Reason.TIMEOUT, observed);
// Do not attach causes: TLS/DNS/parser exceptions can contain signed URLs or response data.
throw failure(Reason.IO, observed);
}
}
private BoundedHttpTransport.Response execute(URI uri, AddressResolver resolver, HttpGet request,
Duration deadline, int maxBytes, AtomicInteger observed) throws IOException {
String host = unbracket(uri.getHost());
InetAddress[] addresses = resolver.resolve(host);
if (addresses == null || addresses.length == 0 || addresses.length > 16
|| java.util.Arrays.stream(addresses).anyMatch(java.util.Objects::isNull)) throw new AddressRejectedException();
// DNS lookup may ignore interruption. Once timed out, it must never resume with a late HTTP GET.
if (request.isCancelled() || Thread.currentThread().isInterrupted()) throw failure(Reason.INTERRUPTED, observed);
InetAddress[] pinned = addresses.clone();
var timeout = Timeout.ofMilliseconds(Math.max(1, deadline.toMillis()));
var boundDns = new DnsResolver() {
@Override public InetAddress[] resolve(String target) throws UnknownHostException {
if (!host.equalsIgnoreCase(unbracket(target))) throw new UnknownHostException("Unexpected target");
return pinned.clone();
}
@Override public String resolveCanonicalHostname(String target) throws UnknownHostException {
if (!host.equalsIgnoreCase(unbracket(target))) throw new UnknownHostException("Unexpected target");
return host;
}
};
var tls = new DefaultClientTlsStrategy(sslContext, HostnameVerificationPolicy.BOTH, new DefaultHostnameVerifier());
// A direct HTTP route alone is insufficient: new Socket() can still use the JVM's SOCKS selector.
var operator = new DefaultHttpClientConnectionOperator(ignored -> new Socket(Proxy.NO_PROXY),
DefaultSchemePortResolver.INSTANCE, boundDns, scheme -> "https".equalsIgnoreCase(scheme) ? tls : null);
var manager = new BasicHttpClientConnectionManager(operator, ManagedHttpClientConnectionFactory.builder()
.http1Config(Http1Config.custom().setMaxLineLength(8192).setMaxHeaderCount(64).build()).build());
manager.setConnectionConfig(ConnectionConfig.custom().setConnectTimeout(timeout).setSocketTimeout(timeout).build());
manager.setTlsConfig(TlsConfig.custom().setHandshakeTimeout(timeout).build());
request.setConfig(RequestConfig.custom().setResponseTimeout(timeout).setConnectionRequestTimeout(timeout)
.setHardCancellationEnabled(true).setAuthenticationEnabled(false).setRedirectsEnabled(false).build());
request.setHeader("Accept", "image/*");
request.setHeader("Accept-Encoding", "identity");
request.setHeader("Connection", "close");
try (var client = HttpClients.custom().setConnectionManager(manager)
.setRoutePlanner(new DefaultRoutePlanner(DefaultSchemePortResolver.INSTANCE))
.disableAutomaticRetries().disableRedirectHandling().disableCookieManagement()
.disableAuthCaching().disableContentCompression().disableDefaultUserAgent().build()) {
try {
var response = client.executeOpen(null, request, null);
try {
observed.set(response.getCode());
var entity = response.getEntity();
int limit = response.getCode() >= 200 && response.getCode() < 300 ? maxBytes : Math.min(maxBytes, 64 * 1024);
if (entity != null && entity.getContentLength() > limit) throw failure(Reason.BODY_TOO_LARGE, observed);
// Compressed images are already encoded; do not accept a decompression bomb or compressed MIME wrapper.
if (entity != null && entity.getContentEncoding() != null
&& !"identity".equalsIgnoreCase(entity.getContentEncoding())) throw failure(Reason.IO, observed);
var body = new ByteArrayOutputStream();
if (entity != null) {
var input = entity.getContent();
byte[] chunk = new byte[8192];
int count;
while ((count = input.read(chunk)) != -1) {
if (request.isCancelled() || Thread.currentThread().isInterrupted()) throw failure(Reason.INTERRUPTED, observed);
if (count > limit - body.size()) throw failure(Reason.BODY_TOO_LARGE, observed);
body.write(chunk, 0, count);
}
}
var headers = new LinkedHashMap<String, List<String>>();
for (var header : response.getHeaders()) headers.computeIfAbsent(header.getName(), ignored -> new ArrayList<>()).add(header.getValue());
return new BoundedHttpTransport.Response(response.getCode(), HttpHeaders.of(headers, (key, value) -> true), body.toByteArray());
} finally {
// Cancel BEFORE closing the entity: close/consume must not drain an oversized/never-ending body.
request.cancel();
try { response.close(); }
catch (IOException expectedAfterCancellation) { /* Never replace the primary outcome with a closed-socket cleanup error. */ }
}
} finally {
request.cancel();
client.close(CloseMode.IMMEDIATE);
}
}
}
private static String unbracket(String host) {
return host.startsWith("[") && host.endsWith("]") ? host.substring(1, host.length() - 1) : host;
}
private static void cancel(HttpGet request, Future<?> attempt) { request.cancel(); attempt.cancel(true); }
private static BoundedHttpTransport.Failure failure(Reason reason, AtomicInteger observed) {
return new BoundedHttpTransport.Failure(reason, observed.get() == 0 ? null : observed.get());
}
}
@@ -0,0 +1,44 @@
package com.moyi.controlplane.provider.transport;
import com.moyi.controlplane.common.ApiException;
/** HTTP boundary facts, not a retry policy. Unknown acceptance must never authorize replay/refund. */
public final class ProviderRequestException extends ApiException {
public enum Acceptance { NOT_SENT, REJECTED, UNKNOWN }
private final Acceptance acceptance;
private final Integer upstreamStatus;
private ProviderRequestException(String code, String message, int status, Acceptance acceptance, Integer upstreamStatus) {
super(code, message, status);
this.acceptance = acceptance;
this.upstreamStatus = upstreamStatus;
}
public static ProviderRequestException notSent(String code, String message, int status) {
return new ProviderRequestException(code, message, status, Acceptance.NOT_SENT, null);
}
public static ProviderRequestException unknown(String code, String message) {
return new ProviderRequestException(code, message, 502, Acceptance.UNKNOWN, null);
}
public static ProviderRequestException http(int status, String message) {
String code = switch (status) {
case 401, 403 -> "UPSTREAM_AUTH_FAILED";
case 402 -> "UPSTREAM_CREDIT_INSUFFICIENT";
case 429 -> "UPSTREAM_RATE_LIMITED";
default -> "UPSTREAM_PROVIDER_ERROR";
};
// Timeout/conflict/5xx responses do not prove a generation was never accepted.
Acceptance acceptance = switch (status) {
case 400, 401, 402, 403, 404, 405, 413, 415, 422, 429 -> Acceptance.REJECTED;
default -> Acceptance.UNKNOWN;
};
return new ProviderRequestException(code, message + "(HTTP " + status + ")", 502, acceptance, status);
}
public Acceptance acceptance() { return acceptance; }
public Integer upstreamStatus() { return upstreamStatus; }
public boolean definitelyNotAccepted() { return acceptance != Acceptance.UNKNOWN; }
}
@@ -0,0 +1,101 @@
package com.moyi.controlplane.provider.transport;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import java.io.BufferedReader;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.CodingErrorAction;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.function.BooleanSupplier;
/** Bounded SSE/JSON framing only; no routing, retry, provider credentials, metering or billing. */
public final class SseEventReader {
private static final int MAX_RESPONSE_BYTES = 4 * 1024 * 1024;
private static final int MAX_FRAME_CHARS = 262_144;
private final ObjectMapper json;
public SseEventReader(ObjectMapper json) {
this.json = json;
}
/** The transport owns and closes the stream; the supplied deadline covers blocked reads. */
public void read(InputStream body, BooleanSupplier expired, boolean requireDone, EventConsumer consumer) throws IOException {
var decoder = StandardCharsets.UTF_8.newDecoder()
.onMalformedInput(CodingErrorAction.REPORT).onUnmappableCharacter(CodingErrorAction.REPORT);
var reader = new BufferedReader(new InputStreamReader(new BoundedBody(body), decoder));
var event = new StringBuilder();
var line = new StringBuilder();
try {
int c;
while ((c = reader.read()) != -1) {
if (expired.getAsBoolean()) throw timeout();
if (c == '\n') {
String text = line.toString();
line.setLength(0);
if (text.endsWith("\r")) text = text.substring(0, text.length() - 1);
if (text.isEmpty()) {
if (!event.isEmpty()) {
String data = event.substring(0, event.length() - 1);
event.setLength(0);
if ("[DONE]".equals(data)) return;
Map<String,Object> chunk = json.readValue(data, new TypeReference<>() {});
if (chunk == null) throw new ApiException("UPSTREAM_STREAM_INVALID", "上游流式事件格式无效", 502);
consumer.accept(chunk);
}
} else if (text.startsWith("data:")) {
if ((long) event.length() + text.length() - 5 + 1 > MAX_FRAME_CHARS) throw tooLarge();
event.append(text.substring(5).stripLeading()).append('\n');
}
} else {
if (line.length() >= MAX_FRAME_CHARS) throw tooLarge();
line.append((char) c);
}
}
} catch (IOException exception) {
if (expired.getAsBoolean()) throw timeout();
throw exception;
}
if (expired.getAsBoolean()) throw timeout();
if (!requireDone && event.isEmpty() && line.isEmpty()) return;
throw new ApiException("UPSTREAM_STREAM_INCOMPLETE", "上游流式响应中断,本次未扣费", 502);
}
private static ApiException tooLarge() {
return new ApiException("UPSTREAM_RESPONSE_TOO_LARGE", "上游响应超过限制", 502);
}
private static ApiException timeout() {
return new ApiException("UPSTREAM_PROVIDER_TIMEOUT", "上游响应超时,本次未扣费", 504);
}
private static final class BoundedBody extends FilterInputStream {
private long bytes;
private BoundedBody(InputStream body) { super(body); }
@Override
public int read() throws IOException {
int value = in.read();
if (value != -1 && ++bytes > MAX_RESPONSE_BYTES) throw tooLarge();
return value;
}
@Override
public int read(byte[] buffer, int offset, int length) throws IOException {
if (length == 0) return 0;
int count = in.read(buffer, offset, (int) Math.min(length, MAX_RESPONSE_BYTES - bytes + 1));
if (count > 0 && (bytes += count) > MAX_RESPONSE_BYTES) throw tooLarge();
return count;
}
}
@FunctionalInterface
public interface EventConsumer {
void accept(Map<String,Object> event) throws IOException;
}
}
@@ -40,6 +40,9 @@ public class SecretCipher {
}
}
/** Readiness only; never expose key material. */
public boolean isConfigured() { return key != null; }
public String decrypt(String encoded) {
if (key == null) throw new ApiException("CREDENTIAL_ENCRYPTION_NOT_CONFIGURED", "请先配置供应商凭证加密密钥", 500);
try {
@@ -5,11 +5,13 @@ import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.security.SecretCipher;
import com.moyi.controlplane.provider.credentials.UpstreamCredentialResolver;
import com.moyi.controlplane.provider.transport.ProviderRequestException;
import com.moyi.controlplane.provider.transport.BoundedHttpTransport;
import com.moyi.controlplane.provider.media.MediaDownloader;
import com.moyi.controlplane.provider.media.MediaDownloader.MediaBytes;
import java.net.URI;
import java.net.InetAddress;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
@@ -40,7 +42,8 @@ public class AsyncProviderClient {
/** Keep rotation state across submit/poll calls; rebuilding a Provider on
* every request otherwise selected the first upstream key forever. */
private final ConcurrentMap<String, AtomicInteger> keyCursors = new ConcurrentHashMap<>();
private final HttpClient client = HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();
private final BoundedHttpTransport transport = new BoundedHttpTransport();
private final MediaDownloader mediaDownloader = new MediaDownloader();
public AsyncProviderClient(JdbcTemplate jdbc, SecretCipher cipher, ObjectMapper mapper, Environment environment) {
this.jdbc = jdbc; this.cipher = cipher; this.mapper = mapper; this.environment = environment;
@@ -59,6 +62,22 @@ public class AsyncProviderClient {
}
public Map<String,Object> submit(String providerId, String upstreamModel, String routeId, String taskType, Map<String,Object> input) {
try {
return submitConfigured(providerId, upstreamModel, routeId, taskType, input);
} catch (ProviderRequestException exception) {
throw exception;
} catch (ApiException exception) {
// Only these local validation/configuration errors are known to precede generation submission.
if (List.of("PROVIDER_NOT_CONFIGURED", "UPSTREAM_KEY_NOT_AVAILABLE", "ASYNC_PROVIDER_NOT_IMPLEMENTED",
"MEDIA_INPUT_UNSUPPORTED", "INVALID_MEDIA_URL", "MEDIA_FETCH_FAILED", "MEDIA_TOO_LARGE",
"MEDIA_TYPE_UNSUPPORTED", "INVALID_TASK_INPUT").contains(exception.getCode())) {
throw ProviderRequestException.notSent(exception.getCode(), exception.getMessage(), exception.getStatus());
}
throw ProviderRequestException.unknown("UPSTREAM_PROVIDER_ERROR", "上游任务提交结果尚未确认");
}
}
private Map<String,Object> submitConfigured(String providerId, String upstreamModel, String routeId, String taskType, Map<String,Object> input) {
if (input == null) input = Map.of();
String prompt = String.valueOf(input.getOrDefault("prompt", ""));
Provider provider = provider(providerId, routeId, upstreamModel);
@@ -92,6 +111,17 @@ public class AsyncProviderClient {
throw new ApiException("ASYNC_PROVIDER_NOT_IMPLEMENTED", "该异步渠道没有可用的执行器", 501);
}
/** Validate a proven polling binding without sending any HTTP request or exposing credential values. */
public void requirePollableBinding(String providerId, String routeId, String upstreamModel, String taskType, String fingerprint) {
if (fingerprint == null || !fingerprint.matches("[0-9a-f]{64}")) throw new ApiException("INVALID_CREDENTIAL_FINGERPRINT", "需要有效的上游账号指纹", 422);
Provider provider = provider(providerId, routeId, upstreamModel);
String code = provider.code().toLowerCase(java.util.Locale.ROOT);
boolean pollable = isTripo(provider.code(), provider.baseUrl()) ? "3d".equals(taskType)
: (code.contains("byte") || code.contains("seedance") || provider.baseUrl().contains("volces.com") || "openai-compatible".equalsIgnoreCase(provider.protocol())) && "video".equals(taskType);
if (!pollable) throw new ApiException("TASK_NOT_POLLABLE", "该执行器没有可恢复的只读任务查询接口,请核实终态", 409);
provider.keyForFingerprint(routeId, upstreamModel, environment, keyCursors, fingerprint);
}
public Map<String,Object> poll(String providerId, String providerTaskId) {
return poll(providerId, null, providerTaskId);
}
@@ -193,7 +223,7 @@ public class AsyncProviderClient {
String media=mediaUrl(input);
if(media!=null) {
try {
MediaBytes file=downloadMedia(media);
MediaBytes file=mediaDownloader.download(media);
parts.add(Map.of("inlineData",Map.of("mimeType",file.contentType(),"data",Base64.getEncoder().encodeToString(file.bytes()))));
} catch(ApiException e) { throw e; }
catch(Exception e) { throw new ApiException("MEDIA_FETCH_FAILED","无法读取参考图",400); }
@@ -260,8 +290,8 @@ public class AsyncProviderClient {
}
private boolean isNotFound(ApiException exception) {
String message = exception.getMessage();
return message != null && (message.contains("HTTP 404") || message.contains("HTTP 405"));
return exception instanceof ProviderRequestException failure
&& (Integer.valueOf(404).equals(failure.upstreamStatus()) || Integer.valueOf(405).equals(failure.upstreamStatus()));
}
/** Normalize OpenAI image/audio response fields to the task result shape. */
@@ -286,19 +316,14 @@ public class AsyncProviderClient {
}
private Map<String,Object> requestAudio(String method, String url, String key, Map<String,Object> body, String message) {
boolean sending = false;
try {
HttpRequest.Builder builder = HttpRequest.newBuilder().uri(URI.create(url)).timeout(UPSTREAM_REQUEST_TIMEOUT)
.header("Authorization", "Bearer " + key).header("Content-Type", "application/json");
HttpResponse<byte[]> response = client.send(builder.POST(HttpRequest.BodyPublishers.ofString(write(body))).build(), HttpResponse.BodyHandlers.ofByteArray());
if (response.statusCode() < 200 || response.statusCode() >= 300) {
String code = switch (response.statusCode()) {
case 401, 403 -> "UPSTREAM_AUTH_FAILED";
case 402 -> "UPSTREAM_CREDIT_INSUFFICIENT";
case 429 -> "UPSTREAM_RATE_LIMITED";
default -> "UPSTREAM_PROVIDER_ERROR";
};
throw new ApiException(code, message + "(HTTP " + response.statusCode() + ")", 502);
}
HttpRequest request = builder.POST(HttpRequest.BodyPublishers.ofString(write(body))).build();
sending = true;
var response = transport.send(request, 8 * 1024 * 1024);
if (response.statusCode() < 200 || response.statusCode() >= 300) throw upstreamStatus(response.statusCode(), message);
byte[] bytes = response.body();
if (bytes.length == 0 || bytes.length > 8 * 1024 * 1024) throw new ApiException("UPSTREAM_MEDIA_TOO_LARGE", message + "(返回音频超过 8 MiB 或为空)", 502);
String contentType = response.headers().firstValue("Content-Type").orElse("audio/mpeg").split(";", 2)[0].trim().toLowerCase(java.util.Locale.ROOT);
@@ -308,8 +333,7 @@ public class AsyncProviderClient {
result.put("mimeType", contentType);
result.put("format", String.valueOf(body.getOrDefault("response_format", "mp3")));
return result;
} catch (ApiException exception) { throw exception; }
catch (Exception exception) { throw new ApiException("UPSTREAM_PROVIDER_ERROR", message, 502); }
} catch (Exception exception) { throw requestFailure(exception, sending, message); }
}
/**
@@ -321,6 +345,7 @@ public class AsyncProviderClient {
*/
private Map<String,Object> requestMultipart(String method, String url, String key, Map<String,Object> fields,
String fileField, String media, String message) {
boolean sending = false;
try {
String boundary = "----moyi-" + java.util.UUID.randomUUID();
ByteArrayOutputStream out = new ByteArrayOutputStream();
@@ -328,7 +353,7 @@ public class AsyncProviderClient {
multipartField(out, boundary, entry.getKey(), String.valueOf(entry.getValue()));
}
if (fileField != null && media != null) {
MediaBytes file = downloadMedia(media);
MediaBytes file = mediaDownloader.download(media);
multipartFile(out, boundary, fileField, file.filename(), file.contentType(), file.bytes());
}
out.write(("--" + boundary + "--\r\n").getBytes(java.nio.charset.StandardCharsets.UTF_8));
@@ -336,7 +361,8 @@ public class AsyncProviderClient {
.header("Authorization", "Bearer " + key)
.header("Content-Type", "multipart/form-data; boundary=" + boundary)
.POST(HttpRequest.BodyPublishers.ofByteArray(out.toByteArray())).build();
HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
sending = true;
var response = transport.send(request, 8 * 1024 * 1024);
if (response.statusCode() < 200 || response.statusCode() >= 300) throw upstreamStatus(response.statusCode(), message);
Map<String,Object> parsed = mapper.readValue(response.body(), new TypeReference<>() {});
Object code = parsed.get("code");
@@ -344,18 +370,23 @@ public class AsyncProviderClient {
throw new ApiException("UPSTREAM_PROVIDER_ERROR", message, 502);
}
return parsed;
} catch (ApiException exception) { throw exception; }
catch (Exception exception) { throw new ApiException("UPSTREAM_PROVIDER_ERROR", message, 502); }
} catch (Exception exception) { throw requestFailure(exception, sending, message); }
}
private ApiException upstreamStatus(int status, String message) {
String code = switch (status) {
case 401, 403 -> "UPSTREAM_AUTH_FAILED";
case 402 -> "UPSTREAM_CREDIT_INSUFFICIENT";
case 429 -> "UPSTREAM_RATE_LIMITED";
default -> "UPSTREAM_PROVIDER_ERROR";
};
return new ApiException(code, message + "(HTTP " + status + ")", 502);
return ProviderRequestException.http(status, message);
}
private ProviderRequestException requestFailure(Exception exception, boolean sending, String message) {
if (exception instanceof InterruptedException) Thread.currentThread().interrupt();
if (exception instanceof ProviderRequestException failure) return failure;
if (exception instanceof BoundedHttpTransport.Failure failure && failure.observedStatus() != null
&& (failure.observedStatus() < 200 || failure.observedStatus() >= 300)) {
return ProviderRequestException.http(failure.observedStatus(), message);
}
if (sending) return ProviderRequestException.unknown("UPSTREAM_PROVIDER_ERROR", message);
if (exception instanceof ApiException error) return ProviderRequestException.notSent(error.getCode(), error.getMessage(), error.getStatus());
return ProviderRequestException.notSent("UPSTREAM_REQUEST_NOT_SENT", "请求准备失败,尚未提交上游生成", 502);
}
private void multipartField(ByteArrayOutputStream out, String boundary, String name, String value) throws IOException {
@@ -371,63 +402,17 @@ public class AsyncProviderClient {
out.write("\r\n".getBytes(java.nio.charset.StandardCharsets.UTF_8));
}
private MediaBytes downloadMedia(String value) throws IOException, InterruptedException {
URI uri = URI.create(value);
if (!"https".equalsIgnoreCase(uri.getScheme()) || uri.getHost() == null || uri.getUserInfo() != null) {
throw new ApiException("INVALID_MEDIA_URL", "媒体地址必须是可公开访问的 HTTPS URL", 400);
}
for (InetAddress address : InetAddress.getAllByName(uri.getHost())) {
if (address.isAnyLocalAddress() || address.isLoopbackAddress() || address.isLinkLocalAddress()
|| address.isSiteLocalAddress() || address.isMulticastAddress()) {
throw new ApiException("INVALID_MEDIA_URL", "媒体地址不能指向内网或本机地址", 400);
}
}
HttpRequest request = HttpRequest.newBuilder(uri).timeout(Duration.ofSeconds(15)).GET().build();
HttpResponse<byte[]> response = client.send(request, HttpResponse.BodyHandlers.ofByteArray());
if (response.statusCode() < 200 || response.statusCode() >= 300) {
throw new ApiException("MEDIA_FETCH_FAILED", "无法读取媒体输入(HTTP " + response.statusCode() + ")", 400);
}
byte[] bytes = response.body();
if (bytes.length == 0 || bytes.length > 8 * 1024 * 1024) {
throw new ApiException("MEDIA_TOO_LARGE", "媒体输入为空或超过 8 MiB", 400);
}
String contentType = response.headers().firstValue("Content-Type").orElse("application/octet-stream")
.split(";", 2)[0].trim().toLowerCase(java.util.Locale.ROOT);
if (!contentType.startsWith("image/")) {
throw new ApiException("MEDIA_TYPE_UNSUPPORTED", "图像编辑和视频参考图必须是 image/* 文件", 400);
}
String filename = "input" + extension(contentType);
return new MediaBytes(bytes, contentType, filename);
}
private String extension(String contentType) {
return switch (contentType) {
case "image/jpeg" -> ".jpg";
case "image/png" -> ".png";
case "image/webp" -> ".webp";
case "image/gif" -> ".gif";
default -> ".bin";
};
}
private record MediaBytes(byte[] bytes, String contentType, String filename) {}
private Map<String,Object> request(String method, String url, String key, Map<String,Object> body, String message) {
boolean sending = false;
try {
HttpRequest.Builder builder = HttpRequest.newBuilder().uri(URI.create(url)).timeout(UPSTREAM_REQUEST_TIMEOUT)
.header("Authorization", "Bearer " + key).header("Content-Type", "application/json");
if ("POST".equals(method)) builder.POST(HttpRequest.BodyPublishers.ofString(write(body)));
else builder.GET();
HttpResponse<String> response = client.send(builder.build(), HttpResponse.BodyHandlers.ofString());
if (response.statusCode() < 200 || response.statusCode() >= 300) {
String code = switch (response.statusCode()) {
case 401, 403 -> "UPSTREAM_AUTH_FAILED";
case 402 -> "UPSTREAM_CREDIT_INSUFFICIENT";
case 429 -> "UPSTREAM_RATE_LIMITED";
default -> "UPSTREAM_PROVIDER_ERROR";
};
throw new ApiException(code, message + "(HTTP " + response.statusCode() + ")", 502);
}
HttpRequest request = builder.build();
sending = true;
var response = transport.send(request, 8 * 1024 * 1024);
if (response.statusCode() < 200 || response.statusCode() >= 300) throw upstreamStatus(response.statusCode(), message);
Map<String,Object> parsed = mapper.readValue(response.body(), new TypeReference<>() {});
Object code = parsed.get("code");
if (parsed.containsKey("error") || code instanceof Number number && number.intValue() != 0) {
@@ -436,8 +421,7 @@ public class AsyncProviderClient {
throw new ApiException("UPSTREAM_PROVIDER_ERROR", message, 502);
}
return parsed;
} catch (ApiException exception) { throw exception; }
catch (Exception exception) { throw new ApiException("UPSTREAM_PROVIDER_ERROR", message, 502); }
} catch (Exception exception) { throw requestFailure(exception, sending, message); }
}
private Provider provider(String id) {
@@ -450,17 +434,15 @@ public class AsyncProviderClient {
String ciphertext = row.get("credential_ciphertext") == null ? "" : String.valueOf(row.get("credential_ciphertext"));
String key = ciphertext.isBlank() ? "" : cipher.decrypt(ciphertext);
String code = String.valueOf(row.get("code"));
List<String> keys = new ArrayList<>();
if (!key.isBlank()) keys.add(key);
addEnvironmentKeys(keys, code);
addEnvironmentKeys(keys, id);
var resolver = new UpstreamCredentialResolver(environment);
List<String> keys = new ArrayList<>(resolver.providerKeys(key, code, id));
// Tripo's official API key may be supplied through the server secret
// environment so it never has to be committed or exposed in SQL.
if (key.isBlank() && isTripo(code, String.valueOf(row.get("base_url")))) {
addEnvironmentKeys(keys, "TRIPO_API_KEY");
addEnvironmentKeys(keys, "MOYI_TRIPO_API_KEY");
keys.addAll(resolver.environmentKeys("TRIPO_API_KEY"));
keys.addAll(resolver.environmentKeys("MOYI_TRIPO_API_KEY"));
}
if (keys.isEmpty() && routeKeys(code, id, routeId, upstreamModel).isEmpty()) throw new ApiException("PROVIDER_NOT_CONFIGURED", "异步渠道尚未配置 API Key", 503);
if (keys.isEmpty() && resolver.scopedKeys(code, routeId, upstreamModel).isEmpty()) throw new ApiException("PROVIDER_NOT_CONFIGURED", "异步渠道尚未配置 API Key", 503);
return new Provider(code, String.valueOf(row.get("protocol")), String.valueOf(row.get("base_url")), keys.stream().distinct().toList(), keyCursors.computeIfAbsent(id, ignored -> new AtomicInteger()));
} catch (EmptyResultDataAccessException exception) { throw new ApiException("PROVIDER_NOT_CONFIGURED", "异步渠道不存在或已停用", 503); }
}
@@ -477,32 +459,6 @@ public class AsyncProviderClient {
private String trim(String value) { return value == null ? "" : value.replaceAll("/+$", ""); }
private String encode(String value) { return java.net.URLEncoder.encode(value, java.nio.charset.StandardCharsets.UTF_8); }
private String write(Object value) throws JsonProcessingException { return mapper.writeValueAsString(value == null ? Map.of() : value); }
private void addEnvironmentKeys(List<String> keys, String alias) {
String name = alias.endsWith("_API_KEY") ? alias : "MOYI_UPSTREAM_API_KEYS_" + alias.toUpperCase(java.util.Locale.ROOT).replace('-', '_');
String configured = environment.getProperty(name);
if (configured != null && !configured.isBlank()) for (String value : configured.split("\\|")) if (!value.isBlank()) keys.add(value.trim());
}
private List<String> routeKeys(String providerCode, String providerId, String routeId, String upstreamModel) {
if (routeId != null && !routeId.isBlank()) {
String configured = environment.getProperty("MOYI_UPSTREAM_API_KEYS_ROUTE_" + normalize(routeId));
if (configured != null && !configured.isBlank()) return parseKeys(configured);
}
if (upstreamModel != null && !upstreamModel.isBlank()) {
String scopedName = "MOYI_UPSTREAM_API_KEYS_MODEL_" + normalize(providerCode) + "_" + normalize(upstreamModel);
String configured = environment.getProperty(scopedName);
if (configured != null && !configured.isBlank()) return parseKeys(configured);
}
return List.of();
}
private List<String> parseKeys(String configured) {
List<String> keys = new ArrayList<>();
for (String key : configured.split("\\|")) if (!key.isBlank()) keys.add(key.trim());
return keys.stream().distinct().toList();
}
private String normalize(String value) { return String.valueOf(value == null ? "" : value).toUpperCase(java.util.Locale.ROOT).replaceAll("[^A-Z0-9]", "_"); }
private Map<String,Object> withKeyFingerprint(Map<String,Object> response, String apiKey) {
Map<String,Object> copy = new LinkedHashMap<>();
if (response != null) copy.putAll(response);
@@ -528,15 +484,15 @@ public class AsyncProviderClient {
return normalizedCode.contains("tripo") || normalizedUrl.contains("tripo3d");
}
private record Provider(String code, String protocol, String baseUrl, List<String> apiKeys, AtomicInteger cursor) {
@Override public String toString() { return "Provider[code=" + code + ", credentialCount=" + apiKeys.size() + "]"; }
private String nextApiKey() {
if (apiKeys.isEmpty()) throw new ApiException("UPSTREAM_KEY_NOT_AVAILABLE", "异步渠道没有可用的上游凭据", 503);
return apiKeys.get(Math.floorMod(cursor.getAndIncrement(), apiKeys.size()));
return UpstreamCredentialResolver.nextKey(apiKeys, cursor);
}
private String nextApiKey(String routeId, String upstreamModel, Environment environment, ConcurrentMap<String, AtomicInteger> cursors) {
List<String> routeKeys = routeKeys(routeId, upstreamModel, environment);
if (routeKeys.isEmpty()) return nextApiKey();
AtomicInteger routeCursor = cursors.computeIfAbsent("route:" + (routeId == null ? upstreamModel : routeId), ignored -> new AtomicInteger());
return routeKeys.get(Math.floorMod(routeCursor.getAndIncrement(), routeKeys.size()));
return UpstreamCredentialResolver.nextKey(routeKeys, routeCursor);
}
private String keyForFingerprint(String routeId, String upstreamModel, Environment environment,
ConcurrentMap<String, AtomicInteger> cursors, String fingerprint) {
@@ -548,22 +504,7 @@ public class AsyncProviderClient {
throw new ApiException("UPSTREAM_KEY_NOT_AVAILABLE", "创建任务时使用的上游凭据已从服务器配置中移除,无法安全轮询", 503);
}
private List<String> routeKeys(String routeId, String upstreamModel, Environment environment) {
if (environment == null) return List.of();
if (routeId != null && !routeId.isBlank()) {
String value = environment.getProperty("MOYI_UPSTREAM_API_KEYS_ROUTE_" + normalize(routeId));
if (value != null && !value.isBlank()) return parseKeys(value);
return new UpstreamCredentialResolver(environment).scopedKeys(code, routeId, upstreamModel);
}
if (upstreamModel != null && !upstreamModel.isBlank()) {
String value = environment.getProperty("MOYI_UPSTREAM_API_KEYS_MODEL_" + normalize(code()) + "_" + normalize(upstreamModel));
if (value != null && !value.isBlank()) return parseKeys(value);
}
return List.of();
}
private static List<String> parseKeys(String value) {
List<String> keys = new ArrayList<>();
for (String key : value.split("\\|")) if (!key.isBlank()) keys.add(key.trim());
return keys.stream().distinct().toList();
}
private static String normalize(String value) { return String.valueOf(value == null ? "" : value).toUpperCase(java.util.Locale.ROOT).replaceAll("[^A-Z0-9]", "_"); }
}
}
@@ -7,14 +7,18 @@ import com.moyi.controlplane.catalog.CatalogService;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.gateway.GatewayModelIds;
import com.moyi.controlplane.gateway.request.ApiRequestContext;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
import com.moyi.controlplane.gateway.request.ApiRequestProtocol;
import com.moyi.controlplane.gateway.request.ApiRequestRiskEngine;
import com.moyi.controlplane.gateway.request.ApiRequestRiskPolicy;
import com.moyi.controlplane.usage.UsageAuditService;
import com.moyi.controlplane.provider.transport.ProviderRequestException;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -30,6 +34,7 @@ import org.springframework.transaction.annotation.Transactional;
@Service
public class AsyncTaskService {
private static final org.slf4j.Logger log = org.slf4j.LoggerFactory.getLogger(AsyncTaskService.class);
private final JdbcTemplate jdbc;
private final ObjectMapper objectMapper;
private final CatalogService catalogService;
@@ -43,11 +48,15 @@ public class AsyncTaskService {
private final AsyncProviderClient providerClient;
private final TransactionTemplate transactions;
private final com.moyi.controlplane.gateway.EnterprisePolicies policies;
private final ApiRequestRiskPolicy requestRisk;
private final TaskOutcomeJournal outcomes;
private final TaskSettlementService settlements;
/** Compatibility constructor for isolated gateway tests that only exercise mock tasks. */
public AsyncTaskService(JdbcTemplate jdbc, ObjectMapper objectMapper, CatalogService catalogService,
int maxQueuedPerModel, GatewayGuard gatewayGuard, UsageAuditService usage) {
this(jdbc, objectMapper, catalogService, maxQueuedPerModel, 4, 4, 3_600_000, true, gatewayGuard, usage, null, null);
this(jdbc, objectMapper, catalogService, maxQueuedPerModel, 4, 4, 3_600_000, true,
gatewayGuard, usage, null, null, new ApiRequestRiskPolicy(ApiRequestRiskEngine.allowAll()));
}
/** Compatibility constructor for isolated tests using explicit limits. */
@@ -58,7 +67,8 @@ public class AsyncTaskService {
AsyncProviderClient providerClient, PlatformTransactionManager transactionManager
) {
this(jdbc, objectMapper, catalogService, maxQueuedPerModel, defaultModelConcurrency,
defaultKeyConcurrency, providerTimeoutMs, true, gatewayGuard, usage, providerClient, transactionManager);
defaultKeyConcurrency, providerTimeoutMs, true, gatewayGuard, usage, providerClient,
transactionManager, new ApiRequestRiskPolicy(ApiRequestRiskEngine.allowAll()));
}
@org.springframework.beans.factory.annotation.Autowired
@@ -74,7 +84,8 @@ public class AsyncTaskService {
GatewayGuard gatewayGuard,
UsageAuditService usage,
AsyncProviderClient providerClient,
PlatformTransactionManager transactionManager
PlatformTransactionManager transactionManager,
ApiRequestRiskPolicy requestRisk
) {
// A model concurrency of 0 means no MOYI-side model-group cap. The
// upstream provider still enforces its own quota, while API-Key
@@ -94,16 +105,64 @@ public class AsyncTaskService {
this.providerClient = providerClient;
this.transactions = transactionManager == null ? null : new TransactionTemplate(transactionManager);
this.policies = new com.moyi.controlplane.gateway.EnterprisePolicies(jdbc,objectMapper);
this.requestRisk = java.util.Objects.requireNonNull(requestRisk, "requestRisk");
this.outcomes = new TaskOutcomeJournal(jdbc, objectMapper, transactions, usage);
this.settlements = new TaskSettlementService(jdbc, objectMapper, gatewayGuard, usage, transactionManager);
}
@Transactional
public Map<String, Object> create(String authorization, String taskType, Map<String, Object> input) {
return createForKey(requireKey(authorization), taskType, input);
return create(authorization, taskType, input, ApiRequestMetadata.internal());
}
@Transactional
public Map<String, Object> create(String authorization, String taskType, Map<String, Object> input, ApiRequestMetadata metadata) {
return create(authorization, taskType, input, null, metadata);
}
@Transactional
public Map<String, Object> create(String authorization, String taskType, Map<String, Object> input,
String idempotencyKey, ApiRequestMetadata metadata) {
return createForKey(requireKey(authorization), taskType, input, idempotencyKey, metadata);
}
@Transactional
public Map<String, Object> createForKey(Map<String, Object> key, String taskType, Map<String, Object> input) {
gatewayGuard.checkRate(UUID.fromString(String.valueOf(key.get("id"))), false);
return createForKey(key, taskType, input, ApiRequestMetadata.internal());
}
@Transactional
public Map<String, Object> createForKey(Map<String, Object> key, String taskType, Map<String, Object> input, ApiRequestMetadata metadata) {
return createForKey(key, taskType, input, null, metadata);
}
@Transactional
public Map<String, Object> createForKey(Map<String, Object> key, String taskType, Map<String, Object> input,
String idempotencyKey, ApiRequestMetadata metadata) {
// Explicit transaction also protects callers that do not go through the Spring proxy.
return inTransaction(() -> createInTransaction(key, taskType, input, idempotencyKey, metadata));
}
private Map<String,Object> createInTransaction(Map<String,Object> key, String taskType, Map<String,Object> input,
String idempotencyKey, ApiRequestMetadata metadata) {
UUID apiKeyId = UUID.fromString(String.valueOf(key.get("id")));
gatewayGuard.checkRate(apiKeyId, false);
if (input == null) throw new ApiException("INVALID_REQUEST", "请提供任务请求");
TaskCreationIdentity identity = TaskCreationIdentity.of(objectMapper, idempotencyKey, taskType, input);
UUID userId = catalogService.keyOwnerId(apiKeyId);
if (identity.keyHash() != null) {
// Serialize the same user/key before checking capacity, budget or wallet balances.
jdbc.queryForList("SELECT pg_advisory_xact_lock(hashtextextended(?,0))",
"moyi:task-idempotency:" + userId + ":" + identity.keyHash());
var existing = jdbc.queryForList("SELECT id,request_hash FROM model_tasks WHERE user_id=? AND idempotency_hash=?", userId, identity.keyHash());
if (!existing.isEmpty()) {
if (!identity.requestHash().equals(existing.getFirst().get("request_hash"))) {
throw new ApiException("IDEMPOTENCY_CONFLICT", "此 Idempotency-Key 已用于不同的任务请求", 409);
}
// Authenticated same-user reads remain valid even if the original route was retired.
return taskView(String.valueOf(existing.getFirst().get("id")));
}
}
String requestedModelId = String.valueOf(input.getOrDefault("model", ""));
Map<String, Object> model = GatewayModelIds.resolve(catalogService, key, requestedModelId);
if (model == null) throw new ApiException("MODEL_NOT_FOUND", "模型不存在或已下线", 404);
@@ -111,19 +170,21 @@ public class AsyncTaskService {
if (!taskType.equals(model.get("type")) || !"async".equals(model.get("mode"))) throw new ApiException("NOT_ASYNC_MEDIA_MODEL", "该模型不是对应的异步媒体模型", 400);
if (Boolean.FALSE.equals(model.get("executionReady"))) throw new ApiException("ASYNC_PROVIDER_NOT_IMPLEMENTED", "该模型的专用执行接口尚未接入", 503);
if (!"provider-mock".equals(model.get("providerId")) && providerClient == null) throw new ApiException("ASYNC_PROVIDER_NOT_IMPLEMENTED", "异步执行器未配置", 501);
// Serialize the queue-capacity check with task insertion for this
// model. Without the transaction advisory lock concurrent creates
// could all observe the same count and exceed maxQueuedPerModel.
jdbc.queryForList("SELECT pg_advisory_xact_lock(hashtextextended('moyi:async-queue:' || ?, 0))", modelId);
Integer queued = jdbc.queryForObject("SELECT COUNT(*) FROM model_tasks WHERE model_id = ? AND status IN ('queued', 'running')", Integer.class, modelId);
if (queued != null && queued >= maxQueuedPerModel) throw new ApiException("TASK_QUEUE_FULL", "该模型当前排队已满,请稍后重试", 429);
String prompt = String.valueOf(input.getOrDefault("prompt", "")).trim();
Object mediaInput = input.get("media_url");
if (prompt.isBlank() && (mediaInput == null || String.valueOf(mediaInput).isBlank())) {
throw new ApiException("PROMPT_REQUIRED", "生成提示词或媒体输入不能为空");
}
UUID apiKeyId = UUID.fromString(String.valueOf(key.get("id")));
UUID userId = catalogService.keyOwnerId(apiKeyId);
requestRisk.enforce(new ApiRequestContext(userId, apiKeyId, ApiRequestProtocol.ASYNC_TASK,
requestedModelId, modelId, false, estimateInputUnits(input), 0, metadata, System.nanoTime()));
String upstreamModel = model.get("upstreamModel") instanceof String value ? value : null;
if (upstreamModel == null || upstreamModel.isBlank()) throw new ApiException("ASYNC_BINDING_MISSING", "模型缺少可冻结的上游执行标识", 503);
// Serialize the queue-capacity check with task insertion for this
// model. Without the transaction advisory lock, concurrent creates
// could all observe the same count and exceed maxQueuedPerModel.
jdbc.queryForList("SELECT pg_advisory_xact_lock(hashtextextended('moyi:async-queue:' || ?, 0))", modelId);
Integer queued = jdbc.queryForObject("SELECT COUNT(*) FROM model_tasks WHERE model_id = ? AND status IN ('queued', 'running', 'reconciling')", Integer.class, modelId);
if (queued != null && queued >= maxQueuedPerModel) throw new ApiException("TASK_QUEUE_FULL", "该模型当前排队已满,请稍后重试", 429);
@SuppressWarnings("unchecked") Map<String,Object> frozenPricing=(Map<String,Object>) model.get("pricing");
if(AsyncPriceSchedule.tokens(frozenPricing) && !"image".equals(taskType)) throw new ApiException("ASYNC_BILLING_UNSUPPORTED","该媒体类型的 Token 用量结算尚未接入",503);
BigDecimal reservedAmount = "provider-mock".equals(model.get("providerId")) ? BigDecimal.ZERO : AsyncPriceSchedule.reserve(frozenPricing,input);
@@ -139,14 +200,14 @@ public class AsyncTaskService {
OffsetDateTime now = now();
try {
jdbc.update(
"INSERT INTO model_tasks (id, user_id, api_key_id, model_id, task_type, provider_task_id, status, progress, prompt, input, reserved_amount, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, 'queued', 0, ?, CAST(? AS jsonb), ?, ?, ?)",
id, userId, apiKeyId, modelId, taskType, providerTaskId, prompt, writeJson(input == null ? Map.of() : input), reservedAmount, now, now
"INSERT INTO model_tasks (id, user_id, api_key_id, model_id, task_type, provider_task_id, status, progress, prompt, input, reserved_amount, created_at, updated_at, idempotency_hash, request_hash, provider_id_snapshot, upstream_model_snapshot) VALUES (?, ?, ?, ?, ?, ?, 'queued', 0, ?, CAST(? AS jsonb), ?, ?, ?, ?, ?, ?, ?)",
id, userId, apiKeyId, modelId, taskType, providerTaskId, prompt, writeJson(input), reservedAmount, now, now, identity.keyHash(), identity.requestHash(), model.get("providerId"), upstreamModel
);
} catch (RuntimeException exception) {
releaseWallet(userId, reservedAmount);
throw exception;
}
usage.recordTask(userId, apiKeyId, modelId, String.valueOf(model.get("providerId")), id, "queued", input, null, null, null);
usage.recordTask(userId, apiKeyId, modelId, String.valueOf(model.get("providerId")), id, "queued", input, null, null, null, metadata);
if (!"provider-mock".equals(model.get("providerId"))) jdbc.update("UPDATE model_tasks SET pricing_snapshot=CAST(? AS jsonb),price_version=(SELECT max(id) FROM model_price_history WHERE model_id=?) WHERE id=?",writeJson(frozenPricing),modelId,id);
catalogService.touchKey(key);
return taskView(id);
@@ -182,6 +243,11 @@ public class AsyncTaskService {
@Scheduled(fixedDelayString = "${moyi.tasks.worker-interval-ms:1000}")
public void completeMockTasks() {
// Retry local settlement from durable observations, never the upstream generation request.
for (var completion : outcomes.pendingCompletions()) {
try { completeObserved(completion); }
catch (RuntimeException exception) { log.warn("Async task {} settlement remains pending", completion.taskId()); }
}
List<String> taskIds = jdbc.query("SELECT id FROM model_tasks WHERE status IN ('queued', 'running') AND provider_task_id = 'mock' AND created_at < now() - INTERVAL '500 milliseconds'", (rs, row) -> rs.getString("id"));
jdbc.update(
"UPDATE model_tasks SET status = 'succeeded', progress = 100, result = CAST(CASE WHEN task_type = '3d' THEN ? ELSE ? END AS jsonb), updated_at = ?, completed_at = ? WHERE status IN ('queued', 'running') AND provider_task_id = 'mock' AND created_at < now() - INTERVAL '500 milliseconds'",
@@ -189,7 +255,7 @@ public class AsyncTaskService {
"{\"format\":\"mp4\",\"url\":\"https://mock.moyi.ai/tasks/result.mp4\"}", now(), now()
);
taskIds.forEach(taskId -> usage.updateTask(taskId, "succeeded", Map.of("format", "result", "url", "https://mock.moyi.ai/tasks/" + taskId + "/result"), null, null));
List<Map<String,Object>> pending = jdbc.queryForList("SELECT t.id, t.model_id, t.provider_task_id, t.provider_key_fingerprint, t.provider_deadline_at, t.created_at, m.provider_id, m.upstream_model FROM model_tasks t JOIN model_catalog m ON m.id=t.model_id WHERE t.status IN ('queued','running') AND t.provider_task_id <> 'mock' AND (t.updated_at < now() - INTERVAL '5 seconds' OR t.provider_deadline_at <= now()) ORDER BY t.updated_at LIMIT 50");
List<Map<String,Object>> pending = jdbc.queryForList("SELECT t.id, t.model_id, t.provider_task_id, t.provider_key_fingerprint, t.provider_deadline_at, t.created_at, t.provider_id_snapshot AS provider_id, t.upstream_model_snapshot AS upstream_model FROM model_tasks t WHERE t.status IN ('queued','running') AND t.completion_status IS NULL AND t.provider_task_id <> 'mock' AND (t.updated_at < now() - INTERVAL '5 seconds' OR t.provider_deadline_at <= now()) ORDER BY t.updated_at LIMIT 50");
for (Map<String,Object> row : pending) pollProviderTask(row);
dispatchQueuedTasks();
}
@@ -202,36 +268,43 @@ public class AsyncTaskService {
*/
private void dispatchQueuedTasks() {
if (providerClient == null) return;
// A process can die after claiming a task but before receiving an
// upstream task id. Fail those claims after a grace period. Retrying
// them blindly could submit the same generation twice when the
// upstream accepted the request just before the process died.
// A lost submit response is ambiguous, not a failed generation.
// Quarantine stale claims without retrying or releasing financial holds.
recoverInterruptedClaims();
for (int i = 0; i < 50; i++) {
Map<String,Object> task = claimQueuedTask();
if (task == null) return;
String taskId = String.valueOf(task.get("id"));
usage.updateTask(taskId, "running", null, null, null);
Map<String,Object> upstream;
try {
Map<String,Object> input = taskInput(task);
Map<String,Object> upstream = submitProviderTask(task, input);
// Some OpenAI-compatible image/audio endpoints return the
// generated media in the submit response rather than creating
// a pollable job. Persist that result and settle the task in
// the same worker transaction.
upstream = submitProviderTask(task, input);
} catch (RuntimeException exception) {
if (exception instanceof ProviderRequestException failure && failure.definitelyNotAccepted()) {
completeObserved(outcomes.observeTerminal(taskId, "failed", null, failure.getCode(), failure.getMessage()));
} else {
outcomes.requireReconciliation(taskId, "UPSTREAM_TASK_ACCEPTANCE_UNKNOWN", "上游任务是否已受理尚未确认;冻结额度保留,请联系管理员核账,勿重复提交", false);
}
continue;
}
TaskOutcomeJournal.Completion completion = null;
try {
// Journal a received result before touching the ledger. A failed settlement
// must only retry locally, never generate again or release its reservation.
if (upstream.get("_moyiInlineResult") instanceof Map<?,?> inline) {
Map<String,Object> result = new LinkedHashMap<>();
inline.forEach((key, value) -> result.put(String.valueOf(key), value));
finishTask(taskId, "succeeded", 100, result, null, null);
continue;
}
completion = outcomes.observeTerminal(taskId, "succeeded", result, null, null);
} else {
OffsetDateTime submittedAt = now();
jdbc.update("UPDATE model_tasks SET provider_task_id=?, provider_key_fingerprint=?, provider_deadline_at=?, updated_at=? WHERE id=? AND status='running'", providerTaskId(upstream), upstream.get("_moyiUpstreamKeyFingerprint"), submittedAt.plus(java.time.Duration.ofMillis(providerTimeoutMs)), submittedAt, taskId);
} catch (RuntimeException exception) {
String code = exception instanceof ApiException error ? error.getCode() : "UPSTREAM_TASK_SUBMIT_FAILED";
String message = exception instanceof ApiException error ? error.getMessage() : "上游异步任务创建失败";
failTask(taskId, code, message);
outcomes.accepted(taskId, providerTaskId(upstream), upstream.get("_moyiUpstreamKeyFingerprint"),
submittedAt, submittedAt.plus(java.time.Duration.ofMillis(providerTimeoutMs)));
}
} catch (RuntimeException exception) {
outcomes.requireReconciliation(taskId, "UPSTREAM_TASK_RECEIPT_UNKNOWN", "已调用上游,但任务回执无法确认或保存;冻结额度保留,等待核账,勿重复提交", false);
}
// Deliberately outside the provider exception handler.
completeObserved(completion);
}
}
@@ -243,26 +316,26 @@ public class AsyncTaskService {
// workers reliably.
jdbc.queryForList("SELECT pg_advisory_xact_lock(hashtextextended('moyi:async-dispatch', 0))");
String sql = """
SELECT t.id, t.user_id, t.api_key_id, t.model_id, m.provider_id, m.upstream_model, t.task_type, t.prompt, t.input
SELECT t.id, t.user_id, t.api_key_id, t.model_id, t.provider_id_snapshot AS provider_id, t.upstream_model_snapshot AS upstream_model, t.task_type, t.prompt, t.input
FROM model_tasks t
JOIN model_catalog m ON m.id=t.model_id
JOIN model_providers p ON p.id=m.provider_id
JOIN model_providers p ON p.id=t.provider_id_snapshot
WHERE t.status='queued'
AND t.provider_task_id IS NULL
AND m.status='active' AND m.deleted_at IS NULL
AND p.status='active' AND p.deleted_at IS NULL
AND (SELECT count(*) FROM model_tasks ru WHERE ru.user_id=t.user_id AND ru.status='running') < coalesce((SELECT max_concurrency FROM gateway_user_policies up WHERE up.user_id=t.user_id),16)
AND (SELECT count(*) FROM model_tasks ru WHERE ru.user_id=t.user_id AND ru.status IN ('running','reconciling')) < coalesce((SELECT max_concurrency FROM gateway_user_policies up WHERE up.user_id=t.user_id),16)
AND (
? <= 0
OR COALESCE(m.max_concurrency, ?) <= 0
OR (SELECT COUNT(*) FROM model_tasks running_model
WHERE running_model.model_id=t.model_id AND running_model.status='running')
WHERE running_model.model_id=t.model_id AND running_model.status IN ('running','reconciling'))
< COALESCE(m.max_concurrency, ?)
)
AND (
? <= 0
OR (SELECT COUNT(*) FROM model_tasks running_key
WHERE running_key.api_key_id=t.api_key_id AND running_key.status='running')
WHERE running_key.api_key_id=t.api_key_id AND running_key.status IN ('running','reconciling'))
< ?
)
ORDER BY t.created_at
@@ -277,51 +350,32 @@ public class AsyncTaskService {
if(!gatewayGuard.tryTaskUserSlot((UUID)row.get("api_key_id"),row.get("id").toString(),providerTimeoutMs+180000)) return null;
try {jdbc.update("UPDATE model_tasks SET status='running', updated_at=now() WHERE id=? AND status='queued'", row.get("id"));}
catch(RuntimeException e) {gatewayGuard.releaseTaskUserSlot((UUID)row.get("user_id"),row.get("id").toString());throw e;}
usage.updateTask(row.get("id").toString(), "running", null, null, null);
return row;
});
}
private void recoverInterruptedClaims() {
List<String> taskIds = inTransaction(() -> jdbc.queryForList(
"SELECT id FROM model_tasks WHERE status='running' AND provider_task_id IS NULL AND updated_at < now() - INTERVAL '2 minutes' FOR UPDATE SKIP LOCKED",
"SELECT id FROM model_tasks WHERE status='running' AND provider_task_id IS NULL AND completion_status IS NULL AND updated_at < now() - INTERVAL '2 minutes' FOR UPDATE SKIP LOCKED",
String.class));
for (String taskId : taskIds) {
failTask(taskId, "UPSTREAM_TASK_SUBMIT_INTERRUPTED", "上游任务提交被中断,请确认上游记录后重新提交");
outcomes.requireReconciliation(taskId, "UPSTREAM_TASK_SUBMIT_INTERRUPTED", "上游提交回执丢失或进程中断;冻结额度保留,等待核账,勿重复提交", true);
}
}
private void completeObserved(TaskOutcomeJournal.Completion completion) {
if (completion == null) return;
finishTask(completion.taskId(), completion.status(), "succeeded".equals(completion.status()) ? 100 : 0,
completion.result(), completion.code(), completion.message());
}
private void failTask(String taskId, String code, String message) {
finishTask(taskId, "failed", 0, null, code, message);
}
private void finishTask(String taskId, String status, int progress, Map<String,Object> result, String code, String message) {
// The terminal state and wallet change must commit together. Otherwise a
// crash after marking failed/succeeded leaves a hold that polling never
// revisits. The conditional update also prevents duplicate settlement.
inTransaction(() -> {
Map<String,Object> task=jdbc.queryForMap("SELECT status,pricing_snapshot::text AS pricing_snapshot FROM model_tasks WHERE id=? FOR UPDATE",taskId);
if(!List.of("queued","running").contains(task.get("status"))) return null;
AsyncPriceSchedule.Bill bill=null;
String finalCode=code,finalMessage=message;
if("succeeded".equals(status) && task.get("pricing_snapshot")!=null) {
try { bill=AsyncPriceSchedule.bill(readMap(String.valueOf(task.get("pricing_snapshot"))),result); }
catch(ApiException e) { finalCode="TASK_USAGE_PENDING"; finalMessage="生成已完成,但上游计费用量未能核对;暂不扣费,等待核账"; }
}
int changed = jdbc.update(
"UPDATE model_tasks SET status=?, progress=?, result=CAST(? AS jsonb), error=CAST(? AS jsonb), updated_at=?, completed_at=? WHERE id=? AND status IN ('queued','running')",
status, progress, result == null ? null : writeJson(result),
finalCode == null ? null : writeJson(Map.of("code", finalCode, "message", finalMessage)), now(), now(), taskId);
if (changed == 0) return null;
if ("TASK_USAGE_PENDING".equals(finalCode)) {
releaseTaskInTransaction(taskId);
jdbc.update("UPDATE model_tasks SET cost_amount=NULL WHERE id=?",taskId);
jdbc.update("UPDATE usage_events SET cost_amount=NULL,cost_status='pending',observed_usage=CAST(? AS jsonb) WHERE provider_task_id=? AND event_kind='task'",writeJson(usageOnly(result)),taskId);
}
else if ("succeeded".equals(status)) settleTaskInTransaction(taskId,bill);
else releaseTaskInTransaction(taskId);
usage.updateTask(taskId, status, result, finalCode, finalMessage);
return null;
});
settlements.finish(taskId, status, progress, result, code, message);
}
private Map<String, Object> requireKey(String authorization) {
@@ -342,6 +396,7 @@ public class AsyncTaskService {
task.put("progress", resultSet.getInt("progress"));
task.put("createdAt", resultSet.getObject("created_at", OffsetDateTime.class));
task.put("updatedAt", resultSet.getObject("updated_at", OffsetDateTime.class));
task.put("requiresReconciliation", "reconciling".equals(resultSet.getString("status")));
putIfPresent(task, "costAmount", resultSet, "cost_amount");
String result = resultSet.getString("result");
if (result != null) task.put("result", readMap(result));
@@ -385,13 +440,20 @@ public class AsyncTaskService {
private String providerTaskId(Map<String,Object> response) {
Object direct = response.get("id");
if (direct == null && response.get("data") instanceof Map<?,?> data) direct = data.get("task_id");
if (direct == null) throw new ApiException("UPSTREAM_PROVIDER_ERROR", "上游没有返回异步任务编号", 502);
return String.valueOf(direct);
if (direct == null) direct = response.get("task_id");
if (direct == null && response.get("data") instanceof Map<?,?> data) direct = data.get("task_id") == null ? data.get("id") : data.get("task_id");
if (!(direct instanceof String value) || value.isBlank() || value.length() > 160 || value.chars().anyMatch(Character::isISOControl)) {
throw new ApiException("UPSTREAM_PROVIDER_ERROR", "上游没有返回有效的异步任务编号", 502);
}
return value;
}
private void pollProviderTask(Map<String,Object> row) {
String taskId = String.valueOf(row.get("id"));
if (row.get("provider_id") == null || row.get("upstream_model") == null) {
outcomes.requireReconciliation(taskId, "ASYNC_BINDING_MISSING", "历史任务缺少可证明的原始执行路由;冻结额度保留,请根据上游记录核账", false);
return;
}
OffsetDateTime deadline = taskTimestamp(row.get("provider_deadline_at"));
if (deadline == null) {
// Defensive fallback for pre-migration tasks. The migration also
@@ -401,7 +463,7 @@ public class AsyncTaskService {
deadline = createdAt.plus(java.time.Duration.ofMillis(providerTimeoutMs));
}
if (!now().isBefore(deadline)) {
failTask(taskId, "UPSTREAM_TASK_TIMEOUT", "异步任务查询已超时,执行结果未知;平台冻结额度已释放。上游可能仍在执行,请先核实后再提交");
outcomes.requireReconciliation(taskId, "UPSTREAM_TASK_TIMEOUT", "异步任务查询已超时,执行结果未知;冻结额度保留,等待管理员核账,勿重复提交", false);
return;
}
Map<String,Object> response;
@@ -409,7 +471,7 @@ public class AsyncTaskService {
response = providerClient.poll(String.valueOf(row.get("provider_id")), String.valueOf(row.get("model_id")), String.valueOf(row.get("upstream_model")), String.valueOf(row.get("provider_task_id")), row.get("provider_key_fingerprint") == null ? null : String.valueOf(row.get("provider_key_fingerprint")));
} catch (RuntimeException exception) {
if (exception instanceof ApiException error && "UPSTREAM_KEY_NOT_AVAILABLE".equals(error.getCode())) {
failTask(taskId, error.getCode(), error.getMessage());
outcomes.requireReconciliation(taskId, error.getCode(), "提交任务使用的上游凭据已不可用,执行结果未知;冻结额度保留,等待核账", false);
return;
}
// Even a permanent authentication/configuration error only proves
@@ -432,9 +494,9 @@ public class AsyncTaskService {
if ("succeeded".equals(status) || "failed".equals(status) || "cancelled".equals(status)) {
// Persistence failures must roll back and remain pollable, rather
// than being mistaken for upstream failures and refunded.
finishTask(taskId, status, progress, result,
completeObserved(outcomes.observeTerminal(taskId, status, result,
"failed".equals(status) ? "UPSTREAM_TASK_FAILED" : null,
"failed".equals(status) ? "上游异步任务失败" : null);
"failed".equals(status) ? "上游异步任务失败" : null));
} else {
updatePollingState(taskId, progress, null, null);
}
@@ -442,7 +504,7 @@ public class AsyncTaskService {
private void updatePollingState(String taskId, Integer progress, String code, String message) {
inTransaction(() -> {
int changed = jdbc.update("UPDATE model_tasks SET status='running', progress=COALESCE(?,progress), error=CAST(? AS jsonb), updated_at=? WHERE id=? AND status IN ('queued','running')",
int changed = jdbc.update("UPDATE model_tasks SET status='running', progress=COALESCE(?,progress), error=CAST(? AS jsonb), updated_at=? WHERE id=? AND status IN ('queued','running') AND completion_status IS NULL",
progress, code == null ? null : writeJson(Map.of("code", code, "message", message, "retryable", true)), now(), taskId);
if (changed > 0) usage.updateTask(taskId, "running", null, code, message);
return null;
@@ -460,10 +522,12 @@ public class AsyncTaskService {
Object raw = response.get("status");
if (raw == null && response.get("data") instanceof Map<?,?> data) raw = data.get("status");
String value = String.valueOf(raw == null ? "running" : raw).toLowerCase(java.util.Locale.ROOT);
if (value.contains("success") || value.equals("completed")) return "succeeded";
if (value.contains("fail") || value.equals("error") || value.equals("banned")) return "failed";
if (value.contains("cancel")) return "cancelled";
return "running";
return switch (value) {
case "success", "succeeded", "completed" -> "succeeded";
case "failed", "failure", "error", "banned" -> "failed";
case "cancelled", "canceled" -> "cancelled";
default -> "running";
};
}
private int progress(Map<String,Object> response, String status) {
@@ -513,6 +577,11 @@ public class AsyncTaskService {
private String writeJson(Object value) { try { return objectMapper.writeValueAsString(value); } catch (JsonProcessingException exception) { return "{}"; } }
private long estimateInputUnits(Map<String,Object> input) {
try { return objectMapper.writeValueAsBytes(input == null ? Map.of() : input).length; }
catch (JsonProcessingException exception) { throw new ApiException("INVALID_REQUEST", "任务请求无法编码"); }
}
private Map<String,Object> usageOnly(Map<String,Object> result) {
var selected=new LinkedHashMap<String,Object>();
if(result!=null) for(String field:List.of("usage","usageMetadata")) if(result.get(field) instanceof Map<?,?> value) selected.put(field,value);
@@ -531,42 +600,10 @@ public class AsyncTaskService {
if (amount.signum() > 0) jdbc.update("UPDATE user_wallets SET reserved_balance=GREATEST(0,reserved_balance-?),updated_at=now() WHERE user_id=?", amount, userId);
}
private void settleTaskInTransaction(String taskId,AsyncPriceSchedule.Bill bill) {
Map<String,Object> task = jdbc.queryForMap("SELECT user_id,reserved_amount,cost_amount FROM model_tasks WHERE id=? FOR UPDATE", taskId);
BigDecimal reserved = (BigDecimal) task.get("reserved_amount");
BigDecimal amount = bill==null ? reserved : bill.amount();
policies.release(taskId);
gatewayGuard.releaseTaskUserSlot((UUID)task.get("user_id"),taskId);
if (amount == null || task.get("cost_amount") != null) return;
UUID userId = (UUID) task.get("user_id");
Map<String,Object> wallet = jdbc.queryForMap("SELECT balance,reserved_balance FROM user_wallets WHERE user_id=? FOR UPDATE", userId);
BigDecimal before = (BigDecimal) wallet.get("balance");
BigDecimal after = before.subtract(amount);
jdbc.update("UPDATE user_wallets SET balance=?,reserved_balance=GREATEST(0,reserved_balance-?),updated_at=now() WHERE user_id=?", after, reserved, userId);
if(amount.signum()>0) jdbc.update("INSERT INTO wallet_entries(id,user_id,request_id,amount,balance_after,currency,reason,operation,requested_amount,balance_before,actor_user_id) VALUES (?,?,?,?,?,'CNY',?,'usage',?,?,NULL)", UUID.randomUUID(), userId, taskRequestId(taskId), amount.negate(), after, "异步模型任务 · " + taskId, amount, before);
jdbc.update("UPDATE model_tasks SET cost_amount=? WHERE id=?", amount, taskId);
jdbc.update("UPDATE usage_events SET cost_amount=? WHERE provider_task_id=? AND event_kind='task'", amount, taskId);
if(bill!=null) jdbc.update("UPDATE usage_events SET input_units=?,output_units=?,observed_usage=jsonb_build_object('input_tokens',?::bigint,'output_tokens',?::bigint,'image_input_tokens',?::bigint) WHERE provider_task_id=? AND event_kind='task'",bill.input(),bill.output(),bill.input(),bill.output(),bill.imageInput(),taskId);
}
private void releaseTaskInTransaction(String taskId) {
Map<String,Object> task = jdbc.queryForMap("SELECT user_id,reserved_amount,cost_amount FROM model_tasks WHERE id=? FOR UPDATE", taskId);
BigDecimal amount = (BigDecimal) task.get("reserved_amount");
policies.release(taskId);
gatewayGuard.releaseTaskUserSlot((UUID)task.get("user_id"),taskId);
if (amount != null && amount.signum() > 0 && task.get("cost_amount") == null) {
releaseWallet((UUID) task.get("user_id"), amount);
jdbc.update("UPDATE model_tasks SET cost_amount=0 WHERE id=?", taskId);
jdbc.update("UPDATE usage_events SET cost_amount=0 WHERE provider_task_id=? AND event_kind='task'", taskId);
}
}
private <T> T inTransaction(Supplier<T> work) {
return transactions == null ? work.get() : transactions.execute(status -> work.get());
}
private UUID taskRequestId(String taskId) { return UUID.nameUUIDFromBytes(("moyi-async:" + taskId).getBytes(StandardCharsets.UTF_8)); }
private void putIfPresent(Map<String,Object> target, String name, ResultSet resultSet, String column) throws SQLException {
Object value = resultSet.getObject(column);
if (value != null) target.put(name, value);
@@ -11,6 +11,8 @@ import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import jakarta.servlet.http.HttpServletRequest;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
@RestController
@RequestMapping("/v1")
@@ -25,19 +27,25 @@ public class TaskController {
public ResponseEntity<Map<String, Object>> create3d(
@org.springframework.web.bind.annotation.PathVariable(required = false) String type,
@RequestHeader(value = "Authorization", required = false) String authorization,
@RequestBody Map<String, Object> input
@RequestHeader(value = "Idempotency-Key", required = false) String idempotencyKey,
@RequestBody Map<String, Object> input,
HttpServletRequest request
) {
String taskType = type == null ? String.valueOf(input == null ? "3d" : input.getOrDefault("taskType", "3d")) : type;
return ResponseEntity.status(HttpStatus.ACCEPTED).body(Map.of("task", tasks.create(authorization, taskType, input)));
return ResponseEntity.status(HttpStatus.ACCEPTED).body(Map.of("task", tasks.create(authorization, taskType, input, idempotencyKey,
ApiRequestMetadata.fromHttp(request.getRemoteAddr()))));
}
@PostMapping("/video/tasks")
public ResponseEntity<Map<String, Object>> createVideo(
@RequestHeader(value = "Authorization", required = false) String authorization,
@RequestBody Map<String, Object> input
@RequestHeader(value = "Idempotency-Key", required = false) String idempotencyKey,
@RequestBody Map<String, Object> input,
HttpServletRequest request
) {
return ResponseEntity.status(HttpStatus.ACCEPTED).body(Map.of("task", tasks.create(authorization, "video", input)));
return ResponseEntity.status(HttpStatus.ACCEPTED).body(Map.of("task", tasks.create(authorization, "video", input, idempotencyKey,
ApiRequestMetadata.fromHttp(request.getRemoteAddr()))));
}
@GetMapping("/tasks/{id}")
@@ -0,0 +1,58 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ArrayNode;
import com.fasterxml.jackson.databind.node.DecimalNode;
import com.fasterxml.jackson.databind.node.ObjectNode;
import com.moyi.controlplane.common.ApiException;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
import java.util.Map;
import java.util.TreeSet;
/** Stable request identity only; never stores or exposes the caller's raw idempotency key. */
record TaskCreationIdentity(String keyHash, String requestHash) {
static TaskCreationIdentity of(ObjectMapper mapper, String key, String type, Map<String,Object> input) {
if (key == null) return new TaskCreationIdentity(null, null);
if (!key.matches("[!-~]{1,128}")) {
throw new ApiException("INVALID_IDEMPOTENCY_KEY", "Idempotency-Key 须为 1–128 个可见 ASCII 字符");
}
try {
ObjectNode request = mapper.createObjectNode();
request.put("taskType", type);
request.set("input", canonical(mapper, mapper.valueToTree(input)));
return new TaskCreationIdentity(hash(key), hash(mapper.writeValueAsString(request)));
} catch (JsonProcessingException | IllegalArgumentException exception) {
throw new ApiException("INVALID_REQUEST", "任务请求无法编码");
}
}
private static JsonNode canonical(ObjectMapper mapper, JsonNode node) {
if (node.isObject()) {
ObjectNode result = mapper.createObjectNode();
var fields = new TreeSet<String>();
node.fieldNames().forEachRemaining(fields::add);
for (String field : fields) result.set(field, canonical(mapper, node.get(field)));
return result;
}
if (node.isArray()) {
ArrayNode result = mapper.createArrayNode();
node.forEach(value -> result.add(canonical(mapper, value)));
return result;
}
if (node.isNumber()) return DecimalNode.valueOf(node.decimalValue().stripTrailingZeros());
return node;
}
private static String hash(String value) {
try {
return HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(value.getBytes(StandardCharsets.UTF_8)));
} catch (NoSuchAlgorithmException exception) {
throw new IllegalStateException("SHA-256 is required by the Java runtime", exception);
}
}
}
@@ -0,0 +1,91 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.usage.UsageAuditService;
import java.time.OffsetDateTime;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.transaction.support.TransactionTemplate;
/** Durable provider observations only. No upstream calls, wallet mutation, retry or refund policy. */
final class TaskOutcomeJournal {
record Completion(String taskId, String status, Map<String,Object> result, String code, String message) { }
private final JdbcTemplate jdbc;
private final ObjectMapper mapper;
private final TransactionTemplate transactions;
private final UsageAuditService usage;
TaskOutcomeJournal(JdbcTemplate jdbc, ObjectMapper mapper, TransactionTemplate transactions, UsageAuditService usage) {
this.jdbc = jdbc;
this.mapper = mapper;
this.transactions = transactions;
this.usage = usage;
}
Completion observeTerminal(String taskId, String status, Map<String,Object> result, String code, String message) {
if (!List.of("succeeded", "failed", "cancelled").contains(status)) throw new IllegalArgumentException("Terminal outcome required");
return transaction(() -> {
var row = jdbc.queryForMap("SELECT status,completion_status,result::text AS result,error::text AS error FROM model_tasks WHERE id=? FOR UPDATE", taskId);
if (!List.of("queued", "running", "reconciling").contains(row.get("status"))) return null;
if (row.get("completion_status") != null) return completion(taskId, row);
jdbc.update("UPDATE model_tasks SET completion_status=?,result=CAST(? AS jsonb),error=CAST(? AS jsonb),updated_at=now() WHERE id=?",
status, json(result), code == null ? null : json(Map.of("code", code, "message", message)), taskId);
return new Completion(taskId, status, result, code, message);
});
}
List<Completion> pendingCompletions() {
return jdbc.queryForList("SELECT id,completion_status,result::text AS result,error::text AS error FROM model_tasks WHERE status IN ('running','reconciling') AND completion_status IS NOT NULL ORDER BY updated_at LIMIT 50")
.stream().map(row -> completion(String.valueOf(row.get("id")), row)).toList();
}
void accepted(String taskId, String providerTaskId, Object keyFingerprint, OffsetDateTime submittedAt, OffsetDateTime deadline) {
transaction(() -> {
int changed = jdbc.update("UPDATE model_tasks SET status='running',provider_task_id=?,provider_key_fingerprint=?,provider_deadline_at=?,error=NULL,reconciliation_required_at=NULL,updated_at=? WHERE id=? AND status IN ('running','reconciling') AND provider_task_id IS NULL AND completion_status IS NULL",
providerTaskId, keyFingerprint, deadline, submittedAt, taskId);
if (changed > 0) usage.updateTask(taskId, "running", null, null, null);
return null;
});
}
void requireReconciliation(String taskId, String code, String message, boolean interruptedClaim) {
transaction(() -> {
String claimGuard = interruptedClaim
? " AND provider_task_id IS NULL AND updated_at < now()-interval '2 minutes'" : "";
int changed = jdbc.update("UPDATE model_tasks SET status='reconciling',error=CAST(? AS jsonb),reconciliation_required_at=now(),updated_at=now() WHERE id=? AND status IN ('queued','running') AND completion_status IS NULL" + claimGuard,
json(Map.of("code", code, "message", message, "retryable", false, "requiresReconciliation", true)), taskId);
if (changed > 0) {
usage.updateTask(taskId, "reconciling", null, code, message);
jdbc.update("UPDATE usage_events SET cost_status='pending',cost_amount=NULL WHERE provider_task_id=? AND event_kind='task'", taskId);
}
return null;
});
}
private Completion completion(String taskId, Map<String,Object> row) {
Map<String,Object> error = map(row.get("error"));
return new Completion(taskId, String.valueOf(row.get("completion_status")), map(row.get("result")),
error == null ? null : (String) error.get("code"), error == null ? null : (String) error.get("message"));
}
private Map<String,Object> map(Object value) {
if (value == null) return null;
try { return mapper.readValue(String.valueOf(value), new TypeReference<>() {}); }
catch (JsonProcessingException exception) { throw new IllegalStateException("Invalid persisted provider observation"); }
}
private String json(Object value) {
if (value == null) return null;
try { return mapper.writeValueAsString(value); }
catch (JsonProcessingException exception) { throw new IllegalStateException("Provider observation cannot be encoded"); }
}
private <T> T transaction(Supplier<T> work) {
return transactions == null ? work.get() : transactions.execute(status -> work.get());
}
}
@@ -0,0 +1,21 @@
package com.moyi.controlplane.task;
import java.math.BigDecimal;
import java.time.OffsetDateTime;
import java.util.Map;
import java.util.UUID;
/** Administrative evidence and optimistic-concurrency preconditions, never a raw provider key. */
public record TaskReconciliationCommand(
UUID requestId,
String action,
OffsetDateTime expectedUpdatedAt,
String evidenceReference,
String note,
String providerTaskId,
String credentialFingerprint,
BigDecimal expectedCost,
Map<String,Object> result
) {
@Override public String toString() { return "TaskReconciliationCommand[requestId=" + requestId + ", action=" + action + "]"; }
}
@@ -0,0 +1,42 @@
package com.moyi.controlplane.task;
import com.moyi.controlplane.auth.AuthController;
import com.moyi.controlplane.common.ApiException;
import jakarta.servlet.http.HttpServletRequest;
import java.util.Map;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/api/admin/task-reconciliation")
public class TaskReconciliationController {
private final TaskReconciliationService reconciliations;
public TaskReconciliationController(TaskReconciliationService reconciliations) { this.reconciliations = reconciliations; }
@GetMapping
public Map<String,Object> list(@RequestParam(defaultValue="1") int page, HttpServletRequest request) {
return reconciliations.list(AuthController.sessionToken(request), page);
}
@GetMapping("/{taskId}")
public Map<String,Object> detail(@PathVariable String taskId, HttpServletRequest request) {
return reconciliations.detail(AuthController.sessionToken(request), taskId);
}
@PostMapping(value="/{taskId}", consumes="application/json")
public Map<String,Object> reconcile(@PathVariable String taskId, @RequestBody TaskReconciliationCommand command, HttpServletRequest request) {
// A custom header + JSON requires CORS preflight; the header is a CSRF barrier, not authentication.
if (!"console".equals(request.getHeader("X-Moyi-Source"))) throw new ApiException("CONSOLE_REQUEST_REQUIRED", "请通过管理控制台发起核账", 403);
return reconciliations.reconcile(AuthController.sessionToken(request), taskId, command, AuthController.meta(request));
}
@PostMapping(value="/{taskId}/quote", consumes="application/json")
public Map<String,Object> quote(@PathVariable String taskId, @RequestBody Map<String,Object> result, HttpServletRequest request) {
return reconciliations.quote(AuthController.sessionToken(request), taskId, result);
}
}
@@ -0,0 +1,64 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import java.math.BigDecimal;
import java.net.URI;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Set;
/** Only normalized assets and numeric usage may enter a manual completion, never raw provider payloads. */
final class TaskReconciliationResult {
private static final Set<String> ASSETS = Set.of("url", "modelUrl", "videoUrl", "imageUrl", "audioUrl", "previewUrl");
private static final Set<String> OPENAI = Set.of("input_tokens", "output_tokens", "total_tokens", "cached_tokens", "input_tokens_details");
private static final Set<String> DETAILS = Set.of("text_tokens", "image_tokens", "cached_tokens", "cached_tokens_details");
private static final Set<String> CACHED = Set.of("text_tokens", "image_tokens");
private static final Set<String> GEMINI = Set.of("promptTokenCount", "candidatesTokenCount", "thoughtsTokenCount", "cachedContentTokenCount", "totalTokenCount");
static Map<String,Object> normalize(Map<String,Object> input, ObjectMapper json) {
if (input == null) throw invalid();
try { if (json.writeValueAsBytes(input).length > 32_768) throw invalid(); }
catch (JsonProcessingException exception) { throw invalid(); }
var result = new LinkedHashMap<String,Object>();
for (var entry : input.entrySet()) {
String field = entry.getKey();
Object value = entry.getValue();
if (ASSETS.contains(field)) {
if (!(value instanceof String text) || text.length() > 4096 || text.chars().anyMatch(Character::isISOControl)) throw invalid();
try {
URI uri = URI.create(text);
if (!("https".equalsIgnoreCase(uri.getScheme()) || "http".equalsIgnoreCase(uri.getScheme())) || uri.getHost() == null || uri.getUserInfo() != null) throw invalid();
} catch (IllegalArgumentException exception) { throw invalid(); }
result.put(field, text);
} else if (Set.of("format", "mimeType").contains(field)) {
if (!(value instanceof String text) || text.isBlank() || text.length() > 128 || text.chars().anyMatch(Character::isISOControl)) throw invalid();
result.put(field, text);
} else if ("usage".equals(field)) result.put(field, numericMap(value, OPENAI));
else if ("usageMetadata".equals(field)) result.put(field, numericMap(value, GEMINI));
else throw invalid();
}
return result;
}
private static Map<String,Object> numericMap(Object value, Set<String> allowed) {
if (!(value instanceof Map<?,?> map)) throw invalid();
var result = new LinkedHashMap<String,Object>();
for (var entry : map.entrySet()) {
if (!(entry.getKey() instanceof String name) || !allowed.contains(name)) throw invalid();
if ("input_tokens_details".equals(name)) result.put(name, numericMap(entry.getValue(), DETAILS));
else if ("cached_tokens_details".equals(name)) result.put(name, numericMap(entry.getValue(), CACHED));
else {
try {
long count = new BigDecimal(String.valueOf(entry.getValue())).longValueExact();
if (count < 0 || count > 100_000_000) throw invalid();
result.put(name, count);
} catch (NumberFormatException | ArithmeticException exception) { throw invalid(); }
}
}
return result;
}
private static ApiException invalid() { return new ApiException("INVALID_RECONCILIATION_RESULT", "只接受有界的 HTTP(S) 结果地址和规范化数值用量,不接受原始上游响应或密钥", 422); }
}
@@ -0,0 +1,216 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.auth.AuthService;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.usage.UsageAuditService;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
/** Evidence-backed administrative decisions; no generation, arbitrary balance adjustment or transparent replay. */
@Service
public class TaskReconciliationService {
private final JdbcTemplate jdbc;
private final ObjectMapper json;
private final AuthService auth;
private final TaskSettlementService settlements;
private final AsyncProviderClient provider;
private final GatewayGuard guard;
private final UsageAuditService usage;
private final TransactionTemplate transactions;
private final long timeoutMs;
// List/detail must not load prompts, original inputs or potentially large generated media.
private static final String TASK_COLUMNS = "SELECT t.id,t.user_id,t.api_key_id,t.model_id,t.task_type,t.status,t.updated_at,t.reserved_amount,t.cost_amount,t.provider_id_snapshot,t.upstream_model_snapshot,t.provider_task_id,t.provider_key_fingerprint,t.completion_status,t.pricing_snapshot::text AS frozen_pricing,t.error::text AS error FROM model_tasks t";
public TaskReconciliationService(JdbcTemplate jdbc, ObjectMapper json, AuthService auth,
TaskSettlementService settlements, AsyncProviderClient provider, GatewayGuard guard,
UsageAuditService usage, PlatformTransactionManager manager,
@Value("${moyi.tasks.provider-timeout-ms:3600000}") long timeoutMs) {
this.jdbc = jdbc; this.json = json; this.auth = auth; this.settlements = settlements;
this.provider = provider; this.guard = guard; this.usage = usage;
this.transactions = new TransactionTemplate(manager);
if (timeoutMs < 1000) throw new IllegalArgumentException("Polling deadline must be positive");
this.timeoutMs = timeoutMs;
}
public Map<String,Object> list(String token, int requestedPage) {
auth.requireAdmin(token);
int page = Math.max(1, Math.min(100000, requestedPage));
return Map.of("records", jdbc.queryForList(TASK_COLUMNS + " WHERE t.status='reconciling' ORDER BY t.reconciliation_required_at NULLS FIRST,t.id LIMIT 50 OFFSET ?", (page - 1) * 50).stream().map(this::view).toList(),
"total", jdbc.queryForObject("SELECT count(*) FROM model_tasks WHERE status='reconciling'", Long.class), "page", page);
}
public Map<String,Object> detail(String token, String taskId) {
auth.requireAdmin(token);
Map<String,Object> task = task(taskId, false);
var history = jdbc.queryForList("SELECT request_id,actor_user_id,action,evidence_reference,note,before_state::text AS before_state,after_state::text AS after_state,created_at FROM task_reconciliations WHERE task_id=? ORDER BY created_at DESC LIMIT 50", taskId);
var records = history.stream().map(row -> {
var item = new LinkedHashMap<String,Object>();
for (String field : List.of("request_id", "actor_user_id", "action", "evidence_reference", "note", "created_at")) item.put(field, row.get(field));
item.put("before_state", decode(row.get("before_state")));
item.put("after_state", decode(row.get("after_state")));
return item;
}).toList();
return Map.of("task", view(task), "history", records);
}
public Map<String,Object> quote(String token, String taskId, Map<String,Object> input) {
auth.requireAdmin(token);
var task = task(taskId, false);
if (!"reconciling".equals(task.get("status")) || task.get("completion_status") != null || task.get("cost_amount") != null) throw conflict("任务不处于可核账状态,请刷新");
var result = TaskReconciliationResult.normalize(input, json);
return Map.of("expectedCost", confirmedCharge(task, result).toPlainString(),
"expectedUpdatedAt", timestamp(task.get("updated_at")).toInstant().toString());
}
public Map<String,Object> reconcile(String token, String taskId, TaskReconciliationCommand command, AuthService.RequestMeta meta) {
var actor = auth.requireAdmin(token); // Authorization is required even for replay.
validate(command);
Map<String,Object> result = "confirm_succeeded".equals(command.action()) ? TaskReconciliationResult.normalize(command.result(), json) : null;
String requestHash = requestHash(command, taskId, result);
return transactions.execute(status -> {
// Keep role/status revocation ordered with this financial decision, not a stale pre-check.
if (jdbc.queryForList("SELECT id FROM users WHERE id=? AND status='active' AND role IN ('admin','super_admin') FOR SHARE", actor.id()).isEmpty()) throw new ApiException("FORBIDDEN", "管理员权限已变化,请重新登录", 403);
auth.requireAdmin(token); // Recheck session revocation/expiry after entering the transaction.
jdbc.queryForList("SELECT pg_advisory_xact_lock(hashtextextended(?,0))", "moyi:task-reconciliation:" + command.requestId());
var prior = jdbc.queryForList("SELECT actor_user_id,request_hash,after_state::text AS after_state FROM task_reconciliations WHERE request_id=?", command.requestId());
if (!prior.isEmpty()) {
var receipt = prior.getFirst();
if (!actor.id().equals(receipt.get("actor_user_id")) || !requestHash.equals(receipt.get("request_hash"))) throw conflict("此核账请求编号已用于不同的操作");
return Map.of("task", decode(receipt.get("after_state")), "requestId", command.requestId(), "replayed", true);
}
var task = task(taskId, true);
if (!"reconciling".equals(task.get("status")) || task.get("completion_status") != null || task.get("cost_amount") != null) throw conflict("任务不处于可人工核账状态,或已有可信终态;请刷新");
if (!timestamp(task.get("updated_at")).toInstant().equals(command.expectedUpdatedAt().toInstant())) throw conflict("任务已被其他操作更新,请刷新后重新核对");
var wallets = jdbc.queryForList("SELECT balance,reserved_balance FROM user_wallets WHERE user_id=? FOR UPDATE", task.get("user_id"));
if (wallets.isEmpty() || ((BigDecimal) wallets.getFirst().get("reserved_balance")).compareTo((BigDecimal) task.get("reserved_amount")) < 0) throw conflict("任务冻结额度与钱包不一致,请先排查账本,不允许直接覆盖");
Map<String,Object> before = view(task);
if ("resume_polling".equals(command.action())) resume(task, command);
else {
if ("confirm_succeeded".equals(command.action())) {
BigDecimal charge = confirmedCharge(task, result);
if (charge == null || charge.compareTo(command.expectedCost()) != 0) throw conflict("确认金额与任务原价格快照及实际用量不一致");
}
String target = "confirm_succeeded".equals(command.action()) ? "succeeded" : "failed";
String code = "failed".equals(target) ? "TASK_RECONCILED_FAILED" : null;
String message = "failed".equals(target) ? "管理员已根据上游记录确认任务失败,冻结额度已释放" : null;
// The observed outcome, shared wallet transaction and both audit receipts commit together.
jdbc.update("UPDATE model_tasks SET completion_status=? WHERE id=?", target, taskId);
settlements.finish(taskId, target, "succeeded".equals(target) ? 100 : 0, result, code, message);
}
Map<String,Object> after = view(task(taskId, false));
jdbc.update("INSERT INTO task_reconciliations(request_id,task_id,actor_user_id,action,request_hash,evidence_reference,note,before_state,after_state) VALUES (?,?,?,?,?,?,?,CAST(? AS jsonb),CAST(? AS jsonb))",
command.requestId(), taskId, actor.id(), command.action(), requestHash, command.evidenceReference().trim(), command.note().trim(), encode(before), encode(after));
auth.audit("task.reconciled", actor.id(), meta, Map.of("taskId", taskId, "requestId", command.requestId(), "action", command.action(), "evidenceReference", command.evidenceReference().trim()));
return Map.of("task", after, "requestId", command.requestId(), "replayed", false);
});
}
private BigDecimal confirmedCharge(Map<String,Object> task, Map<String,Object> result) {
try { return settlements.confirmedCharge(task, result); }
catch (ApiException exception) {
if ("TASK_USAGE_PENDING".equals(exception.getCode())) throw new ApiException("TASK_USAGE_REQUIRED", "成功核账必须提供完整可核验用量;任务与冻结额度保持不变", 422);
throw exception;
}
}
private void resume(Map<String,Object> task, TaskReconciliationCommand command) {
String originalId = (String) task.get("provider_task_id");
if (originalId != null && !originalId.equals(command.providerTaskId())) throw conflict("不能将已受理任务改绑到另一个上游编号");
String fingerprint = (String) task.get("provider_key_fingerprint");
if (fingerprint != null && !fingerprint.equals(command.credentialFingerprint())) throw conflict("必须继续使用创建任务时的账号指纹,不能切换账号");
if (task.get("provider_id_snapshot") == null || task.get("upstream_model_snapshot") == null) throw conflict("历史任务原执行路由无法证明,禁止猜测后恢复查询;请核实终态");
provider.requirePollableBinding(String.valueOf(task.get("provider_id_snapshot")), String.valueOf(task.get("model_id")),
String.valueOf(task.get("upstream_model_snapshot")), String.valueOf(task.get("task_type")), command.credentialFingerprint());
String id = String.valueOf(task.get("id"));
if (!guard.tryTaskUserSlot((java.util.UUID) task.get("api_key_id"), id, timeoutMs + 180000)) throw new ApiException("TASK_USER_BUSY", "用户当前并发已满,尚未恢复轮询,请稍后重试", 429);
jdbc.update("UPDATE model_tasks SET status='running',provider_task_id=?,provider_key_fingerprint=?,provider_deadline_at=?,error=NULL,reconciliation_required_at=NULL,updated_at=now() WHERE id=?",
command.providerTaskId(), command.credentialFingerprint(), OffsetDateTime.now(ZoneOffset.UTC).plus(java.time.Duration.ofMillis(timeoutMs)), id);
usage.updateTask(id, "running", null, null, null);
}
private void validate(TaskReconciliationCommand command) {
if (command == null || command.requestId() == null || command.expectedUpdatedAt() == null) throw new ApiException("RECONCILIATION_PRECONDITION_REQUIRED", "必须提供操作编号和任务版本", 422);
if (command.action() == null || !List.of("resume_polling", "confirm_failed", "confirm_succeeded").contains(command.action())) throw new ApiException("INVALID_RECONCILIATION_ACTION", "核账操作无效", 422);
text(command.evidenceReference(), 3, 500); text(command.note(), 10, 2000);
if ("resume_polling".equals(command.action())) {
text(command.providerTaskId(), 1, 160);
if (!command.providerTaskId().matches("[A-Za-z0-9][A-Za-z0-9._:-]{0,159}")) throw new ApiException("INVALID_PROVIDER_TASK_ID", "上游任务编号不能包含路径、查询串或空白", 422);
if (command.credentialFingerprint() == null || !command.credentialFingerprint().matches("[0-9a-f]{64}")) throw new ApiException("INVALID_CREDENTIAL_FINGERPRINT", "必须提供 SHA-256 账号指纹,不能填写 API Key", 422);
if (command.result() != null || command.expectedCost() != null) throw conflict("恢复轮询不能同时修改结果或扣费金额");
} else {
if (command.providerTaskId() != null || command.credentialFingerprint() != null) throw conflict("终态核账不能改绑上游编号或账号");
if ("confirm_succeeded".equals(command.action())) {
if (command.expectedCost() == null || command.expectedCost().signum() < 0 || command.expectedCost().precision() > 20) throw new ApiException("EXPECTED_COST_REQUIRED", "请确认按原价格快照核算的非负扣费金额", 422);
try { command.expectedCost().setScale(8, RoundingMode.UNNECESSARY); }
catch (ArithmeticException exception) { throw new ApiException("INVALID_EXPECTED_COST", "确认金额最多 8 位小数", 422); }
} else if (command.result() != null || command.expectedCost() != null) throw conflict("确认失败不能附带成功结果或扣费金额");
}
}
private void text(String value, int min, int max) {
if (value == null || value.trim().length() < min || value.trim().length() > max || value.chars().anyMatch(Character::isISOControl)) throw new ApiException("RECONCILIATION_EVIDENCE_REQUIRED", "必须填写有效的上游证据引用与 10–2000 字核账说明;不得包含控制字符", 422);
}
private String requestHash(TaskReconciliationCommand command, String taskId, Map<String,Object> result) {
var body = new LinkedHashMap<String,Object>();
body.put("taskId", taskId); body.put("action", command.action()); body.put("expectedUpdatedAt", command.expectedUpdatedAt().toInstant().toString());
body.put("evidenceReference", command.evidenceReference().trim()); body.put("note", command.note().trim());
body.put("providerTaskId", command.providerTaskId()); body.put("credentialFingerprint", command.credentialFingerprint());
body.put("expectedCost", command.expectedCost()); body.put("result", result);
return TaskCreationIdentity.of(json, command.requestId().toString(), "task-reconciliation", body).requestHash();
}
private Map<String,Object> task(String taskId, boolean lock) {
if (taskId == null || taskId.isBlank() || taskId.length() > 160) throw new ApiException("TASK_NOT_FOUND", "任务不存在", 404);
var rows = jdbc.queryForList(TASK_COLUMNS + " WHERE t.id=?" + (lock ? " FOR UPDATE" : ""), taskId);
if (rows.isEmpty()) throw new ApiException("TASK_NOT_FOUND", "任务不存在", 404);
var task = rows.getFirst();
task.put("pricing_snapshot", task.get("frozen_pricing"));
return task;
}
private Map<String,Object> view(Map<String,Object> task) {
var view = new LinkedHashMap<String,Object>();
view.put("id", task.get("id")); view.put("userId", task.get("user_id")); view.put("modelId", task.get("model_id"));
view.put("taskType", task.get("task_type")); view.put("status", task.get("status"));
view.put("updatedAt", timestamp(task.get("updated_at")).toInstant().toString());
view.put("reservedAmount", ((BigDecimal) task.get("reserved_amount")).toPlainString());
view.put("costAmount", task.get("cost_amount") == null ? null : ((BigDecimal) task.get("cost_amount")).toPlainString());
view.put("providerId", task.get("provider_id_snapshot")); view.put("upstreamModel", task.get("upstream_model_snapshot"));
view.put("providerTaskId", task.get("provider_task_id")); view.put("credentialFingerprint", task.get("provider_key_fingerprint"));
view.put("completionStatus", task.get("completion_status")); view.put("requiresReconciliation", "reconciling".equals(task.get("status")));
view.put("pricing", task.get("frozen_pricing") == null ? null : decode(task.get("frozen_pricing")));
view.put("error", task.get("error") == null ? null : decode(task.get("error")));
return view;
}
private OffsetDateTime timestamp(Object value) {
if (value instanceof OffsetDateTime time) return time;
if (value instanceof java.sql.Timestamp time) return time.toInstant().atOffset(ZoneOffset.UTC);
throw new IllegalStateException("Persisted task timestamp missing");
}
private Map<String,Object> decode(Object value) {
try { return json.readValue(String.valueOf(value), new TypeReference<>() {}); }
catch (JsonProcessingException exception) { throw new IllegalStateException("Persisted reconciliation record cannot be read"); }
}
private String encode(Object value) {
try { return json.writeValueAsString(value); }
catch (JsonProcessingException exception) { throw new IllegalStateException("Reconciliation record cannot be encoded"); }
}
private ApiException conflict(String message) { return new ApiException("RECONCILIATION_CONFLICT", message, 409); }
}
@@ -0,0 +1,125 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.EnterprisePolicies;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.usage.UsageAuditService;
import java.math.BigDecimal;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.function.Supplier;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.support.TransactionTemplate;
/** The one task/wallet/usage settlement transaction, shared by workers and audited reconciliation. */
@Service
public class TaskSettlementService {
private final JdbcTemplate jdbc;
private final ObjectMapper json;
private final GatewayGuard guard;
private final UsageAuditService usage;
private final EnterprisePolicies policies;
private final TransactionTemplate transactions;
public TaskSettlementService(JdbcTemplate jdbc, ObjectMapper json, GatewayGuard guard,
UsageAuditService usage, PlatformTransactionManager manager) {
this.jdbc = jdbc;
this.json = json;
this.guard = guard;
this.usage = usage;
this.policies = new EnterprisePolicies(jdbc, json);
this.transactions = manager == null ? null : new TransactionTemplate(manager);
}
void finish(String taskId, String status, int progress, Map<String,Object> result, String code, String message) {
if (!List.of("succeeded", "failed", "cancelled").contains(status)) throw new IllegalArgumentException("Terminal task state required");
transaction(() -> {
var task = jdbc.queryForMap("SELECT status,completion_status,pricing_snapshot::text AS pricing_snapshot FROM model_tasks WHERE id=? FOR UPDATE", taskId);
if (!List.of("queued", "running", "reconciling").contains(task.get("status"))) return null;
if (task.get("completion_status") != null && !status.equals(task.get("completion_status"))) return null;
AsyncPriceSchedule.Bill bill = null;
String finalCode = code, finalMessage = message;
if ("succeeded".equals(status) && task.get("pricing_snapshot") != null) {
try { bill = AsyncPriceSchedule.bill(pricing(task.get("pricing_snapshot")), result); }
catch (ApiException exception) {
finalCode = "TASK_USAGE_PENDING";
finalMessage = "生成已完成,但上游计费用量未能核对;暂不扣费,等待核账";
}
}
jdbc.update("UPDATE model_tasks SET status=?,progress=?,result=CAST(? AS jsonb),error=CAST(? AS jsonb),reconciliation_required_at=NULL,updated_at=now(),completed_at=now() WHERE id=?",
status, progress, encode(result), finalCode == null ? null : encode(Map.of("code", finalCode, "message", finalMessage)), taskId);
if ("TASK_USAGE_PENDING".equals(finalCode)) {
release(taskId);
jdbc.update("UPDATE model_tasks SET cost_amount=NULL WHERE id=?", taskId);
var observed = new LinkedHashMap<String,Object>();
if (result != null) for (String field : List.of("usage", "usageMetadata")) if (result.get(field) instanceof Map<?,?> value) observed.put(field, value);
jdbc.update("UPDATE usage_events SET cost_amount=NULL,cost_status='pending',observed_usage=CAST(? AS jsonb) WHERE provider_task_id=? AND event_kind='task'", encode(observed), taskId);
} else if ("succeeded".equals(status)) settle(taskId, bill);
else release(taskId);
usage.updateTask(taskId, status, result, finalCode, finalMessage);
return null;
});
}
/** Strict calculation for explicit administrator confirmation: missing usage is an error, not a refund. */
BigDecimal confirmedCharge(Map<String,Object> task, Map<String,Object> result) {
if (task.get("pricing_snapshot") == null) return (BigDecimal) task.get("reserved_amount");
return AsyncPriceSchedule.bill(pricing(task.get("pricing_snapshot")), result).amount();
}
private void settle(String taskId, AsyncPriceSchedule.Bill bill) {
var task = jdbc.queryForMap("SELECT user_id,reserved_amount,cost_amount FROM model_tasks WHERE id=? FOR UPDATE", taskId);
BigDecimal reserved = (BigDecimal) task.get("reserved_amount");
BigDecimal amount = bill == null ? reserved : bill.amount();
releasePolicies(taskId, task);
if (amount == null || task.get("cost_amount") != null) return;
UUID userId = (UUID) task.get("user_id");
var wallet = jdbc.queryForMap("SELECT balance,reserved_balance FROM user_wallets WHERE user_id=? FOR UPDATE", userId);
BigDecimal before = (BigDecimal) wallet.get("balance"), after = before.subtract(amount);
jdbc.update("UPDATE user_wallets SET balance=?,reserved_balance=GREATEST(0,reserved_balance-?),updated_at=now() WHERE user_id=?", after, reserved, userId);
if (amount.signum() > 0) jdbc.update("INSERT INTO wallet_entries(id,user_id,request_id,amount,balance_after,currency,reason,operation,requested_amount,balance_before,actor_user_id) VALUES (?,?,?,?,?,'CNY',?,'usage',?,?,NULL)",
UUID.randomUUID(), userId, taskRequestId(taskId), amount.negate(), after, "异步模型任务 · " + taskId, amount, before);
jdbc.update("UPDATE model_tasks SET cost_amount=? WHERE id=?", amount, taskId);
jdbc.update("UPDATE usage_events SET cost_amount=? WHERE provider_task_id=? AND event_kind='task'", amount, taskId);
if (bill != null) jdbc.update("UPDATE usage_events SET input_units=?,output_units=?,observed_usage=jsonb_build_object('input_tokens',?::bigint,'output_tokens',?::bigint,'image_input_tokens',?::bigint) WHERE provider_task_id=? AND event_kind='task'",
bill.input(), bill.output(), bill.input(), bill.output(), bill.imageInput(), taskId);
}
private void release(String taskId) {
var task = jdbc.queryForMap("SELECT user_id,reserved_amount,cost_amount FROM model_tasks WHERE id=? FOR UPDATE", taskId);
BigDecimal amount = (BigDecimal) task.get("reserved_amount");
releasePolicies(taskId, task);
if (amount != null && task.get("cost_amount") == null) {
if (amount.signum() > 0) jdbc.update("UPDATE user_wallets SET reserved_balance=GREATEST(0,reserved_balance-?),updated_at=now() WHERE user_id=?", amount, task.get("user_id"));
jdbc.update("UPDATE model_tasks SET cost_amount=0 WHERE id=?", taskId);
jdbc.update("UPDATE usage_events SET cost_amount=0 WHERE provider_task_id=? AND event_kind='task'", taskId);
}
}
private void releasePolicies(String taskId, Map<String,Object> task) {
policies.release(taskId);
guard.releaseTaskUserSlot((UUID) task.get("user_id"), taskId);
}
private Map<String,Object> pricing(Object value) {
try { return json.readValue(String.valueOf(value), new TypeReference<>() {}); }
catch (JsonProcessingException exception) { throw new ApiException("INVALID_PRICE_SNAPSHOT", "任务价格快照无法核验", 409); }
}
private String encode(Object value) {
if (value == null) return null;
try { return json.writeValueAsString(value); }
catch (JsonProcessingException exception) { throw new IllegalStateException("Task settlement cannot be encoded"); }
}
private UUID taskRequestId(String taskId) { return UUID.nameUUIDFromBytes(("moyi-async:" + taskId).getBytes(StandardCharsets.UTF_8)); }
private <T> T transaction(Supplier<T> work) { return transactions == null ? work.get() : transactions.execute(status -> work.get()); }
}
@@ -8,6 +8,7 @@ import com.fasterxml.jackson.databind.node.ObjectNode;
import com.moyi.controlplane.auth.AuthService;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
import java.nio.charset.StandardCharsets;
import java.sql.ResultSet;
import java.sql.SQLException;
@@ -50,14 +51,22 @@ public class UsageAuditService {
}
public void startChat(UUID id, UUID userId, UUID keyId, Map<String,Object> model, Map<String,Object> input) {
String agent=null, source="api";
startChat(id,userId,keyId,model,input,ApiRequestMetadata.internal());
}
public void startChat(UUID id, UUID userId, UUID keyId, Map<String,Object> model, Map<String,Object> input,
ApiRequestMetadata metadata) {
String agent=null, source=source(metadata);
String ip=metadata == null ? null : metadata.clientIp();
var attributes=org.springframework.web.context.request.RequestContextHolder.getRequestAttributes();
if(attributes instanceof org.springframework.web.context.request.ServletRequestAttributes servlet) {
agent=servlet.getRequest().getHeader("User-Agent");
if("playground".equals(servlet.getRequest().getHeader("X-Moyi-Source"))) source="playground";
var request=servlet.getRequest();
agent=request.getHeader("User-Agent");
if(ip==null) ip=request.getRemoteAddr();
if(metadata==null || metadata.source()==ApiRequestMetadata.Source.INTERNAL) source=isPlaygroundRequest(request)?"playground":"api";
}
jdbc.update("INSERT INTO usage_events(id,request_id,user_id,api_key_id,model_id,provider_id,event_kind,status,input_payload,endpoint,call_mode,request_source,user_agent,reasoning_effort) VALUES (?,?,?,?,?,?,'call','running',?::jsonb,'/v1/chat/completions',?,?,?,?)",
id,id.toString(),userId,keyId,model.get("id"),model.get("providerId"),json(input),Boolean.TRUE.equals(input.get("stream"))?"stream":"sync",source,
jdbc.update("INSERT INTO usage_events(id,request_id,user_id,api_key_id,model_id,provider_id,event_kind,status,input_payload,endpoint,call_mode,request_source,source_ip,user_agent,reasoning_effort) VALUES (?,?,?,?,?,?,'call','running',?::jsonb,'/v1/chat/completions',?,?,CAST(? AS inet),?,?)",
id,id.toString(),userId,keyId,model.get("id"),model.get("providerId"),json(input),Boolean.TRUE.equals(input.get("stream"))?"stream":"sync",source,blankToNull(ip),
agent==null?null:agent.substring(0,Math.min(1000,agent.length())),input.get("reasoning_effort")==null?null:String.valueOf(input.get("reasoning_effort")).substring(0,Math.min(32,String.valueOf(input.get("reasoning_effort")).length())));
}
@@ -83,9 +92,18 @@ public class UsageAuditService {
public void recordTask(
UUID userId, UUID apiKeyId, String modelId, String providerId, String taskId,
String status, Object input, Object output, String errorCode, String errorMessage
) {
recordTask(userId,apiKeyId,modelId,providerId,taskId,status,input,output,errorCode,errorMessage,ApiRequestMetadata.internal());
}
public void recordTask(
UUID userId, UUID apiKeyId, String modelId, String providerId, String taskId,
String status, Object input, Object output, String errorCode, String errorMessage,
ApiRequestMetadata metadata
) {
record(userId, apiKeyId, modelId, providerId, "task", status, null, 0, 0,
input, output, taskId, errorCode, errorMessage, null, null);
input, output, taskId, errorCode, errorMessage, metadata == null ? null : metadata.clientIp(), null,
metadata == null ? null : source(metadata));
}
public void updateTask(String taskId, String status, Object output, String errorCode, String errorMessage) {
@@ -139,15 +157,17 @@ public class UsageAuditService {
WITH pending AS (
SELECT model_id,
COUNT(*) FILTER (WHERE status='running') AS running,
COUNT(*) FILTER (WHERE status='queued') AS queued
FROM model_tasks WHERE status IN ('queued','running') GROUP BY model_id
COUNT(*) FILTER (WHERE status='queued') AS queued,
COUNT(*) FILTER (WHERE status='reconciling') AS reconciling
FROM model_tasks WHERE status IN ('queued','running','reconciling') GROUP BY model_id
)
SELECT m.id, m.public_name, m.source_label, p.name AS provider_name, m.call_mode,
CASE WHEN m.max_concurrency = 0 THEN 0
ELSE COALESCE(m.max_concurrency, ?) END AS max_concurrency,
(m.status='active' AND m.deleted_at IS NULL AND p.status='active' AND p.deleted_at IS NULL
AND d.status='active' AND d.deleted_at IS NULL) AS route_enabled,
COALESCE(pending.running, 0) AS running, COALESCE(pending.queued, 0) AS queued
COALESCE(pending.running, 0) AS running, COALESCE(pending.queued, 0) AS queued,
COALESCE(pending.reconciling, 0) AS reconciling
FROM model_catalog m JOIN model_providers p ON p.id=m.provider_id
JOIN model_definitions d ON d.id=m.logical_model_id
LEFT JOIN pending ON pending.model_id=m.id
@@ -164,6 +184,7 @@ public class UsageAuditService {
group.put("limit", modelGuardEnabled ? rs.getInt("max_concurrency") : 0);
group.put("running", rs.getLong("running"));
group.put("queued", rs.getLong("queued"));
group.put("reconciling", rs.getLong("reconciling"));
return group;
}, defaultModelLimit);
List<String> synchronousIds = groups.stream().filter(group -> !"async".equals(group.get("mode"))).map(group -> group.get("id").toString()).toList();
@@ -175,12 +196,13 @@ public class UsageAuditService {
group.put("running", current == null ? null : current.running());
group.put("queued", current == null ? null : current.queued());
}
// Keep disabled groups visible while their queued or running work still exists.
// Keep disabled groups visible while any work or unresolved financial hold still exists.
groups = groups.stream().filter(group -> Boolean.TRUE.equals(group.get("routeEnabled"))
|| positive(group.get("running")) || positive(group.get("queued"))).toList();
|| positive(group.get("running")) || positive(group.get("queued")) || positive(group.get("reconciling"))).toList();
result.put("capacityGroups", groups);
result.put("running", totalCapacity(groups, "running"));
result.put("waiting", totalCapacity(groups, "queued"));
result.put("reconciling", totalCapacity(groups, "reconciling"));
result.put("capacityConfig", Map.of(
"queueWaitMs", gatewayGuard == null ? 30000 : gatewayGuard.queueWaitMs(),
"defaultModelConcurrency", defaultModelLimit,
@@ -345,14 +367,20 @@ public class UsageAuditService {
}
private void record(UUID userId, UUID keyId, String modelId, String providerId, String kind, String status, Integer latency, long inputUnits, long outputUnits, Object input, Object output, String taskId, String errorCode, String errorMessage, String ip, String userAgent) {
record(userId,keyId,modelId,providerId,kind,status,latency,inputUnits,outputUnits,input,output,taskId,errorCode,errorMessage,ip,userAgent,null);
}
private void record(UUID userId, UUID keyId, String modelId, String providerId, String kind, String status, Integer latency, long inputUnits, long outputUnits, Object input, Object output, String taskId, String errorCode, String errorMessage, String ip, String userAgent, String explicitSource) {
String endpoint=null, source=null;
var attributes=org.springframework.web.context.request.RequestContextHolder.getRequestAttributes();
if(attributes instanceof org.springframework.web.context.request.ServletRequestAttributes servlet) {
var request=servlet.getRequest();
endpoint=request.getRequestURI();
userAgent=request.getHeader("User-Agent");
source="playground".equals(request.getHeader("X-Moyi-Source")) ? "playground" : "api";
if(ip==null) ip=request.getRemoteAddr();
source=explicitSource == null ? (isPlaygroundRequest(request) ? "playground" : "api") : explicitSource;
}
if(source==null) source=explicitSource;
JsonNode payload=objectMapper.valueToTree(output);
Long cached=usageNumber(payload,"/usage/prompt_tokens_details/cached_tokens");
if(cached==null) cached=usageNumber(payload,"/usage/prompt_cache_hit_tokens");
@@ -370,6 +398,15 @@ public class UsageAuditService {
return value.isIntegralNumber() && value.canConvertToLong() && value.longValue()>=0 ? value.longValue() : null;
}
/** Source attribution is based on the matched server route, not a spoofable client header. */
private boolean isPlaygroundRequest(jakarta.servlet.http.HttpServletRequest request) {
return request.getRequestURI() != null && request.getRequestURI().startsWith("/api/playground/keys/");
}
private String source(ApiRequestMetadata metadata) {
return metadata != null && metadata.source()==ApiRequestMetadata.Source.PLAYGROUND ? "playground" : "api";
}
private String json(Object value) {
if (value == null) return null;
try {
@@ -3,6 +3,7 @@ package com.moyi.controlplane.wallet;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.ChatMeter;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
import com.moyi.controlplane.usage.UsageAuditService;
import java.math.BigDecimal;
import java.nio.charset.StandardCharsets;
@@ -12,19 +13,40 @@ import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.transaction.support.TransactionTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@Service
public class ChatBillingService {
private static final Logger log = LoggerFactory.getLogger(ChatBillingService.class);
private final JdbcTemplate jdbc;
private final ObjectMapper json;
private final UsageAuditService usage;
private final com.moyi.controlplane.gateway.EnterprisePolicies policies;
public ChatBillingService(JdbcTemplate jdbc, ObjectMapper json, UsageAuditService usage) { this.jdbc=jdbc; this.json=json; this.usage=usage; this.policies=new com.moyi.controlplane.gateway.EnterprisePolicies(jdbc,json); }
private final TransactionTemplate recoveryTransactions;
public ChatBillingService(JdbcTemplate jdbc, ObjectMapper json, UsageAuditService usage) {
this.jdbc=jdbc; this.json=json; this.usage=usage; this.policies=new com.moyi.controlplane.gateway.EnterprisePolicies(jdbc,json);
this.recoveryTransactions = jdbc.getDataSource() == null ? null
: new TransactionTemplate(new DataSourceTransactionManager(jdbc.getDataSource()));
}
public record Ticket(UUID id, UUID userId, ChatMeter.Rates rates) {}
public void endpoint(UUID id,String endpoint) {jdbc.update("UPDATE usage_events SET endpoint=? WHERE id=?",endpoint,id);}
public void observedUsage(Ticket ticket,String protocol,Object observed) {
if(observed==null) return;
usage.observeUsage(ticket.id(),protocol,observed);
try { usage.observeUsage(ticket.id(),protocol,observed); }
catch (RuntimeException exception) {
// Usage diagnostics are best-effort; canonical normalized usage is persisted before settlement.
log.warn("Observed provider usage could not be persisted ({})", exception.getClass().getSimpleName());
}
}
/** Persist a minimal, normalized meter before attempting the atomic ledger transaction. */
@Transactional
public void prepareSettlement(Ticket ticket, Map<String,Object> normalizedUsage, int latencyMs, Integer firstTokenMs) {
jdbc.update("UPDATE chat_billing_requests SET settlement_usage=?::jsonb, settlement_latency_ms=?, settlement_first_token_ms=?, expires_at=GREATEST(expires_at,now()+interval '5 minutes') WHERE id=? AND status='pending'",
encode(normalizedUsage), Math.max(0,latencyMs), firstTokenMs, ticket.id());
}
@Transactional public void actualRoute(Ticket ticket,Map<String,Object> route) {
var snapshots=jdbc.queryForList("SELECT procurement_pricing FROM model_price_history WHERE model_id=? ORDER BY id DESC LIMIT 1",route.get("id"));
@@ -36,6 +58,12 @@ public class ChatBillingService {
@Transactional
public Ticket reserve(Map<String,Object> key, Map<String,Object> model, ChatMeter.Request request, String idempotencyKey) {
return reserve(key,model,request,idempotencyKey,ApiRequestMetadata.internal());
}
@Transactional
public Ticket reserve(Map<String,Object> key, Map<String,Object> model, ChatMeter.Request request,
String idempotencyKey, ApiRequestMetadata metadata) {
UUID userId=UUID.fromString(key.get("userId").toString());
var versions=jdbc.queryForList("SELECT id,pricing,procurement_pricing FROM model_price_history WHERE model_id=? ORDER BY id DESC LIMIT 1",model.get("id"));
var version=versions.isEmpty()?Map.<String,Object>of():versions.getFirst();
@@ -58,7 +86,7 @@ public class ChatBillingService {
if(available.compareTo(hold)<0) throw new ApiException("INSUFFICIENT_BALANCE","可用余额不足以预留本次调用费用,请充值或调低输出 Token 上限",402);
UUID id=UUID.randomUUID();
policies.reserve(id.toString(),userId,UUID.fromString(key.get("id").toString()),hold);
usage.startChat(id,userId,UUID.fromString(key.get("id").toString()),model,request.body());
usage.startChat(id,userId,UUID.fromString(key.get("id").toString()),model,request.body(),metadata);
// Longer than the upstream hard deadline and the concurrency lease, including cleanup margin.
jdbc.update("INSERT INTO chat_billing_requests(id,user_id,idempotency_hash,request_hash,reserved_amount,pricing,expires_at) VALUES (?,?,?,?,?,?::jsonb,now()+interval '5 minutes')",id,userId,idem,requestHash,hold,encode(rates.snapshot()));
jdbc.update("UPDATE chat_billing_requests SET procurement_pricing=?::jsonb,price_version=? WHERE id=?",version.get("procurement_pricing")==null?null:version.get("procurement_pricing").toString(),version.get("id"),id);
@@ -84,7 +112,7 @@ public class ChatBillingService {
BigDecimal procurement=ChatMeter.rates(decode(call.get("procurement_pricing").toString()),false).cost(tokens);
jdbc.update("UPDATE usage_events SET procurement_cost=?,profit_amount=?,cost_status='estimated' WHERE id=?",procurement,cost.subtract(procurement),ticket.id());
}
jdbc.update("UPDATE chat_billing_requests SET status='succeeded',cost_amount=? WHERE id=?",cost,ticket.id());
jdbc.update("UPDATE chat_billing_requests SET status='succeeded',cost_amount=?,settlement_usage=NULL,settlement_latency_ms=NULL,settlement_first_token_ms=NULL WHERE id=?",cost,ticket.id());
policies.release(ticket.id().toString());
return cost;
}
@@ -96,17 +124,39 @@ public class ChatBillingService {
var call=jdbc.queryForMap("SELECT * FROM chat_billing_requests WHERE id=? FOR UPDATE",id);
if(!"pending".equals(call.get("status"))) return;
jdbc.update("UPDATE user_wallets SET reserved_balance=reserved_balance-?,updated_at=now() WHERE user_id=?",call.get("reserved_amount"),call.get("user_id"));
jdbc.update("UPDATE chat_billing_requests SET status='failed',cost_amount=0 WHERE id=?",id);
jdbc.update("UPDATE chat_billing_requests SET status='failed',cost_amount=0,settlement_usage=NULL,settlement_latency_ms=NULL,settlement_first_token_ms=NULL WHERE id=?",id);
usage.finishChat(id,"failed",new ChatMeter.Tokens(0,0,0),latency,null,BigDecimal.ZERO,null,code,message);
jdbc.update("UPDATE usage_events SET cost_status='pending' WHERE id=?",id);
policies.release(id.toString());
}
@Scheduled(fixedDelayString="${moyi.billing.recovery-interval-ms:60000}")
@Transactional
public void recoverExpired() {
var ids=jdbc.queryForList("SELECT id FROM chat_billing_requests WHERE status='pending' AND expires_at<now() ORDER BY expires_at LIMIT 100 FOR UPDATE SKIP LOCKED",UUID.class);
for(UUID id:ids) release(id,"CALL_INTERRUPTED","调用中断,未取得可结算用量,预留额度已释放",0);
var ids=jdbc.queryForList("SELECT id FROM chat_billing_requests WHERE status='pending' AND expires_at<now() ORDER BY expires_at LIMIT 100",UUID.class);
for(UUID id:ids) {
try {
if (recoveryTransactions == null) recoverOne(id);
else recoveryTransactions.executeWithoutResult(status -> recoverOne(id));
} catch (RuntimeException exception) {
// Keep both the pending ticket and its wallet hold. A later scheduler tick retries the durable meter.
log.warn("Pending chat settlement recovery failed ({})", exception.getClass().getSimpleName());
}
}
}
private void recoverOne(UUID id) {
var rows=jdbc.queryForList("SELECT user_id,settlement_usage::text AS settlement_usage,settlement_latency_ms,settlement_first_token_ms FROM chat_billing_requests WHERE id=? AND status='pending' AND expires_at<now() FOR UPDATE SKIP LOCKED",id);
if(rows.isEmpty()) return;
var row=rows.getFirst();
Object usageSnapshot=row.get("settlement_usage");
if(usageSnapshot!=null) {
Object firstToken=row.get("settlement_first_token_ms");
settle(new Ticket(id,(UUID)row.get("user_id"),null),decode(usageSnapshot.toString()),
row.get("settlement_latency_ms")==null?0:((Number)row.get("settlement_latency_ms")).intValue(),
firstToken==null?null:((Number)firstToken).intValue());
} else {
release(id,"CALL_INTERRUPTED","调用中断,未取得可结算用量,预留额度已释放",0);
}
}
private String encode(Object value) { try { return json.writeValueAsString(value); } catch(Exception e) { throw new ApiException("INVALID_REQUEST","请求无法编码"); } }
private Map<String,Object> decode(String value) { try { return json.readValue(value,new com.fasterxml.jackson.core.type.TypeReference<>(){}); } catch(Exception e) { throw new IllegalStateException("Invalid billing snapshot",e); } }
@@ -43,6 +43,7 @@ management:
moyi:
gateway:
redis-prefix: ${MOYI_REDIS_KEY_PREFIX:moyi:}
risk-mode: ${GATEWAY_RISK_MODE:shadow}
requests-per-minute: ${GATEWAY_REQUESTS_PER_MINUTE:60}
reads-per-minute: ${GATEWAY_READS_PER_MINUTE:300}
# 0 disables the MOYI-side model-group cap so channel concurrency can be
@@ -68,7 +69,7 @@ moyi:
secure-cookie: ${AUTH_SECURE_COOKIE:false}
bootstrap:
admin-account: ${ADMIN_ACCOUNT:adimin}
admin-password: ${ADMIN_PASSWORD:admin123}
admin-password: ${ADMIN_PASSWORD:}
admin-display-name: ${ADMIN_DISPLAY_NAME:平台管理员}
credentials:
encryption-key: ${CREDENTIAL_ENCRYPTION_KEY:}
@@ -0,0 +1,5 @@
-- Persist only the normalized meter needed to retry settlement after a transient ledger failure.
-- Prompts, generated content and provider credentials are deliberately excluded.
ALTER TABLE chat_billing_requests ADD COLUMN settlement_usage jsonb;
ALTER TABLE chat_billing_requests ADD COLUMN settlement_latency_ms integer;
ALTER TABLE chat_billing_requests ADD COLUMN settlement_first_token_ms integer;
@@ -0,0 +1,9 @@
-- User-scoped durable identities survive completion; raw Idempotency-Key values are never persisted.
ALTER TABLE model_tasks ADD COLUMN idempotency_hash varchar(64);
ALTER TABLE model_tasks ADD COLUMN request_hash varchar(64);
ALTER TABLE model_tasks ADD CONSTRAINT model_tasks_request_identity_check
CHECK ((idempotency_hash IS NULL AND request_hash IS NULL)
OR (idempotency_hash ~ '^[0-9a-f]{64}$' AND request_hash ~ '^[0-9a-f]{64}$'
AND idempotency_hash IS NOT NULL AND request_hash IS NOT NULL));
CREATE UNIQUE INDEX model_tasks_user_idempotency_idx ON model_tasks(user_id, idempotency_hash)
WHERE idempotency_hash IS NOT NULL;
@@ -0,0 +1,16 @@
-- Unknown provider acceptance is not a failed generation and must retain its financial holds.
ALTER TABLE model_tasks DROP CONSTRAINT model_tasks_status_check;
ALTER TABLE model_tasks ADD CONSTRAINT model_tasks_status_check
CHECK (status IN ('queued','running','reconciling','succeeded','failed','cancelled'));
ALTER TABLE usage_events DROP CONSTRAINT usage_events_status_check;
ALTER TABLE usage_events ADD CONSTRAINT usage_events_status_check
CHECK (status IN ('queued','running','reconciling','succeeded','failed','cancelled'));
ALTER TABLE model_tasks ADD COLUMN reconciliation_required_at timestamptz;
-- A normalized observed terminal outcome survives a crash or transient ledger/audit write failure.
-- The result itself uses the existing result column; no second copy of generated media is stored.
ALTER TABLE model_tasks ADD COLUMN completion_status varchar(16)
CHECK (completion_status IN ('succeeded','failed','cancelled'));
CREATE INDEX model_tasks_pending_completion_idx ON model_tasks(updated_at)
WHERE completion_status IS NOT NULL AND status IN ('running','reconciling');
CREATE INDEX model_tasks_reconciliation_idx ON model_tasks(reconciliation_required_at)
WHERE status='reconciling';
@@ -0,0 +1,23 @@
-- Freeze route identity without copying credentials or mutable provider configuration.
ALTER TABLE model_tasks ADD COLUMN provider_id_snapshot varchar(64) REFERENCES model_providers(id);
ALTER TABLE model_tasks ADD COLUMN upstream_model_snapshot varchar(160);
ALTER TABLE model_tasks ADD CONSTRAINT model_tasks_execution_binding_check CHECK (
(provider_id_snapshot IS NULL AND upstream_model_snapshot IS NULL)
OR (provider_id_snapshot IS NOT NULL AND upstream_model_snapshot IS NOT NULL
AND length(trim(provider_id_snapshot)) > 0 AND length(trim(upstream_model_snapshot)) > 0)
);
-- Never guess the original route of an already-submitted legacy task.
-- Unsubmitted queued work can safely freeze its route at migration time.
UPDATE model_tasks t SET provider_id_snapshot=m.provider_id, upstream_model_snapshot=m.upstream_model
FROM model_catalog m
WHERE t.model_id=m.id AND t.status='queued' AND t.provider_task_id IS NULL;
COMMENT ON COLUMN model_tasks.provider_id_snapshot IS 'Provider identity at admission; NULL for ambiguous legacy accepted tasks';
-- New proven bindings cannot associate the same provider/account task with two local wallets.
-- Legacy accepted rows are deliberately not guessed/backfilled, so this index does not rewrite their history.
CREATE UNIQUE INDEX model_tasks_proven_upstream_identity_idx
ON model_tasks(provider_id_snapshot,provider_key_fingerprint,provider_task_id)
WHERE provider_id_snapshot IS NOT NULL AND provider_key_fingerprint IS NOT NULL
AND provider_task_id IS NOT NULL AND provider_task_id <> 'mock';
@@ -0,0 +1,20 @@
-- Append-only operation receipts; no prompts, generated media or raw credentials.
CREATE TABLE task_reconciliations (
request_id uuid PRIMARY KEY,
task_id varchar(160) NOT NULL REFERENCES model_tasks(id),
actor_user_id uuid NOT NULL REFERENCES users(id),
action varchar(24) NOT NULL CHECK (action IN ('resume_polling','confirm_failed','confirm_succeeded')),
request_hash varchar(64) NOT NULL CHECK (request_hash ~ '^[0-9a-f]{64}$'),
evidence_reference varchar(500) NOT NULL CHECK (length(trim(evidence_reference)) BETWEEN 3 AND 500),
note varchar(2000) NOT NULL CHECK (length(trim(note)) BETWEEN 10 AND 2000),
before_state jsonb NOT NULL,
after_state jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX task_reconciliations_task_idx ON task_reconciliations(task_id,created_at DESC);
CREATE FUNCTION protect_task_reconciliation_history() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
RAISE EXCEPTION 'Task reconciliation history is append-only';
END $$;
CREATE TRIGGER task_reconciliation_history_immutable BEFORE UPDATE OR DELETE ON task_reconciliations
FOR EACH ROW EXECUTE FUNCTION protect_task_reconciliation_history();
@@ -0,0 +1,8 @@
-- Shared across replicas. Raw accounts, IPs and passwords are never written here.
CREATE TABLE auth_login_windows (
bucket_hash varchar(64) PRIMARY KEY CHECK (bucket_hash ~ '^[0-9a-f]{64}$'),
window_started_at timestamptz NOT NULL DEFAULT clock_timestamp(),
expires_at timestamptz NOT NULL DEFAULT (clock_timestamp()+interval '60 seconds'),
attempts integer NOT NULL CHECK (attempts BETWEEN 1 AND 1000)
);
CREATE INDEX auth_login_windows_expiry_idx ON auth_login_windows(expires_at);
@@ -0,0 +1,21 @@
-- One immutable generation per delivery. No plaintext recipient or OTP is stored in this queue.
ALTER TABLE registration_codes ADD COLUMN delivery_id uuid;
ALTER TABLE password_reset_codes ADD COLUMN delivery_id uuid;
CREATE INDEX registration_codes_expiry_idx ON registration_codes(expires_at);
CREATE TABLE verification_delivery_outbox (
id uuid PRIMARY KEY,
purpose varchar(24) NOT NULL CHECK (purpose IN ('registration','password-reset')),
payload_ciphertext text,
status varchar(16) NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','sending','sent','failed','discarded')),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
expires_at timestamptz NOT NULL DEFAULT clock_timestamp()+interval '5 minutes',
started_at timestamptz,
finished_at timestamptz,
failure_code varchar(32),
CHECK ((status IN ('queued','sending') AND payload_ciphertext IS NOT NULL)
OR (status IN ('sent','failed','discarded') AND payload_ciphertext IS NULL))
);
CREATE INDEX verification_delivery_queue_idx ON verification_delivery_outbox(created_at,id) WHERE status='queued';
CREATE INDEX verification_delivery_expiry_idx ON verification_delivery_outbox(expires_at) WHERE status IN ('queued','sending');
CREATE INDEX verification_delivery_finished_idx ON verification_delivery_outbox(finished_at) WHERE status IN ('sent','failed','discarded');
@@ -0,0 +1,14 @@
-- A resumed task renews its own lease instead of consuming a second slot.
local time = redis.call('TIME')
local now = tonumber(time[1]) * 1000 + math.floor(tonumber(time[2]) / 1000)
redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', now)
local existing = redis.call('ZSCORE', KEYS[1], ARGV[3])
if not existing and redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[1]) then
return 0
end
redis.call('ZADD', KEYS[1], now + tonumber(ARGV[2]), ARGV[3])
if redis.call('PTTL', KEYS[1]) < tonumber(ARGV[2]) then
redis.call('PEXPIRE', KEYS[1], ARGV[2])
end
if existing then return 2 end
return 1
@@ -54,6 +54,7 @@ class PlatformAdminPostgresTest {
private CatalogService catalog;
private WalletService wallet;
private VerificationService verification;
private VerificationDispatchService dispatch;
private TestSender sender;
@BeforeEach void setup() throws Exception {
@@ -87,12 +88,14 @@ class PlatformAdminPostgresTest {
context.registerBean(WalletService.class, () -> new WalletService(jdbc, context.getBean(AuthService.class)));
context.registerBean(UsageAuditService.class, () -> new UsageAuditService(jdbc,json));
context.registerBean(ChatBillingService.class, () -> new ChatBillingService(jdbc,json,context.getBean(UsageAuditService.class)));
context.registerBean(VerificationService.class, () -> new VerificationService(jdbc, sender, "test-pepper"));
context.registerBean(VerificationDispatchService.class, () -> new VerificationDispatchService(jdbc,sender,new SecretCipher("test-delivery-encryption"),json,context.getBean(DataSourceTransactionManager.class)));
context.registerBean(VerificationService.class, () -> new VerificationService(jdbc, context.getBean(VerificationDispatchService.class), "test-pepper"));
context.refresh();
auth = context.getBean(AuthService.class);
catalog = context.getBean(CatalogService.class);
wallet = context.getBean(WalletService.class);
verification = context.getBean(VerificationService.class);
dispatch = context.getBean(VerificationDispatchService.class);
auth.ensureAdmin("test_owner", "local-password", "Test Owner");
ownerToken = auth.login("test_owner", "local-password", meta).token();
memberId = auth.createUser(ownerToken, "test_member", "local-password", "Test Member", meta).id();
@@ -308,7 +311,7 @@ class PlatformAdminPostgresTest {
} finally { org.springframework.web.context.request.RequestContextHolder.resetRequestAttributes(); }
var saved=jdbc.queryForMap("SELECT * FROM usage_events WHERE user_id=?",memberId);
assertEquals("/v1/chat/completions",saved.get("endpoint"));
assertEquals("playground",saved.get("request_source"));
assertEquals("api",saved.get("request_source"),"a caller-supplied source header must not reclassify a public API request");
assertEquals("Local test browser",saved.get("user_agent"));
assertEquals(80L,saved.get("cached_input_units"));
assertEquals(5L,saved.get("cache_write_units"));
@@ -323,7 +326,8 @@ class PlatformAdminPostgresTest {
}
@Test void directCreationCannotEscalateAndOtpIsRequiredOnBothChannels() throws Exception {
MockMvc http = MockMvcBuilders.standaloneSetup(new AuthController(auth,verification,168,false)).setControllerAdvice(new ApiExceptionHandler()).build();
var verifiedAccounts = new VerifiedAccountService(auth, verification, context.getBean(DataSourceTransactionManager.class));
MockMvc http = MockMvcBuilders.standaloneSetup(new AuthController(auth,verification,verifiedAccounts,168,false)).setControllerAdvice(new ApiExceptionHandler()).build();
http.perform(post("/api/admin/users").cookie(new Cookie("moyi_session",ownerToken)).contentType("application/json").content("{\"account\":\"created_user\",\"password\":\"test-password\",\"role\":\"super_admin\"}"))
.andExpect(status().isCreated()).andExpect(jsonPath("$.user.role").value("user")).andExpect(jsonPath("$.user.password").doesNotExist());
http.perform(post("/api/admin/users").cookie(new Cookie("moyi_session",memberToken)).contentType("application/json").content("{\"account\":\"bad_user\",\"password\":\"test-password\"}"))
@@ -334,6 +338,7 @@ class PlatformAdminPostgresTest {
http.perform(post("/api/auth/register").contentType("application/json").content(json.writeValueAsBytes(input))).andExpect(status().isBadRequest());
http.perform(post("/api/auth/verification").contentType("application/json").content(json.writeValueAsBytes(input)))
.andExpect(status().isOk()).andExpect(jsonPath("$.code").doesNotExist()).andExpect(jsonPath("$.sent").value(true));
assertTrue(VerificationDeliveryFixture.deliverOne(dispatch));
String code = sender.codes.get(account);
assertNotEquals(code,jdbc.queryForObject("SELECT code_hash FROM registration_codes WHERE account=?",String.class,account));
input.put("verificationCode",code);
@@ -349,16 +354,19 @@ class PlatformAdminPostgresTest {
assertThrows(ApiException.class, () -> verification.send(account,"bad-invite","127.0.0.1"));
verification.send(account,invite,"127.0.0.1");
assertThrows(ApiException.class, () -> verification.send(account,invite,"127.0.0.1"));
assertTrue(VerificationDeliveryFixture.deliverOne(dispatch));
String correct=sender.codes.get(account);
String wrong=correct.equals("000000")?"111111":"000000";
for(int i=0;i<5;i++) assertThrows(ApiException.class, () -> verification.consume(account,wrong));
assertEquals(5,jdbc.queryForObject("SELECT attempts FROM registration_codes WHERE account=?",Integer.class,account));
assertThrows(ApiException.class, () -> verification.consume(account,correct));
verification.send("expired@example.test",invite,"127.0.0.1");
assertTrue(VerificationDeliveryFixture.deliverOne(dispatch));
jdbc.update("UPDATE registration_codes SET expires_at=now()-interval '1 second' WHERE account='expired@example.test'");
assertThrows(ApiException.class, () -> verification.consume("expired@example.test",sender.codes.get("expired@example.test")));
sender.fail=true;
assertThrows(ApiException.class, () -> verification.send("failed@example.test",invite,"127.0.0.1"));
assertEquals(true,verification.send("failed@example.test",invite,"127.0.0.1").get("sent"));
assertTrue(VerificationDeliveryFixture.deliverOne(dispatch));
assertTrue(jdbc.queryForObject("SELECT consumed FROM registration_codes WHERE account='failed@example.test'",Boolean.class));
assertEquals(1,jdbc.queryForObject("SELECT count(*) FROM verification_send_events WHERE account='failed@example.test'",Integer.class));
sender.enabled=false;
@@ -482,7 +490,13 @@ class PlatformAdminPostgresTest {
.withProperty("moyi.providers.deepseek.base-url","http://127.0.0.1:"+upstream.getAddress().getPort()+"/v1");
var provider=new com.moyi.controlplane.provider.OpenAiCompatibleProvider(json,environment,jdbc,new SecretCipher("local-test"));
try {
var controller=new com.moyi.controlplane.gateway.GatewayController(catalog,provider,org.mockito.Mockito.mock(com.moyi.controlplane.gateway.GatewayGuard.class),f.billing(),json);
var guard=org.mockito.Mockito.mock(com.moyi.controlplane.gateway.GatewayGuard.class);
org.mockito.Mockito.doReturn(org.mockito.Mockito.mock(com.moyi.controlplane.gateway.GatewayGuard.Admission.class))
.when(guard).acquire(org.mockito.ArgumentMatchers.anyMap(),org.mockito.ArgumentMatchers.any(java.util.UUID.class),org.mockito.ArgumentMatchers.anyLong());
var requestExecution=new com.moyi.controlplane.gateway.request.GatewayRequestExecutionService(
f.billing(),new com.moyi.controlplane.gateway.request.ApiRequestRiskPolicy(
com.moyi.controlplane.gateway.request.ApiRequestRiskEngine.allowAll()));
var controller=new com.moyi.controlplane.gateway.GatewayController(catalog,provider,guard,requestExecution,json);
var http=MockMvcBuilders.standaloneSetup(controller).setControllerAdvice(new ApiExceptionHandler()).build();
var input=new LinkedHashMap<>(f.request().body());
http.perform(post("/v1/chat/completions").header("Authorization","Bearer "+key).header("Idempotency-Key","retry-test").contentType("application/json").content(json.writeValueAsBytes(input)))
@@ -494,7 +508,7 @@ class PlatformAdminPostgresTest {
.andExpect(status().isOk()).andExpect(content().contentTypeCompatibleWith("text/event-stream")).andReturn().getResponse().getContentAsString();
assertTrue(streamed.endsWith("data: [DONE]\n\n")); assertFalse(streamed.contains("private-alias"));
assertEquals(new BigDecimal("0.99932000"),jdbc.queryForObject("SELECT balance FROM user_wallets WHERE user_id=?",BigDecimal.class,memberId));
assertEquals(1,jdbc.queryForObject("SELECT count(*) FROM usage_events WHERE call_mode='stream' AND request_source='playground' AND cost_amount=0.00034 AND first_token_ms IS NOT NULL",Integer.class));
assertEquals(1,jdbc.queryForObject("SELECT count(*) FROM usage_events WHERE call_mode='stream' AND request_source='api' AND cost_amount=0.00034 AND first_token_ms IS NOT NULL",Integer.class));
mode.set("no-usage");
var failed=http.perform(post("/v1/chat/completions").header("Authorization","Bearer "+key).contentType("application/json").content(json.writeValueAsBytes(input))).andReturn().getResponse().getContentAsString();
assertTrue(failed.contains("UPSTREAM_USAGE_MISSING"));
@@ -530,4 +544,62 @@ class PlatformAdminPostgresTest {
f.billing().fail(ticket,"SETTLEMENT_FAILED","Test rollback",5);
assertEquals(0,jdbc.queryForObject("SELECT reserved_balance FROM user_wallets WHERE user_id=?",BigDecimal.class,memberId).signum());
}
@Test void explicitRequestMetadataPersistsWithoutThreadLocalHttpContext() {
var f=chatFixture(); wallet.grant(ownerToken,memberId,UUID.randomUUID(),"1","metadata test",meta);
var metadata=com.moyi.controlplane.gateway.request.ApiRequestMetadata.playground("198.51.100.7");
var ticket=f.billing().reserve(f.key(),f.model(),f.request(),"metadata",metadata);
var saved=jdbc.queryForMap("SELECT request_source,host(source_ip) AS client_ip FROM usage_events WHERE id=?",ticket.id());
assertEquals("playground",saved.get("request_source"));
assertEquals("198.51.100.7",saved.get("client_ip"));
f.billing().fail(ticket,"TEST_COMPLETE","metadata fixture",1);
var audit=context.getBean(UsageAuditService.class);
audit.recordTask(memberId,UUID.fromString(String.valueOf(f.key().get("id"))),String.valueOf(f.model().get("id")),
String.valueOf(f.model().get("providerId")),"metadata-task","queued",Map.of(),null,null,null,
com.moyi.controlplane.gateway.request.ApiRequestMetadata.fromHttp("203.0.113.8"));
saved=jdbc.queryForMap("SELECT request_source,host(source_ip) AS client_ip FROM usage_events WHERE provider_task_id='metadata-task'");
assertEquals("api",saved.get("request_source"));
assertEquals("203.0.113.8",saved.get("client_ip"));
}
@Test void gatewaySettlementFailureKeepsDurableMeterAndRecoversExactlyOnce() {
var f=chatFixture(); wallet.grant(ownerToken,memberId,UUID.randomUUID(),"1","recoverable settlement test",meta);
var billing=f.billing();
var execution=new com.moyi.controlplane.gateway.request.GatewayRequestExecutionService(billing,
new com.moyi.controlplane.gateway.request.ApiRequestRiskPolicy(
com.moyi.controlplane.gateway.request.ApiRequestRiskEngine.allowAll()));
var requestContext=new com.moyi.controlplane.gateway.request.ApiRequestContext(memberId,
UUID.fromString(String.valueOf(f.key().get("id"))),
com.moyi.controlplane.gateway.request.ApiRequestProtocol.CHAT_COMPLETIONS,
String.valueOf(f.model().get("id")),String.valueOf(f.model().get("id")),false,
f.request().reservedInputTokens(),f.request().maxOutputTokens(),
com.moyi.controlplane.gateway.request.ApiRequestMetadata.internal(),System.nanoTime());
jdbc.execute("ALTER TABLE usage_events ADD CONSTRAINT test_reject_recoverable_cost CHECK (cost_amount IS NULL OR cost_amount=0)");
try {
String result=execution.execute(requestContext,f.key(),f.model(),f.request(),null,
() -> (com.moyi.controlplane.gateway.request.ApiRequestAdmission)()->{},
ticket -> { execution.settle(ticket,meteredResult(),5,null); return "unexpected"; },
failure -> failure instanceof ApiException api ? api : new ApiException("CALL_INTERRUPTED","interrupted",502),
ApiException::getCode);
assertEquals("SETTLEMENT_PENDING",result);
UUID ticketId=jdbc.queryForObject("SELECT id FROM chat_billing_requests WHERE user_id=? ORDER BY created_at DESC LIMIT 1",UUID.class,memberId);
assertEquals("pending",jdbc.queryForObject("SELECT status FROM chat_billing_requests WHERE id=?",String.class,ticketId));
assertNotNull(jdbc.queryForObject("SELECT settlement_usage FROM chat_billing_requests WHERE id=?",Object.class,ticketId));
assertTrue(jdbc.queryForObject("SELECT reserved_balance FROM user_wallets WHERE user_id=?",BigDecimal.class,memberId).signum()>0);
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM wallet_entries WHERE request_id=?",Integer.class,ticketId));
jdbc.execute("ALTER TABLE usage_events DROP CONSTRAINT test_reject_recoverable_cost");
jdbc.update("UPDATE chat_billing_requests SET expires_at=now()-interval '1 second' WHERE id=?",ticketId);
billing.recoverExpired();
billing.recoverExpired();
assertEquals("succeeded",jdbc.queryForObject("SELECT status FROM chat_billing_requests WHERE id=?",String.class,ticketId));
assertEquals(new BigDecimal("0.99966000"),jdbc.queryForObject("SELECT balance FROM user_wallets WHERE user_id=?",BigDecimal.class,memberId));
assertEquals(0,jdbc.queryForObject("SELECT reserved_balance FROM user_wallets WHERE user_id=?",BigDecimal.class,memberId).signum());
assertEquals(1,jdbc.queryForObject("SELECT count(*) FROM wallet_entries WHERE request_id=?",Integer.class,ticketId));
} finally {
jdbc.execute("ALTER TABLE usage_events DROP CONSTRAINT IF EXISTS test_reject_recoverable_cost");
}
}
}
@@ -65,7 +65,9 @@ class AuthPostgresTest {
assertThrows(DataIntegrityViolationException.class, () -> jdbc.update("UPDATE users SET role = 'super_admin' WHERE id = ?", operatorId));
assertThrows(IllegalStateException.class, () -> auth.ensureAdmin("operator", "test-password", "Other"));
MockMvc http = MockMvcBuilders.standaloneSetup(new AuthController(auth, org.mockito.Mockito.mock(VerificationService.class), 168, true))
var verification = org.mockito.Mockito.mock(VerificationService.class);
var verifiedAccounts = new VerifiedAccountService(auth, verification, context.getBean(DataSourceTransactionManager.class));
MockMvc http = MockMvcBuilders.standaloneSetup(new AuthController(auth, verification, verifiedAccounts, 168, true))
.setControllerAdvice(new ApiExceptionHandler()).build();
var ownerCookie = login(http, "adimin", "super_admin");
var operatorCookie = login(http, "operator", "admin");
@@ -32,6 +32,15 @@ class AuthRolesTest {
verify(hasher).hash("test-password");
}
@Test
void firstBootstrapCannotUsePublishedDefaultsOrExamplePlaceholders() {
for (String password : List.of("admin123", "GENERATE_ADMIN_PASSWORD", "replace-with-a-strong-initial-password")) {
assertEquals("INVALID_ADMIN_CONFIG", assertThrows(ApiException.class, () -> auth.ensureAdmin("adimin", password, "Owner")).getCode());
}
verify(jdbc, never()).update(anyString(), any(Object[].class));
verifyNoInteractions(hasher);
}
@Test
void bootstrapPromotesLegacyAdminWithoutReplacingPassword() throws Exception {
lookupUser(ownerId, "admin");
@@ -0,0 +1,180 @@
package com.moyi.controlplane.auth;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.common.ApiExceptionHandler;
import com.moyi.controlplane.security.SecretCipher;
import jakarta.servlet.http.Cookie;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import javax.sql.DataSource;
import org.flywaydb.core.Flyway;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import org.springframework.transaction.support.TransactionTemplate;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
@EnabledIfEnvironmentVariable(named="MOYI_AUTH_TEST_JDBC_URL", matches="jdbc:postgresql://127\\.0\\.0\\.1:[0-9]+/.+")
class AuthSecurityPostgresTest {
@Configuration @EnableTransactionManagement(proxyTargetClass=true) static class Transactions {}
private final UUID userId = UUID.randomUUID();
private final ObjectMapper json = new ObjectMapper().findAndRegisterModules();
private final AuthService.RequestMeta meta = new AuthService.RequestMeta("127.0.0.1", "isolated-security-test");
private DriverManagerDataSource base, source;
private String schema;
private JdbcTemplate jdbc;
private AnnotationConfigApplicationContext context;
private AuthService auth;
private PasswordHasher hasher;
private MockMvc http;
@BeforeEach void setup() {
String url = System.getenv("MOYI_AUTH_TEST_JDBC_URL");
schema = "auth_security_" + UUID.randomUUID().toString().replace("-", "");
base = new DriverManagerDataSource(url, "moyi_role_test", "");
source = new DriverManagerDataSource(url + "?currentSchema=" + schema, "moyi_role_test", "");
Flyway.configure().dataSource(base).schemas(schema).defaultSchema(schema).load().migrate();
jdbc = new JdbcTemplate(source); hasher = spy(new PasswordHasher("isolated-pepper"));
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name,role) VALUES (?,'security_member','username',?,'Test','super_admin')", userId, hasher.hash("original-password"));
context = new AnnotationConfigApplicationContext();
context.register(Transactions.class);
context.registerBean(DataSource.class, () -> source);
context.registerBean(DataSourceTransactionManager.class, () -> new DataSourceTransactionManager(source));
context.registerBean(AuthService.class, () -> new AuthService(jdbc, hasher, json, 168));
context.registerBean(TotpService.class, () -> new TotpService(jdbc, new SecretCipher("isolated-mfa-key")));
context.refresh(); auth = context.getBean(AuthService.class);
var verification = mock(VerificationService.class);
var verifiedAccounts = new VerifiedAccountService(auth, verification, context.getBean(DataSourceTransactionManager.class));
http = MockMvcBuilders.standaloneSetup(new AuthController(auth, verification, verifiedAccounts, 168, true)).setControllerAdvice(new ApiExceptionHandler()).build();
}
@AfterEach void cleanup() {
if (context != null) context.close();
if (base != null && schema != null) new JdbcTemplate(base).execute("DROP SCHEMA " + schema + " CASCADE");
}
private String session() { return auth.login("security_member", "original-password", meta).token(); }
@Test void passwordChangeRevokesEverySessionAndClearsTheBrowserCookie() throws Exception {
String first = session(), second = session();
http.perform(patch("/api/auth/account").cookie(new Cookie("moyi_session", first)).contentType("application/json")
.content("{\"displayName\":\"Test\",\"currentPassword\":\"original-password\",\"newPassword\":\"replacement-password\"}"))
.andExpect(status().isOk()).andExpect(cookie().maxAge("moyi_session", 0));
assertEquals(401, assertThrows(ApiException.class, () -> auth.requireUser(first)).getStatus());
assertEquals(401, assertThrows(ApiException.class, () -> auth.requireUser(second)).getStatus());
assertEquals(401, assertThrows(ApiException.class, () -> session()).getStatus());
assertNotNull(auth.login("security_member", "replacement-password", meta).token());
}
@Test void allRolesHaveTheSamePasswordBoundsAndProfileOnlySaveKeepsSessions() {
String token = session();
assertEquals("WEAK_PASSWORD", assertThrows(ApiException.class, () -> auth.updateAccount(token, "Test", "original-password", "x", meta)).getCode());
assertEquals("WEAK_PASSWORD", assertThrows(ApiException.class, () -> auth.resetPassword("security_member", "short", meta)).getCode());
assertEquals("WEAK_PASSWORD", assertThrows(ApiException.class, () -> auth.register("unused", "new@example.test", "x".repeat(257), "New", meta)).getCode());
auth.updateAccount(token, "Renamed", null, null, meta);
assertEquals("Renamed", auth.requireUser(token).displayName());
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM auth_sessions WHERE revoked_at IS NOT NULL", Integer.class));
}
@Test void failedLoginBudgetIsCommittedAndHttpReturnsRetryAfter() throws Exception {
for (int i = 0; i < 10; i++) assertEquals(401, assertThrows(ApiException.class, () -> auth.login("missing_member", "wrong-password", meta)).getStatus());
http.perform(post("/api/auth/login").contentType("application/json").content("{\"account\":\"missing_member\",\"password\":\"wrong-password\"}"))
.andExpect(status().isTooManyRequests()).andExpect(header().exists("Retry-After"));
assertEquals(2, jdbc.queryForObject("SELECT count(*) FROM auth_login_windows", Integer.class));
assertFalse(jdbc.queryForObject("SELECT string_agg(bucket_hash,',') FROM auth_login_windows", String.class).contains("missing_member"));
jdbc.update("UPDATE auth_login_windows SET expires_at=clock_timestamp()-interval '1 second'");
assertEquals(401, assertThrows(ApiException.class, () -> auth.login("missing_member", "wrong-password", meta)).getStatus());
}
@Test void concurrentLoginBudgetCannotExceedTheAccountLimit() throws Exception {
var start = new CountDownLatch(1);
try (var executor = Executors.newFixedThreadPool(12)) {
var results = new ArrayList<java.util.concurrent.Future<Integer>>();
for (int i = 0; i < 20; i++) results.add(executor.submit(() -> {
assertTrue(start.await(10, TimeUnit.SECONDS));
try { auth.login("missing_member", "wrong-password", meta); return 200; }
catch (ApiException error) { return error.getStatus(); }
}));
start.countDown();
var codes = new ArrayList<Integer>();
for (var result : results) codes.add(result.get(30, TimeUnit.SECONDS));
assertEquals(10, codes.stream().filter(code -> code == 401).count());
assertEquals(10, codes.stream().filter(code -> code == 429).count());
}
}
@Test void sourceBudgetAlsoLimitsChangingUnknownAccounts() {
var limiter = new LoginAttemptLimiter(jdbc);
for (int i = 0; i < 50; i++) limiter.check("different-" + i, "127.0.0.1");
assertEquals(429, assertThrows(ApiException.class, () -> limiter.check("yet-another", "127.0.0.1")).getStatus());
assertDoesNotThrow(() -> limiter.check("yet-another", "127.0.0.2"));
}
@Test void passwordResetWinningTheUserLockPreventsOldPasswordSessionIssuance() throws Exception {
try (var executor = Executors.newSingleThreadExecutor()) {
var started = new CountDownLatch(1);
var future = new java.util.concurrent.atomic.AtomicReference<java.util.concurrent.Future<Integer>>();
new TransactionTemplate(new DataSourceTransactionManager(source)).execute(status -> {
jdbc.queryForList("SELECT id FROM users WHERE id=? FOR UPDATE", userId);
future.set(executor.submit(() -> {
started.countDown();
try { session(); return 200; } catch (ApiException error) { return error.getStatus(); }
}));
try { assertTrue(started.await(10, TimeUnit.SECONDS)); } catch (InterruptedException error) { throw new AssertionError(error); }
auth.resetPassword("security_member", "replacement-password", meta);
return null;
});
assertEquals(401, future.get().get(20, TimeUnit.SECONDS));
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM auth_sessions WHERE revoked_at IS NULL", Integer.class));
}
}
@Test void loginWinningTheLockStillGetsRevokedByFollowingPasswordReset() throws Exception {
var verified = new CountDownLatch(1); var allow = new CountDownLatch(1);
doAnswer(call -> {
verified.countDown(); assertTrue(allow.await(15, TimeUnit.SECONDS)); return call.callRealMethod();
}).when(hasher).matches(eq("original-password"), anyString());
try (var executor = Executors.newFixedThreadPool(2)) {
var login = executor.submit(this::session);
assertTrue(verified.await(10, TimeUnit.SECONDS));
var reset = executor.submit(() -> auth.resetPassword("security_member", "replacement-password", meta));
allow.countDown();
String token = login.get(20, TimeUnit.SECONDS); reset.get(20, TimeUnit.SECONDS);
assertEquals(401, assertThrows(ApiException.class, () -> auth.requireUser(token)).getStatus());
} finally { allow.countDown(); }
}
@Test void mfaFailuresRemainCommittedInsideTheLoginTransaction() {
var cipher = new SecretCipher("isolated-mfa-key");
jdbc.update("INSERT INTO user_totp(user_id,secret_ciphertext,enabled) VALUES (?,?,true)", userId, cipher.encrypt("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"));
for (int i = 0; i < 5; i++) assertEquals("INVALID_MFA_CODE", assertThrows(ApiException.class, () -> auth.login("security_member", "original-password", "not-a-code", meta)).getCode());
assertEquals("MFA_RATE_LIMITED", assertThrows(ApiException.class, () -> auth.login("security_member", "original-password", "not-a-code", meta)).getCode());
assertEquals(5, jdbc.queryForObject("SELECT attempts FROM user_totp WHERE user_id=?", Integer.class, userId));
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM auth_sessions", Integer.class));
}
@Test void auditFailureRollsBackPasswordAndRevocationsTogether() {
String token = session();
jdbc.execute("CREATE FUNCTION reject_password_audit() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.event_type='user.account_updated' THEN RAISE EXCEPTION 'isolated audit failure'; END IF; RETURN NEW; END $$");
jdbc.execute("CREATE TRIGGER reject_password_audit BEFORE INSERT ON auth_audit_log FOR EACH ROW EXECUTE FUNCTION reject_password_audit()");
assertThrows(org.springframework.dao.DataAccessException.class, () -> auth.updateAccount(token, "Test", "original-password", "replacement-password", meta));
assertNotNull(auth.requireUser(token));
assertNotNull(session());
}
}
@@ -0,0 +1,18 @@
package com.moyi.controlplane.auth;
import com.moyi.controlplane.common.ApiException;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class PasswordPolicyTest {
@Test void everyNewPasswordHasTheSameBoundsWithoutTrimming() {
for (String value : new String[]{null, "", " ".repeat(12), "short", "x".repeat(257)}) assertThrows(ApiException.class, () -> PasswordPolicy.validate(value));
assertDoesNotThrow(() -> PasswordPolicy.validate("a".repeat(8)));
assertDoesNotThrow(() -> PasswordPolicy.validate("a".repeat(256)));
}
@Test void malformedHashesCannotSelectUnboundedWorkAndNullIsNeverAPassword() {
var hasher = new PasswordHasher("pepper");
for (String encoded : new String[]{null, "", "pbkdf2$2147483647$AAECAwQFBgcICQoLDA0ODw$" + "A".repeat(43), "pbkdf2$210000$AA$AA"}) assertFalse(hasher.matches("test-password", encoded));
assertFalse(hasher.matches(null, hasher.hash("null")));
}
}
@@ -0,0 +1,7 @@
package com.moyi.controlplane.auth;
/** Deterministic test-only access; no public production dispatch endpoint. */
public final class VerificationDeliveryFixture {
private VerificationDeliveryFixture() {}
public static boolean deliverOne(VerificationDispatchService dispatch) { return dispatch.dispatchOne(); }
}
@@ -0,0 +1,343 @@
package com.moyi.controlplane.auth;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.common.ApiExceptionHandler;
import com.moyi.controlplane.security.SecretCipher;
import java.util.ArrayList;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import javax.sql.DataSource;
import org.flywaydb.core.Flyway;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Configuration;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.transaction.IllegalTransactionStateException;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import org.springframework.transaction.support.TransactionSynchronizationManager;
import org.springframework.transaction.support.TransactionTemplate;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
@EnabledIfEnvironmentVariable(named="MOYI_AUTH_TEST_JDBC_URL",matches="jdbc:postgresql://127[.]0[.]0[.]1:[0-9]+/.+")
class VerificationDispatchPostgresTest {
@Configuration @EnableTransactionManagement(proxyTargetClass=true) static class Transactions {}
private static final String ACTIVE="active@example.test", UNKNOWN="unknown@example.test", DISABLED="disabled@example.test";
private final ObjectMapper json=new ObjectMapper().findAndRegisterModules();
private final SecretCipher cipher=new SecretCipher("isolated-delivery-encryption");
private DriverManagerDataSource base;
private String schema, invitation;
private JdbcTemplate jdbc;
private AnnotationConfigApplicationContext context;
private VerificationDispatchService dispatch;
private VerificationService verification;
private MockMvc http;
private final TestSender sender=new TestSender();
static class TestSender implements VerificationSender {
final Map<String,String> codes=new ConcurrentHashMap<>();
final AtomicInteger calls=new AtomicInteger();
volatile boolean available=true, fail=false;
volatile CountDownLatch arrived, release;
volatile VerificationPurpose lastPurpose;
public boolean available(String channel) { return available; }
public void send(String channel,String account,String code) { send(channel,account,code,VerificationPurpose.REGISTRATION); }
public void send(String channel,String account,String code,VerificationPurpose purpose) {
assertFalse(TransactionSynchronizationManager.isActualTransactionActive(),"Network delivery must not hold a DB transaction");
calls.incrementAndGet(); lastPurpose=purpose;
if (arrived!=null) arrived.countDown();
if (release!=null) try { assertTrue(release.await(10,TimeUnit.SECONDS)); }
catch (InterruptedException error) { Thread.currentThread().interrupt(); throw new IllegalStateException("Interrupted fixture"); }
if (fail) throw new ApiException("VERIFICATION_SEND_FAILED","Synthetic provider detail must stay private",502);
codes.put(account,code);
}
}
@BeforeEach void setup() {
String url=System.getenv("MOYI_AUTH_TEST_JDBC_URL");
schema="verification_dispatch_"+UUID.randomUUID().toString().replace("-","");
base=new DriverManagerDataSource(url,"moyi_role_test","");
var source=new DriverManagerDataSource(url+"?currentSchema="+schema,"moyi_role_test","");
Flyway.configure().dataSource(base).schemas(schema).defaultSchema(schema).load().migrate();
jdbc=new JdbcTemplate(source); UUID creator=UUID.randomUUID(); invitation="synthetic-invite-"+UUID.randomUUID();
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name,status) VALUES (?,?,'email','not-a-login-hash','Synthetic','active'),(?,?,'email','not-a-login-hash','Synthetic','disabled')",creator,ACTIVE,UUID.randomUUID(),DISABLED);
jdbc.update("INSERT INTO invitation_codes(id,code_hash,code_prefix,label,max_uses,created_by) VALUES (?,?,?,'Synthetic',100,?)",UUID.randomUUID(),AuthService.hashSecret(invitation),"synthetic",creator);
context=new AnnotationConfigApplicationContext(); context.register(Transactions.class);
context.registerBean(DataSource.class,()->source);
context.registerBean(DataSourceTransactionManager.class,()->new DataSourceTransactionManager(source));
context.registerBean(VerificationDispatchService.class,()->new VerificationDispatchService(jdbc,sender,cipher,json,context.getBean(DataSourceTransactionManager.class)));
context.registerBean(VerificationService.class,()->new VerificationService(jdbc,context.getBean(VerificationDispatchService.class),"isolated-pepper"));
context.refresh(); dispatch=context.getBean(VerificationDispatchService.class); verification=context.getBean(VerificationService.class);
http=MockMvcBuilders.standaloneSetup(new AuthController(mock(AuthService.class),verification,mock(VerifiedAccountService.class),168,true))
.setControllerAdvice(new ApiExceptionHandler()).build();
}
@AfterEach void cleanup() {
if (sender.release!=null) sender.release.countDown();
if (context!=null) context.close();
if (base!=null && schema!=null) new JdbcTemplate(base).execute("DROP SCHEMA "+schema+" CASCADE");
}
private UUID delivery(String account,VerificationPurpose purpose) {
return jdbc.queryForObject("SELECT delivery_id FROM "+purpose.table+" WHERE account=?",UUID.class,account);
}
private void releaseQuota(String account) { jdbc.update("UPDATE verification_send_events SET created_at=clock_timestamp()-interval '61 seconds' WHERE account=?",account); }
private void assertTerminal(UUID id,String state) {
var row=jdbc.queryForMap("SELECT status,payload_ciphertext,finished_at FROM verification_delivery_outbox WHERE id=?",id);
assertEquals(state,row.get("status")); assertNull(row.get("payload_ciphertext")); assertNotNull(row.get("finished_at"));
}
@Test void resetRequestsHaveIdenticalReceiptsAndNoRequestSideDeliveryForAllAccountStates() throws Exception {
String reference=null;
for (String account : java.util.List.of(ACTIVE,UNKNOWN,DISABLED)) {
var response=http.perform(post("/api/auth/password-reset/code").contentType("application/json").content(json.writeValueAsBytes(Map.of("account",account))))
.andExpect(status().isOk()).andExpect(jsonPath("$.sent").value(true)).andExpect(jsonPath("$.code").doesNotExist())
.andExpect(jsonPath("$.token").doesNotExist()).andExpect(cookie().doesNotExist("moyi_session")).andReturn().getResponse().getContentAsString();
if (reference==null) reference=response; else assertEquals(json.readTree(reference),json.readTree(response));
}
assertEquals(0,sender.calls.get());
assertEquals(3,jdbc.queryForObject("SELECT count(*) FROM verification_send_events",Integer.class));
assertEquals(3,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox WHERE status='queued'",Integer.class));
assertEquals(3,jdbc.queryForObject("SELECT count(*) FROM password_reset_codes",Integer.class));
}
@Test void knownAndUnknownRepeatedRequestsExposeTheSameLimit() throws Exception {
for (String account : java.util.List.of(ACTIVE,UNKNOWN,DISABLED)) {
verification.sendReset(account,"127.0.0.1");
http.perform(post("/api/auth/password-reset/code").contentType("application/json").content(json.writeValueAsBytes(Map.of("account",account))))
.andExpect(status().isTooManyRequests()).andExpect(jsonPath("$.error.code").value("VERIFICATION_RATE_LIMIT"));
}
assertEquals(3,jdbc.queryForObject("SELECT count(*) FROM verification_send_events",Integer.class));
}
@Test void workerDeliversOnlyToActiveUsersAndPurgesAllFinishedPayloads() {
for (String account : java.util.List.of(ACTIVE,UNKNOWN,DISABLED)) verification.sendReset(account,"127.0.0.1");
for (int i=0;i<3;i++) assertTrue(dispatch.dispatchOne());
assertEquals(1,sender.calls.get()); assertEquals(VerificationPurpose.PASSWORD_RESET,sender.lastPurpose);
assertTrue(sender.codes.containsKey(ACTIVE)); assertFalse(sender.codes.containsKey(UNKNOWN)); assertFalse(sender.codes.containsKey(DISABLED));
assertTerminal(delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET),"sent");
assertTerminal(delivery(UNKNOWN,VerificationPurpose.PASSWORD_RESET),"discarded");
assertTerminal(delivery(DISABLED,VerificationPurpose.PASSWORD_RESET),"discarded");
}
@Test void payloadContainsNoPlaintextAccountOrOtpAndIsBoundToItsGeneration() throws Exception {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID id=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
String stored=jdbc.queryForObject("SELECT payload_ciphertext FROM verification_delivery_outbox WHERE id=?",String.class,id);
var decoded=json.readTree(cipher.decrypt(stored));
assertTrue(stored.startsWith("v1$")); assertFalse(stored.contains(ACTIVE)); assertFalse(stored.contains("account"));
assertEquals(id.toString(),decoded.path("id").asText()); assertEquals("password-reset",decoded.path("purpose").asText());
assertTrue(decoded.path("code").asText().matches("[0-9]{6}"));
assertNotEquals(decoded.path("code").asText(),jdbc.queryForObject("SELECT code_hash FROM password_reset_codes WHERE account=?",String.class,ACTIVE));
}
@Test void providerFailureStaysOutOfTheReceiptAndIsNotRetried() {
sender.fail=true;
assertEquals(true,verification.sendReset(ACTIVE,"127.0.0.1").get("sent"));
UUID id=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET); assertTrue(dispatch.dispatchOne());
assertTerminal(id,"failed"); assertEquals(1,sender.calls.get()); assertFalse(dispatch.dispatchOne()); assertEquals(1,sender.calls.get());
assertTrue(jdbc.queryForObject("SELECT consumed FROM password_reset_codes WHERE delivery_id=?",Boolean.class,id));
assertEquals("DELIVERY_FAILED",jdbc.queryForObject("SELECT failure_code FROM verification_delivery_outbox WHERE id=?",String.class,id));
assertEquals(1,jdbc.queryForObject("SELECT count(*) FROM verification_send_events",Integer.class));
}
@Test void sendingDoesNotHoldTheGlobalQuotaLockOrADatabaseTransaction() throws Exception {
verification.sendReset(ACTIVE,"127.0.0.1");
sender.arrived=new CountDownLatch(1); sender.release=new CountDownLatch(1);
try (var executor=Executors.newFixedThreadPool(2)) {
var worker=executor.submit(dispatch::dispatchOne); assertTrue(sender.arrived.await(3,TimeUnit.SECONDS));
var request=executor.submit(()->verification.sendReset(UNKNOWN,"127.0.0.2"));
assertEquals(true,request.get(3,TimeUnit.SECONDS).get("sent"));
assertEquals("sending",jdbc.queryForObject("SELECT status FROM verification_delivery_outbox WHERE id=?",String.class,delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET)));
sender.release.countDown(); assertTrue(worker.get(3,TimeUnit.SECONDS));
}
}
@Test void supersededQueueEntriesCannotDeliverOldCodesOrInvalidateNewOnes() {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID old=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
releaseQuota(ACTIVE); verification.sendReset(ACTIVE,"127.0.0.1"); UUID current=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
assertNotEquals(old,current); assertTrue(dispatch.dispatchOne()); assertTerminal(old,"discarded"); assertEquals(0,sender.calls.get());
assertFalse(jdbc.queryForObject("SELECT consumed FROM password_reset_codes WHERE delivery_id=?",Boolean.class,current));
assertTrue(dispatch.dispatchOne()); assertTerminal(current,"sent"); assertEquals(1,sender.calls.get());
assertEquals(ACTIVE,verification.consumeReset(ACTIVE,sender.codes.get(ACTIVE)));
}
@Test void oldInFlightFailureDoesNotConsumeTheReplacementCode() throws Exception {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID old=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
sender.fail=true; sender.arrived=new CountDownLatch(1); sender.release=new CountDownLatch(1);
try (var executor=Executors.newSingleThreadExecutor()) {
var worker=executor.submit(dispatch::dispatchOne); assertTrue(sender.arrived.await(3,TimeUnit.SECONDS));
releaseQuota(ACTIVE); verification.sendReset(ACTIVE,"127.0.0.2"); UUID replacement=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
sender.release.countDown(); assertTrue(worker.get(3,TimeUnit.SECONDS)); assertTerminal(old,"failed");
assertFalse(jdbc.queryForObject("SELECT consumed FROM password_reset_codes WHERE delivery_id=?",Boolean.class,replacement));
sender.fail=false; sender.arrived=null; sender.release=null; assertTrue(dispatch.dispatchOne()); assertTerminal(replacement,"sent");
}
}
@Test void expiredAndAmbiguousSendingRowsAreErasedWithoutAutomaticReplay() {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID id=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
jdbc.update("UPDATE verification_delivery_outbox SET status='sending',started_at=clock_timestamp() WHERE id=?",id);
assertFalse(dispatch.dispatchOne()); assertEquals(0,sender.calls.get());
jdbc.update("UPDATE verification_delivery_outbox SET expires_at=clock_timestamp()-interval '1 second' WHERE id=?",id);
assertFalse(dispatch.dispatchOne()); assertTerminal(id,"discarded"); assertEquals(0,sender.calls.get());
jdbc.update("UPDATE verification_delivery_outbox SET finished_at=clock_timestamp()-interval '3 days' WHERE id=?",id);
assertFalse(dispatch.dispatchOne()); assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox",Integer.class));
}
@Test void disabledAfterEnqueueAndAlreadyConsumedCodesAreNotSent() {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID id=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
jdbc.update("UPDATE users SET status='disabled' WHERE account=?",ACTIVE); assertTrue(dispatch.dispatchOne()); assertTerminal(id,"discarded");
jdbc.update("UPDATE users SET status='active' WHERE account=?",ACTIVE); releaseQuota(ACTIVE); verification.sendReset(ACTIVE,"127.0.0.1");
id=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET); jdbc.update("UPDATE password_reset_codes SET consumed=true WHERE delivery_id=?",id);
assertTrue(dispatch.dispatchOne()); assertTerminal(id,"discarded"); assertEquals(0,sender.calls.get());
}
@Test void missingEncryptionOrDisabledSenderIsAccountIndependentAndCreatesNoWork() {
var missingKey=new VerificationDispatchService(jdbc,sender,new SecretCipher(""),json,context.getBean(DataSourceTransactionManager.class));
try { assertFalse(missingKey.available("email")); assertFalse(missingKey.available("phone")); } finally { missingKey.close(); }
sender.available=false;
for (String account : java.util.List.of(ACTIVE,UNKNOWN,DISABLED)) {
assertEquals("VERIFICATION_UNAVAILABLE",assertThrows(ApiException.class,()->verification.sendReset(account,"127.0.0.1")).getCode());
}
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox",Integer.class));
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM verification_send_events",Integer.class));
}
@Test void queueFailureRollsBackTheCodeAndQuotaRecordTogether() {
jdbc.execute("ALTER TABLE verification_delivery_outbox ADD CONSTRAINT injected_failure CHECK (purpose <> 'password-reset') NOT VALID");
assertThrows(DataIntegrityViolationException.class,()->verification.sendReset(ACTIVE,"127.0.0.1"));
for (String table : java.util.List.of("verification_delivery_outbox","verification_send_events","password_reset_codes")) {
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM "+table,Integer.class));
}
jdbc.execute("ALTER TABLE verification_delivery_outbox DROP CONSTRAINT injected_failure");
assertEquals(true,verification.sendReset(ACTIVE,"127.0.0.1").get("sent"));
}
@Test void concurrentRequestsAcrossConnectionsCannotDuplicateTheSameAccountAdmission() throws Exception {
var start=new CountDownLatch(1); var results=new ArrayList<String>();
try (var executor=Executors.newFixedThreadPool(2)) {
var requests=new ArrayList<java.util.concurrent.Future<String>>();
for (int i=0;i<2;i++) requests.add(executor.submit(()->{
assertTrue(start.await(3,TimeUnit.SECONDS));
try { verification.sendReset(UNKNOWN,"127.0.0.1"); return "accepted"; } catch (ApiException error) { return error.getCode(); }
}));
start.countDown(); for (var request : requests) results.add(request.get(5,TimeUnit.SECONDS));
}
assertEquals(1,results.stream().filter("accepted"::equals).count()); assertEquals(1,results.stream().filter("VERIFICATION_RATE_LIMIT"::equals).count());
assertEquals(1,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox",Integer.class));
}
@Test void concurrentWorkerClaimsSendAGenerationAtMostOnce() throws Exception {
verification.sendReset(ACTIVE,"127.0.0.1"); var start=new CountDownLatch(1);
try (var executor=Executors.newFixedThreadPool(2)) {
var first=executor.submit(()->{start.await(); return dispatch.dispatchOne();});
var second=executor.submit(()->{start.await(); return dispatch.dispatchOne();}); start.countDown();
assertNotEquals(first.get(5,TimeUnit.SECONDS),second.get(5,TimeUnit.SECONDS));
}
assertEquals(1,sender.calls.get()); assertTerminal(delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET),"sent");
}
@Test void accountDailySourceHourlyAndGlobalBudgetsIncludeUnknownAccounts() {
for (int i=0;i<10;i++) {
verification.sendReset(UNKNOWN,"127.0.0.1"); releaseQuota(UNKNOWN);
}
assertEquals("VERIFICATION_RATE_LIMIT",assertThrows(ApiException.class,()->verification.sendReset(UNKNOWN,"127.0.0.2")).getCode());
jdbc.update("DELETE FROM verification_send_events");
jdbc.execute("INSERT INTO verification_send_events(id,account,ip_address) SELECT md5('source-'||g)::uuid,'synthetic-'||g||'@example.test','127.0.0.9' FROM generate_series(1,30) g");
assertEquals("VERIFICATION_RATE_LIMIT",assertThrows(ApiException.class,()->verification.sendReset(ACTIVE,"127.0.0.9")).getCode());
assertEquals("VERIFICATION_RATE_LIMIT",assertThrows(ApiException.class,()->verification.sendReset("other@example.test","127.0.0.9")).getCode());
jdbc.update("DELETE FROM verification_send_events");
jdbc.execute("INSERT INTO verification_send_events(id,account,ip_address) SELECT md5('global-'||g)::uuid,'synthetic-'||g||'@example.test','192.0.2.1' FROM generate_series(1,1000) g");
assertEquals("VERIFICATION_RATE_LIMIT",assertThrows(ApiException.class,()->verification.sendReset(ACTIVE,"127.0.0.1")).getCode());
assertEquals("VERIFICATION_RATE_LIMIT",assertThrows(ApiException.class,()->verification.sendReset("another@example.test","127.0.0.2")).getCode());
}
@Test void crossPurposeQuotaIsSharedWhileCodesRemainPurposeBound() {
verification.send(UNKNOWN,invitation,"127.0.0.1");
assertEquals("VERIFICATION_RATE_LIMIT",assertThrows(ApiException.class,()->verification.sendReset(UNKNOWN,"127.0.0.1")).getCode());
assertTrue(dispatch.dispatchOne()); String code=sender.codes.get(UNKNOWN);
assertEquals(VerificationPurpose.REGISTRATION,sender.lastPurpose); verification.consume(UNKNOWN,code);
assertEquals("VERIFICATION_INVALID",assertThrows(ApiException.class,()->verification.consumeReset(UNKNOWN,code)).getCode());
}
@Test void allInvalidResetCodeStatesHaveTheSamePublicError() {
String expected=assertThrows(ApiException.class,()->verification.consumeReset(UNKNOWN,"000000")).getMessage();
verification.sendReset(ACTIVE,"127.0.0.1"); assertTrue(dispatch.dispatchOne());
String correct=sender.codes.get(ACTIVE), wrong="000000".equals(correct)?"111111":"000000";
assertEquals(expected,assertThrows(ApiException.class,()->verification.consumeReset(ACTIVE,wrong)).getMessage());
jdbc.update("UPDATE password_reset_codes SET attempts=5 WHERE account=?",ACTIVE);
assertEquals(expected,assertThrows(ApiException.class,()->verification.consumeReset(ACTIVE,correct)).getMessage());
jdbc.update("UPDATE password_reset_codes SET attempts=0,expires_at=clock_timestamp()-interval '1 second' WHERE account=?",ACTIVE);
assertEquals(expected,assertThrows(ApiException.class,()->verification.consumeReset(ACTIVE,correct)).getMessage());
}
@Test void ciphertextTamperingAndEnvelopeSwappingNeverReachTheSender() {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID first=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
jdbc.update("UPDATE verification_delivery_outbox SET payload_ciphertext='v1$invalid$ciphertext' WHERE id=?",first);
assertTrue(dispatch.dispatchOne()); assertTerminal(first,"failed"); assertEquals(0,sender.calls.get());
releaseQuota(ACTIVE); verification.sendReset(ACTIVE,"127.0.0.1");
UUID second=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET); String valid=jdbc.queryForObject("SELECT payload_ciphertext FROM verification_delivery_outbox WHERE id=?",String.class,second);
verification.sendReset(UNKNOWN,"127.0.0.2"); UUID third=delivery(UNKNOWN,VerificationPurpose.PASSWORD_RESET);
jdbc.update("UPDATE verification_delivery_outbox SET payload_ciphertext=? WHERE id=?",valid,third);
assertTrue(dispatch.dispatchOne()); assertTerminal(second,"sent"); assertTrue(dispatch.dispatchOne()); assertTerminal(third,"failed");
assertEquals(1,sender.calls.get());
}
@Test void enqueuingRequiresTheCallingTransaction() {
assertThrows(IllegalTransactionStateException.class,()->dispatch.enqueue(VerificationPurpose.REGISTRATION,UNKNOWN,"123456","a".repeat(64)));
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox",Integer.class));
}
@Test void failedReceiptPersistenceNeverResendsTheMessage() {
verification.sendReset(ACTIVE,"127.0.0.1"); UUID id=delivery(ACTIVE,VerificationPurpose.PASSWORD_RESET);
jdbc.execute("ALTER TABLE verification_delivery_outbox ADD CONSTRAINT injected_receipt_failure CHECK (status <> 'sent') NOT VALID");
assertThrows(DataIntegrityViolationException.class,dispatch::dispatchOne); assertEquals(1,sender.calls.get());
assertEquals("sending",jdbc.queryForObject("SELECT status FROM verification_delivery_outbox WHERE id=?",String.class,id));
jdbc.execute("ALTER TABLE verification_delivery_outbox DROP CONSTRAINT injected_receipt_failure");
assertFalse(dispatch.dispatchOne()); assertEquals(1,sender.calls.get());
}
@Test void scheduledDeliveryHasOnlyTwoWorkersAndNoUnboundedWaitingQueue() throws Exception {
for (int i=0;i<6;i++) verification.send("queued-"+i+"@example.test",invitation,"127.0.0.1");
sender.arrived=new CountDownLatch(2); sender.release=new CountDownLatch(1);
try {
dispatch.dispatchPending(); assertTrue(sender.arrived.await(3,TimeUnit.SECONDS));
for (int i=0;i<10;i++) dispatch.dispatchPending();
assertEquals(2,sender.calls.get());
assertEquals(2,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox WHERE status='sending'",Integer.class));
assertEquals(4,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox WHERE status='queued'",Integer.class));
} finally { sender.release.countDown(); dispatch.close(); }
}
@Test void blockedQuotaLockTimesOutAndDoesNotLeavePartialAdmission() throws Exception {
var locked=new CountDownLatch(1); var unlock=new CountDownLatch(1);
try (var executor=Executors.newSingleThreadExecutor()) {
var blocker=executor.submit(()->new TransactionTemplate(context.getBean(DataSourceTransactionManager.class)).executeWithoutResult(status->{
jdbc.queryForList("SELECT pg_advisory_xact_lock(734092615)"); locked.countDown();
try { assertTrue(unlock.await(8,TimeUnit.SECONDS)); } catch (InterruptedException error) { Thread.currentThread().interrupt(); throw new IllegalStateException(error); }
}));
assertTrue(locked.await(3,TimeUnit.SECONDS)); long start=System.nanoTime();
try {
assertThrows(org.springframework.dao.DataAccessException.class,()->verification.sendReset(UNKNOWN,"127.0.0.1"));
assertTrue(TimeUnit.NANOSECONDS.toMillis(System.nanoTime()-start)<5000);
} finally { unlock.countDown(); }
blocker.get(3,TimeUnit.SECONDS);
}
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM verification_send_events",Integer.class));
assertEquals(0,jdbc.queryForObject("SELECT count(*) FROM verification_delivery_outbox",Integer.class));
assertEquals(true,verification.sendReset(UNKNOWN,"127.0.0.1").get("sent"));
}
}
@@ -0,0 +1,227 @@
package com.moyi.controlplane.auth;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.common.ApiExceptionHandler;
import com.moyi.controlplane.security.SecretCipher;
import jakarta.servlet.http.Cookie;
import java.util.ArrayList;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import javax.sql.DataSource;
import org.flywaydb.core.Flyway;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Configuration;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import org.springframework.transaction.annotation.EnableTransactionManagement;
import org.springframework.transaction.support.TransactionTemplate;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
@EnabledIfEnvironmentVariable(named="MOYI_AUTH_TEST_JDBC_URL", matches="jdbc:postgresql://127[.]0[.]0[.]1:[0-9]+/.+")
class VerifiedAccountPostgresTest {
@Configuration @EnableTransactionManagement(proxyTargetClass=true) static class Transactions {}
private static final String ACCOUNT = "verified@example.test", CODE = "123456", PASSWORD = "synthetic-original-password";
private final UUID invitationCreator = UUID.randomUUID();
private final AuthService.RequestMeta meta = new AuthService.RequestMeta("127.0.0.1", "isolated-verified-workflow");
private final ObjectMapper json = new ObjectMapper().findAndRegisterModules();
private DriverManagerDataSource base;
private String schema;
private JdbcTemplate jdbc;
private AnnotationConfigApplicationContext context;
private AuthService auth;
private VerifiedAccountService verified;
private VerificationService verification;
private PasswordHasher hasher;
private MockMvc http;
@BeforeEach void setup() {
String url = System.getenv("MOYI_AUTH_TEST_JDBC_URL");
schema = "verified_account_" + UUID.randomUUID().toString().replace("-", "");
base = new DriverManagerDataSource(url, "moyi_role_test", "");
var source = new DriverManagerDataSource(url + "?currentSchema=" + schema, "moyi_role_test", "");
Flyway.configure().dataSource(base).schemas(schema).defaultSchema(schema).load().migrate();
jdbc = new JdbcTemplate(source); hasher = new PasswordHasher("isolated-verified-pepper");
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name,role) VALUES (?,'invitation_creator','username','not-a-login-hash','Synthetic Creator','admin')", invitationCreator);
context = new AnnotationConfigApplicationContext(); context.register(Transactions.class);
context.registerBean(DataSource.class, () -> source);
context.registerBean(DataSourceTransactionManager.class, () -> new DataSourceTransactionManager(source));
context.registerBean(AuthService.class, () -> new AuthService(jdbc, hasher, json, 168));
context.registerBean(TotpService.class, () -> new TotpService(jdbc, new SecretCipher("isolated-totp")));
context.registerBean(VerificationService.class, () -> new VerificationService(jdbc, mock(VerificationDispatchService.class), "isolated-verified-pepper"));
context.registerBean(VerifiedAccountService.class); context.refresh();
auth = context.getBean(AuthService.class); verification = context.getBean(VerificationService.class); verified = context.getBean(VerifiedAccountService.class);
http = MockMvcBuilders.standaloneSetup(new AuthController(auth, verification, verified, 168, true))
.setControllerAdvice(new ApiExceptionHandler()).build();
}
@AfterEach void cleanup() {
if (context != null) context.close();
if (base != null && schema != null) new JdbcTemplate(base).execute("DROP SCHEMA " + schema + " CASCADE");
}
private String invitation() {
String value = "synthetic-invite-" + UUID.randomUUID();
jdbc.update("INSERT INTO invitation_codes(id,code_hash,code_prefix,label,max_uses,used_count,created_by) VALUES (?,?,?,'test',1,0,?)", UUID.randomUUID(), AuthService.hashSecret(value), "synthetic", invitationCreator);
return value;
}
private UUID user() {
UUID id = UUID.randomUUID();
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name) VALUES (?,?,'email',?,'Test')", id, ACCOUNT, hasher.hash(PASSWORD));
return id;
}
private void code(boolean reset) {
String table = reset ? "password_reset_codes" : "registration_codes";
String purpose = reset ? "password-reset" : "registration";
jdbc.update("INSERT INTO " + table + "(account,code_hash,expires_at) VALUES (?,?,clock_timestamp()+interval '5 minutes')",
ACCOUNT, AuthService.hashSecret("isolated-verified-pepper:" + purpose + ":" + ACCOUNT + ":" + CODE));
}
private void assertUnconsumed(boolean reset, int attempts) {
var row = jdbc.queryForMap("SELECT consumed,attempts FROM " + (reset ? "password_reset_codes" : "registration_codes") + " WHERE account=?", ACCOUNT);
assertEquals(false, row.get("consumed")); assertEquals(attempts, ((Number)row.get("attempts")).intValue());
}
private void rejectAudit(String event) {
// Constraint is isolated to this random test schema; only synthetic event types are supplied.
jdbc.execute("ALTER TABLE auth_audit_log ADD CONSTRAINT injected_audit_failure CHECK (event_type <> '" + event + "') NOT VALID");
}
@Test void invalidInvitationKeepsTheCodeUsableForTheSameAccount() {
code(false); String invite = invitation();
assertEquals("INVITATION_INVALID", assertThrows(ApiException.class, () -> verified.register("invalid", ACCOUNT, PASSWORD, "Test", CODE, meta)).getCode());
assertUnconsumed(false, 0);
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM users WHERE account=?", Integer.class, ACCOUNT));
assertEquals(0, jdbc.queryForObject("SELECT sum(used_count) FROM invitation_codes", Integer.class));
var result = verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta);
assertEquals(ACCOUNT, result.user().account()); assertNotNull(auth.requireUser(result.token()));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM invitation_redemptions", Integer.class));
assertEquals("VERIFICATION_EXPIRED", assertThrows(ApiException.class, () -> verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta)).getCode());
}
@Test void invalidDisplayNameAndDuplicateAccountDoNotBurnAValidCode() {
code(false); String invite = invitation();
assertEquals("INVALID_NAME", assertThrows(ApiException.class, () -> verified.register(invite, ACCOUNT, PASSWORD, "x".repeat(81), CODE, meta)).getCode());
assertUnconsumed(false, 0);
user();
assertEquals("ACCOUNT_EXISTS", assertThrows(ApiException.class, () -> verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta)).getCode());
assertUnconsumed(false, 0);
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM invitation_redemptions", Integer.class));
}
@Test void registrationAuditFailureRollsBackCodeInvitationUserAndSessionTogether() {
code(false); String invite = invitation(); rejectAudit("user.registered");
assertThrows(DataIntegrityViolationException.class, () -> verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta));
assertUnconsumed(false, 0);
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM users WHERE account=?", Integer.class, ACCOUNT));
for (String table : java.util.List.of("auth_sessions", "invitation_redemptions", "auth_audit_log")) {
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM " + table, Integer.class), table);
}
assertEquals(0, jdbc.queryForObject("SELECT sum(used_count) FROM invitation_codes", Integer.class));
jdbc.execute("ALTER TABLE auth_audit_log DROP CONSTRAINT injected_audit_failure");
assertNotNull(verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta).token());
}
@Test void resetAuditFailureRollsBackCodeHashAndSessionRevocations() {
UUID id = user(); code(true);
String first = auth.login(ACCOUNT, PASSWORD, meta).token(), second = auth.login(ACCOUNT, PASSWORD, meta).token();
String original = jdbc.queryForObject("SELECT password_hash FROM users WHERE id=?", String.class, id);
rejectAudit("user.password_reset");
assertThrows(DataIntegrityViolationException.class, () -> verified.resetPassword(ACCOUNT, CODE, "replacement-password", meta));
assertUnconsumed(true, 0);
assertEquals(original, jdbc.queryForObject("SELECT password_hash FROM users WHERE id=?", String.class, id));
assertNotNull(auth.requireUser(first)); assertNotNull(auth.requireUser(second));
jdbc.execute("ALTER TABLE auth_audit_log DROP CONSTRAINT injected_audit_failure");
verified.resetPassword(ACCOUNT, CODE, "replacement-password", meta);
assertEquals(401, assertThrows(ApiException.class, () -> auth.requireUser(first)).getStatus());
assertEquals(401, assertThrows(ApiException.class, () -> auth.requireUser(second)).getStatus());
assertNotNull(auth.login(ACCOUNT, "replacement-password", meta).token());
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM auth_audit_log WHERE event_type='user.password_reset'", Integer.class));
}
@ParameterizedTest @ValueSource(booleans = {false, true})
void wrongCodeAttemptsCommitEvenWhenAnOuterCallerRollsBack(boolean reset) {
if (reset) user(); code(reset); String invite = invitation();
var outer = new TransactionTemplate(context.getBean(DataSourceTransactionManager.class));
outer.executeWithoutResult(status -> {
for (int i = 0; i < 5; i++) {
var failure = assertThrows(ApiException.class, () -> {
if (reset) verified.resetPassword(ACCOUNT, "000000", "replacement-password", meta);
else verified.register(invite, ACCOUNT, PASSWORD, "Test", "000000", meta);
});
assertEquals("VERIFICATION_INVALID", failure.getCode());
}
status.setRollbackOnly();
});
assertUnconsumed(reset, 5);
assertEquals(reset ? "VERIFICATION_INVALID" : "VERIFICATION_EXPIRED", assertThrows(ApiException.class, () -> {
if (reset) verified.resetPassword(ACCOUNT, CODE, "replacement-password", meta);
else verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta);
}).getCode());
}
@ParameterizedTest @ValueSource(booleans = {false, true})
void weakNewPasswordsNeverConsumeCodes(boolean reset) {
code(reset); String invite = invitation();
assertEquals("WEAK_PASSWORD", assertThrows(ApiException.class, () -> {
if (reset) verified.resetPassword(ACCOUNT, CODE, "short", meta);
else verified.register(invite, ACCOUNT, "short", "Test", CODE, meta);
}).getCode());
assertUnconsumed(reset, 0);
}
@Test void disabledResetAccountLeavesTheValidCodeUnconsumed() {
UUID id = user(); code(true); jdbc.update("UPDATE users SET status='disabled' WHERE id=?", id);
assertEquals("ACCOUNT_NOT_FOUND", assertThrows(ApiException.class, () -> verified.resetPassword(ACCOUNT, CODE, "replacement-password", meta)).getCode());
assertUnconsumed(true, 0);
}
@ParameterizedTest @ValueSource(booleans = {false, true})
void concurrentCodeUseCompletesExactlyOneAccountOperation(boolean reset) throws Exception {
if (reset) user(); code(reset); String invite = invitation();
var start = new CountDownLatch(1); var results = new ArrayList<String>();
try (var executor = Executors.newFixedThreadPool(2)) {
var futures = new ArrayList<java.util.concurrent.Future<String>>();
for (int i = 0; i < 2; i++) futures.add(executor.submit(() -> {
assertTrue(start.await(3, TimeUnit.SECONDS));
try {
if (reset) verified.resetPassword(ACCOUNT, CODE, "replacement-password", meta);
else verified.register(invite, ACCOUNT, PASSWORD, "Test", CODE, meta);
return "completed";
} catch (ApiException error) { return error.getCode(); }
}));
start.countDown();
for (var future : futures) results.add(future.get(15, TimeUnit.SECONDS));
}
assertEquals(1, results.stream().filter("completed"::equals).count());
assertEquals(1, results.stream().filter((reset ? "VERIFICATION_INVALID" : "VERIFICATION_EXPIRED")::equals).count());
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM users WHERE account=?", Integer.class, ACCOUNT));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM auth_audit_log WHERE event_type=?", Integer.class, reset ? "user.password_reset" : "user.registered"));
if (!reset) assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM invitation_redemptions", Integer.class));
}
@Test void publicResetClearsTheCookieAndRevokesTheServerSessionOnlyAfterSuccess() throws Exception {
user(); code(true); String token = auth.login(ACCOUNT, PASSWORD, meta).token();
http.perform(post("/api/auth/password-reset").cookie(new Cookie("moyi_session", token)).contentType("application/json")
.content(json.writeValueAsBytes(Map.of("account", ACCOUNT, "verificationCode", "000000", "newPassword", "replacement-password"))))
.andExpect(status().isBadRequest()).andExpect(cookie().doesNotExist("moyi_session"));
assertNotNull(auth.requireUser(token)); assertUnconsumed(true, 1);
http.perform(post("/api/auth/password-reset").cookie(new Cookie("moyi_session", token)).contentType("application/json")
.content(json.writeValueAsBytes(Map.of("account", ACCOUNT, "verificationCode", CODE, "newPassword", "replacement-password"))))
.andExpect(status().isNoContent()).andExpect(cookie().maxAge("moyi_session", 0))
.andExpect(cookie().httpOnly("moyi_session", true)).andExpect(cookie().secure("moyi_session", true));
assertEquals(401, assertThrows(ApiException.class, () -> auth.requireUser(token)).getStatus());
}
}
@@ -7,6 +7,9 @@ import com.moyi.controlplane.provider.OpenAiCompatibleProvider;
import com.moyi.controlplane.task.AsyncTaskService;
import com.moyi.controlplane.usage.UsageAuditService;
import com.moyi.controlplane.wallet.ChatBillingService;
import com.moyi.controlplane.gateway.request.ApiRequestRiskEngine;
import com.moyi.controlplane.gateway.request.ApiRequestRiskPolicy;
import com.moyi.controlplane.gateway.request.GatewayRequestExecutionService;
import org.springframework.mock.web.MockHttpServletResponse;
import java.util.List;
import java.util.Map;
@@ -20,6 +23,13 @@ import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
class GatewayDeploymentTest {
private static GatewayController controller(CatalogService catalog, OpenAiCompatibleProvider provider,
GatewayGuard guard, ChatBillingService billing, ObjectMapper json) {
doReturn(mock(GatewayGuard.Admission.class)).when(guard).acquire(anyMap(), any(UUID.class), anyLong());
var requests = new GatewayRequestExecutionService(billing,
new ApiRequestRiskPolicy(ApiRequestRiskEngine.allowAll()));
return new GatewayController(catalog, provider, guard, requests, json);
}
@Test void failedTaskClaimReleasesRedisLeaseAfterDatabaseRollback() {
var redis=mock(StringRedisTemplate.class);doReturn(1L).when(redis).execute(any(RedisScript.class),anyList(),any(Object[].class));
var policies=mock(EnterprisePolicies.class);UUID key=UUID.randomUUID(),user=UUID.randomUUID();
@@ -124,7 +134,7 @@ class GatewayDeploymentTest {
when(catalog.authenticateKey("test-key")).thenReturn(key);
when(catalog.listModels(false)).thenReturn(List.of(route));
when(catalog.keyCanInvoke(key, "route-1")).thenReturn(true);
GatewayController controller = new GatewayController(catalog, mock(OpenAiCompatibleProvider.class), guard, mock(ChatBillingService.class), new ObjectMapper());
GatewayController controller = controller(catalog, mock(OpenAiCompatibleProvider.class), guard, mock(ChatBillingService.class), new ObjectMapper());
Map<String, Object> response = controller.models("Bearer test-key");
@SuppressWarnings("unchecked")
@@ -163,7 +173,7 @@ class GatewayDeploymentTest {
when(catalog.activeModel("glm-5--apihub")).thenReturn(null);
when(catalog.listModels(false)).thenReturn(List.of(route));
when(catalog.keyCanInvoke(key, "route-readable")).thenReturn(true);
GatewayController controller = new GatewayController(catalog, mock(OpenAiCompatibleProvider.class), guard, mock(ChatBillingService.class), new ObjectMapper());
GatewayController controller = controller(catalog, mock(OpenAiCompatibleProvider.class), guard, mock(ChatBillingService.class), new ObjectMapper());
assertSame(route, controller.resolveModel(key, "glm-5--apihub--" + tag("route-readable")));
verify(catalog).keyCanInvoke(key, "route-readable");
}
@@ -236,7 +246,7 @@ class GatewayDeploymentTest {
when(catalog.authenticateKey("test-key")).thenReturn(key);
when(catalog.activeModel("model-real")).thenReturn(Map.of("id", "model-real", "providerId", "provider-deepseek", "mode", "sync", "type", "text"));
when(catalog.keyCanInvoke(key, "model-real")).thenReturn(true);
GatewayController controller = new GatewayController(catalog, provider, guard, mock(ChatBillingService.class),new ObjectMapper());
GatewayController controller = controller(catalog, provider, guard, mock(ChatBillingService.class),new ObjectMapper());
ApiException error = assertThrows(ApiException.class, () -> controller.chatCompletions("Bearer test-key", Map.of("model", "model-real"),null,new MockHttpServletResponse()));
assertEquals("PROVIDER_NOT_CONFIGURED", error.getCode());
assertEquals(503, error.getStatus());
@@ -251,9 +261,9 @@ class GatewayDeploymentTest {
when(catalog.activeModel("mock")).thenReturn(Map.of("id", "mock", "providerId", "provider-mock", "mode", "sync", "type", "text"));
when(catalog.keyCanInvoke(key, "mock")).thenReturn(true);
var billing=mock(ChatBillingService.class);
when(billing.reserve(anyMap(),anyMap(),any(),any())).thenReturn(new ChatBillingService.Ticket(UUID.randomUUID(),UUID.fromString(key.get("userId").toString()),ChatMeter.rates(null,true)));
when(billing.reserve(anyMap(),anyMap(),any(),any(),any())).thenReturn(new ChatBillingService.Ticket(UUID.randomUUID(),UUID.fromString(key.get("userId").toString()),ChatMeter.rates(null,true)));
when(billing.settle(any(),anyMap(),anyInt(),any())).thenReturn(java.math.BigDecimal.ZERO);
GatewayController controller = new GatewayController(catalog, mock(OpenAiCompatibleProvider.class), mock(GatewayGuard.class), billing,new ObjectMapper());
GatewayController controller = controller(catalog, mock(OpenAiCompatibleProvider.class), mock(GatewayGuard.class), billing,new ObjectMapper());
assertEquals(200, controller.chatCompletions("Bearer test-key", Map.of("model", "mock","messages",List.of(Map.of("role","user","content","test"))),null,new MockHttpServletResponse()).getStatusCode().value());
}
@@ -276,10 +286,10 @@ class GatewayDeploymentTest {
"usage", Map.of("prompt_tokens", 2, "completion_tokens", 16, "total_tokens", 18)
));
when(provider.chat(eq("provider-relay"), eq("reasoning-model"), eq("reasoning-route"), anyMap())).thenReturn(upstreamResponse);
when(billing.reserve(anyMap(), anyMap(), any(), isNull())).thenReturn(new ChatBillingService.Ticket(UUID.randomUUID(), UUID.fromString(userId), ChatMeter.rates(null, true)));
when(billing.reserve(anyMap(), anyMap(), any(), isNull(), any())).thenReturn(new ChatBillingService.Ticket(UUID.randomUUID(), UUID.fromString(userId), ChatMeter.rates(null, true)));
when(billing.settle(any(), anyMap(), anyInt(), isNull())).thenReturn(java.math.BigDecimal.ZERO);
GatewayController controller = new GatewayController(catalog, provider, guard, billing, new ObjectMapper());
GatewayController controller = controller(catalog, provider, guard, billing, new ObjectMapper());
var response = controller.chatCompletions("Bearer test-key", Map.of("model", "reasoning-route", "messages", List.of(Map.of("role", "user", "content", "hello"))), null, new MockHttpServletResponse());
assertEquals(200, response.getStatusCode().value());
Map<?, ?> choice = (Map<?, ?>)((List<?>)response.getBody().get("choices")).getFirst();
@@ -307,8 +317,8 @@ class GatewayDeploymentTest {
"choices", List.of(Map.of("index", 0, "message", Map.of("role", "assistant", "content", ""), "finish_reason", "stop"))
));
when(provider.chat(eq("provider-relay"), eq("empty-model"), eq("empty-route"), anyMap())).thenReturn(upstreamResponse);
when(billing.reserve(anyMap(), anyMap(), any(), isNull())).thenReturn(new ChatBillingService.Ticket(UUID.randomUUID(), UUID.fromString(userId), ChatMeter.rates(null, true)));
GatewayController controller = new GatewayController(catalog, provider, guard, billing, new ObjectMapper());
when(billing.reserve(anyMap(), anyMap(), any(), isNull(), any())).thenReturn(new ChatBillingService.Ticket(UUID.randomUUID(), UUID.fromString(userId), ChatMeter.rates(null, true)));
GatewayController controller = controller(catalog, provider, guard, billing, new ObjectMapper());
ApiException error = assertThrows(ApiException.class, () -> controller.chatCompletions("Bearer test-key", Map.of("model", "empty-route", "messages", List.of(Map.of("role", "user", "content", "hello"))), null, new MockHttpServletResponse()));
assertEquals("UPSTREAM_RESPONSE_INVALID", error.getCode());
@@ -0,0 +1,64 @@
package com.moyi.controlplane.gateway;
import java.util.UUID;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.*;
@EnabledIfEnvironmentVariable(named="MOYI_GATEWAY_TEST_REDIS_PORT", matches="[0-9]{2,5}")
class TaskLeaseRedisTest {
private LettuceConnectionFactory factory;
private StringRedisTemplate redis;
private GatewayGuard guard;
private final UUID userId = UUID.randomUUID(), keyId = UUID.randomUUID();
private final String prefix = "moyi:test:" + UUID.randomUUID() + ":";
private String bucket() { return prefix + "gateway:slots:user:" + userId; }
@BeforeEach void setup() {
factory = new LettuceConnectionFactory("127.0.0.1", Integer.parseInt(System.getenv("MOYI_GATEWAY_TEST_REDIS_PORT")));
factory.afterPropertiesSet(); factory.start();
redis = new StringRedisTemplate(factory); redis.afterPropertiesSet();
guard = new GatewayGuard(redis, 60, 300, 4, prefix);
var policies = mock(EnterprisePolicies.class);
when(policies.taskLimit(keyId)).thenReturn(new EnterprisePolicies.UserLimit(userId, 120, 1));
guard.setEnterprisePolicies(policies);
}
@AfterEach void cleanup() {
if (TransactionSynchronizationManager.isSynchronizationActive()) TransactionSynchronizationManager.clearSynchronization();
if (redis != null) redis.delete(bucket());
if (factory != null) factory.destroy();
}
@Test void renewalDoesNotConsumeAnotherSlotAndDifferentTaskIsBlocked() {
assertTrue(guard.tryTaskUserSlot(keyId, "original-task", 180000));
assertTrue(guard.tryTaskUserSlot(keyId, "original-task", 300000));
assertEquals(1L, redis.opsForZSet().zCard(bucket()));
assertFalse(guard.tryTaskUserSlot(keyId, "different-task", 180000));
assertTrue(redis.getExpire(bucket(), java.util.concurrent.TimeUnit.MILLISECONDS) > 250000);
}
@Test void rolledBackRenewalKeepsThePreexistingLease() {
assertTrue(guard.tryTaskUserSlot(keyId, "original-task", 180000));
TransactionSynchronizationManager.initSynchronization();
assertTrue(guard.tryTaskUserSlot(keyId, "original-task", 300000));
for (var synchronization : TransactionSynchronizationManager.getSynchronizations()) synchronization.afterCompletion(TransactionSynchronization.STATUS_ROLLED_BACK);
TransactionSynchronizationManager.clearSynchronization();
assertNotNull(redis.opsForZSet().score(bucket(), "original-task"));
}
@Test void rolledBackNewClaimRemovesOnlyItsOwnLease() {
TransactionSynchronizationManager.initSynchronization();
assertTrue(guard.tryTaskUserSlot(keyId, "new-task", 180000));
for (var synchronization : TransactionSynchronizationManager.getSynchronizations()) synchronization.afterCompletion(TransactionSynchronization.STATUS_ROLLED_BACK);
TransactionSynchronizationManager.clearSynchronization();
assertNull(redis.opsForZSet().score(bucket(), "new-task"));
}
}
@@ -0,0 +1,96 @@
package com.moyi.controlplane.gateway.request;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.catalog.CatalogService;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.GatewayController;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.gateway.NativeGatewayController;
import com.moyi.controlplane.provider.OpenAiCompatibleProvider;
import com.moyi.controlplane.wallet.ChatBillingService;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicReference;
import org.junit.jupiter.api.Test;
import org.springframework.mock.web.MockHttpServletRequest;
import org.springframework.mock.web.MockHttpServletResponse;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
class ApiRequestEntryPointsTest {
private final UUID userId=UUID.randomUUID(), keyId=UUID.randomUUID();
private final Map<String,Object> key=Map.of("id",keyId.toString(),"userId",userId.toString());
private final CatalogService catalog=mock(CatalogService.class);
private final GatewayGuard guard=mock(GatewayGuard.class);
private final ChatBillingService billing=mock(ChatBillingService.class);
private final OpenAiCompatibleProvider provider=mock(OpenAiCompatibleProvider.class);
private final AtomicReference<ApiRequestContext> observed=new AtomicReference<>();
private final GatewayRequestExecutionService requests=new GatewayRequestExecutionService(billing,
new ApiRequestRiskPolicy(context -> {
observed.set(context);
return ApiRequestRiskDecision.deny(new ApiRequestRiskSignal(ApiRequestRiskSignal.Action.DENY,
"test.entry-block","RISK_BLOCKED","Blocked",0));
}));
@Test void publicChatIgnoresSpoofedPlaygroundHeaderAndDenialNeverAcquiresOrReserves() {
Map<String,Object> model=Map.of("id","chat-route","providerId","provider-mock","type","text","mode","sync");
when(catalog.authenticateKey("test-key")).thenReturn(key);
when(catalog.activeModel("chat-route")).thenReturn(model);
when(catalog.keyCanInvoke(key,"chat-route")).thenReturn(true);
var controller=new GatewayController(catalog,provider,guard,requests,new ObjectMapper());
var request=http("/v1/chat/completions");
assertEquals("RISK_BLOCKED",assertThrows(ApiException.class,() -> controller.chatCompletions("Bearer test-key",
Map.of("model","chat-route","messages",List.of(Map.of("role","user","content","hello"))),null,
request,new MockHttpServletResponse())).getCode());
assertEquals(ApiRequestMetadata.Source.API,observed.get().metadata().source());
assertEquals("198.51.100.7",observed.get().metadata().clientIp());
verify(guard).checkRate(keyId,false);
verify(guard,never()).acquire(anyMap(),any(UUID.class),anyLong());
verifyNoInteractions(billing,provider);
}
@Test void nativePublicAndPlaygroundBothEnterRiskBoundaryWithEndpointOwnedSource() {
Map<String,Object> model=Map.of("id","embedding-route","providerId","provider-a","upstreamModel","embedding-model","type","embedding","mode","sync");
when(catalog.authenticateKey("test-key")).thenReturn(key);
when(catalog.activeModel("embedding-route")).thenReturn(model);
when(catalog.keyCanInvoke(key,"embedding-route")).thenReturn(true);
when(provider.isConfigured("provider-a","embedding-route","embedding-model")).thenReturn(true);
var controller=new NativeGatewayController(catalog,guard,requests,provider,new ObjectMapper());
var request=http("/v1/embeddings");
request.addHeader("Authorization","Bearer test-key");
Map<String,Object> input=Map.of("model","embedding-route","input","hello");
assertEquals("RISK_BLOCKED",assertThrows(ApiException.class,
() -> controller.invoke(input,request,new MockHttpServletResponse())).getCode());
assertEquals(ApiRequestProtocol.EMBEDDINGS,observed.get().protocol());
assertEquals(ApiRequestMetadata.Source.API,observed.get().metadata().source());
assertEquals("RISK_BLOCKED",assertThrows(ApiException.class,
() -> controller.executeForKey("embeddings",input,false,key,request,new MockHttpServletResponse(),
ApiRequestMetadata.playground(request.getRemoteAddr()))).getCode());
assertEquals(ApiRequestMetadata.Source.PLAYGROUND,observed.get().metadata().source());
verify(guard,never()).acquire(anyMap(),any(UUID.class),anyLong());
verify(provider,never()).invoke(anyString(),anyString(),anyString(),anyMap(),anyString());
verifyNoInteractions(billing);
}
@Test void malformedRequestsStillConsumeTheirEarlyRateBudget() {
var controller=new NativeGatewayController(catalog,guard,requests,provider,new ObjectMapper());
assertThrows(ApiException.class,() -> controller.executeForKey("embeddings",Map.of(),false,key,
http("/v1/embeddings"),new MockHttpServletResponse()));
verify(guard).checkRate(keyId,false);
verifyNoInteractions(billing);
}
private MockHttpServletRequest http(String path) {
var request=new MockHttpServletRequest("POST",path);
request.setRemoteAddr("198.51.100.7");
request.addHeader("X-Moyi-Source","playground");
request.addHeader("X-Forwarded-For","203.0.113.99"); // Adapter never parses this untrusted header itself.
return request;
}
}
@@ -0,0 +1,59 @@
package com.moyi.controlplane.gateway.request;
import java.util.UUID;
import java.util.Optional;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import static org.junit.jupiter.api.Assertions.*;
class ApiRequestModuleConfigurationTest {
private final ApplicationContextRunner runner = new ApplicationContextRunner()
.withUserConfiguration(ApiRequestModuleConfiguration.class);
private final ApiRequestContext request = new ApiRequestContext(UUID.randomUUID(), UUID.randomUUID(),
ApiRequestProtocol.EMBEDDINGS, "public", "route", false, 12, 0,
ApiRequestMetadata.fromHttp("198.51.100.7"), System.nanoTime());
@Test void noRulesStartsSafelyAndDefaultsToShadow() {
runner.run(context -> {
assertNull(context.getStartupFailure());
assertEquals(ApiRequestRiskDecision.Action.ALLOW, context.getBean(ApiRequestRiskEngine.class).evaluate(request).action());
});
}
@Test void installedRuleIsShadowedByDefaultAndEnforcedOnlyWhenConfigured() {
var configured = runner.withBean("blockedRule", ApiRequestRiskRule.class, () -> rule("test.block", 0));
configured.run(context -> assertEquals(ApiRequestRiskDecision.Action.SHADOW,
context.getBean(ApiRequestRiskEngine.class).evaluate(request).action()));
configured.withPropertyValues("moyi.gateway.risk-mode=enforce").run(context -> assertEquals(
ApiRequestRiskDecision.Action.DENY, context.getBean(ApiRequestRiskEngine.class).evaluate(request).action()));
}
@Test void throttleAggregationHonorsTheLongestRetryAndRuleFailureClosesOnlyInEnforce() {
var engine = new CompositeApiRequestRiskEngine(java.util.List.of(rule("test.short", 2), rule("test.long", 30)),
CompositeApiRequestRiskEngine.Mode.ENFORCE);
assertEquals(30, engine.evaluate(request).retryAfterSeconds());
ApiRequestRiskRule broken = new ApiRequestRiskRule() {
public String id() { return "test.broken"; }
public Optional<ApiRequestRiskSignal> evaluate(ApiRequestContext context) { throw new IllegalStateException(); }
};
assertEquals(ApiRequestRiskDecision.Action.SHADOW, new CompositeApiRequestRiskEngine(java.util.List.of(broken),
CompositeApiRequestRiskEngine.Mode.SHADOW).evaluate(request).action());
assertEquals(ApiRequestRiskDecision.Action.UNAVAILABLE, new CompositeApiRequestRiskEngine(java.util.List.of(broken),
CompositeApiRequestRiskEngine.Mode.ENFORCE).evaluate(request).action());
assertFalse(request.metadata().toString().contains("198.51.100.7"));
}
@Test void invalidModeDoesNotSilentlyDisableRisk() {
runner.withPropertyValues("moyi.gateway.risk-mode=typo").run(context -> assertNotNull(context.getStartupFailure()));
}
private ApiRequestRiskRule rule(String id, int retry) {
return new ApiRequestRiskRule() {
public String id() { return id; }
public Optional<ApiRequestRiskSignal> evaluate(ApiRequestContext context) {
return Optional.of(new ApiRequestRiskSignal(retry == 0 ? ApiRequestRiskSignal.Action.DENY : ApiRequestRiskSignal.Action.THROTTLE,
id, "RISK_BLOCKED", "Blocked", retry));
}
};
}
}
@@ -0,0 +1,197 @@
package com.moyi.controlplane.gateway.request;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.gateway.ChatMeter;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.catalog.CatalogService;
import com.moyi.controlplane.task.AsyncTaskService;
import com.moyi.controlplane.usage.UsageAuditService;
import com.moyi.controlplane.wallet.ChatBillingService;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.atomic.AtomicBoolean;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.jdbc.core.JdbcTemplate;
class GatewayRequestExecutionServiceTest {
private final UUID userId = UUID.randomUUID();
private final UUID keyId = UUID.randomUUID();
private final ApiRequestContext context = new ApiRequestContext(userId, keyId,
ApiRequestProtocol.CHAT_COMPLETIONS, "public-model", "route-1", false,
256, 128, ApiRequestMetadata.fromHttp("127.0.0.1"), System.nanoTime());
private final Map<String, Object> key = Map.of("id", keyId.toString(), "userId", userId.toString());
private final Map<String, Object> model = Map.of("id", "route-1");
private final ChatMeter.Request normalized = new ChatMeter.Request(Map.of("model", "public-model"), 256, 128);
@Test
void denyHappensBeforeAdmissionAndWalletReservation() {
ChatBillingService billing = mock(ChatBillingService.class);
var service = new GatewayRequestExecutionService(billing,
new ApiRequestRiskPolicy(ignored -> ApiRequestRiskDecision.deny(
new ApiRequestRiskSignal(ApiRequestRiskSignal.Action.DENY, "test.block", "RISK_BLOCKED", "Blocked", 0))));
AtomicBoolean admitted = new AtomicBoolean();
ApiException error = assertThrows(ApiException.class, () -> service.execute(context, key, model, normalized,
null, () -> { admitted.set(true); return mock(GatewayGuard.Admission.class); },
ticket -> "unreachable", failure -> new ApiException("CALL_FAILED", "failed", 502),
ApiException::getCode));
assertEquals(403, error.getStatus());
assertFalse(admitted.get());
verifyNoInteractions(billing);
}
@Test
void shadowSignalDoesNotChangeRequestExecution() {
ChatBillingService billing = mock(ChatBillingService.class);
var ticket = new ChatBillingService.Ticket(UUID.randomUUID(), userId, null);
when(billing.reserve(key, model, normalized, null, context.metadata())).thenReturn(ticket);
var service = new GatewayRequestExecutionService(billing,
new ApiRequestRiskPolicy(ignored -> ApiRequestRiskDecision.shadow(List.of("test.observe"))));
ApiRequestAdmission admission = mock(ApiRequestAdmission.class);
String result = service.execute(context, key, model, normalized, null, () -> admission,
ignored -> "executed", failure -> new ApiException("CALL_FAILED", "failed", 502), ApiException::getCode);
assertEquals("executed", result);
verify(billing).reserve(key, model, normalized, null, context.metadata());
verify(admission).close();
}
@Test
void enforceModeFailsClosedWhenRiskEngineThrowsBeforeAdmission() {
ChatBillingService billing = mock(ChatBillingService.class);
var service = new GatewayRequestExecutionService(billing,
new ApiRequestRiskPolicy(ignored -> { throw new IllegalStateException("private detail"); }));
AtomicBoolean admitted = new AtomicBoolean();
ApiException error = assertThrows(ApiException.class, () -> service.execute(context, key, model, normalized,
null, () -> { admitted.set(true); return mock(GatewayGuard.Admission.class); },
ticket -> "unreachable", failure -> new ApiException("CALL_FAILED", "failed", 502),
ApiException::getCode));
assertEquals("RISK_ENGINE_UNAVAILABLE", error.getCode());
assertEquals(503, error.getStatus());
assertFalse(admitted.get());
assertFalse(error.getMessage().contains("private detail"));
verifyNoInteractions(billing);
}
@Test
void admissionLeaseClosesWhenReservationFails() {
ChatBillingService billing = mock(ChatBillingService.class);
when(billing.reserve(key, model, normalized, null, context.metadata())).thenThrow(new ApiException("INSUFFICIENT_BALANCE", "low", 402));
var service = new GatewayRequestExecutionService(billing, new ApiRequestRiskPolicy(ApiRequestRiskEngine.allowAll()));
ApiRequestAdmission admission = mock(ApiRequestAdmission.class);
assertEquals("INSUFFICIENT_BALANCE", assertThrows(ApiException.class, () -> service.execute(context, key, model,
normalized, null, () -> admission, ticket -> "unreachable",
failure -> new ApiException("CALL_FAILED", "failed", 502), ApiException::getCode)).getCode());
verify(admission).close();
}
@Test
void workFailureIsRecordedAndAdmissionLeaseCloses() {
ChatBillingService billing = mock(ChatBillingService.class);
var ticket = new ChatBillingService.Ticket(UUID.randomUUID(), userId, null);
when(billing.reserve(key, model, normalized, null, context.metadata())).thenReturn(ticket);
var service = new GatewayRequestExecutionService(billing, new ApiRequestRiskPolicy(ApiRequestRiskEngine.allowAll()));
ApiRequestAdmission admission = mock(ApiRequestAdmission.class);
assertEquals("CALL_INTERRUPTED", service.execute(context, key, model, normalized, null, () -> admission,
ignored -> { throw new IllegalStateException("failure"); },
failure -> new ApiException("CALL_INTERRUPTED", "failed", 502), ApiException::getCode));
verify(billing).fail(eq(ticket), eq("CALL_INTERRUPTED"), eq("failed"), anyInt());
verify(admission).close();
}
@Test
void asyncTaskRiskDenialPrecedesQueueAndWalletDatabaseWork() {
JdbcTemplate jdbc = mock(JdbcTemplate.class);
CatalogService catalog = mock(CatalogService.class);
GatewayGuard guard = mock(GatewayGuard.class);
UsageAuditService usage = mock(UsageAuditService.class);
var route = new java.util.LinkedHashMap<String,Object>();
route.put("id", "async-route");
route.put("providerId", "provider-mock");
route.put("type", "3d");
route.put("mode", "async");
route.put("executionReady", true);
var apiKey = Map.<String,Object>of("id", keyId.toString(), "userId", userId.toString());
when(catalog.listModels(false)).thenReturn(List.<Map<String,Object>>of(route));
when(catalog.activeModel("async-route")).thenReturn(route);
when(catalog.keyCanInvoke(apiKey, "async-route")).thenReturn(true);
when(catalog.keyOwnerId(keyId)).thenReturn(userId);
var risk = new ApiRequestRiskPolicy(ignored -> ApiRequestRiskDecision.deny(
new ApiRequestRiskSignal(ApiRequestRiskSignal.Action.DENY, "test.async-block", "RISK_BLOCKED", "Blocked", 0)));
var tasks = new AsyncTaskService(jdbc, new ObjectMapper(), catalog, 100, 4, 4, 3_600_000,
false, guard, usage, null, null, risk);
ApiException error = assertThrows(ApiException.class,
() -> tasks.createForKey(apiKey, "3d", Map.of("model", "async-route", "prompt", "generate"), ApiRequestMetadata.playground("127.0.0.1")));
assertEquals("RISK_BLOCKED", error.getCode());
verify(guard).checkRate(keyId,false);
verifyNoMoreInteractions(guard);
verifyNoInteractions(jdbc, usage);
}
@Test
void publicMetadataCannotBePromotedToPlaygroundByRequestHeaderAndContextStringOmitsIp() {
var publicMetadata = ApiRequestMetadata.fromHttp("198.51.100.7");
var playgroundMetadata = ApiRequestMetadata.playground("198.51.100.7");
assertEquals(ApiRequestMetadata.Source.API, publicMetadata.source());
assertEquals(ApiRequestMetadata.Source.PLAYGROUND, playgroundMetadata.source());
assertFalse(context.toString().contains("127.0.0.1"));
}
@Test
void compositeRiskModeReportsSignalsInShadowAndAppliesThemOnlyInEnforce() {
ApiRequestRiskRule rule = new ApiRequestRiskRule() {
@Override public String id() { return "test.rule"; }
@Override public java.util.Optional<ApiRequestRiskSignal> evaluate(ApiRequestContext ignored) {
return java.util.Optional.of(new ApiRequestRiskSignal(ApiRequestRiskSignal.Action.DENY,
id(), "RISK_BLOCKED", "Blocked", 0));
}
};
var shadow = new CompositeApiRequestRiskEngine(List.of(rule), CompositeApiRequestRiskEngine.Mode.SHADOW)
.evaluate(context);
var enforce = new CompositeApiRequestRiskEngine(List.of(rule), CompositeApiRequestRiskEngine.Mode.ENFORCE)
.evaluate(context);
assertEquals(ApiRequestRiskDecision.Action.SHADOW, shadow.action());
assertEquals(List.of("test.rule"), shadow.ruleIds());
assertEquals(ApiRequestRiskDecision.Action.DENY, enforce.action());
assertEquals("RISK_BLOCKED", enforce.code());
}
@Test
void settlementPersistenceFailureDoesNotMarkCompletedProviderWorkAsFailed() {
ChatBillingService billing = mock(ChatBillingService.class);
var ticket = new ChatBillingService.Ticket(UUID.randomUUID(), userId, null);
when(billing.reserve(key, model, normalized, null, context.metadata())).thenReturn(ticket);
doThrow(new org.springframework.dao.DataIntegrityViolationException("ledger unavailable"))
.when(billing).settle(eq(ticket), anyMap(), anyInt(), any());
var service = new GatewayRequestExecutionService(billing, new ApiRequestRiskPolicy(ApiRequestRiskEngine.allowAll()));
ApiRequestAdmission admission = mock(ApiRequestAdmission.class);
String result = service.execute(context, key, model, normalized, null, () -> admission,
current -> {
service.settle(current, Map.of("usage", Map.of("prompt_tokens", 1, "completion_tokens", 1)), 2, null);
return "unexpected";
}, failure -> new ApiException("CALL_INTERRUPTED", "failed", 502), ApiException::getCode);
assertEquals("SETTLEMENT_PENDING", result);
verify(billing, never()).fail(any(), anyString(), anyString(), anyInt());
verify(admission).close();
}
}
@@ -0,0 +1,37 @@
package com.moyi.controlplane.gateway.request;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
/** Lightweight source dependency gates, not a substitute for design review or a sandbox. */
class ModuleDependencyTest {
private final Path root = Path.of(System.getProperty("basedir", "."), "src/main/java/com/moyi/controlplane");
@Test void requestApplicationBoundaryDoesNotOwnHttpDatabaseOrProviderCredentials() throws Exception {
disallow(root.resolve("gateway/request"), List.of("java.net.http", "org.springframework.jdbc", ".security.SecretCipher", ".provider.credentials", ".task.AsyncProviderClient"));
}
@Test void transportDoesNotOwnRiskRoutingOrTheWallet() throws Exception {
disallow(root.resolve("provider/transport"), List.of(".gateway.", ".wallet.", ".auth.", ".task.", "org.springframework.jdbc", ".security.SecretCipher"));
}
@Test void taskSettlementDoesNotCallAnUpstream() throws Exception {
assertTrue(Files.exists(root.resolve("task/TaskSettlementService.java")));
disallow(root.resolve("task/TaskSettlementService.java"), List.of("java.net.http", "AsyncProviderClient", "OpenAiCompatibleProvider"));
}
@Test void mediaInputDoesNotOwnCredentialsRoutingOrBilling() throws Exception {
disallow(root.resolve("provider/media"), List.of(".gateway.", ".wallet.", ".auth.", ".task.",
".provider.credentials", "org.springframework.jdbc", ".security.SecretCipher", "java.net.http.HttpClient"));
assertFalse(Files.readString(root.resolve("task/AsyncProviderClient.java")).contains("InetAddress"), "DNS policy belongs to the media boundary");
}
private void disallow(Path path, List<String> forbidden) throws Exception {
assertTrue(Files.exists(path), "Expected source boundary: " + path);
try (var paths = Files.isDirectory(path) ? Files.walk(path) : java.util.stream.Stream.of(path)) {
for (Path file : paths.filter(p -> p.toString().endsWith(".java")).toList()) {
String source = Files.readString(file);
for (String dependency : forbidden) assertFalse(source.contains(dependency), file + " crosses boundary via " + dependency);
}
}
}
}
@@ -112,6 +112,31 @@ class OpenAiCompatibleProviderHttpTest {
assertEquals(List.of("Bearer route-key", "Bearer model-key"), authorization);
}
@Test
void dynamicDatabaseChannelKeepsRotationAcrossRepeatedResolution() throws Exception {
var authorization = new ArrayList<String>();
server.createContext("/v1/chat/completions", exchange -> {
authorization.add(exchange.getRequestHeaders().getFirst("Authorization"));
exchange.getRequestBody().readAllBytes();
write(exchange, 200, "{\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1}}");
});
server.start();
var jdbc = org.mockito.Mockito.mock(JdbcTemplate.class);
var cipher = new SecretCipher("local-test-only");
org.mockito.Mockito.when(jdbc.queryForMap(org.mockito.ArgumentMatchers.anyString(), org.mockito.ArgumentMatchers.eq("dynamic-provider")))
.thenReturn(Map.of("code", "dynamic-code", "base_url", "http://127.0.0.1:" + server.getAddress().getPort() + "/v1",
"credential_ciphertext", cipher.encrypt("first-key")));
environment.withProperty("MOYI_UPSTREAM_API_KEYS_DYNAMIC_CODE", "second-key");
var dynamic = new OpenAiCompatibleProvider(json, environment, jdbc, cipher);
try {
dynamic.chat("dynamic-provider", "test-model", "route-no-override", textInput(128));
dynamic.chat("dynamic-provider", "test-model", "route-no-override", textInput(128));
dynamic.chat("dynamic-provider", "test-model", "route-no-override", textInput(128));
} finally { dynamic.close(); }
assertEquals(List.of("Bearer first-key", "Bearer second-key", "Bearer first-key"), authorization);
}
private Map<String, Object> textInput(int maxTokens) {
var input = new LinkedHashMap<String, Object>();
input.put("messages", List.of(Map.of("role", "user", "content", "Reply with OK.")));
@@ -0,0 +1,64 @@
package com.moyi.controlplane.provider.credentials;
import com.moyi.controlplane.common.ApiException;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Test;
import org.springframework.mock.env.MockEnvironment;
import static org.junit.jupiter.api.Assertions.*;
class UpstreamCredentialResolverTest {
@Test
void scopedPoolUsesRouteThenProviderModelAndNeverGlobalModelKeys() {
var env = new MockEnvironment()
.withProperty("MOYI_UPSTREAM_API_KEYS_ROUTE_ROUTE_A", "route-first | route-second | route-first")
.withProperty("MOYI_UPSTREAM_API_KEYS_MODEL_RELAY_CODE_MODEL_1", "model-first|model-second")
.withProperty("MOYI_UPSTREAM_API_KEYS_MODEL_MODEL_1", "unscoped-key");
var resolver = new UpstreamCredentialResolver(env);
assertEquals(List.of("route-first", "route-second"), resolver.scopedKeys("relay-code", "route-a", "model-1"));
assertEquals(List.of("model-first", "model-second"), resolver.scopedKeys("relay-code", "route-b", "model-1"));
assertTrue(resolver.scopedKeys("other-provider", "route-b", "model-1").isEmpty());
assertTrue(resolver.scopedKeys("relay-code", null, null).isEmpty());
}
@Test
void channelPoolPreservesFallbackAndExplicitAliasOrderAndDeduplicates() {
var resolver = new UpstreamCredentialResolver(new MockEnvironment()
.withProperty("MOYI_UPSTREAM_API_KEYS_PROVIDER_A", "a|shared")
.withProperty("MOYI_UPSTREAM_API_KEYS_CHANNEL_CODE", "b|shared"));
assertEquals(List.of("fallback", "a", "shared", "b"),
resolver.providerKeys(" fallback ", "provider-a", "channel-code"));
assertEquals(List.of("fallback", "b", "shared", "a"),
resolver.providerKeys("fallback", "channel-code", "provider-a"));
}
@Test
void parsingHandlesBlankEntriesAndReturnsImmutablePools() {
assertEquals(List.of("a", "b"), UpstreamCredentialResolver.parseKeys(" a | |b|a| "));
assertTrue(UpstreamCredentialResolver.parseKeys(null).isEmpty());
assertTrue(UpstreamCredentialResolver.parseKeys(" || ").isEmpty());
assertThrows(UnsupportedOperationException.class, () -> UpstreamCredentialResolver.parseKeys("a").add("b"));
}
@Test
void rotationUsesPersistentCallerCursorAndSurvivesIntegerOverflow() {
var cursor = new AtomicInteger(Integer.MAX_VALUE);
var keys = List.of("a", "b", "c");
assertEquals(keys.get(Math.floorMod(Integer.MAX_VALUE, keys.size())), UpstreamCredentialResolver.nextKey(keys, cursor));
assertEquals(keys.get(Math.floorMod(Integer.MIN_VALUE, keys.size())), UpstreamCredentialResolver.nextKey(keys, cursor));
assertEquals(Integer.MIN_VALUE + 1, cursor.get());
assertEquals("UPSTREAM_KEY_NOT_AVAILABLE", assertThrows(ApiException.class,
() -> UpstreamCredentialResolver.nextKey(List.of(), cursor)).getCode());
}
@Test
void officialFallbackKeysAreReadOnlyWhenRequestedExplicitly() {
var resolver = new UpstreamCredentialResolver(new MockEnvironment().withProperty("TRIPO_API_KEY", "tripo-a|tripo-b"));
assertTrue(resolver.providerKeys("", "provider-tripo").isEmpty());
assertEquals(List.of("tripo-a", "tripo-b"), resolver.environmentKeys("TRIPO_API_KEY"));
}
}
@@ -0,0 +1,58 @@
package com.moyi.controlplane.provider.media;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.provider.transport.BoundedHttpTransport;
import com.moyi.controlplane.provider.transport.PinnedHttpsTransport;
import java.net.URI;
import java.net.http.HttpHeaders;
import java.time.Duration;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
class MediaDownloaderTest {
private final PinnedHttpsTransport transport = mock(PinnedHttpsTransport.class);
private final MediaDownloader downloader = new MediaDownloader(transport, host -> { throw new AssertionError("Unexpected network"); });
@Test void normalizesImagesAndPreservesBoundedTransportContract() throws Exception {
byte[] bytes = {1, 2, 3};
when(transport.get(any(), any(), any(), anyInt())).thenReturn(response(200, "IMAGE/PNG; charset=utf-8", bytes));
var image = downloader.download("https://media.test/a?signature=synthetic");
assertArrayEquals(bytes, image.bytes()); assertEquals("image/png", image.contentType()); assertEquals("input.png", image.filename());
verify(transport).get(eq(URI.create("https://media.test/a?signature=synthetic")), any(), eq(Duration.ofSeconds(15)), eq(8 * 1024 * 1024));
assertFalse(image.toString().contains("synthetic"));
}
@Test void invalidUrlsFailBeforeTransport() {
assertEquals("INVALID_MEDIA_URL", assertThrows(ApiException.class, () -> downloader.download("https://user:secret@media.test/a")).getCode());
verifyNoInteractions(transport);
}
@ParameterizedTest @CsvSource({"BODY_TOO_LARGE,MEDIA_TOO_LARGE", "INVALID_ADDRESS,INVALID_MEDIA_URL", "TIMEOUT,MEDIA_FETCH_FAILED", "INTERRUPTED,MEDIA_FETCH_FAILED", "BUSY,MEDIA_FETCH_FAILED", "IO,MEDIA_FETCH_FAILED"})
void transportFailuresExposeOnlySafeDomainCodes(BoundedHttpTransport.Reason reason, String code) throws Exception {
var failure = mock(BoundedHttpTransport.Failure.class); when(failure.reason()).thenReturn(reason);
when(transport.get(any(), any(), any(), anyInt())).thenThrow(failure);
var error = assertThrows(ApiException.class, () -> downloader.download("https://media.test/a?signature=private"));
assertEquals(code, error.getCode()); assertFalse(error.getMessage().contains("private")); assertNull(error.getCause());
}
@ParameterizedTest @CsvSource({"302,image/png,MEDIA_FETCH_FAILED", "404,image/png,MEDIA_FETCH_FAILED", "200,text/html,MEDIA_TYPE_UNSUPPORTED", "200,image/invalid:type,MEDIA_TYPE_UNSUPPORTED"})
void redirectsErrorsAndUnsupportedTypesAreRejected(int status, String type, String code) throws Exception {
when(transport.get(any(), any(), any(), anyInt())).thenReturn(response(status, type, new byte[] {1}));
assertEquals(code, assertThrows(ApiException.class, () -> downloader.download("https://media.test/a")).getCode());
}
@Test void emptyBodyIsRejected() throws Exception {
when(transport.get(any(), any(), any(), anyInt())).thenReturn(response(200, "image/png", new byte[0]));
assertEquals("MEDIA_TOO_LARGE", assertThrows(ApiException.class, () -> downloader.download("https://media.test/a")).getCode());
}
private BoundedHttpTransport.Response response(int status, String type, byte[] body) {
return new BoundedHttpTransport.Response(status, HttpHeaders.of(Map.of("Content-Type", List.of(type)), (key, value) -> true), body);
}
}
@@ -0,0 +1,65 @@
package com.moyi.controlplane.provider.media;
import com.moyi.controlplane.provider.transport.PinnedHttpsTransport;
import java.net.InetAddress;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.*;
class PublicMediaAddressPolicyTest {
private final PublicMediaAddressPolicy policy = new PublicMediaAddressPolicy();
@ParameterizedTest
@ValueSource(strings = {
"0.0.0.0", "0.255.255.255", "10.0.0.1", "127.0.0.1", "127.255.255.254",
"100.64.0.1", "100.127.255.254", "169.254.169.254", "172.16.0.1", "172.31.255.254",
"192.0.0.1", "192.0.0.9", "192.0.2.1", "192.88.99.1", "192.168.1.1",
"198.18.0.1", "198.19.255.254", "198.51.100.1", "203.0.113.1", "224.0.0.1",
"239.255.255.255", "240.0.0.1", "255.255.255.255", "168.63.129.16",
"::", "::1", "::127.0.0.1", "::ffff:127.0.0.1", "::ffff:169.254.169.254",
"fc00::1", "fd00::1", "fe80::1", "fec0::1", "ff02::1", "64:ff9b::a00:1", "64:ff9b:1::1",
"100::1", "2001::1", "2001:1ff:ffff::1", "2001:2::1", "2001:20::1", "2001:db8::1",
"2002:7f00:1::1", "2002:0808:0808::1", "3ffe::1", "3fff::1", "3fff:fff:ffff::1", "4000::1"
})
void rejectsNonGlobalSpecialAndTranslationTargets(String literal) throws Exception {
InetAddress address = InetAddress.getByName(literal);
assertFalse(policy.isPublic(address), literal);
assertThrows(PinnedHttpsTransport.AddressRejectedException.class, () -> policy.requirePublic(new InetAddress[] {address}));
}
@ParameterizedTest
@ValueSource(strings = {"1.1.1.1", "8.8.8.8", "100.63.255.254", "100.128.0.1", "172.15.255.254", "172.32.0.1",
"192.0.1.1", "198.17.255.254", "198.20.0.1", "223.255.255.254", "2606:4700:4700::1111", "2001:4860:4860::8888", "3fff:1000::1"})
void permitsOrdinaryGlobalAddressesWithoutMakingInternetRequests(String literal) throws Exception {
var address = InetAddress.getByName(literal);
assertTrue(policy.isPublic(address), literal);
assertArrayEquals(new InetAddress[] {address}, policy.requirePublic(new InetAddress[] {address}));
}
@Test void rejectsTheWholeMixedAnswerSetInEitherOrder() throws Exception {
var external = InetAddress.getByName("8.8.8.8");
var internal = InetAddress.getByName("fd00::1");
assertThrows(PinnedHttpsTransport.AddressRejectedException.class, () -> policy.requirePublic(new InetAddress[] {external, internal}));
assertThrows(PinnedHttpsTransport.AddressRejectedException.class, () -> policy.requirePublic(new InetAddress[] {internal, external}));
assertThrows(PinnedHttpsTransport.AddressRejectedException.class, () -> policy.requirePublic(new InetAddress[0]));
assertThrows(PinnedHttpsTransport.AddressRejectedException.class, () -> policy.requirePublic(new InetAddress[] {null}));
assertThrows(PinnedHttpsTransport.AddressRejectedException.class, () -> policy.requirePublic(new InetAddress[17]));
}
@ParameterizedTest
@ValueSource(strings = {"http://example.test/a", "file:///etc/passwd", "data:image/png,a", "//example.test/a",
"https://user:secret@example.test/a", "https://example.test/a#fragment", "https://example.test:0/a", "https://example.test:65536/a",
"https://[fe80::1%25eth0]/a", "https://example.test\\@127.0.0.1/a", "https://example.test/%0d%0a/x\r\n", "https:///a", ""})
void rejectsAmbiguousOrNonHttpsUris(String value) {
var failure = assertThrows(IllegalArgumentException.class, () -> policy.requireUri(value));
assertFalse(failure.getMessage().contains("secret"));
assertNull(failure.getCause());
}
@Test void permitsSignedHttpsUrlsAndExplicitHttpsPortsWithoutRewritingAuthority() {
String value = "https://media.example.test:8443/a.png?signature=synthetic";
assertEquals(value, policy.requireUri(value).toString());
assertEquals("[2606:4700:4700::1111]", policy.requireUri("https://[2606:4700:4700::1111]/a").getHost());
}
}
@@ -0,0 +1,113 @@
package com.moyi.controlplane.provider.transport;
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.HttpRequest;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class BoundedHttpTransportTest {
private HttpServer server;
private final BoundedHttpTransport transport = new BoundedHttpTransport();
private final CountDownLatch release = new CountDownLatch(1);
@BeforeEach void setup() throws Exception { server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); }
@AfterEach void cleanup() { release.countDown(); server.stop(0); }
private HttpRequest request(String path, long timeoutMs) { return HttpRequest.newBuilder(URI.create("http://127.0.0.1:" + server.getAddress().getPort() + path)).timeout(Duration.ofMillis(timeoutMs)).GET().build(); }
@Test void returnsBoundedBytesWithoutExposingBodyInDiagnostics() throws Exception {
server.createContext("/", exchange -> {
byte[] bytes = "private-provider-result".getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(200, bytes.length);
try (var output = exchange.getResponseBody()) { output.write(bytes); }
}); server.start();
var response = transport.send(request("/", 2000), 1024);
assertEquals("private-provider-result", new String(response.body(), StandardCharsets.UTF_8));
assertFalse(response.toString().contains("private-provider-result"));
}
@Test void oversizedChunkedResponseIsCancelledBeforeUnboundedBuffering() {
var calls = new AtomicInteger();
server.createContext("/", exchange -> {
calls.incrementAndGet(); exchange.sendResponseHeaders(200, 0);
try (var output = exchange.getResponseBody()) { for (int i = 0; i < 100; i++) output.write(new byte[4096]); }
catch (java.io.IOException expectedAfterCancellation) { }
}); server.start();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> transport.send(request("/", 2000), 8192));
assertEquals(BoundedHttpTransport.Reason.BODY_TOO_LARGE, failure.reason());
assertEquals(200, failure.observedStatus()); assertEquals(1, calls.get());
}
@Test void declaredOversizeIsRejectedWithoutWaitingForTheBody() {
server.createContext("/", exchange -> {
exchange.sendResponseHeaders(200, 1_000_000);
try { release.await(3, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
exchange.close();
}); server.start();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> transport.send(request("/", 1000), 8192));
assertEquals(BoundedHttpTransport.Reason.BODY_TOO_LARGE, failure.reason());
}
@Test void totalDeadlineIncludesBodyReadsAfterHeaders() {
var calls = new AtomicInteger();
server.createContext("/", exchange -> {
calls.incrementAndGet(); exchange.sendResponseHeaders(200, 0);
try (var output = exchange.getResponseBody()) {
output.write('x'); output.flush();
try { release.await(3, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
} catch (java.io.IOException expectedAfterCancellation) { }
}); server.start();
long start = System.nanoTime();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> transport.send(request("/", 200), 8192));
assertEquals(BoundedHttpTransport.Reason.TIMEOUT, failure.reason());
assertTrue(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) < 2000);
assertEquals(1, calls.get());
}
@Test void errorBodiesUseASmallerLimitAndRetainOnlyObservedStatus() {
server.createContext("/", exchange -> { exchange.sendResponseHeaders(401, 1_000_000); exchange.close(); }); server.start();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> transport.send(request("/?private-query", 1000), 8 * 1024 * 1024));
assertEquals(BoundedHttpTransport.Reason.BODY_TOO_LARGE, failure.reason());
assertEquals(401, failure.observedStatus());
assertFalse(failure.toString().contains("private-query"));
}
@Test void redirectsAreReturnedAndNeverFollowed() throws Exception {
var redirected = new AtomicInteger();
server.createContext("/start", exchange -> { exchange.getResponseHeaders().set("Location", "/internal"); exchange.sendResponseHeaders(302, -1); exchange.close(); });
server.createContext("/internal", exchange -> { redirected.incrementAndGet(); exchange.sendResponseHeaders(200, -1); exchange.close(); }); server.start();
assertEquals(302, transport.send(request("/start", 2000), 1024).statusCode());
assertEquals(0, redirected.get());
}
@Test void interruptionCancelsTheAttemptAndRestoresInterruptFlag() throws Exception {
var arrived = new CountDownLatch(1); var result = new AtomicReference<BoundedHttpTransport.Reason>(); var interrupted = new AtomicReference<Boolean>();
server.createContext("/", exchange -> {
exchange.sendResponseHeaders(200, 0); arrived.countDown();
try { release.await(5, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
exchange.close();
}); server.start();
var worker = new Thread(() -> {
try { transport.send(request("/", 5000), 1024); }
catch (BoundedHttpTransport.Failure failure) { result.set(failure.reason()); interrupted.set(Thread.currentThread().isInterrupted()); }
}); worker.setDaemon(true); worker.start();
try {
assertTrue(arrived.await(2, TimeUnit.SECONDS)); worker.interrupt(); worker.join(2000);
assertFalse(worker.isAlive()); assertEquals(BoundedHttpTransport.Reason.INTERRUPTED, result.get()); assertEquals(true, interrupted.get());
} finally { release.countDown(); worker.interrupt(); }
}
@Test void callersCannotOmitDeadlinesOrRequestUnboundedResponses() {
var request = HttpRequest.newBuilder(URI.create("http://127.0.0.1:1/")).GET().build();
assertThrows(IllegalArgumentException.class, () -> transport.send(request, 1024));
assertThrows(IllegalArgumentException.class, () -> transport.send(request, Integer.MAX_VALUE));
}
}
@@ -0,0 +1,279 @@
package com.moyi.controlplane.provider.transport;
import com.sun.net.httpserver.HttpsConfigurator;
import com.sun.net.httpserver.HttpsServer;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.time.Duration;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import javax.net.ssl.ExtendedSSLSession;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SNIHostName;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import static org.junit.jupiter.api.Assertions.*;
class PinnedHttpsTransportTest {
@TempDir static Path temporary;
private static SSLContext serverTls, trustedTls;
private HttpsServer server;
private ExecutorService workers;
private PinnedHttpsTransport transport;
private final CountDownLatch release = new CountDownLatch(1);
@BeforeAll static void createEphemeralTestCertificate() throws Exception {
// No committed key, no trust-all TLS: mint a .test certificate and trust only this test key.
String password = java.util.UUID.randomUUID().toString();
Path store = temporary.resolve("media-test.p12");
var process = new ProcessBuilder(Path.of(System.getProperty("java.home"), "bin", "keytool").toString(),
"-genkeypair", "-alias", "test", "-keyalg", "RSA", "-keysize", "2048", "-validity", "2",
"-dname", "CN=media.test", "-ext", "SAN=dns:media.test", "-storetype", "PKCS12",
"-keystore", store.toString(), "-storepass", password, "-keypass", password, "-noprompt")
.redirectErrorStream(true).start();
assertTrue(process.waitFor(15, TimeUnit.SECONDS));
assertEquals(0, process.exitValue(), () -> "Unable to create ephemeral TLS fixture");
var keys = KeyStore.getInstance("PKCS12");
try (var input = Files.newInputStream(store)) { keys.load(input, password.toCharArray()); }
var kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(keys, password.toCharArray());
serverTls = SSLContext.getInstance("TLS");
serverTls.init(kmf.getKeyManagers(), null, null);
var trust = KeyStore.getInstance(KeyStore.getDefaultType()); trust.load(null, null);
trust.setCertificateEntry("test", keys.getCertificate("test"));
var tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm()); tmf.init(trust);
trustedTls = SSLContext.getInstance("TLS"); trustedTls.init(null, tmf.getTrustManagers(), null);
}
@BeforeEach void setup() throws Exception {
server = HttpsServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.setHttpsConfigurator(new HttpsConfigurator(serverTls));
workers = new ThreadPoolExecutor(0, 2, 5, TimeUnit.SECONDS, new SynchronousQueue<>(),
runnable -> { var thread = new Thread(runnable, "test-pinned-get"); thread.setDaemon(true); return thread; });
transport = new PinnedHttpsTransport(trustedTls, workers);
}
@AfterEach void cleanup() throws Exception {
release.countDown(); workers.shutdownNow(); server.stop(0);
assertTrue(workers.awaitTermination(3, TimeUnit.SECONDS));
}
private URI uri(String path) { return URI.create("https://media.test:" + server.getAddress().getPort() + path); }
private InetAddress[] loopback(String host) throws java.io.IOException { return new InetAddress[] {InetAddress.getByName("127.0.0.1")}; }
private BoundedHttpTransport.Response get(String path, long timeoutMs, int maxBytes) throws Exception {
return transport.get(uri(path), this::loopback, Duration.ofMillis(timeoutMs), maxBytes);
}
@Test void bindsDnsOnceWhilePreservingHostSniAndCredentialIsolation() throws Exception {
var lookups = new AtomicInteger(); var host = new AtomicReference<String>(); var sni = new AtomicReference<String>();
server.createContext("/", exchange -> {
host.set(exchange.getRequestHeaders().getFirst("Host"));
var session = (ExtendedSSLSession) ((com.sun.net.httpserver.HttpsExchange) exchange).getSSLSession();
sni.set(((SNIHostName) session.getRequestedServerNames().getFirst()).getAsciiName());
assertNull(exchange.getRequestHeaders().getFirst("Authorization"));
assertNull(exchange.getRequestHeaders().getFirst("Cookie"));
assertNull(exchange.getRequestHeaders().getFirst("Proxy-Authorization"));
assertEquals("identity", exchange.getRequestHeaders().getFirst("Accept-Encoding"));
exchange.getResponseHeaders().set("Content-Type", "image/png");
byte[] bytes = "synthetic-image".getBytes(StandardCharsets.UTF_8); exchange.sendResponseHeaders(200, bytes.length);
try (var output = exchange.getResponseBody()) { output.write(bytes); }
}); server.start();
var response = transport.get(uri("/?signature=synthetic-private-query"), target -> {
assertEquals("media.test", target);
if (lookups.incrementAndGet() != 1) throw new AssertionError("DNS was resolved twice");
return loopback(target);
}, Duration.ofSeconds(3), 1024);
assertEquals(200, response.statusCode()); assertEquals(1, lookups.get());
assertEquals("media.test:" + server.getAddress().getPort(), host.get()); assertEquals("media.test", sni.get());
assertEquals("synthetic-image", new String(response.body(), StandardCharsets.UTF_8));
assertFalse(response.toString().contains("synthetic"));
}
@Test void aTrustedCertificateForAnotherHostnameStillFails() throws Exception {
var calls = new AtomicInteger();
server.createContext("/", exchange -> { calls.incrementAndGet(); exchange.sendResponseHeaders(200, -1); exchange.close(); }); server.start();
URI wrong = URI.create("https://wrong.test:" + server.getAddress().getPort() + "/?signature=private");
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> transport.get(wrong, this::loopback, Duration.ofSeconds(3), 1024));
assertEquals(BoundedHttpTransport.Reason.IO, failure.reason()); assertEquals(0, calls.get());
assertNull(failure.getCause()); assertFalse(failure.toString().contains("private"));
}
@Test void theDefaultTrustStoreDoesNotAcceptOurSelfSignedTestCertificate() throws Exception {
server.createContext("/", exchange -> { exchange.sendResponseHeaders(200, -1); exchange.close(); }); server.start();
var strict = new PinnedHttpsTransport(org.apache.hc.core5.ssl.SSLContexts.createDefault(), workers);
assertEquals(BoundedHttpTransport.Reason.IO, assertThrows(BoundedHttpTransport.Failure.class,
() -> strict.get(uri("/"), this::loopback, Duration.ofSeconds(3), 1024)).reason());
}
@Test void redirectsAreNeverFollowed() throws Exception {
var calls = new AtomicInteger();
server.createContext("/", exchange -> {
calls.incrementAndGet(); exchange.getResponseHeaders().set("Location", "https://127.0.0.1/private");
exchange.sendResponseHeaders(302, -1); exchange.close();
}); server.start();
assertEquals(302, get("/", 3000, 1024).statusCode()); assertEquals(1, calls.get());
}
@Test void declaredOversizeDoesNotWaitForTheBodyOrDrainItOnClose() {
server.createContext("/", exchange -> {
exchange.sendResponseHeaders(200, 1_000_000);
try { release.await(3, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
exchange.close();
}); server.start();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 2000, 8192));
assertEquals(BoundedHttpTransport.Reason.BODY_TOO_LARGE, failure.reason()); assertEquals(200, failure.observedStatus());
}
@Test void chunkedOversizeIsStoppedDuringRead() {
server.createContext("/", exchange -> {
exchange.sendResponseHeaders(200, 0);
try (var output = exchange.getResponseBody()) { for (int i = 0; i < 100; i++) output.write(new byte[4096]); }
catch (java.io.IOException expectedAfterCancellation) { }
}); server.start();
assertEquals(BoundedHttpTransport.Reason.BODY_TOO_LARGE, assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 3000, 8192)).reason());
}
@Test void deadlineIncludesSlowResponseBodyAndRestoresWorkerCapacity() throws Exception {
var arrived = new CountDownLatch(1);
server.createContext("/", exchange -> {
exchange.sendResponseHeaders(200, 0);
try (var output = exchange.getResponseBody()) {
output.write('x'); output.flush(); arrived.countDown();
try { release.await(5, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
} catch (java.io.IOException expectedAfterCancellation) { }
}); server.start();
long start = System.nanoTime();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 1000, 8192));
assertEquals(BoundedHttpTransport.Reason.TIMEOUT, failure.reason()); assertEquals(200, failure.observedStatus());
assertEquals(0, arrived.getCount()); assertTrue(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start) < 2500);
workers.shutdown(); assertTrue(workers.awaitTermination(2, TimeUnit.SECONDS));
}
@Test void lateDnsResolutionCannotStartHttpAfterTheDeadline() throws Exception {
var resolving = new CountDownLatch(1); var finished = new CountDownLatch(1); var calls = new AtomicInteger();
server.createContext("/", exchange -> { calls.incrementAndGet(); exchange.sendResponseHeaders(200, -1); exchange.close(); }); server.start();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> transport.get(uri("/"), host -> {
resolving.countDown();
boolean interrupted = false;
while (release.getCount() != 0) try { release.await(1, TimeUnit.SECONDS); }
catch (InterruptedException ignoredForSimulatedNativeResolver) { interrupted = true; }
if (interrupted) Thread.currentThread().interrupt();
finished.countDown(); return loopback(host);
}, Duration.ofMillis(200), 1024));
assertEquals(BoundedHttpTransport.Reason.TIMEOUT, failure.reason()); assertEquals(0, resolving.getCount());
release.countDown(); assertTrue(finished.await(2, TimeUnit.SECONDS));
workers.shutdown(); assertTrue(workers.awaitTermination(2, TimeUnit.SECONDS)); assertEquals(0, calls.get());
}
@Test void interruptionCancelsSocketAndRestoresCallerInterruptFlag() throws Exception {
var arrived = new CountDownLatch(1); var observed = new AtomicReference<BoundedHttpTransport.Reason>(); var flag = new AtomicReference<Boolean>();
server.createContext("/", exchange -> {
exchange.sendResponseHeaders(200, 0);
try (var output = exchange.getResponseBody()) {
output.write('x'); output.flush(); arrived.countDown();
try { release.await(5, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
} catch (java.io.IOException expectedAfterCancellation) { }
}); server.start();
var caller = new Thread(() -> {
try { get("/", 5000, 1024); }
catch (BoundedHttpTransport.Failure failure) { observed.set(failure.reason()); flag.set(Thread.currentThread().isInterrupted()); }
catch (Exception unexpected) { throw new AssertionError(unexpected); }
}); caller.setDaemon(true); caller.start();
try {
assertTrue(arrived.await(3, TimeUnit.SECONDS)); caller.interrupt(); caller.join(2000);
assertFalse(caller.isAlive()); assertEquals(BoundedHttpTransport.Reason.INTERRUPTED, observed.get()); assertEquals(true, flag.get());
} finally { release.countDown(); caller.interrupt(); }
}
@Test void refusedOrDroppedRequestsAreNotAutomaticallyRetried() {
var calls = new AtomicInteger();
server.createContext("/", exchange -> { calls.incrementAndGet(); exchange.close(); }); server.start();
assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 3000, 1024)); assertEquals(1, calls.get());
}
@Test void compressedBodiesAreNotDecompressedOrTreatedAsImages() {
server.createContext("/", exchange -> {
exchange.getResponseHeaders().set("Content-Encoding", "gzip"); exchange.sendResponseHeaders(200, 0);
try (var output = exchange.getResponseBody()) { output.write(new byte[16]); }
catch (java.io.IOException expectedAfterCancellation) { }
}); server.start();
assertEquals(BoundedHttpTransport.Reason.IO, assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 3000, 1024)).reason());
}
@Test void busyPoolRejectsImmediatelyRatherThanQueueingUnlimitedDnsRequests() throws Exception {
var occupied = new CountDownLatch(1);
var oneWorker = new ThreadPoolExecutor(0, 1, 5, TimeUnit.SECONDS, new SynchronousQueue<>());
try {
oneWorker.submit(() -> {
occupied.countDown();
try { release.await(5, TimeUnit.SECONDS); } catch (InterruptedException error) { Thread.currentThread().interrupt(); }
});
assertTrue(occupied.await(2, TimeUnit.SECONDS));
var rejected = new PinnedHttpsTransport(trustedTls, oneWorker);
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> rejected.get(uri("/"), this::loopback, Duration.ofSeconds(3), 1024));
assertEquals(BoundedHttpTransport.Reason.BUSY, failure.reason()); assertNull(failure.observedStatus());
assertEquals(0, oneWorker.getQueue().size());
} finally { release.countDown(); oneWorker.shutdownNow(); assertTrue(oneWorker.awaitTermination(2, TimeUnit.SECONDS)); }
}
@Test void defaultSystemProxySelectorCannotBypassDnsBinding() throws Exception {
var original = java.net.ProxySelector.getDefault();
var selections = new AtomicInteger();
server.createContext("/", exchange -> { exchange.sendResponseHeaders(200, 1); exchange.getResponseBody().write('x'); exchange.close(); }); server.start();
try {
java.net.ProxySelector.setDefault(new java.net.ProxySelector() {
@Override public java.util.List<java.net.Proxy> select(URI target) {
selections.incrementAndGet();
return java.util.List.of(new java.net.Proxy(java.net.Proxy.Type.SOCKS, new InetSocketAddress("127.0.0.1", 1)));
}
@Override public void connectFailed(URI target, java.net.SocketAddress address, java.io.IOException failure) { }
});
assertEquals(200, get("/", 3000, 1024).statusCode()); assertEquals(0, selections.get());
} finally { java.net.ProxySelector.setDefault(original); }
}
@Test void independentAttemptsRevalidateDnsAndNeverReuseCookiesOrThePreviousConnection() throws Exception {
var calls = new AtomicInteger(); var lookups = new AtomicInteger();
server.createContext("/", exchange -> {
calls.incrementAndGet(); assertNull(exchange.getRequestHeaders().getFirst("Cookie"));
exchange.getResponseHeaders().set("Set-Cookie", "synthetic=value; Secure");
exchange.sendResponseHeaders(200, 1); exchange.getResponseBody().write('x'); exchange.close();
}); server.start();
PinnedHttpsTransport.AddressResolver resolver = host -> {
if (lookups.incrementAndGet() == 3) throw new PinnedHttpsTransport.AddressRejectedException();
return loopback(host);
};
assertEquals(200, transport.get(uri("/"), resolver, Duration.ofSeconds(3), 1024).statusCode());
assertEquals(200, transport.get(uri("/"), resolver, Duration.ofSeconds(3), 1024).statusCode());
assertEquals(BoundedHttpTransport.Reason.INVALID_ADDRESS, assertThrows(BoundedHttpTransport.Failure.class,
() -> transport.get(uri("/"), resolver, Duration.ofSeconds(3), 1024)).reason());
assertEquals(3, lookups.get()); assertEquals(2, calls.get());
}
@Test void errorBodyLimitRetainsStatusAndIsSmallerThanTheImageLimit() {
server.createContext("/", exchange -> { exchange.sendResponseHeaders(401, 1_000_000); exchange.close(); }); server.start();
var failure = assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 3000, 8 * 1024 * 1024));
assertEquals(BoundedHttpTransport.Reason.BODY_TOO_LARGE, failure.reason()); assertEquals(401, failure.observedStatus());
}
@Test void oversizedResponseHeadersAreNotBufferedWithoutLimits() {
server.createContext("/", exchange -> {
exchange.getResponseHeaders().set("X-Large", "x".repeat(16 * 1024));
exchange.sendResponseHeaders(200, 1); exchange.getResponseBody().write('x'); exchange.close();
}); server.start();
assertEquals(BoundedHttpTransport.Reason.IO, assertThrows(BoundedHttpTransport.Failure.class, () -> get("/", 3000, 1024)).reason());
}
}
@@ -0,0 +1,80 @@
package com.moyi.controlplane.provider.transport;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import java.io.ByteArrayInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class SseEventReaderTest {
private final SseEventReader reader = new SseEventReader(new ObjectMapper());
@Test
void fragmentedUtf8CrLfMultilineDataAndCommentsHaveOneEvent() throws Exception {
String data = ": heartbeat\r\nevent: result\r\ndata: {\"content\":\r\ndata: \"你好\"}\r\n\r\ndata: [DONE]\r\n\r\n";
var input = new FilterInputStream(bytes(data)) {
@Override public int read(byte[] value, int offset, int length) throws IOException {
return super.read(value, offset, Math.min(1, length));
}
};
var events = new ArrayList<Map<String,Object>>();
reader.read(input, () -> false, true, events::add);
assertEquals(1, events.size());
assertEquals("你好", events.getFirst().get("content"));
}
@Test
void chatRequiresDoneAndNativeRequiresWholeFramesEvenWithoutDone() throws Exception {
String event = "data: {\"type\":\"message_stop\"}\n\n";
assertEquals("UPSTREAM_STREAM_INCOMPLETE", assertThrows(ApiException.class,
() -> reader.read(bytes(event), () -> false, true, ignored -> {})).getCode());
assertDoesNotThrow(() -> reader.read(bytes(event), () -> false, false, ignored -> {}));
assertEquals("UPSTREAM_STREAM_INCOMPLETE", assertThrows(ApiException.class,
() -> reader.read(bytes("data: {\"type\":\"message_stop\"}"), () -> false, false, ignored -> {})).getCode());
}
@Test
void malformedJsonAndUtf8NeverReachConsumer() {
var delivered = new AtomicBoolean();
assertThrows(IOException.class, () -> reader.read(bytes("data: {not json}\n\n"), () -> false, true,
ignored -> delivered.set(true)));
assertThrows(IOException.class, () -> reader.read(new ByteArrayInputStream(new byte[]{(byte) 0xc3, 0x28}),
() -> false, true, ignored -> delivered.set(true)));
assertFalse(delivered.get());
}
@Test
void frameAndActualUtf8ByteBudgetsAreBothBounded() {
assertEquals("UPSTREAM_RESPONSE_TOO_LARGE", assertThrows(ApiException.class,
() -> reader.read(bytes("data: " + "x".repeat(262_145)), () -> false, true, ignored -> {})).getCode());
// Fewer than 2M characters, but more than 4MiB on the wire; comments cannot bypass the budget.
String multibyte = (": " + "中".repeat(1024) + "\n\n").repeat(1400);
assertEquals("UPSTREAM_RESPONSE_TOO_LARGE", assertThrows(ApiException.class,
() -> reader.read(bytes(multibyte), () -> false, true, ignored -> {})).getCode());
}
@Test
void transportDeadlineMapsReadFailureAndTransportRetainsStreamOwnership() throws Exception {
var closed = new AtomicBoolean();
var input = new ByteArrayInputStream("data: [DONE]\n\n".getBytes(StandardCharsets.UTF_8)) {
@Override public void close() { closed.set(true); }
};
reader.read(input, () -> false, true, ignored -> {});
assertFalse(closed.get());
assertEquals(504, assertThrows(ApiException.class,
() -> reader.read(bytes("data: {}\n\n"), () -> true, true, ignored -> {})).getStatus());
}
private ByteArrayInputStream bytes(String text) {
return new ByteArrayInputStream(text.getBytes(StandardCharsets.UTF_8));
}
}
@@ -11,6 +11,7 @@ import static org.mockito.Mockito.when;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.security.SecretCipher;
import com.moyi.controlplane.provider.transport.ProviderRequestException;
import com.sun.net.httpserver.HttpServer;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
@@ -207,7 +208,7 @@ class AsyncProviderClientTest {
}
@ParameterizedTest
@CsvSource({"401,UPSTREAM_AUTH_FAILED", "403,UPSTREAM_AUTH_FAILED", "402,UPSTREAM_CREDIT_INSUFFICIENT", "429,UPSTREAM_RATE_LIMITED", "503,UPSTREAM_PROVIDER_ERROR"})
@CsvSource({"401,UPSTREAM_AUTH_FAILED", "403,UPSTREAM_AUTH_FAILED", "402,UPSTREAM_CREDIT_INSUFFICIENT", "429,UPSTREAM_RATE_LIMITED", "408,UPSTREAM_PROVIDER_ERROR", "409,UPSTREAM_PROVIDER_ERROR", "503,UPSTREAM_PROVIDER_ERROR"})
void httpErrorsExposeOnlySafeCodes(int status, String code) throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", exchange -> {
@@ -221,6 +222,7 @@ class AsyncProviderClientTest {
ApiException submit = assertThrows(ApiException.class, () -> client.submit("provider-vast", "v3.1-20260211", "3d", "chair"));
assertEquals(code, submit.getCode());
assertEquals("上游 3D 任务创建失败(HTTP " + status + ")", submit.getMessage());
assertEquals(status != 408 && status != 409 && status < 500, ((ProviderRequestException) submit).definitelyNotAccepted());
ApiException poll = assertThrows(ApiException.class, () -> client.poll("provider-vast", "task-test"));
assertEquals(code, poll.getCode());
assertEquals("上游异步任务查询失败(HTTP " + status + ")", poll.getMessage());
@@ -229,6 +231,115 @@ class AsyncProviderClientTest {
}
}
@Test void successfulHttpResponseWithMalformedBodyHasUnknownAcceptance() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", exchange -> {
byte[] body = "not-json; private upstream detail".getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(200, body.length);
try (var output = exchange.getResponseBody()) { output.write(body); }
});
server.start();
try {
var client = clientFor("http://127.0.0.1:" + server.getAddress().getPort());
var failure = assertThrows(ProviderRequestException.class, () -> client.submit("provider-vast", "tripo", "3d", "chair"));
assertEquals(ProviderRequestException.Acceptance.UNKNOWN, failure.acceptance());
org.junit.jupiter.api.Assertions.assertFalse(failure.getMessage().contains("private upstream detail"));
} finally { server.stop(0); }
}
@ParameterizedTest
@CsvSource({"image,gpt-image-2", "video,sora-2", "audio,tts-1"})
void compatibleSubmit5xxRemainsUnknownAndNeverUsesFallbackRoute(String type, String model) throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
var calls = new java.util.concurrent.atomic.AtomicInteger();
server.createContext("/", exchange -> {
calls.incrementAndGet();
exchange.getRequestBody().readAllBytes();
byte[] body = "private account details".getBytes(StandardCharsets.UTF_8);
exchange.sendResponseHeaders(503, body.length);
try (var output = exchange.getResponseBody()) { output.write(body); }
});
server.start();
try {
var jdbc = mock(JdbcTemplate.class);
var cipher = new SecretCipher("test-encryption");
when(jdbc.queryForMap(any(String.class), eq("relay"))).thenReturn(Map.of(
"code", "relay", "protocol", "openai-compatible", "base_url", "http://127.0.0.1:" + server.getAddress().getPort(), "credential_ciphertext", cipher.encrypt("test-key")));
var client = new AsyncProviderClient(jdbc, cipher, new ObjectMapper(), mock(Environment.class));
var failure = assertThrows(ProviderRequestException.class, () -> client.submit("relay", model, type, "chair"));
assertEquals(ProviderRequestException.Acceptance.UNKNOWN, failure.acceptance());
assertEquals(1, calls.get());
} finally { server.stop(0); }
}
@Test void acceptedPostWithDroppedResponseIsUnknownAndIsNotTransparentlyRepeated() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
var calls = new java.util.concurrent.atomic.AtomicInteger();
server.createContext("/", exchange -> {
exchange.getRequestBody().readAllBytes();
calls.incrementAndGet();
exchange.close(); // Simulate acceptance followed by loss of the HTTP response.
});
server.start();
try {
var client = clientFor("http://127.0.0.1:" + server.getAddress().getPort());
var failure = assertThrows(ProviderRequestException.class, () -> client.submit("provider-vast", "tripo", "3d", "chair"));
assertEquals(ProviderRequestException.Acceptance.UNKNOWN, failure.acceptance());
assertEquals(1, calls.get());
} finally { server.stop(0); }
}
@Test void pollingBindingValidationChecksTheExistingCredentialWithoutAnHttpCall() throws Exception {
var client = clientFor("http://127.0.0.1:1");
String fingerprint = java.util.HexFormat.of().formatHex(java.security.MessageDigest.getInstance("SHA-256").digest("test-key".getBytes(StandardCharsets.UTF_8)));
org.junit.jupiter.api.Assertions.assertDoesNotThrow(() -> client.requirePollableBinding("provider-vast", "route-a", "tripo", "3d", fingerprint));
assertEquals("UPSTREAM_KEY_NOT_AVAILABLE", assertThrows(ApiException.class, () -> client.requirePollableBinding("provider-vast", "route-a", "tripo", "3d", "0".repeat(64))).getCode());
assertEquals("TASK_NOT_POLLABLE", assertThrows(ApiException.class, () -> client.requirePollableBinding("provider-vast", "route-a", "tripo", "image", fingerprint)).getCode());
assertEquals("INVALID_CREDENTIAL_FINGERPRINT", assertThrows(ApiException.class, () -> client.requirePollableBinding("provider-vast", "route-a", "tripo", "3d", "test-key")).getCode());
}
@ParameterizedTest
@CsvSource({"200,UNKNOWN", "401,REJECTED"})
void responseLimitKeepsTheObservedAcceptanceBoundary(int status, ProviderRequestException.Acceptance expected) throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
var calls = new java.util.concurrent.atomic.AtomicInteger();
server.createContext("/", exchange -> {
calls.incrementAndGet(); exchange.getRequestBody().readAllBytes();
exchange.sendResponseHeaders(status, 9 * 1024 * 1024); exchange.close();
}); server.start();
try {
var client = clientFor("http://127.0.0.1:" + server.getAddress().getPort());
var failure = assertThrows(ProviderRequestException.class, () -> client.submit("provider-vast", "tripo", "3d", "test"));
assertEquals(expected, failure.acceptance());
assertEquals(1, calls.get());
} finally { server.stop(0); }
}
@Test void invalidMediaIsKnownToFailBeforeSendingGenerationRequest() {
var client = clientFor("http://127.0.0.1:1");
var failure = assertThrows(ProviderRequestException.class, () -> client.submit("provider-vast", "tripo", "3d", Map.of("media_url", "http://127.0.0.1/private")));
assertEquals(ProviderRequestException.Acceptance.NOT_SENT, failure.acceptance());
assertEquals("INVALID_MEDIA_URL", failure.getCode());
}
@ParameterizedTest
@CsvSource({"image,gpt-image-2", "video,sora-2", "image,gemini-3-pro-image-preview"})
void localReferenceImageIsRejectedBeforeAnyGenerationPost(String type, String model) throws Exception {
var calls = new java.util.concurrent.atomic.AtomicInteger();
HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/", exchange -> { calls.incrementAndGet(); exchange.sendResponseHeaders(500, -1); exchange.close(); }); server.start();
try {
var jdbc = mock(JdbcTemplate.class); var cipher = new SecretCipher("test-encryption");
when(jdbc.queryForMap(any(String.class), eq("relay"))).thenReturn(Map.of("code", "relay", "protocol", "openai-compatible",
"base_url", "http://127.0.0.1:" + server.getAddress().getPort(), "credential_ciphertext", cipher.encrypt("test-key")));
var client = new AsyncProviderClient(jdbc, cipher, new ObjectMapper(), mock(Environment.class));
var failure = assertThrows(ProviderRequestException.class, () -> client.submit("relay", model, type,
Map.of("prompt", "synthetic", "media_url", "https://127.0.0.1/private")));
assertEquals(ProviderRequestException.Acceptance.NOT_SENT, failure.acceptance()); assertEquals("INVALID_MEDIA_URL", failure.getCode());
assertEquals(0, calls.get());
} finally { server.stop(0); }
}
private AsyncProviderClient clientFor(String baseUrl) {
JdbcTemplate jdbc = mock(JdbcTemplate.class);
SecretCipher cipher = new SecretCipher("test-encryption");
@@ -10,6 +10,12 @@ import java.time.Duration;
import java.time.OffsetDateTime;
import java.util.Map;
import java.util.UUID;
import java.util.ArrayList;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import com.moyi.controlplane.gateway.request.ApiRequestMetadata;
import com.moyi.controlplane.provider.transport.ProviderRequestException;
import org.flywaydb.core.Flyway;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
@@ -36,6 +42,8 @@ class AsyncTaskPostgresTest {
private UsageAuditService usage;
private AsyncProviderClient provider;
private AsyncTaskService tasks;
private CatalogService catalog;
private GatewayGuard guard;
@BeforeEach void setup() {
String url = System.getenv("MOYI_AUTH_TEST_JDBC_URL");
@@ -46,9 +54,10 @@ class AsyncTaskPostgresTest {
jdbc = new JdbcTemplate(source);
usage = spy(new UsageAuditService(jdbc, json));
provider = mock(AsyncProviderClient.class);
GatewayGuard guard=mock(GatewayGuard.class);
guard=mock(GatewayGuard.class);
when(guard.tryTaskUserSlot(any(),anyString(),anyLong())).thenReturn(true);
tasks = new AsyncTaskService(jdbc, json, mock(CatalogService.class), 100, 4, 4, 120_000,
catalog = mock(CatalogService.class);
tasks = new AsyncTaskService(jdbc, json, catalog, 100, 4, 4, 120_000,
guard, usage, provider, new DataSourceTransactionManager(source));
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name) VALUES (?,'async-test','username','not-a-password','Test')", userId);
jdbc.update("INSERT INTO platform_api_keys(id,user_id,name,key_prefix,key_hash) VALUES (?,?,'Test','test-prefix',?)", keyId, userId, "b".repeat(64));
@@ -56,6 +65,11 @@ class AsyncTaskPostgresTest {
jdbc.update("INSERT INTO model_tasks(id,user_id,api_key_id,model_id,task_type,status,prompt,reserved_amount,provider_task_id) VALUES (?,?,?,'model-mock-3d','3d','running','chair',1.44,'upstream-task')", taskId,userId,keyId);
usage.recordTask(userId,keyId,"model-mock-3d","provider-vast",taskId,"running",Map.of("prompt","chair"),null,null,null);
Flyway.configure().dataSource(base).schemas(schema).defaultSchema(schema).load().migrate();
// This fixture knows its original binding; a migration must not guess it for real legacy jobs.
jdbc.update("UPDATE model_tasks SET provider_id_snapshot='provider-vast',upstream_model_snapshot='v3.1-20260211' WHERE id=?", taskId);
when(catalog.keyOwnerId(keyId)).thenReturn(userId);
when(catalog.keyCanInvoke(anyMap(), eq("model-mock-3d"))).thenReturn(true);
when(catalog.activeModel("model-mock-3d")).thenReturn(Map.of("id", "model-mock-3d", "providerId", "provider-vast", "upstreamModel", "v3.1-20260211", "type", "3d", "mode", "async", "pricing", Map.of("unit", "CNY / task", "price", 1.44)));
}
@AfterEach void cleanup() {
@@ -81,11 +95,109 @@ class AsyncTaskPostgresTest {
return jdbc.queryForObject("SELECT cost_amount FROM model_tasks WHERE id=?",BigDecimal.class,taskId);
}
private Map<String,Object> create(String idempotencyKey, Map<String,Object> input) {
return tasks.createForKey(Map.of("id", keyId.toString()), "3d", input, idempotencyKey, ApiRequestMetadata.internal());
}
@Test void queuedTaskRetainsAdmittedExecutionBindingWhenRouteIsEdited() {
String created = String.valueOf(create("frozen-queue", Map.of("model", "model-mock-3d", "prompt", "chair")).get("id"));
jdbc.update("UPDATE model_catalog SET provider_id='provider-bytedance',upstream_model='different-upstream' WHERE id='model-mock-3d'");
jdbc.update("UPDATE model_tasks SET updated_at=now() WHERE id=?", taskId);
when(provider.submit(eq("provider-vast"), eq("v3.1-20260211"), eq("model-mock-3d"), eq("3d"), anyMap())).thenReturn(Map.of("id", "original-route-task"));
tasks.completeMockTasks();
assertEquals("original-route-task", jdbc.queryForObject("SELECT provider_task_id FROM model_tasks WHERE id=?", String.class, created));
verify(provider).submit(eq("provider-vast"), eq("v3.1-20260211"), eq("model-mock-3d"), eq("3d"), anyMap());
}
@Test void acceptedTaskPollsFrozenProviderRatherThanEditedCatalog() {
jdbc.update("UPDATE model_catalog SET provider_id='provider-bytedance',upstream_model='different-upstream' WHERE id='model-mock-3d'");
jdbc.update("UPDATE model_tasks SET updated_at=now()-interval '6 seconds' WHERE id=?", taskId);
when(provider.poll("provider-vast", "model-mock-3d", "v3.1-20260211", "upstream-task", null)).thenReturn(Map.of("status", "running", "progress", 55));
tasks.completeMockTasks();
assertEquals("running", status());
assertEquals(55, jdbc.queryForObject("SELECT progress FROM model_tasks WHERE id=?", Integer.class, taskId));
verify(provider).poll("provider-vast", "model-mock-3d", "v3.1-20260211", "upstream-task", null);
}
@Test void acceptedLegacyTaskWithoutProofOfBindingIsNotPolledOnAGuessedRoute() {
jdbc.update("UPDATE model_tasks SET provider_id_snapshot=NULL,upstream_model_snapshot=NULL,updated_at=now()-interval '6 seconds' WHERE id=?", taskId);
tasks.completeMockTasks();
assertEquals("reconciling", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
verifyNoInteractions(provider);
}
@Test void taskReadsAreOwnerScopedAndCannotExposeAnotherUsersResult() {
UUID anotherUser = UUID.randomUUID(), anotherKey = UUID.randomUUID();
when(catalog.keyOwnerId(anotherKey)).thenReturn(anotherUser);
var foreignKey = Map.<String,Object>of("id", anotherKey.toString());
assertEquals("TASK_NOT_FOUND", assertThrows(ApiException.class, () -> tasks.getForKey(foreignKey, taskId)).getCode());
when(catalog.authenticateKey("foreign-token")).thenReturn(foreignKey);
assertTrue(tasks.list("Bearer foreign-token", 100).isEmpty());
assertEquals(taskId, tasks.getForKey(Map.of("id", keyId.toString()), taskId).get("id"));
}
@Test void idempotentReplayReturnsOriginalEvenAfterCompletionAndRouteRetirement() {
var input = Map.<String,Object>of("model", "model-mock-3d", "prompt", "one chair");
var first = create("client-operation-1", input);
var replay = create("client-operation-1", new java.util.LinkedHashMap<>(input));
assertEquals(first.get("id"), replay.get("id"));
String created = String.valueOf(first.get("id"));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM usage_events WHERE provider_task_id=?", Integer.class, created));
BigDecimal held = wallet("reserved_balance");
ReflectionTestUtils.invokeMethod(tasks, "finishTask", created, "succeeded", 100, Map.of("url", "https://example.test/result.glb"), null, null);
when(catalog.activeModel("model-mock-3d")).thenReturn(null);
var finishedReplay = create("client-operation-1", input);
assertEquals(created, finishedReplay.get("id"));
assertEquals("succeeded", finishedReplay.get("status"));
assertTrue(wallet("reserved_balance").compareTo(held) < 0);
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM wallet_entries WHERE request_id IS NOT NULL AND user_id=?", Integer.class, userId));
String stored = jdbc.queryForObject("SELECT idempotency_hash FROM model_tasks WHERE id=?", String.class, created);
assertEquals(64, stored.length());
assertNotEquals("client-operation-1", stored);
}
@Test void sameKeyDifferentRequestConflictsWithoutNewReservation() {
create("client-conflict", Map.of("model", "model-mock-3d", "prompt", "chair"));
BigDecimal held = wallet("reserved_balance");
ApiException conflict = assertThrows(ApiException.class,
() -> create("client-conflict", Map.of("model", "model-mock-3d", "prompt", "table")));
assertEquals("IDEMPOTENCY_CONFLICT", conflict.getCode());
assertEquals(409, conflict.getStatus());
assertEquals(held, wallet("reserved_balance"));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM model_tasks WHERE idempotency_hash IS NOT NULL", Integer.class));
}
@Test void concurrentSameKeyCreatesOneTaskOneAuditAndOneHold() throws Exception {
var input = Map.<String,Object>of("model", "model-mock-3d", "prompt", "concurrent chair");
BigDecimal held = wallet("reserved_balance");
var start = new CountDownLatch(1);
try (var executor = Executors.newFixedThreadPool(8)) {
var results = new ArrayList<java.util.concurrent.Future<Map<String,Object>>>();
for (int i = 0; i < 8; i++) results.add(executor.submit(() -> {
assertTrue(start.await(10, TimeUnit.SECONDS));
return create("concurrent-operation", input);
}));
start.countDown();
var ids = new java.util.HashSet<Object>();
for (var future : results) ids.add(future.get(30, TimeUnit.SECONDS).get("id"));
assertEquals(1, ids.size());
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM usage_events WHERE provider_task_id=?", Integer.class, ids.iterator().next()));
}
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM model_tasks WHERE idempotency_hash IS NOT NULL", Integer.class));
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance").subtract(held));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM gateway_budget_holds", Integer.class));
verifyNoInteractions(provider);
}
@ParameterizedTest
@ValueSource(strings={"PROVIDER_NOT_CONFIGURED", "UPSTREAM_AUTH_FAILED"})
void submissionFailureReleasesExactlyOnceWithoutDebitingBalance(String code) {
jdbc.update("UPDATE model_tasks SET status='queued',provider_task_id=NULL WHERE id=?",taskId);
when(provider.submit(anyString(),anyString(),anyString(),anyString(),anyString())).thenThrow(new ApiException(code,"上游任务未接受",503));
when(provider.submit(anyString(),anyString(),anyString(),anyString(),anyString())).thenThrow(
"PROVIDER_NOT_CONFIGURED".equals(code)
? ProviderRequestException.notSent(code, "上游任务未接受", 503)
: ProviderRequestException.http(401, "上游任务未接受"));
tasks.completeMockTasks();
tasks.completeMockTasks();
assertEquals("failed",status());
@@ -149,19 +261,167 @@ class AsyncTaskPostgresTest {
assertNull(jdbc.queryForObject("SELECT error_code FROM usage_events WHERE provider_task_id=?",String.class,taskId));
}
@Test void deadlineReleasesHoldWithoutClaimingTheUpstreamFailed() {
@Test void deadlineRetainsHoldWithoutClaimingTheUpstreamFailed() {
jdbc.update("UPDATE model_tasks SET provider_deadline_at=now()-interval '1 second' WHERE id=?",taskId);
poll(); poll();
assertEquals("failed",status());
assertEquals("reconciling",status());
assertEquals(new BigDecimal("10.00000000"),wallet("balance"));
assertEquals(0,wallet("reserved_balance").signum());
assertEquals(0,cost().signum());
assertEquals(new BigDecimal("1.44000000"),wallet("reserved_balance"));
assertNull(cost());
String error = jdbc.queryForObject("SELECT error::text FROM model_tasks WHERE id=?",String.class,taskId);
assertTrue(error.contains("UPSTREAM_TASK_TIMEOUT"));
assertTrue(error.contains("执行结果未知"));
verifyNoInteractions(provider);
}
@ParameterizedTest
@ValueSource(ints={408,409,500,502,503,504})
void uncertainSubmitResponseNeverRefundsOrResubmits(int httpStatus) {
jdbc.update("UPDATE model_tasks SET status='queued',provider_task_id=NULL WHERE id=?", taskId);
when(provider.submit(anyString(), anyString(), anyString(), anyString(), anyString()))
.thenThrow(ProviderRequestException.http(httpStatus, "safe local message"));
tasks.completeMockTasks();
tasks.completeMockTasks();
assertEquals("reconciling", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertEquals(new BigDecimal("10.00000000"), wallet("balance"));
assertNull(cost());
assertEquals("reconciling", jdbc.queryForObject("SELECT status FROM usage_events WHERE provider_task_id=?", String.class, taskId));
assertEquals("pending", jdbc.queryForObject("SELECT cost_status FROM usage_events WHERE provider_task_id=?", String.class, taskId));
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM wallet_entries WHERE user_id=?", Integer.class, userId));
verify(provider, times(1)).submit(anyString(), anyString(), anyString(), anyString(), anyString());
}
@Test void lostSubmitResponseAndInterruptedClaimRequireReconciliationWithoutReplay() {
jdbc.update("UPDATE model_tasks SET provider_task_id=NULL,updated_at=now()-interval '3 minutes' WHERE id=?", taskId);
tasks.completeMockTasks();
tasks.completeMockTasks();
assertEquals("reconciling", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertNull(cost());
verifyNoInteractions(provider);
}
@Test void missingTaskReceiptRetainsFinancialAndBudgetHolds() {
var created = create("missing-receipt", Map.of("model", "model-mock-3d", "prompt", "chair"));
String id = String.valueOf(created.get("id"));
when(provider.submit(anyString(), anyString(), anyString(), anyString(), anyMap())).thenReturn(Map.of("accepted", true));
tasks.completeMockTasks();
tasks.completeMockTasks();
assertEquals("reconciling", jdbc.queryForObject("SELECT status FROM model_tasks WHERE id=?", String.class, id));
assertEquals(new BigDecimal("2.88000000"), wallet("reserved_balance"));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM gateway_budget_holds WHERE request_id=?", Integer.class, id));
verify(provider, times(1)).submit(anyString(), anyString(), anyString(), anyString(), anyMap());
}
@Test void genericSubmitFailureIsNotTreatedAsEvidenceOfRejection() {
jdbc.update("UPDATE model_tasks SET status='queued',provider_task_id=NULL WHERE id=?", taskId);
when(provider.submit(anyString(), anyString(), anyString(), anyString(), anyString()))
.thenThrow(new IllegalStateException("network connection lost; private details"));
tasks.completeMockTasks();
assertEquals("reconciling", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertFalse(jdbc.queryForObject("SELECT error::text FROM model_tasks WHERE id=?", String.class, taskId).contains("private details"));
}
@Test void missingPollingCredentialDoesNotRefundAnAlreadyAcceptedJob() {
when(provider.poll("provider-vast", "model-mock-3d", "v3.1-20260211", "upstream-task", null))
.thenThrow(new ApiException("UPSTREAM_KEY_NOT_AVAILABLE", "sensitive detail", 503));
poll();
assertEquals("reconciling", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertNull(cost());
}
@Test void inlineSuccessSurvivesSettlementFailureAndRecoversWithoutRegenerating() {
jdbc.update("UPDATE model_tasks SET status='queued',provider_task_id=NULL WHERE id=?", taskId);
when(provider.submit(anyString(), anyString(), anyString(), anyString(), anyString()))
.thenReturn(Map.of("_moyiInlineResult", Map.of("url", "https://example.test/generated.glb")));
doThrow(new IllegalStateException("simulated audit failure")).when(usage).updateTask(eq(taskId), eq("succeeded"), anyMap(), isNull(), isNull());
assertThrows(IllegalStateException.class, () -> tasks.completeMockTasks());
assertEquals("running", status());
assertEquals("succeeded", jdbc.queryForObject("SELECT completion_status FROM model_tasks WHERE id=?", String.class, taskId));
fail(); // A stale error cannot refund an already observed success before the retry settles it.
assertEquals("running", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertEquals(new BigDecimal("10.00000000"), wallet("balance"));
assertNull(cost());
jdbc.update("UPDATE model_tasks SET updated_at=now()-interval '3 minutes' WHERE id=?", taskId);
doCallRealMethod().when(usage).updateTask(eq(taskId), eq("succeeded"), anyMap(), isNull(), isNull());
tasks.completeMockTasks();
tasks.completeMockTasks();
assertEquals("succeeded", status());
assertEquals(new BigDecimal("8.56000000"), wallet("balance"));
assertEquals(0, wallet("reserved_balance").signum());
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM wallet_entries WHERE user_id=?", Integer.class, userId));
verify(provider, times(1)).submit(anyString(), anyString(), anyString(), anyString(), anyString());
}
@ParameterizedTest
@ValueSource(strings={"unsuccessful","not_failed","cancelling","unknown"})
void ambiguousProviderStatusCannotTriggerSettlementOrRefund(String state) {
when(provider.poll("provider-vast", "model-mock-3d", "v3.1-20260211", "upstream-task", null))
.thenReturn(Map.of("status", state));
poll();
assertEquals("running", status());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertNull(cost());
}
@Test void interruptedClaimCheckCannotQuarantineALaterAcceptedReceipt() {
var journal = new TaskOutcomeJournal(jdbc, json, new org.springframework.transaction.support.TransactionTemplate(new DataSourceTransactionManager(jdbc.getDataSource())), usage);
jdbc.update("UPDATE model_tasks SET updated_at=now()-interval '3 minutes' WHERE id=?", taskId);
journal.requireReconciliation(taskId, "STALE_WORKER", "stale claim check", true);
assertEquals("running", status(), "An upstream task ID proves this is no longer an interrupted claim");
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
}
@Test void reconciliationAndUsageAuditRollBackTogetherWithoutTouchingTheHold() {
var journal = new TaskOutcomeJournal(jdbc, json, new org.springframework.transaction.support.TransactionTemplate(new DataSourceTransactionManager(jdbc.getDataSource())), usage);
doThrow(new IllegalStateException("audit offline")).when(usage).updateTask(eq(taskId), eq("reconciling"), isNull(), anyString(), anyString());
assertThrows(IllegalStateException.class, () -> journal.requireReconciliation(taskId, "UNKNOWN", "result unknown", false));
assertEquals("running", status());
assertEquals("running", jdbc.queryForObject("SELECT status FROM usage_events WHERE provider_task_id=?", String.class, taskId));
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertNull(cost());
}
@Test void failedTaskReplayDoesNotRequeueOrReserveAgain() {
var input = Map.<String,Object>of("model", "model-mock-3d", "prompt", "will be rejected");
var first = create("rejected-task", input);
String id = String.valueOf(first.get("id"));
when(provider.submit(anyString(), anyString(), anyString(), anyString(), anyMap()))
.thenThrow(ProviderRequestException.http(429, "provider rejected"));
tasks.completeMockTasks();
BigDecimal held = wallet("reserved_balance");
var replay = create("rejected-task", input);
assertEquals(id, replay.get("id"));
assertEquals("failed", replay.get("status"));
assertEquals(held, wallet("reserved_balance"));
assertEquals(0, jdbc.queryForObject("SELECT count(*) FROM gateway_budget_holds WHERE request_id=?", Integer.class, id));
verify(provider, times(1)).submit(anyString(), anyString(), anyString(), anyString(), anyMap());
}
@Test void pendingReconciliationConsumesAsyncConcurrencyRatherThanOpeningAnotherSlot() {
tasks = new AsyncTaskService(jdbc, json, catalog, 100, 4, 4, 120_000, false,
guard, usage, provider, new DataSourceTransactionManager(jdbc.getDataSource()),
new com.moyi.controlplane.gateway.request.ApiRequestRiskPolicy(com.moyi.controlplane.gateway.request.ApiRequestRiskEngine.allowAll()));
jdbc.update("UPDATE model_tasks SET status='reconciling' WHERE id=?", taskId);
jdbc.update("UPDATE model_catalog SET max_concurrency=1 WHERE id='model-mock-3d'");
String id = String.valueOf(create("next-while-unknown", Map.of("model", "model-mock-3d", "prompt", "another chair")).get("id"));
tasks.completeMockTasks();
assertEquals("queued", jdbc.queryForObject("SELECT status FROM model_tasks WHERE id=?", String.class, id));
verifyNoInteractions(provider);
}
@Test void documentedSucceededStatusSettlesInsteadOfPollingForever() {
when(provider.poll("provider-vast", "model-mock-3d", "v3.1-20260211", "upstream-task", null))
.thenReturn(Map.of("status", "succeeded", "url", "https://example.test/generated.glb"));
poll();
assertEquals("succeeded", status());
assertEquals(new BigDecimal("8.56000000"), wallet("balance"));
}
@Test void confirmedUpstreamFailureReleasesHoldImmediately() {
when(provider.poll("provider-vast","model-mock-3d","v3.1-20260211","upstream-task",null)).thenReturn(Map.of("data",Map.of("status","failed")));
poll();
@@ -0,0 +1,36 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class TaskCreationIdentityTest {
private final ObjectMapper mapper = new ObjectMapper();
@Test void canonicalizesObjectOrderAndNumbersButPreservesArrayOrderAndContent() {
var first = new LinkedHashMap<String,Object>();
first.put("model", "route");
first.put("settings", Map.of("quality", 1, "images", List.of("a", "b")));
var reordered = new LinkedHashMap<String,Object>();
reordered.put("settings", Map.of("images", List.of("a", "b"), "quality", 1.0));
reordered.put("model", "route");
var original = TaskCreationIdentity.of(mapper, "same-key", "image", first);
assertEquals(original, TaskCreationIdentity.of(mapper, "same-key", "image", reordered));
assertNotEquals(original.requestHash(), TaskCreationIdentity.of(mapper, "same-key", "video", reordered).requestHash());
assertNotEquals(original.requestHash(), TaskCreationIdentity.of(mapper, "same-key", "image", Map.of("model", "route", "settings", Map.of("quality", 1, "images", List.of("b", "a")))).requestHash());
assertEquals(64, original.keyHash().length());
assertFalse(original.toString().contains("same-key"));
}
@Test void rejectsBlankControlNonAsciiAndOversizeKeys() {
for (String value : List.of("", " ", "has space", "line\nbreak", "中文", "a".repeat(129))) {
assertEquals("INVALID_IDEMPOTENCY_KEY", assertThrows(ApiException.class,
() -> TaskCreationIdentity.of(mapper, value, "image", Map.of())).getCode());
}
assertNull(TaskCreationIdentity.of(mapper, null, "image", Map.of()).keyHash());
}
}
@@ -0,0 +1,295 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.moyi.controlplane.auth.AuthService;
import com.moyi.controlplane.auth.PasswordHasher;
import com.moyi.controlplane.common.ApiException;
import com.moyi.controlplane.common.ApiExceptionHandler;
import com.moyi.controlplane.gateway.GatewayGuard;
import com.moyi.controlplane.usage.UsageAuditService;
import jakarta.servlet.http.Cookie;
import java.math.BigDecimal;
import java.time.OffsetDateTime;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import org.flywaydb.core.Flyway;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.springframework.dao.DataAccessException;
import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.test.web.servlet.setup.MockMvcBuilders;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.ArgumentMatchers.*;
import static org.mockito.Mockito.*;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
@EnabledIfEnvironmentVariable(named="MOYI_AUTH_TEST_JDBC_URL", matches="jdbc:postgresql://127\\.0\\.0\\.1:[0-9]+/.+")
class TaskReconciliationPostgresTest {
private final ObjectMapper json = new ObjectMapper().findAndRegisterModules().disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS);
private final UUID userId = UUID.randomUUID(), keyId = UUID.randomUUID(), adminId = UUID.randomUUID(), secondAdminId = UUID.randomUUID();
private final String taskId = "reconcile-" + UUID.randomUUID();
private final String fingerprint = "c".repeat(64);
private final AuthService.RequestMeta meta = new AuthService.RequestMeta("127.0.0.1", "isolated-reconciliation-test");
private DriverManagerDataSource base;
private JdbcTemplate jdbc;
private String schema;
private AuthService auth;
private GatewayGuard guard;
private AsyncProviderClient provider;
private TaskReconciliationService service;
private MockMvc http;
@BeforeEach void setup() {
String url = System.getenv("MOYI_AUTH_TEST_JDBC_URL");
schema = "reconciliation_test_" + UUID.randomUUID().toString().replace("-", "");
base = new DriverManagerDataSource(url, "moyi_role_test", "");
Flyway.configure().dataSource(base).schemas(schema).defaultSchema(schema).load().migrate();
var source = new DriverManagerDataSource(url + "?currentSchema=" + schema, "moyi_role_test", "");
jdbc = new JdbcTemplate(source);
var manager = new DataSourceTransactionManager(source);
for (var entry : Map.of(userId, "user", adminId, "admin", secondAdminId, "admin").entrySet()) {
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name,role) VALUES (?,?,'username','not-loginable','Test',?)", entry.getKey(), entry.getKey().toString(), entry.getValue());
}
session("admin-session", adminId); session("second-admin-session", secondAdminId); session("user-session", userId);
jdbc.update("INSERT INTO platform_api_keys(id,user_id,name,key_prefix,key_hash) VALUES (?,?,'Test','test-key',?)", keyId, userId, "b".repeat(64));
jdbc.update("INSERT INTO user_wallets(user_id,balance,reserved_balance) VALUES (?,10,1.44)", userId);
jdbc.update("INSERT INTO model_tasks(id,user_id,api_key_id,model_id,task_type,status,prompt,reserved_amount,provider_task_id,provider_key_fingerprint,provider_id_snapshot,upstream_model_snapshot,pricing_snapshot,reconciliation_required_at) VALUES (?,?,?,'model-mock-3d','3d','reconciling','private prompt',1.44,'original-provider-task',?,'provider-vast','v3.1-20260211','{\"unit\":\"CNY / task\",\"price\":1.44}'::jsonb,now())", taskId, userId, keyId, fingerprint);
jdbc.update("INSERT INTO gateway_budget_holds(request_id,user_id,key_id,amount) VALUES (?,?,?,1.44)", taskId, userId, keyId);
var usage = new UsageAuditService(jdbc, json);
usage.recordTask(userId, keyId, "model-mock-3d", "provider-vast", taskId, "reconciling", Map.of("prompt", "private prompt"), null, "UNKNOWN", "pending");
auth = spy(new AuthService(jdbc, mock(PasswordHasher.class), json, 168));
guard = mock(GatewayGuard.class);
when(guard.tryTaskUserSlot(any(), anyString(), anyLong())).thenReturn(true);
provider = mock(AsyncProviderClient.class);
var settlements = new TaskSettlementService(jdbc, json, guard, usage, manager);
service = new TaskReconciliationService(jdbc, json, auth, settlements, provider, guard, usage, manager, 120_000);
http = MockMvcBuilders.standaloneSetup(new TaskReconciliationController(service))
.setControllerAdvice(new ApiExceptionHandler()).setMessageConverters(new MappingJackson2HttpMessageConverter(json)).build();
}
@AfterEach void cleanup() { if (base != null && schema != null) new JdbcTemplate(base).execute("DROP SCHEMA " + schema + " CASCADE"); }
private void session(String token, UUID user) {
jdbc.update("INSERT INTO auth_sessions(id,token_hash,user_id,expires_at) VALUES (?,?,?,now()+interval '1 hour')", UUID.randomUUID(), AuthService.hashSecret(token), user);
}
private OffsetDateTime version() { return jdbc.queryForObject("SELECT updated_at FROM model_tasks WHERE id=?", OffsetDateTime.class, taskId); }
private String state() { return jdbc.queryForObject("SELECT status FROM model_tasks WHERE id=?", String.class, taskId); }
private BigDecimal wallet(String column) { return jdbc.queryForObject("SELECT " + column + " FROM user_wallets WHERE user_id=?", BigDecimal.class, userId); }
private int count(String table) { return jdbc.queryForObject("SELECT count(*) FROM " + table, Integer.class); }
private TaskReconciliationCommand command(String action) {
return new TaskReconciliationCommand(UUID.randomUUID(), action, version(), "provider-bill/test-001", "Verified against the provider task history and billing record",
"resume_polling".equals(action) ? "original-provider-task" : null,
"resume_polling".equals(action) ? fingerprint : null,
"confirm_succeeded".equals(action) ? new BigDecimal("1.44") : null,
"confirm_succeeded".equals(action) ? Map.of("url", "https://example.test/generated.glb") : null);
}
private Map<String,Object> reconcile(TaskReconciliationCommand command) { return service.reconcile("admin-session", taskId, command, meta); }
private void unchanged() {
assertEquals("reconciling", state()); assertEquals(new BigDecimal("10.00000000"), wallet("balance"));
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertEquals(0, count("wallet_entries")); assertEquals(0, count("task_reconciliations"));
assertEquals(1, count("gateway_budget_holds"));
}
@Test void realSessionsEnforceAdministratorAndCsrfBarrier() throws Exception {
var command = command("confirm_failed");
http.perform(get("/api/admin/task-reconciliation")).andExpect(status().isUnauthorized());
http.perform(get("/api/admin/task-reconciliation").cookie(new Cookie("moyi_session", "user-session"))).andExpect(status().isForbidden());
http.perform(post("/api/admin/task-reconciliation/" + taskId).cookie(new Cookie("moyi_session", "user-session")).header("X-Moyi-Source", "console").contentType("application/json").content(json.writeValueAsBytes(command))).andExpect(status().isForbidden());
http.perform(post("/api/admin/task-reconciliation/" + taskId).cookie(new Cookie("moyi_session", "admin-session")).contentType("application/json").content(json.writeValueAsBytes(command))).andExpect(status().isForbidden());
unchanged();
verifyNoInteractions(provider, guard);
}
@Test void failureConfirmationReleasesOnceAndReplayReturnsTheOperationReceipt() throws Exception {
var command = command("confirm_failed");
http.perform(post("/api/admin/task-reconciliation/" + taskId).cookie(new Cookie("moyi_session", "admin-session")).header("X-Moyi-Source", "console").contentType("application/json").content(json.writeValueAsBytes(command)))
.andExpect(status().isOk()).andExpect(jsonPath("$.task.status").value("failed")).andExpect(jsonPath("$.replayed").value(false));
assertEquals(true, reconcile(command).get("replayed"));
assertEquals(new BigDecimal("10.00000000"), wallet("balance"));
assertEquals(0, wallet("reserved_balance").signum());
assertEquals(0, count("wallet_entries")); assertEquals(0, count("gateway_budget_holds"));
assertEquals(1, count("task_reconciliations"));
assertEquals(1, jdbc.queryForObject("SELECT count(*) FROM auth_audit_log WHERE event_type='task.reconciled'", Integer.class));
verifyNoInteractions(provider);
}
@Test void successUsesFrozenPriceAndDebitsExactlyOnce() {
jdbc.update("UPDATE model_catalog SET pricing='{\"unit\":\"CNY / task\",\"price\":99}'::jsonb WHERE id='model-mock-3d'");
var command = command("confirm_succeeded");
reconcile(command); reconcile(command);
assertEquals("succeeded", state());
assertEquals(new BigDecimal("8.56000000"), wallet("balance"));
assertEquals(0, wallet("reserved_balance").signum());
assertEquals(1, count("wallet_entries")); assertEquals(1, count("task_reconciliations"));
assertEquals(new BigDecimal("1.44000000"), jdbc.queryForObject("SELECT cost_amount FROM model_tasks WHERE id=?", BigDecimal.class, taskId));
verifyNoInteractions(provider);
}
@Test void quoteIsReadOnlyAndUsesTheFrozenVersionNotEditableCatalogPrices() throws Exception {
jdbc.update("UPDATE model_catalog SET pricing='{\"unit\":\"CNY / task\",\"price\":999}'::jsonb WHERE id='model-mock-3d'");
http.perform(post("/api/admin/task-reconciliation/" + taskId + "/quote").cookie(new Cookie("moyi_session", "admin-session")).contentType("application/json").content("{}"))
.andExpect(status().isOk()).andExpect(jsonPath("$.expectedCost").value("1.44"));
unchanged();
assertEquals(0, count("auth_audit_log"));
}
@Test void concurrentIdenticalOperationsShareOneReceiptAndOneDebit() throws Exception {
var command = command("confirm_succeeded");
var start = new CountDownLatch(1);
try (var executor = Executors.newFixedThreadPool(8)) {
var results = new ArrayList<java.util.concurrent.Future<Map<String,Object>>>();
for (int i = 0; i < 8; i++) results.add(executor.submit(() -> { assertTrue(start.await(10, TimeUnit.SECONDS)); return reconcile(command); }));
start.countDown();
int originals = 0;
for (var result : results) if (Boolean.FALSE.equals(result.get(30, TimeUnit.SECONDS).get("replayed"))) originals++;
assertEquals(1, originals);
}
assertEquals(1, count("wallet_entries")); assertEquals(1, count("task_reconciliations"));
assertEquals(new BigDecimal("8.56000000"), wallet("balance"));
}
@Test void concurrentDifferentDecisionsHaveExactlyOneWinner() throws Exception {
var success = command("confirm_succeeded"); var failure = command("confirm_failed");
var start = new CountDownLatch(1);
try (var executor = Executors.newFixedThreadPool(2)) {
var results = new ArrayList<java.util.concurrent.Future<Integer>>();
for (var command : List.of(success, failure)) results.add(executor.submit(() -> {
assertTrue(start.await(10, TimeUnit.SECONDS));
try { reconcile(command); return 200; } catch (ApiException exception) { return exception.getStatus(); }
}));
start.countDown();
assertEquals(List.of(200, 409), results.stream().map(f -> { try { return f.get(30, TimeUnit.SECONDS); } catch (Exception exception) { throw new AssertionError(exception); } }).sorted().toList());
}
assertEquals(1, count("task_reconciliations")); assertEquals(0, wallet("reserved_balance").signum());
assertEquals("succeeded".equals(state()) ? new BigDecimal("8.56000000") : new BigDecimal("10.00000000"), wallet("balance"));
}
@Test void operationIdCannotBeReusedForDifferentActionOrAnotherAdministrator() {
var command = command("confirm_failed"); reconcile(command);
var changed = new TaskReconciliationCommand(command.requestId(), "confirm_succeeded", command.expectedUpdatedAt(), command.evidenceReference(), command.note(), null, null, new BigDecimal("1.44"), Map.of());
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(changed)).getStatus());
assertEquals(409, assertThrows(ApiException.class, () -> service.reconcile("second-admin-session", taskId, command, meta)).getStatus());
assertEquals(1, count("task_reconciliations"));
}
@Test void staleVersionAndWrongExpectedAmountLeaveFinancialStateUnchanged() {
var command = command("confirm_succeeded");
var wrong = new TaskReconciliationCommand(command.requestId(), command.action(), command.expectedUpdatedAt(), command.evidenceReference(), command.note(), null, null, new BigDecimal("1.45"), command.result());
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(wrong)).getStatus()); unchanged();
jdbc.update("UPDATE model_tasks SET updated_at=updated_at+interval '1 second' WHERE id=?", taskId);
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(command)).getStatus()); unchanged();
}
@Test void missingEvidenceCannotAuthorizeARefund() {
var command = command("confirm_failed");
var invalid = new TaskReconciliationCommand(command.requestId(), command.action(), command.expectedUpdatedAt(), "", "short", null, null, null, null);
assertEquals(422, assertThrows(ApiException.class, () -> reconcile(invalid)).getStatus()); unchanged();
}
@Test void missingTokenUsageCannotReleaseTheReservationUnderASuccessLabel() {
jdbc.update("UPDATE model_tasks SET pricing_snapshot='{\"unit\":\"CNY / 1M tokens\",\"input\":1,\"output\":10,\"taskBilling\":\"openai-image-tokens\"}'::jsonb WHERE id=?", taskId);
assertEquals("TASK_USAGE_REQUIRED", assertThrows(ApiException.class, () -> reconcile(command("confirm_succeeded"))).getCode()); unchanged();
}
@Test void auditFailureRollsBackTaskWalletUsageAndReceiptTogether() {
var command = command("confirm_succeeded");
doThrow(new IllegalStateException("audit database unavailable")).when(auth).audit(eq("task.reconciled"), eq(adminId), any(), anyMap());
assertThrows(IllegalStateException.class, () -> reconcile(command)); unchanged();
assertNull(jdbc.queryForObject("SELECT completion_status FROM model_tasks WHERE id=?", String.class, taskId));
assertEquals("reconciling", jdbc.queryForObject("SELECT status FROM usage_events WHERE provider_task_id=?", String.class, taskId));
}
@Test void resumeOnlyQueriesTheOriginalBindingWithoutChangingTheWallet() {
var command = command("resume_polling"); reconcile(command); reconcile(command);
assertEquals("running", state());
assertEquals(new BigDecimal("1.44000000"), wallet("reserved_balance"));
assertEquals(new BigDecimal("10.00000000"), wallet("balance"));
assertEquals(1, count("gateway_budget_holds")); assertEquals(0, count("wallet_entries"));
assertEquals(1, count("task_reconciliations"));
verify(provider, times(1)).requirePollableBinding("provider-vast", "model-mock-3d", "v3.1-20260211", "3d", fingerprint);
verifyNoMoreInteractions(provider);
}
@Test void differentProviderTaskCannotBeAttachedToAnAcceptedJob() {
var command = command("resume_polling");
var other = new TaskReconciliationCommand(command.requestId(), command.action(), command.expectedUpdatedAt(), command.evidenceReference(), command.note(), "some-other-task", fingerprint, null, null);
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(other)).getStatus()); unchanged(); verifyNoInteractions(provider);
}
@Test void sameProvenUpstreamTaskCannotBeAssociatedWithTwoLocalTasks() {
assertThrows(DataAccessException.class, () -> jdbc.update("INSERT INTO model_tasks(id,user_id,api_key_id,model_id,task_type,status,prompt,reserved_amount,provider_task_id,provider_key_fingerprint,provider_id_snapshot,upstream_model_snapshot) VALUES ('duplicate-local-task',?,?,'model-mock-3d','3d','running','',0,'original-provider-task',?,'provider-vast','v3.1-20260211')", userId, keyId, fingerprint));
unchanged();
}
@Test void validTokenUsageIsQuotedAndSettledOnceUsingTheFrozenSchedule() {
jdbc.update("UPDATE model_tasks SET pricing_snapshot='{\"unit\":\"CNY / 1M tokens\",\"input\":1,\"output\":10,\"taskBilling\":\"openai-image-tokens\"}'::jsonb WHERE id=?", taskId);
var result = Map.<String,Object>of("usage", Map.of("input_tokens", 100, "output_tokens", 100));
var quote = service.quote("admin-session", taskId, result);
assertEquals("0.00110000", quote.get("expectedCost"));
var command = new TaskReconciliationCommand(UUID.randomUUID(), "confirm_succeeded", version(), "provider-token-bill", "Verified exact input and output token usage from provider record", null, null, new BigDecimal("0.0011"), result);
reconcile(command); reconcile(command);
assertEquals(new BigDecimal("9.99890000"), wallet("balance"));
assertEquals(0, wallet("reserved_balance").signum());
assertEquals(100L, jdbc.queryForObject("SELECT input_units FROM usage_events WHERE provider_task_id=?", Long.class, taskId));
assertEquals(1, count("wallet_entries"));
}
@Test void unavailableCredentialAndAmbiguousLegacyRouteCannotResumePolling() {
doThrow(new ApiException("UPSTREAM_KEY_NOT_AVAILABLE", "safe message", 503)).when(provider).requirePollableBinding(anyString(), anyString(), anyString(), anyString(), anyString());
assertEquals(503, assertThrows(ApiException.class, () -> reconcile(command("resume_polling"))).getStatus()); unchanged();
jdbc.update("UPDATE model_tasks SET provider_id_snapshot=NULL,upstream_model_snapshot=NULL WHERE id=?", taskId);
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(command("resume_polling"))).getStatus()); unchanged();
}
@Test void existingObservedOutcomeCannotBeOverruledManually() {
jdbc.update("UPDATE model_tasks SET completion_status='succeeded' WHERE id=?", taskId);
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(command("confirm_failed"))).getStatus()); unchanged();
}
@Test void insufficientHoldIsAConflictNotABalancingPlug() {
jdbc.update("UPDATE user_wallets SET reserved_balance=0 WHERE user_id=?", userId);
assertEquals(409, assertThrows(ApiException.class, () -> reconcile(command("confirm_failed"))).getStatus());
assertEquals("reconciling", state()); assertEquals(new BigDecimal("10.00000000"), wallet("balance")); assertEquals(0, count("task_reconciliations"));
}
@Test void receiptsAreAppendOnlyAndNeverContainPromptOrGeneratedContent() {
reconcile(command("confirm_succeeded"));
String stored = jdbc.queryForObject("SELECT row_to_json(r)::text FROM task_reconciliations r", String.class);
assertFalse(stored.contains("private prompt")); assertFalse(stored.contains("generated.glb"));
assertThrows(DataAccessException.class, () -> jdbc.update("DELETE FROM task_reconciliations"));
assertThrows(DataAccessException.class, () -> jdbc.update("UPDATE task_reconciliations SET note='rewritten history'"));
assertEquals(1, count("task_reconciliations"));
}
@Test void listAndHistoryAreBoundedAdministratorViewsWithoutRawPayloads() {
var list = service.list("admin-session", 1);
assertEquals(1L, list.get("total"));
String encoded = assertDoesNotThrow(() -> json.writeValueAsString(list));
assertFalse(encoded.contains("private prompt")); assertFalse(encoded.contains("input_payload"));
reconcile(command("confirm_failed"));
assertEquals(1, ((List<?>) service.detail("admin-session", taskId).get("history")).size());
assertEquals(0L, service.list("admin-session", 1).get("total"));
}
@Test void freeFailureHasAnExplicitZeroCostAndNoLedgerEntry() {
jdbc.update("UPDATE model_tasks SET reserved_amount=0 WHERE id=?", taskId);
jdbc.update("UPDATE user_wallets SET reserved_balance=0 WHERE user_id=?", userId);
reconcile(command("confirm_failed"));
assertEquals(0, jdbc.queryForObject("SELECT cost_amount FROM model_tasks WHERE id=?", BigDecimal.class, taskId).signum());
assertEquals(0, count("wallet_entries"));
}
}
@@ -0,0 +1,34 @@
package com.moyi.controlplane.task;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.moyi.controlplane.common.ApiException;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class TaskReconciliationResultTest {
private final ObjectMapper json = new ObjectMapper();
@Test void acceptsOnlyNormalizedAssetsAndIntegralUsage() {
var result = TaskReconciliationResult.normalize(Map.of("url", "https://example.test/image.png", "usage", Map.of("input_tokens", "10", "output_tokens", 20,
"input_tokens_details", Map.of("text_tokens", 2, "image_tokens", 8))), json);
assertEquals(10L, ((Map<?,?>) result.get("usage")).get("input_tokens"));
}
@Test void rejectsCredentialsRawPayloadsAndUnsafeAssetUrls() {
for (Map<String,Object> value : List.<Map<String,Object>>of(
Map.of("Authorization", "secret"), Map.of("base64", "large media"),
Map.of("url", "javascript:alert(1)"), Map.of("url", "https://user:password@example.test/a"),
Map.of("usage", Map.of("api_key", "secret")), Map.of("usage", Map.of("input_tokens", -1)),
Map.of("usage", Map.of("output_tokens", 1.5)), Map.of("usage", Map.of("input_tokens", 100_000_001)))) {
assertEquals(422, assertThrows(ApiException.class, () -> TaskReconciliationResult.normalize(value, json)).getStatus());
}
}
@Test void rejectsOversizeObjectsAndCommandLoggingOmitsEvidenceAndPayload() {
assertThrows(ApiException.class, () -> TaskReconciliationResult.normalize(Map.of("url", "https://example.test/" + "x".repeat(40_000)), json));
var command = new TaskReconciliationCommand(java.util.UUID.randomUUID(), "confirm_succeeded", null, "private bill", "private note", null, null, null, Map.of("url", "private result"));
assertFalse(command.toString().contains("private"));
}
}
@@ -39,7 +39,7 @@ class UsageCapacityPostgresTest {
UUID user = UUID.randomUUID(), key = UUID.randomUUID();
jdbc.update("INSERT INTO users(id,account,account_kind,password_hash,display_name) VALUES (?,'capacity-user','username','not-loginable','Capacity user')", user);
jdbc.update("INSERT INTO platform_api_keys(id,user_id,name,key_prefix,key_hash) VALUES (?,?,'Capacity test','local-test',?)", key, user, "c".repeat(64));
for (String state : List.of("running", "queued", "succeeded")) {
for (String state : List.of("running", "queued", "reconciling", "succeeded")) {
jdbc.update("INSERT INTO model_tasks(id,user_id,api_key_id,model_id,task_type,status,created_at) VALUES (?,?,?,'capacity-route-105','3d',?,now()-interval '1 day')", "task-" + state, user, key, state);
}
}
@@ -67,9 +67,11 @@ class UsageCapacityPostgresTest {
Map<String, Object> async = groups.stream().filter(group -> group.get("id").equals("capacity-route-105")).findFirst().orElseThrow();
assertEquals(1L, async.get("running"), "A task older than ten minutes must remain in live capacity");
assertEquals(1L, async.get("queued"), "Running tasks must not also count as queued");
assertEquals(1L, async.get("reconciling"), "Unknown upstream outcomes must remain visible, not be counted as failed or idle");
assertEquals(7, async.get("limit"), "Null route limits use the real configured model default");
assertEquals(3L, summary.get("running"));
assertEquals(4L, summary.get("waiting"));
assertEquals(1L, summary.get("reconciling"));
Map<?, ?> config = (Map<?, ?>) summary.get("capacityConfig");
assertEquals(4500L, config.get("queueWaitMs"));
assertEquals(3, config.get("defaultKeyConcurrency"));
@@ -8,6 +8,8 @@
- 更新时间:2026-08-20
- 当前节点:D4,密钥与模型目录 Mock 闭环
> 本文为 2026-08-20 的历史设计基线,保留当时的选型与节点记录,不代表最新实施状态。当前 Java/PostgreSQL/Redis 实现、已验证范围及 CPR 差距请以 `README.md` 和 `24-security-and-cpr-task-tracker.md` 为准。
## 一、开源项目使用决策
模亦灵枢不直接修改某个中转站项目并把它整体作为产品,而采用“开源网关能力 + MOYI 自研控制面与双端控制台”的组合方式。这样可以复用成熟的模型代理、虚拟密钥和用量能力,同时保留多类型模型、异步 3D 任务、额度台账和未来 MaaS 能力的产品自主性。
@@ -138,6 +140,6 @@ POST /v1/tasks
| 邮件/短信 | 控制台验证码 | 服务商、模板、签名、发送限制和风控 |
| 地区与合规 | 不对公网开放 | 服务地区、隐私政策、日志和内容安全要求 |
## 六、当前结论
## 六、当时的阶段结论(历史记录)
当前 D3 已完成认证与权限基线。D4 已提供平台 Key 生命周期、统一模型目录、标准同步调用和异步 3D 任务接口,并完成首个 Tripo 真实 Provider Adapter 与文生 3D 验收。目录和真实任务归属当前仍使用内存仓库,PostgreSQL 表结构已准备;下一步进入 D5 调用计量,并在 D6 持久化异步任务、队列和结果文件。详见 [D4 密钥与模型目录实施计划](06-d4-integration-plan.md)。
+70
View File
@@ -0,0 +1,70 @@
# API 请求模块化与风控扩展边界
更新日期:2026-10-21。本文对应 Java 后端的第一阶段模块化改造,不代表全部安全审计问题已解决或已上线。
## 模块分工
采用同一 Spring Boot 应用内的模块化单体,暂不拆微服务或 Maven 子工程,以保留钱包事务的一致性。
| 模块 | 负责 | 不负责 |
| --- | --- | --- |
| Gateway / Native / Playground / Task HTTP 入口 | 身份认证、Key 所属权检查、来源标记、早期 RPM 限流、协议规范化 | 风控规则的业务实现、单独复制账务流程 |
| `gateway.request` | 不可变请求上下文、风控规则与决策、同步请求执行编排、准入租约、结算持久化异常分类 | 上游凭据、HTTP/SSE 解析、异步轮询状态机 |
| `gateway.ChatMeter` / `NativeProtocolMeter` | 请求合法性、用量规范化、协议完成条件 | 选择账号、重试、修改余额 |
| `provider.credentials` | 渠道/路由/模型密钥池解析、去重、原子轮转 | 账号冷却、是否安全重试、任务凭据指纹绑定 |
| `provider.transport.SseEventReader` | SSE 解帧、严格 UTF-8、单行/事件字符上限和总响应字节上限 | 扣费、HTTP 响应写出、路由切换 |
| Provider / Failover / UpstreamHealth | 上游协议适配、HTTP 生命周期、Key 健康、授权约束下的安全换路 | 绕过风险策略、复制钱包事务 |
| Wallet / Usage | 价格快照、预留、结算、释放、持久化审计与恢复 | 估算用量后扣费、透明重放上游生成 |
| AsyncTaskService | 通过共享风险门禁后排队,worker 提交、轮询和终态事务 | 复用同步请求的短生命周期并发租约 |
同步调用顺序:认证 → 基础 RPM → 路由与协议校验 → 风控 → 并发准入 → 钱包预留 → 上游调用 → 精简用量快照 → 原子结算 → 释放准入。
异步创建顺序:认证 → 基础 RPM → 路由与输入校验 → 风控 → 数据库队列准入及价格快照/预留 → worker。异步的长任务租约继续由原有任务调度管理,不能使用同步 Controller 的生命周期。
## 风控规则接口
- 通过 Spring 注册 `ApiRequestRiskRule`,每个规则需要唯一、稳定的小写 ID。规则只接收 `ApiRequestContext`,不能读取原始 API Key、Authorization 或请求正文。
- `GATEWAY_RISK_MODE=shadow` 为默认值:规则命中或规则异常只形成影子信号,不阻断请求;`enforce` 下才会拒绝、限速或因策略异常返回 503。
- 未注册规则时不会凭空产生反欺诈能力。部署配置中写 `enforce` 也不等于已实现有效风控。
- 多规则时拒绝优先于限速,限速采用命中规则中的最长等待时间。非法模式在启动时失败,不能默默关闭安全检查。
- 原有基础频率与用户/Key/模型并发策略继续生效。不能因为增加高级风控而让非法请求绕过基础 RPM。
## 来源与隐私
- 公网入口固定标记为 `API`;只有已经验证会话和 Key 所属权的 Playground 入口使用 `PLAYGROUND`。`X-Moyi-Source` 仅保留为 Playground 的跨站请求约束之一,不作为可信来源证明。
- IP 由 Servlet 的 `getRemoteAddr()` 获取。不直接读取客户端 `X-Forwarded-For`;需要真实 IP 时,应由明确可信的反向代理/Tomcat 配置提供。
- 风控日志不写 IP、User-Agent、密钥或原始正文;上下文/元数据的默认字符串表示也不含 IP。已有用量审计可保存经脱敏的请求响应,保留策略与访问权限需要独立评估。
- 来源/IP 显式传入钱包及任务用量记录,避免仅依赖线程局部 HTTP 上下文。数据库中的 IP 属于敏感审计数据,不应出现在普通风控日志中。
## 结算与恢复
- `V44__recoverable_chat_settlement.sql` 新增归一化用量与耗时快照。快照独立于最终扣款事务提交,不保存提示词、工具参数、生成内容或凭据。
- 正常路径仍用原始响应完成审计;账本写入失败后使用精简用量恢复,不能保证重放原始生成内容。
- 已有有效快照时,账本异常返回 `SETTLEMENT_PENDING`,不能直接调用 `fail()` 释放成功调用的冻结额度。过期恢复任务逐笔事务重试,并利用行锁与原有终态检查确保只扣一次。
- 无完整结果或有效用量时仍不能估算扣费;缺少快照的过期孤儿请求沿用释放策略。
- **剩余窗口:**数据库在快照提交前不可用、或进程在收到上游结果后而快照提交前崩溃,仍可能没有可自动恢复的用量。需配合上游 request ID、对账记录和后续持久执行状态设计,不能宣称端到端 exactly-once 已实现。
- `SETTLEMENT_PENDING` 后不能用新的幂等键自动重新生成。同步既有幂等语义保持为相同用户、相同幂等键不重复调用;结果查询/重放接口尚未实现。
## 保持不变的上游约束
密钥顺序保留为路由池 → provider+model 池 → 渠道池;渠道池内保留原实现各自的 fallback/alias 顺序。轮转状态保存在客户端实例中,不随动态 DB 渠道读取重置。异步轮询继续绑定提交时 Key 的指纹,禁止随意换账号查询已有任务。
超时和发送结果不明时不透明重放;SSE 已取得后不换路。确定的请求参数错误兼容重试、明确未受理的账号拒绝及备用路由授权约束仍由原 Provider/Failover 决定。HTTP 生命周期、整体截止时间、健康冷却目前仍在 Provider 门面内,后续可继续提取为独立 Transport。
## 验证
本轮在 Docker 内使用 Java 21、Maven 3.9.9 和独立 PostgreSQL 15.18 执行 `clean verify`:157 项测试,0 失败、0 错误、0 跳过;生成 Spring Boot JAR 成功。临时数据库已清理,未执行线上发布或业务库迁移。该结果不等于真实上游、生产数据库或真实 Redis 并发的完整验收。
运行 Java 21 / Maven:`mvn -B -ntp -f control-plane-java/pom.xml verify`。
PostgreSQL 集成测试由 `MOYI_AUTH_TEST_JDBC_URL` 启用,当前测试限制为本机回环地址,并使用专用测试账号 `moyi_role_test`。必须使用独立临时数据库;测试会创建/删除随机 schema,不能指向业务或生产数据库。Docker 内运行时,`127.0.0.1` 指向容器自身;应使用与测试数据库容器共享的 network namespace,而非将宿主机回环地址直接传入普通 Maven 容器。
回归范围包括:风控 shadow/enforce/异常、所有主要调用入口拒绝时无准入和预留、早期 RPM、来源伪造、IP 显式审计、结算失败与恢复一次扣费、密钥优先级/轮转/任务指纹、SSE 分片/多行/字节上限/完成标记/超时。
## 下一阶段优先级
1. 已完成 `V45`:异步创建 `Idempotency-Key`、请求哈希和数据库唯一性/锁语义,客户端同键重试不重复排队或预留。
2. 已完成 `V46` 的未知受理持久状态与冻结保护;V47/V48 后续补人工核账与执行绑定,详见 `23-audited-task-reconciliation.md`。本阶段原始 190 项后端验证结果见 `22-async-task-safety.md`。
3. 已增加异步已观察终态的结算恢复与待核账提示;继续完善执行路由快照、重试退避、人工对账和不可原子提交窗口。
4. HTTP Transport 生命周期继续拆分;真实 Redis 并发、租约回收与队列公平性回归。
5. 再进入用户/角色/会话、管理操作与密钥生命周期的后续安全优化;本轮模块改造不替代完整审计。
+61
View File
@@ -0,0 +1,61 @@
# 异步任务幂等与执行安全
## 阶段一:创建幂等(2026-10-21)
- 公网 `/v1/{3d|video|image|audio}/tasks` 与 Playground 创建入口支持可选 `Idempotency-Key`。
- 键为 1–128 个可见 ASCII 字符;只保存 SHA-256,不保存原始键。幂等命名空间为用户,不局限于单个 API Key。
- 请求摘要包括任务类型和完整请求体,递归排序 JSON 对象键、规范化数值,保留数组顺序及文本内容;不忽略未知字段或悄悄修剪提示词。
- 相同用户、相同键、相同请求返回原任务的当前状态,包括成功或失败后的重放;相同键和不同请求返回 HTTP 409 / `IDEMPOTENCY_CONFLICT`。
- 有效凭据校验、基础 RPM 检查先执行;幂等重放不再进入生成流程,因此不受后来变更的模型价格、队列容量、余额或路由下线影响。这与同一用户查询历史任务的权限范围一致。
- 同键创建在数据库事务中取得用户/键摘要 advisory lock;部分唯一索引提供最终约束。检查原任务发生在队列计数、预算占用、钱包冻结之前。
- 不提供键失效后自动重用,避免客户端晚到重试再次生成。无幂等键的调用保持原有语义。
验证:Java 21 / Maven,隔离 PostgreSQL 15.18(不连接业务数据库),26 项针对性测试通过,0 失败、0 跳过。包含 8 路并发同键请求、重放、请求冲突、完成后路由下线重放、键摘要规范化与风控顺序。
## 阶段二:未知受理与可恢复结算(2026-10-21)
### 模块边界
- `provider.transport.ProviderRequestException` 只描述 HTTP 边界事实:`NOT_SENT`、`REJECTED`、`UNKNOWN`,保留可安全展示的错误码及 HTTP 状态,不附带上游响应体、凭据或异常原文;不决定重试和退款。
- `task.TaskOutcomeJournal` 持久化已观察的终态和待核账状态,不发送请求、不修改钱包、不制定退款策略。
- `AsyncTaskService` 编排提交、只读轮询和本地结算。收到生成结果后先提交日志,再在原有钱包/任务/审计事务中结算;账本失败不能被提交请求异常处理器误认为上游生成失败。
### 状态与账务
| 观察到的事实 | 任务状态 / 后续动作 | 钱包与预算 |
| --- | --- | --- |
| 本地输入/配置校验失败、请求尚未发出 | `failed` | 原子释放预留,不扣费 |
| HTTP 400/401/402/403/404/405/413/415/422/429 明确拒绝 | `failed` | 原子释放预留,不自动重发 |
| HTTP 408/409、5xx、网络异常、成功响应不可解析 | `reconciling`,不自动重发 | 保留冻结及预算占用,等待证据核账 |
| 回执缺少合法任务 ID,或回执无法保存 | `reconciling` | 保留冻结,不假定生成失败 |
| worker 已认领、两分钟后仍无 ID 且无终态日志 | `reconciling` | 保留冻结,不重新派发 |
| 已受理任务的轮询期限到期,或原凭据已移除 | `reconciling`,停止自动轮询 | 保留冻结,不将查询失败视为生成失败 |
| 已收到上游终态,但本地结算失败 | 保存 `completion_status` 和归一化结果,下轮只重试本地结算 | 保留冻结,成功后只扣一次 |
| 已确认上游失败/取消 | 原子提交终态与审计 | 释放预留,只执行一次 |
兼容视频路径的备用接口只在可结构化识别的 HTTP 404/405 拒绝后尝试,不再解析错误文案判断重试;5xx 或结果未知不换接口。轮询是只读操作,瞬时鉴权/限流/网络异常可继续尝试至持久化期限,但不能由此推导退款。
`V46` 扩展 `model_tasks` / `usage_events` 状态约束并增加终态观察日志。生成内容复用已有 `result` 列,不重复存两份。日志一旦观察到终态,晚到的冲突状态不能覆盖它;过期认领检查也不能覆盖已经保存的任务编号。
`reconciling` 仍计入异步模型/API Key/用户的数据库并发与队列容量,避免未知任务被无限替换为新任务。用户任务页展示“待核账”,提示额度仍冻结、勿重复提交;使用记录可筛选,管理员容量分组保留待核账计数,即使渠道已停用也不隐藏。
状态解析改为明确白名单:`success` / `succeeded` / `completed` 才能结算;不再把 `unsuccessful` 当成功,也不把 `cancelling` 当取消完成。其他未知状态继续只读查询,期限到后进入核账状态。
### 上线与核账边界
1. 本阶段只验证隔离测试数据库,没有迁移业务数据库或部署线上服务。发布时先停旧 worker,再迁移 `V45` / `V46` 并启新 worker,避免混跑旧的自动退款逻辑。
2. 不回溯改写历史失败任务或重新冻结曾经释放的资金。历史的提交中断、超时、凭据丢失记录须独立审计,不能自动扣款。
3. 待核账状态不会按 TTL 自动退款,不会自动重发。收集本地任务 ID、用户、时间、模型路由、预留金额、上游任务/账单记录;不要把完整凭据放进工单或日志。
4. 后续 V47/V48 已补受审计的人工核账入口:恢复只读查询、确认失败释放原冻结、按原快照确认成功,详见 `23-audited-task-reconciliation.md`。不提供无证据一键退款、重新生成、任意改余额或追扣已释放资金。
5. 成功的 token 图片任务缺少可核验用量时,仍沿用既有 `TASK_USAGE_PENDING` 规则:保留结果、暂不扣费、释放该次预留并标记待核账。本阶段没有把“已知完成但计量缺失”与“是否受理未知”混为一类。
6. HTTP 受理至观察日志提交之间仍有无法跨外部系统原子提交的窗口。数据库完全不可用时也不能承诺保存回执;恢复后转入待核账,而非声称精确恢复所有生成结果。
后续人工核账与 provider/model 执行绑定已在 V47/V48 完成,范围及剩余限制见 `23-audited-task-reconciliation.md`。继续重点处理共享有界 HTTP Transport、媒体下载安全、完整配置版本绑定与上游有证明的幂等能力适配。
### 最终验证
2026-10-21:Docker / Java 21 / Maven 3.9.9 执行 `mvn -B -o clean verify`,后端 190 项测试全部通过(0 失败、0 错误、0 跳过),Spring Boot JAR 构建成功。使用独立 PostgreSQL 15.18,Maven 容器共享测试数据库的 network namespace;未访问业务数据库或真实供应商 API。
前端 `npm run build`(含 `vue-tsc --noEmit`)和 `npm test` 通过,22 项测试全部通过。前端“待核账”提示已通过类型检查和构建,未将这些测试描述为完整浏览器交互验收。`git diff --check` 无错误。
重点回归覆盖:8 路同键并发、终态/失败后重放、不同请求冲突、HTTP 408/409/5xx、收到 POST 后断开响应且不重发、成功响应 JSON 无法解析、无任务 ID、旧认领检查竞态、凭据丢失、轮询期限、待核账并发占位、审计事务回滚、成功结果先保存后结算、晚到失败不得退款、结算恢复一次扣费,以及模糊状态白名单。
+73
View File
@@ -0,0 +1,73 @@
# 人工任务核账与执行绑定
日期:2026-10-21。本阶段只在本地隔离环境实现和验证,未迁移业务数据库、未调用真实供应商生成接口、未发布线上服务。
## 模块边界
| 模块 | 职责 | 明确禁止 |
| --- | --- | --- |
| `TaskReconciliationController` | 管理会话 HTTP 适配,JSON / 自定义头防跨站表单屏障 | 上游生成、直接改账 |
| `TaskReconciliationService` | 管理员校验、证据引用、版本检查、幂等收据、人工决策事务 | 任意指定扣款金额、自动重发生成 |
| `TaskSettlementService` | worker 与人工核账共享的任务/钱包/用量原子结算及预留释放 | 另起一套人工钱包加减逻辑 |
| `TaskOutcomeJournal` | 记录已观察终态和未知结果 | 代替财务决策 |
| `AsyncProviderClient.requirePollableBinding` | 本地验证查询能力及已有凭据指纹 | 验证时发 HTTP、接受原始 API Key |
| `TaskReconciliationResult` | 有界结果地址与规范化数值用量白名单 | 原始上游响应、任意字段、base64、大段内容 |
`AsyncTaskService` 的钱包结算逻辑已移动到共享 `TaskSettlementService`,worker 门面只委派调用;不复制两份扣费逻辑。已知成功但用量缺失的既有 `TASK_USAGE_PENDING` 语义暂不改变;人工成功确认则必须严格校验用量,校验失败不修改状态、不释放冻结。
## 管理端入口
“价格版本与成本对账”页面增加“异步任务人工核账”区块。它与原采购成本核对是不同流程:采购成本核对不改用户消费,任务核账可能将原冻结额转为消费或释放。
- `GET /api/admin/task-reconciliation?page=1`:每页 50 笔待核账任务,返回总数。
- `GET /api/admin/task-reconciliation/{taskId}`:最小任务视图与最近 50 条操作历史。查询不加载任务提示词、输入或生成媒体。
- `POST /api/admin/task-reconciliation/{taskId}/quote`:只读金额预览,请求体为规范化结果/用量对象;使用任务原价格快照,返回金额和版本,不写账。
- `POST /api/admin/task-reconciliation/{taskId}`:执行有证据的核账决定;须有 `Content-Type: application/json`、`X-Moyi-Source: console` 和有效管理员 Cookie 会话。自定义头只是与已有严格 CORS 白名单配合的跨站屏障,不是身份或风控来源证明。
每次操作必须包含 UUID `requestId`、原 `expectedUpdatedAt`(保留服务器完整精度)、`evidenceReference`(3–500 字)、`note`(10–2000 字)。不得将凭据粘入证据/说明。证据为管理员人工核对后填写的引用,服务端不自动证明引用真实,不会访问或下载证据 URL。
### 三类决定
1. **`resume_polling`**:恢复原任务的只读轮询,不生成、不退款、不扣款。需要已证实的上游编号和原账号 SHA-256 指纹;账号须仍在服务端凭据池中,执行器须支持该媒体类型的查询。已知编号/账号不能改绑;原执行绑定无法证明的历史任务禁止猜测恢复。允许对有证据的新式回执丢失任务补原编号。
2. **`confirm_failed`**:管理员根据上游失败或明确未受理的记录,确认失败;共享结算事务释放原冻结和预算占用,不额外给钱包充值。不能附带金额或成功结果。
3. **`confirm_succeeded`**:管理员核实成功与计费用量,先预览并确认 `expectedCost`;金额必须等于原价格快照结合该用量的计算结果。没有快照的历史固定价任务沿用原预留额,不拿当前目录价补算。不能任意填写价格、改变上游绑定,缺少 token 用量时拒绝操作。实际费用可超过原预留,沿用现有真实用量记债语义,界面须显式确认。
结果/用量 JSON 最多 32 KiB,仅允许 HTTP(S) 资产地址、format/mimeType 和数值 usage/usageMetadata。URL 不会被后台下载;禁止 userinfo 和非 HTTP(S) scheme。结果字段仍保存在任务既有 result 列,操作收据不复制媒体或提示词。
## 并发、账务与审计
- 操作前使用真实会话鉴权;事务内锁住仍启用的管理员角色并再次检查会话,防止使用过期的预检查结果。普通用户和无会话请求不可访问,幂等重放也要鉴权。
- 同一操作编号取 PostgreSQL advisory lock,以 `request_id` 主键兜底;请求摘要包含任务、动作、版本、证据、说明、规范化结果及确认金额。只有同一管理员原请求可重放;编号用于不同动作/内容或不同管理员返回 409。
- 锁任务行,验证 `reconciling`、无已观察终态、无既有 cost 和完整 updatedAt 版本;锁钱包验证原冻结额仍存在。不同管理员竞争只有一个成功,另一方刷新后再决定,不覆盖可信终态,不用平衡数填补资金缺口。
- 状态、原价结算/释放、用量审计、`task_reconciliations` 收据及 `auth_audit_log` 在同一数据库事务提交;审计失败时全部回滚。Redis 新租约回滚只释放新租约,不删除事务前已存在的任务租约。
- `V48` 收据保存操作者、操作编号、时间、动作、证据引用、说明和最小前后状态。数据库触发器禁止 UPDATE / DELETE;这不是防数据库所有者、超级用户或禁用触发器的防篡改承诺,生产仍需最小权限和独立审计备份。
- 页面提交后如果响应丢失,原样重试保留同一操作编号;服务端返回原收据,不重复扣款。页面不能在缺少报价/明确确认时提交成功核账;展示超过预留金额的风险。
## 执行绑定 `V47`
- 新创建任务冻结 `provider_id_snapshot` 与 `upstream_model_snapshot`;派发和轮询使用快照,不再从改过的模型目录读取另一个渠道/上游模型。当前目录开关仍可停止新任务派发。
- 升级时只对尚未发往上游的 queued 任务冻结可用的当前绑定。已经受理的历史任务不猜测原渠道,缺少原绑定时保留冻结并进入待核账。
- 已有证明的 provider + 原账号指纹 + 上游任务编号有部分唯一索引,阻止同一上游任务关联两笔本地任务。原始密钥不落库。
- 这不是供应商全部配置的不可变快照:provider 的 base URL、协议、别名与凭据池仍可能被管理员修改。生产应避免修改有在途任务的 provider,后续补配置版本引用、执行适配器版本以及更强关联证据。
## Redis 任务租约
独立 `acquire-task-slot.lua` 用 Redis TIME:同一个任务续期不占第二个槽位,满额时另一任务不能进入。返回值区分“新建租约”和“既有租约续期”;数据库回滚不能误删旧租约。既有租约续期后回滚可能保留延长后的 TTL(保守占位),不是承诺 Redis 与数据库跨系统原子提交。
## 上线要求与剩余范围
1. 先停所有旧 worker,备份数据库并检查在途任务,再按顺序迁移 V47/V48、发布新后端和前端;本轮没有执行这些线上步骤。不能混跑按可变目录轮询的旧 worker。
2. 历史任务没有证明的执行绑定会变为待核账,这是安全失败,不是自动补绑或自动退款。安排运营逐笔核实,不批量猜测。
3. 当前是单管理员明确确认模式,未实现双人审批、强制二次验证、证据真实性校验或自动供应商账单同步;不要将它宣传成已完成完整财务审批。
4. 当前人工入口只覆盖“受理/结果未知且原冻结仍在”的 reconciling 任务;不重开已成功/失败任务、不追扣已释放额度、不解决既有 TASK_USAGE_PENDING 的后补消费。
5. 后续继续共享有界 HTTP Transport、媒体下载 SSRF/DNS 重绑定与响应体读取限制、配置版本绑定、真实供应商端到端联调,以及用户会话/角色/管理操作全面安全复核。本阶段不代表整个仓库已安全。
## 验证记录
先前基础改造 74 项回归和核账首批 21 项专项测试已通过。前端 `npm run build`(含 vue-tsc)与 `npm test` 通过,25 项测试零失败。Browser 技能用于隔离 fixture 的真实交互检查:未报价禁提交、金额确认、模拟提交已落地但响应丢失、原编号重放、历史回执、处理后移出待核账列表;两次模拟 HTTP 提交只产生一条收据。未接触真实钱包、会话或供应商。截图保存在被忽略的 `.tools/reconciliation-ui-proof.png`。
最终验证在 2026-10-21 20:06(Asia/Shanghai)完成:Docker / Java 21 / Maven 3.9.9 执行 `mvn -B -o clean verify`,后端 **221 项测试,0 失败、0 错误、0 跳过**,Spring Boot JAR 构建成功。包含 21 项真实 PostgreSQL 核账集成测试、38 项异步任务事务测试和 3 项真实 Redis 租约测试。独立 PostgreSQL 15.18 与 Redis 7 测试容器和 Maven 共享隔离 network namespace,没有映射到业务数据库。
新增关键覆盖:管理员会话和普通用户越权拒绝、自定义头屏障、8 路同操作编号并发一次扣费、不同决定并发一个胜者、幂等编号跨动作/管理员冲突、任务版本过期、冻结不足不补平、原快照报价与 Token 精确结算、缺少用量不释放、审计失败整体回滚、收据禁止改删、同一上游身份不能关联两笔任务、目录修改不改变已冻结执行绑定、旧绑定未知不猜测轮询、Redis 同任务续期及回滚保护。
本地通过不等于生产发布验收、供应商真实账单正确性或完整财务安全审计。测试日志、构建产物和截图均为忽略文件,不进入 Git 提交。
+47
View File
@@ -0,0 +1,47 @@
# 安全与 CPR 差距任务追踪
最新增量审查日期:2026-10-22。对照用户粘贴的 CPR 比对报告与本地代码,不将建议清单视为已完成验收。
## 已交付基线
- `5533a48`:gateway.request 请求边界、风险策略接口、可信来源元数据、共享凭据解析、有界 SSE 与同步结算恢复。默认 shadow,没有实际规则就不代表风控拦截已上线。
- `7626584`:异步创建用户级幂等、哈希身份、事务锁、唯一性约束。
- `5be06a9`:上游受理未知不重发不退款,持久终态日志和本地结算恢复。
- `c716ec7`:人工核账、共享任务结算、provider/model 绑定、幂等审计回执、Redis 同任务续期。221 项后端与 25 项前端测试的结果仅对应该提交,不代表整个建议清单完成。
- `d994ffb`:本轮认证修复,包含共享登录窗口、统一新密码策略、会话撤销与竞态保护、首次管理员配置检查及隔离界面回归。
- `32f6f67`:服务端参考图 DNS 绑定与无凭据直连传输,补齐媒体本地下载边界及 109 项回归,生产网络验收仍未完成。
## 待补任务与当前证据
| ID | 工作项 | 对照结果 / 当前状态 | 验收依据 |
| --- | --- | --- | --- |
| SEC-01 | 登录防爆破与账号/IP 限速 | 本轮已实现并验证:独立提交的账号/IP 固定窗口,未知账号同样限制 | 真实数据库并发、限额持久化、429/Retry-After,见 25 号文档 |
| SEC-02 | 所有角色统一新密码策略 | 本轮已实现并验证:所有角色新密码 8–256,保留既有有界哈希登录 | 各入口相同规则,KDF 参数上界,不静默截断 |
| SEC-03 | 改密与会话撤销、登录竞态 | 本轮已实现并验证:撤销全部会话、清 Cookie、行锁与二次校验 | 改密两种锁顺序、MFA 失败计数及审计回滚 |
| SEC-04 | 任务用户隔离与回执安全 | 已实现核心用户级查询/幂等/冻结/核账,并补跨用户读取回归;完整授权审计仍待继续 | 跨用户读取/列表不可见;核账拒绝普通用户 |
| SEC-05 | Provider 有界 HTTP Transport | 部分完成:异步 JSON/multipart/音频/媒体已接有界读取和总期限;同步/SSE 生命周期尚未统一抽取 | 超大 chunk、慢响应体、已观察拒绝状态、取消、重定向及原适配器回归 |
| SEC-06 | 媒体下载出站 / SSRF | 服务端参考图下载的代码级 DNS 绑定已实现并通过本地回归;生产网络隔离与实际部署验收仍未完成 | 一次解析、整组地址校验、固定地址直连、原域名 TLS/SNI、禁代理与跳转;109 项新增回归,见 26 号文档 |
| SEC-07 | 整体安全审查与生产验收 | 未完成:已补验证码/账号原子性;本轮进一步分离加密投递、统一已知/未知账号重置受理和限额,外部投递不再持 DB 事务;管理接口全审查、时序残余、默认配置、依赖与备份恢复仍需继续 | 本轮新增 22 项投递数据库回归,全量 390 项通过;真实邮件/短信与规格驱动验收未执行,禁止宣称“全仓库安全” |
| COD-01 | Codex 协议能力及验收矩阵 | 未实现完整兼容;保留有限文本 Responses 的明确拒绝语义 | 指定客户端版本、模型、工具和计费联合验收 |
| COD-02 | Responses WebSocket 与响应归属续接 | 未实现,不可宣传完整 Codex 支持 | 原账号/凭据版本归属、断线、取消、租约、费用一致性 |
| COD-03 | 长请求连接/空闲/总期限与续租 | 未完成,不只修改 90 秒常量 | 并发租约、钱包预留恢复、长流/取消共同验证 |
| SCH-01 | 独立上游账号生命周期与调度 | 仅共享 Key 轮换/冷却/指纹,非完整账号实体 | 凭据版本、额度、模型范围、权重及解释能力 |
| SCH-02 | 公平排队 / 取消 / 共享等待预算 | 未完成,现有同步 Redis 竞争不保证 FIFO | 多客户端饥饿/FIFO/取消回收与横向并发验证 |
| OPS-01 | request/attempt 诊断、配置版本、质量门禁 | 部分:已有最终审计/健康/部分快照;本轮加依赖方向测试、CI/Dependabot 配置,远端未运行;逐 attempt/完整配置版本仍缺 | 本地门禁不等于 GitHub CI、CVE 扫描或真实协议验收 |
| OPS-02 | CPR 可选部署 / OAuth 账号池 | 未部署;属于可选方案,不默认新建生产基础设施 | 用户确认部署方式及授权来源后才接入;本地不发付费生成 |
## 实施边界
已补 SEC-01/02/03、SEC-06 的本地下载代码边界、验证码/账号事务,以及请求侧一致的验证码受理和加密异步投递。下一优先级是管理授权审查、同步 Transport 与完整 provider 配置版本,再处理 COD-01/02/03 的协议/续接/长请求合同。投递真实环境、残余时序与生产网络隔离仍属待验收。Codex WebSocket 与 OAuth 不是添加 URL 即可安全完成的能力,须连同状态归属、计量、预留续期和客户端验收逐步实施。未部署、未真实供应商验证或待设计的项必须保持未完成标记。
本轮设计与验证细节见 `25-auth-and-bounded-transport.md`。这份表按用户粘贴报告的锁定基线追踪本地实施,不把旧的 CPR 版本号冒充实时最新版,也没有重新部署 CPR。只做本地 Git commit,不 push,不修改业务数据库。
2026-10-21 本轮最终验证:247 项后端测试全部通过(含真实 PostgreSQL/Redis,0 跳过),前端构建与 25 项测试通过,另完成隔离改密界面交互检查。新增 GitHub 配置只保存本地,远端未执行。本轮没有把剩余 COD/SCH/出站/运营验收项目标为完成。
2026-10-21 媒体安全后续增量:使用 HttpClient 5.6.5 / HttpCore 5.4.4 重新执行 `mvn -B -o clean verify`,356 项后端测试通过(0 失败/错误/跳过),前端构建及 25 项测试通过。服务端本地下载不再使用 DNS 预检后交给 JDK 客户端二次解析的路径。用户指定的规格驱动验收尚未执行:`testingflow` 的 `.venv` 指向不存在的 Python,首个 `qa project list` 失败;没有绕过工具门禁,也没有自行批准规格。细节及下一步审阅范围见 `26-media-dns-binding-and-acceptance.md`。
2026-10-21 验证码事务后续增量:新增 `VerifiedAccountService`,公开注册/重置不再在业务成功前永久消费有效验证码;错误验证码尝试仍提交,审计失败时成功消费、账号、邀请码和会话一起回滚。`mvn -B -o clean verify` 最终 368 项全部通过(0 失败/错误/跳过,2026-10-21 23:05:46 Asia/Shanghai),包含新增 12 项数据库验证;前端沿用本轮已通过的构建及 25 项测试,未修改前端代码。详见 `27-verified-account-transactions.md`。规格驱动验收环境阻塞未解除,没有生成通过报告。
2026-10-22 投递隔离增量:新建 V50 加密 outbox、`VerificationDispatchService` 与封闭目的枚举;发送请求与投递分离,未知/停用账号不触发真实发送,但参与完全相同的请求端限额与写入路径。错误重置码统一公开错误,后台最多双线程、无内存等待队列,旧代次失败不影响新码,投递结果不明不自动重发。后端 `mvn -B -o clean verify` **390 项,0 失败/错误/跳过**,2026-10-22 08:41:38(Asia/Shanghai)结束;前端构建与 25 项测试通过,并通过独立容器的完整 Spring Boot 启动/健康检查。见 `28-verification-delivery-isolation.md`。
用户于 2026-10-22 明确取消每半小时提交检查,聊天自动化 `moyi` 已删除,不再自动轮询或提交;不停止产品内部 worker。当前开发仍只做经审查验证的本地提交,不 push、不部署、不迁移业务数据库;规格驱动验收环境未修复,未自行批准或执行业务验收。
+44
View File
@@ -0,0 +1,44 @@
# 认证安全与有界 HTTP 增量
日期:2026-10-21。范围是补齐用户清单中的可独立验证安全底座,不代表完整 Codex 兼容或全部 CPR 能力已完成;总任务状态见 `24-security-and-cpr-task-tracker.md`。
## 认证边界
- `LoginAttemptLimiter` 在昂贵密码校验前,使用 PostgreSQL 共享固定窗口:同一规范化账号每 60 秒最多 10 次,同一来源每 60 秒最多 50 次,包括未知账号、成功与失败尝试。超过返回 429 / `AUTH_RATE_LIMITED` 和 `Retry-After`。
- `V49` 仅存 scope+身份摘要、计数和期限,不保存原始账号、IP 或密码。摘要是伪名化,不等于抵抗字典反推的匿名化。过期数据在后续尝试中分批清理;这是应用防爆破,不替代 WAF/网关整体流量防护。
- 限速计数独立事务提交,失败登录/外层回滚不能抹去次数;稳定的桶锁顺序防死锁。先完成限速再进入登录事务,不让所有请求各占一条连接后再争抢 REQUIRES_NEW 连接。
- 已知和未知账号均执行有界 PBKDF2 验证;未知账号使用非用户凭据的固定摘要形状,降低明显的是否存在计时差异,不承诺严格恒定响应时长。
- 登录、改密、密码重置通过用户行锁协调。先登录的会话会被随后改密撤销;先改密后,旧密码不能继续创建有效会话。MFA 失败计数、动态码重放标记和恢复码消费与会话签发保持事务语义。
- 改密撤销所有浏览器会话,包括当前会话;Controller 清理 Cookie,界面回到登录入口。仅改显示名称不撤销会话。API Key 不自动撤销,泄露时仍需单独停用或轮换。
- `PasswordPolicy` 统一注册、管理端创建、所有角色改密/重置以及新建初始管理员的新密码规则:8–256 个 UTF-16 code unit、不能仅为空白,不静默裁切/修剪。超级管理员不再豁免长度策略。8 位是兼容当前产品规则,不是声称达到所有部署的最佳口令强度;更强策略、泄露口令检测、强制 MFA 仍需按部署要求补齐。
- 兼容既有哈希和有界旧密码登录,不自动重置已有管理员。匹配端拒绝超大输入、异常盐/密钥尺寸和不合理迭代数,防止损坏哈希控制无界 KDF 工作。新建管理员必须显式配置密码,默认 `admin123` 及示例环境文件中的占位密码被拒绝;已有管理员启动不因缺少新的初始密码配置被重置。
注册/验证码整体流程、重置发送的账号枚举侧信道、管理敏感动作二次验证、会话管理面板等仍在剩余审查中;不能把本增量描述为“认证系统已无漏洞”。
## 有界 HTTP 模块
`provider.transport.BoundedHttpTransport` 只执行一次非流式 HTTP 传输,不选择账号/渠道、不读写钱包、不决定重试、也不携带业务风险策略:
1. 强制调用方提供总期限,期限覆盖响应头和整个响应体;超时或线程中断时取消 future 和 body subscription,恢复中断标志。
2. 使用限量 BodySubscriber,在继续累计之前检查 chunk 总量;声明 Content-Length 已超限时提前拒绝。错误体最多 64 KiB,不是“先读完整体再判断大小”。
3. 禁止自动跟随重定向;返回已观察 HTTP 状态及字节,诊断对象不输出 URL、头、密钥或原始响应体。
4. 异步 JSON、multipart、音频接入 8 MiB 上限;媒体输入也使用同一读取上限和 15 秒期限。媒体获取在生成发送前失败时不应被视为已发送生成请求。
5. `AsyncProviderClient` 继续负责协议和受理语义:已观察到明确拒绝状态时保留该事实;成功响应超限、读取超时或接收结果未知时仍按 UNKNOWN 处理,不自动重发、不自动退款。原有 404/405 视频路径兼容边界不放宽。
这还不是全项目统一 Transport:同步/SSE 的现有有界读取和整体 deadline 仍在原 Provider 生命周期内;长请求续租、逐 attempt 交付状态、统一取消与跨路由共享预算未完成。
**该提交时的媒体 SSRF 尚未闭环。** 本阶段约束了响应体、总期限和重定向,但当时旧 DNS 预检与 JDK HTTP 实际解析之间的重绑定窗口仍存在。2026-10-21 的后续增量已用固定地址直连替换该路径,并补特殊地址范围回归,详见 `26-media-dns-binding-and-acceptance.md`;生产网络策略仍未验收。不能将本地下载修复扩大为全项目出站安全保证,也没有擅自更改业务网络或把任意用户 URL 加入可信名单。
## 工程验证与限制
- 新增 3 项模块依赖方向测试:请求应用边界不直接持有 HTTP/JDBC/凭据;transport 不依赖风险/路由/钱包;结算模块不调用上游。它们是轻量源码门禁,不替代代码审查或沙箱。
- 新增跨用户任务查询回归,其他用户不能读取任务结果或列出其任务。
- `.github/workflows/verification.yml` 配置 Java 21、Node 22、锁文件安装、前后端构建与独立 PostgreSQL/Redis 测试;Actions 固定完整 commit SHA(2026-10-21 通过各上游 GitHub API 核对 v4 引用,pnpm 的 annotated tag 另行解引用),避免仅依赖可移动主版本标签。此项不等于审计了 Actions 全部源码。`.github/dependabot.yml` 配置 Maven/npm/Actions 更新检查。仅本地保存配置,未 push、未在 GitHub 运行,不能声称远端 CI 或依赖漏洞扫描已通过。
- 本机 pnpm 包装器提示 esbuild 构建脚本待审批。没有全局关闭审批,也没有批准未知脚本;移除了本次自动生成的待审批占位配置。使用现有安装依赖执行 `npm run build` / `npm test`,前端构建与 25 项测试通过;CI 的锁文件安装仍需首次远端实际验证。
- 使用 Browser 技能在全部 fetch 被拦截的隔离 fixture 中验证改密事件使设置页退回登录入口;密码是测试字符串,没有操作真实账号或浏览器会话。截图 `.tools/auth-security-ui-proof.png` 被忽略,不进入 Git。
最终全量验证于 2026-10-21 22:01(Asia/Shanghai)完成:Java 21 / Maven 3.9.9 的 `mvn -B -o clean verify`,**247 项测试,0 失败、0 错误、0 跳过**,Spring Boot JAR 成功构建。使用独立 PostgreSQL 15.18、Redis 7 容器和共享测试 network namespace,没有连接业务数据库。包含 9 项认证数据库安全回归、既有认证/角色/MFA/钱包/异步核账回归、8 项传输测试、2 项异步超限受理语义回归、3 项依赖方向门禁和跨用户任务读取回归。
前端 `npm run build`(vue-tsc + Vite)和 `npm test` 再次通过,25 项测试零失败;隔离界面验证通过。`git diff --check` 无错误。日志 `.tools/security-final-verify.log` 和截图被忽略,不进入提交。测试通过只证明本次本地已覆盖范围;CI 未远端执行,媒体 DNS 绑定、WebSocket/续接、长请求续租、公平调度、完整配置版本、依赖漏洞与备份恢复验收仍未完成。
不部署、不迁移业务数据库、不进行付费上游生成;只创建本地阶段性 Git commit,不 push。发布此版本前必须先迁移 V49,再启动新后端;首次启动须替换环境示例占位密码。不要将旧 TypeScript 参考服务当成与 Java 同时对外写账的第二套生产后端。
@@ -0,0 +1,46 @@
# 媒体 DNS 绑定与功能验收状态
日期:2026-10-21。增量范围是服务端为 Gemini 图像输入、OpenAI 兼容图像编辑和视频参考图下载的 HTTPS 文件。不是完整生产安全审计,不涉及 CPR 部署、业务数据迁移或付费上游生成。
## 已实现边界
- `provider.media.PublicMediaAddressPolicy`:校验 HTTPS URI,拒绝 userinfo、fragment、zone ID 与非法端口。DNS 答案整组校验;只要存在非公开/特殊用途地址就全部拒绝,不选择其中的公网地址继续。答案为空、含 null 或超过 16 个地址同样拒绝。IPv4 覆盖本机、私网、共享地址、链路本地、基准测试、文档、组播、保留地址和平台虚拟地址;IPv6 仅接受原生 global-unicast 候选,排除特殊、文档、6to4、已退役 6bone 等前缀。策略采用保守拒绝,某些特殊用途的实际可达地址也可能被拒绝。
- `provider.transport.PinnedHttpsTransport`:每次调用只向系统解析一次,经策略验证后使用这份不可变地址集合连接。HTTP 库自己的 DNS 回调只返回该集合,不调用系统 DNS,也不反向解析。URI 始终保留原域名,Host、TLS 证书链、主机名校验和 SNI 不替换为 IP;证书校验未放宽。
- 路由层固定直连,底层 Socket 显式 `Proxy.NO_PROXY`,防止 JVM 的系统 SOCKS 代理绕过本地 DNS 绑定。没有自动跳转、HTTP 重试、Cookie 管理、认证缓存或跨下载连接复用;不接受任意请求头、上游 Key 或浏览器 Cookie。
- 15 秒总期限覆盖 DNS、连接、TLS 和整个响应体;超时/中断取消 socket,调用方中断标志保留。DNS 原生实现可能不能及时中断,迟到的解析结果会检查取消状态,不能在超时后再发起 HTTP。最多 8 个后台工作线程、无等待队列;饱和直接安全失败,避免无限积压。
- 成功体最多 8 MiB,错误体最多 64 KiB;声明超限提前拒绝,chunked 在读取过程中计数。HTTP/1 响应行最多 8192 字符、头数量最多 64。禁自动解压,拒绝非 identity 的 Content-Encoding。先取消再关闭,关闭时不排空超大/无限响应,清理异常不覆盖主要超限原因。
- `provider.media.MediaDownloader` 只负责文件输入、MIME 类型及域错误映射;`AsyncProviderClient` 不再自行执行 DNS 策略。模块不依赖凭据、路由、风控、认证或账务,新增依赖方向检查。媒体下载失败仍是生成之前的 `NOT_SENT`,不制造“上游已受理未知”的结果。
## 依赖与本地验证
新模块使用 Apache HttpClient **5.6.5** 和配套 HttpCore **5.4.4**,在 POM 中固定属性,不沿用 Spring Boot 3.4.5 管理的旧组合,也不采用 5.7 alpha。版本在 2026-10-21 通过 Maven Central metadata/POM 与 Apache 官方发布记录核对;这不是完整依赖漏洞扫描或零漏洞承诺。参考资料:
- Apache 发布记录:`https://hc.apache.org/news.html`
- HttpClient 父 POM:`https://repo.maven.apache.org/maven2/org/apache/httpcomponents/client5/httpclient5-parent/5.6.5/httpclient5-parent-5.6.5.pom`
最终后端命令为 `mvn -B -o clean verify`,Java 21 / Maven 3.9.9 容器,共享本机依赖缓存。**356 项测试,0 失败、0 错误、0 跳过,JAR 构建成功**;结束时间 2026-10-21 22:44:03(Asia/Shanghai)。相对前一基线增加 109 项:76 项 URI/地址策略、13 项下载域边界、16 项真实本地 HTTPS 传输、3 项适配器发送前语义,以及 1 项模块依赖方向检查。
TLS 测试用 keytool 临时生成 `.test` 证书,仅测试客户端信任该证书;默认客户端必须拒绝它,可信但域名不匹配同样拒绝。测试还验证原 Host/SNI、DNS 只查一次、后续下载重新校验、Cookie 不复用、系统 SOCKS 代理不被读取、重定向不跟随、超限/响应头/压缩/慢体/中断/DNS 迟到和无队列饱和。没有通过关闭证书校验来获得绿灯。
数据库测试使用本轮独立 PostgreSQL 15.18 和 Redis 7 容器,共享隔离 network namespace,未映射业务数据库端口、未连接业务库。前端 `npm run build` 和 `npm test` 也通过,25 项测试零失败。日志 `.tools/media-dns-final-verify.log` 与构建目录忽略,不进入 Git。提交前再执行 `git diff --check`、逐路径检查差异与暂存区,只本地提交,不 push。
## 尚未完成的安全边界
本增量只证明上述本地下载路径。传给供应商由其下载的 URL、同步聊天输入中的第三方 URL、浏览器展示结果文件、管理员配置的 Provider Base URL 不属于这条下载路径;不能将它们标成已通过该策略。未知 NAT64/自定义翻译前缀、透明代理、组织路由将公网 IP 引向内部服务等部署行为,需要网络 egress ACL/防火墙控制;地址表也要随地址分配变化维护。生产出站允许范围、部署连通性和真实供应商兼容性均未验收。
图片目前校验媒体类型和字节大小,不解码文件、扫描内容或限制解码后像素;MIME 声明不是内容真实性保证。并发数 8 是保守传输资源上限,不是租户公平队列;长请求续租、完整 Provider 配置版本、WebSocket 续接、公平调度与备份恢复仍在 24 号任务表中保持待完成。
## 规格驱动验收的真实状态
已按用户指定的 `spec-driven-acceptance-testing` 读取完整技能、方法决策表、字段格式、框架 README 和模板。技能记录的 `E:/PROJECT/xpp_dev_agent` 在当前主机不存在,实际框架位于 `D:/PROJECT/xpp_dev_agent/dev-test`,档案目录位于同级 `qa-projects`。
首先执行了框架规定的项目发现命令:
```powershell
# 工作目录 D:\PROJECT\xpp_dev_agent\dev-test
.\.venv\Scripts\python.exe -m qa project list
```
退出码 **1**,提示 Python 路径 `C:/Users/xpp/.conda/envs/ctf/python.exe` 不存在;`.venv/pyvenv.cfg` 仍记录旧的 E 盘创建路径。因此尚未完成项目登记/体检,**未创建或批准业务规格、未执行 `qa run --project`,没有验收报告可交付**。没有手动读取档案代替项目命令、改写共享 Python 环境、绕过批准门禁,或把本轮 Maven/前端测试冒称规格驱动验收。
后续先确认并修复共享框架环境,成功运行项目发现/体检;项目不存在时按 `qa project init` 建档并只填写源码核实的事实。首批建议审阅范围是隔离环境下匿名访问拒绝、普通用户/管理员权限、任务跨用户隔离、改密撤销会话、错误响应不暴露凭据,以及媒体失败不发起生成。规格始终先 draft,必须向用户展示具体输入、预期与清理步骤,获明确批准后才记录批准并执行。变更真实账号、付费模型调用和生产环境测试不在当前授权范围。
+37
View File
@@ -0,0 +1,37 @@
# 验证码与账号事务
日期:2026-10-21。范围为公开注册和密码重置的本地事务一致性,不包含真实短信/邮件发送、生产用户操作或完整认证安全验收。
## 缺陷与修复
此前 `AuthController` 先调用 `VerificationService.consume/consumeReset` 并提交消费,再调用独立事务里的账号操作。即使邀请码无效、显示名称不合法、账号已存在、账号停用或后续审计失败,正确验证码也已经消耗,造成失败后无法重试。简单给外层加默认事务又会让错误验证码次数随异常一起回滚,破坏原有五次上限。
新增 `auth.VerifiedAccountService` 作为公开注册/恢复流程的应用事务边界:
1. 新密码策略先校验,不因弱密码消耗验证码。
2. 开启本次操作拥有的 `REQUIRES_NEW` 事务,验证码行锁、消费和账号业务在同一事务中完成。入口仅用于公开操作,不应嵌入其他业务事务;若未来复用,须重新审查连接池预算及锁顺序,避免持外层连接等待新连接。
3. 验证码失败时,仅将已知校验异常作为结果返回,让事务正常提交尝试计数,再在事务结束后向调用方抛出。尚未调用任何账号修改,不会创建用户或撤销会话;外层调用者回滚也不能抹去错误次数。`VerificationService` 原有 `noRollbackFor=ApiException` 语义保持不变。
4. 验证成功后不捕获业务/审计/存储异常。邀请码消费、兑换记录、用户/密码、会话和审计任一步失败,连同本次成功验证码消费一起回滚;没有补偿式重置验证码,也没有失去行锁后再尝试恢复状态。
5. Controller 不再自行拼接消费和业务写入。注册 Cookie 只在事务成功后签发;重置成功撤销原账号全部服务端浏览器会话,并清理响应中的会话 Cookie,失败不清 Cookie。API Key 不因本增量自动撤销。
## 验证
新增 `VerifiedAccountPostgresTest`,共 12 项真实 PostgreSQL 回归:
- 无效邀请码、显示名称过长、重复账号时有效验证码保持可用。
- 注册审计故障注入后,验证码、用户、会话、邀请码计数和兑换记录全部回滚;排除故障后同一码可成功。
- 重置审计故障注入后,验证码、密码哈希和会话撤销全部回滚;排除故障后可成功,旧会话失效。
- 注册与重置的错误码尝试计数各达到五次,外层事务回滚仍不丢计数,正确码随后被拒绝。
- 注册与重置弱密码不消费验证码;停用账号重置失败也不消耗正确码。
- 注册与重置并发消费同一码都恰好成功一次,另一次返回验证码已失效,审计/兑换不重复。
- HTTP 重置失败保留原 Cookie 和有效会话;成功返回 204、清 Cookie 且服务端会话撤销。
针对测试 `mvn -B -o -Dtest=VerifiedAccountPostgresTest test`:12 项通过,0 失败/错误/跳过。修正了新测试中的 Java 正则转义和遗漏邀请码创建者的 fixture 后重新执行,未跳过失败用例、未降低断言或放宽数据库约束。
最终完整后端命令 `mvn -B -o clean verify`:**368 项,0 失败、0 错误、0 跳过,JAR 构建成功**。结束时间为 2026-10-21 23:05:46(Asia/Shanghai)。Java 21 / Maven 3.9.9 与独立 PostgreSQL 15.18、Redis 7 测试容器,不连接业务库、不对外映射测试数据库端口。原有认证、角色、MFA、媒体 TLS/DNS、任务隔离、幂等、核账和钱包回归均参与执行。日志 `.tools/verified-account-final-verify.log` 被忽略。前端未修改,本轮先前 `npm run build` 和 `npm test` 已通过,25 项测试。
## 保留的待办
这不是注册/找回密码的全部安全闭环。该提交时 `sendReset` 对不存在账号提前返回,对存在账号可能暴露限速/发送错误差异,且短信/邮件调用持事务锁。2026-10-22 的后续增量已分离加密异步投递、统一请求端处理与配额、清理过期记录,详见 `28-verification-delivery-isolation.md`。真实投递时序、SDK 期限、管理敏感操作二次验证及完整授权审查仍需继续;未做付费请求、部署或生产迁移。
用户指定的规格驱动功能验收仍受共享 `testingflow` 虚拟环境缺失 Python 阻塞,详见 26 号文档。本次仅为源码、单元/数据库集成回归,不能称作 `qa run --project` 验收通过。修复共享环境与人工审阅 draft 规格仍是正式执行前置条件。只对已审阅且验证通过的本任务改动创建本地 Git 提交,绝不 push。
@@ -0,0 +1,48 @@
# 验证码投递隔离与重置受理一致性
日期:2026-10-22(Asia/Shanghai)。本增量继续处理 24 号任务表中的认证逻辑缺陷,不代表所有系统开发任务完成。
## 发现与设计
此前重置发送对未知账号立即返回,对已有账号等待邮件/短信网络响应,并只对已有账号执行限额。单次成功 JSON 相同并不能消除 429、发送失败和网络等待带来的账号枚举差异。注册/重置发送还持有全局 PostgreSQL 事务锁等待外部服务。
本轮依照 OWASP Forgot Password Cheat Sheet 的统一消息、相似执行路径、异步处理和配额原则,将受理、投递、账号变更分为三个边界:
- `VerificationService`:仅做输入规范化、公开配置可用性、共享配额、随机码摘要和原子入队。重置请求**不查 users**,已知、未知、停用账号均执行相同 DB/加密写入路径;未知账号同样计入账户、来源及全局预算,不再快速返回。注册仍须有效邀请码。
- `VerificationDispatchService`:事务内原子写码和加密 outbox;后台领取提交后才解密、复查当前代次和账号可用性并调用 sender。网络调用期间没有数据库事务或全局配额锁。`VerifiedAccountService` 原有验证码/账号业务原子性和错误码计数保留。
- `VerificationPurpose`:封闭的注册/密码重置用途枚举,同时提供固定 SQL 表标识及消息目的。邮件不再把重置验证码误写成注册验证码;SMS 沿用服务器配置的通用模板,生产须核对其适用范围。
## 数据与资源边界
V50 给两张验证码表增加 `delivery_id`,并新建 `verification_delivery_outbox`。payload 使用已有 `SecretCipher` 的 AES-GCM,加密账户、验证码、用途与随机代次 ID;工作者检查密文内容中的 ID/用途与队列行一致,防止跨行替换投递。队列表不存明文账户或 OTP;原有码表继续保存摘要,原有发送事件仍含账户/IP,因此**并未声称所有认证表均匿名化或加密**。
共享预算沿用每账户 60 秒 1 次/24 小时 10 次、每来源每小时 30 次、全平台每日 1000 次,注册与重置共同计数。限额锁只覆盖 DB 工作,锁等待最多 2 秒;使用 `clock_timestamp()` 防止排队事务沿用早先开始时间。仍需部署侧 WAF/入口请求上限,不能把发送配额当成全站 DoS 防护。配额计算保留全局序列化与事件扫描,适用于当前低容量内测,未来扩容需专用窗口桶及多租户公平策略。
后台每秒调度,最多两个本机工作线程,`SynchronousQueue` 无内存等待队列;重复调度在繁忙时直接退出。多副本用 `FOR UPDATE SKIP LOCKED` 原子领取,领取状态先提交为 `sending`。领取后的消息只尝试一次:崩溃、网络结果不明或成功回执保存失败都不自动重发。该策略是**至多一次尝试**而不是可靠必达;某些消息可能丢失,用户可在受理冷却期后重新申请。
发送前复查当前 `delivery_id`、未消费、未超过五次、未过期;重置再查账号是否仍 active,未知/停用账号丢弃且不发送。投递失败只按原代次作废,不会消耗后来生成的新码;消费成功/新申请后迟到的旧消息不具备新码效力。状态终结即清除密文 payload,不保留收件人、OTP、异常堆栈或原供应商错误消息到队列表/公开响应。终态保留固定故障码。未处理消息五分钟过期,终态两天后、过期码一天后分批删除;清理当前由空闲 worker 驱动,持续繁忙时物理清理可能延后,过期判断仍生效。
原 SMTP 连接/读写超时和短信 SDK 禁自动重试保持不变。本轮没有证明底层 SDK 的 DNS 或整体调用具备不可绕过的绝对墙钟期限;双线程上限只能限制资源规模,不能保证阻塞线程及时释放。任意网络迟到、数据缓存、不同历史状态造成的可测时序仍未做生产统计验收;本轮不承诺严格恒定时间或消除所有枚举侧信道。
## 对外兼容与上线要求
- 发送接口仍返回 200、`sent: true`、60 秒冷却和 300 秒有效期;`sent` 是历史兼容字段,**只表示受理,并非邮件/短信送达证明**。不返回任务 ID、验证码、内部丢弃原因或凭据。公开重置页改为“若账号存在且可用,将收到重置验证码”。
- 未获取、已消费、已过期、次数耗尽及错误的重置验证码统一返回 `VERIFICATION_INVALID` 与同一消息;不再以“缺失/过期”差异表达内部状态。注册验证码既有错误语义不变。
- 部署本版本前先应用 V50,再启动新后端;所有副本必须使用相同且安全持久化的 `CREDENTIAL_ENCRYPTION_KEY`,并配置 SMTP 或 SMS。缺失加密密钥时能力查询关闭该功能,发送安全失败,不回退明文或同步发送。不要在仍有待发送消息时直接更换密钥。
- 本轮仅在隔离测试库执行迁移;没有修改业务库、真实发送、部署或 push。数据库备份中的历史密文不因在线清理被擦除,仍需备份保留和密钥管理策略。
## 验证证据
新增 `VerificationDispatchPostgresTest` 22 项真实 PostgreSQL 回归,覆盖请求一致性、未知账号同额限制、只向 active 发送、加密/代次校验、供应商失败不外泄和不重试、网络期间无事务/锁、新旧代次隔离、崩溃/过期清理、缺加密密钥、写入失败原子回滚、多连接领取、跨用途预算、统一错误、密文篡改、成功回执保存失败不重发、双线程无队列和 2 秒锁超时。测试用专用合成账户和内存 sender,真实网络只用于本地应用健康检查,没有邮件/短信付费调用。
最终后端命令:`mvn -B -o clean verify`,Java 21 / Maven 3.9.9、独立 PostgreSQL 15.18/Redis 7。**390 项,0 失败、0 错误、0 跳过,JAR 构建成功**,结束于 2026-10-22 08:41:38(Asia/Shanghai)。新增迁移和全部既有认证/MFA/验证码事务/媒体出站/账务/任务回归均参与。日志 `.tools/verification-delivery-final.log` 被忽略。
前端重新运行 `npm run build` 与 `npm test`:构建通过、25 项测试通过。另在本轮独立容器完整启动打包 JAR,Flyway 迁移至 V50,Spring 组件和定时方法成功初始化,`/health` 返回正常;未配置真实邮件/短信,`/api/auth/verification` 返回两通道均关闭,符合安全关闭状态。这只是本地启动冒烟,不替代真实投递或产品验收。
`spec-driven-acceptance-testing` 的共享 Python 环境阻塞仍未解除;本轮没有获授权修改其环境,未自行批准规格,也没有执行或声称 `qa run --project` 通过。仍须用户审阅 draft 后走正式验收。
参考原则:`https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html`。该文档是设计参考,不是对本产品的认证或第三方审计结果。
## 自动化与后续
按用户本轮明确要求,已删除聊天内 `moyi` 每半小时本地提交检查自动化。没有删除产品内部任务/投递 worker。当前按 `git-commit-conventions` 审查完整差异和暂存区,只提交本任务已验证文件,不 push。后续按 24 号表继续管理授权、配置版本、请求传输统一、长请求租约和协议能力;不把本次增量称作全部开发完成。
+1
View File
@@ -47,6 +47,7 @@ GATEWAY_READS_PER_MINUTE=300
GATEWAY_DEFAULT_MODEL_CONCURRENCY=0
GATEWAY_DEFAULT_KEY_CONCURRENCY=4
GATEWAY_DISABLE_MODEL_CONCURRENCY=false
GATEWAY_RISK_MODE=shadow
GATEWAY_QUEUE_WAIT_MS=30000
GATEWAY_QUEUE_POLL_MS=100
TASK_MAX_QUEUED_PER_MODEL=100
Loaded 100 of 121 files, more files were not shown because too many files have changed in this diff. Show more