mirror of
https://github.com/Momoko-Ayase/Senbei.git
synced 2026-09-19 03:57:59 -04:00
refactor: consolidate platform engines into senbei-engine
This commit is contained in:
bfloat16
+34
-99
@@ -2,125 +2,60 @@
|
||||
|
||||
## Building
|
||||
|
||||
Requires a Rust toolchain (MSVC backend is the default on Windows;
|
||||
`rustup-init.exe` from <https://rustup.rs> installs it). The pinned toolchain
|
||||
and targets are in `rust-toolchain.toml`.
|
||||
The pinned Rust toolchain is defined in `rust-toolchain.toml`. Build the CLI with `cargo build --release`; the binary is written to `target/release/senbei.exe` on Windows.
|
||||
|
||||
```cmd
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
Output: `target\release\senbei.exe`. The binary is self-contained — no driver,
|
||||
no proxy DLL, no external assets.
|
||||
|
||||
The library and CLI also build for Linux/macOS (`cfg`-gated platform code
|
||||
only) and for `wasm32-unknown-unknown` (see the [web version](../web/README.md)).
|
||||
The workspace crates are portable where their APIs are pure. The browser binding is outside the workspace and is checked with `cargo check --manifest-path senbei-wasm/Cargo.toml` or built with `wasm-pack`.
|
||||
|
||||
## Testing
|
||||
|
||||
```cmd
|
||||
cargo test --release
|
||||
cargo test --release --workspace
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
cargo fmt --all -- --check
|
||||
```
|
||||
|
||||
The suite covers CLI behavior, detection, the folder driver, the run log, and
|
||||
byte-exact golden tests over `samples/` — a user-managed corpus (git-ignored,
|
||||
see `samples/README.md`) of real Crackproof inputs plus `<base>.golden.<ext>`
|
||||
reference outputs. Every input goes through `job::unpack_bytes` — the same
|
||||
routing the CLI uses, so an `<input>._` companion in the corpus is spliced and
|
||||
the stub export/TLS overlays run — and is gated on **two** checks: the static
|
||||
integrity check (catches runtime-broken outputs even when a stale golden would
|
||||
still byte-match) and, when a golden exists, a bit-for-bit comparison. il2cpp
|
||||
`*.dat` inputs are routed through `metadata::deobfuscate` instead. An empty or
|
||||
absent corpus is a no-op pass; set `SENBEI_REQUIRE_SAMPLES` to make it fail
|
||||
instead (useful on a private CI that has the corpus — public CI never does,
|
||||
since binaries are not committed).
|
||||
The tracked test suite is safe without protected samples. The optional local `samples/` corpus is user-managed and the ignored `test/` folder can be used for real Windows and Android runs.
|
||||
|
||||
> **Note:** goldens encode expected *bytes*, not runtime behavior. A golden
|
||||
> produced before a pipeline fix may byte-match while still being wrong — the
|
||||
> integrity check is the second gate for exactly this reason. Re-verify
|
||||
> goldens against real runs when touching the affected pipeline stages.
|
||||
>
|
||||
> **The corpus only protects what it contains.** Wire the test to the routing
|
||||
> the CLI actually takes (it is), and keep a sample for every layout family —
|
||||
> marker-based, marker-less, external-companion, PE32, PE32+, native, managed,
|
||||
> metadata. An unrepresented family has no regression gate at all, which is
|
||||
> how a "re-run the golden corpus" rule can pass while silently covering
|
||||
> nothing.
|
||||
For an Android package, use one command at a time because a protected `.so` can be hundreds of megabytes. APK, APKS, and XAPK tests read the ZIP manifest first and extract only `.so` and `global-metadata.dat` entries.
|
||||
|
||||
## Debugging levers (environment variables)
|
||||
## Environment Variables
|
||||
|
||||
- `DD8_SHIFT` — override the `decrypt_data8` page-XOR shift (`99` skips dd8
|
||||
entirely).
|
||||
- `SEL_DIAG` — print the dd8 selector's scores: the per-shift `0xCC` counts and
|
||||
the plaintext baseline they are compared against (PE32+), and the per-formula
|
||||
counts, baseline and net gain (PE32).
|
||||
- `SENBEI_THREADS` — cap the block-parallel fan-out (`1` forces the fully
|
||||
sequential path).
|
||||
- `SENBEI_SCAN_ALL` — same as `--scan-all` (probe every file in a folder).
|
||||
- `SENBEI_ANDROID_SAMPLES` — override the Android corpus location (default
|
||||
`samples/android/`; see `samples/README.md`). The Android corpus test pins
|
||||
restored outputs with SHA-256 sidecar files next to each protected input
|
||||
and documents known restore gaps with empty `<base>.restore-fails` markers.
|
||||
- `DD8_SHIFT` overrides the PE page-XOR shift; `99` skips that stage.
|
||||
- `SEL_DIAG` prints PE layout-selector diagnostics.
|
||||
- `SENBEI_THREADS` caps deterministic block fan-out; `1` forces the sequential reference path.
|
||||
- `SENBEI_SCAN_ALL` enables the explicit scan-all mode for selected target names.
|
||||
- `SENBEI_ANDROID_SAMPLES` overrides the Android sample corpus location.
|
||||
|
||||
## Conventions
|
||||
|
||||
- The `senbei-pe/` core (and its `senbei-crypto/` base) is pure: no file I/O,
|
||||
no panics across the public boundary, no `unsafe`. Keep it that way — it is
|
||||
what the WebAssembly build embeds.
|
||||
- Layout heuristics must **trial-and-validate**: never pick a candidate offset
|
||||
on shape alone and trust it; validate by decryption/checksum and fall
|
||||
through to the next candidate on failure. A silent wrong offset produces a
|
||||
silently broken output, which is worse than an error.
|
||||
- Output must remain byte-identical against the golden corpus for every
|
||||
supported layout. When fixing one build family, re-run the full golden
|
||||
corpus to prove no other family regressed.
|
||||
- Folder scanning uses a size floor plus an extension **deny**-list, never an
|
||||
allow-list: targets are recognised by content, not extension, and can carry
|
||||
arbitrary names, so only known bulk-asset extensions are excluded. The
|
||||
pre-filter exists because folder-scan cost is per-file I/O latency, not the
|
||||
walk — probe fewer files, don't parallelize the probe loop.
|
||||
- `cargo fmt` and `cargo clippy` must stay clean (CI enforces both).
|
||||
Format crates stay free of filesystem I/O and protection-specific logic. Windows engine code lives below `senbei-engine/src/windows/`, Android engine code below `senbei-engine/src/android/`, and shared code stays directly under each crate's `src/`.
|
||||
|
||||
## Repository layout
|
||||
Layout heuristics must trial and validate every candidate. A failed validation is an error or a fall-through, never a silently accepted offset.
|
||||
|
||||
```
|
||||
senbei/
|
||||
├── Cargo.toml workspace root (members: the senbei-* crates)
|
||||
├── rust-toolchain.toml pinned toolchain + targets
|
||||
├── senbei-cli/ senbei binary (default member)
|
||||
│ └── tests/ CLI, detection, golden, and folder tests
|
||||
├── senbei-pe/ pure unpacker core (see docs/design.md)
|
||||
├── senbei-crypto/ crypto/compression primitives
|
||||
├── senbei-metadata/ il2cpp metadata de-obfuscation
|
||||
├── senbei-io/ filesystem, scanning, CLI orchestration
|
||||
├── senbei-wasm/ WebAssembly bindings crate (own Cargo.lock,
|
||||
│ outside the workspace; builds into web/pkg/)
|
||||
├── samples/ local-only test corpus (git-ignored)
|
||||
├── web/ static browser frontend assets (+ built pkg/)
|
||||
├── docs/ usage, design, and development documentation
|
||||
└── .github/ CI workflows and issue templates
|
||||
Outputs must remain byte-identical against the available golden corpus. Run the full workspace tests after changing a pipeline or a metadata layout.
|
||||
|
||||
Folder scanning uses explicit target names to avoid opening bulk assets. External `.exe._` and `.dll._` files are auxiliary data for their sibling stubs and are not independent scan targets.
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```text
|
||||
senbei-cli/ command-line binary and integration tests
|
||||
senbei-crypto/ shared crypto and Android crypto primitives
|
||||
senbei-elf/ basic ELF parsing
|
||||
senbei-engine/ Windows and Android unpacking engines
|
||||
senbei-io/ filesystem, package, scanning, and CLI orchestration
|
||||
senbei-metadata/ Windows and Android metadata restoration
|
||||
senbei-pe/ basic PE parsing
|
||||
senbei-wasm/ browser bindings and its own lockfile
|
||||
web/ static browser frontend
|
||||
samples/ optional local corpus
|
||||
```
|
||||
|
||||
## Web build
|
||||
|
||||
See [web/README.md](../web/README.md). In short:
|
||||
## Web Build
|
||||
|
||||
```cmd
|
||||
cd senbei-wasm
|
||||
wasm-pack build --target web --release --out-dir ../web/pkg
|
||||
```
|
||||
|
||||
then serve `web/` statically and open `index.html`. Everything runs
|
||||
client-side; no file leaves the browser.
|
||||
|
||||
## Contributing
|
||||
|
||||
Issues and pull requests are welcome. A few ground rules:
|
||||
|
||||
- **Never commit binaries** (protected or decrypted) to the repository —
|
||||
the only corpus is the local git-ignored `samples/`. Attaching a protected
|
||||
input file to an issue is welcome if it helps diagnose the problem; only
|
||||
attach files you are authorized to share.
|
||||
- Run `cargo test --release`, `cargo clippy`, and `cargo fmt` before
|
||||
submitting.
|
||||
- Keep the unpacker core free of I/O, `unsafe`, and platform-specific code.
|
||||
Serve `web/` with a static HTTP server after the build. The browser never uploads input files.
|
||||
|
||||
Reference in New Issue
Block a user