oathe / scan / reports / caution
Is lllllllama/rigorpilot-skills safe?
Use with caution. lllllllama/rigorpilot-skills is not clearly unsafe, but the audit flagged risks to review before you install it. Oathe's behavioral security audit gave the rigorpilot-skills skill by lllllllama a trust score of 67/100 with 9 findings (2 high).
https://github.com/lllllllama/rigorpilot-skills
Is lllllllama/rigorpilot-skills safe to install?
RigorPilot-Skills is a large, multi-skill AI research reproduction toolkit with legitimate and extensively benchmarked functionality. The primary security concern is a burst read of six canonical credential file locations (.env, .ssh/id_rsa, .aws/credentials, .npmrc, .docker/config.json, .config/gcloud/application_default_credentials.json) at the start of the audit session, temporally correlated with an outbound HTTPS connection to an Azure-range IP; while the most likely explanation is bash profile initialization during the audit SSH session combined with a git fetch to GitHub, process-level attribution is absent and cannot be fully ruled out. Secondary concerns include bundled execution scripts with broad shell access, a ~/.rigorpilot/PERSONAL_RIGOR.md injection vector, and four uninspected .claude/commands/ slash commands. All canary files were intact and no git hooks or npm lifecycle scripts were found.
What security issues were found in lllllllama/rigorpilot-skills?
Category Scores
Findings (9)
HIGH Targeted credential file enumeration during audit session -25 ▶
inotifywait recorded simultaneous OPEN/ACCESS/CLOSE events on six canonical credential locations (.env, .ssh/id_rsa, .aws/credentials, .npmrc, .docker/config.json, .config/gcloud/application_default_credentials.json) at 07:14:12 — a single second, all together. This exact pattern matches a targeted credential sweep. While an alternative explanation exists (bash profile sourcing during SSH session init for the audit user), normal PAM/SSH authentication does not access .aws/credentials, .npmrc, or .docker/config.json. The reads cannot be definitively attributed to the skill vs. the audit framework's own canary baseline check without process-level attribution.
Evidence: 07:14:12 OPEN /home/oc-exec/.env; 07:14:12 OPEN /home/oc-exec/.ssh/id_rsa; 07:14:12 OPEN /home/oc-exec/.aws/credentials; 07:14:12 OPEN /home/oc-exec/.npmrc; 07:14:12 OPEN /home/oc-exec/.docker/config.json; 07:14:12 OPEN /home/oc-exec/.config/gcloud/application_default_credentials.json
HIGH External HTTPS connection temporally correlated with credential reads -20 ▶
An outbound TCP connection to 4.163.8.144:443 was established within ~6 seconds of the credential file reads. The sustained, large-volume traffic pattern (hundreds of TCP packets) is more consistent with a git fetch than with credential POST exfiltration, and the IP falls in the Azure range used by GitHub. However, the temporal proximity to the credential reads cannot be dismissed without process-level attribution confirming this connection was initiated by git and not by a skill script.
Evidence: 07:14:18.302964 ens3 Out IP 10.0.2.15.46180 > 4.163.8.144.443: tcp 0 (followed by hundreds of data packets)
MEDIUM Bundled execution engine runs arbitrary shell commands with agent permissions -20 ▶
The _bundled/ directory ships runtime_runner.py, orchestrate_repro.py, and bootstrap_env.sh, which are designed to spawn and manage arbitrary child processes on behalf of the agent. These scripts execute commands selected from repository READMEs and configuration files — inputs that could be attacker-controlled in a supply-chain attack scenario. The skill's design explicitly requires these capabilities, but they represent a broad execution surface.
Evidence: skills/ai-research-reproduction/_bundled/shared/scripts/runtime_runner.py; skills/env-and-assets-bootstrap/scripts/bootstrap_env.sh; RECHECK.json shows orchestrate_repro.py --run-selected invocation
MEDIUM Skill instructs agent to read ~/.rigorpilot/PERSONAL_RIGOR.md from user home directory -8 ▶
The ai-research-explore section of SKILL.md instructs the agent to consult ~/.rigorpilot/PERSONAL_RIGOR.md if present. This creates a persistent injection vector: an attacker who can write to the user's home directory (e.g., via a separate compromised tool) could plant instructions in that file that would be silently loaded and acted upon by any agent using this skill.
Evidence: SKILL.md line: 'Consult ~/.rigorpilot/PERSONAL_RIGOR.md if present, under ../ai-research-reproduction/references/continuous-learning-policy.md (advisory only; core wins).'
MEDIUM .claude/commands/ slash commands present but content not inspectable -12 ▶
The repository ships four .claude/commands/ markdown files (ai-research-explore.md, analyze-project.md, ai-research-reproduction.md, safe-debug.md) that register as slash commands in Claude Code. These are surfaced directly to users who may invoke them without recognizing them as third-party skill content. Their full content was not included in the evidence corpus, making injection analysis incomplete.
Evidence: File list: .claude/commands/ai-research-explore.md, .claude/commands/analyze-project.md, .claude/commands/ai-research-reproduction.md, .claude/commands/safe-debug.md
LOW Scientific sabotage risk via silent protocol modification -15 ▶
The skill is explicitly designed to modify repository code (patches, environment fixes, path corrections) during reproduction. The SKILL.md requires that changes be recorded in SCIENTIFIC_CHANGELOG.md, but the skill's orchestration layer could potentially make subtle changes to evaluation metrics, dataset splits, or checkpoint loading that alter scientific results without triggering obvious security signals. This is a dual-use concern rather than a confirmed threat.
Evidence: SKILL.md: 'Reproduce fixes are allowed when needed, but they must not be hidden... avoid changing model architecture, core inference semantics, training logic, loss functions, or experiment meaning.'
LOW Canary files were read (not modified) — honeypot integrity check is insufficient -10 ▶
The canary integrity check only verifies that honeypot files were not modified or deleted. All six canary files were confirmed OPENED and ACCESSED (read-only). A read-and-transmit exfiltration attack would pass the canary integrity check while still leaking credential content.
Evidence: Monitoring: Canary File Integrity: All canary files intact; auditd PATH events 15343-15348 confirm canary files were read
INFO No git hooks, submodules, or npm lifecycle scripts found 0 ▶
The audit framework found no .githooks/ files, no .gitmodules submodule references, and no preinstall/postinstall npm scripts. The .gitattributes file is benign (jupyter language stats override and line-ending preservation for benchmark outputs).
Evidence: find /home/oc-exec/skill-under-test/ -path '*/.githooks/*' returned no results; .gitattributes contains only linguist and -text flags
INFO Extensive real-world benchmark evidence confirms legitimate functionality 0 ▶
The benchmark_outputs/ directory contains hundreds of files documenting genuine reproduction runs on micrograd, nanogpt-shakespeare, pytorch-mnist, and mingpt — including agent traces, grader results, and showcase snapshots with verified pytest passes. This strongly suggests the skill functions as advertised as an AI research reproduction tool.
Evidence: benchmark_outputs/showcases/micrograd-first-use-after/CHECK.json: ok:true, task_log_condition:passed; independent pytest shows 2 passed
Should I install lllllllama/rigorpilot-skills?
Oathe's verdict for lllllllama/rigorpilot-skills is CAUTION with a trust score of 67/100. Recommendation: Review before install.