Detecting Secrets in a Git Repository for Beginners

The best tools and script to search threw git history

Detecting Secrets in a Git Repository for Beginners

Introduction

Discovering secrets like API keys, passwords, or access tokens in a Git repository is crucial for security. Leaked secrets can lead to unauthorized access and potential data breaches. We’ll explore three tools to scan your Git repository for secrets: Gitleaks, TruffleHog, and Detect Secrets. All can be installed with Homebrew.

The Importance of Finding Secrets in Git History

Secrets in Git history pose risks:

1. Persistence of Data: Secrets remain in history even after removal in a later commit.
2. Exposure during Cloning and Forking: Secrets are copied during cloning or forking.
3. Security Risks: Malicious actors can exploit secrets if they access the repository.
4. Hard to Mitigate: Removing secrets from history is complex.
5. Compliance and Audits: Secrets in history can lead to compliance issues.
6. Reputation Damage: Leaked secrets can damage an organization’s reputation.

Finding and Removing Secrets

Address risks by finding/removing secrets from both current code and Git history. Use tools like BFG Repo-Cleaner or git filter-branch for history, but coordinate with collaborators due to history rewrite.

Proactive Measures

Prevent secrets from being committed:

• Education: Teach about risks and best practices.
• Pre-commit Hooks: Use tools like pre-commit to scan before committing.
• Environment Variables: Use them or secret management tools instead of hardcoding.
• Regular Audits: Conduct them to check for secrets.

Gitleaks

Installation: brew install gitleaks

Navigate to your project directory and run:

gitleaks detect --source=. --config-path=../gitleaks.toml --report-format=json

Create a gitleaks.toml file for custom rules and exclusions.

[allowlist]
  description = "Exclude files in icons directories"
  files = [
    "(?i)icons/.*",
  ]t

TruffleHog

Installation: brew install trufflehog

Navigate to your project directory and run:

trufflehog git file://. --exclude_paths exclude-patterns.txt --json

Create an exclude-patterns.txt file for path exclusions.

.*\site-config.json
.*\.ttf$
.*\.eot$
.env

Detect Secrets

Installation: brew install detect-secrets

Navigate to your project directory and run:

detect-secrets scan --exclude-files '.*icons/.*' --disable-plugin Base64HighEntropyString --disable-plugin HexHighEntropyString

I disable Base64HighEntropyString and HexHighEntropyString as I think there is too much false positive.

Bonus: Searching for Specific Keywords in Git History (without tools)

Manually search for secret-related keywords:

git rev-list --all | (
  while read revision; do
    git grep -E 'password|private_key' $revision # you can change keywords here
  done
)

Conclusion

This is a basic approach, you will get a lot of results and I recommend to check them one by one. Read the docs to each tools see how to put the false positive in verify state.

If you want to use only one tool, I would recommend TruffleHog.