Amazon Web Services: Performing a Source Code Security Scan Using git-secrets in AWS
EC2 environment. In this tutorial you will install Git, clone a repository, install `git-secrets`, and scan for a vulnerability which has been deliberately introduced into the code.
After entering into a SSH session of your server you will need to update the server first.
Last login: Tue Sep 24 14:07:23 on ttys000
austinsonger@Songer ~ % ssh cloud_user@35.172.109.218
The authenticity of host '35.172.109.218 (35.172.109.218)' can't be established.
ECDSA key fingerprint is SHA256:7koLywItBCNqYYmWN3n/r7fHm8UJpk3DmGKqmQ+RMvg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '35.172.109.218' (ECDSA) to the list of known hosts.
Password:
__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
3 package(s) needed for security, out of 7 available
Run "sudo yum update" to apply all updates.
[cloud_user@ip-10-0-1-15 ~]$ sudo yum update
[sudo] password for cloud_user:
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main | 2.1 kB 00:00:00
amzn-updates | 2.5 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:4.14.143-91.122.amzn1 will be installed
---> Package kernel-tools.x86_64 0:4.14.138-89.102.amzn1 will be updated
---> Package kernel-tools.x86_64 0:4.14.143-91.122.amzn1 will be an update
---> Package libblkid.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libblkid.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package libjpeg-turbo.x86_64 0:1.2.90-5.14.amzn1 will be updated
---> Package libjpeg-turbo.x86_64 0:1.2.90-8.16.amzn1 will be an update
---> Package libmount.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libmount.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package libuuid.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libuuid.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package util-linux.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package util-linux.x86_64 0:2.23.2-59.29.amzn1 will be an update
--> Processing Dependency: libsmartcols = 2.23.2-59.29.amzn1 for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Processing Dependency: libsmartcols.so.1(SMARTCOLS_2.25)(64bit) for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Processing Dependency: libsmartcols.so.1()(64bit) for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Running transaction check
---> Package libsmartcols.x86_64 0:2.23.2-59.29.amzn1 will be installed
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:4.14.97-74.72.amzn1 will be erased
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================================
Package Arch Version Repository Size
==========================================================================================
Installing:
kernel x86_64 4.14.143-91.122.amzn1 amzn-updates 22 M
Updating:
kernel-tools x86_64 4.14.143-91.122.amzn1 amzn-updates 133 k
libblkid x86_64 2.23.2-59.29.amzn1 amzn-updates 187 k
libjpeg-turbo x86_64 1.2.90-8.16.amzn1 amzn-updates 144 k
libmount x86_64 2.23.2-59.29.amzn1 amzn-updates 192 k
libuuid x86_64 2.23.2-59.29.amzn1 amzn-updates 84 k
util-linux x86_64 2.23.2-59.29.amzn1 amzn-updates 3.1 M
Removing:
kernel x86_64 4.14.97-74.72.amzn1 @amzn-updates 90 M
Installing for dependencies:
libsmartcols x86_64 2.23.2-59.29.amzn1 amzn-updates 146 k
Transaction Summary
==========================================================================================
Install 1 Package (+1 Dependent package)
Upgrade 6 Packages
Remove 1 Package
Total download size: 26 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): libblkid-2.23.2-59.29.amzn1.x86_64.rpm | 187 kB 00:00:00
(2/8): kernel-tools-4.14.143-91.122.amzn1.x86_64.rpm | 133 kB 00:00:00
(3/8): libjpeg-turbo-1.2.90-8.16.amzn1.x86_64.rpm | 144 kB 00:00:00
(4/8): libmount-2.23.2-59.29.amzn1.x86_64.rpm | 192 kB 00:00:00
(5/8): libsmartcols-2.23.2-59.29.amzn1.x86_64.rpm | 146 kB 00:00:00
(6/8): libuuid-2.23.2-59.29.amzn1.x86_64.rpm | 84 kB 00:00:00
(7/8): util-linux-2.23.2-59.29.amzn1.x86_64.rpm | 3.1 MB 00:00:00
(8/8): kernel-4.14.143-91.122.amzn1.x86_64.rpm | 22 MB 00:00:15
------------------------------------------------------------------------------------------
Total 1.7 MB/s | 26 MB 00:00:15
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : libuuid-2.23.2-59.29.amzn1.x86_64 1/15
Updating : libblkid-2.23.2-59.29.amzn1.x86_64 2/15
Updating : libmount-2.23.2-59.29.amzn1.x86_64 3/15
Installing : libsmartcols-2.23.2-59.29.amzn1.x86_64 4/15
Updating : util-linux-2.23.2-59.29.amzn1.x86_64 5/15
Updating : libjpeg-turbo-1.2.90-8.16.amzn1.x86_64 6/15
Installing : kernel-4.14.143-91.122.amzn1.x86_64 7/15
Updating : kernel-tools-4.14.143-91.122.amzn1.x86_64 8/15
Cleanup : util-linux-2.23.2-33.28.amzn1.x86_64 9/15
Cleanup : kernel-4.14.97-74.72.amzn1.x86_64 10/15
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.order: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.networking: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.modesetting: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.drm: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.builtin: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.block: remove failed: No such file or directory
Cleanup : libmount-2.23.2-33.28.amzn1.x86_64 11/15
Cleanup : libblkid-2.23.2-33.28.amzn1.x86_64 12/15
Cleanup : libuuid-2.23.2-33.28.amzn1.x86_64 13/15
Cleanup : libjpeg-turbo-1.2.90-5.14.amzn1.x86_64 14/15
Cleanup : kernel-tools-4.14.138-89.102.amzn1.x86_64 15/15
intel: model '', path ' intel-ucode/*', kvers ''
intel: blacklist ''
intel-06-4f-01: model 'GenuineIntel 06-4f-01', path ' intel-ucode/06-4f-01', kvers ' 4.14.42'
intel-06-4f-01: blacklist ''
Verifying : util-linux-2.23.2-59.29.amzn1.x86_64 1/15
Verifying : kernel-tools-4.14.143-91.122.amzn1.x86_64 2/15
Verifying : libsmartcols-2.23.2-59.29.amzn1.x86_64 3/15
Verifying : libblkid-2.23.2-59.29.amzn1.x86_64 4/15
Verifying : libmount-2.23.2-59.29.amzn1.x86_64 5/15
Verifying : libuuid-2.23.2-59.29.amzn1.x86_64 6/15
Verifying : kernel-4.14.143-91.122.amzn1.x86_64 7/15
Verifying : libjpeg-turbo-1.2.90-8.16.amzn1.x86_64 8/15
Verifying : libjpeg-turbo-1.2.90-5.14.amzn1.x86_64 9/15
Verifying : libmount-2.23.2-33.28.amzn1.x86_64 10/15
Verifying : libblkid-2.23.2-33.28.amzn1.x86_64 11/15
Verifying : libuuid-2.23.2-33.28.amzn1.x86_64 12/15
Verifying : kernel-tools-4.14.138-89.102.amzn1.x86_64 13/15
Verifying : util-linux-2.23.2-33.28.amzn1.x86_64 14/15
Verifying : kernel-4.14.97-74.72.amzn1.x86_64 15/15
Removed:
kernel.x86_64 0:4.14.97-74.72.amzn1
Installed:
kernel.x86_64 0:4.14.143-91.122.amzn1
Dependency Installed:
libsmartcols.x86_64 0:2.23.2-59.29.amzn1
Updated:
kernel-tools.x86_64 0:4.14.143-91.122.amzn1 libblkid.x86_64 0:2.23.2-59.29.amzn1
libjpeg-turbo.x86_64 0:1.2.90-8.16.amzn1 libmount.x86_64 0:2.23.2-59.29.amzn1
libuuid.x86_64 0:2.23.2-59.29.amzn1 util-linux.x86_64 0:2.23.2-59.29.amzn1
Complete!
Install Git
sudo yum install git -y
Output
[cloud_user@ip-10-0-1-15 ~]$ sudo yum install git -y Loaded plugins: priorities, update-motd, upgrade-helper Resolving Dependencies --> Running transaction check ---> Package git.x86_64 0:2.14.5-1.60.amzn1 will be installed --> Processing Dependency: perl-Git = 2.14.5-1.60.amzn1 for package: git-2.14.5-1.60.amzn1.x86_64 --> Processing Dependency: perl(Term::ReadKey) for package: git-2.14.5-1.60.amzn1.x86_64 --> Processing Dependency: perl(Git::I18N) for package: git-2.14.5-1.60.amzn1.x86_64 --> Processing Dependency: perl(Git) for package: git-2.14.5-1.60.amzn1.x86_64 --> Processing Dependency: perl(Error) for package: git-2.14.5-1.60.amzn1.x86_64 --> Running transaction check ---> Package perl-Error.noarch 1:0.17020-2.9.amzn1 will be installed ---> Package perl-Git.noarch 0:2.14.5-1.60.amzn1 will be installed ---> Package perl-TermReadKey.x86_64 0:2.30-20.9.amzn1 will be installed --> Finished Dependency Resolution Dependencies Resolved ========================================================================================== Package Arch Version Repository Size ========================================================================================== Installing: git x86_64 2.14.5-1.60.amzn1 amzn-updates 12 M Installing for dependencies: perl-Error noarch 1:0.17020-2.9.amzn1 amzn-main 33 k perl-Git noarch 2.14.5-1.60.amzn1 amzn-updates 69 k perl-TermReadKey x86_64 2.30-20.9.amzn1 amzn-main 33 k Transaction Summary ========================================================================================== Install 1 Package (+3 Dependent packages) Total download size: 12 M Installed size: 29 M Downloading packages: (1/4): perl-Error-0.17020-2.9.amzn1.noarch.rpm | 33 kB 00:00:00 (2/4): perl-TermReadKey-2.30-20.9.amzn1.x86_64.rpm | 33 kB 00:00:00 (3/4): perl-Git-2.14.5-1.60.amzn1.noarch.rpm | 69 kB 00:00:00 (4/4): git-2.14.5-1.60.amzn1.x86_64.rpm | 12 MB 00:00:01 ------------------------------------------------------------------------------------------ Total 6.3 MB/s | 12 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : 1:perl-Error-0.17020-2.9.amzn1.noarch 1/4 Installing : perl-TermReadKey-2.30-20.9.amzn1.x86_64 2/4 Installing : perl-Git-2.14.5-1.60.amzn1.noarch 3/4 Installing : git-2.14.5-1.60.amzn1.x86_64 4/4 Verifying : git-2.14.5-1.60.amzn1.x86_64 1/4 Verifying : 1:perl-Error-0.17020-2.9.amzn1.noarch 2/4 Verifying : perl-Git-2.14.5-1.60.amzn1.noarch 3/4 Verifying : perl-TermReadKey-2.30-20.9.amzn1.x86_64 4/4 Installed: git.x86_64 0:2.14.5-1.60.amzn1 Dependency Installed: perl-Error.noarch 1:0.17020-2.9.amzn1 perl-Git.noarch 0:2.14.5-1.60.amzn1 perl-TermReadKey.x86_64 0:2.30-20.9.amzn1 Complete!
Clone Github Repoistory
Type the following in the terminal
git clone https://github.com/austinsonger/aws-security
Output
[cloud_user@ip-10-0-1-15 ~]$ git clone https://github.com/austinsonger/aws-security Cloning into 'aws-security'... remote: Enumerating objects: 128, done. remote: Counting objects: 100% (128/128), done. remote: Compressing objects: 100% (69/69), done. remote: Total 128 (delta 38), reused 128 (delta 38), pack-reused 0 Receiving objects: 100% (128/128), 1.20 MiB | 37.29 MiB/s, done. Resolving deltas: 100% (38/38), done.
Install git-secrets
Follow the following steps
git clone https://github.com/awslabs/git-secrets.git cd git-secrets sudo make install git secrets cd ..
Output
[cloud_user@ip-10-0-1-15 ~]$ git clone https://github.com/awslabs/git-secrets.git
Cloning into 'git-secrets'...
remote: Enumerating objects: 341, done.
remote: Total 341 (delta 0), reused 0 (delta 0), pack-reused 341
Receiving objects: 100% (341/341), 109.22 KiB | 13.65 MiB/s, done.
Resolving deltas: 100% (194/194), done.
[cloud_user@ip-10-0-1-15 ~]$ cd git-secrets
[cloud_user@ip-10-0-1-15 git-secrets]$ sudo make install
[cloud_user@ip-10-0-1-15 git-secrets]$ git secrets
usage: git secrets --scan [-r|--recursive] [--cached] [--no-index] [--untracked] [<files>...]
or: git secrets --scan-history
or: git secrets --install [-f|--force] [<target-directory>]
or: git secrets --list [--global]
or: git secrets --add [-a|--allowed] [-l|--literal] [--global] <pattern>
or: git secrets --add-provider [--global] <command> [arguments...]
or: git secrets --register-aws [--global]
or: git secrets --aws-provider [<credentials-file>]
--scan Scans <files> for prohibited patterns
--scan-history Scans repo for prohibited patterns
--install Installs git hooks for Git repository or Git template directory
--list Lists secret patterns
--add Adds a prohibited or allowed pattern, ensuring to de-dupe with existing patterns
--add-provider Adds a secret provider that when called outputs secret patterns on new lines
--aws-provider Secret provider that outputs credentials found in an ini file
--register-aws Adds common AWS patterns to the git config and scans for ~/.aws/credentials
-r, --recursive --scan scans directories recursively
--cached --scan scans searches blobs registered in the index file
--no-index --scan searches files in the current directory that is not managed by Git
--untracked In addition to searching in the tracked files in the working tree, --scan also in untracked files
-f, --force --install overwrites hooks if the hook already exists
-l, --literal --add and --add-allowed patterns are escaped so that they are literal
-a, --allowed --add adds an allowed pattern instead of a prohibited pattern
--global Uses the --global git config
[cloud_user@ip-10-0-1-15 git-secrets]$ cd ..
[cloud_user@ip-10-0-1-15 ~]$ ls
aws-security git-secrets
Scan Repository
Change to the repository directory:
Follow these steps
cd aws-security
cd git_secrets_activity
Output
[cloud_user@ip-10-0-1-15 ~]$ cd aws-security [cloud_user@ip-10-0-1-15 aws-security]$ ls 311 - CloudFront 312 - restrict to s3 316 - CRR Automatic-Resource-Remediation-with-AWS-Config Enabling-VPC-Flow-Logs-with-Automation git_secrets_activity README.md S3Events Troubleshooting-Detection-Alerting-Response-Workflow Troubleshooting-Logging-with-CloudTrail-and-S3 [cloud_user@ip-10-0-1-15 aws-security]$ cd git_secrets_activity [cloud_user@ip-10-0-1-15 git_secrets_activity]$ ls example.sh
Register the AWS rule set (the “Git hooks”):
Follow these steps
git secrets --register-aws
Output
[cloud_user@ip-10-0-1-15 git_secrets_activity]$ git secrets --register-aws OK
Scan the repository:
Follow these steps
git secrets --scan
Output
[cloud_user@ip-10-0-1-15 git_secrets_activity]$ git secrets --scan example.sh:4:AWS_SECRET_ACCESS_KEY = Z3ofnVlFTH9DFmulF3uDO7BCDxGYD4nIG92oeymX [ERROR] Matched one or more prohibited patterns Possible mitigations: - Mark false positives as allowed using: git config --add secrets.allowed ... - Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory - List your configured patterns: git config --get-all secrets.patterns - List your configured allowed patterns: git config --get-all secrets.allowed - List your configured allowed patterns in .gitallowed at repository's root directory - Use --no-verify if this is a one-time false positive