Amazon Web Services: Performing a Source Code Security Scan Using git-secrets in AWS

Amazon Web Services: Performing a Source Code Security Scan Using git-secrets in AWS

EC2 environment. In this tutorial you will install Git, clone a repository, install `git-secrets`, and scan for a vulnerability which has been deliberately introduced into the code.

After entering into a SSH session of your server you will need to update the server first.

Last login: Tue Sep 24 14:07:23 on ttys000
austinsonger@Songer ~ % ssh cloud_user@35.172.109.218
The authenticity of host '35.172.109.218 (35.172.109.218)' can't be established.
ECDSA key fingerprint is SHA256:7koLywItBCNqYYmWN3n/r7fHm8UJpk3DmGKqmQ+RMvg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '35.172.109.218' (ECDSA) to the list of known hosts.
Password: 

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
3 package(s) needed for security, out of 7 available
Run "sudo yum update" to apply all updates.
[cloud_user@ip-10-0-1-15 ~]$ sudo yum update
[sudo] password for cloud_user: 
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                          | 2.1 kB  00:00:00     
amzn-updates                                                       | 2.5 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:4.14.143-91.122.amzn1 will be installed
---> Package kernel-tools.x86_64 0:4.14.138-89.102.amzn1 will be updated
---> Package kernel-tools.x86_64 0:4.14.143-91.122.amzn1 will be an update
---> Package libblkid.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libblkid.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package libjpeg-turbo.x86_64 0:1.2.90-5.14.amzn1 will be updated
---> Package libjpeg-turbo.x86_64 0:1.2.90-8.16.amzn1 will be an update
---> Package libmount.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libmount.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package libuuid.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libuuid.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package util-linux.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package util-linux.x86_64 0:2.23.2-59.29.amzn1 will be an update
--> Processing Dependency: libsmartcols = 2.23.2-59.29.amzn1 for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Processing Dependency: libsmartcols.so.1(SMARTCOLS_2.25)(64bit) for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Processing Dependency: libsmartcols.so.1()(64bit) for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Running transaction check
---> Package libsmartcols.x86_64 0:2.23.2-59.29.amzn1 will be installed
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:4.14.97-74.72.amzn1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================
 Package             Arch         Version                       Repository           Size
==========================================================================================
Installing:
 kernel              x86_64       4.14.143-91.122.amzn1         amzn-updates         22 M
Updating:
 kernel-tools        x86_64       4.14.143-91.122.amzn1         amzn-updates        133 k
 libblkid            x86_64       2.23.2-59.29.amzn1            amzn-updates        187 k
 libjpeg-turbo       x86_64       1.2.90-8.16.amzn1             amzn-updates        144 k
 libmount            x86_64       2.23.2-59.29.amzn1            amzn-updates        192 k
 libuuid             x86_64       2.23.2-59.29.amzn1            amzn-updates         84 k
 util-linux          x86_64       2.23.2-59.29.amzn1            amzn-updates        3.1 M
Removing:
 kernel              x86_64       4.14.97-74.72.amzn1           @amzn-updates        90 M
Installing for dependencies:
 libsmartcols        x86_64       2.23.2-59.29.amzn1            amzn-updates        146 k

Transaction Summary
==========================================================================================
Install  1 Package  (+1 Dependent package)
Upgrade  6 Packages
Remove   1 Package

Total download size: 26 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): libblkid-2.23.2-59.29.amzn1.x86_64.rpm                      | 187 kB  00:00:00     
(2/8): kernel-tools-4.14.143-91.122.amzn1.x86_64.rpm               | 133 kB  00:00:00     
(3/8): libjpeg-turbo-1.2.90-8.16.amzn1.x86_64.rpm                  | 144 kB  00:00:00     
(4/8): libmount-2.23.2-59.29.amzn1.x86_64.rpm                      | 192 kB  00:00:00     
(5/8): libsmartcols-2.23.2-59.29.amzn1.x86_64.rpm                  | 146 kB  00:00:00     
(6/8): libuuid-2.23.2-59.29.amzn1.x86_64.rpm                       |  84 kB  00:00:00     
(7/8): util-linux-2.23.2-59.29.amzn1.x86_64.rpm                    | 3.1 MB  00:00:00     
(8/8): kernel-4.14.143-91.122.amzn1.x86_64.rpm                     |  22 MB  00:00:15     
------------------------------------------------------------------------------------------
Total                                                     1.7 MB/s |  26 MB  00:00:15     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : libuuid-2.23.2-59.29.amzn1.x86_64                                     1/15 
  Updating   : libblkid-2.23.2-59.29.amzn1.x86_64                                    2/15 
  Updating   : libmount-2.23.2-59.29.amzn1.x86_64                                    3/15 
  Installing : libsmartcols-2.23.2-59.29.amzn1.x86_64                                4/15 
  Updating   : util-linux-2.23.2-59.29.amzn1.x86_64                                  5/15 
  Updating   : libjpeg-turbo-1.2.90-8.16.amzn1.x86_64                                6/15 
  Installing : kernel-4.14.143-91.122.amzn1.x86_64                                   7/15 
  Updating   : kernel-tools-4.14.143-91.122.amzn1.x86_64                             8/15 
  Cleanup    : util-linux-2.23.2-33.28.amzn1.x86_64                                  9/15 
  Cleanup    : kernel-4.14.97-74.72.amzn1.x86_64                                    10/15 
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.order: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.networking: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.modesetting: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.drm: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.builtin: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.block: remove failed: No such file or directory
  Cleanup    : libmount-2.23.2-33.28.amzn1.x86_64                                   11/15 
  Cleanup    : libblkid-2.23.2-33.28.amzn1.x86_64                                   12/15 
  Cleanup    : libuuid-2.23.2-33.28.amzn1.x86_64                                    13/15 
  Cleanup    : libjpeg-turbo-1.2.90-5.14.amzn1.x86_64                               14/15 
  Cleanup    : kernel-tools-4.14.138-89.102.amzn1.x86_64                            15/15 
intel: model '', path ' intel-ucode/*', kvers ''
intel: blacklist ''
intel-06-4f-01: model 'GenuineIntel 06-4f-01', path ' intel-ucode/06-4f-01', kvers ' 4.14.42'
intel-06-4f-01: blacklist ''
  Verifying  : util-linux-2.23.2-59.29.amzn1.x86_64                                  1/15 
  Verifying  : kernel-tools-4.14.143-91.122.amzn1.x86_64                             2/15 
  Verifying  : libsmartcols-2.23.2-59.29.amzn1.x86_64                                3/15 
  Verifying  : libblkid-2.23.2-59.29.amzn1.x86_64                                    4/15 
  Verifying  : libmount-2.23.2-59.29.amzn1.x86_64                                    5/15 
  Verifying  : libuuid-2.23.2-59.29.amzn1.x86_64                                     6/15 
  Verifying  : kernel-4.14.143-91.122.amzn1.x86_64                                   7/15 
  Verifying  : libjpeg-turbo-1.2.90-8.16.amzn1.x86_64                                8/15 
  Verifying  : libjpeg-turbo-1.2.90-5.14.amzn1.x86_64                                9/15 
  Verifying  : libmount-2.23.2-33.28.amzn1.x86_64                                   10/15 
  Verifying  : libblkid-2.23.2-33.28.amzn1.x86_64                                   11/15 
  Verifying  : libuuid-2.23.2-33.28.amzn1.x86_64                                    12/15 
  Verifying  : kernel-tools-4.14.138-89.102.amzn1.x86_64                            13/15 
  Verifying  : util-linux-2.23.2-33.28.amzn1.x86_64                                 14/15 
  Verifying  : kernel-4.14.97-74.72.amzn1.x86_64                                    15/15 

Removed:
  kernel.x86_64 0:4.14.97-74.72.amzn1                                                     

Installed:
  kernel.x86_64 0:4.14.143-91.122.amzn1                                                   

Dependency Installed:
  libsmartcols.x86_64 0:2.23.2-59.29.amzn1                                                

Updated:
  kernel-tools.x86_64 0:4.14.143-91.122.amzn1    libblkid.x86_64 0:2.23.2-59.29.amzn1     
  libjpeg-turbo.x86_64 0:1.2.90-8.16.amzn1       libmount.x86_64 0:2.23.2-59.29.amzn1     
  libuuid.x86_64 0:2.23.2-59.29.amzn1            util-linux.x86_64 0:2.23.2-59.29.amzn1   

Complete!

Install Git

sudo yum install git -y

Output

[cloud_user@ip-10-0-1-15 ~]$ sudo yum install git -y
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:2.14.5-1.60.amzn1 will be installed
--> Processing Dependency: perl-Git = 2.14.5-1.60.amzn1 for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Git::I18N) for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Git) for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Error) for package: git-2.14.5-1.60.amzn1.x86_64
--> Running transaction check
---> Package perl-Error.noarch 1:0.17020-2.9.amzn1 will be installed
---> Package perl-Git.noarch 0:2.14.5-1.60.amzn1 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.9.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================
 Package                Arch         Version                     Repository          Size
==========================================================================================
Installing:
 git                    x86_64       2.14.5-1.60.amzn1           amzn-updates        12 M
Installing for dependencies:
 perl-Error             noarch       1:0.17020-2.9.amzn1         amzn-main           33 k
 perl-Git               noarch       2.14.5-1.60.amzn1           amzn-updates        69 k
 perl-TermReadKey       x86_64       2.30-20.9.amzn1             amzn-main           33 k

Transaction Summary
==========================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 12 M
Installed size: 29 M
Downloading packages:
(1/4): perl-Error-0.17020-2.9.amzn1.noarch.rpm                     |  33 kB  00:00:00     
(2/4): perl-TermReadKey-2.30-20.9.amzn1.x86_64.rpm                 |  33 kB  00:00:00     
(3/4): perl-Git-2.14.5-1.60.amzn1.noarch.rpm                       |  69 kB  00:00:00     
(4/4): git-2.14.5-1.60.amzn1.x86_64.rpm                            |  12 MB  00:00:01     
------------------------------------------------------------------------------------------
Total                                                     6.3 MB/s |  12 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:perl-Error-0.17020-2.9.amzn1.noarch                                  1/4 
  Installing : perl-TermReadKey-2.30-20.9.amzn1.x86_64                                2/4 
  Installing : perl-Git-2.14.5-1.60.amzn1.noarch                                      3/4 
  Installing : git-2.14.5-1.60.amzn1.x86_64                                           4/4 
  Verifying  : git-2.14.5-1.60.amzn1.x86_64                                           1/4 
  Verifying  : 1:perl-Error-0.17020-2.9.amzn1.noarch                                  2/4 
  Verifying  : perl-Git-2.14.5-1.60.amzn1.noarch                                      3/4 
  Verifying  : perl-TermReadKey-2.30-20.9.amzn1.x86_64                                4/4 

Installed:
  git.x86_64 0:2.14.5-1.60.amzn1                                                          

Dependency Installed:
  perl-Error.noarch 1:0.17020-2.9.amzn1          perl-Git.noarch 0:2.14.5-1.60.amzn1     
  perl-TermReadKey.x86_64 0:2.30-20.9.amzn1     

Complete!

Clone Github Repoistory

Type the following in the terminal

git clone https://github.com/austinsonger/aws-security

Output

[cloud_user@ip-10-0-1-15 ~]$ git clone https://github.com/austinsonger/aws-security
Cloning into 'aws-security'...
remote: Enumerating objects: 128, done.
remote: Counting objects: 100% (128/128), done.
remote: Compressing objects: 100% (69/69), done.
remote: Total 128 (delta 38), reused 128 (delta 38), pack-reused 0
Receiving objects: 100% (128/128), 1.20 MiB | 37.29 MiB/s, done.
Resolving deltas: 100% (38/38), done.

Install git-secrets

Follow the following steps

git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
sudo make install
git secrets
cd ..

Output

[cloud_user@ip-10-0-1-15 ~]$ git clone https://github.com/awslabs/git-secrets.git
Cloning into 'git-secrets'...
remote: Enumerating objects: 341, done.
remote: Total 341 (delta 0), reused 0 (delta 0), pack-reused 341
Receiving objects: 100% (341/341), 109.22 KiB | 13.65 MiB/s, done.
Resolving deltas: 100% (194/194), done.
[cloud_user@ip-10-0-1-15 ~]$ cd git-secrets
[cloud_user@ip-10-0-1-15 git-secrets]$ sudo make install
[cloud_user@ip-10-0-1-15 git-secrets]$ git secrets
usage: git secrets --scan [-r|--recursive] [--cached] [--no-index] [--untracked] [<files>...]
   or: git secrets --scan-history
   or: git secrets --install [-f|--force] [<target-directory>]
   or: git secrets --list [--global]
   or: git secrets --add [-a|--allowed] [-l|--literal] [--global] <pattern>
   or: git secrets --add-provider [--global] <command> [arguments...]
   or: git secrets --register-aws [--global]
   or: git secrets --aws-provider [<credentials-file>]

    --scan                Scans <files> for prohibited patterns
    --scan-history        Scans repo for prohibited patterns
    --install             Installs git hooks for Git repository or Git template directory
    --list                Lists secret patterns
    --add                 Adds a prohibited or allowed pattern, ensuring to de-dupe with existing patterns
    --add-provider        Adds a secret provider that when called outputs secret patterns on new lines
    --aws-provider        Secret provider that outputs credentials found in an ini file
    --register-aws        Adds common AWS patterns to the git config and scans for ~/.aws/credentials
    -r, --recursive       --scan scans directories recursively
    --cached              --scan scans searches blobs registered in the index file
    --no-index            --scan searches files in the current directory that is not managed by Git
    --untracked           In addition to searching in the tracked files in the working tree, --scan also in untracked files
    -f, --force           --install overwrites hooks if the hook already exists
    -l, --literal         --add and --add-allowed patterns are escaped so that they are literal
    -a, --allowed         --add adds an allowed pattern instead of a prohibited pattern
    --global              Uses the --global git config

[cloud_user@ip-10-0-1-15 git-secrets]$ cd ..
[cloud_user@ip-10-0-1-15 ~]$ ls
aws-security  git-secrets

Scan Repository

Change to the repository directory:

Follow these steps

cd aws-security

cd git_secrets_activity

Output

[cloud_user@ip-10-0-1-15 ~]$ cd aws-security
[cloud_user@ip-10-0-1-15 aws-security]$ ls
311 - CloudFront
312 - restrict to s3
316 - CRR
Automatic-Resource-Remediation-with-AWS-Config
Enabling-VPC-Flow-Logs-with-Automation
git_secrets_activity
README.md
S3Events
Troubleshooting-Detection-Alerting-Response-Workflow
Troubleshooting-Logging-with-CloudTrail-and-S3
[cloud_user@ip-10-0-1-15 aws-security]$ cd git_secrets_activity
[cloud_user@ip-10-0-1-15 git_secrets_activity]$ ls
example.sh

Register the AWS rule set (the “Git hooks”):

Follow these steps

git secrets --register-aws

Output

[cloud_user@ip-10-0-1-15 git_secrets_activity]$ git secrets --register-aws
OK

Scan the repository:

Follow these steps

git secrets --scan

Output

[cloud_user@ip-10-0-1-15 git_secrets_activity]$ git secrets --scan
example.sh:4:AWS_SECRET_ACCESS_KEY = Z3ofnVlFTH9DFmulF3uDO7BCDxGYD4nIG92oeymX

[ERROR] Matched one or more prohibited patterns

Possible mitigations:
- Mark false positives as allowed using: git config --add secrets.allowed ...
- Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory
- List your configured patterns: git config --get-all secrets.patterns
- List your configured allowed patterns: git config --get-all secrets.allowed
- List your configured allowed patterns in .gitallowed at repository's root directory
- Use --no-verify if this is a one-time false positive