Amazon Web Services: Performing a Source Code Security Scan Using git-secrets in AWS

Amazon Web Services: Performing a Source Code Security Scan Using git-secrets in AWS

EC2 environment. In this tutorial you will install Git, clone a repository, install `git-secrets`, and scan for a vulnerability which has been deliberately introduced into the code.

After entering into a SSH session of your server you will need to update the server first.

Last login: Tue Sep 24 14:07:23 on ttys000
austinsonger@Songer ~ % ssh cloud_user@35.172.109.218
The authenticity of host '35.172.109.218 (35.172.109.218)' can't be established.
ECDSA key fingerprint is SHA256:7koLywItBCNqYYmWN3n/r7fHm8UJpk3DmGKqmQ+RMvg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '35.172.109.218' (ECDSA) to the list of known hosts.
Password: 

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
3 package(s) needed for security, out of 7 available
Run "sudo yum update" to apply all updates.
[cloud_user@ip-10-0-1-15 ~]$ sudo yum update
[sudo] password for cloud_user: 
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main                                                          | 2.1 kB  00:00:00     
amzn-updates                                                       | 2.5 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:4.14.143-91.122.amzn1 will be installed
---> Package kernel-tools.x86_64 0:4.14.138-89.102.amzn1 will be updated
---> Package kernel-tools.x86_64 0:4.14.143-91.122.amzn1 will be an update
---> Package libblkid.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libblkid.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package libjpeg-turbo.x86_64 0:1.2.90-5.14.amzn1 will be updated
---> Package libjpeg-turbo.x86_64 0:1.2.90-8.16.amzn1 will be an update
---> Package libmount.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libmount.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package libuuid.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package libuuid.x86_64 0:2.23.2-59.29.amzn1 will be an update
---> Package util-linux.x86_64 0:2.23.2-33.28.amzn1 will be updated
---> Package util-linux.x86_64 0:2.23.2-59.29.amzn1 will be an update
--> Processing Dependency: libsmartcols = 2.23.2-59.29.amzn1 for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Processing Dependency: libsmartcols.so.1(SMARTCOLS_2.25)(64bit) for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Processing Dependency: libsmartcols.so.1()(64bit) for package: util-linux-2.23.2-59.29.amzn1.x86_64
--> Running transaction check
---> Package libsmartcols.x86_64 0:2.23.2-59.29.amzn1 will be installed
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:4.14.97-74.72.amzn1 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================
 Package             Arch         Version                       Repository           Size
==========================================================================================
Installing:
 kernel              x86_64       4.14.143-91.122.amzn1         amzn-updates         22 M
Updating:
 kernel-tools        x86_64       4.14.143-91.122.amzn1         amzn-updates        133 k
 libblkid            x86_64       2.23.2-59.29.amzn1            amzn-updates        187 k
 libjpeg-turbo       x86_64       1.2.90-8.16.amzn1             amzn-updates        144 k
 libmount            x86_64       2.23.2-59.29.amzn1            amzn-updates        192 k
 libuuid             x86_64       2.23.2-59.29.amzn1            amzn-updates         84 k
 util-linux          x86_64       2.23.2-59.29.amzn1            amzn-updates        3.1 M
Removing:
 kernel              x86_64       4.14.97-74.72.amzn1           @amzn-updates        90 M
Installing for dependencies:
 libsmartcols        x86_64       2.23.2-59.29.amzn1            amzn-updates        146 k

Transaction Summary
==========================================================================================
Install  1 Package  (+1 Dependent package)
Upgrade  6 Packages
Remove   1 Package

Total download size: 26 M
Is this ok [y/d/N]: y
Downloading packages:
(1/8): libblkid-2.23.2-59.29.amzn1.x86_64.rpm                      | 187 kB  00:00:00     
(2/8): kernel-tools-4.14.143-91.122.amzn1.x86_64.rpm               | 133 kB  00:00:00     
(3/8): libjpeg-turbo-1.2.90-8.16.amzn1.x86_64.rpm                  | 144 kB  00:00:00     
(4/8): libmount-2.23.2-59.29.amzn1.x86_64.rpm                      | 192 kB  00:00:00     
(5/8): libsmartcols-2.23.2-59.29.amzn1.x86_64.rpm                  | 146 kB  00:00:00     
(6/8): libuuid-2.23.2-59.29.amzn1.x86_64.rpm                       |  84 kB  00:00:00     
(7/8): util-linux-2.23.2-59.29.amzn1.x86_64.rpm                    | 3.1 MB  00:00:00     
(8/8): kernel-4.14.143-91.122.amzn1.x86_64.rpm                     |  22 MB  00:00:15     
------------------------------------------------------------------------------------------
Total                                                     1.7 MB/s |  26 MB  00:00:15     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : libuuid-2.23.2-59.29.amzn1.x86_64                                     1/15 
  Updating   : libblkid-2.23.2-59.29.amzn1.x86_64                                    2/15 
  Updating   : libmount-2.23.2-59.29.amzn1.x86_64                                    3/15 
  Installing : libsmartcols-2.23.2-59.29.amzn1.x86_64                                4/15 
  Updating   : util-linux-2.23.2-59.29.amzn1.x86_64                                  5/15 
  Updating   : libjpeg-turbo-1.2.90-8.16.amzn1.x86_64                                6/15 
  Installing : kernel-4.14.143-91.122.amzn1.x86_64                                   7/15 
  Updating   : kernel-tools-4.14.143-91.122.amzn1.x86_64                             8/15 
  Cleanup    : util-linux-2.23.2-33.28.amzn1.x86_64                                  9/15 
  Cleanup    : kernel-4.14.97-74.72.amzn1.x86_64                                    10/15 
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.order: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.networking: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.modesetting: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.drm: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.builtin: remove failed: No such file or directory
warning: file /lib/modules/4.14.97-74.72.amzn1.x86_64/modules.block: remove failed: No such file or directory
  Cleanup    : libmount-2.23.2-33.28.amzn1.x86_64                                   11/15 
  Cleanup    : libblkid-2.23.2-33.28.amzn1.x86_64                                   12/15 
  Cleanup    : libuuid-2.23.2-33.28.amzn1.x86_64                                    13/15 
  Cleanup    : libjpeg-turbo-1.2.90-5.14.amzn1.x86_64                               14/15 
  Cleanup    : kernel-tools-4.14.138-89.102.amzn1.x86_64                            15/15 
intel: model '', path ' intel-ucode/*', kvers ''
intel: blacklist ''
intel-06-4f-01: model 'GenuineIntel 06-4f-01', path ' intel-ucode/06-4f-01', kvers ' 4.14.42'
intel-06-4f-01: blacklist ''
  Verifying  : util-linux-2.23.2-59.29.amzn1.x86_64                                  1/15 
  Verifying  : kernel-tools-4.14.143-91.122.amzn1.x86_64                             2/15 
  Verifying  : libsmartcols-2.23.2-59.29.amzn1.x86_64                                3/15 
  Verifying  : libblkid-2.23.2-59.29.amzn1.x86_64                                    4/15 
  Verifying  : libmount-2.23.2-59.29.amzn1.x86_64                                    5/15 
  Verifying  : libuuid-2.23.2-59.29.amzn1.x86_64                                     6/15 
  Verifying  : kernel-4.14.143-91.122.amzn1.x86_64                                   7/15 
  Verifying  : libjpeg-turbo-1.2.90-8.16.amzn1.x86_64                                8/15 
  Verifying  : libjpeg-turbo-1.2.90-5.14.amzn1.x86_64                                9/15 
  Verifying  : libmount-2.23.2-33.28.amzn1.x86_64                                   10/15 
  Verifying  : libblkid-2.23.2-33.28.amzn1.x86_64                                   11/15 
  Verifying  : libuuid-2.23.2-33.28.amzn1.x86_64                                    12/15 
  Verifying  : kernel-tools-4.14.138-89.102.amzn1.x86_64                            13/15 
  Verifying  : util-linux-2.23.2-33.28.amzn1.x86_64                                 14/15 
  Verifying  : kernel-4.14.97-74.72.amzn1.x86_64                                    15/15 

Removed:
  kernel.x86_64 0:4.14.97-74.72.amzn1                                                     

Installed:
  kernel.x86_64 0:4.14.143-91.122.amzn1                                                   

Dependency Installed:
  libsmartcols.x86_64 0:2.23.2-59.29.amzn1                                                

Updated:
  kernel-tools.x86_64 0:4.14.143-91.122.amzn1    libblkid.x86_64 0:2.23.2-59.29.amzn1     
  libjpeg-turbo.x86_64 0:1.2.90-8.16.amzn1       libmount.x86_64 0:2.23.2-59.29.amzn1     
  libuuid.x86_64 0:2.23.2-59.29.amzn1            util-linux.x86_64 0:2.23.2-59.29.amzn1   

Complete!

Install Git

sudo yum install git -y

Output

[cloud_user@ip-10-0-1-15 ~]$ sudo yum install git -y
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:2.14.5-1.60.amzn1 will be installed
--> Processing Dependency: perl-Git = 2.14.5-1.60.amzn1 for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Git::I18N) for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Git) for package: git-2.14.5-1.60.amzn1.x86_64
--> Processing Dependency: perl(Error) for package: git-2.14.5-1.60.amzn1.x86_64
--> Running transaction check
---> Package perl-Error.noarch 1:0.17020-2.9.amzn1 will be installed
---> Package perl-Git.noarch 0:2.14.5-1.60.amzn1 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.9.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================
 Package                Arch         Version                     Repository          Size
==========================================================================================
Installing:
 git                    x86_64       2.14.5-1.60.amzn1           amzn-updates        12 M
Installing for dependencies:
 perl-Error             noarch       1:0.17020-2.9.amzn1         amzn-main           33 k
 perl-Git               noarch       2.14.5-1.60.amzn1           amzn-updates        69 k
 perl-TermReadKey       x86_64       2.30-20.9.amzn1             amzn-main           33 k

Transaction Summary
==========================================================================================
Install  1 Package (+3 Dependent packages)

Total download size: 12 M
Installed size: 29 M
Downloading packages:
(1/4): perl-Error-0.17020-2.9.amzn1.noarch.rpm                     |  33 kB  00:00:00     
(2/4): perl-TermReadKey-2.30-20.9.amzn1.x86_64.rpm                 |  33 kB  00:00:00     
(3/4): perl-Git-2.14.5-1.60.amzn1.noarch.rpm                       |  69 kB  00:00:00     
(4/4): git-2.14.5-1.60.amzn1.x86_64.rpm                            |  12 MB  00:00:01     
------------------------------------------------------------------------------------------
Total                                                     6.3 MB/s |  12 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:perl-Error-0.17020-2.9.amzn1.noarch                                  1/4 
  Installing : perl-TermReadKey-2.30-20.9.amzn1.x86_64                                2/4 
  Installing : perl-Git-2.14.5-1.60.amzn1.noarch                                      3/4 
  Installing : git-2.14.5-1.60.amzn1.x86_64                                           4/4 
  Verifying  : git-2.14.5-1.60.amzn1.x86_64                                           1/4 
  Verifying  : 1:perl-Error-0.17020-2.9.amzn1.noarch                                  2/4 
  Verifying  : perl-Git-2.14.5-1.60.amzn1.noarch                                      3/4 
  Verifying  : perl-TermReadKey-2.30-20.9.amzn1.x86_64                                4/4 

Installed:
  git.x86_64 0:2.14.5-1.60.amzn1                                                          

Dependency Installed:
  perl-Error.noarch 1:0.17020-2.9.amzn1          perl-Git.noarch 0:2.14.5-1.60.amzn1     
  perl-TermReadKey.x86_64 0:2.30-20.9.amzn1     

Complete!

Clone Github Repoistory

Type the following in the terminal

git clone https://github.com/austinsonger/aws-security

Output

[cloud_user@ip-10-0-1-15 ~]$ git clone https://github.com/austinsonger/aws-security
Cloning into 'aws-security'...
remote: Enumerating objects: 128, done.
remote: Counting objects: 100% (128/128), done.
remote: Compressing objects: 100% (69/69), done.
remote: Total 128 (delta 38), reused 128 (delta 38), pack-reused 0
Receiving objects: 100% (128/128), 1.20 MiB | 37.29 MiB/s, done.
Resolving deltas: 100% (38/38), done.

Install git-secrets

Follow the following steps

git clone https://github.com/awslabs/git-secrets.git
cd git-secrets
sudo make install
git secrets
cd ..

Output

[cloud_user@ip-10-0-1-15 ~]$ git clone https://github.com/awslabs/git-secrets.git
Cloning into 'git-secrets'...
remote: Enumerating objects: 341, done.
remote: Total 341 (delta 0), reused 0 (delta 0), pack-reused 341
Receiving objects: 100% (341/341), 109.22 KiB | 13.65 MiB/s, done.
Resolving deltas: 100% (194/194), done.
[cloud_user@ip-10-0-1-15 ~]$ cd git-secrets
[cloud_user@ip-10-0-1-15 git-secrets]$ sudo make install
[cloud_user@ip-10-0-1-15 git-secrets]$ git secrets
usage: git secrets --scan [-r|--recursive] [--cached] [--no-index] [--untracked] [<files>...]
   or: git secrets --scan-history
   or: git secrets --install [-f|--force] [<target-directory>]
   or: git secrets --list [--global]
   or: git secrets --add [-a|--allowed] [-l|--literal] [--global] <pattern>
   or: git secrets --add-provider [--global] <command> [arguments...]
   or: git secrets --register-aws [--global]
   or: git secrets --aws-provider [<credentials-file>]

    --scan                Scans <files> for prohibited patterns
    --scan-history        Scans repo for prohibited patterns
    --install             Installs git hooks for Git repository or Git template directory
    --list                Lists secret patterns
    --add                 Adds a prohibited or allowed pattern, ensuring to de-dupe with existing patterns
    --add-provider        Adds a secret provider that when called outputs secret patterns on new lines
    --aws-provider        Secret provider that outputs credentials found in an ini file
    --register-aws        Adds common AWS patterns to the git config and scans for ~/.aws/credentials
    -r, --recursive       --scan scans directories recursively
    --cached              --scan scans searches blobs registered in the index file
    --no-index            --scan searches files in the current directory that is not managed by Git
    --untracked           In addition to searching in the tracked files in the working tree, --scan also in untracked files
    -f, --force           --install overwrites hooks if the hook already exists
    -l, --literal         --add and --add-allowed patterns are escaped so that they are literal
    -a, --allowed         --add adds an allowed pattern instead of a prohibited pattern
    --global              Uses the --global git config

[cloud_user@ip-10-0-1-15 git-secrets]$ cd ..
[cloud_user@ip-10-0-1-15 ~]$ ls
aws-security  git-secrets

Scan Repository

Change to the repository directory:

Follow these steps

cd aws-security

cd git_secrets_activity

Output

[cloud_user@ip-10-0-1-15 ~]$ cd aws-security
[cloud_user@ip-10-0-1-15 aws-security]$ ls
311 - CloudFront
312 - restrict to s3
316 - CRR
Automatic-Resource-Remediation-with-AWS-Config
Enabling-VPC-Flow-Logs-with-Automation
git_secrets_activity
README.md
S3Events
Troubleshooting-Detection-Alerting-Response-Workflow
Troubleshooting-Logging-with-CloudTrail-and-S3
[cloud_user@ip-10-0-1-15 aws-security]$ cd git_secrets_activity
[cloud_user@ip-10-0-1-15 git_secrets_activity]$ ls
example.sh

Register the AWS rule set (the “Git hooks”):

Follow these steps

git secrets --register-aws

Output

[cloud_user@ip-10-0-1-15 git_secrets_activity]$ git secrets --register-aws
OK

Scan the repository:

Follow these steps

git secrets --scan

Output

[cloud_user@ip-10-0-1-15 git_secrets_activity]$ git secrets --scan
example.sh:4:AWS_SECRET_ACCESS_KEY = Z3ofnVlFTH9DFmulF3uDO7BCDxGYD4nIG92oeymX

[ERROR] Matched one or more prohibited patterns

Possible mitigations:
- Mark false positives as allowed using: git config --add secrets.allowed ...
- Mark false positives as allowed by adding regular expressions to .gitallowed at repository's root directory
- List your configured patterns: git config --get-all secrets.patterns
- List your configured allowed patterns: git config --get-all secrets.allowed
- List your configured allowed patterns in .gitallowed at repository's root directory
- Use --no-verify if this is a one-time false positive


Share Tweet Send
0 Comments
Loading...
You've successfully subscribed to Songer Tech
Great! Next, complete checkout for full access to Songer Tech
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.