How can Jenkinsfile extract value from console output?

How can I get Jenkins to extract a value from the console output that is generated by a command in a Jenkinsfile?

Specifically, how can I get Jenkins to extract the commit hash from the results of the checkout scm command in the following very simple Jenkinsfile?


Jenkinsfile


node {
    // Clean workspace before doing anything
    deleteDir()
    try {
        stage ('Clone') {
            checkout scm    
        }
    } catch (err) {
        currentBuild.result = 'FAILED'
        throw err
    }
}  


Output Of checkout scm command:


The Jenkins log prints the following as a result of running the checkout scm command in the above simplified Jenkinsfile:

Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git
 > git init /var/jenkins_home/workspace/le-repo_sample-issue-branch-2WOFDGRDQWR367VAM7O26H2DKPTRVPDKRTGNRIS4AQNNFP7QIX2Q # timeout=10
Fetching upstream changes from http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git  
 > git --version # timeout=10  
 using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git +refs/heads/sample-issue-branch:refs/remotes/origin/sample-issue-branch
 > git config remote.origin.url http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/sample-issue-branch:refs/remotes/origin/sample-issue-branch # timeout=10
 > git config remote.origin.url http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git # timeout=10
Fetching without tags
Fetching upstream changes from http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git  
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress http://<bitbucket-ip-on-lan>:7990/scm/JSP/jenkinsfile-simple-repo.git +refs/heads/sample-issue-branch:refs/remotes/origin/sample-issue-branch
Checking out Revision 77cf12f42136efc77fecbcd1f761a54254278cb3 (sample-issue-branch)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 77cf12f42136efc77fecbcd1f761a54254278cb3
Commit message: "add whitespace"
 > git rev-list --no-walk e975fb4391677bc09f2056b3e8a6be62eda0b222 # timeout=10
[Bitbucket] Notifying commit build result


Restated Question:


Specifically, what do we add to the Jenkinsfile in order for the log to additionally print out the following at the end:

Commit hash is:  77cf12f42136efc77fecbcd1f761a54254278cb3  

This is obviously over simplified. The printed output in real life will go into a variable and be passed as an argument into a script. But for this OP, what specific syntax will enable Jenkins to extract the commit hash?

Solution:

If you need commit hash in particular, the checkout step’s documentation mentions an easy way to get it (rather than parse console output etc):

This step returns a Map of any variables the SCM plugin would set in a
Freestyle job, so if your SCM is git, you can do:

    def scmVars = checkout scm
    def commitHash = scmVars.GIT_COMMIT

    // or

    def commitHash = checkout(scm).GIT_COMMIT

Once you have it in a variable it’s just a matter of echoing it to see it in the log.

echo "Commit hash is: ${commitHash}"
Advertisements

Git doesn't add my files, even though I add them

After adding files in a bunch of different ways, the files are still not added to the commit, no matter which way I am trying to add them. This is my console output so you understand better:

daniel@padamtuts:~/github/dotfiles$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working    directory)
  (commit or discard the untracked or modified content in submodules)

    modified:   .vim/bundle/auto-pairs (modified content)
    modified:   .vim/bundle/ctrlp.vim (modified content)
    modified:   .vim/bundle/emmet-vim (modified content)
    modified:   .vim/bundle/nerdtree (modified content)
    <a lot of files>

no changes added to commit (use "git add" and/or "git commit -a")
daniel@padamtuts:~/github/dotfiles$ git add *
daniel@padamtuts:~/github/dotfiles$ git add -A
daniel@padamtuts:~/github/dotfiles$ git commit -m "Vim bundles"
On branch master
Changes not staged for commit:
    modified:   .vim/bundle/auto-pairs (modified content)
    ...
    <same files>

no changes added to commitPdaniel@padamtuts:~/github/dotfiles$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working    directory)
  (commit or discard the untracked or modified content in submodules)

    modified:   .vim/bundle/auto-pairs (modified content)
    <same files>

no changes added to commit (use "git add" and/or "git commit -a")
daniel@padamtuts:~/github/dotfiles$ git add *
daniel@padamtuts:~/github/dotfiles$ git add -A
daniel@padamtuts:~/github/dotfiles$ git commit -m "Vim bundles"
On branch master
Changes not staged for commit:
    modified:   .vim/bundle/auto-pairs (modified content)
    <same files>

no changes added to commit

“`

Solution:

Files/directories that begin with . are ‘hidden’ and will be ignored by git when you add with the glob character. You can read about these ‘dotfiles’ if you want to learn more; usually they are configuration files (e.g. for vim) and such that you may not want to interact with directly and by default will not appear in a file explorer, or in the output of ls, etc.

For your purposes, git add . should do the trick.

Jenkins pipline pruned branches are not deleted from disk

Jenkins will create a new build job within the pipline when a new branch is created. When that branch is merged and pruned jenkins will remove it from the UI, as the scan picks it up. But the job folder for that old branch remains on the disk. Some of my repos are 2g+ by the time we clone, pull in dependencies and build. The pipelines are quickly using all the disk space.

How can I force jenkins to delete the folders belonging to pruned branches?

Update

It looks like the intended behaviour of Jenkins is to delete the job, but not the workspace files for piplines, you must delete those yourself. The finally block of my pipeline now has the following:

cleanWs() 

Solution:

If this is an option for you, Simply delete the workspace as last step of your build with cleanWs. Of course you’ll need to install the plugin.

You could also delete the a specific directory using deleteDir. This could be the directory in which the nuget packages are stored. Also run nuger restore without nocache parameter

One more option is to simply run git clean -xfd as a bat step at the end, this is deleting
all thr folders and files not under source control. Of course this implies that workspace Root is the got repo Rroot. be careful if you are saving some build logs or something that you need but it’s not under git

Setting credentials for https git clone in AWS CodeBuild

I am running a CodeBuild on a project that has private requirements stored in CodeCommit.

My requriements.txt is

boto3==1.4.4
git+https://git-codecommit.us-west-2.amazonaws.com/v1/repos/my-req@master#egg=my-req

My IAM credentials are setup correctly and I have a username and password that works locally for both pip install -r requirements.txt and git clone git+https://....

I could use ssh but I don’t think that works with CodeBuild.

My buildspec.yml has

...
phases:
  install:
    commands:
      - pip install -r requirements.txt
...

I need to add a command in buildspec.yml that loads the https git credentials and then the git clone will work in pip install.

Currently it fails with:

fatal: could not read Username for 'https://git-codecommit.us-west-2.amazonaws.com': No such device or address

I can do git@git-code... but I still need the password. I don’t want to hard code it into requirements.txt.

What’s the best way to set https git credentials in buildspec.yml?

Solution:

Since the CodeBuild environment uses an IAM role for credentials (not a username and password), you will need to configure the CodeCommit credential helper in your buildspec:

- git config --global credential.helper '!aws codecommit credential-helper $@'
- git config --global credential.UseHttpPath true

Access linux paths in windows git powershell

I am running a Windows Powershell provided through the git for windows installation. This shell provides many unix style commands (i.e. “ls”, “mv”, etc.).

My question is: How do I access Unix style paths from the powershell cmd line on Windows?

Consider this example: the “ls” program is installed and works in the powershell. The path is shown as “/usr/bin/ls” if I type “which ls” as the cmd prompt. But if I try to change my current directory using “cd /usr/bin/”, the shell complains that the path is not found.

enter image description here

I can’t see any mounted volumes or anything like that using “mount” (perhaps in PowerShell it is a different command?).

I’m asking this question because I have other files that I need to get to which are listed under unix-style paths, and right now I can’t get to anything. I figure if I can get to /usr/bin, then I can figure out how to get where I really need to go.

Solution:

Powershell is not Unix. It may have a few familiar commands like “ls” and “ps”, but that’s where the similarity ends.

When you installed Git For Windows, you likely installed the Git Bash shell as well. Run that instead to get a more Unix like atmosphere. (Re-install Git For Windows if you didn’t select this option on install).

But even with Git Bash, there’s still no such folder as /usr/bin. That folder doesn’t exist on Windows. If you want a Unix emulation on Windows that includes the traditional folder structure, use Cygwin. And you can run Git on that environment too and access an emulated /usr/bin folder.

why ssh always ask me for passwords even I have set the public key in VPS?

I just order a VPS in Vultr, and want set to ssh without passwords. after configured the ssh key things, I think it should be work, but failed.
yes, I have multiple ssh configures like github, gitlab or some other things and this is the configure file

Host gitlab
 HostName cd.i.foo.com
 IdentityFile ~/.ssh/id_rsa
Host github
 HostName github.com
 IdentityFile ~/.ssh/id_rsa_gayhub
Host vultr
 HostName 198.13.59.103
 Port 22
 User root
 IdentityFile ~/.ssh/id_rsa_vultr

but when I try with specified path ssh root@198.13.59.103 -i ~/.ssh/id_rsa_vultr, it can work!!!

below is some more log with ssh -v -v root@198.13.59.103

OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/xdguo/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: Connecting to 198.13.59.103 port 22.
debug1: Connection established.
debug1: identity file /Users/xdguo/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/xdguo/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 198.13.59.103:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:s8qKj8CKXc6tp7zmVV32hiqHvdrxk46JVJQlQfpOQ0Q
debug1: Host '198.13.59.103' is known and matches the ECDSA host key.
debug1: Found key in /Users/xdguo/.ssh/known_hosts:40
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:LBGwqZmXcUwd6kqIuDAxIj+jbPBeLbU1uwrxN1mQRKQ /Users/xdguo/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /Users/xdguo/.ssh/id_dsa
debug1: Trying private key: /Users/xdguo/.ssh/id_ecdsa
debug1: Trying private key: /Users/xdguo/.ssh/id_ed25519
debug1: Next authentication method: password
root@198.13.59.103's password:

I think I spend lots of time here, it really maked me confused, thanks in advance for any help

Solution:

ssh -v -v root@198.13.59.103 would not work, as it would default to id_rsa.

You need to test:

ssh -Tvv vultr

Only that “URL” (vultr) would be the equivalent of ssh root@198.13.59.103 -i ~/.ssh/id_rsa_vultr: it would find everything it needs in your ~/.ssh/config, under the Host vultr entry.