Useful ansible stuff

inventory_hostname

inventory_hostname‘ contains the name of the current node being worked on…. (as in, what it is defined in your hosts file as) so if you want to skip a task for a single node –

- name: Restart amavis
  service: name=amavis state=restarted
  when: inventory_hostname != "boris"

(Don’t restart Amavis for boris,  do for all others).

You could also use :

...
  when: inventory_hostname not in groups['group_name']
...

if your aim was to (perhaps skip) a task for some nodes in the specified group.

 

Need to check whether you need to reboot for a kernel update?

  1. If /vmlinuz doesn’t resolve to the same kernel as we’re running
  2. Reboot
  3. Wait 45 seconds before carrying on…
- name: Check for reboot hint.
  shell: if [ $(readlink -f /vmlinuz) != /boot/vmlinuz-$(uname -r) ]; then echo 'reboot'; else echo 'no'; fi
  ignore_errors: true
  register: reboot_hint

- name: Rebooting ...
  command: shutdown -r now "Ansible kernel update applied"
  async: 0
  poll: 0
  ignore_errors: true
  when: kernelup|changed or reboot_hint.stdout.find("reboot") != -1
  register: rebooting

- name: Wait for thing to reboot...
  pause: seconds=45
  when: rebooting|changed

Fixing ~/.ssh/known_hosts

Often an ansible script may create a remote node – and often it’ll have the same IP/name as a previous entity. This confuses SSH — so after creating :

- name: Fix .ssh/known_hosts. (1)
  local_action: command  ssh-keygen -f "~/.ssh/known_hosts" -R hostname

If you’re using ec2, for instance, you could do something like :

- name: Fix .ssh/known_hosts.
  local_action: command  ssh-keygen -f "~/.ssh/known_hosts" -R {{ item.public_ip }} 
  with_items: ec2_info.instances

Where ec2_info is your registered variable from calling the ‘ec2’ module.

Debug/Dump a variable?

- name: What's in reboot_hint?
  debug: var=reboot_hint

which might output something like :

"reboot_hint": {
        "changed": true, 
        "cmd": "if [ $(readlink -f /vmlinuz) != /boot/vmlinuz-$(uname -r) ]; then echo 'reboot'; else echo 'no'; fi", 
        "delta": "0:00:00.024759", 
        "end": "2014-07-29 09:05:06.564505", 
        "invocation": {
            "module_args": "if [ $(readlink -f /vmlinuz) != /boot/vmlinuz-$(uname -r) ]; then echo 'reboot'; else echo 'no'; fi", 
            "module_name": "shell"
        }, 
        "rc": 0, 
        "start": "2014-07-29 09:05:06.539746", 
        "stderr": "", 
        "stdout": "reboot", 
        "stdout_lines": [
            "reboot"
        ]
    }

Which leads on to —

Want to run a shell command do something with the output?

Registered variables have useful attributes like :

  • changed – set to boolean true if something happened (useful to tell when a task has done something on a remote machine).
  • stderr – contains stringy output from stderr
  • stdout – contains stringy output from stdout
  • stdout_lines – contains a list of lines (i.e. stdout split on \n).

(see above)

- name: Do something
  shell: /usr/bin/something | grep -c foo || true
  register: shell_output

So – we could :

- name: Catch some fish (there are at least 5)
  shell: /usr/bin/somethingelse 
  when: shell_output.stdout > "5"

Default values for a Variable, and host specific values.

Perhaps you’ll override a variable, or perhaps not … so you can do something like the following in a template :

...
max_allowed_packet = {{ mysql_max_allowed_packet|default('128M') }}
...

And for the annoying hosts that need a larger mysql_max_allowed_packet, just define it within the inventory hosts file like :

[linux_servers]
beech
busy-web-server mysql_max_allowed_packet=256M
Advertisements

Creating a pipeline in GoCD

If you haven’t installed GoCD yet, you can follow the GoCD Installation Guide to install the GoCD Server and at least one GoCD Agent. This is a good point to stop and learn about the first concept in GoCD.

Concept 1: Server and agents

GoCD Server mapped to three agents

In the GoCD ecosystem, the server is the one that controls everything. It provides the user interface to users of the system and provides work for the agents to do. The agents are the ones that do any work (run commands, do deployments, etc) that is configured by the users or administrators of the system.

The server does not do any user-specified “work” on its own. It will not run any commands or do deployments. That is the reason you need a GoCD Server and at least one GoCD Agent installed before you proceed.

Once you have them installed and running, you should see a screen similar to this, if you navigate to the home page in a web browser (defaults to: http://localhost:8153):

GoCD new pipeline page
GoCD’s new pipeline page

If you have installed the GoCD Agent properly and click on the “Agents” link in the header, you should see an idle GoCD agent waiting (as shown below). If you do not, head over to the troubleshooting page to figure out why.

GoCD Agent page
Agents page

Congratulations! You’re on your way to using GoCD. If you now click “Pipelines”, you’ll get back to the “Add pipeline” screen you saw earlier.

Creating a pipeline

Before creating a pipeline, it might help to know what it is and concepts around it.

Concept 2: Pipelines and materials

Multiple materials mapped to a Pipeline with multiple stages within it

A pipeline, in GoCD, is a representation of workflow or a part of a workflow. For instance, if you are trying to automatically run tests, build an installer and then deploy an application to a test environment, then those steps can be modeled as a pipeline. GoCD provides different modeling constructs within a pipeline, such as stages, jobs and tasks. We will see these in more detail soon. For the purpose of this part of the guide, you need to know that a pipeline can be configured to run a command or set of commands.

Another equally important concept is that of a material. A material is a cause for a pipeline to “trigger” or to start doing what it is configured to do. Typically, a material is a source control repository (like Git, Subversion, etc) and any new commit to the repository is a cause for the pipeline to trigger. A pipeline needs to have at least one material and can have as many materials of different kinds as you want.

The concept of a pipeline is extemely central to Continuous Delivery. Together with the concepts of stages, jobs and tasks, GoCD provides important modeling blocks which allow you to build up very complex workflows, and get feedback quicker. You’ll learn more about GoCD pipelines and Deployment pipelines in the upcoming parts of this guide. In case you cannot wait, Martin Fowler has a nice and succint article here.

Now, back at the “Add pipeline” screen, let’s provide the pipeline a name, without spaces, and ignore the “Pipeline Group” field for now.

Step 1 - Screen to name your pipeline
Step 1: Name our pipeline

Pressing “Next” will take you to step 2, which can be used to configure a material.

Step 2 A - Screen to choose a material
Step 2a: Point it to a material – Where to look for changes?

You can choose your own material here [1], or use a Git repository available on GitHub for this guide. The URL of that repository is: https://github.com/gocd-contrib/getting-started-repo.git. You can change “Material Type” to “Git” and provide the URL in the “URL” textbox. If you now click on the “Check Connection” button, it should tell you everything is OK.

Step 2 B - Checking that the material exists
Step 2b: Check that the material exists

This step assumes that you have git installed on the GoCD Server and Agent. Like git, any other commands you need for running your scripts need to be installed on the GoCD Agent nodes.

If you had trouble with this step, and it failed, take a look at the troubleshooting page in the documentation. If everything went well, press “Next” to be taken to step 3, which deals with stages, jobs and tasks.

Step 3 A - Use the predefined stage and job
Step 3a: Use the predefined stage and job for now

Since a stage name and a job name are populated, and in the interest of quickly achieving our goal of creating and running a pipeline in GoCD, let us delay understanding the (admittedly very important) concepts of stage and job and focus on a task instead. Scrolling down a bit, you’ll see the “Initial Job and Task” section.

Step 3 B - Take a closer look at the initial job and task
Step 3b: Take a closer look at the initial job and task

Since we don’t want to use “Ant” right now, let’s change the “Task Type” to “More…”. You should see a screen like this:

Step 3 C - Choose a custom command
Step 3c: Choose a custom command

Change the text of the “Command” text box to “./build” (that is, dot forward-slash build) and press “Finish”. If all went well, you just created your first pipeline! Leaving you in a screen similar to the one shown below.

Your first pipeline (paused)
Your first pipeline (paused)

Helpfully, the pipeline has been paused for you (see the pause button and the text next to it, right next to the pipeline name). This allows you to make more changes to the pipeline before it triggers. Usually, pipeline administrators can take this opportunity to add more stages, jobs, etc. and model their pipelines better. For the purpose of this guide, let’s just un-pause this pipeline and get it to run. Click on the blue “pause” button and then click on the “Pipelines” link in the header.

If you give it a minute, you’ll see your pipeline building (yellow) and then finish successfully (green):

The pipeline is building
The pipeline is building
The pipeline finished successfully
The pipeline finished successfully

Clicking on the bright green bar will show you information about the stage:

Information about the stage
Information about the stage

and then clicking on the job will take you to the actual task and show you what it did:

The output of the job run
The output of the job run

Scrolling up a bit, you can see it print out the environment variables for the task and the details of the agent this task ran on (remember “Concept 1”?).

The environment variables used for the job
The environment variables used for the job
Job run details
Job run details

How to Trace Execution of Commands in Shell Script with Shell Tracing

In this article of the shell script debugging series, we will explain the third shell script debugging mode, that is shell tracing and look at some examples to demonstrate how it works, and how it can be used.

The previous part of this series clearly throws light upon the two other shell script debugging modes: verbose mode and syntax checking mode with easy-to-understand examples of how to enable shell script debugging in these modes.

Shell tracing simply means tracing the execution of the commands in a shell script. To switch on shell tracing, use the -x debugging option.

This directs the shell to display all commands and their arguments on the terminal as they are executed.

We will use the sys_info.sh shell script below, which briefly prints your system date and time, number of users logged in and the system uptime. However, it contains syntax errors that we need to find and correct.

#!/bin/bash
#script to print brief system info
ROOT_ID="0"
DATE=`date`
NO_USERS=`who | wc -l`
UPTIME=`uptime`
check_root(){
if [ "$UID" -ne "$ROOT_ID" ]; then
echo "You are not allowed to execute this program!"
exit 1;    
}
print_sys_info(){
echo "System Time    : $DATE"
echo "Number of users: $NO_USERS"
echo "System Uptime  : $UPTIME
}
check_root
print_sys_info
exit 0

Save the file and make the script executable. The script can only be run by root, therefore employ the sudo command to run it as below:

$ chmod +x sys_info.sh
$ sudo bash -x sys_info.sh

Shell Tracing - Show Error in Script

From the output above, we can observe that, a command is first executed before its output is substituted as the value of a variable.

For example, the date was first executed and the its output was substituted as the value of the variable DATE.

We can perform syntax checking to only display the syntax errors as follows:

$ sudo bash -n sys_info.sh 

Syntax Checking in Script

If we look at the shell script critically, we will realize that the if statement is missing a closing fi word. Therefore, let us add it and the new script should now look like below:

#!/bin/bash
#script to print brief system info
ROOT_ID="0"
DATE=`date`
NO_USERS=`who | wc -l`
UPTIME=`uptime`
check_root(){
if [ "$UID" -ne "$ROOT_ID" ]; then
echo "You are not allowed to execute this program!"
exit 1;
fi    
}
print_sys_info(){
echo "System Time    : $DATE"
echo "Number of users: $NO_USERS"
echo "System Uptime  : $UPTIME
}
check_root
print_sys_info
exit 0

Save the file again and invoke it as root and do some syntax checking:

$ sudo bash -n sys_info.sh

Perform Syntax Check in Shell Scripts

The result of our syntax checking operation above still shows that there is one more bug in our script on line 21. So, we still have some syntax correction to do.

If we look through the script analytically one more time, the error on line 21 is due to a missing closing double quote (”) in the last echo command inside the print_sys_info function.

We will add the closing double quote in the echo command and save the file. The changed script is below:

#!/bin/bash
#script to print brief system info
ROOT_ID="0"
DATE=`date`
NO_USERS=`who | wc -l`
UPTIME=`uptime`
check_root(){
if [ "$UID" -ne "$ROOT_ID" ]; then
echo "You are not allowed to execute this program!"
exit 1;
fi
}
print_sys_info(){
echo "System Time    : $DATE"
echo "Number of users: $NO_USERS"
echo "System Uptime  : $UPTIME"
}
check_root
print_sys_info
exit 0

Now syntactically check the script one more time.

$ sudo bash -n sys_info.sh

The command above will not produce any output because our script is now syntactically correct. We can as well trace the execution of the script all for a second time and it should work well:

$ sudo bash -x sys_info.sh

Trace Shell Script Execution

Now run the script.

$ sudo ./sys_info.sh

Shell Script to Show Date, Time and Uptime

Importance of Shell Script Execution Tracing

Shell script tracing helps us identify syntax errors and more importantly, logical errors. Take for instance the check_root function in the sys_info.sh shell script, which is intended to determine if a user is root or not, since the script is only allowed to be executed by the superuser.

check_root(){
if [ "$UID" -ne "$ROOT_ID" ]; then
echo "You are not allowed to execute this program!"
exit 1;
fi
}

The magic here is controlled by the if statement expression [ "$UID" -ne "$ROOT_ID" ], once we do not use the suitable numerical operator (-ne in this case, which means not equal ), we end up with a possible logical error.

Assuming that we used -eq ( means equal to), this would permit any system user as well as the root user to run the script, hence a logical error.

check_root(){
if [ "$UID" -eq "$ROOT_ID" ]; then
echo "You are not allowed to execute this program!"
exit 1;
fi
}

Note: As we looked at before at the start of this series, the set shell built-in command can activate debugging in a particular section of a shell script.

Therefore, the line below will help us find this logical error in the function by tracing its execution:

The script with a logical error:

#!/bin/bash
#script to print brief system info
ROOT_ID="0"
DATE=`date`
NO_USERS=`who | wc -l`
UPTIME=`uptime`
check_root(){
if [ "$UID" -eq "$ROOT_ID" ]; then
echo "You are not allowed to execute this program!"
exit 1;
fi
}
print_sys_info(){
echo "System Time    : $DATE"
echo "Number of users: $NO_USERS"
echo "System Uptime  : $UPTIME"
}
#turning on and off debugging of check_root function
set -x ; check_root;  set +x ;
print_sys_info
exit 0

Save the file and invoke the script, we can see that a regular system user can run the script without sudo as in the output below. This is because the value of USER_ID is 100 which is not equal to root ROOT_ID which is 0.

$ ./sys_info.sh

Run Shell Script Without Sudo

Linux commands

sudo!! : Forgot to run a command with sudo? You need not re-write the whole command, just type “sudo!!” and the last command will run with sudo.

2. Python -m SimpleHTTPServer : Creates a simple web page for the current working directory over port 8000.

3. mtr : A command which is a combination of ‘ping’ and ‘traceroute’ command.

4. Ctrl+x+e : This key combination fires up, an editor in the terminal, instantaneously.

5. nl : Outputs the content of text file with lines Numbered.

6. shuf : Randomly selects line/file/folder from a file/folder.

7. ss : Outputs Socket Statistics.

8. Last: Want to know history of last logged in users? This command comes to rescue here.

9. curl ifconfig.me : Shows machine’s external IP Address.

10. tree : Prints files and folders in tree like fashion, recursively.

11. Pstree : Prints running processes with child processes, recursively.

13. stat : Shows the status information of a file as well as of a file system.

15. Pv : outputs simulating text, similar to hollywood movies.

16. Mount | column -t : Lists mounted file system, in nice formatting with specification.

17. Ctrl + l: clear shell prompt, instantaneously.

18. curl -u gmail_id –silent “https://mail.google.com/mail/feed/atom” | perl -ne ‘print “\t” if //; print “$2\n” if /(.*)/;’. This simple scripts, opens up, unread mail of an user, in the terminal itself.

19. screen : Detach and Reattach, long running process from a session.

20. file : Outputs information, regarding types of file.

21. id : Print User and Group Id.
22. ^foo^bar : Run last command with modification, without the need of rewriting the whole command again.

24. at : Run a particular command, time based.

25. du -h –max-depth=1 Command : Outputs the size of all the files and folders within current folder, in human readable format.

26. expr : Solve simple mathematical calculations from the terminal.

27. look: Check for an English word, from the dictionary, in case of confusion, right from the shell.

28. yes : continues to print a sting, till interrupt instruction is given.

29. factor: Gives all the possible factors of a decimal number.

30. ping -i 60 -a IP_address : Pings the provided IP_address, and gives audible sound when host comes alive.

31. tac : Prints content of a file, in reverse order.

32. strace : A debugging tool.

33. disown -a && exit Command : Run a command in background, even after terminal session is closed.

34. getconf LONG_BIT Command : Output Machine Architecture, very clearly.

35. while sleep 1;do tput sc;tput cup 0 $(($(tput cols)-29));date;tput rc;done & : The script outputs date and time on the top right corner of shell/ terminal.

36. convert : converts the output of a command in picture, automatically.

37. watch -t -n1 “date +%T|figlet” : Show animated digital clock at the prompt.

38. host and dig : DNS lookup utility.

39. dstat : Generates statistics regarding system resource.

40. bind -p : Shows all the shortcuts available in Bash.

41. Touch /forcefsck : Force file-system check on next boot.

42. lsb_release : Prints distribution specification information.

43. nc -ZV localhost port_number : Check if a specific port is open or not.

44. curl ipinfo.io : Outputs Geographical Information, regarding an ip_address.

45. find .-user xyz : Lists all file owned by user ‘xyz’

46. apt-get build-dep package_name: Build all the dependency, automatically while installing any specific package.

47. lsof -iTCP:80 -sTCP:LISTEN. The script, outputs all the service/process using port 80.

48. find -size +100M : This command combination, Lists all the files/folders the size of which is 100M or more.

49. pdftk : A nice way to concatenate a lot of pdf files, into one.

50. ps -LF -u user_name : Outputs Processes and Threads of a user.

51. Startx — :1 (This command creates another new X session).

top 6 files that eat up your space:

du -hsx * | sort -rh | head -6
  • Alt+Backspace: Deletes the previous word.
  • Alt+F: Skips ahead to the next space.
  • Alt+B: Skips back to the previous space.
  • Ctrl+U: Cuts all text up to the cursor.
  • Ctrl+K: Cuts all text after the cursor until end of line.
  • Ctrl+A: Moves the cursor to the start of line.
  • Ctrl+E: Moves the cursor to the end of line.