Prometheus pod consuming a lot of memory

I can use ‘–storage.local.memory-chunks’ to limit the memory usage when prometheus 1.X.

 

Prometheus 2.0 uses the OS page cache for data.  It will only use as much memory as it needs to operate. The good news is the memory use is far efficient than 1.x.  The amount needed to collect more data is minimal.

There is some extra memory needed if you do a large amount of queries, or queries that require a large amount of data.
You will want to monitor the memory use of the Prometheus process (process_resident_memory_bytes) and how much page cache the node has left (node_exporter, node_memory_Cached).

Continue reading Prometheus pod consuming a lot of memory

Advertisements

Why the address of the pointer variable printed differently between two printf statements without any modification to the variable?

In below simple c test program two printf statements return different values. ( check last four printf statements).

int main ()
{
    char c, *cc;
    int i;
    long l;
    float f;
    c = 'Z';
    i = 15;
    l = 7777;
    f = 9999999312;
    cc = &c;
    printf("\nc = %c, cc= %u", *cc, cc);
    cc = &i;
    printf("\nc = %d, cc= %u", *cc, cc);
    cc = &l;
    printf("\nc = %ld, cc= %u", *( long* )cc, cc);
    printf("\nc = %ld, cc= %u", *cc, cc);
    cc = &f;
    printf("\nc = %f, cc= %u", *(float *)cc, cc);
    printf("\n cc= %u", cc);
    printf("\nc = %f, cc= %u", *cc, cc);
    printf("\nc = %f, cc= %u", *(float *)cc, cc);

    printf("\nc = %f, cc using pointer  = %p", *(float *)cc, cc);
    printf("\nc = %f, cc using pointer  =%p", *cc, cc);

    return 0;
}

output : –

c = Z, cc= 755585903
c = 15, cc= 755585904
c = 7777, cc= 755585912
c = 97, cc= 755585912
c = 9999998976.000000, cc= 755585908
cc= 755585908
c = 9999998976.000000, cc= 4294967288
c = 9999998976.000000, cc= 755585908
c = 9999998976.000000, cc using pointer  = 0x7ffc37f4ace4
c = 9999998976.000000, cc using pointer  =0xfffffff8

I am running it in eclipse ide and using Ubuntu Linux.

Why does it behave differently?

Solution:

In several places you are using the wrong format specifier to printf. In particular, on this line:

printf("\nc = %f, cc= %u", *cc, cc);

The first parameter has type char, but you use %f which expects a double. Also, the second parameter has type char * but %u is expecting an unsigned int. Using the wrong format specifiers invokes undefined behavior.

That being said, here is what is most likely happening under the hood:

On most hosted implementations, floating point numbers are not pushed onto the stack the way integer types and pointers are, but are instead stored in a floating point register.

When you run the above printf command, both *cc and cc are pushed onto the stack because neither of them are floating point numbers. When printf then looks for the first parameter it sees %f, so it retrieves a value from a floating point register. Since you did actually pass a floating point value the prior time you called printf that value is still there, and that value happens to be the value you actually wanted to print, so that’s what gets printed.

Then when printf goes to print the next parameter, it sees %u in the format so the value is pulled from the stack. That value is *cc, which points to the first byte in the representation of f.

Assuming a float is a IEEE754 single precision floating point number, the value it contains is represented as 0xf8021550. The first byte of that is 0xf8. Since cc points to a char, which in your case appears to be signed, it is interpreted as a negative value. When passed to printf the value is promoted to type int, so the actual value passed in is 0xfffffff8, which is what you see being printed.

To reiterate however, the output you’re seeing is undefined behavior. The output could change if you build it on a different machine, use a different compiler, or just use a different optimization setting.

How to switch gdb byte output grouping

When I try to see what’s inside stack space, I use the following command:

x/100x $sp

However, sometimes output is formatted like this, grouping by 4 bytes:

0xbffff0ac: 0x00000000  0xb7fbc000  0xb7fbc000  0xbffff4e8
...

While sometimes I get this:

0xbffff0ac: 00 00 00 00 00 c0 fb b7 00 c0 fb b7 e8 f4 ff bf

But I can’t determine how to switch between these formats and how gdb desides which format to use for output. Any suggestions?

Solution:

sometimes output is formatted like this
However, sometimes output is formatted like this

This is because the x command remembers the last size you used.

If you want particular size with your x, just specify it directly:

(gdb) x/100wx $sp

Documentation.

Are elements of ArrayList in Java reallocated in the memory when new ones are added If the size of the ArrayList has been passed to the constructor?

I am trying to write the method that efficiently wraps each element in the List passed to this method and returns the created ArrayList with wrapped elements.

According to the documentation:

The size(), isEmpty(), get(), set(), iterator(), and listIterator() operations run in constant time. The add operation runs in amortized constant time, that is, adding n elements requires O(n) time. All of the other operations run in linear time (roughly speaking). The constant factor is low compared to that for the LinkedList implementation.

Do I understand it right that If I create an ArrayList and pass the initial capacity to the constructor, the elements in ArrayList won’t be reallocated in memory when new ones are added?

Example:

public static <T> ArrayList<RequestToExternalSource<T>> wrapExternalSources(List<ExternalSource<T>> externalSources, BiConsumer<Integer, T> publishResult) {
    ArrayList<RequestToExternalSource<T>> requests = new ArrayList<>(externalSources.size());

    ListIterator<ExternalSource<T>> externalSourcesIterator = externalSources.listIterator();
    int index = 0;

    while (externalSourcesIterator.hasNext()) {
        requests.add(new RequestToExternalSource<>(
                index++,
                externalSourcesIterator.next(),
                publishResult));
    }
    return requests;
}  

Solution:

To answer this, we can look directly at the source code of ArrayList#add. We first see the following method:

public boolean add(E e) {
    modCount++;
    add(e, elementData, size);
    return true;
}

The method above calls the following private, overloaded add method:

private void add(E e, Object[] elementData, int s) {
    if (s == elementData.length)
        elementData = grow();
    elementData[s] = e;
    size = s + 1;
}

We can see that elementData (the Object[] that holds the data) will only grow when s (the size parameter, equal to ArrayList#size in our case) equals the length of the data array. For this reason, elementData is not grown even if we add n elements to an ArrayList initialized with a capacity of n, which is good!

Do I understand it right that If I create an ArrayList and pass the initial capacity to the constructor, the elements in ArrayList won’t be reallocated in memory when new ones are added?

For these reasons, yes, you’re correct, until you add more elements than the capacity specified.

How can I get a guarantee that when a memory is freed, the OS will reclaim that memory for it's use?

I noticed that this program:

#include <stdio.h>

int main() {
  const size_t alloc_size = 1*1024*1024;
  for (size_t i = 0; i < 3; i++) {
    printf("1\n");
    usleep(1000*1000);
    void *p[3];
    for (size_t j = 3; j--; )
      memset(p[j] = malloc(alloc_size),0,alloc_size); // memset for de-virtualize the memory
    usleep(1000*1000);
    printf("2\n");
    free(p[i]);
    p[i] = NULL;
    usleep(1000*1000*4);
    printf("3\n");
    for (size_t j = 3; j--; )
      free(p[j]);
  }
}

which allocates 3 memories, 3 times and each time frees different memory, frees the memory according to watch free -m, which means that the OS reclaimed the memory for every free regardless of the memory’s position inside the program’s address space. Can I somehow get a guarantee for this effect? Or is there already anything like that (like a rule of >64KB allocations)?

Solution:

The short answer is: In general, you cannot guarantee that the OS will reclaim the freed memory, but there may be an OS specific way to do it or a better way to ensure such behavior.

The long answer:

  • Your code has undefined behavior: there is an extra free(p[i]); after the printf("2\n"); which accesses beyond the end of the p array.

  • You allocate large blocks (1 MB) for which your library makes individual system calls (for example mmap in linux systems), and free releases these blocks to the OS, hence the observed behavior.

  • Various OSes are likely to implement such behavior for a system specific threshold (typically 128KB), but the C standard gives guarantee about this, so relying on such behavior is system specific.

  • Read the manual page for malloc() on your system to see if this behavior can be controlled. For example, the C library on Linux uses an environment variable MMAP_THRESHOLD to override the default setting for this threshold.

  • If you program to a Posix target, you might want to use mmap() directly instead of malloc to guarantee that the memory is returned to the system once deallocated with munmap(). Note that the block returned by mmap() will have been initialized to all bits zero before the first access, so you may avoid such explicit initialization to take advantage of on demand paging, on perform explicit initialization to ensure the memory is mapped to try and minimize latency in later operations.

Shared memory segment with different varable types C

Would be grateful if anyone could help me out with the problem I’m facing. I’m using Linux by the way.

Using shared memory to store a static array of structs between two different programs.

Here’s a code snippet showing how I create the shared memory block.

typedef struct {
 int ID;
 int nData;
 int time;
} strPrintJob;

size = (sizeof(strPrintJob) * lRetMaxJobs) + (sizeof(int) * 2);
strPrintJob *shmPrintJob;

    //Create data segment
    if((nShmid = shmget(nKey, size, IPC_CREAT | 0666)) < 0)
    {
        perror("shmget");
        exit(1);
    }

    shmPrintJob = (strPrintJob *) shmat(nShmid, NULL, 0);
    if (shmPrintJob == (strPrintJob *)(-1))
    {
        perror("shmat");
        exit(1);
    }

So far everything is working fine, and the two programs communicate: One modifying data within the structs and the other printing it out.

I would also like to use two integers within the shared memory to act as ‘flags’ but how would I attach and access them?
Something along the lines of?

int *shmnFlagOne, *nPtr;

if((shmnFlagOne = shmat(nShmid, NULL, 0)) == -1)
{
   perror("shmat");
   exit(1);
}

nPtr = shmnFlagOne;

And then set the pointer to go after the array of structs in the shared memory?

Sorry if I’m getting things completely muddled up here, tried to do as much on my own before asking on here. Thanks for taking a look.

Solution:

You are right on track. You can place these items within your memory as you see fit. It looks like you have already allocated the storage for your flags with

2 * sizeof(int)

You can access them many ways all of which involve pointer casting.

It would look something like:

void *shared_memory = shmat(nShmid, NULL, 0);
strPrintJob *shmPrintJob = (strPrintJob *) shared_memory;
int *flags = (int *) shared_memory[sizeof(strPrintJob) * lRetMaxJobs / sizeof(int)];

OK, totally gross. But that would be the minimal change.
An alternative is to create another struct to wrap your struct:

typedef struct {
    strPrintJob[lRetMaxJobs] printJobs;
    int flags[2];
} PrintJobWrapper;

Now use PrintJobWrapper to access your shared memory as you were before:

PrintJobWrapper *print_wrapper = (PrintJobWrapper *) shmat(nShmid, NULL, 0);
print_wrapper->flags[0] = xxx;

Install and configure chef server on centos 7

  1. Go to this link: https://downloads.chef.io/chef-server
  2. In the on-premise section select your OS and download your package.
  3. Here I will be using centos 7. When the rpm is downloaded you can install it via the command:
    sudo rpm -Uvh
  4. Execute the following command: chef-server-ctl reconfigure
    Because the Chef server is composed of many different services that work together to create a functioning system, this step may take a few minutes to complete.
  5. Run the following command to create an administrator:
    $ chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename FILE_NAME
    

    An RSA private key is generated automatically. This is the user’s private key and should be saved to a safe location. The --filename option will save the RSA private key to the specified absolute path.

    For example:

    $ chef-server-ctl user-create stevedanno Steve Danno steved@chef.io 'abc123' --filename /path/to/stevedanno.pem

    If you get the following error:

    Specified config file /etc/opscode/pivotal.rb does not exist

    Then check if you have RAM available. Free some memory and rerun the

    chef-server-ctl reconfigure

  6. If you get following error: /users resource does not exist

    Then check the chef logs with following command:

    chef-server-ctl tail

    You might see an error similar to the following:
    ==> /var/log/opscode/nginx/error.log <==
    2017/09/08 11:38:28 [emerg] 29689#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
    2017/09/08 11:38:28 [emerg] 29689#0: bind() to 0.0.0.0:443 failed (98: Address already in use).

    Then execute the following command:
    netstat -ntlp | grep 80
    Check which service is using the port 80 and 443. Majorly it is the httpd service. Stop the httpd service. Then the chef nginx service will start automatically without any issues.

    Then execute the create-user command it will get executed without any error.

  7. Run the following command to create an organization:
    $ chef-server-ctl org-create short_name 'full_organization_name' --association_user user_name --filename ORGANIZATION-validator.pem
    

    The name must begin with a lower-case letter or digit, may only contain lower-case letters, digits, hyphens, and underscores, and must be between 1 and 255 characters. For example: 4thcoffee.

    The full name must begin with a non-white space character and must be between 1 and 1023 characters. For example: 'Fourth Coffee, Inc.'.

    The --association_user option will associate the user_name with the admins security group on the Chef server.

    An RSA private key is generated automatically. This is the chef-validator key and should be saved to a safe location. The --filename option will save the RSA private key to the specified absolute path.

    For example:

    $ chef-server-ctl org-create 4thcoffee 'Fourth Coffee, Inc.' --association_user stevedanno --filename /path/to/4thcoffee-validator.pem
    
  8. Enable additional features of the Chef server! The packages may be downloaded directly as part of the installation process or they may be first downloaded to a local directory, and then installed.Use DownloadsThe install subcommand downloads packages from https://packages.chef.io/ by default. For systems that are not behind a firewall (and have connectivity to https://packages.chef.io/), these packages can be installed as described below.
    Feature Command
    Chef Manage

    Use Chef management console to manage data bags, attributes, run-lists, roles, environments, and cookbooks from a web user interface.

    On the Chef server, run:

    $ chef-server-ctl install chef-manage
    

    then:

    $ chef-server-ctl reconfigure
    

    and then:

    $ chef-manage-ctl reconfigure
    

    Note

    Starting with the Chef management console 2.3.0, the Chef MLSA must be accepted when reconfiguring the product. If the Chef MLSA has not already been accepted, the reconfigure process will prompt for a yes to accept it. Or run chef-manage-ctl reconfigure --accept-license to automatically accept the license.

  9. Open the chef-manage UI in the browser with https:///login
    Login with the credentials of the user you just created in the above step.
    On the nodes panel if you see the following error:  Error An error occurred, please try again
    Then look at the chef logs with the command chef-server-ctl tail
    The error will be majorly with the nginx. The error that I was facing was as follows:
    FAILED SocketConnector@127.0.0.1:8983: java.net.BindException: Address already in use (Bind failed). The issue was that solr service was already running on my server before installing chef. I had to stop the solr service, the issue was resolved. I was able to see the nodes panel without any error.2017-09-11 11_03_01-Mail
  10. Chef Push Jobs

    Use Chef push jobs to run jobs—an action or a command to be executed—against nodes independently of a chef-client run.

    On the Chef server, run:

    $ chef-server-ctl install opscode-push-jobs-server
    

    then:

    $ chef-server-ctl reconfigure
    

    and then:

    $ opscode-push-jobs-server-ctl reconfigure
  11. Reporting

    Use Reporting to keep track of what happens during every chef-client runs across all of the infrastructure being managed by Chef. Run Reporting with Chef management console to view reports from a web user interface.

    On the Chef server, run:

    $ chef-server-ctl install opscode-reporting
    

    then:

    $ chef-server-ctl reconfigure
    

    and then:

    $ opscode-reporting-ctl reconfigure
  12. If you want to login into postgres database of chef and see the data present in there execute the following command:
    chef-server-ctl psql
    You will get an output similar to:
    [ERROR] You must supply a service name. Valid names include: bifrost, bookshelf, oc-id, oc_erchef, oc_id, opscode-erchef, opscode_chef, push-jobs, reporting
    This is the list of databases you can login and see the data from.
    The main database where the chef data is present is opscode-erchef (previously opscode_chef). You can login to the database with the following command:
    chef-server-ctl psql opscode-erchef
    To see the tables in the database execute the following psql command:
    \dt
    This will give you a list as below:
    Schema | Name | Type | Owner
    ——–+——————————————–+——-+—————
    public | checksums | table | opscode-pgsql
    public | clients | table | opscode-pgsql
    public | containers | table | opscode-pgsql
    public | cookbook_artifact_version_checksums | table | opscode-pgsql
    public | cookbook_artifact_versions | table | opscode-pgsql
    public | cookbook_artifacts | table | opscode-pgsql
    public | cookbook_version_checksums | table | opscode-pgsql
    public | cookbook_versions | table | opscode-pgsql
    public | cookbooks | table | opscode-pgsql
    public | data_bag_items | table | opscode-pgsql
    public | data_bags | table | opscode-pgsql
    public | environments | table | opscode-pgsql
    public | groups | table | opscode-pgsql
    public | keys | table | opscode-pgsql
    public | node_policy | table | opscode-pgsql
    public | nodes | table | opscode-pgsql
    public | opc_customers | table | opscode-pgsql
    public | opc_users | table | opscode-pgsql
    public | org_migration_state | table | opscode-pgsql
    public | org_user_associations | table | opscode-pgsql
    public | org_user_invites | table | opscode-pgsql
    public | orgs | table | opscode-pgsql
    public | policies | table | opscode-pgsql
    public | policy_groups | table | opscode-pgsql
    public | policy_revisions | table | opscode-pgsql
    public | policy_revisions_policy_groups_association | table | opscode-pgsql
    public | roles | table | opscode-pgsql
    public | sandboxed_checksums | table | opscode-pgsql
    public | users | table | opscode-pgsqlYou can see the data in every table with the follwoing command:
    select * from <table-name>;
  13. If you want to see all the settings and configurations chef is using you can see the file:
    vi /etc/opscode/chef-server-running.json
  14. Find chef version:
    head -n1 /opt/opscode/version-manifest.txt
  15. Go to the chef dashboard -> Administration-> Organizations-> Starter Kit – > download starter kit.
    unzip chef-starter.zip
    Then unzip the starter kit in workstation. You can also use your chef server for this purpose. A chef-repo directory will be created.
    cd chef-repo/.chef
    cat knife.rb
    Check the configurations in knife.rb file.
  16. Install chefdk
    rpm -ivh https://packages.chef.io/files/stable/chefdk/2.2.1/el/7/chefdk-2.2.1-1.el7.x86_64.rpm
  17. Execute the following command
    knife ssl fetch 
    If you get an permission denied error then execute the command with sudo privileges.
    After execuing the above command you will get an output similar to the above:
    WARNING: Certificates from vrushabh.novalocal will be fetched and placed in your trusted_cert
    directory (/root/chef-repo/.chef/trusted_certs)……
  18. Knife is used to create cookbooks and to upload the cookbooks to chef server, upload roles, runlist etc.
    Knife allows us to communicate with our chef server.
  19. Lets bootstrap our node to chef server. For this you will need another machine which is accessible from the machine you have unzipped the chef-repo directory. Create a user in the node machine with sudo privileges, which can be ‘ssh’ed from our workstation and chef server.
    Execute the command:
    knife bootstrap 192.168.1.240 -N chef-node –ssh-user user1 –sudo
    Where ‘chef-node’ is the node name i am giving to the node. This can be any name. If -N option is not provided the default nodename will be the hostname of the node.
    ‘user1’ is the user I created on node.
    You might get an error: Your private key could not be loaded.
    cd chef-repo then execute the bootstrap command.
    You will see an output similar to the following:
    ……
    192.168.1.240 Converging 0 resources
    192.168.1.240
    192.168.1.240 Running handlers:
    192.168.1.240 Running handlers complete
    192.168.1.240 Chef Client finished, 0/0 resources updated in 16 seconds
  20. Then goto the chef dashboard and on the nodes panel page you can see the newly added node. In the attributes panel you can see the attributes belonging to the node like CPU,RAM etc.
  21. Create a project in github(ex: chef-fluency-badge). Then cd chef-repo and execute the following command:
    git init
    git add ./
    git config –global user.email “”
    git config –global user.name “”
    git commit -am “Adding chef-repo”
    git remote add origin https://github.com//chef-fluency-badge.git
    git push -u origin master
  22. Create a cookbook with the followig command:
    knife cookbook create learn for chef < 12
    chef generate cookbook for chef => 12
    Add this learn cookbook to git repo with the following commands:
    git add ./
    git commit -m “adding learn cookbook”
    git push -u origin master
  23. Lets create apache cookbook:
    cd chef-repo
    chef generate cookbook cookbooks/apache
    cd cookbooks/apache
    Edit the metadata.rb file and enter your details. Save and close.
    cd recipes
    Edit default.rb and add the following lines:
    ##

    package 'apache2' do
            package_name 'httpd'
            action :install
    end
    
    service 'apache2' do
            service_name 'httpd'
            action [:start, :enable]
    end
    
    
  24. Check the ruby syntax by running the command:
    ruby -c default.rb
    foodcritic default.rb 

    foodcritic should not give any error message.
    Now we want to change the default apache webpage.
    In the same recipes directory create websites.rb with following content:
    ##

    file "default www" do
            path '/var/www/html/index.html'
            content 'Hello world'
    end
    &lt;span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			&gt;&amp;#65279;&lt;/span&gt;

    Check it with foodcritic. You can also comeback to cookbooks directory and execute:
    foodcritic apache

  25. Upload the cookbook to the chef server with the following command:
    knife cookbook upload apache
    You can also add this code to git repo.
  26. Then goto the chef-manage UI, in the policies section you will see the apache cookbook. Policy is basically configuration management for a node.
  27. In your workstation machine execute the knife node list command to get a list of nodes.
  28. Then we need to add our cookbook to run_list. Execute the following command for it:
    knife node run_list add chef-node ‘recipe[apache]’

    Then execute the following command to retreive the list of run_list:
    knife node show chef-node
    The above command gives a summary of the node like recipes, OS version etc. To get a detailed list execute the following command:
    knife node show -l chef-node

  29. Now we will dry-run our cookbook on chef-client:
    chef-client –why-run
    OR
    chef-client -W
  30. After the why-run is successful then execute the cookbook with the following command:
    chef-client

    You can check the httpd service status, it should be running. Then go to the browser and hit the <IP> of the client. It will not show our “Hello World” page but the default apache “Testing 123” page.
    This was because our websites recipe did not execute. Edit the default.rb and add the following line:
    include_recipe ‘apache::websites’

  31. Then upload the apache cookbook to chef-server:
    knife cookbook upload apache
  32. Then run chef-client your websites recipe will get executed.
  33. Suppose if we want to run the websites recipe manually everytime.
    Edit the websites.rb file and remove the include_recipe line we just added above. Then again upload the cookbook to the chef-server.
  34. Then we need to add the websites recipe to the run_list otherwise after running chef-client the websites recipe will not get executed.
    You can add the recipe with the following command:
    knife node run_list add chef-node ‘recipe[apache::websites]’ -a recipe[apache]
    or you can also execute knife node run_list add ‘apache’
  35. Now you can run chef-client it will execute the websites recipe.
  36. The client configuration can be found at /etc/chef/client.rb on the client machine.