ansible: with_items key and value

- name: Tuning sysctl
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
  with_items:
    - { key: net.core.somaxconn, value: 65536 }
Advertisements

AWS S3 list keys containing a string

I am using python in AWS Lambda function to list keys in a s3 bucket that contains a specific id

for object in mybucket.objects.all():
            file_name = os.path.basename(object.key)
            match_id = file_name.split('_', 1)[0]

The problem is if a s3 bucket has several thousand files the iteration is very inefficient and sometimes lambda function times out

Here is an example file name

https://s3.console.aws.amazon.com/s3/object/bucket-name/012345_abc_happy.jpg

i want to only iterate objects that contains “012345” in the key name
Any good suggestion on how i can accomplish that

Solution:

Here is how you need to solve it.

S3 stores everything as objects and there is no folder or filename. It is all for user convenience.

aws s3 ls s3://bucket/folder1/folder2/filenamepart –recursive

will get all s3 objects name that matches to that name.

import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucketname')
for obj in my_bucket.objects.filter(Prefix='012345'):
    print(obj)

To speed up the list you can run multiple scripts parallelly.

Hope it helps.

Saltstack and Vault integration

First install and configure vault using this tutorial:
https://apassionatechie.wordpress.com/2017/03/05/hashicorp-vault/

Use the latest version of vault.

Then install salt using the steps given here:
https://docs.saltstack.com/en/latest/topics/installation/

If you face any issues then refer these links:
https://apassionatechie.wordpress.com/2017/07/31/salt-issues/

https://apassionatechie.wordpress.com/2017/08/03/salt-stack-formulas/

Now let’s integrate vault and salt so that we can access vault secrets from inside salt state.

    1. First let’s add some key values into our vault.
      vault write secret/ssh/user1 password=”abc123″
      Then you can check it by reading: vault read secret/ssh/user1
    2. To allow salt to access your secrets you must firstly create a policy as follows:
      salt-policy.hcl

      path "secret/*" {
        capabilities = ["read", "list"]
      }
      
      path "auth/*" {
        capabilities = ["read", "list","sudo","create","update","delete"]
      }
      

      You can also point to your secret like secret/ssh/*
      We have added auth/* so that our token can create other tokens.

    3. Then create a new policy with the following command:
      vault policy-write salt-policy salt-policy.hcl
    4. Then we will create a token from the new salt-policy
      vault token-create -policy=salt-policy
      Save the token created.
    5. Then in the salt-master create a file:
      /etc/salt/master.d/vault.conf with the follwoing contents:

      vault:
        url: http://127.0.0.1:8200
        auth:
          method: token
          token: xxxxxx48-xxxx-xxxx-xxxx-xxxx1xxxx<span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			></span>c4a
        policies:
            - salt-policy
      
      

      Then create a file /etc/salt/master.d//peer_run.conf

      peer_run:
          .*:
              - vault.generate_token
      
      

      Then restart the salt-master with service salt-master restart

    6. Then execute the following command to access the secret stored in vault:
      salt ‘*’ vault.read_secret “secret/ssh/user1”
    7. To access the secret from inside jinja:
      my-secret: {{ salt[‘vault’].read_secret(‘secret/ssh/user1’, ‘password’) }}
      OR
      {% set supersecret = salt[‘vault’].read_secret(‘secret/ssh/user1’) %}
      secrets:
          my_secret: {{ supersecret.password }}
    8. If you want to access the secret as pillar then add the following in salt master configuration:
      ext_pillar:
       – vault: sdb_vault path=secret/ssh/user1
      Restart the salt-master and salt-minion
      Then access the data with the following command:
      salt ‘*’ pillar.get ‘password’
      Then refresh the pillar data with: salt ‘*’ saltutil.refresh_pillar
    9. If your vault policy is not configured correctly you might get an error as:
      ERROR: {‘error’: ‘Forbidden’}
      2017-09-21 06:51:39,320 [salt.loaded.int.utils.vault][ERROR ][26333] Failed to get token from master! An error was returned: Forbidden
      2017-09-21 06:51:39,350 [salt.pillar ][ERROR ][26333] Execption caught loading ext_pillar ‘vault’:
      File “/usr/lib/python2.7/site-packages/salt/pillar/__init__.py”, line 822, in ext_pillar
      key)
      File “/usr/lib/python2.7/site-packages/salt/pillar/__init__.py”, line 765, in _external_pillar_data
      val)
      File “/usr/lib/python2.7/site-packages/salt/pillar/vault.py”, line 91, in ext_pillar
      response = __utils__[‘vault.make_request’](‘GET’, url)
      File “/usr/lib/python2.7/site-packages/salt/utils/vault.py”, line 124, in make_request
      connection = _get_vault_connection()
      File “/usr/lib/python2.7/site-packages/salt/utils/vault.py”, line 113, in _get_vault_connection
      return _get_token_and_url_from_master()
      File “/usr/lib/python2.7/site-packages/salt/utils/vault.py”, line 89, in _get_token_and_url_from_master
      raise salt.exceptions.CommandExecutionError(result)2017-09-21 06:51:39,351 [salt.pillar ][CRITICAL][26333] Pillar render error: Failed to load ext_pillar vault: {‘error’: ‘Forbidden’}

      Make sure you have added auth/* in the policy.

    10. If you get the following error:
      Failed to get token from master! No result returned – is the peer publish configuration correct?
      OR
      ERROR: {}
      Then make sure you have peer_run.conf created and configured.
    11. You can also access your secret with command:
      salt-call sdb.get ‘sdb://vault/secret/ssh/user1?password’