get_job_info() giving item not found error : python-jenkins

I am using python-jenkins and python 3.6

Trying to get the information regarding the jobs in jenkins but facing an error.

File “/usr/local/lib/python3.6/dist-packages/jenkins/init.py”,
line 359, in get_job_info
self._build_url(JOB_INFO, locals()) File “/usr/local/lib/python3.6/dist-packages/jenkins/init.py”, line
451, in jenkins_open
raise NotFoundException(‘Requested item could not be found’) jenkins.NotFoundException: Requested item could not be found

jobs = server.get_all_jobs()
for j in jobs:
    jobName = j['name'] # get job name
    print(jobName)
    lastJobId = getLastJobId(session, jobName) # get last locally stored job of this name
    print(lastJobId)
    lastBuildNumber = server.get_job_info(jobName)

get_all_jobs() is working fine but get_job_info() is not working.

Any help will be appreciated.

Solution:

You might need to use the fullname key instead of the name key.

jobName = j['fullname'] # get job name

There could be multiple jobs with the same job name, but in different folders/views. And fullname will fully qualify a specific job.

Advertisements

DynamoDB AutoIncrement Primary key Python flask

I am working on python flask with Dynamo DB ,is there any functionality to auto increment the primary key in AWS Dynamo DB (like SQL.AUTOINCREMENT), Any suggestion on this? how to handle the AUTOINCREMENT Feature in AWS DynamoDB.

Solution:

Generally speaking, using an auto-increment primary key is not going to be best practice. If you really want to do it, you could use a lambda function and another dynamodb table that stores the last used value, but you would be much better off picking a better primary key so that you don’t run into performance problems down the road.

Generally speaking, a GUID is a very easy to use alternative for a primary key where you don’t have another obvious field to use.

Aws::Errors::MissingCredentialsError in PostsController#update with Paperclip and AWS

I’ve been working on this for hours and haven’t been able to figure it out. I ran git clean and then realized that my s3.yml file got deleted. I created new keys and put them in a new s3.yml file. I haven’t changed anything else from before which was working just fine. What am I missing?

I’m getting: Aws::Errors::MissingCredentialsError (unable to sign request without credentials set):

Here is my code:

development.rb

  config.paperclip_defaults = {
    storage: :s3,
    s3_region: 'us-east-1',
    bucket:'mybucket',
    s3_credentials: "#{Rails.root}/config/s3.yml"
  }

production.rb

config.paperclip_defaults = {
      storage: :s3,
      s3_credentials: {
        bucket: ENV.fetch('S3_BUCKET_NAME'),
        access_key_id: ENV.fetch('AWS_ACCESS_KEY_ID'),
        secret_access_key: ENV.fetch('AWS_SECRET_ACCESS_KEY'),
        s3_region: ENV.fetch('AWS_REGION'),
      }
    }
    # Secret key base for non-existent secrets.yml file
    config.secret_key_base = ENV["SECRET_KEY_BASE"]

s3.yml (with new access keys)

S3_BUCKET_NAME: mybucket
AWS_ACCESS_KEY_ID: ***
AWS_SECRET_ACCESS_KEY: ***
AWS_REGION: us-east-1

Post class

class Post < ApplicationRecord
  belongs_to :user, counter_cache: true
  belongs_to :category

  validates :title, :presence => true
  validates :content, :presence => true

  has_attached_file :thumbnail, styles: {
      medium: '270x170#',
      large: '560x280#',
      large2: '540x340#'
    }

    # Validate the attached image is image/jpg, image/png, etc
    validates_attachment_content_type :thumbnail, :content_type => /\Aimage\/.*\Z/
end

Solution:

As you are getting the error on dev environment I think that the error is due to s3_credentials: “#{Rails.root}/config/s3.yml”.

Given the s3.yml you have posted I don’t think that s3_credentials will be filled with a json structure like the one you have in production environment.

I suggest you to load the file and fill the json like you do in your production environment.

What should be done when the provisioned throughput is exceeded?

I’m using AWS SDK for Javascript (Node.js) to read data from a DynamoDB table. The auto scaling feature does a great job during most of the time and the consumed Read Capacity Units (RCU) are really low most part of the day. However, there’s a programmed job that is executed around midnight which consumes about 10x the provisioned RCU and since the auto scaling takes some time to adjust the capacity, there are a lot of throttled read requests. Furthermore, I suspect my requests are not being completed (though I can’t find any exceptions in my error log).

In order to handle this situation, I’ve considered increasing the provisioned RCU using the AWS API (updateTable) but calculating the number of RCU my application needs may not be straightforward.

So my second guess was to retry failed requests and simply wait for auto scale increase the provisioned RCU. As pointed out by AWS docs and some Stack Overflow answers (particularlly about ProvisionedThroughputExceededException):

The AWS SDKs for Amazon DynamoDB automatically retry requests that receive this exception. So, your request is eventually successful, unless the request is too large or your retry queue is too large to finish.

I’ve read similar questions (this one, this one and this one) but I’m still confused: is this exception raised if the request is too large or the retry queue is too large to finish (therefore after the automatic retries) or actually before the retries?

Most important: is that the exception I should be expecting in my context? (so I can catch it and retry until auto scale increases the RCU?)

Solution:

Yes.

Every time your application sends a request that exceeds your capacity you get ProvisionedThroughputExceededException message from Dynamo. However your SDK handles this for you and retries. The default Dynamo retry time starts at 50ms, the default number of retries is 10, and backoff is exponential by default.

This means you get retries at:

  • 50ms
  • 100ms
  • 200ms
  • 400ms
  • 800ms
  • 1.6s
  • 3.2s
  • 6.4s
  • 12.8s
  • 25.6s

If after the 10th retry your request has still not succeeded, the SDK passes the ProvisionedThroughputExceededException back to your application and you can handle it how you like.

You could handle it by increasing throughput provision but another option would be to change the default retry times when you create the Dynamo connection. For example

new AWS.DynamoDB({maxRetries: 13, retryDelayOptions: {base: 200}});

This would mean you retry 13 times, with an initial delay of 200ms. This would give your request a total of 819.2s to complete rather than 25.6s.

How Can I use Python dictionary Grouping AWS instances by tag's name?

For example :
I have 50 AWS instances.
These instances’s name of key in tags is AP1 , AP2 , AP3 , AP4 , AP5

Now I want to use the Python dictionary Grouping AWS instances by tag’s name.
Like this : {AP1:[1,2,3….10] , AP2:[11,12…..20] , AP3:[21,22,….30], AP4:[31…..40] , AP5:[41,42,…..50]}

According to this article :
Obtaining tags from AWS instances with boto

I use this python script like this :

#!/usr/bin/env python
# -*- encoding: utf8 -*-

import boto.ec2
conn = boto.ec2.connect_to_region('us-west-1')
reservations = conn.get_all_instances()
InstanceMap={}
for reservation in reservations:
    for instance in reservation.instances:
        if 'Name' in instance.tags:
            InstanceMap[instance.tags['Name']].append(instance.id)

When I run this script, it show :
[root@Redhat script]# python group.py

Traceback (most recent call last):

File “group.py”, line 11, in

InstanceMap[instance.tags['Name']].append(instance.id)

KeyError: u’AP1′

What’s wrong with my script ?
Please provide me a correct python script.

Solution:

When you try to append to InstanceMap[instance.tags['Name']] you are trying to append to a list that the key (instance.tags['Name']) doesn’t exists in InstanceMap yet, as it is empty.
First, you need to check if that key already exists in InstanceMap, and if it does, use append. Else, create it.

#!/usr/bin/env python
# -*- encoding: utf8 -*-

import boto.ec2
conn = boto.ec2.connect_to_region('us-west-1')
reservations = conn.get_all_instances()
InstanceMap={}
for reservation in reservations:
    for instance in reservation.instances:
        if 'Name' in instance.tags:
            tag_name = instance.tags['Name']
            if tag_name in InstanceMap
                InstanceMap[tag_name].append(instance.id)
            else:
                InstanceMap[tag_name] = [instance.id,]

Cannot install ZeroMQ / 0mq / ØMQ properly

I want to play around with network programming using ZeroMQ

I’m trying to install it on my Linux Mint 18.2 machine. Maybe I’m just dumb (which is why I come here for help) but I can’t seem to get these instructions work.

Mainly following this, from the hyper-linked instructions:

To build on UNIX-like systems

If you have free choice, the most comfortable OS for developing with ZeroMQ is probably Ubuntu.

  1. Make sure that libtool, pkg-config, build-essential, autoconf, and
    automake are installed.
  2. Check whether uuid-dev package, uuid/e2fsprogs RPM or equivalent on
    your system is installed.
  3. Unpack the .tar.gz source archive.
  4. Run ./configure, followed by make.
  5. To install ZeroMQ system-wide run sudo make install.
  6. On Linux, run sudo ldconfig after installing ZeroMQ.

Using the tarball from release 4.2.2.

To test, I used the hello world example from 0mq (in examples/c++):

git clone --depth=1 https://github.com/imatix/zguide.git

I tried to compile hwclient.cpp using g++ but I get a bunch of errors, I assume because I cannot find the included zmq.hpp anywhere on my system (used locate zmq.hpp). Here are the errors:

/tmp/ccsb8olx.o: In function `zmq::error_t::error_t()':
hwclient.cpp:(.text._ZN3zmq7error_tC2Ev[_ZN3zmq7error_tC5Ev]+0x26): undefined reference to `zmq_errno'
/tmp/ccsb8olx.o: In function `zmq::error_t::what() const':
hwclient.cpp:(.text._ZNK3zmq7error_t4whatEv[_ZNK3zmq7error_t4whatEv]+0x16): undefined reference to `zmq_strerror'
/tmp/ccsb8olx.o: In function `zmq::message_t::message_t()':
hwclient.cpp:(.text._ZN3zmq9message_tC2Ev[_ZN3zmq9message_tC5Ev]+0x23): undefined reference to `zmq_msg_init'
/tmp/ccsb8olx.o: In function `zmq::message_t::message_t(unsigned long)':
hwclient.cpp:(.text._ZN3zmq9message_tC2Em[_ZN3zmq9message_tC5Em]+0x2e): undefined reference to `zmq_msg_init_size'
/tmp/ccsb8olx.o: In function `zmq::message_t::~message_t()':
hwclient.cpp:(.text._ZN3zmq9message_tD2Ev[_ZN3zmq9message_tD5Ev]+0x14): undefined reference to `zmq_msg_close'
/tmp/ccsb8olx.o: In function `zmq::message_t::data()':
hwclient.cpp:(.text._ZN3zmq9message_t4dataEv[_ZN3zmq9message_t4dataEv]+0x14): undefined reference to `zmq_msg_data'
/tmp/ccsb8olx.o: In function `zmq::context_t::context_t(int)':
hwclient.cpp:(.text._ZN3zmq9context_tC2Ei[_ZN3zmq9context_tC5Ei]+0x18): undefined reference to `zmq_init'
/tmp/ccsb8olx.o: In function `zmq::context_t::~context_t()':
hwclient.cpp:(.text._ZN3zmq9context_tD2Ev[_ZN3zmq9context_tD5Ev]+0x23): undefined reference to `zmq_term'
/tmp/ccsb8olx.o: In function `zmq::socket_t::socket_t(zmq::context_t&, int)':
hwclient.cpp:(.text._ZN3zmq8socket_tC2ERNS_9context_tEi[_ZN3zmq8socket_tC5ERNS_9context_tEi]+0x26): undefined reference to `zmq_socket'
/tmp/ccsb8olx.o: In function `zmq::socket_t::close()':
hwclient.cpp:(.text._ZN3zmq8socket_t5closeEv[_ZN3zmq8socket_t5closeEv]+0x26): undefined reference to `zmq_close'
/tmp/ccsb8olx.o: In function `zmq::socket_t::connect(char const*)':
hwclient.cpp:(.text._ZN3zmq8socket_t7connectEPKc[_ZN3zmq8socket_t7connectEPKc]+0x25): undefined reference to `zmq_connect'
/tmp/ccsb8olx.o: In function `zmq::socket_t::send(zmq::message_t&, int)':
hwclient.cpp:(.text._ZN3zmq8socket_t4sendERNS_9message_tEi[_ZN3zmq8socket_t4sendERNS_9message_tEi]+0x2b): undefined reference to `zmq_send'
hwclient.cpp:(.text._ZN3zmq8socket_t4sendERNS_9message_tEi[_ZN3zmq8socket_t4sendERNS_9message_tEi]+0x46): undefined reference to `zmq_errno'
/tmp/ccsb8olx.o: In function `zmq::socket_t::recv(zmq::message_t*, int)':
hwclient.cpp:(.text._ZN3zmq8socket_t4recvEPNS_9message_tEi[_ZN3zmq8socket_t4recvEPNS_9message_tEi]+0x2b): undefined reference to `zmq_recv'
hwclient.cpp:(.text._ZN3zmq8socket_t4recvEPNS_9message_tEi[_ZN3zmq8socket_t4recvEPNS_9message_tEi]+0x46): undefined reference to `zmq_errno'
collect2: error: ld returned 1 exit status

And for convenience, here is the hwclient.cpp code:

//
//  Hello World client in C++
//  Connects REQ socket to tcp://localhost:5555
//  Sends "Hello" to server, expects "World" back
//
#include <zmq.hpp>
#include <string>
#include <iostream>

int main ()
{
    //  Prepare our context and socket
    zmq::context_t context (1);
    zmq::socket_t socket (context, ZMQ_REQ);

    std::cout << "Connecting to hello world server..." << std::endl;
    socket.connect ("tcp://localhost:5555");

    //  Do 10 requests, waiting each time for a response
    for (int request_nbr = 0; request_nbr != 10; request_nbr++) {
        zmq::message_t request (5);
        memcpy (request.data (), "Hello", 5);
        std::cout << "Sending Hello " << request_nbr << "..." << std::endl;
        socket.send (request);

        //  Get the reply.
        zmq::message_t reply;
        socket.recv (&reply);
        std::cout << "Received World " << request_nbr << std::endl;
    }
    return 0;
}

I’m trying to figure out why zmq.hpp was not being installed.

Can anyone offer any advice on what I might be doing wrong?

Thanks.

Solution:

As discussed over the comments, you need to give the correct linking parameter to g++

g++ hwclient.cpp -lzmq

Explanation:
g++ compilation works in phases, the last of which is “linking”, where your file is linked with “libraries” containing the pre-compiled “definitions” (of for example, functions that you’re calling).

This is in contrast to the “pre-processing” stage that handles including of header files. Header files (most of the time) contain the “declarations” of functions.

Long story short:
if you see undefined reference to errors, you’re missing some library during linking.

Loop until choice within list

I am trying to write a script which loops until the user chooses a value within a list (single digit numbers from 0 to 9). This is my .sh script which I try to run in the ubuntu 16.04 shell using the sh command:

choice=999
echo $choice
until [[ $choice in 0 1 2 3 4 5 6 7 8 9 ]]
do
  read -p "How many would you like to add? " choice
done

No matter what I do, I just can’t get it to work. Here’s a test, to give you an idea of the errors at hand:

sh test2.sh
999
test2.sh: 3: test2.sh: [[: not found
How many would you like to add? f
test2.sh: 3: test2.sh: [[: not found
How many would you like to add? 2
test2.sh: 3: test2.sh: [[: not found
How many would you like to add? 3
test2.sh: 3: test2.sh: [[: not found
How many would you like to add? r
test2.sh: 3: test2.sh: [[: not found

I’ve tried so many things:

  • Avoiding until and using while
  • Using just singular square brackets [ condition ], or no brackets at all
  • Using =~ to match the regular expression ^[0-9]

Nothing works. Always that same error. What’s going on? 😦

Solution:

Firstly, your [[: not found suggests that you’re not using Bash. Either add a #!/bin/bash at the top of your script, or run it with bash test2.sh, or use the standard [.

Either way, you can’t use in like that. One alternative would be to use a case statement:

while :; do
  read -p "How many would you like to add? " choice
  case $choice in
    [0-9])
      break
      ;;
  esac
done

The nice thing about case statements is that they allow you to use glob patterns, so [0-9] matches any number from 0 to 9.

If you are planning on using Bash in the end, you can also go for something like this:

#!/bin/bash

until [[ $choice =~ ^[0-9]$ ]]; do
  read -p "How many would you like to add? " choice
done

Here, a regular expression is used to match a since digit from 0 to 9.