AWS S3 Can't do anything with one file

I’m having issues trying to remove a file from my s3 bucket with the following name: Patrick bla bla 1 PV@05-06-2018-19:42:01.jpg

If I try to rename it through the s3 console, it just says that the operation failed. If I try to delete it, the operation will “succeed” but the file will still be there.

I’ve tried removing it through the aws cli, when listing the object I get this back

 {
        "LastModified": "2018-06-05T18:42:05.000Z",
        "ETag": "\"b67gcb5f8166cab8145157aa565602ab\"",
        "StorageClass": "STANDARD",
        "Key": "test/\bPatrick bla bla 1 PV@05-06-2018-19:42:01.jpg",
        "Owner": {
            "DisplayName": "dev",
            "ID": "bd65671179435c59d01dcdeag231786bbf6088cb1ca4881adf3f5e17ea7e0d68"
        },
        "Size": 1247277
    },

But if I try to delete or head it, the cli won’t find it.

s3api head-object --bucket mybucket --key "test/\bPatrick bla bla 1 PV@05-06-2018-20:09:37.jpg"

An error occurred (404) when calling the HeadObject operation: Not Found

Is there any way to remove, rename or just move this image from the folder?

Regards

Solution:

It looks like your object’s key begins with a backspace (\b) character. I’m sure there is a way to manage this using the awscli but I haven’t worked out what it is yet.

Here’s a Python script that works for me:

import boto3 
s3 = boto3.client('s3')
Bucket ='avondhupress'
Key='test/\bPatrick bla bla 1 PV@05-06-2018-19:42:01.jpg'
s3.delete_object(Bucket=bucket, Key=key)

Or the equivalent in node.js:

const aws = require('aws-sdk');
const s3 = new aws.S3({ region: 'us-east-1', signatureVersion: 'v4' });

const params = {
  Bucket: 'avondhupress',
  Key: '\bPatrick bla bla 1 PV@05-06-2018-19:42:01.jpg',
};

s3.deleteObject(params, (err, data) => {
  if (err) console.error(err, err.stack);
});
Advertisements

How can I use same tasks yum with CentOS and dnf with Fedora on Ansible?

I use CentOS 7 on work and Fedora 28 on hobby, and I’m making an ansible’s playbook that installs some pakcage for them.

But CentOS uses yum and Fedora uses dnf. I know there exists yum module and dnf module, but separeted.

I’d like to write simple, how can I solve it? Could you please tell me.

Solution:

You could use the package module. It’ll sort out dnf vs yum behind the scenes. https://docs.ansible.com/ansible/latest/modules/package_module.html#package-module

Run script command on parallel

i’ve bash script which I need to run on it two command in parallel

For example I’m executing a command of npm install which takes some time (20 -50 secs)

and I run it on two different folders in sequence first npm install on books folder and the second
is for orders folder, is there a way to run both in parallel in shell script ?

For example assume the script is like following:

#!/usr/bin/env bash

   dir=$(pwd)

  cd $tmpDir/books/  

  npm install

  grunt

  npm prune production 
  cd $tmpDir/orders/

  npm install

  grunt

 npm prune production 

Solution:

You could use & to run the process in the background, for example:

#!/bin/sh

cd $HOME/project/books/
npm install &

cd $HOME/project/orders/
npm install &

# if want to wait for the processes to finish
wait

To run and wait for nested/multiple processes you could use a subshell () for example:

#!/bin/sh

(sleep 10 && echo 10 && sleep 1 && echo 1) &

cd $HOME/project/books/
(npm install && grunt && npm prune production ) &

cd $HOME/project/orders/
(npm install && grunt && npm prune production ) &

# waiting ...
wait

In this case, notice the that the commands are within () and using && that means that only the right side will be evaluated if the left size succeeds (exit 0) so for the example:

(sleep 10 && echo 10 && sleep 1 && echo 1) &
  • It creates a subshell putting things between ()
  • runs sleep 10 and if succeeds && then runs echo 10, if succeeds && then run sleep 1 and if succeeds && then runs echo 1
  • run all this in the background by ending the command with &

Check if All Values Exist as Keys in Dictionary

I have a list of values, and a dictionary. I want to ensure that each value in the list exists as a key in the dictionary. At the moment I’m using two sets to figure out if any values don’t exist in the dictionary

unmapped = set(foo) - set(bar.keys())

Is there a more pythonic way to test this though? It feels like a bit of a hack?

Solution:

Your approach will work, however, there will be overhead from the conversion to set.

Another solution with the same time complexity would be:

all(i in bar for i in foo)

Both of these have time complexity O(len(foo))

bar = {str(i): i for i in range(100000)}
foo = [str(i) for i in range(1, 10000, 2)]

%timeit all(i in bar for i in foo)
462 µs ± 14.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit set(foo) - set(bar)
14.6 ms ± 174 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# The overhead is all the difference here:

foo = set(foo)
bar = set(bar)

%timeit foo - bar
213 µs ± 1.48 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The overhead here makes a pretty big difference, so I would choose all here.

IndentationError: unexpected indent after comment

I am trying to write some Python example code with a line commented out:

user_by_email = session.query(User)\
    .filter(Address.email=='one')\
    #.options(joinedload(User.addresses))\
    .first()

I also tried:

user_by_email = session.query(User)\
    .filter(Address.email=='one')\
#    .options(joinedload(User.addresses))\
    .first()

But I get IndentationError: unexpected indent.
If I remove the commented out line, the code works.
I am decently sure that I use only spaces (Notepad++ screenshot):

enter image description here

Solution:

Enclose the statement in paranthesis

user_by_email = (session.query(User)
     .filter(Address.email=='one')
     #.options(joinedload(User.addresses))
     .first())

Fastest way to compute 9^(9^9) exactly all digits?

Is there a way to exactly compute all of the ca. 370 million decimal digits of 9^(9^9) very fast? I used an out of the box bignumber algorithms library(*) which took 16 minutes.

(*) I used Java BigInteger the pow() method:

?- time((_ is 9^(9^9))).
% Up 974,318 ms, GC 9,302 ms, Thread Cpu 962,688 ms (Current 06/01/18
19:54:01)
Yes

?- statistics.
Max Memory           7,635,730,432 Bytes
Used Memory           765,913,440 Bytes
Free Memory          2,891,739,304 Bytes
Uptime                 2,466,670 Millis
GC Time                    9,315 Millis
Thread Cpu Time          963,812 Millis
Current Time          06/01/18 20:18:37 

The Java BigInteger implementation uses Karatsuba and Toom Cook:
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/math/BigInteger.java

P.S.: To additionally output the ca. 370 million decimal digits, if it would use 1 ms per digit, would only need 370’000 ms. Thats a third of the time I needed to compute the exact digits in the above, i.e. 974,318 ms without displaying.

Solution:

Try https://raw.githubusercontent.com/tbuktu/bigint/master/src/main/java/java/math/BigInteger.java for an improved BigInteger that switches to Schoenhage-Strassen once integers get past 74,000 digits. My back of the envelope says that that should be an order of magnitude faster after you get to hundreds of millions of digits.

Crash when trying to run WebGoat with a Java Agent

I am currently learning how to boot a web application with a java agent for monitoring.

The Web Application I chose was WebGoat, and running WebGoat with java -jar webgoat-server-8.0.0.M17.jar as stated in WebGoat’s README works perfectly fine.

However, when I try to add my agent, I get the following mess of an error log:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v1.5.12.RELEASE)

2018-06-06 22:36:08.528  INFO 3741 --- [           main] org.owasp.webgoat.StartWebGoat           : Starting StartWebGoat v8.0.0.M17 on MacBook-Pro.local with PID 3741 (/Users/andrewfan/Desktop/Lang Agent Dev Proj help info/webgoat-server-8.0.0.M17.jar started by andrewfan in /Users/andrewfan/Desktop/Lang Agent Dev Proj help info)
2018-06-06 22:36:08.531  INFO 3741 --- [           main] org.owasp.webgoat.StartWebGoat           : No active profile set, falling back to default profiles: default
2018-06-06 22:36:08.844  INFO 3741 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@1376c05c: startup date [Wed Jun 06 22:36:08 EDT 2018]; root of context hierarchy
2018-06-06 22:36:11.354  INFO 3741 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$8e12590a] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2018-06-06 22:36:11.442  WARN 3741 --- [           main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mvcConfiguration': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'webgoat.user.directory' in value "${webgoat.user.directory}"
2018-06-06 22:36:11.455  INFO 3741 --- [           main] utoConfigurationReportLoggingInitializer : 

Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2018-06-06 22:36:11.464 ERROR 3741 --- [           main] o.s.boot.SpringApplication               : Application startup failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mvcConfiguration': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'webgoat.user.directory' in value "${webgoat.user.directory}"
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:372) ~[spring-beans-4.3.16.RELEASE.jar!/:4.3.16.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1268) ~[spring-beans-4.3.16.RELEASE.jar!/:4.3.16.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) ~[spring-beans-4.3.16.RELEASE.jar!/:4.3.16.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.16.RELEASE.jar!/:4.3.16.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312) ~[spring-beans-4.3.16.RELEASE.jar!/:4.3.16.RELEASE]
    at 

I cut the error messages short since the trace is a few pages long, but the main error seems to be org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mvcConfiguration': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'webgoat.user.directory' in value "${webgoat.user.directory}"


I am running my agent as follows:
java -javaagent:/Users/path/to/jar/Spn-LangAgent-0.0.jar -jar webgoat-server-8.0.0.M17.jar --server.port=8080 --server.address=localhost

My agent is as follows:

package com.spnlangagent.langagent;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.StandardCopyOption;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLClassLoader;
import java.lang.reflect.Field;
import java.lang.instrument.Instrumentation;
import java.lang.instrument.UnmodifiableClassException;

import com.google.monitoring.runtime.instrumentation.AllocationRecorder;

public class LangAgent {

    public static void premain(String agentArgs, Instrumentation inst) throws Exception {
        System.out.println("LangAgent: premain now running");
        setupInstrumentation(agentArgs, inst);
        startRuntime(agentArgs);
    }

    private static void setupInstrumentation(String agentArgs, Instrumentation inst) throws Exception {
        System.out.println("setupInstrumentation: now running with agentArgs: " + agentArgs);
    }

    private static void startRuntime(String agentArgs) throws Exception {
        System.out.println("startRuntime: now running with agentArgs: " + agentArgs);
    }
}

The original contents of the agent were commented out except for a few print statements, and yet even with this, WebGoat is crashing on startup.

I tried another agent with WebGoat and it worked fine, so the only thing I can think of is that something is wrong with either my agent, or the way it is being packaged.

I am using Maven, and my MANIFEST.MF is as follows:

Manifest-Version: 1.0
Premain-Class: com.spnlangagent.langagent.LangAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true

After running mvn package, the MANIFEST packaged in the .jar is as follows:

Manifest-Version: 1.0
Premain-Class: com.spnlangagent.langagent.LangAgent
Built-By: andrewfan
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Created-By: Apache Maven 3.5.3
Build-Jdk: 1.8.0_172

In my pom.xml, I am doing the following to reach the manifest:

         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>3.1.0</version>
            <configuration>
                <archive>
                    <manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
                </archive>
            </configuration>
        </plugin>

If someone could point me in the right direction in regards to figuring out why WebGoat is crashing, or if someone could provide more insight into why what I am currently doing is wrong, that would be greatly appreciated.

Thank you.

Note: If the rest of my pom.xml is necessary for debugging, I will gladly provide it; it’s just that the question is already very long as-is.

Solution:

Webgoat (and also in most Spring-based application) relies on properties file (in properties or yaml format usually) to perform placeholder lookup.

The symptom in your failure indicate that Spring failed to lookup properties for placeholder processing.

Given that placeholder lookup works well without presence of your agent JAR, and with information you mentioned in comment, the problem is caused by

  • Your agent JAR provided application.properties (which has name collision with the properties file used by Webgoat for placeholder)
  • Agent JAR will be part of classpath, and probably even appear earlier than the main JAR
  • your empty application.properties “shadowed” the one in Webgoat main JAR. Which means, when Webgoat starts, Spring picked up your empty application.properties for its placeholder processing, hence failed.