Access S3 files meta data and other details using Python Boto using Assume Role

Here is the code

import sys
import boto
from boto.sts import STSConnection
from boto.s3.connection import S3Connection
from boto.s3.key import Key


sts_connection = STSConnection()
assumedRoleObject = sts_connection.assume_role(
    role_arn="arn:aws:iam::0000000000:role/xxxx/xxxx-yyyy-zzz",
    role_session_name="AssumeRoleSession1"
)

s3_connection = S3Connection(
    aws_access_key_id=assumedRoleObject.credentials.access_key,
    aws_secret_access_key=assumedRoleObject.credentials.secret_key,
    security_token=assumedRoleObject.credentials.session_token
)


bname="BUCKETNAME"
fname="FILENAME"
bucket = s3_connection.get_bucket(bname)
files=bucket.list(prefix=fname)

for file in files:
        k=bucket.get_key(file)
        print k.get_metadata('key-name')
        print file.name
        print file.size
        print file.last_modified
        print file.get_acl()
        print



1: Lets assume your key name is x-amz-meta-keyname then in the code pass your keyname as "keyname". No need to mention x-amz-meta

Let me know in comments section if his works/doesn't work for you

"Unable to execute HTTP request: Connection to https://ec2.amazonaws.com refused" error while executing AWS Java SDK Sample projects

If you are behind proxy then these sample projects will not read proxy from eclipse proxy settings.

to fix this issue you have to add some extra lines of code into your sample projects.


Create a object of ClientConfiguration class and set proxyhost and proxyport properties and then pass this object while creating client (ec2,S3  or sdb etc)

so your init method should look something like this after modification


 private static void init() throws Exception {
    /*
* This credentials provider implementation loads your AWS credentials
* from a properties file at the root of your classpath.
*
*/
   
    ClientConfiguration clientCfg = new ClientConfiguration();
        clientCfg.setProtocol(Protocol.HTTP);

        //setup proxy connection:
        clientCfg.setProxyHost("HTTP_PROXY_IP");
        clientCfg.setProxyPort(8080);
       
        AWSCredentialsProvider credentialsProvider = new ClasspathPropertiesFileCredentialsProvider();

        ec2 = new AmazonEC2Client(credentialsProvider,clientCfg);
        s3  = new AmazonS3Client(credentialsProvider,clientCfg);
        sdb = new AmazonSimpleDBClient(credentialsProvider,clientCfg);
    }

"Missing requirement: Eclipse Data Tools Platform Amazon SimpleDB UI Plug-in 1.0.0.v201308121803" Error while installing AWS Toolkit for Eclipse

In Eclipse follow below steps to fix this

1: Help -> install new software
2: Type "http://download.eclipse.org/releases/kepler" in the url field
3: Select "Database Development"  and install from the list.
4:  Try to re install  AWS Toolkit, it will work..   :)

Get count of messages in a jboss message queue from command prompt

Hi All,

You can also retrieve message count in a particular message queue of jboss, from command prompt, without accessing jmx console.

With command prompt you can automate and send alerts in case message count is above your application threshold

The command to do this is


./twiddle.sh -s localhost:1099 get jboss.messaging.destination:service=Queue,name=ResponseQueue MessageCount


Run this command from jboss bin directory and you can change queue name (in above example it is ResponseQueue) to your application specific queue name

Run Multiple Jboss on Same server

Hi All,

Its very easy to run multiple jboss on same server.

You just need to add one parameter wen you are starting your jboss.

the paramter is

-Djboss.service.binding.set=ports-01

example

./run.sh -c myproj -Djboss.service.binding.set=ports-01 &

to start another jboss on same server use

./run.sh -c myproject -Djboss.service.binding.set=ports-02 &

Just change -Djboss.service.binding.set parameter and you can start many jboss on same server, without any changes in any conf file.

Install perl GD module

Hi all,

If you are getting below error while instaling perl GD module module

Could not find gdlib-config in the search path. Please install libgd 2.0.28 or higher


Follow the below steps

1:do yum install gd gd-devel
2: go to cpan
3:then fire install GD

You are all set to go, your GD module should install now

How to Log slow queries in postgresql database

Hi All,

Postgresql db can log sql queries that takes more time to execute then specified time limit.


Suppose you want to log sql queries that takes more than 100ms to execute.

Follow the below steps

1: Open postgresql.conf file. You will find this file in data folder of postgres.
2: Modify the below paratmeters

a:logging_collector = on
b:log_min_duration_statement = 100
c:log_duration = on

3: Save and close this file.
4: Restart the DB

Now execute the queries again. P{ostgres will log all the queries in pg_log folder
pg_log folder will be inside your data folder of postgresql.
Name of log file will be in the following format

postgresql-%Y-%m-%d_%H%M%S.log


If you want to change log directory and log file name, just modify these parameters to desired value

#log_directory = 'pg_log' # directory where log files are written,
# can be absolute or relative to PGDATA
#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,

Get Index statistics in Postgresql database

I always observed that around 30 to 40% indexes created on a database are never used by any queries.

Having extra indexes on a DB, can degrade the performance (update and insert statements) , consumes lot of space and can increase the time of backup and restoring the DB.

It make sense to check the index utilization.

To check the same in Postgresql Database follow below steps.


1: Open postgresql.conf file. This file is inside your data directory of postgres.
2: Change following parameters

A: logging_collector = on
B:track_activities = on
C:track_counts = on
3: save and close the file
4: Restart the postgresql server

Now do all possible operartions using application, just to make sure all possible queries got fired to database.

Now just run the following query



select * from pg_stat_all_indexes where schemaname= order by relname,idx_scan desc;


You will find statistics of all indexes.

Always make sure your JVM is runnig is SERVER mode not in client mode

Hi,

JVM can run in many modes and selecting a proper mode can help improve your application performance

In servers, make sure JVM is runnig in server mode . This activates the server hotspot JIT.
It may need more time to start. But this delivers best performance.

Other modes of JVM are client and classic mode.

make sure -server, should be the first argument to java

Too Many Open Files problem in Linux

Hi All,

Most of the time during stress testing on linux server, you may encounter the following problem

SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=80] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files

We get this error when system reaches the the max file limit set on your Linux OS

first check the limit set on your OS with the following command
cat /proc/sys/fs/file-max

if you want to check the current open files .
1:lsof |wc -l   ( to check number of all open files)
2:lsof -p  | wc -l  ( to check number of opened files by a process)

in Ubuntu
ulimit -n



to change the value
1 edit /etc/sysctl.conf and set fs.file-max to any desired value

then run

sysctl -p

you may have to restart your system after this

troubleshoot mysql database connection pool problems

If you are getting following error messages, then its time to tune your mysql max_connection value

1:User 'mysqldba' has exceeded the 'max_user_connections' resource
2:Too many connections
3:GenericJDBCException: Cannot open connection
4:message from server: "Too many connections"

Now once u got these messages, we need to change value of max_connection variable in mysql database.

There are two ways to do this.

Method 1:Without restarting the mysql DB.
a;login to the mysql db.
b: run the query ----
show variables like "max_connections";
c: this will display the current value to max_connections.
d: Now to change the value of max_connection, run this query
set GLOBAL max_connections=NewValue;

max_connection value is changed , now again run the query
'show variables like "max_connections";'
to check if the value is reflecting.

But this value will be temporary, if you restart the mysql DB, max_connection value will again set to its default value

Method 2: This requires restarting the DB, but this change will be permanent and will not go even after restart of the db

a: For this you need to change the global setting file of mysql ie "my.cnf" file
b: By default, path of my.cnf will be under etc directory. If you dont know the path, run the
following command to get the path of my.cnf file
find / -name my.cnf
c: Once you got the path . open the file in vi editor.
d: You will find [mysqld] inside this file
e: Just add one line below this

max_connections=NewValue
f: restart the db.
g: login to the mysql and run the following query to check the value of max_connection

show variables like "max_connections";

Monitor cpu utilization of all threads in a process in windows

Hi All,

We can monitor cpu utilization of all threads for a process in windows using perfmon tool and this tool also writes logs to a file.
We can also use process explorer for this but this tool can not write logs.

To monitor threads in a process in windows,
1: Launch perfmon
2:Click Performance Logs and Alerts tree on left most window
3:Select Counter Logs from tree structure
4: Right click Counter Logs and select "New Log setting"
5: Give any name for new logs.
6: Click on Add general tab at top.
7: click "add counters" button.
8:From "Performane objects" drop down, select thread.
9:At right hand side, it will display all currently executing threads in windows.
10:Select the threads, you want to monitor and counter to monitor from lft hand side window.
11:Click close and go to "Log Files" tab.
12:Set the log file path and type.

Tomcat Performance parameters

1:DNS Lookups

When your webserver logs the information about the client, it can either log
the client’s numeric IP address or look up the actual host name in the Domain Name
Service. DNS lookups require network traffic, involving a round-trip response
from multiple servers, possibly far away and possibly inoperative, resulting in delays.

find this part of the server.xml file:

Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="true" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />







Just change the enableLookups value from "true" to "false", and restart Tomcat. No
more DNS lookups and their resulting delays!

2: Number of threads

We can control the number of threads that are allocated by changing a Connector’s
minThreads and maxThreads values.This setting is totally depend upon traffic that your application handles.

3: Precompile your JSP

When a JSP is first accessed, it is converted into Java servlet source code, which must then be compiled into Java bytecode.

For example, if you want index.jsp (in the root of your webapp’s directory) to always
be precompiled at webapp startup time, you can add a tag for this file in
your web.xml file, like this:

?servlet?
?servlet-name?index.jsp?/servlet-name?
?jsp-file?/index.jsp?/jsp-file?
?load-on-startup?0?/load-on-startup?
?/servlet?


Enjoy tomcat!!!!!!!!!

run a command in solaris from winodw and output will save in window through batch file

Hi All,

some time we need to run some command in solaris or Unix machine and need to collect output in window.
This all can be automated and directly we can invoke a shell scrpit in solaris from windows. You need putty to do this. Your shell script should be in windows machine.

CODE:

Create a batch file (suppose putty is saved in d: drive of windows and your shell script that you want to run is saved in d: drive).shell script name is "shellscript.sh"


D:\putty -ssh username@servername -pw mypassword -m "D:\shellscript.sh"
ftp -i -s:"D:\ftp.txt"



now create a ftp.txt file in D: drive as follows

OPEN servername
username
password
get filename
BYE




Thanks
Shashank

Write a parameter to a text file in loadrunner script

char *filename = "c:\\myfilename.txt";
long file_stream;
vuser_init()
{

if ((file_stream = fopen(filename, "a+")) == NULL) //open file in append mode
{
lr_error_message ("Cannot open %s", filename);
return -1;
}


fprintf (file_stream, "%s\n", lr_eval_string("THE PARAMETER IS :{Parameter}")); //Parameter is parameter name to be written in a file

Return 0;

}

vuser_end()
{
fclose(file_stream);//dont forget to close file_stream in vuser_end scetion
return 0;
}

monitor cpu and memory utilization of an individual process in windows

Sometime it is required to measure cpu utilization of a process instead to cpu utilization of whole machine in windows.
This is very much possible with PerfMon tool in windows
Steps to follow
1: Launch perfmon
2:Click Performance Logs and Alerts tree on left most window
3:Select Counter Logs from tree structure
4: Right click Counter Logs and select "New Log setting"
5: Give any name for new logs.
6: Click on Add objects tab.
7:Select Process from the list then click on Add then close.
8:Click "Log Files" tab at the top.
9:Select Log File Type as csv or tsv,as per your requirement
10: You are all set to go by default log file will be saved in c:\perfLogs folder

Code for puttting custome Pacing in script

Action()
{
merc_timer_handle_t timer;
double duration;
double pacing_time;

// Read desired pacing timer from a text file.
pacing_time = (double)jds_read_pacing_time("C:\\TEMP\\vugen_pacing.txt");

// Start the timer.
timer = lr_start_timer();

//Your Action Code

// Stop the timer
duration = lr_end_timer(timer);

// Wait for the necessary number of seconds to elapse
if (duration < pacing_time) {
lr_think_time(pacing_time - duration);
} else {
lr_error_message("Pacing time exceeded. Target: %G seconds. Actual: %g seconds", pacing_time, duration);
}

return 0;
}








// Read pacing time from a file. Returns time in seconds (whole seconds only).
// Note that file can be on a shared network drive.
int jds_read_pacing_time(char* file_name) {
long fs; // file stream
int number_from_file;

// Open the file for reading
fs = fopen(file_name, "r+");
if (fs == NULL) {
lr_error_message("Error opening file: %s", file_name);
return -1;
}

// Read number from file.
if (fscanf(fs, "%d", &number_from_file) != 1) {
lr_error_message("Error reading number from file: %s", file_name);
return -1;
}

fclose(fs);

return number_from_file;
}

URL vs HTML recording mode

HTML based mode, records script for every user action that is
performed during recording, while URL based mode records each and
every browser request to the server and resources received from the
server.

IN HTML recording mode the execution will be fast..as all
the server responses will not be captured in the script.
In URL all the server responses will be captured in the
script and hence the execution will be slow

how to monitor tomcat 6.0.16 with jconsole

go to tomcat bin folder
click on tomcat6w.exe
go to java tab
go to java options
type :
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
you are all set to go
now you can monitor your tomcat through jconsole and give port as 8090

back or referesh error with siebel protocol in loadrunner

hi this is a very common problem everyone facing while working on
siebel protocol of loadrunner.
this problem is coming from 2nd iteration .

solution is very simple

run the script and see in which web_url or web_submit, this problem is
coming soo just change mode to http from html and resources from 0
to 1.

I hope this will surely help you