Introduction ~

I was planning to write article on Multi Master MySQL replication since long time; Finally started now!. Please refer the the article on “How to configure MySQL replication with one Master” URL ~ http://www.indiangnu.org/2007/mysql-replication-one-master-multiple-slave/

* Let me inform you all that Multi Master replication in MySQL is purely based on following two variables. It has nothing to do with replication technology used in MySQL replication….

mysql> show variables  like  ‘%increment_%’;

+—————————————+——-+
| Variable_name                                   | Value |
+—————————————+——-+
| auto_increment_increment    |  1    |
| auto_increment_offset               |  1    |
+—————————————+—–+
2 rows in set (0.00 sec)

mysql>

** Requirements ~

a) Master Hosts (2 master in my case) ~

master-1 => 10.66.66.194
master-2 => 10.66.90.135
b) Replication Slave (1 slave) ~
slave => 10.66.75.137

c) MySQL server (with replication support)

** Let us understand how it works ?

* Master-1 Server =>

Set following variables…

mysql> set auto_increment_increment=5;
mysql> set auto_increment_offset=1;

mysql> show variables like ‘%increment_%’;
+————————–+——-+
| Variable_name            | Value |
+————————–+——-+
| auto_increment_increment | 2     |
| auto_increment_offset    | 1     |
+————————–+——-+
2 rows in set (0.00 sec)

mysql>

** Create Table ~

mysql> create table class ( rollno INT(5) NOT NULL PRIMARY KEY AUTO_INCREMENT , name VARCHAR(30) );

** Add Record now ~

mysql> INSERT INTO class VALUES (”,’Arun Bagul’);
mysql> INSERT INTO class VALUES (”,’Ravi Bhure’);
mysql> INSERT INTO class VALUES (”,’Karthik Appigita’);
mysql> INSERT INTO class VALUES (”,’Ameya Pandit’);

mysql> SELECT * FROM class;
+——–+——————+
| rollno | name             |
+——–+——————+
|      1 | Arun Bagul       |
|      3 | Ravi Bhure       |
|      5 | Karthik Appigita |
|      7 | Ameya Pandit     |
+——–+——————+
4 rows in set (0.00 sec)

mysql>

* Master-2 Server =>

Set following variables…

mysql> set auto_increment_increment=2;
mysql> set auto_increment_offset=2;

mysql> show variables like ‘%increment_%’;
+————————–+——-+
| Variable_name            | Value |
+————————–+——-+
| auto_increment_increment | 2     |
| auto_increment_offset    | 2     |
+————————–+——-+
2 rows in set (0.00 sec)

mysql>

** Create Table ~

mysql> create table class ( rollno INT(5) NOT NULL PRIMARY KEY AUTO_INCREMENT , name VARCHAR(30) );

** Add Record now ~

mysql> INSERT INTO class VALUES (”,’Nilkanth Parab’);
mysql> INSERT INTO class VALUES (”,’Nishit Shah’);
mysql> INSERT INTO class VALUES (”,’Ram Krishna’);
mysql> INSERT INTO class VALUES (”,’Suhail Thakur’);

mysql> SELECT * FROM class;
+——–+——————+
| rollno | name             |
+——–+——————+
|      2 | Nilkanth Parab   |
|      4 | Nishit Shah      |
|      6 | Ram Krishna      |
|      8 | Suhail Thakur    |
+——–+——————+
4 rows in set (0.00 sec)

mysql>

** What is the importance of “auto_increment_increment” and “auto_increment_offset” ~

mysql> desc class;
+——–+————-+——+—–+———+—————-+
| Field  | Type        | Null | Key | Default | Extra          |
+——–+————-+——+—–+———+—————-+
| rollno | int(5)      | NO   | PRI | NULL    | auto_increment |
| name   | varchar(30) | YES  |     | NULL    |                |
+——–+————-+——+—–+———+—————-+

auto_increment_offset => This is BASE value for column with “auto_increment” attribute (please refer the above example)
auto_increment_increment => This is the increment value for column with “auto_increment” attribute

** If you combine the both tables (master-1 and master-2) the final table will look like this ~

mysql> SELECT * FROM class;
+——–+——————+
| rollno | name             |
+——–+——————+
|      1 | Arun Bagul       |
|      2 | Nilkanth Parab   |
|      3 | Ravi Bhure       |
|      4 | Nishit Shah      |
|      5 | Karthik Appigita |
|      6 | Ram Krishna      |
|      7 | Ameya Pandit     |
|      8 | Suhail Thakur    |
+——–+——————+
8 rows in set (0.00 sec)

mysql>

** This is how Multi master replication works….

auto_increment_offset=Nth master server
auto_increment_increment=M

Where -
N => nth number of master server (on master-1 keep it 1 and on master-2 keep it 2 and so on..)
M => Total number of Master Server (2 in our case but better to keep this value high so that we can add new master server easily)

log-slave-updates => Slave server does not log to its own binary log any updates that are received from a Master server. This option tells the slave to log the updates performed by its SQL thread to its own binary log.

** Make sure that MySQL is running and up on all master servers and slave server-

How to setup Multi Master MySQL replication ? –

Step 1] Create Database/Tables on  Master  & Slave Servers –

You can create DB on all master & slave server or  create on one server and export that DB on  rest of all servers…

Master-1 => Create DB and Table

mysql> create database student;

mysql> use student;

mysql> create table class ( rollno INT(5) NOT NULL PRIMARY KEY AUTO_INCREMENT , name VARCHAR(30) );

mysql> show tables;
+——————-+
| Tables_in_student |
+——————-+
| class             |
+——————-+
1 row in set (0.00 sec)

mysql> desc class;
+——–+————-+——+—–+———+—————-+
| Field  | Type        | Null | Key | Default         |      Extra          |
+——–+————-+——+—–+———+—————-+
| rollno | int(5)      | NO   | PRI | NULL    | auto_increment |
| name   | varchar(30) | YES  |     | NULL    |                |
+——–+————-+——+—–+———+—————-+
2 rows in set (0.00 sec)

mysql> SELECT * FROM class;
Empty set (0.00 sec)
mysql>

* Now take dump of “student” DB and export it on all master and Slave server…

[root@master-1~]# mysqldump -u root -p  -d student > /home/arunsb/student.sql

* SCP the dump file on master-2 and slave server ~
[root@master-1~]# scp /home/arunsb/student.sql arunsb@10.66.90.135:/tmp/student.sql
[root@master-1~]# scp /home/arunsb/student.sql arunsb@10.66.75.137:/tmp/student.sql

Login on master-2 and slave ~

mysql> create database student;

[root@master-2~]# mysql -u root -p student < /tmp/student.sql
Enter password:
[root@master-2~]#

[root@master-2~]# mysql -u root -p
Enter password:

mysql> use student

mysql> SELECT * FROM class;
Empty set (0.00 sec)
mysql>

** Please repeat the same steps on Slave server as well…

Step 2] Update “my.cnf” config file on master-1,master-2 and slave server –

[root@master-1~]# cat /etc/my.cnf

###########################
##MySQL replication setting

#Master setting(1)
server-id = 1
log-bin = /var/log/mysql/binary/mysql-bin.log
binlog-do-db = student
binlog-ignore-db = mysql
log = /var/log/mysql/mysql.log
auto_increment_offset=1
auto_increment_increment=5
log-slave-updates

##slave setting
master-port=3306
master-host=10.66.90.135
master-user=replication
master-password=mypwd
master-connect-retry=60
replicate-do-db=student
###########################

[root@master-1~]#

[root@master-2~]# cat /etc/mysql/my.cnf

###########################
##MySQL replication setting

#Master setting(2)
server-id = 2
log-bin = /var/log/mysql/binary/mysql-bin.log
binlog-do-db=student
binlog-ignore-db = mysql
log = /var/log/mysql/mysql.log
auto_increment_offset=2
auto_increment_increment=5
log-slave-updates

##slave setting
master-port=3306
master-host=10.66.66.194
master-user=replication
master-password=mypwd
master-connect-retry=60
replicate-do-db=student
###########################

[root@master-2~]#

* please create directory for binary log and set permission…

[root@master-1~]# mkdir -p /var/log/mysql/binary/
[root@master-1~]# chown mysql:adm  /var/log/mysql/ /var/log/mysql/binary/

[root@master-2~]# mkdir -p /var/log/mysql/binary/
[root@master-2~]# chown mysql:adm  /var/log/mysql/ /var/log/mysql/binary/

** MySQL Replication Slave ~

[root@slave~]# cat  /etc/my.cnf

[mysqld]

########################################
##slave setting
server-id=4
master-port=3306
master-host=10.66.90.135
master-user=replication
master-password=mypwd
master-connect-retry=60
replicate-do-db=student
########################################

[root@slave~]#

Step 3] Give Replication permission on both masters ~

** Master (1 & 2) ~

mysql> GRANT REPLICATION SLAVE ON *.* TO ‘replication’@'10.66.%.%’ IDENTIFIED BY ‘mypwd’;
Query OK, 0 rows affected (0.00 sec)

mysql>

Step 4] Restart MySQL on both master as well as replication slave server ~

** Please verify setting on master-1 and master-2 server…

* Master-1

mysql> show variables like ‘%increment_%’;
+————————–+——-+
| Variable_name            | Value |
+————————–+——-+
| auto_increment_increment | 5     |
| auto_increment_offset    | 1     |
+————————–+——-+
2 rows in set (0.00 sec)

* Master-2

mysql> show variables like ‘%increment_%’;
+————————–+——-+
| Variable_name            | Value |
+————————–+——-+
| auto_increment_increment | 5     |
| auto_increment_offset    | 2     |
+————————–+——-+
2 rows in set (0.00 sec)

** Please verify ‘master’ & ’slave’ status on both masters(1 & 2) and slave -

mysql> show master status;
mysql> show slave status;

** Multi Master replication is started…

Step 5] Add few records on Master-1 & Master-2 server at same time ~

Add records on both master server at same time and check master and replication slave status as shown above….

** Add following on master-1

mysql> INSERT INTO class VALUES (”,’Arun Bagul’);
mysql> INSERT INTO class VALUES (”,’Ravi Bhure’);
mysql> INSERT INTO class VALUES (”,’Karthik Appigita’);
mysql> INSERT INTO class VALUES (”,’Ameya Pandit’);

** Add following on master-2

mysql> INSERT INTO class VALUES (”,’Nilkanth Parab’);
mysql> INSERT INTO class VALUES (”,’Nishit Shah’);
mysql> INSERT INTO class VALUES (”,’Ram Krishna’);
mysql> INSERT INTO class VALUES (”,’Suhail Thakur’);

** Please verify the numbers of records on both masters and slave….

mysql> SELECT * FROM class;
+——–+——————+
| rollno | name             |
+——–+——————+
|      1 | Arun Bagul       |
|      2 | Nilkanth Parab   |
|      6 | Ravi Bhure       |
|     11 | Karthik Appigita |
|     16 | Ameya Pandit     |
|     17 | Nishit Shah      |
|     22 | Ram Krishna      |
|     27 | Suhail Thakur    |
+——–+——————+
8 rows in set (0.00 sec)

mysql>

* So we all learned to configure multi-master MySQL replication. Enjoy!!

Regards,
Arun Bagul

Introduction -

* “rpmbuild” tool is used to build both…

1) Binary Package ~ used to install the software and supporting scripts. It contains the files that comprise the application, along with any additional information needed to install and erase it.
2) Source Package ~ contains the original compressed tar file of source code, patches and Specification File.

* What is RPM & RPM package Manager?

The RPM Package Manager (RPM) is a powerful command line package management system capable of installing, uninstalling, verifying, querying, and updating software packages.

RPM package consists of an archive of files and meta-data used to install and erase the archive files. The meta-data includes helper scripts, file attributes, and descriptive information about the package.

* To build RPM package you need to specify three things ~

1) Source of application - In any case, you should not modify the sources used in the package building process.

2) Patches - RPM gives you the ability to automatically apply patches to them. The patch addresses an issue specific to the target system. This could include changing makefiles to install the application into the appropriate directories, or resolving cross-platform conflicts. Patches create the environment required for proper compilation.

3) Specification File - The specification file is at the heart of RPM package building process. It contains information required by RPM to build the package, as well as instructions telling RPM how to build it. The specification file also dictates exactly what files are a part of the package, and where they should be installed.

** Specification File ~ is divided in to 8 sections as shown below

a) Preamble ~ contains information that will be displayed when users request information about the package. This would include a description of the package’s function, the version number of the software etc.

b) Preparation ~ where the actual work of building a package starts. As the name implies, this section is where the necessary preparations are made prior to the actual building of the software. In general, if anything needs to be done to the sources prior to building the software, it needs to happen in the preparation section. The contents of this section are an ordinary shell script. However, RPM does provide two macros to make life easier. One macro can unpack a compressed tar file and cd into the source directory. The other macro easily applies patches to the unpacked sources.

c) Build ~ This section consists of a shell script. It is used to perform whatever commands are required to actually compile the sources like single make  command, or be more complex if the build process requires it. There are no macros available in this section.

d) Install ~ This section also containing a shell script, the install section is used to perform the commands required to actually install the software.

e) Install and Uninstall Scripts ~ It consists of scripts that will be run, on the user’s system, when the package is actually installed or removed. RPM can execute a script pre/post installation/removal of package.

f) Verify Script ~ script that is executed on the user’s system. It is executed when RPM verifies the package’s proper installation.

g) Clean Section ~ script that can clean things up after the build. This script is rarely used, since RPM normally does a good job of clean-up in most build environments.

h) File List ~ consists of a list of files that will comprise the package. Additionally, a number of macros can be used to control file attributes when installed, as well as to denote which files are documentation, and which contain configuration information. The file list is very important.

*** RPM’s requirement for build environment ~

A] RPM requires a set of directories to perform the build. While the directories’ locations and names can be changed. Default layout is shown below -

root@arunsb:~# ls -l /usr/src/redhat/

drwxr-xr-x 2 root root 4096 Aug 25  2007 SOURCES => Contains the original sources, patches, and icon files
drwxr-xr-x 2 root root 4096 Aug 25  2007 SPECS => Contains the specification files
drwxr-xr-x 2 root root 4096 Aug 25  2007 BUILD => Directory in which the sources are unpacked, & software is built
drwxr-xr-x 8 root root 4096 May 28  2008 RPMS => Contains the binary package files created by the build process
drwxr-xr-x 2 root root 4096 Aug 25  2007 SRPMS => Contains the source package files created by the build process

root@arunsb:~#

B] Need to export few global variables used by RPM -

root@arunsb:~# export  RPM_BUILD_DIR=/usr/src/redhat/BUILD/
root@arunsb:~# export  RPM_SOURCE_DIR=/usr/src/redhat/SOURCES/

Step 1] Create Specification (spec) File ~

root@arunsb:~# head -n 50  /usr/src/redhat/SPECS/openlsm.spec
# Authority: Arun Bagul

#RPM_BUILD_DIR    /usr/src/redhat/BUILD/
#RPM_SOURCE_DIR    /usr/src/redhat/SOURCES/

%define MY_PREFIX    /usr/local/openlsm/

## Preamble Section-
Name: openlsm
Version: 0.99
Vendor: IndianGNU.org & openlsm
Release: r45
Group: System Environment/Daemons
Packager: IndianGNU.org (http://www.indiangnu.org)
URL: http://openlsm.sourceforge.net/
Summary: openlsm Admin Server
License: GPL

%description
openlsm Admin Server is free & open source web based control panel for Linux,Unix systems.

## Preparation Section-
%prep

rm -rf ${RPM_BUILD_DIR}/openlsm-0.99-r45/
tar xvfz ${RPM_SOURCE_DIR}/openlsm-0.99-r45.tar.gz -C ${RPM_BUILD_DIR}

## Build Section-
%build

cd ./openlsm-0.99-r45/
./configure –prefix=/usr/local/openlsm –with-mysql=/usr/bin/mysql_config –enable-internal-pcre –with-geoip=/usr –with-ldap=/usr –enable-trace
make

## Install Section-
%install

cd ./openlsm-0.99-r45/
make install

## Files Section-
%files

/usr/local/openlsm
/usr/local/openlsm/etc/openlsm/openlsm.conf
/usr/local/openlsm/etc/openlsm/openlsm.conf.perf_sample
/usr/local/openlsm/etc/openlsm/ssl/
/usr/local/openlsm/bin/openlsm-config
….
…..
….. list of files installed by pkg
root@arunsb:~#

* How do you create the File List?

Creating the file list is manual process. What I did is I took the file list from my manual installed prefix directory with find command as shown below…

root@arunsb:~# find /usr/local/openlsm/ -type f -or  -type d

Step 2] Starting the Build ~

root@arunsb:~# cd /usr/src/redhat/SPECS
root@arunsb:/usr/src/redhat/SPECS# ls -l  openlsm.spec
-rw-r–r– 1 root root 12938 Dec  2 17:21 openlsm.spec
root@arunsb:/usr/src/redhat/SPECS#

root@arunsb:/usr/src/redhat/SPECS# rpmbuild   -ba  openlsm.spec
…
….
…..

Checking for unpackaged file(s): /usr/lib/rpm/check-files %{buildroot}
Wrote: /usr/src/redhat/SRPMS/openlsm-0.99-r45.src.rpm
Wrote: /usr/src/redhat/RPMS/i386/openlsm-0.99-r45.i386.rpm
root@arunsb:/usr/src/redhat/SPECS# echo $?
0

root@arunsb:/usr/src/redhat/SPECS# ls -l  /usr/src/redhat/SRPMS/openlsm-0.99-r45.src.rpm
-rw-r–r– 1 root root 3206 Dec  2 17:50 /usr/src/redhat/SRPMS/openlsm-0.99-r45.src.rpm
root@arunsb:/usr/src/redhat/SPECS# ls -l  /usr/src/redhat/RPMS/i386/openlsm-0.99-r45.i386.rpm
-rw-r–r– 1 root root 3052868 Dec  2 17:50 /usr/src/redhat/RPMS/i386/openlsm-0.99-r45.i386.rpm
root@arunsb:/usr/src/redhat/SPECS#

* Source and Binary package created !!

** Let’s see what happened in “/usr/src/redhat/” directory

root@arunsb:/usr/src/redhat# pwd
/usr/src/redhat
root@arunsb:/usr/src/redhat# ls
BUILD  RPMS  SOURCES  SPECS  SRPMS
root@arunsb:/usr/src/redhat# ls BUILD/
openlsm-0.99-r45                                               <== Source extracted here as part of build instructions from specification file ie “openlsm.spec”
root@arunsb:/usr/src/redhat# ls SOURCES/
openlsm-0.99-r45.tar.gz                                 <== original ‘openlsm-0.99-r45.tar.gz’ source file copied by me
root@arunsb:/usr/src/redhat# ls RPMS/
athlon  i386  i486  i586  i686  noarch
root@arunsb:/usr/src/redhat# ls RPMS/i386/
openlsm-0.99-r45.i386.rpm <== Binary rpm package created!
root@arunsb:/usr/src/redhat# ls SRPMS/
openlsm-0.99-r45.src.rpm <== Source rpm package created!
root@arunsb:/usr/src/redhat#

Step 3] Now install the package and test it ~

root@arunsb:/usr/src/redhat# cp RPMS/i386/openlsm-0.99-r45.i386.rpm  /home/arunsb/

root@arunsb:/usr/src/redhat# cd /home/arunsb/
root@arunsb:~# ls
openlsm-0.99-r45.i386.rpm
root@arunsb:~# rpm -ivh openlsm-0.99-r45.i386.rpm
Preparing…                ########################################### [100%]
1:openlsm                ########################################### [100%]
root@arunsb:~# ls /usr/local/openlsm/
bin  contrib  etc  include  lib  sbin  scripts  share  var
root@arunsb:~#

** Starting the openlsm server ~

root@arunsb:~# /usr/local/openlsm/contrib/openlsm-redhat  start
* Starting openlsm admin server: openlsm
.                                                         [  OK  ]
root@arunsb:~#
root@arunsb:~# /usr/local/openlsm/contrib/openlsm-redhat  status
openlsm (pid 21607) is running…
root@arunsb:~#

root@arunsb:~# rpm -q openlsm-0.99-r45
openlsm-0.99-r45
root@arunsb:~#

root@arunsb:~# rpm -ql  openlsm-0.99-r45
..
…

root@arunsb:~# rpm  -qiv  openlsm-0.99-r45
Name        : openlsm                      Relocations: (not relocatable)
Version     : 0.99                              Vendor: IndianGNU.org & openlsm
Release     : r45                           Build Date: Wed 02 Dec 2009 05:50:54 PM IST
Install Date: Wed 02 Dec 2009 06:06:23 PM IST      Build Host: alongseveral-dr.eglbp.corp.yahoo.com
Group       : System Environment/Daemons    Source RPM: openlsm-0.99-r45.src.rpm
Size        : 14877918                         License: GPL
Signature   : (none)
Packager    : IndianGNU.org (http://www.indiangnu.org)
URL         : http://openlsm.sourceforge.net/
Summary     : openlsm Admin Server
Description :
openlsm Admin Server is free & open source web based control panel for Linux,Unix systems.
root@arunsb:~#

** NOTE ~ This article does not contain information on how to define micros,how to copy docs,man pages to default location, how to set permision and ownership etc. I will cover this topics in next article on RPM.

Regards,
Arun Bagul

Introduction ~

GNU Privacy Guard (GnuPG or GPG) is open source/free software encryption and signing tool, alternative to the Pretty Good Privacy (PGP) suite of cryptographic software. Enigmail is an extension for Mozilla Thunderbird and other Mozilla applications. It provides public key e-mail encryption. Actual cryptographic functionality is handled by GNU Privacy Guard (GnuPG,GPG).

Step 1] Install Gnupg or GPG –

* Ubuntu/Debian ~

root@arun:~# apt-get  install gnupg  gnupg2

* Redhat/Fedora ~

root@arun:~# yum  install gnupg  gnupg2

Step 2] How to Install Enigmail ?

I assume that Mozilla Thunderbird is already installed on your system. To install  “Enigmail” follow following steps

a) Download “Enigmail” from  url “http://enigmail.mozdev.org/download/”

Note ~ select OS and Thunderbird version properly.

b) In menu bar of the main Thunderbird window you will see “Tools”. Select this, and then “Add-ons” option. This will bring up a new window listing all of your Thunderbird plug-ins. In the lower left-hand corner of this new window you’ll see a button marked “Install”. Click this button. Tell  Thunderbird where you saved the Enigmail .XPI file. and just say “Install” that’s it!!

* Once ‘Enigmail’ is installed restart the Thunderbird. Then you will see “OpenPGP” tab in main menu of Thunderbird.

Step 3] Setup private/public key ~

Enigmail uses public key cryptography to ensure privacy between you and your correspondents. To generate the public/private keys, there is two methods either generate them with the help of “gpg” command line tool or use “enigmail”….

* We will generate private/public cryptographic keys with the help of “enigmail” as shown below….

a) Click on “OpenPGP” in the menu bar of the Thunderbird main window. Select “Key Management”.
b) In Enigmail Key Manager ~ click on “Generate” tab in the menu bar and select “New key pair”.
c) At the very top of the window you will see a combo box showing all of your email addresses. GnuPG will associate your new key with an email address.
Enigmail is just asking you which address you want to use for this key. Select whichever account will be receiving encrypted mail.

NOTE ~ We can use same keys for multiple accounts.

d) You can use passphrase or just check “No passphrase” checkbox to generate keys without passphrase.
e) Create  directory to save “Revocation Certificates”…

arunsb@arun:~$ mkdir /home/arunsb/.gpg_key/

f) Click on “Generate key” button to generate keys. done..

To share keys easily you can publish your keys with keyserver.

a) In “Key Management” window select your keys and then  click on ‘Keyserver’ tab in main menu and then click on “Upload Public Keys”
Note ~ make sure to check  “Display All Keys by Default” checkbox (to list all keys)

Step 4] Compose the mail and sign it ~

Compose the mail and tell Enigmail to sign it.  At the top of your Compose window you will see a button reading “OpenPGP”. Click on this. Make sure that the “Sign” option, and only that, is checked. Finally “Send” the mail! (You will be asked for your passphrase. Once you enter it, Enigmail will sign your email and send it if you have generate keys with passphrase else it will not ask)

Enjoy!!

Regards,
Arun Bagul

Dear All,

Today , we are celebrating third anniversary!!. We have successfully completed three years of social contribution in open source and free software movement. I am very much sure that our existence and contribution is recognized and encouraged by many open source and free software contributors (gurus). It was simply impossible without your support, contribution and hard work.

We started with blogging, sharing our knowledge, ideas and even problems!! also. This is helped us to learn new technologies, problem solving skills and many more things.

openlsm Project ~

openslm-0.99 development platform was released on 10th Jun 2009. We got positive response from users,community members, and other open source communities. Please refer the article for more information http://www.indiangnu.org/2009/openlsm-099-released/

Cheers,
IndianGNU.org

What is ETL ?

Extract, Transform, and Load (ETL) is procedure to

a) Extracting data from outside sources
b) Transforming the data as ther operational requirements
c) Loading it into the end target (database or data warehouse)

1) Talend Open Studio ~ http://www.talend.com

Talend Open Studio operates as a code generator allowing data transformation scripts and underlying programs to be generated either in Perl or in Java.
Its GUI is made of a metadata repository and a graphical designer. The jobs are designed using graphical components, for transformation, connectivity or other operations. The threads created can be executed from within the studio or as standalone scripts.

2) CloverETL ~     http://www.cloveretl.com/

CloverETL is a Java-based data integration framework used to transform, cleanse, standardize and distribute data to applications, databases or warehouses. Its component-based structure allows easy customization and embeddability.

* CloverETL Designer – the graphical user interface to create and modify data transformations for CloverETL Server and Engine.
* CloverETL Engine – executes the transformations (run-time); can be embedded as a library. Available under LGPL.
* CloverETL Server – full-fledged server application with a rich WEB-based administrative interface, which leverages the existing CloverETL Engine.

3) Pentaho ~ http://www.pentaho.com/

The Pentaho BI Project is Open Source application software for enterprise reporting, analysis, dashboard, data mining, workflow and ETL capabilities for Business Intelligence (BI) needs.

4) Apatar ~ http://www.apatar.com/

Apatar is an open source ETL (Extract-Transform-Load) and mashup data integration software application. Other open source data integration projects are Clover.ETL, Pentaho Project, Talend Open Studio or Enhydra Octopus.

5) http://www.azinta.com/
Regards,

Arun

Introduction ~

Please refer the following article to install/configure Jabberd2…

http://www.indiangnu.org/2009/how-to-configure-jabber-jabberd2-with-mysqlpam-as-auth-database/

Requirement ~

*  Following packages are require to compile MU-Conference…

root@laptop:~# apt-get install  libglib2.0-0  libglib2.0-dev
root@laptop:~# apt-get install  libidn11  libidn11-dev
root@laptop:~# apt-get install  expat  lib64expat1  lib64expat1-dev libexpat1-dev  liblua5.1-expat-dev  liblua5.1-expat0

Step 1] How compile “MU-Conference” –

* Download “MU-Conference” from following URL – https://gna.org/projects/mu-conference/

root@laptop:/var/src# wget -c http://download.gna.org/mu-conference/mu-conference_0.8.tar.gz
root@laptop:/var/src# tar xvfz mu-conference_0.8.tar.gz
root@laptop:/var/src# cd mu-conference_0.8/
root@laptop:/var/src/mu-conference_0.8#

* compile MU-Conference

root@laptop:/var/src/mu-conference_0.8# make
cd src/ ; make
make[1]: Entering directory `/var/src/mu-conference_0.8/src’
cd jabberd ; make
make[2]: Entering directory `/var/src/mu-conference_0.8/src/jabberd’
…
…..

root@laptop:/var/src/mu-conference_0.8# echo $?
0
root@laptop:/var/src/mu-conference_0.8#

Step 2] Configure MU-Conference –

* Now copy the “MU-Conference” binary to Jabberd2 installation directory -

root@laptop:~# cp /var/src/mu-conference_0.8/src/mu-conference  /usr/local/jabberd-2.2.9/bin/
root@laptop:~# chown jabber:jabber /usr/local/jabberd-2.2.9/bin/mu-conference
root@laptop:~# ls -l /usr/local/jabberd-2.2.9/bin/mu-conference
-rwxr-xr-x 1 jabber jabber 191904 2009-10-12 18:59 /usr/local/jabberd-2.2.9/bin/mu-conference
root@laptop:~#

root@laptop:~# /usr/local/jabberd-2.2.9/bin/mu-conference –help
Jabber Component Runtime — 0.2.4
(c) 2003-2004 Paul Curtis

/usr/local/jabberd-2.2.9/bin/mu-conference: invalid option — ‘-’
Usage: mu-conference [-B] [-s] [-h] [-d LEVEL] -c FILE
-B         Put the daemon in background
-s         Show debug messages on stderr
-h         Print this help
-d LEVEL   Set the level of debug output
-c FILE    Set the config file, mandatory argument
root@laptop:~#

* Create spool directory for “MU-Conference”. The mu-conference component requires a spool directory to in which to store conference room information.

root@laptop:~# mkdir /usr/local/jabberd-2.2.9/var/spool
root@laptop:~# chown jabber:jabber /usr/local/jabberd-2.2.9/var/spool

NOTE ~  “jabber:jabber” user/group name of Jabberd2 server.

* Copy the config file of “MU-Conference” to Jabberd2  installation directory and edit the setting –

root@laptop:~# cp /var/src/mu-conference_0.8/muc-default.xml  /usr/local/jabberd-2.2.9/etc/mu-conference.xml
root@laptop:~# chown jabber:jabber /usr/local/jabberd-2.2.9/etc/mu-conference.xml
root@laptop:~#

root@laptop:~# vi /usr/local/jabberd-2.2.9/etc/mu-conference.xml

<name>conf.laptop.ubuntu.me</name>
<host>conf.laptop.ubuntu.me</host>
<ip>localhost</ip>
<port>5347</port>
<secret>secret</secret>

<spool>/usr/local/jabberd-2.2.9/var/spool</spool>
<logdir>/usr/local/jabberd-2.2.9/var/log</logdir>
<pidfile>/usr/local/jabberd-2.2.9/var/run/mu-conference.pid</pidfile>


<loglevel>255</loglevel>

<sadmin>
<user>admin@laptop.ubuntu.me</user>
</sadmin>

…
…..
root@laptop:~#

* Now restart the Jabberd2 server and then  start “MU-Conference”…

root@laptop:~# su -l jabber -s /bin/bash -c “/usr/local/jabberd-2.2.9/bin/mu-conference  -B -c /usr/local/jabberd-2.2.9/etc/mu-conference.xml”
root@laptop:~#

* Please check above article, init startup script ie ‘/etc/init.d/jabberd2′ will start mu-conference.

root@laptop:~# tail -f /usr/local/jabberd-2.2.9/var/log/mu-conference.log
Mon Oct 12 19:19:40 2009 main.c:168 (main): Jabber Component Runtime — 0.2.4  starting.
Mon Oct 12 19:19:40 2009 MU-Conference: [conference.c:1076 (conference)] mu-conference loading  – Service ID: conf.laptop.ubuntu.me
…
Mon Oct 12 19:19:40 2009 MU-Conference: [conference.c:1157 (conference)] Adding sadmin admin@laptop.ubuntu.me
Mon Oct 12 19:19:40 2009 MU-Conference: [xdb.c:319 (xdb_rooms_get)] asked to get rooms from xdb
Mon Oct 12 19:19:40 2009 MU-Conference: [xdb.c:418 (xdb_rooms_get)] skipping .. no results
Mon Oct 12 19:19:40 2009 main.c:219 (main): Main loop starting.
Mon Oct 12 19:19:40 2009 jcr_base_connect.c:34 (jcr_socket_connect): Attempting connection to localhost:5347
Mon Oct 12 19:19:40 2009 jcr_base_connect.c:87 (jcr_send_start_stream): Opening XML stream: sent 173 bytes
Mon Oct 12 19:19:40 2009 jcr_main_stream_error.c:50 (jcr_main_new_stream): Server stream connected.
Mon Oct 12 19:19:40 2009 jcr_deliver.c:51 (jcr_queue_deliver): packet delivery thread starting.

done!!

Step 3] Test “MU-Conference” -

* In “PSI” IM client , goto ‘General’ menu and then click on ‘Service Discovery’ and check the room list…
* In “Pidgin” IM client, goto ‘Tools’ and then click on ‘Room List’ section.

Once you detected the ‘mu-conference’ server. Please click on  ‘+ Add Chat’ from ‘Buddies’ menu of pidgin. In case of PSI click on ‘Join Groupchat’ from ‘General’ menu to add “Conference/Room” and then join the ‘Confernece Room’.

Enjoy,
Arun Bagul

Introduction -

Jabberd2 is XMPP protocol based Instants Messaging (IM) server. Jabberd2 is highly scalable,high performance jabber server. The beauty of the Jabberd2 architecture lies in the fact that its component architecture distributes services across six components, each of which communicates over TCP/IP.

1) Router - is the backbone of Jabber server. It accepts connections from Jabberd components and passes XML packets between components
2) Server to Server (S2S) - component handles communications with external servers. S2S passes packets between other components and external servers, and it performs dial-back to authenticate remote Jabber servers.

3) Resolver  - acts in support of the S2S component. It resolves hostnames for S2S as part of dialback authentication.
4) Session Manager (SM)  - component implements instant messaging features like message passing,presence,roster and subscription etc. + DB connection
5) Client to Server (C2S) - component handles communication with Jabber clients like connection,passing packets to SM, authenticate and register users.
6) Jabber core  - logging and third party plugin communication.

** To compile/install Jabberd-2.2.9 we need following packages on Debian/Ubuntu (similar on Redhat/Fedora or other OS)

- libpam0g  libpam0g-dev (PAM support)
- openssl libssl-dev (TLS/SSL support)
- libudns0 libudns-dev (DNS Resolver Library)
- libidn11  libidn11-dev libnet-libidn-perl (libidn provides necessary string manipulation functionality for Jabberd2)
- mysql-common libdbd-mysql-perl  mysql-server-5.1  mysql-client-5.1 libmysqlclient16-dev (MySQL DB authentication)

**  Jabberd2 supports five authentication (user) mechanism -

* PAM
* MySQL Database
* Berkeley DB
* PostgreSQL Database
* SQLite DB
* OpenLDAP

** Following ports are used by jabberd2 -
* port 5222 - non-SSL client connection
* port 5223 – SSL client connection
* port 5269 – server to server connection
* port 5347 – jabberd2 router

Step 1] Create system User and Group for Jabberd 2  Server (http://codex.xiaoka.com/wiki/jabberd2:start) –

root@laptop:~# addgroup  –system  jabber
Adding group `jabber’ (GID 61) …
Done.
root@laptop:~#

root@laptop:~# adduser  –system  –home /usr/local/jabberd-2.2.9/ –shell /bin/false  –gid 61 jabber

* Verify system User and Group…  (steps for Ubuntu)

root@laptop:~# id jabber
uid=125(jabber) gid=61(jabber) groups=61(jabber)
root@laptop:~#

Step 2] Download the latest version of  Jabberd2 -

* Extract the source then compile/install it as shown below with PAM/MYSQL DB for authentication with SSL

root@laptop:/var/src/# wget -c http://codex.xiaoka.com/pub/jabberd2/releases/jabberd-2.2.9.tar.bz2

root@laptop:/var/src# tar xvfj  jabberd-2.2.9.tar.bz2

root@laptop:/var/src# cd jabberd-2.2.9

root@laptop:/var/src/jabberd-2.2.9# ./configure –prefix=/usr/local/jabberd-2.2.9/ –enable-debug  –enable-mysql  –enable-ssl –enable-pam  –enable-ssl
…
…..
checking for Libidn version >= 0.3.0… yes
checking for dns_init in -ludns… yes
checking gsasl.h usability… yes
checking gsasl.h presence… yes
checking for gsasl.h… yes
checking for gsasl_check_version in -lgsasl… yes
checking for GnuSASL version >= 0.2.27… no
configure: error: no SASL backend available out of: gsasl
root@laptop:/var/src/jabberd-2.2.9#

Step 3] Facing problem  like “configure: error: no SASL backend available out of: gsasl” ~

Don’t worry download latest version of gsasl library from URL ~ http://alpha.gnu.org/gnu/gsasl/

* Download latest version of GNU SASL (gsasl) …

root@laptop:/var/src/# wget -c  http://alpha.gnu.org/gnu/gsasl/gsasl-0.2.29.tar.gz

* Extract the source then compile/install it …

root@laptop:/var/src# tar xvfz  gsasl-0.2.29.tar.gz
root@laptop:/var/src# cd gsasl-0.2.29/
root@laptop:/var/src/gsasl-0.2.29# ./configure –prefix=/usr/local/gsasl/
root@laptop:/var/src/gsasl-0.2.29# make
root@laptop:/var/src/gsasl-0.2.29# make  install
root@laptop:/var/src/gsasl-0.2.29#

* Verify the “gsasl” version

root@laptop:/var/src/gsasl-0.2.29# /usr/local/gsasl/bin/gsasl –version
gsasl (GNU SASL) 0.2.29
Copyright (C) 2008 Simon Josefsson.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Simon Josefsson.
root@laptop:/var/src/gsasl-0.2.29#

Step 4]  Go back to Jabberd2  source and start compiling/installing as shown in below -

root@laptop:/var/src/jabberd-2.2.9# ./configure –prefix=/usr/local/jabberd-2.2.9/  –enable-debug  –enable-mysql  –enable-ssl –enable-pam  –enable-ssl  –with-extra-include-path=/usr/local/gsasl/include/ –with-extra-library-path=/usr/local/gsasl/lib/
root@laptop:/var/src/jabberd-2.2.9# make
root@laptop:/var/src/jabberd-2.2.9# make install

**  Create log and runtime directories ~

root@laptop:/usr/local/jabberd-2.2.9# mkdir /usr/local/jabberd-2.2.9/var
root@laptop:/usr/local/jabberd-2.2.9# mkdir /usr/local/jabberd-2.2.9/var/run

root@laptop:/usr/local/jabberd-2.2.9# ls -l
total 20
drwxr-xr-x 2 jabber jabber 4096 2009-10-11 18:21 bin
drwxr-xr-x 3 jabber jabber 4096 2009-10-11 18:21 etc
drwxr-xr-x 3 jabber jabber 4096 2009-10-11 18:21 lib
drwxr-xr-x 3 jabber jabber 4096 2009-10-11 18:21 share
drwxr-xr-x 3 jabber jabber 4096 2009-10-11 18:42 var
root@laptop:/usr/local/jabberd-2.2.9#

Step 5] Configure jabberd-2.2.9 ~

* Setup (jabberid@laptop.ubuntu.me) Domain Name (hostname of server),IP address,port and log setting in client (c2s.xml) & server (sm.xml) configuration file -

NOTE ~ Domain Name not necessary to be hostname of server. But it should be resolvable (DNS) to one of the IP of server.

root@laptop:/usr/local/jabberd-2.2.9# hostname
laptop.ubuntu.me
root@laptop:/usr/local/jabberd-2.2.9#

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/sm.xml

<pidfile>/usr/local/jabberd-2.2.9/var/run/sm.pid</pidfile>

<id>laptop.ubuntu.me</id>

<ip>0.0.0.0</ip>            <!– default: 127.0.0.1 –>
<port>5347</port>             <!– default: 5347 –>

<log type=’file’>
<file>/usr/local/jabberd-2.2.9/var/log/sm.log</file>

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/c2s.xml

<pidfile>/usr/local/jabberd-2.2.9/var/run/c2s.pid</pidfile>

** To auto enable registration (in c2s.xml file ‘register-enable=’true’ is required)
<id register-enable=’true’>laptop.ubuntu.me</id>

<ip>0.0.0.0</ip>
<port>5222</port>

<log type=’file’>
<file>/usr/local/jabberd-2.2.9/var/log/c2s.log</file>

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/s2s.xml

<pidfile>/usr/local/jabberd-2.2.9/var/run/s2s.pid</pidfile>
<log type=’file’>
<file>/usr/local/jabberd-2.2.9/var/log/s2s.log</file>

Step 6 ] Configure Jabberd-2.2.9 for Storage and Authentication 9using MySQL DB) -

* Make sure that database “Jabberd2″ doesn’t exist (if exist either drop db or change DB name in db-setup.mysql file). If not export MySQL DB dump from Jabberd2 source…..

root@laptop:/usr/local/jabberd-2.2.9# mysql -u root -p  <  /var/src/jabberd-2.2.9/tools/db-setup.mysql
Enter password:
root@laptop:/usr/local/jabberd-2.2.9# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 176
Server version: 5.1.31-1ubuntu2 (Ubuntu)

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> show databases;
+———————————-+
| Database                         |
+———————————-+
| information_schema                |
| jabberd2                                    |
| mysql                                             |
+———————————-+
3 rows in set (0.00 sec)

mysql> use jabberd2;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+——————–+
| Tables_in_jabberd2 |
+——————–+
| active             |
| authreg            |
| disco-items        |
| logout             |
| motd-message       |
| motd-times         |
| privacy-default    |
| privacy-items      |
| private            |
| queue              |
| roster-groups      |
| roster-items       |
| status             |
| vacation-settings  |
| vcard              |
+——————–+
15 rows in set (0.00 sec)

mysql>

*  Creating mysql user for jabberd2 ie ‘jabberd2′ with access to DB “jabberd2″ -

mysql> GRANT select,insert,delete,update ON jabberd2.* to ‘jabber’@'localhost’ IDENTIFIED by ‘mypassword’;
Query OK, 0 rows affected (0.00 sec)

mysql>quit
Bye
root@laptop:/usr/local/jabberd-2.2.9#

* Now verify access to DB ~

root@laptop:/usr/local/jabberd-2.2.9# mysql -u jabberd2 -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 178
Server version: 5.1.31-1ubuntu2 (Ubuntu)

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> show databases;
+——————–+
| Database           |
+——————–+
| information_schema |
| jabberd2           |
+——————–+
2 rows in set (0.00 sec)

mysql> quit
Bye
root@laptop:/usr/local/jabberd-2.2.9#

Step 7] Change c2s.xml and sm.xml config file for MySQL DB support –

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/c2s.xml

<!– Authentication/registration database configuration –>
<authreg>

<!– Backend module to use –>
<module>mysql</module>

<!– MySQL module configuration –>
<mysql>
<!– Database server host and port –>
<host>localhost</host>
<port>3306</port>

<!– Database name –>
<dbname>jabberd2</dbname>
<!– Database username and password –>
<user>jabberd2</user>
<pass>mypassword</pass>

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/sm.xml

<!– Storage database configuration –>
<storage>
<!– Dynamic storage modules path –>
<path>/usr/local/jabberd-2.2.9/lib/jabberd</path>

<!– By default, we use the SQLite driver for all storage –>
<driver>mysql</driver>

<!– MySQL driver configuration –>
<mysql>
<!– Database server host and port –>
<host>localhost</host>
<port>3306</port>

<!– Database name –>
<dbname>jabberd2</dbname>
<!– Database username and password –>
<user>jabberd2</user>
<pass>mypassword</pass>

** To auto enable registration (in sm.xml file)

<auto-create/>

———————————-

NOTE ~ It is not enough to add users to the ‘authreg’ table because this only introduces users to the c2s component, but not to the sm component. Correct entries are required in the ‘active’ table as well. It is best to use a Jabber client to register users.

Step 8] Let’s start Jabberd-2 server (Test configuration) –

root@laptop:~# su -l jabber -s /bin/bash -c “/usr/local/jabberd-2.2.9/bin/jabberd -b”
root@laptop:~#

* check whether ports are open or not

root@laptop:~# netstat -nlp

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5347            0.0.0.0:*               LISTEN      31662/router
tcp        0      0 0.0.0.0:5222            0.0.0.0:*               LISTEN      13883/c2s
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2892/mysqld
tcp        0      0 0.0.0.0:5269            0.0.0.0:*               LISTEN      13886/s2s
…
…..

root@laptop:~#

** Now Register the user “jabberd@laptop.ubuntu.me” and password “secret” using Jabber IM client

root@laptop:/usr/local/jabberd-2.2.9# tail -f var/log/c2s.log
…
Mon Oct 12 00:43:15 2009 [notice] [8] registration succeeded, requesting user creation: jid=jabberd@laptop.ubuntu.me
Mon Oct 12 00:43:15 2009 [notice] [8] SASL authentication succeeded: mechanism=DIGEST-MD5; authzid=jabberd@laptop.ubuntu.me
Mon Oct 12 00:43:15 2009 [notice] [8] bound: jid=jabberd@laptop.ubuntu.me/Telepathy
Mon Oct 12 00:44:20 2009 [notice] [9] [192.168.0.1, port=48307] connect

* Checking DB entry -

mysql> SELECT * FROM  active;
+————————–+—————–+————+
| collection-owner         | object-sequence | time       |
+————————–+—————–+————+
| jabberd@laptop.ubuntu.me |               1 | 1255288395 |
+————————–+—————–+————+
1 row in set (0.00 sec)

mysql> SELECT * FROM  authreg;
+———-+——————+———-+
| username | realm            | password |
+———-+——————+———-+
| jabberd  | laptop.ubuntu.me | secret   |
+———-+——————+———-+
1 row in set (0.00 sec)

mysql>

=> Testing completed successfully….

Step 9] Configuring Jabberd2 for SSL/TLS Connections –

Let’s configure jabberd2 for SSL/TLS connection. Jabberd2 is designed to provide for SSL/TLS connections not only between Jabber clients and the server, but also between the Jabberd server components (sm, s2s and c2s) and the Jabberd router. A single SSL certificate may be used for these two functions (Jabber client to Jabberd and Jabberd component to router), or two separate keys may be used.

* Generate Self signed SSL Certificate…

root@laptop:/usr/local/jabberd-2.2.9# openssl req -new -x509 -newkey rsa:1024 -days 365 -keyout privkey.pem -out server.pem
Generating a 1024 bit RSA private key
.++++++
…..++++++
writing new private key to ‘privkey.pem’
Enter PEM pass phrase:
….
Common Name (eg, YOUR name) []:laptop.ubuntu.me
root@laptop:/usr/local/jabberd-2.2.9#

* Remove Passphrase from private key

root@laptop:/usr/local/jabberd-2.2.9# openssl rsa -in privkey.pem -out privkey.pem

** Combine the Private and Public Key and delete private key

root@laptop:/usr/local/jabberd-2.2.9# cat privkey.pem >> server.pem

root@laptop:/usr/local/jabberd-2.2.9# rm privkey.pem

* Change permission…

root@laptop:/usr/local/jabberd-2.2.9# chown jabber:jabber /usr/local/jabberd-2.2.9/server.pem
root@laptop:/usr/local/jabberd-2.2.9# ls  -l   /usr/local/jabberd-2.2.9
total 24
drwxr-xr-x 2 jabber jabber 4096 2009-10-11 22:17 bin
drwxr-xr-x 3 jabber jabber 4096 2009-10-12 01:03 etc
drwxr-xr-x 3 jabber jabber 4096 2009-10-11 20:16 lib
-rw-r–r– 1 jabber jabber 2217 2009-10-12 01:17 server.pem
drwxr-xr-x 3 jabber jabber 4096 2009-10-11 20:16 share
drwxr-xr-x 4 jabber jabber 4096 2009-10-12 00:20 var
root@laptop:/usr/local/jabberd-2.2.9#

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/c2s.xml
<ssl-port>5223</ssl-port>
<pemfile>/usr/local/jabberd-2.2.9/server.pem</pemfile>

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/s2s.xml

<pemfile>/usr/local/jabberd-2.2.9/server.pem</pemfile>

root@laptop:/usr/local/jabberd-2.2.9# vi /usr/local/jabberd-2.2.9/etc/sm.xml

<pemfile>/usr/local/jabberd-2.2.9/server.pem</pemfile>

<pemfile>/usr/local/jabberd-2.2.9/server.pem</pemfile>

** Now restart the server and check log …

root@laptop:/usr/local/jabberd-2.2.9# tail -f var/log/c2s.log

Mon Oct 12 01:28:57 2009 [notice] connection to router established
Mon Oct 12 01:28:57 2009 [notice] [0.0.0.0, port=5222] listening for connections
Mon Oct 12 01:28:57 2009 [notice] [0.0.0.0, port=5223] listening for SSL connections

* While registering user ~

1) Required SSL/TLS
2) Force old SSL (5223 port)

Please enable above two setting and uncheck “Allow plaintext auth unecrypted streams”

NOTE ~ While login first time (auto registration mode) make sure to check “Create this new account on the server” checkbox in pidgin (bottom)

** It works !!

Step 10] Init.d startup script for Jabberd2 and Mu-Conference –

root@laptop:~# /etc/init.d/jabberd2 start
Starting the Jabberd2 IM Server…
router 11095 | sm 11099 | s2s 11102 | c2s 11106 |mu-conf 11149
…
Done.
root@laptop:~# /etc/init.d/jabberd2 status
Jabberd2 IM Server status -
router – 11095 | sm – 11099 | s2s – 11102 | c2s – 11106 | mu-conf 11149
root@laptop:~#

* Now check network setting…

root@laptop:~# netstat -nlp

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5347            0.0.0.0:*               LISTEN      11095/router
tcp        0      0 0.0.0.0:5222            0.0.0.0:*               LISTEN      11106/c2s
tcp        0      0 0.0.0.0:5223            0.0.0.0:*               LISTEN      11106/c2s
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2899/mysqld
tcp        0      0 0.0.0.0:5269            0.0.0.0:*               LISTEN      11102/s2s

root@laptop:~# /etc/init.d/jabberd2 stop
Stoping the Jabberd2 IM Server…
Done.
root@laptop:~#

** Want to see the script ~

root@laptop:~# cat /etc/init.d/jabberd2
#!/bin/bash

## Jabberd2 IM Server
## Jabber User/Group – jabber/jabber
## command to srart ~ su -l jabber -s /bin/bash -c “${BASE_PATH}/bin/jabberd -b”
##
#c2s
BASE_PATH=”/usr/local/jabberd-2.2.9″
c2s_pid=”${BASE_PATH}/var/run/c2s.pid”
#s2s
s2s_pid=”${BASE_PATH}/var/run/s2s.pid”
#sm
sm_pid=”${BASE_PATH}/var/run/sm.pid”
#router
router_pid=”${BASE_PATH}/var/run/router.pid”
#Mu-Conference
mu_conf_pid=”${BASE_PATH}/var/run/mu-conference.pid”

case “$1″ in

start)
## checking whether Jabberd2 is running or not
if [ -f ${c2s_pid} ];then
c2spid=$(cat ${c2s_pid})
echo “Jabberd2 IM Server ~ ‘c2s’ is running (pid ${c2spid})”
elif [ -f ${s2s_pid} ];then
s2spid=$(cat ${s2s_pid})
echo “Jabberd2 IM Server ~ ’s2s’ is running (pid ${s2spid})”
elif [ -f ${sm_pid} ];then
smpid=$(cat ${sm_pid})
echo “Jabberd2 IM Server ~ ’sm’ is running (pid ${smpid})”
elif [ -f ${router_pid} ];then
routerpid=$(cat ${router_pid})
echo “Jabberd2 IM Server ~ ‘router’ is running (pid ${routerpid})”
else
echo “Starting the Jabberd2 IM Server…”
su -l jabber -s /bin/bash -c “${BASE_PATH}/bin/jabberd -b”
su -l jabber -s /bin/bash -c “${BASE_PATH}/bin/mu-conference  -B -c ${BASE_PATH}/etc/mu-conference.xml” > /dev/null 2>&1
echo “router $(cat ${router_pid}) | sm $(cat ${sm_pid}) | s2s $(cat ${s2s_pid}) | c2s $(cat ${c2s_pid}) |mu-conf $(cat ${mu_conf_pid})”
echo “…”
echo “Done.”
fi
;;

stop)
echo “Stoping the Jabberd2 IM Server…”
if [ -f ${sm_pid} ];then
kill -9 $(cat ${sm_pid})
fi
if [ -f ${router_pid} ];then
kill -9 $(cat ${router_pid})
fi
if [ -f ${c2s_pid} ];then
kill -9 $(cat ${c2s_pid}) > /dev/null 2>&1
fi
if [ -f ${s2s_pid} ];then
kill -9 $(cat ${s2s_pid}) $(cat ${mu_conf_pid})  > /dev/null 2>&1
fi
##
killall -9  -u jabber
rm -f ${router_pid} ${sm_pid} ${s2s_pid} ${c2s_pid} ${mu_conf_pid}  > /dev/null  2>&1
echo “Done.”
;;

status)
echo “Jabberd2 IM Server status -”
if [ -f ${c2s_pid} ];then
c2spid=$(cat ${c2s_pid})
fi
if [ -f ${s2s_pid} ];then
s2spid=$(cat ${s2s_pid})
fi
if [ -f ${sm_pid} ];then
smpid=$(cat ${sm_pid})
fi
if [ -f ${router_pid} ];then
routerpid=$(cat ${router_pid})
fi
if [ -f ${router_pid} ];then
mupid=$(cat ${mu_conf_pid})
fi
echo “router – ${routerpid} | sm – ${smpid} | s2s – ${s2spid} | c2s – ${c2spid} | mu-conf ${mupid}”
;;
*)
echo “Usage: $0 {start|stop|status}”
exit 1
esac
exit 0
#DONE
root@laptop:~#

Thank you,
Arun Bagul

Introduction -

“Attansic Technology Corp. L1 Gigabit Ethernet Adapte” network (NIC) card or Adapter was not detected by RHEL4 (redhat) system. I tried running kudzu and other commands to detect device, but no use. So finally I have to install drivers for my network card…

Step 1] Device status (network card) –

* See below device status from hardware conf file ~ “/etc/sysconfig/hwconf”
* Attansic Technology Corp. L1 Gigabit Ethernet Adapter  not detected – Unknown device 8226

03:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit Ethernet Adapter (rev b0)
Subsystem: ASUSTeK Computer Inc.: Unknown device 8226
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR-
Latency: 0, Cache Line Size 10
Interrupt: pin A routed to IRQ 201
…
…..

[root@desktop ~]# lspci

03:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit Ethernet Adapter (rev b0)
…
…..
[root@desktop ~]# lspci -n

03:00.0 Class 0200: 1969:1048 (rev b0)
…
…..
[root@desktop ~]#

* Make sure to download drivers for above  venderID & deviceId “1969:1048″….

Step 2] download and extract the source  –

First, download vendor* driver from here

ftp://ftp.hogchain.net/pub/linux/attansic/vendor_driver/l1-linux-v1.2.40.3.tar.gz

OR
open-source (http://atl1.sourceforge.net/)

[root@desktop ~]# tar xvfz l1-linux-v1.2.40.3.tar.gz

[root@desktop ~]# cd  l1-linux-v1.2.40.3

[root@desktop src]# ls
at_ethtool.c  at.h  at_hw.c  at_hw.h  at_main.c  at_osdep.h  at_param.c  kcompat.c  kcompat_ethtool.c  kcompat.h  Makefile
[root@desktop src]#

* Now compile and install the drivers

[root@desktop src]# make
make -C /lib/modules/2.6.9-78.ELsmp/build SUBDIRS=/root/l1-linux-v1.2.40.3/src modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-78.EL-smp-i686′
…
…..
make[1]: Leaving directory `/usr/src/kernels/2.6.9-78.EL-smp-i686′
[root@desktop src]# echo $?
0

[root@desktop src]# make install
make -C /lib/modules/2.6.9-78.ELsmp/build SUBDIRS=/root/l1-linux-v1.2.40.3/src modules
…
…..
man -c -P’cat > /dev/null’ atl1 || true
[root@desktop src]# echo $?
0

* Now load the kernel module….

[root@desktop src]# modprobe   atl1

Step 3] Now verify whether kernel driver is working or not  –

[root@desktop src]# modinfo   atl1
filename:       /lib/modules/2.6.9-78.ELsmp/kernel/drivers/net/atl1/atl1.ko
author:         Atheros Corporation, <xiong.huang@atheros.com>
description:    Atheros 1000M Ethernet Network Driver
license:        GPL
version:        1.2.40.3 1FC4E58EBDF31F49BFD33E8
parm:           TxDescriptors:Number of transmit descriptors
parm:           RxDescriptors:Number of receive descriptors
parm:           MediaType:MediaType Select
parm:           IntModTimer:Interrupt Moderator Timer
parm:           FlashVendor:SPI Flash Vendor
vermagic:       2.6.9-78.ELsmp SMP 686 REGPARM 4KSTACKS gcc-3.4
depends:
alias:          pci:v00001969d00001048sv*sd*bc*sc*i*
[root@desktop src]#

[root@desktop src]# netconfig
[root@desktop src]# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:AD:54:0A:XX:WW
inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
inet6 addr: fe80::223:54ff:fe0a:616b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b)  TX bytes:498 (498.0 b)
Memory:feac0000-feb00000
…
…..

[root@desktop src]#

[root@desktop ~]# vi /etc/sysconfig/hwconf

class: NETWORK
bus: PCI
detached: 0
device: eth0
driver: atl1
desc: “Attansic Technology Corp. L1 Gigabit Ethernet Adapter”
network.hwaddr: 00:AD:54:0A:XX:WW
vendorId: 1969
deviceId: 1048
subVendorId: 1043
subDeviceId: 8226
pciType: 1
pcidom:    0
pcibus:  3
pcidev:  0
pcifn:  0
[root@desktop ~]#

Enjoy,
Arun Bagul

Introduction ~

This article is about how to configure TATA Indicom,BSNL and Reliance Broadband+ Netconnect ( EDVO usb modem ) on Ubuntu Linux.

To configure Reliance,BSNL and Tataindicom epi valley usb modem please refer the following article ~

http://www.indiangnu.org/2008/tata-indicom-usb-modem-epi-valley-on-ubuntu-linux/

To configure Tataindicom,Reliance Huawei datacard refer the following article ~

http://www.indiangnu.org/2008/tata-indicom-datacard-huawei-cdma-on-linuxubuntu/

** EVDO ?

EVolution-Data Optimized (EVDO) is a telecommunications standard for the wireless transmission of data through radio signals, typically for broadband Internet access. It uses multiplexing techniques including code division multiple access (CDMA) as well as time division multiple access (TDMA)
to maximize both individual user’s throughput and the overall system throughput. It is standardized by (3G) 3rd Generation Partnership Project 2 (3GPP2) as part of the CDMA2000 family of standards and has been adopted by many mobile phone service providers around the world – particularly those previously employing CDMA networks.

How to configure Reliance Broadband+ Netconnect -

Step 1] Mount USB file system to “/proc/bus/usb” –

root@laptop:~# ls  /proc/bus/usb/
root@laptop:~#

* It shows that usbfs is not mounted on “/proc/bus/usb”. To mount run following command….

root@laptop:/var/src/usb_modeswitch-1.0.5# mount -t usbfs  none /proc/bus/usb
root@laptop:/var/src/usb_modeswitch-1.0.5# ls /proc/bus/usb/
001  002  003  004  005  006  007  devices
root@laptop:/var/src/usb_modeswitch-1.0.5#

Step 2] Get the status of Reliance Broadband+ USB device ~

* lsusb list USB devices connected to PC as well as information about USB buses in the system and the devices connected to them.

* Output before connecting Reliance Broadband+ Netconnect usb modem -

root@laptop:~# lsusb
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
…..
root@laptop:~#

* Let’s connect Reliance Broadband+ Netconnect! usb modem -

root@laptop:~# lsusb
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 004: ID 19d2:fff5
Bus 005 Device 003: ID 08ff:2580 AuthenTec, Inc. AES2501 Fingerprint Sensor
…..
root@laptop:~#

* Bus 005 Device 004: ID 19d2:fff5  – This shows that the Reliance (EVDO) usb device is detected with Vendor_id  19d2  and product_id fff5

root@laptop:~# cat /proc/bus/usb/devices

T:  Bus=07 Lev=01 Prnt=01 Port=01 Cnt=01 Dev#= 23 Spd=12  MxCh= 0
D:  Ver= 1.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=19d2 ProdID=fff5 Rev= 0.00
S:  Manufacturer=ZTE, Incorporated
S:  Product=USB Storage
S:  SerialNumber=000000000002
C:* #Ifs= 1 Cfg#= 1 Atr=c0 MxPwr=100mA
I:* If#= 0 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=usbserial_generic
E:  Ad=89(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=0a(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms

root@laptop:~# dmesg | tail
[15035.656075] usb 7-2: new full speed USB device using uhci_hcd and address 24
[15035.814188] usb 7-2: configuration #1 chosen from 1 choice
[15035.827708] scsi10 : SCSI emulation for USB Mass Storage devices
[15035.828851] usb-storage: device found at 24
[15035.828856] usb-storage: waiting for device to settle before scanning
[15040.831095] usb-storage: device scan complete
[15040.834105] scsi 10:0:0:0: Direct-Access     ZTE      USB Storage FFF1 2.31 PQ: 0 ANSI: 2
[15040.839233] sd 10:0:0:0: [sdb] Attached SCSI removable disk
[15040.839378] sd 10:0:0:0: Attached scsi generic sg2 type 0
root@laptop:~#

*** Reliance Broadband+ EVDO USB is detected as “USB storage device” as shown above…

Step 3] How to use Reliance Broadband+ Netconnect as USB Modem -

To use Reliance Broadband+ usb as USB Modem. We need to switch the usb mode of this device with the help of “usb_modeswitch” tool.

Download ~ http://www.draisberghof.de/usb_modeswitch/usb_modeswitch-1.0.5.tar.bz2

Help – http://www.draisberghof.de/usb_modeswitch/

* Download and extract the “usb_modeswitch” –

root@laptop:/var/src# wget -c  http://www.draisberghof.de/usb_modeswitch/usb_modeswitch-1.0.5.tar.bz2

root@laptop:/var/src# tar xvfj usb_modeswitch-1.0.5.tar.bz2

* Now compile and install –

root@laptop:/var/src/usb_modeswitch-1.0.5# gcc -l usb -o usb_modeswitch usb_modeswitch.c
root@laptop:/var/src/usb_modeswitch-1.0.5# make install
mkdir -p /usr/sbin
install ./usb_modeswitch /usr/sbin
mkdir -p /etc
install –mode=644 ./usb_modeswitch.conf /etc
root@laptop:/var/src/usb_modeswitch-1.0.5#

**** Now configure USB mode switching -

a) Edit configuration file “/etc/usb_modeswitch.conf” -

Just search for vendor and product id eg (19d2 and fff5) in config “/etc/usb_modeswitch.conf” file….
This file contains most of the details. I choose following setting and that work’s for me

root@laptop:~# cat  /etc/usb_modeswitch.conf

###################
# ZTE AC8710
#

DefaultVendor=  0×19d2
DefaultProduct= 0xfff5

TargetVendor=   0×19d2
TargetProduct=  0xfff1

MessageContent=”5553424312345678c00000008000069f030000000000000000000000000000″

root@laptop:~#

b) Now run “usb_modeswitch” command to switch the mode of USB device -

root@laptop:~# usb_modeswitch -c /etc/usb_modeswitch.conf

Step 4] Load “usbserial” module

* Remember in Ubuntu 09.04 the “usbserial” is buildin kernel. To load that module we need to modify “grub.conf” or “menu.lst” GRUB config file

root@laptop:~# cat /boot/grub/menu.lst

title        Ubuntu 9.04, kernel 2.6.28-11-generic
uuid        c98db8a7-0a2e-4cea-b9d5-43a30c892fb0
kernel        /vmlinuz-2.6.28-11-generic root=/dev/sda5  ro quiet splash  usbserial.vendor=0×19d2  usbserial.product=0xfff1
initrd        /initrd.img-2.6.28-11-generic
quiet
….
……
root@laptop:~#

**** Reboot the machine and run the following command

* Output before switch….

root@laptop:~# lsusb
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 023: ID 19d2:fff5
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
…..
root@laptop:~#

root@laptop:~# usb_modeswitch -c /etc/usb_modeswitch.conf

* Output after switch….

root@laptop:~# lsusb
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 024: ID 19d2:fff1
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
…..
root@laptop:~#

root@laptop:~# usb_modeswitch  -v 19d2 -p fff1

Looking for default devices …
Found default devices (1)
Accessing device 004 on bus 005 …
Using endpoints 0×02 (out) and 0×82 (in)
Not a storage device, skipping SCSI inquiry

Device description data (identification)
————————-
Manufacturer: ZTE, Incorporated
Product: ZTE CDMA Tech
Serial No.: not provided
————————-
Warning: no switching method given.
-> Run lsusb to note any changes. Bye.

root@laptop:~#

root@laptop:~# cat /proc/bus/usb/devices

T:  Bus=07 Lev=01 Prnt=01 Port=01 Cnt=01 Dev#= 24 Spd=12  MxCh= 0
D:  Ver= 1.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=19d2 ProdID=fff1 Rev= 0.00
S:  Manufacturer=ZTE, Incorporated
S:  Product=ZTE CDMA Tech
C:* #Ifs= 6 Cfg#= 1 Atr=a0 MxPwr=500mA
I:* If#= 0 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
E:  Ad=81(I) Atr=03(Int.) MxPS=  16 Ivl=128ms

root@laptop:~# dmesg  | tail
[   70.985075] usbserial_generic 5-1:1.1: generic converter detected
[   70.985109] usb 5-1: generic converter now attached to ttyUSB1
[   70.987028] usbserial_generic 5-1:1.2: generic converter detected
[   70.987064] usb 5-1: generic converter now attached to ttyUSB2
[   70.989589] usbserial_generic 5-1:1.3: generic converter detected
[   70.989623] usb 5-1: generic converter now attached to ttyUSB3
[   70.991023] usbserial_generic 5-1:1.4: generic converter detected
[   70.991061] usb 5-1: generic converter now attached to ttyUSB4
[   70.993066] usbserial_generic 5-1:1.5: generic converter detected
[   70.993109] usb 5-1: generic converter now attached to ttyUSB5
root@laptop:~#

*** It shows that Reliance Broadband+ EVDO usb  is detected as CDMA modem

Step 5] Configure wvdail –

* Run “wvdialconf” to detect and edit “/etc/wvdial.conf” confile

root@laptop:~# wvdialconf
Editing `/etc/wvdial.conf’.
Scanning your serial ports for a modem.
Modem Port Scan<*1>: S0   S1

root@laptop:~# cat /etc/wvdial.conf

[Dialer Defaults]
Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0
Password = your_mobile_no
Username = your_mobile_no
Phone = #777
PPPP Path = /usr/sbin/pppd
Modem Type = Analog Modem
Stupid Mode = 1
Tonline = 0
Baud = 9600
New PPPD = 1
Modem = /dev/ttyUSB0
ISDN = 0
root@laptop:~#

* Now it’s time to start surfing…..

root@laptop:~# wvdial &
[1] 21710
root@laptop:~#

root@laptop:~# ifconfig

ppp0 Link encap:Point-to-Point Protocol
inet addr:115.184.XX.BB  P-t-P:220.224.CC.DD  Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
RX packets:4310 errors:0 dropped:0 overruns:0 frame:0
TX packets:4347 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:2268618 (2.2 MB)  TX bytes:445276 (445.2 KB)

Enjoy,
Arun Bagul