Performing Linear Regression in Perl

Linear regression is a very commonly used data analysis technique that can easily be performed in Perl using the module Statistics::Regression. This module is capable of performing multivariate linear regressions, but this code snippet will demonstrate a simple regression to the basic equation of a line y=mx +b, using the following data set:

X	Y
1	1.3
2	2.9
3	4.2
4	5.4

#!usr/bin/perl

use Statistics::Regression;
use strict;
use warnings;

my $reg=Statistics::Regression->new(
   "Title", ["Intercept", "Slope"]
   );
 
#the use of 1.0 in the includes allows
#for the computation of a y intercept     
$reg->include(1.3, [1.0, 1.0]);
$reg->include(2.9, [1.0, 2.0]);
$reg->include(4.2, [1.0, 3.0]);
$reg->include(5.4, [1.0, 4.0]);

$reg->print;

To understand how the module works, let’s consider the equation in a more multivariate form by rewriting it as y=bx₁ + mx₂.  Considering the equation in this form, will make the “include” statements that provide the data points to the module much easier to understand.  The include statements could be considered as using the following data entry format:

$reg->include(y, [x₁, x₂]);

Thus by always putting 1.0 in the x₁ position it allows for the computation of a Y-intercept since it is telling the regression module that the x₁ variable does not influence the outcome of bx₁, whereas the x₂ values correspond to the X values, since we want mx₂ to change with a change in X.

A look at the output generated by the Perl module demonstrates the following results:

****************************************************************

Regression 'Title'

****************************************************************

Name                   Theta          StdErr     T-stat

[0='Intercept']       0.0500          0.1775       0.28

[1='Slope']           1.3600          0.0648      20.99

R^2= 0.995, N= 4, K= 2

****************************************************************

The plot of the line that would result from the regression is as follows:

 

Split an Array into Two Equal Parts

This is a small snippet of Perl code which demonstrates a quick and easy way to take an array that contains an even number of elements and split it up into two smaller arrays.  Assuming the array values are read left to right, the array called @Left will store the first half of the array elements, while the array called @Right will store the second half of the array elements.  

#!usr/bin/perl
use strict;
use warnings;

#generate 10 random numbers between 0 and 9
my $range=10;
my @nums = map { int(rand($range)) } ( 1..10 );

#splits array in half
my @Left = @nums;
my @Right = splice @Left, scalar(@nums)/2;

#formats data to visually show split
my $Lefthalf=join('',@Left);
my $Righthalf=join('',@Right);
print "$Lefthalf\n";
print "$Righthalf\n";

A Simple Perl-Based RSS to Email Program

While many sites offer RSS feeds as a means of keeping up to date with new content, not everyone is as familiar with RSS and RSS feed readers as they are with Email and Email clients.  To deal with this issue, here is a small Perl code snippet which will download an RSS feed, parse out the title and the description associated with each RSS entry, and email out the extracted information.  In the interest of simplicity for the sample code, it extracts the item titles and description from every element in the feed, but with a little modification it could be adapted to only utilize the most recent feed entries.  Running the script periodically to check for new RSS entries could be easily handled via cron jobs or other job scheduler. 

The Email sending portion of the code was tested using Gmail.  

 #!usr/bin/perl

use Email::Sender::Simple qw(sendmail);
use Email::Sender::Transport::SMTP::TLS;
use Email::Simple;
use Email::Simple::Creator;
use XML::RSS::Parser;
use LWP;
use strict;
use warnings;

my $feedurl='http://feeds.feedburner.com/SansInstituteNewsbites?format=xml';

my $ua=LWP::UserAgent->new;
my $response=$ua->get($feedurl);
my $rss=$response->content;

my $parser= new XML::RSS::Parser;
my $feed=$parser->parse_string($rss);
my $feed_title = $feed->query('/channel/title');
my $message =  $feed_title->text_content;
my $count = $feed->item_count;
$message.=" ($count)\n";
foreach my $i ( $feed->query('//item') ) {
  my $node = $i->query('title');
  $message.= '  '.$node->text_content . "\n";
  $node = $i->query('description');
  $message.='  '.$node->text_content;
  $message.= "\n\n";
}
#strips out feed specific html tags because I chose to send plain text email
$message=~s/((<p>)|(<\/p>))//gio;

my $transport = Email::Sender::Transport::SMTP::TLS->new(
   host => 'smtp.gmail.com',
   port => 587,
   username => 'somebody@gmail.com',
   password => 'password'
);

my $email = Email::Simple->create(
    header => [
      To      => '"Somebody" <somebody@gmail.com>',
      From    => '"Somebody Else" <s_else@gmail.com>',
      Subject => "The Latest Updates",
    ],
    body => "$message",
);
sendmail($email, {transport => $transport});

Companies That Use Perl

I recently saw this site mentioned in one of the LinkedIn Perl forums and thought it provides an interesting way to promote Perl.  It is a listing of many of the companies in the Los Angeles area that make use of Perl for major product development- http://perl.la/ One thing that I think is also interesting about this site is the way they present the list of companies.  While I can think of many reasons a “text-like” list of hyperlinks (used by other such lists) can be advantageous, the graphical way perl.la displays its data is probably an interface with more marketing appeal. 

The list contains a number of companies that should be widely recognizable to those that are not already involved in the Perl community or even a computer related occupation – companies like CBS television, DreamWorks, Ticketmaster, and Experian. 

Seeing this site also made me think of some other lists of companies that use Perl such as these from the Perl 5 Wiki and other sources:

A list of companies using Perl - https://www.socialtext.net/perl5/Companies%20using%20Perl

A list of applications written in Perl - https://www.socialtext.net/perl5/applications

A list of companies that use Catalyst - http://wiki.catalystframework.org/wiki/sitesrunningcatalyst

A list of companies that use Moose - http://moose.iinteractive.com/about.html

A list of companies that use Mojolicious - https://github.com/kraih/mojo/wiki/Projects-and-Companies-Using-Mojolicious

Perl is clearly still thriving!!!

A Perl PDL Demonstration Using the Hill Cipher Algorithm

PDL is an excellent Perl module for anyone seeking to do any type of numerical computing that involves matrix math.  As an illustration of this, let’s take a look at a PDL implementation of a simple encryption algorithm that uses matrix math, the Hill Cipher.  As a starting point, the Hill Cipher assigns each letter in the alphabet a numerical value (e.g. a=0, b=1, c=2, … z=25).  The cipher works by taking n plain text letters and converting them to their numeric representation.  In the example code a n of 3 is used. We will refer to this set of numerically represented plain text letters as P. It then takes an nXn matrix, which is used as an encryption key (K), and performs the following matrix math operation to yield the numerical representation of n cipher text letters (C).  

C=P x K mod 26

If you look at the Perl code below, notice how easy PDL makes this operation.  There is no need to loop through the elements of an array and perform mathematical operations on each element.   One can simply process entire matrices as a single entity and as such this is one of the features that makes PDL such a huge asset to anyone that performs numerical computing in Perl. 

The reverse of this process is performed by taking the inverse matrix (K1) and performing the following math operation:

P=C x K1 mod 26

As a means of testing the example code, a known set of values from the book “Cryptography and Network Security” by William Stallings are used.  If run as is, the sample code should result in the cipher text RRLMWBKASPDH after encryption and the decryption should return the original plain text value.  

#usr/bin/perl

use PDL;
use strict;
use warnings;

my @letters = ('a'..'z');
my (%encoding,%encoding2);
my $i=0;
foreach my $letter (@letters){
    $encoding{$letter}=$i;
    $encoding2{$i}=$letter;
    $i++;
}

#encryption
my $plaintext='paymoremoney'; #length=multiple of 3
$plaintext=lc($plaintext);
my $ciphertext='';

#encryption key
my $k = pdl [[17,17,5],[21,18,21],[2,2,19]];

while($plaintext=~/(\w)(\w)(\w)/g){
    my $x=$encoding{$1};
    my $y=$encoding{$2};
    my $z=$encoding{$3};
    my $p= pdl [$x,$y,$z];
    my $c= $p x $k % 26;
    foreach (0 .. $c->nelem-1){
        my $j=$c->flat->index($_);
        $ciphertext.=$encoding2{$j};
    }
}
print uc($ciphertext)."\n\n";

#decryption
$plaintext='';
#inverse matrix
#hardcoded and not computed to simplify example code
my $k1= pdl [[4,9,15],[15,17,6],[24,0,17]];
while($ciphertext=~/(\w)(\w)(\w)/g){
    my $x=$encoding{$1};
    my $y=$encoding{$2};
    my $z=$encoding{$3};
    my $c= pdl [$x,$y,$z];
    my $p= $c x $k1 % 26;
    foreach (0 .. $p->nelem-1){
        my $j=$p->flat->index($_);
        $plaintext.=$encoding2{$j};
    }
}
print $plaintext;