NY Perl Mongers Talk Slides

Information Retrieval and Extraction from cfrenz

Slides from a talk I recently gave at a NY Perl Mongers Meetup 

Perl and Open Government

For any interested in initiatives to make government data more accessible, I ran across an interesting API put out by Civic Impulse – the GovTrack API (http://www.govtrack.us/developers/api).  The API allows queries to be made regarding many of the bills being debated about in the US congress as well as the ability to programmatically look up information pertaining to members of Congress.  The available documentation for the API is located at the link already provided, but a small Perl script below will demonstrate the basics of accessing the API in Perl.

This script will query the API for bills regarding “fracking” in the “112” session of congress and will simply print out the retrieved results.

#!usr/bin/perl

use LWP;
use strict;
use warnings;

#sets query and congress session
my $query='fracking';
my $congress=112;

my $ua = LWP::UserAgent->new;

my $url="http://www.govtrack.us/api/v2/bill?q=$query&congress=$congress";

my $response=$ua->get($url);
my $result=$response->content;
print $result;

The results are returned in the JSON format be default, but the API does allow for parameters to be specified for XML or CSV based results as alternatives.  For the code above the returned JSON would look as follows:

 {
 "meta": {
  "limit": 100,
  "offset": 0,
  "total_count": 2
 },
 "objects": [
  {
   "bill_resolution_type": "bill",
   "bill_type": "senate_bill",
   "bill_type_label": "S.",
   "congress": 112,
   "current_status": "referred",
   "current_status_date": "2012-03-28",
   "current_status_description": "This bill was introduced on March 28, 2012, in a previous session of Congress, but was not enacted.",
   "current_status_label": "Referred to Committee",
   "display_number": "S. 2248",
   "docs_house_gov_postdate": null,
   "id": 251518,
   "introduced_date": "2012-03-28",
   "is_alive": false,
   "is_current": false,
   "link": "http://www.govtrack.us/congress/bills/112/s2248",
   "major_actions": [
    [
     "datetime.datetime(2012, 3, 28, 0, 0)",
     1,
     "Sponsor introductory remarks on measure. (CR S2166-2167)"
    ],
    [
     "datetime.datetime(2012, 3, 28, 0, 0)",
     2,
     "Read twice and referred to the Committee on Energy and Natural Resources."
    ]
   ],
   "noun": "bill",
   "number": 2248,
   "senate_floor_schedule_postdate": null,
   "sliplawnum": null,
   "sliplawpubpriv": null,
   "sponsor": {
    "bioguideid": "I000024",
    "birthday": "1934-11-17",
    "cspanid": 5619,
    "firstname": "James",
    "gender": "male",
    "gender_label": "Male",
    "id": 300055,
    "lastname": "Inhofe",
    "link": "http://www.govtrack.us/congress/members/james_inhofe/300055",
    "middlename": "M.",
    "name": "Sen. James “Jim” Inhofe [R-OK]",
    "namemod": "",
    "nickname": "Jim",
    "osid": "N00005582",
    "pvsid": "27027",
    "sortname": "Inhofe, James “Jim” (Sen.) [R-OK]",
    "twitterid": "InhofePress",
    "youtubeid": "jiminhofepressoffice"
   },
   "sponsor_role": {
    "congress_numbers": [
     111,
     112,
     113
    ],
    "current": true,
    "description": "Senator from Oklahoma",
    "district": null,
    "enddate": "2015-01-03",
    "id": 4082,
    "party": "Republican",
    "person": 300055,
    "role_type": "senator",
    "role_type_label": "Senator",
    "senator_class": "class2",
    "senator_class_label": "Class 2",
    "startdate": "2009-01-06",
    "state": "OK",
    "title": "Sen.",
    "title_long": "Senator",
    "website": "http://www.inhofe.senate.gov"
   },
   "thomas_link": "http://thomas.loc.gov/cgi-bin/bdquery/z?d112:s2248:",
   "title": "S. 2248 (112th): Fracturing Regulations are Effective in State Hands Act",
   "title_without_number": "Fracturing Regulations are Effective in State Hands Act",
   "titles": [
    [
     "short",
     "introduced",
     "Fracturing Regulations are Effective in State Hands Act"
    ],
    [
     "official",
     "introduced",
     "A bill to clarify that a State has the sole authority to regulate hydraulic fracturing on Federal land within the boundaries of the State."
    ]
   ]
  },
  {
   "bill_resolution_type": "bill",
   "bill_type": "house_bill",
   "bill_type_label": "H.R.",
   "congress": 112,
   "current_status": "referred",
   "current_status_date": "2012-03-29",
   "current_status_description": "This bill was introduced on March 29, 2012, in a previous session of Congress, but was not enacted.",
   "current_status_label": "Referred to Committee",
   "display_number": "H.R. 4322",
   "docs_house_gov_postdate": null,
   "id": 251585,
   "introduced_date": "2012-03-29",
   "is_alive": false,
   "is_current": false,
   "link": "http://www.govtrack.us/congress/bills/112/hr4322",
   "major_actions": [
    [
     "datetime.datetime(2012, 3, 29, 0, 0)",
     2,
     "Referred to the Committee on Natural Resources, and in addition to the Committees on Agriculture, Transportation and Infrastructure, and Energy and Commerce, for a period to be subsequently determined by the Speaker, in each case for consideration of such provisions as fall within the jurisdiction of the committee concerned."
    ]
   ],
   "noun": "bill",
   "number": 4322,
   "senate_floor_schedule_postdate": null,
   "sliplawnum": null,
   "sliplawpubpriv": null,
   "sponsor": {
    "bioguideid": "G000552",
    "birthday": "1953-08-18",
    "cspanid": 1011394,
    "firstname": "Louie",
    "gender": "male",
    "gender_label": "Male",
    "id": 400651,
    "lastname": "Gohmert",
    "link": "http://www.govtrack.us/congress/members/louie_gohmert/400651",
    "middlename": "B.",
    "name": "Rep. Louie Gohmert [R-TX1]",
    "namemod": "Jr.",
    "nickname": "",
    "osid": "N00026148",
    "pvsid": "50029",
    "sortname": "Gohmert, Louie (Rep.) [R-TX1]",
    "twitterid": "replouiegohmert",
    "youtubeid": "GohmertTX01"
   },
   "sponsor_role": {
    "congress_numbers": [
     112
    ],
    "current": false,
    "description": "Representative for Texas's 1st congressional district",
    "district": 1,
    "enddate": "2013-01-03",
    "id": 5197,
    "party": "Republican",
    "person": 400651,
    "role_type": "representative",
    "role_type_label": "Representative",
    "senator_class": null,
    "startdate": "2011-01-05",
    "state": "TX",
    "title": "Rep.",
    "title_long": "Representative",
    "website": "http://gohmert.house.gov"
   },
   "thomas_link": "http://thomas.loc.gov/cgi-bin/bdquery/z?d112:hr4322:",
   "title": "H.R. 4322 (112th): Fracturing Regulations are Effective in State Hands Act",
   "title_without_number": "Fracturing Regulations are Effective in State Hands Act",
   "titles": [
    [
     "short",
     "introduced",
     "Fracturing Regulations are Effective in State Hands Act"
    ],
    [
     "official",
     "introduced",
     "To clarify that a State has the sole authority to regulate hydraulic fracturing on Federal land within the boundaries of the State."
    ]
   ]
  }
 ]
}

Perl and Shodan

The search engine Shodan (http://www.shodanhq.com/) has recently drawn a lot of attention as the “scariest search engine on the internet” since it lets you search for computers and other devices by IP, OS, location, etc, and in doing so often reveals information that the computer owner’s might not have ever intended to be public.  For security professionals it makes for an interesting tool for pen-testing and forensics.  For any Perl developers that are interested in such work the good news is that they have an API and they have a nice Perl tutorial already in place documenting how to use it (http://docs.shodanhq.com/perl/tutorial.html#).  They also provide one sample Perl script that lets you generate a list of IP addresses that match your query terms (http://docs.shodanhq.com/perl/examples.html).  The API does make use of an API key that requires registration to use. 

Randomize the Elements of an Array

Sometimes it is useful to be able to randomize the elements of an array and this posting will demonstrate a simple way of accomplishing that using the shuffle subroutine of the List::Util Perl module.  As a sample use case let’s consider a simplistic random password generator that was designed to generate passwords that contain two uppercase characters, two numbers, and four lowercase characters. A sample code snippet that would accomplish such a task might look as follows:

#!usr/bin/perl

use List::Util 'shuffle';

use strict;

use warnings;

#Ensure number of each character type requirements are met

my @randuc=map{('A'..'Z')[rand(26)]}(1..2);

my @randlc=map{('a'..'z')[rand(26)]}(1..4);

my @randnum=map{int(rand(10))}(1..2);

my @pass=(@randuc, @randlc, @randnum);

my $passwd=join("", @pass);

print "$passwd \n";

If executed, this code would create random passwords such as “SEadzs55”, which clearly meet the complexity requirements, but if we run the password generator over and over again a pattern should emerge.  The two upper case characters will always be first, followed by the four lowercase characters, followed by the two numbers.  This pattern greatly reduces the randomness of our passwords and is a common mistake you see in many random password generators.  We could greatly improve this code by adding some additional code that will randomize the elements of the password, so there is no definitive pattern as to the ordering of characters.  We could do this with the following code snippet:

#Shuffles the generated characters so uc characters not always first

#followed by lc characters, etc to improve randomness

my @mixed = shuffle(@pass);

$passwd=join("", @mixed);

print "$passwd \n";

If we now execute the code with this addition multiple times, will see that after each execution the ordering of the characters is randomized, thereby improving the entropy of our password.