Monday, October 26, 2009

Calling a Perl Script from a web page using JavaScript

Ever wanted to include some dynamic data on your web page, but didn't want to (or restrictions didn't allow you to) code the entire page server-side? There is actually a pretty straightforward way to accomplish this, by making the Perl script return valid JavaScript and then calling the Perl script as if it were JavaScript. Confused? I was too until I developed a working example of how it works.

For this example, I will combine the random quotation Perl script I outlined in a previous post with cowsay, also outlined in a previous post on this blog. The Perl script will call the random quotation script, pipe it through cowsay, then return the output in JavaScript to be executed by the browser.

First, the Perl code; I'll call it "cow_quote.pl":
#!/usr/bin/perl 

use strict;
my $line;

# Get a random quotation from www.quotationspage.com
# using random_quote.pl, then pipe the output through
# the cowsay program, limiting the output to 30 columns
my $cowsay = `perl random_quote.pl | cowsay -W 30`;

# Substitute all occurrences of the backslash ("\")
# with the HTML escape code ("\")
$cowsay =~ s/\\/\/g;

# Substitute all occurrences of the space (" ")
# with the HTML escape code (" ")
$cowsay =~ s/ / /g;

# Split the resulting cowsay text into individual lines
my @cowsay = split(/\n/, $cowsay);

# Declare the Internet Media (MIME) type  - The browser
# REQUIRES this to be declared; without it this example
# WILL NOT WORK!
print "Content-type: text/html\n\n";

# Using CSS, declare a style for our DIV tag, setting the
# font to bold courier, with the size of the font being
# 83% of the normal size.  This will permit us to utilize
# all 30 columns of the cowsay text in the narrow sidebar. 
print "document.write(\"<style>\\n\");\n";
print "document.write(\"  \#cowsay\\n\");\n";
print "document.write(\"  {\\n\");\n";
print "document.write(\"    font-family:courier;\\n\");\n";
print "document.write(\"    font-weight:bold;\\n\");\n";
print "document.write(\"    font-size:83%;\\n\");\n";
print "document.write(\"  }\\n\");\n";
print "document.write(\"</style>\\n\");\n";

# Create the cowsay DIV tag, and output the multiple lines
# of the cowsay text, separating each line with an HTML
# break, then close the DIV tag.
print "document.write(\"<div id=\'cowsay\'>\\n\");\n";
foreach $line (@cowsay)
{
  print "document.write(\"  $line\\n\");\n";
  print "document.write(\"  <br />\\n\");\n";
}
print "document.write(\"</div>\\n\");\n";
As you can see, all the output from the Perl script is in JavaScript, allowing the code to be executed client-side by the browser.

Now all that we need to do is to call the script from the web page, using the following code:
<script type="text/javascript" src="http://yourdomain.com/cgi-bin/cow_quote.pl"> 
</script>
Of course, you would substitute your domain for the one listed above.

This is by no means the limit of what could be done utilizing this method. As long as the output is in JavaScript, the sky is the limit!

The Perl source for cow_quote.pl is available here. In order for function properly, it does require that cowsay and random_quote.pl (and it's dependent Perl modules) be installed on your server.

Read More...

Wednesday, October 21, 2009

Using Perl's LWP::UserAgent and HTML::Form Modules to Extract Data From a Web Page

I wanted a script that would display a random quotation each time I logged into my server from the command line. I suppose I could have used Linux's fortune program, but thats been done before, and besides, I wanted something different.

Here's how I used two modules from libwww-perl (LWP::UserAgent and HTML::Form) to extract random quotations from www.quotationspage.com. The libwww-perl collection is a set of Perl modules which provides a simple and consistent application programming interface to the World-Wide Web. The main focus of the library is to provide classes and functions that allow you to write WWW clients.

With amazingly little code, a perl script can be be written that retrieves the HTML returned when a web page is requested, just like a browser. This is accomplished thru the use of perl's LMP::UserAgent, a perl module that can be used to dispatch web requests.

In my case, I also needed to fill out an HTML form on a web page, click a Submit button, and extract specific data from the returned page. Again, very easy using another Perl module, HTML::Form.

In the following example, I hope to show how easy it can be to extract specific data from a web page using just these two Perl modules.

The two Perl modules used in this example are not installed by default. By far the easiest way to install them is by using your distro's package manager to install the libwww-perl package. That way you will get not only the two modules needed for this particular example, but the entire library of Perl www related modules. If you end up doing a lot of web related Perl programming, you will find uses for a lot more of the modules in the libwww-perl collection, so you might as well just install them all now.

I am going to break the script into four units and explain the function of each unit. This should allow anyone reading this post to get a pretty good understanding of how the script works.

#!/usr/bin/perl 
use strict; 
 
use LWP::UserAgent; 
use HTML::Form; 
 
my ($quote_count,$return_count,$user_agent,$response,@forms, 
    @checkbox_values,@form_out,$form,@inputs,$inp,$type, 
    $value,$check_value,$name,$page,$display_count,$quote, 
    $quote_string,$author); 
 
# Store the web form's checkbox names in an array  
@checkbox_values = ("mgm","motivate","classic","coles", 
                    "lindsly","poorc","altq","20thcent", 
                    "bywomen","devils","contrib"); 
 
# If the number of quotes to display is passed as a  
# command line parameter, store it in $quote_count,  
# otherwise set $quote_count to 1 
if (@ARGV) {$quote_count = $ARGV[0]}  
else {$quote_count = 1}  
 
# www.quotationspage.com's form has a minimum number of 4 
# and a maximum of 15 quotes.  If the number of requested  
# quotes is less than 4 or greater than 15, set the number 
# of returned quotes within those limits 
$return_count = $quote_count;  
if ($quote_count < 4) {$return_count = 4}  
if ($quote_count > 15) {$return_count = 15} 
 
# Create the UserAgent object 
$user_agent = LWP::UserAgent->new;
The beginning section lists the Perl modules used by the program (LWP::UserAgent and HTML::Form), declares all the variables, sets the number of quotes to be displayed (also checks if number is passed from the command line), and creates an instance of the UserAgent object.


# Retrieve the form from the webpage and store the form  
# in an HTML::Form hash 
$response = $user_agent->get("http://www.quotationspage.com/random.php3");  
@forms = HTML::Form->parse($response);  
 
# Clear the array that the modified form will be "pushed"  
# into, ignore the first form (it's not the one we want) 
# and store the form hash in an array (@inputs) 
undef(@form_out);  
$form = shift(@forms);  
$form = shift(@forms);  
@inputs = $form->inputs; 
This section retrieves the form from the web page and stores the form data in an array that can be modified.


# Parse the array containing the form data, entering the  
# number of quotes to request, checking all the checkboxes, 
# and filling out the outgoing form to be returned to the 
# web page's php program that processes the form 
for (my $i=0 ; $i<=$#inputs ; $i++)  
{ 
  $inp = $inputs[$i]; 
  $type = $inp->type; 
  if ($type eq "option") # Set the quote count  
  {$inp->value($return_count)} 
  if ($type eq "checkbox") # Check the checkboxes 
  { 
    $check_value = shift(@checkbox_values); 
    $inp->value($check_value); 
  }  
  $value = $inp->value; 
  $name = $inp->name; 
  if ($type ne "submit") {push(@form_out,$name,$value)}  
} 
 
# Send the completed form to the php script that processes 
# the web form, and store the HTML that comes back in a  
# string called $page 
$response = $user_agent->post('http://www.quotationspage.com/random.php3',\@form_out); 
$page = $response->as_string;
Now the form data is modified and "submitted" on the web page


# Parse the HTML stored in $page, extracting the quotations 
# formatting the text, and displaying the requested number 
# of quotes 
 
$display_count = 0; 
 
# Look for a quote in the HTML 
while ($page =~ m/<dt class=\"quote\">(.*?)<\/dd>/gs) 
{ 
  $quote = $1;  
  if ($quote =~ m/\.html">(.*?)<\/a>/) # Extract the quote 
  {  
    $quote_string = $1; 
 
    # Replace any HTML break statements with Newlines 
    $quote_string =~ s/<br>/\n/gs;  
  }  
  if ($quote =~ m/<b>(.*?)<\/b>/) # Extract the author 
  {  
    $author = $1; 
    # If the author is imbedded in a link, remove the link HTML 
    if ($author =~ m/\/">(.*?)$/)  
    {  
      $author = $1; 
      $author =~ s/<\/a>//;  
    } 
    $display_count++; 
    if ($display_count <= $quote_count)  
    {print $quote_string." - ".$author."\n\n"}  
  } 
}
The final section extracts the quotes from the HTML of the page that was returned when the "Sumbit" button was pressed and displays the quotes.


The entire Perl script is available here, if you would like to simply download the script.

Special thanks to Vivian's Tech Blog for the code that permitted me to display the code in the shaded boxes!

Read More...

Sunday, October 11, 2009

cowsay - A Configurable Speaking/Thinking Cow

cowsay is a silly little command line text filter that displays a cow saying (or thinking) whatever text you feed it.  For example, each time I log into my server via ssh I am greeted by a friendly cow who gives me a random quote.

cowsay has a number of options that change how the text is displayed and the appearance of the cow (dead, greedy, paranoid, etc.).  There is also a library of "cowfiles" - alternate pictures that can be displayed in place of the cow.

Some of the options available:

The -n option turns off word wrap. If it is specified, the given message will not be word wrapped.

The  -W specifies roughly (where the message should be wrapped.  The default is equivalent to -W 40 i.e. wrap words at or before the 40th column.

There  are  several  provided modes which change the appearance of the cow depending on its particular emotional/physical state.
The -b option initiates Borg mode;
       -d causes the cow to appear dead;
       -g invokes greedy mode;
       -p causes a state of paranoia to come over the cow;
       -s makes the cow appear thoroughly stoned;
       -t yields a tired cow;
       -w is somewhat the opposite of -t, and initiates wired mode;
       -y brings on the cow's youthful appearance.
The -f option  specifies  a particular cow picture file (``cowfile'') to use.
The -l option lists all the cowfiles available

Making the cow say what you want is easy: cowsay moo displays a cow saying "moo".  You can also "pipe" the text to be displayed, like this: echo moo | cowsay .  To display a dragon saying the content of a text file called cowtext, enter cat ~/cowtext | cowsay -f dragon


Installing cowsay

If cowsay isn't already installed in your distribution, installing cowsay is a snap.  Almost all distros include cowsay in their repositories, so its a simple matter of installing via your distro's package manager. (apt/Synaptic, yum, pacman, YasT, etc.)

In the event you can't find a cowsay package for your flavor of Linux, you can download the source tarball from http://www.nog.net/~tony/warez/cowsay-3.03.tar.gz and build cowsay from source.

Cowsay Homepage: http://www.nog.net/~tony/warez/cowsay.shtml

Read More...

Saturday, October 10, 2009

Where Wizards Stay Up Late - The Origins of the Internet


While perusing the computer books at my local library I came across a book entitled Where Wizards Stay Up Late - The Origins of the Internet, by Katie Hafner & Matthew Lyon

It looked interesting, so I checked it out.

What a fascinating read. I couldn't put the book down. I ended up reading the entire book in one sitting.

From early meetings discussing the need to link distant computers together, all the way through the development of HTML and Mosaic (the first browser) this book details not only the technologies that were developed, but also the people and organizations that made it all happen.

It is written so even a person with no background in the underlying hardware, software, and protocols involved with the internet can read it and understand, but yet someone with a basic understanding of those concepts isn't put off.

This book is a really good read, and anyone with even the slightest interest in how the internet as we know it came to be should read it.

Read More...

Friday, October 9, 2009

Beginner's Linux Command Cheat Sheet


  As a new (or experienced) Linux user, ever wish you had a handy cheat sheet filled with the most commonly used terminal commands for things like file operations, process management, permissions, ssh, searching, system information, compression and networking?

  Now you can, thanks to Jacob Peddicord and FOSSwire.  Designed as a reference to assist both new and seasoned Linux users remember the syntax for common cli commands, the Unix/Linux Command Cheat Sheet is downloadable in PDF format in english as well as a variety of other languages.

Read More...

GNU nano - A small, free and friendly text editor


  GNU nano is a cli (command line interface) editor for the Linux platform.  It was originally developed as a free replacement for the non-free Pico editor included as a part of the email client Pine.

  Unlike vim or emacs, nano allows even the most inexperienced user to easily create or edit text files.  Typing nano filename at the command line opens nano and either loads the file filename if it exists, or will create the file upon saving/exiting nano.

  Nano has a very short learning curve.  Immediately upon opening nano you can begin editing text, and simple,easy to remember commands control things like cut/paste and find/replace.

  Nano also includes a multitude of other options for setting things like word wrap, mouse control, etc.  These options can be specified when nano is run, or be configured to default settings by editing the .nanorc file in the user's home folder.


Here is a short list of  basic nano commands:

Saving and exiting
   If you want to save the changes you've made, press Ctrl+O. To exit nano, type Ctrl+X. If you ask nano to exit from a modified file, it will ask you if you want to save it. Just press N in case you don't, or Y in case you do. It will then ask you for a filename. Just type it in and press Enter.

  If you accidentally confirmed that you want to save the file but you actually don't, you can always cancel by pressing Ctrl+C when you're prompted for a filename.

Cutting and pasting
   To cut a single line, you use Ctrl+K (hold down Ctrl and then press K). The line disappears. To paste it, you simply move the cursor to where you want to paste it and press Ctrl+U. The line reappears. To move multiple lines, simply cut them with several Ctrl+Ks in a row, then paste them with a single Ctrl+U. The whole paragraph appears wherever you want it.

  If you need a little more control, then you have to mark the text. Move the cursor to the beginning of the text you want to cut. Hit Alt+A. Now move your cursor to the end of the text you want to cut: the marked text gets highlighted. If you need to cancel your text marking, simply hit Alt+A again. Press Ctrl+K to cut the marked text. Use Ctrl+U to paste it.

Searching for text
   Searching for a string is easy as well.  Simply hit Ctrl+W, type in your search string, and press Enter. To search for the same string again, hit Alt+W.

Getting nano
  Nano is already installed on most modern Linux distributions, and can usually easily be added via a package manager if it isn't already installed.  If your particular flavor of Linux doesn't have a nano package available, you can build it from source by downloading the source from the nano homepage, http://www.nano-editor.org

Read More...

Wednesday, October 7, 2009

Slax, the modular "Build it Yourself" Linux Distribution

  As an attendee of Ohio LinuxFest 2009, I received a complimentary copy of the September 2009 issue of LinuxPro magazine.  One of the articles in the issue was about Slax,  a unique Linux distribution with a modular design that allows you to build a custom distro directly from the Slax website.  The article intrigued me, and being the "distro whore" that I am, I had to play around with Slax.

  Building a custom Slax distro was ridiculously easy.  I visited the Build Slax page on the Slax website,  chose the modules I wanted to be included in my custom ISO, and clicked Download ISO.  It was that easy!


  If you decide later that you have additional modules you want to add, you can return to the website, add the additional modules, and download a new ISO.

  It's a nifty concept with lots of interesting possibilites.  You can easily create a sub-200MB distro that fits on a mini CD, or create a bootable USB installation as well.

Read More...

Tuesday, October 6, 2009

My weekend at Ohio LinuxFest 2009

I recently attended Ohio LinuxFest 2009, held September 25-27, 2009 at the Greater Columbus Convention Center in Columbus, Ohio.  The annual event draws Linux users from Ohio and the surrounding states. This was my third year as an attendee.

In past years, I went primarily to wander through the exhibitor area and bond with my fellow Linux geeks, but this year I expanded my LinuxFest schedule, attending both Friday and Saturday sessions.

On Friday I attended Zenoss' Community Day, a no-cost training day offered by Zenoss, an Open Source (with an enterprise edition available) network monitoring tool.  Zenoss goes well beyond what a simple home network administrator like me would ever need to manage my six computer network, but I still learned a lot at the session.

Before attending I never really gave much thought to the issues that come with managing a large scale network, and I realized that monitoring a large network is extremely complex.  The object driven network model that Zenoss builds really makes managing a complex network pretty easy.  

On Saturday I returned and brought my 14 year old daughter and 12 year old son along as well.  I suspect the main reason the two of them really want to go each year is so they can collect the freebies from the exhibitor area, but they do enjoy attending.

We spent about an hour people watching and wandering past exhibitor booths, stopping occasionally to talk to an exhibitor.  I sat down for 10 minutes or so and chatted with the fine folks at TheLinuxLinkTechShow (TLLTS).  They broadcast a live streaming Linux Tech show on the internet each Wednesday.

From there we attended a talk given by 19 year old Elizabeth Garbee entitled "How to Use Open Source to Pay For a College Education".  I thought Ms. Garbee gave an excellent presentation and I hope that my kids paid at least a little attention to what she said.

To round out our day at OLF 2009, we attended a talk by Mike Badger, "Programming for the Young and the Young At Heart.  Mr. Badger is the author of a book on Scratch, "Scratch 1.4 Beginner's Guide".

The talk was an hour long introduction to Scratch, the modular programming language.  I discovered Scratch on my own several months ago, (see my previous post on Scratch for more information) and the kids and I were really looking forward to the presentation.

To be honest I was somewhat disappointed by the presentation.  I felt that too much time was spent discussing the purpose and origins of Scratch, time that could have been better spent demonstrating Scratch's ease of use and abilities.  I thought Mr. Badgers presentation was really sort of dull, and his talk didn't hold my kids attention at all.

I really enjoyed the two days I spent at OLF 2009, and look forward to next year!  Here are some pictures taken by my daughter at the event:


Pictures from Ohio LinuxFest 2009

Ohio LinuxFest 2009

Read More...

>