Hide Notes 

Slide 1: Perl 5 Tutorial First Edition Chan Bernard Ki Hong
Slide 2: Perl is copyright by Larry Wall. Linux is a trademark of Linus Torvalds. Unix is a trademark of AT&T Bell Laboratories. Perl 5 Tutorial First Edition Author: Chan Bernard Ki Hong (webmaster@cbkihong.com) Web site: http://www.cbkihong.com Date of Printing: December 24, 2003 Total Number of Pages: 241 A Prepared from LTEX source files by the author. © 2001–2003 by Chan Bernard Ki Hong. While the author of this document has taken the best effort in enhancing the technical accuracy as well as readability of this publication, please note that this document is released “as is” without guarantee for accuracy or suitability of any kind. The full text of the terms and conditions of usage and distribution can be found at the end of this document. In order to further enhance the quality of this publication, the author would like to hear from you, the fellow readers. Comments or suggestions on this publication are very much appreciated. Please feel free to forward me your comments through the email feedback form or the feedback forum on the author’s Web site.
Slide 3: Contents 1 Introduction to Programming 1.1 What is Perl? . . . . . . . . . . . . . . . . . . . . 1.2 A Trivial Introduction to Computer Programming 1.3 Scripts vs. Programs . . . . . . . . . . . . . . . . 1.4 An Overview of the Software Development Process Getting Started 2.1 What can Perl do? . . . . . . . . . . . . . . . . . . 2.2 Comparison with Other Programming Languages 2.2.1 C/C++ . . . . . . . . . . . . . . . . . . . 2.2.2 PHP . . . . . . . . . . . . . . . . . . . . . 2.2.3 Java/JSP . . . . . . . . . . . . . . . . . . . 2.2.4 ASP . . . . . . . . . . . . . . . . . . . . . 2.3 What do I need to learn Perl? . . . . . . . . . . . . 2.4 Make Good Use of Online Resources . . . . . . . . 2.5 The Traditional “Hello World” Program . . . . . . 2.6 How A Perl Program Is Executed . . . . . . . . . . 2.7 Literals . . . . . . . . . . . . . . . . . . . . . . . 2.7.1 Numbers . . . . . . . . . . . . . . . . . . 2.7.2 Strings . . . . . . . . . . . . . . . . . . . 2.8 Introduction to Data Structures . . . . . . . . . . 1 1 1 3 4 7 7 8 8 8 9 9 9 11 12 15 16 16 16 17 23 23 23 24 25 26 26 26 27 28 29 30 31 32 33 35 38 38 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Manipulation of Data Structures 3.1 Scalar Variables . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Assignment . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Nomenclature . . . . . . . . . . . . . . . . . . . . . 3.1.3 Variable Substitution . . . . . . . . . . . . . . . . . . 3.1.4 substr() — Extraction of Substrings . . . . . . . . 3.1.5 length() — Length of String . . . . . . . . . . . . . 3.2 Lists and Arrays . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Creating an Array . . . . . . . . . . . . . . . . . . . 3.2.2 Adding Elements . . . . . . . . . . . . . . . . . . . . 3.2.3 Getting the number of Elements in an Array . . . . . 3.2.4 Accessing Elements in an Array . . . . . . . . . . . . 3.2.5 Removing Elements . . . . . . . . . . . . . . . . . . 3.2.6 splice(): the Versatile Function . . . . . . . . . . . 3.2.7 Miscellaneous List-Related Functions . . . . . . . . . 3.2.8 Check for Existence of Elements in an Array (Avoid!) 3.3 Hashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Assignment . . . . . . . . . . . . . . . . . . . . . . . i . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Slide 4: ii CONTENTS 3.3.2 Accessing elements in the Hash . . 3.3.3 Removing Elements from a Hash . 3.3.4 Searching for an Element in a Hash Contexts . . . . . . . . . . . . . . . . . . . Miscellaneous Issues with Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 40 41 42 44 47 47 48 48 50 51 53 54 56 57 58 59 64 67 67 67 69 69 69 70 71 73 75 78 80 80 81 81 82 82 86 88 88 89 89 91 91 3.4 3.5 4 Operators 4.1 Introduction . . . . . . . . . . . . . . . 4.2 Description of some Operators . . . . . . 4.2.1 Arithmetic Operators . . . . . . 4.2.2 String Manipulation Operators . 4.2.3 Comparison Operators . . . . . 4.2.4 Equality Operators . . . . . . . . 4.2.5 Logical Operators . . . . . . . . 4.2.6 Bitwise Operators . . . . . . . . 4.2.7 Assignment Operators . . . . . . 4.2.8 Other Operators . . . . . . . . . 4.3 Operator Precedence and Associativity . . 4.4 Constructing Your Own sort() Routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conditionals, Loops & Subroutines 5.1 Breaking Up Your Code . . . . . . . . . . . . . . 5.1.1 Sourcing External Files with require() 5.2 Scope and Code Blocks . . . . . . . . . . . . . . 5.2.1 Introduction to Associations . . . . . . . 5.2.2 Code Blocks . . . . . . . . . . . . . . . 5.3 Subroutines . . . . . . . . . . . . . . . . . . . . 5.3.1 Creating and Using A Subroutine . . . . 5.3.2 Prototypes . . . . . . . . . . . . . . . . 5.3.3 Recursion . . . . . . . . . . . . . . . . . 5.3.4 Creating Context-sensitive Subroutines . 5.4 Packages . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Declaring a Package . . . . . . . . . . . 5.4.2 Package Variable Referencing . . . . . . 5.4.3 Package Variables and Symbol Tables . . 5.4.4 Package Constructors with BEGIN {} . . 5.5 Lexical Binding and Dynamic Binding . . . . . . 5.6 Conditionals . . . . . . . . . . . . . . . . . . . 5.7 Loops . . . . . . . . . . . . . . . . . . . . . . . 5.7.1 for loop . . . . . . . . . . . . . . . . . 5.7.2 while loop . . . . . . . . . . . . . . . . 5.7.3 foreach loop . . . . . . . . . . . . . . 5.7.4 Loop Control Statements . . . . . . . . 5.8 Leftovers . . . . . . . . . . . . . . . . . . . . . References 6.1 Introduction . . . . . . . . . 6.2 Creating a Reference . . . . . 6.3 Using References . . . . . . . 6.4 Pass By Reference . . . . . . . 6.5 How Everything Fits Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 . 95 . 95 . 97 . 100 . 101
Slide 5: CONTENTS 6.6 7 iii Typeglobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 105 105 105 105 106 107 107 110 111 113 119 119 119 120 121 122 122 122 123 124 126 127 127 128 128 128 128 128 131 132 137 137 138 138 139 139 141 141 142 143 143 144 144 145 Object-Oriented Programming 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Object-Oriented Concepts . . . . . . . . . . . . . . . . . . . . . 7.2.1 Programming Paradigms . . . . . . . . . . . . . . . . . 7.2.2 Basic Ideas . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.3 Fundamental Elements of Object-Oriented Programming 7.3 OOP Primer: Statistics . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Creating and Using A Perl Class . . . . . . . . . . . . . . 7.3.2 How A Class Is Instantiated . . . . . . . . . . . . . . . . 7.4 Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Files and Filehandles 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Filehandles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 open a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Output Redirection . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 File Input and Output Functions . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 readline() — Read A Line from Filehandle . . . . . . . . . . . . . 8.3.2 binmode() — Binary Mode Declaration . . . . . . . . . . . . . . . 8.3.3 read() — Read A Specified Number of Characters from Filehandle . 8.3.4 print()/printf() — Output To A FileHandle . . . . . . . . . . . 8.3.5 seek() — Set File Pointer Position . . . . . . . . . . . . . . . . . . 8.3.6 tell() — Return File Pointer Position . . . . . . . . . . . . . . . . 8.3.7 close() — Close An opened File . . . . . . . . . . . . . . . . . . . 8.4 Directory Traversal Functions . . . . . . . . . . . . . . . . . . . . . . . . . 8.4.1 opendir() — Open A Directory . . . . . . . . . . . . . . . . . . . 8.4.2 readdir() — Read Directory Content . . . . . . . . . . . . . . . . 8.4.3 closedir() — Close A Directory . . . . . . . . . . . . . . . . . . . 8.4.4 Example: File Search . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 File Test Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 File Locking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Regular Expressions 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . 9.2 Building a Pattern . . . . . . . . . . . . . . . . . . . . 9.2.1 Getting your Foot Wet . . . . . . . . . . . . . 9.2.2 Introduction to m// and the Binding Operator 9.2.3 Metacharacters . . . . . . . . . . . . . . . . . 9.2.4 Quantifiers . . . . . . . . . . . . . . . . . . . 9.2.5 Character Classes . . . . . . . . . . . . . . . . 9.2.6 Backtracking . . . . . . . . . . . . . . . . . . 9.3 Regular Expression Operators . . . . . . . . . . . . . 9.3.1 m// — Pattern Matching . . . . . . . . . . . . 9.3.2 s/// — Search and Replace . . . . . . . . . . 9.3.3 tr/// — Global Character Transliteration . . 9.4 Constructing Complex Regular Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Slide 6: iv CONTENTS 149 149 149 150 151 151 152 152 155 159 159 159 160 163 169 169 169 169 170 172 174 178 180 182 182 183 184 186 187 187 190 193 193 198 199 199 201 201 201 201 202 203 10 Runtime Evaluation & Error Trapping 10.1 Warnings and Exceptions . . . . . . . . . . . . . 10.2 Error-Related Functions . . . . . . . . . . . . . 10.3 eval . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Backticks and system() . . . . . . . . . . . . . 10.5 Why Runtime Evaluation Should Be Restricted . 10.6 Next Generation Exception Handling . . . . . . 10.6.1 Basic Ideas . . . . . . . . . . . . . . . . 10.6.2 Throwing Different Kinds of Errors . . . 10.6.3 Other Handlers . . . . . . . . . . . . . . 10.7 Other Methods To Catch Programming Errors . 10.7.1 The -w Switch — Enable Warnings . . . 10.7.2 Banning Unsafe Constructs With strict 10.7.3 The -T Switch — Enable Taint Checking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 CGI Programming 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Static Content and Dynamic Content . . . . . . . . . . . . 11.2.1 The Hypertext Markup Language . . . . . . . . . . 11.2.2 The World Wide Web . . . . . . . . . . . . . . . . 11.3 What is CGI? . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Your First CGI Program . . . . . . . . . . . . . . . . . . . 11.5 GET vs. POST . . . . . . . . . . . . . . . . . . . . . . . . . 11.6 File Upload . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7 Important HTTP Header Fields and Environment Variables 11.7.1 CGI-Related Environment Variables . . . . . . . . . 11.7.2 HTTP Header Fields . . . . . . . . . . . . . . . . . 11.8 Server Side Includes . . . . . . . . . . . . . . . . . . . . . . 11.9 Security Issues . . . . . . . . . . . . . . . . . . . . . . . . 11.9.1 Why Should I Care? . . . . . . . . . . . . . . . . . 11.9.2 Some Forms of Attack Explained . . . . . . . . . . 11.9.3 Safe CGI Scripting Guidelines . . . . . . . . . . . . A How A Hash Works A.1 Program Listing of Example Implementation A.2 Overview . . . . . . . . . . . . . . . . . . . A.3 Principles . . . . . . . . . . . . . . . . . . . A.4 Notes on Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B Administration B.1 CPAN . . . . . . . . . . . . . . . . . . . . . . . . B.1.1 Accessing the Module Database on the Web B.1.2 Package Managers . . . . . . . . . . . . . B.1.3 Installing Modules using CPAN.pm . . . . B.1.4 Installing Modules — The Traditional Way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C Setting Up A Web Server 205 C.1 Apache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 C.1.1 Microsoft Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 C.1.2 Unix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
Slide 7: CONTENTS D A Unix Primer D.1 Introduction . . . . . . . . . . . . . . D.1.1 Why Should I Care About Unix? D.1.2 What Is Unix? . . . . . . . . . D.1.3 The Overall Structure . . . . . D.2 Filesystems and Processes . . . . . . . . D.2.1 Overview . . . . . . . . . . . . D.2.2 Symbolic Links and Hard Links D.2.3 Permission and Ownership . . D.2.4 Processes . . . . . . . . . . . . D.2.5 The Special Permission Bits . . E In The Next Edition Index v 213 213 213 213 214 215 215 216 220 222 223 225 226 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Slide 8: vi CONTENTS
Slide 9: Preface If you are looking for a free Perl tutorial that is packed with everything you need to know to get started on Perl programming, look no further. Presenting before you is probably the most comprehensive Perl tutorial on the Web, the product of two years of diligence seeking reference from related books and Web sites. Perl is a programming language that is offered at no cost. So wouldn’t it be nice if you can also learn it at no cost? Packed with some background knowledge of programming in C++ and Visual Basic, when I started learning Perl several years ago, I couldn’t even find one good online tutorial that covered at least the basics of the Perl language and was free. Most Perl tutorials I could find merely covered the very basic topics such as scalar/list assignment, operators and some flow control structures etc. On the other hand, although I have accumulated certain levels of experience in a number of programming languages, the official Perl manual pages are quite technical with whole pages of jargons that I was not very familiar with. As a result, the book “Learning Perl” written by Larry Wall, the inventor of the Perl language, naturally became the only Perl textbook available. The O’Reilly Perl Series present the most authoritative and well-written resources on the subject written by the core developers of Perl. While you are strongly recommended to grab one copy of each if you have the money, they are not so cheap, though, and that’s the motive behind my writing of this tutorial — so that more people with no programming background can start learning this stupendous and powerful language in a more cost-effective way. Although this tutorial covers a rather wide range of topics, similar to what you can find in some other Perl guidebooks, you are strongly encouraged to read those books too, since their paedagogies of teaching may suit you more. Here are several features of this tutorial: As this is not a printed book, I will constantly add new materials to this tutorial as needed, thus enriching the content of this tutorial. Moreover, in order to help me improve the quality of this tutorial, it is crucial for you to forward me your comments and suggestions so that I can make further improvements to it. Earlier drafts of this tutorial were published in HTML format on my Web site. In response to requests made from several visitors, this tutorial has been made available in PDF format for download. I hope this will help those who are charged on time basis for connecting to the Internet. This A tutorial is typeset in LTEX, a renowned document typesetting system that has been widely used in the academic community on Unix-compatible systems (although it is available on nearly any operating systems you can think of). The HTML version has been discontinued, until a solution can be found which allows both versions to be generated from the same source base. You will find a list of Web links and references to book chapters after each chapter where applicable which contains additional materials that ambitious learners will find helpful to further your understanding of the subject. vii
Slide 10: viii Preface Throughout the text there would be many examples. In this tutorial, you will find two types of examples — examples and illustrations. Illustrations are intended to demonstrate a particular concept just mentioned, and are shorter in general. You will find them embedded inline throughout the tutorial. On the other hand, examples are more functional and resemble practical scripts, and are usually simplified versions of such. They usually demonstrate how different parts of a script can work together to realize the desired functionalities or consolidate some important concepts learned in a particular chapter. If applicable, there will be some exercises in the form of concept consolidation questions as well as programming exercises at the end of each chapter to give readers chances to test how much they understand the materials learned from this tutorial and apply their knowledge through practice. This is the First Edition of the Perl 5 Tutorial. It primarily focuses on fundamental Perl programming knowledge that any Perl programmer should be familiar with. I start with some basic ideas behind computer programming in general, and then move on to basic Perl programming with elementary topics such as operators and simple data structures. The chapter on scoping and subroutines is the gateway to subsequent, but more advanced topics such as references and object-oriented programming. The remaining chapters are rather assorted in topic, covering the use of filehandles, file I/O and regular expressions in detail. There is also a dedicated chapter on error handling which discusses facilities that you can use to locate logical errors and enhance program security. The final chapter on CGI programming builds on knowledge covered in all earlier chapters. Readers will learn how to write a Perl program that can be used for dynamic scripting on the World Wide Web. However short, the main text already embraces the most important fundamental subjects in the Perl programming language. In the appendices, instructions are given on acquiring and installing Perl modules, setting up a basic but functional CGI-enabled Web server for script testing, and there is a voluminous coverage of Unix fundamentals. As much of Perl is based on Unix concepts, I believe a brief understanding of this operating system is beneficial to Perl programmers. An appendix is also prepared to give my readers an idea of the internal structure of general hashes. While authoring of this tutorial cannot proceed indefinitely, topics that were planned but cannot be included in this edition subject to time constraints are deferred to the Second Edition. A list of these topics appear at the end of this document for your reference. In the second release candidate of this tutorial I made an audacious attempt of adding into it two topics that are rarely discussed in most Perl literature. The first is the Error CPAN module for exception handling. The second attempt, which is an even more audacious one, is an introduction of the finite-state automaton (FSA) for construction of complex regular expressions for pattern matching. While FSA is a fundamental topic in Computer Science (CS) studies, this is seldom mentioned outside the CS circle. Although there is a high degree of correspondence between regular expressions and FSA, this may not be at all obvious to a reader without relevant background, despite I have avoided rigorous treatment of the topic and tried to explain it in a manner that would be more easily communicable to fellow readers. I would like to emphasize this topic is not essential in Perl programming, and I only intend to use it as a tool to formulate better patterns. Feel free to skip it if that is not comfortable to you and I require your feedback of whether these topics can help you as I originally intended. It is important for me to reiterate that this document is not intended to be a substitute for the official Perl manual pages (aka man pages) and other official Perl literature. In fact, it is the set of manual pages that covers the Perl language in sufficiently fine detail, and it will be the most important set of document after you have accumulated certain level of knowledge and programming experience. The Perl man pages are written in the most concise and correct technical parlance, and as a result they are not very suitable for new programmers to understand. The primary objective of this tutorial is to bridge the gap so as to supplement readers with sufficient knowledge to understand the man pages. Therefore, this tutorial presents a different perspective compared with some other Perl guidebooks available at your local bookstores from the mainstream computer book publishers. With a Computer Science background, I
Slide 11: ix intend to go more in-depth into the principles which are central to the study of programming languages in general. Apart from describing the syntax and language features of Perl, I also tried to draw together the classical programming language design theories and explained how they are applied in Perl. With this knowledge, it is hoped that readers can better understand the jargons presented in manual pages and the principles behind. Perl is attributed by some as a very cryptic language and is difficult to learn. However, those who are knowledgeable about programming language design principles would agree Perl implements a very rich set of language features, and therefore is an ideal language for students to experiment with different programming language design principles taught in class in action. I do hope that after you have finished reading this tutorial you will be able to explore the Perl horizons on your own with confidence and experience the exciting possibilities associated with the language more easily. “To help you learn how to learn” has always been the chief methodology followed in this tutorial. Time flies. Today when I am revising this preface, which was actually written before I made my initial preview release in late 2001 according to the timestamp, I am aghast to find that it has already been nearly two years since I started writing it. Indeed, a lot of things have changed in two years. Several Perl manpages written in tutorial-style have been included into the core distribution, which are written in a more gentle way targeted at beginners. There are also more Perl resources online today than there were two years ago. However, I believe through preparing this tutorial I have also learnt a lot in the process. Despite I started this project two years ago, a major part of the text was actually written in a window of 3 months. As a result, many parts of the tutorial were unfortunately completed in a hasty manner. However, through constant refinement and rewriting of certain parts of the tutorial, I believe the Second Edition will be more well-organized and coherent, while more advanced topics can be accommodated as more ample time is available. At last, thank you very much for choosing this tutorial. Welcome to the exciting world of Perl programming! Bernard Chan in Hong Kong, China 15th September, 2003
Slide 12: x Preface Typographical Conventions Although care has been taken towards establishing a consistent typographical convention throughout A this tutorial, considering this is the first time I try to publish in LTEX, slight deviations may be found in certain parts of this document. Here I put down the convention to which I tried to adhere: Elements in programming languages are typeset in monospace font. Important terms are typeset in bold. Profound sayings or quotes are typeset in italic. In source code listings, very long lines are broken into several lines. ¶ is placed wherever a line break occurs. Release History 30th August, 2003 15th September, 2003 01st October, 2003 31st December, 2003 First Edition, Release Candidate 1 First Edition, Release Candidate 2 First Edition, Release Candidate 3 First Edition About The Author Bernard Chan was born and raised in Hong Kong, China. He received his Bachelor’s Degree in Computer Engineering from the University of Hong Kong. His major interests in the field include information system security, networking and Web technologies. He carries a long-term career objective of becoming an avid technical writer in the area of programming. His most notable publication is the Perl 5 Tutorial.
Slide 13: Chapter 1 Introduction to Programming 1.1 What is Perl? Extracted from the perl manpage, “Perl is an interpreted high-level programming language developed by Larry Wall.” If you have not learnt any programming languages before, as this is not a prerequisite of this tutorial, this definition may appear exotic for you. The two keywords that you may not understand are “interpreted” and “high-level”. Because this tutorial is mainly for those who do not have any programming experience, it is better for me to give you a general picture as to how a computer program is developed. This helps you understand this definition. 1.2 A Trivial Introduction to Computer Programming You should know that, regardless of the programming language you are using, you have to write something that we usually refer to as source code, which include a set of instructions for the computer to perform some operations dictated by the programmer. There are two ways as to how the source code can be executed by the Central Processing Unit (CPU) inside your computer. The first way is to go through two processes, compilation and linking, to transform the source code into machine code, which is a file consisting of a series of numbers only. This file is in a format that can be recognized by the CPU readily, and does not require any external programs for execution. Syntax errors are detected when the program is being compiled. We describe this executable file as a compiled program. Most software programs (e.g. most EXEs for MS-DOS/Windows) installed in your computer fall within this type. NOTES There are some subtleties, though. For example, the compiler that comes with Visual Basic 6 Learning Edition translates source code into p-code (pseudo code) which has to be further converted to machine code at runtime. Such an EXE is described as interpreted instead. Therefore, not all EXEs are compiled. On the other hand, although Java is customarily considered an interpreted language, Java source files are first compiled into bytecode by the programmer, so syntactical errors can be checked at compile time. Another way is to leave the program uncompiled (or translate the source code to an intermediate level 1
Slide 14: 2 Chapter 1 Introduction to Programming between machine code and source code, e.g. Java). However, the program cannot be executed on its own. Instead, an external program has to be used to execute the source code. This external program is known as an interpreter, because it acts as an intermediary to interpret the source code in a way the CPU can understand. Compilation is carried out by the interpreter before execution to check for syntax errors and convert the program into certain internal form for execution. Therefore, the main difference between compiled programs and interpreted languages is largely only the time of compilation phase. Compilation of compiled programs is performed early, while for interpreted programs it is usually performed just before the execution phase. Every approach has its respective merits. Usually, a compiled program only has to be compiled once, and thus syntax checking is only performed once. What the operating system only needs to do is to read the compiled program and the instructions encoded can be arranged for execution by the CPU directly. However, interpreted programs usually have to perform syntax check every time the program is executed, and a further compilation step is needed. Therefore, startup time of compiled programs are usually shorter and execution of the program is usually faster. For two functionally equivalent programs, a compiled program generally gives higher performance than the interpreted program. Therefore, performance-critical applications are generally compiled. However, there are a number of factors, e.g. optimization, that influence the actual performance. Also, the end user of a compiled program does not need to have any interpreters installed in order to run the program. This convenience factor is important to some users. On the other hand, interpreters have to be installed in order to execute a program that is interpreted. One example is the Java Virtual Machine (JVM) that is an interpreter plugged into your browser to support Java applets. Java source files are translated into Java bytecode, which is then executed by the interpreter. There are some drawbacks for a compiled program. For example, every time you would like to test your software to see if it works properly, you have to compile and link the program. This makes it rather annoying for programmers to fix the errors in the program (debug), although the use of makefiles alleviates most of this hassle from you. Because compilation translates the source code to machine code which can be executed by the hardware circuitry in the CPU, this process creates a file in machine code that depends on the instruction set of the computer (machine-dependent). On the other hand, interpreted programs are usually platform-independent, that is, the program is not affected by the operating system on which the program is executed. Therefore, for example, if you have a Java applet on a Web site, it can most probably be executed correctly regardless of the operating system or browser a visitor is using. It is also easier to debug an interpreted program because repeated compilation is waived. TERMINOLOGY Instruction set refers to the set of instructions that the CPU executes. There are a number of types of microprocessors nowadays. For example, IBM-compatible PCs are now using the Intel-based instruction set. This is the instruction set that most computer users are using. Another prominent example is the Motorola 68000 series microprocessors in Macintosh computers. There are some other microprocessor types which exist in minority. The instruction sets of these microprocessors are different and, therefore, a Windows program cannot be executed unadapted on a Macintosh computer. In a more technical parlance, different microprocessors have different instruction set architectures. Recall that I mentioned that a compiled program consists entirely of numbers. Because a CPU is actually an electronic circuit, and a digital circuit mainly deals with Booleans (i.e. 0 and 1), so it is obvious that programs used by this circuit have to be sequences of 0s and 1s. This is what machine code actually is. However, programming entirely with numbers is an extreme deterrent to computer
Slide 15: 1.3 Scripts vs. Programs 3 programming, because numeric programming is highly prone to mistakes and debugging is very difficult. Therefore, assembly language was invented to allow programmers to use mnemonic names to write programs. An assembler is used to translate the assembly language source into machine code. Assembly language is described as a low-level programming language, because the actions of an assembly language program are mainly hardware operations, for example, moving bits of data from one memory location to another. Programming using assembly language is actually analogous to that of machine code in disguise, so it is still not programmer friendly enough. Some mathematicians and computer scientists began to develop languages which were more machineindependent and intuitive to programmers that today we refer to as high-level programming languages. The first several high-level languages, like FORTRAN, LISP, COBOL, were designed for specialized purposes. It was not until BASIC (Beginner’s All-purpose Symbolic Instruction Code) was invented in 1966 that made computer programming unprecedentedly easy and popular. It was the first widely-used high-level language for general purpose. Many programmers nowadays use C++, another high-level language, to write software programs. The reason why we call these “high-level languages” is that they were built on top of low-level languages and hid the complexity of low-level languages from the programmers. All such complexities are handled by the interpreters or compilers automatically. This is an important design concept in Computer Science called abstraction. That’s enough background information and we can now apply the concepts learned above to Perl. Perl (Practical Extraction and Reporting Language) was designed by Larry Wall, who continues to develop newer versions of the Perl language for the Perl community today. Perl does not create standalone programs and Perl programs have to be executed by a Perl interpreter. Perl interpreters are now available for virtually any operating system, including but not limited to Microsoft Windows (Win32) and many flavours of Unix. As I quoted above, “Perl is a language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information.” This precise description best summarizes the strength of Perl, mainly because Perl has a powerful set of regular expressions with which programmers can specify search criteria (patterns) precisely. You are going to see a whole chapter devoted to regular expression in Chapter 9. Perl is installed on many Web servers nowadays for dynamic Web CGI scripting. Perl programs written as CGI applications are executed on servers where the Perl source files are placed. Therefore, there is no need to transfer the Perl source to and from the server (as opposed to client-side scripts like JavaScript or Java applets). Guestbooks, discussion forums and many powerful applications for the Web can be developed using Perl. There is one point which makes Perl very flexible — there is always more than one approach to accomplish a certain task, and programmers can pick whatever approach that best suits the purpose. 1.3 Scripts vs. Programs There has always been some arguments over whether to use the term “script” or “program” for Perl source files. In general, a piece of code that is executed by hardware or a software interpreter, written in some kind of programming languages, is formally called a “program”. This is a general term that applies to programs written in machine instructions, or any programs that are compiled or interpreted. However, it is also common today to hear that people use the term “script” to refer to programs that are interpreted, especially those executed on the command line. In my opinion, it is not important to draw a distinction between the two terms as both are considered equally acceptable and understandable nowadays. In many situations people just use both interchangeably.
Slide 16: 4 Chapter 1 Introduction to Programming I am more inclined towards calling Perl programs and CGI programs running on a Perl backend as scripts, so I will adhere to this terminology in this tutorial. 1.4 An Overview of the Software Development Process An intuitive software development process is outlined below. Note that this process is not tailored for Perl programming in particular. It is a general development process that can be applied to any programming projects with any programming languages. For additional notes specific to Perl, please refer to the next chapter. Because this tutorial does not assume readers to have any programming experience, it is appropriate for me to give you an idea as to the procedure you will most probably follow when you write your programs. In general, the process of development of a software project could be broken down into a number of stages. Here is an outline of the stages involved: Requirements Analysis First you need to identify the requirements of the project. Simply speaking, you will need to decide what your program should do (known as functional requirements), and note down other requirements that are important but not related to the functions of your program (known as non-functional requirements), for example, a requirement that the user interface should be user friendly. You have to make a list of the requirements, and from it you will need to decide whether you have the capability to complete them. You may also want to prioritize them such that the most important functionalities are developed first, and other parts can be added subsequently. Systems Design From the requirements determined you can then define the scope of the project. Instead of putting the whole program in one piece, we will now organize the program into several components (or subsystems — a part of the entire system). As we will discuss later in this tutorial, modularization facilitates code reuse and make correction of bugs (debug) easier. Two major models exist today — decomposition based on functions and decomposition based on objects. After you have fixed the model, you decide on which functions or object methods are to be associated with which source file or object, and determine how these components interact with each other to perform the functionalities. Note that you don’t need to decide on how these source files or objects are implemented in real source code at this stage — it is just an overall view of the interaction between the components. We emphasize functional decomposition in the first part of the tutorial, while object-oriented programming will be covered in a later part of this tutorial. Program Design After we have determined how the components interact with each other, we can now decide how each function or object method is implemented. For each function, based on the actions to perform you have to develop an algorithm, which is a well-defined programming-languageindependent procedure to carry out the actions specified. You may want to use a flowchart or some pseudocode to illustrate the flow of the program. Pseudocode is expressed in a way resembling real programming source code, except language-dependent constructs are omitted. As pseudocode is language independent, you can transform an idea from pseudocode to source code in any programming languages very easily. There isn’t a single standardized pseudocode syntax. In many cases, pseudocode can even be written in English-like statements because pseudocode is written to demonstrate how a program is supposed to work, and provided it communicates the idea clearly it suffices. It is up to you as the author to express pseudocode in whatever way the algorithm is best illustrated.
Slide 17: 1.4 An Overview of the Software Development Process 5 Coding This is largely the continuation of the Program Design stage to transform your algorithm into programming language constructs. If you have worked out the algorithm properly this should be a piece of cake. Unit Testing Unit testing corresponds to Program Design. As each function or object method has a predefined behaviour, they can be tested individually to see if such behaviour agree to that defined. Most of the time when we are talking about debugging, we are describing this stage. Systems Testing Systems Testing corresponds to System Design. This is to test if the components interact with each other in exactly the same way as designed in the Systems Design stage. Requirements Validation This corresponds to requirements analysis. The software developed is compared against the requirements to ensure each functionality has been incorporated into the system and works as expected. Maintenance By now the software has been developed but you cannot simply abandon it. Most probably we still need to develop later versions, or apply patches to the current one as new bugs are found in the program. Software for commercial distribution especially needs investment of a lot of time and effort at this stage capturing user feedback, but software not distributed commercially should also pay attention to this stage as this affects how well your software can be further developed. Of course, for the examples in this tutorial that are so short and simple we don’t need such an elaborate development procedure. However, you will find when you develop a larger-scale project that having a well-defined procedure is essential to keep your development process in order. This is just one of the many process models in existence today. Discussion of such process models can be found in many fundamental text for Software Engineering, and are beyond the scope of this tutorial. Actually, what I have presented was a variant of the Waterfall process model, and is considered one that, if employed, is likely to delay project schedules and result in increased costs of software development. The reason I present it here is that the Waterfall model is the easiest model to understand. Because presentation of process models is out of the scope of the tutorial, some Web links will be presented at the end of this chapter from which you will find selected texts describing process models, including the Rational Unified Process which I recommend as an improved process model for larger-scale development projects. Adoption of an appropriate process model helps guide the development process with optimized usage of resources, increased productivity and software that are more fault-tolerant. Summary ” ” ” ” Source code include a set of instructions to be executed by the computer through levels of translation. Low-level programming languages involves architecture-dependent programming with machine code or assembly language. High-level programming languages are built on top of low-level languages and source programs can usually be platform-independent. Source programs are either explicitly precompiled into object code, or are implicitly compiled before execution by the interpreter.
Slide 18: 6 ” ” Chapter 1 Introduction to Programming Perl is an interpreted high-level programming language developed by Larry Wall. Following a well-defined software development process model helps keep the development process systematic and within budgets. Web Links 1.1 1.2 Evolution of Programming Languages http://lycoskids.infoplease.com/ce6/sci/A0860536.html Rational Unified Process Whitepapers http://www.rational.com/products/rup/whitepapers.jsp
Slide 19: Chapter 2 Getting Started 2.1 What can Perl do? I understand it is a full wastage of time for you to have read through half of a book to find that it is not the one you are looking for. Therefore, I am going to let you know what you will learn by following this tutorial as early as possible. If you are looking for a programming language to write an HTML editor that runs on the Windows platform, or if you would like to write a Web browser or office suite, then Perl does not seem to be an appropriate language for you. C/C++, Java or (if you are using Windows) Visual Basic are likely to be more appropriate choices for you. Although it appears that Perl is not the optimum language for developing applications with a graphical user interface (but you can, with Perl/Tk or native modules like WIN::GUI), it is especially strong in doing text manipulation and extraction of useful information. Therefore, with database interfacing it is possible to build robust applications that require a lot of text processing as well as database management. Perl is the most popular scripting language used to write scripts that utilize the Common Gateway Interface (CGI), and this is how most of us got to know this language in the first place. A cursory look at the CGI Resource Index Web site provided me with a listing of about 3000 Perl CGI scripts, compared with only 220 written in C/C++, as of this writing. There are quite many free Web hosts that allow you to deploy custom Perl CGI scripts, but in general C/C++ CGI scripts are virtually only allowed unless you pay. In particular, there are several famous programs written in Perl worth mentioning here: YaBB is an open source bulletin board system. While providing users with many advanced features that could only be found on commercial products, it remains as a free product. Many webmasters use YaBB to set up their BBS. Another popular BBS written in Perl is ikonboard, featuring a MySQL/PostgreSQL database back-end. Thanks to the powerful pattern matching functions in Perl, search engines can also be written in Perl with unparalleled ease. Perlfect Search is a very good Web site indexing and searching system written in Perl. You will learn more about Perl CGI programming in Chapter 11 of this tutorial. 7
Slide 20: 8 Chapter 2 Getting Started 2.2 Comparison with Other Programming Languages There are many programming languages in use today, each of which placing its emphasis on certain application domains and features. In the following section I will try to compare Perl with several popular programming languages for the readers to decide whether Perl is appropriate for you. 2.2.1 C/C++ Perl is written in the C programming language. C is extensively used to develop many system software. C++ is an extension of C, adding various new features such as namespaces, templates, object-oriented programming and exception handling etc. Because C and C++ programs are compiled to native code, startup times of C/C++ programs are usually very short and they can be executed very efficiently. Perl allows you to delegate part of your program in C through the Perl-C XS interface. This Perl-C binding is extensively used by cryptographic modules to implement the core cryptographic algorithms, because such modules are computation-intensive. While C/C++ is good for performance-critical applications, C/C++ suffers a number of drawbacks. First, C/C++ programs are platform dependent. A C/C++ program written on Unix is different from one on Windows because the libraries available on different platforms are different. Second, because C/C++ is a very structured language, its syntax is not as flexible as scripting languages such as Perl, Tcl/Tk or (on Unix platforms) bash. If you are to write two functionally equivalent programs in C/C++ and Perl, very likely the C/C++ version requires more lines of code compared with Perl. And also, improperly written C/C++ programs are vulnerable to memory leak problems where heap memory allocated are not returned when the program exits. On a Web server running 24×7 with a lot of visitors, a CGI script with memory leak is sufficient to paralyze the machine. 2.2.2 PHP Perl has been the traditional language of choice for writing server-side CGI scripts. However, in recent years there has been an extensive migration from Perl to PHP. Many programmers, especially those who are new to programming, have chosen PHP instead of Perl. What are the advantages of PHP over Perl? PHP is from its infancy Web-scripting oriented. Similar to ASP or JSP, it allows embedding of inline PHP code inside HTML documents that makes it very convenient to embed small snippets of PHP code, e.g. to update a counter when a visitor views a page. Perl needs Server Side Includes (SSI) or an additional package “eperl” to implement a similar functionality. Also, it inherits its language syntax from a number of languages so that it has the best features of many different languages. It mainly inherits from C/C++, and portions from Perl and Java. It uses I/O functions similar to that in C, that are also inherited into Perl, so it is relatively easy for Perl programmers to migrate to PHP. While PHP supports the object-oriented paradigm, most of its functionalities are provided through functions. When PHP is compiled the administrator decides the sets of functionalities to enable. This in turn determines the sets of functions enabled in the PHP installation. I’m personally sceptical of this approach, because in practice only a small subset of these functions is frequently used. On the other hand, Perl only has a small set of intrinsic functions covering the most frequently used functionalities. Other functionalities are delegated to modules which are only installed and invoked as needed. As I will introduce shortly and in Appendix B, the Comprehensive Perl Archive Network (CPAN) contains a comprehensive and well-organized listing of ready-made Perl modules that you can install and use very easily.
Slide 21: 2.3 What do I need to learn Perl? 9 2.2.3 Java/JSP Sun Microsystems developed the Java language and intended to target it as a general purpose programming language. It is from the ground up object-oriented and platform independent. Functionalities are accessed through the Java API, consisting of hierarchies of classes similar to that of Perl. Java Server Pages (JSP) is a Web scripting environment similar to ASP except with a Java syntax. Similar to C/C++, the Java syntax is very structured and thus are not as flexible as scripting languages like Perl. Also, Java itself is not just an interpreter, it is a virtual machine over which programmers are totally abstracted from the underlying operating system platforms, which allows the Java API to be implemented on top of this platform-independent layer. For those who have programmed in Java before, you will probably find that the Java Virtual Machine takes rather long time to load, especially on lower-end systems with limited computational power. This defers the possibility of widespread deployment of Java programs. While Perl is not strictly a general-purpose programming language like Java, I found it difficult to compare Perl and Java because of their different natures. However, if confined to the purpose of Web server scripting, I generally prefer Perl to JSP for its flexibility and lightweight performance. Despite this, I feel that Java is a language that is feature-rich and if time allows, you are strongly encouraged to find out more about this stupendous language, which is expecting increasing attention in mobile and embedded devices because of its platform independence. 2.2.4 ASP Active Server Pages (ASP) is only available on Windows NT-series operating systems where Internet Information Services (IIS) is installed (although alternative implementations of ASP on other system architectures exist, e.g. Sun Chili!Soft ASP, which is a commercial product that runs on Unix, but generally considered not very stable). Running on a Windows Web server, ASP can impose a tighter integration with Microsoft technologies, so that the use of, say, ActiveX data objects (ADO) for database access can be made a lot easier. However, IIS is especially vulnerable to remote attacks when operated as a Web server. Numerous service packs have been released to patch the security holes in IIS and Windows NT. However, new holes are still being discovered from time to time that makes the deployment of Windows NT/IIS as the Web server of choice not very favourable. On the other hand, Apache, the renowned Web server for Unix and now for other operating systems as well, has far less security concerns and are less susceptible to remote attacks. Apache also has the largest installation base among all Web server software, taking up more than 60% of the market share. 2.3 What do I need to learn Perl? You don’t need to pay a penny to learn and use Perl. Basically, a text editor that handles text-only files and a working installation of the Perl interpreter are all that you will need. Under Microsoft Windows, Notepad meets the minimum requirement. However, a whole page of code in black is not visually attractive in terms of readability. Some text editors have the feature of syntax highlighting, with different parts of a statement displayed in different colours. Good colouring makes the source files more pleasurable to look at (such colouring is used for display only and will not be saved to file). However, avoid using word processors like Microsoft Word or Wordpad which add proprietary control codes on file save by default. The Perl interpreter does not recognize these special formats. If you have to use these word processors, ensure that your files are saved as plain text ASCII format so
Slide 22: 10 Chapter 2 Getting Started Figure 2.1: Editing a Perl source file with GVIM, running on GNU/Linux that the Perl interpreter can access them. AnyEdit and UltraEdit are nice text editors on the Windows platform. On Unix variants, emacs and vim are stupendous text editors featuring syntax highlighting profiles for most programming languages with a lot of powerful features. Fig. 2.1 shows a screenshot of a Perl source file edited with GVIM, a port of vim that runs on Windows, X-Windows with the GTK library on Unix/Linux and many other platforms. This is my favourite text editor and is used to construct my entire Web site. If you are using one of the mainstream operating systems, the perl interpreter can be downloaded from the download section of perl.com. perl.com is the official Web site for the Perl language and you can find the download links to all available interpreter versions there. Choose the version which matches your operating system. When you go to the download page you will see two versions, namely the stable production release and the experimental developer’s release. The stable release is the version I recommend to new users, because the developer’s version is for more advanced users to beta test the new version. It may still contain bugs and may give incorrect results. The files you have to download are under the heading “binary distribution”. Do not download the source code distribution unless you know exactly how to compile and install them. In case you are using an operating system that is not listed, a good place to find a binary distribution for your operating system is the CPAN, located at here, which contains a fairly comprehensive list of platforms on which Perl can run. For Windows users, most probably you should download the Activestate distribution of Perl. It is very easy to install, with some extra tools bundled in the package for easy installation of new modules. For GNU/Linux users, most probably Perl is already installed or available as RPM (Redhat Package Manager) or DEB (Debian packages) formats. As many Linux distributions already have builtin support for RPM packages, you may look at your installation discs and you are likely to find some RPM binaries for Perl if it is not yet installed. For other Unix systems, you may find tarballs containing the Perl binaries. If no binaries are available for your system, you can still build from sources by downloading the source distribution from the CPAN. To check if perl is installed on your system, simply open a
Slide 23: 2.4 Make Good Use of Online Resources 11 terminal and type perl -v. If Perl is installed you will have the version information of Perl installed displayed on screen. If error messages appear, you will need to install it. Installation of Perl will not be covered in this tutorial, and you should look for the installation help documents for details. NOTES Because Perl is an open source software, which releases the source code to the public for free, you will see the source code distribution listed. Yet for usual programming purposes there is no need to download the source files unless binary distributions are not available for your system. An exception is if you are using one of the operating systems in the Unix family (including Linux). There are already compilation tools in these operating systems and you can manually compile Perl from sources and install it afterwards. However, note that compilation can be a very time-consuming process, depending on the performance of your system. If you are using Linux, binary distributions in the form of RPM or DEB packages can be installed very easily. Only if you cannot find a binary distribution for your platform that you are encouraged to install from source package. 2.4 Make Good Use of Online Resources You may need to seek reference while you are learning the language. As a new user you are not recommended to start learning Perl by reading the man-pages or the reference manuals. They are written in strict technical parlance that beginners, especially those who do not have prior programming experience or basic knowledge in Computer Science, would find reading them real headaches. You are recommended to follow this tutorial (or other tutorials or books) to acquire some basic knowledge first, and these reference documents will become very useful for ambitious learners to know more about the language, or when you have doubt on a particular subject you may be able to find the answers inside. In this course I will try to cover some important terms used in the reference materials to facilitate your understanding of the text. For the time being, you may want to have several books on Perl for cross-referencing purposes. I have tried to write this tutorial in a way that beginners should find it easy to follow, yet you may need to consult these books if you have any points that you don’t understand fully. Although you are not advised to read the official reference documents too early, in some later parts I may refer you to read a certain manpage. A manpage, on Unix/Linux systems, is a help document on a particular aspect. To read a particular manpage, (bring up a terminal if you are in X-Windows) type man followed by the name of the manpage, for example, man perlvar, the perlvar manpage will be displayed. For other platforms, manpages may usually come in the format of HTML files. Consult the documentation of your distribution for details. There is an online version of the Perl official documentation at perldoc.com. It contains the Perl man pages as well as documentation of the modules shipped with Perl. In fact, there are now several manpages that are targeted at novice programmers. For instance, the perlintro manpage is a brief introduction to the fundamental aspects of the Perl language that you should master fully in order to claim yourself a Perl programmer. You are also reminded of the vast varieties of Perl resources online. There are many Perl newsgroups on the USENET and mailing lists devoted to Perl. Your questions may be readily answered by expert Perl
Slide 24: 12 Chapter 2 Getting Started programmers there. Of course, try to look for a solution from all the resources you can find including the FAQs before you post! Otherwise, your question may simply be ignored. Perl Monks is also a very useful resource to Perl users. dmoz.org contains a nice selection of Perl-related sites. You can find a similar list of entries on Yahoo!. Google is the best search engine for programmers. You can usually get very accurate search results that deliver what you need. For example, by specifying the terms “Perl CGI.pm example”, you will get screenfuls of links to examples demonstrating the various uses of the CGI.pm module. As you will see later, this module is the central powerhouse allowing you to handle most operations involving CGI programming with unparalleled ease. For materials not covered in this tutorial or other books, a search phrase can be constructed in a similar manner that allows you to find documentation and solutions to your questions at your fingertips. Of course, don’t forget to experiment yourself! CPAN, the Comprehensive Perl Archive Network is a nice place where you can download a lot of useful modules contributed by other Perl programmers. By using these modules you can enforce code reuse, rather than always inventing code from scratch again. There are so many modules on the CPAN available that you would be surprised at how active the Perl community has been. Some CPAN modules are well-documented, some are not. You may need to try to fit the bits and pieces together and see if it works. This requires much time and effort, but you can learn quite a lot from this process. 2.5 The Traditional “Hello World” Program Traditionally, the first example most book authors use to introduce a programming language is what is customarily called a “Hello World” program. The action of this program is extremely simple — simply displays the text “Hello World” to the screen and does nothing else. For all examples in this tutorial of which source code are given in the text, you are encouraged to type them in yourself instead of executing the examples downloaded from my Web site, since it is more likely that by doing so you would understand the materials more quickly. Let’s write a “Hello World” program to see the procedures we take to develop a Perl program. If you are on Windows, it is a good practice to check if the path to the Perl interpreter has been added to the path list in C:\Autoexec.bat. In this way, you can change to the path containing your Perl source files and can run the interpreter without specifying its path. The setup program of your distribution would probably have done it for you. If it hasn’t, append the path to the end of the list and end it with a semicolon. A typical path list looks like this, the last one in this example is the path to the perl interpreter (note that your path may be different): SET PATH = C:\WINDOWS; C:\WINDOWS\COMMAND; C:\WINDOWS\SYSTEM; C:\PERL\BIN; For Unix/Linux, check your PATH variable and see if the directory containing the perl executable is present (usually /usr/bin). You can look at the list of paths by typing echo $PATH on the command line (be careful of exact capitalization!). Look for “/usr/bin” in the colon-separated values. On some systems, the path to perl would be “/usr/local/bin” or something else, so please check carefully. You may need to modify the startup login scripts like .login, .bashrc, .profile etc. so that you don’t need to set PATH or specify the full path to perl every time if perl is installed at some weird locations. A convenient workaround is to create a symbolic link in a directory included in PATH, e.g. /usr/bin that points to the perl executable.
Slide 25: 2.5 The Traditional “Hello World” Program NOTES The use of Autoexec.bat is now obsolete starting from Windows 2000. Setting of environment variables should be carried out by right-clicking on the “My Computer” icon, and then choose the “Properties” option. Now select the “Advanced” tab and then click on the “Environment Variables” button at the bottom. To make the perl interpreter available to all users on the system, the path should be appended to the PATH variable in the “System variables” section. If you modify the PATH variable in the “User variables” section, only the user concerned (presumably you) will be affected. 13 EXAMPLE 2.1 1 2 3 4 5 #!/usr/bin/perl -w # Example 2.1 - Hello World # print the text to the screen print "Hello, World!\n"; Here we outline the steps to create this simple program on Windows and Linux. Microsoft Windows 1. Open Notepad (or any other text editor you choose) and type in the source code shown above. Note that the line numbers on the left are for identification of lines only and do NOT type them into the text editor. Please make sure word wrap is disabled. Save the file as hello.pl. A few text editors, like Notepad, usually append the “.txt” extension to the filename when saving. You may put a pair of double quotes around the filename to circumvent this behaviour. Also, if you are using Windows 2000 or above and would like to use Notepad, please ensure that the file encoding is set to ANSI instead of Unicode. Bring up an MS-DOS prompt window and change to the directory containing your newly created file. Say if you have saved to ”C:\perl examples”, then type cd C:\perl examples and press Enter. Put a pair of double quotes around the path if any directories in the path contains spaces (In fact I don’t recommend placing Perl source files in directories with names containing spaces. It only complicates matters). Execute the program by typing perl -w hello.pl and press enter. 2. 3. 4. Unix or GNU/Linux 1. Open any text editor (vim, emacs, pico, kedit ....) and type in the source code shown above. Note that the line numbers on the left are for identification only and do NOT type them into the text editor. Please make sure word wrap is disabled. Save the file as hello.pl. Note that the path on line 1 has to match the path to perl on your system. Also, no spaces should precede the ‘#’ character and no empty lines are allowed before this special line (traditionally known as the ‘shebang’ line). 2.
Slide 26: 14 3. 4. Chapter 2 Getting Started If you are in X-Windows environment, bring up a terminal window. Change to the directory containing the newly created file using the cd command. In order to run it without specifying the perl interpreter, set the file access privilege to user executable by using the chmod command. The command should be chmod u+x hello.pl and press Enter. Execute the program by typing ./hello.pl and then press Enter. NOTES Even if you are using Unix/Linux, it is not absolutely needed to chmod your perl source files. In fact, you only need to make those source files executable if you want them to be directly invoked without specifying the name of the interpreter (i.e. perl). In this case, the shell will look at the first line of the file to determine which interpreter is used, so the #!/usr/bin/perl line must exist. Otherwise, the file cannot be executed. If you only intend to execute the file in Unix or Linux using perl -w filename.pl, then filename.pl need not be given an executable permission. As you will learn later, you may have some Perl source files that are not invoked directly. Instead, they are being sourced from another source file and don’t need to be executable themselves. For these files, you don’t need to chmod them and the default permission is adequate. 5. If there is not any errors, you should see the words “Hello, World!” under the command prompt. For such a simple program it is not easy to make mistakes. If error messages appear, check carefully if you have left out anything, because a trivial mistake is sufficient to end up with some error messages if you are not careful. Also check if you are using the latest stable version of Perl 5. All examples in this tutorial have been tested with Perl 5.8.0 Win32 (ActiveState distribution) and Perl 5.8.0 on GNU/Linux, but it should work for other distributions or versions as well unless otherwise noted. The -w is an example of a switch. You specify a switch to enable a particular interpreter feature. -w is specified so that warning messages, if any, are displayed on screen. Under no circumstances should this switch be omitted because it is important, especially as a beginner, to ensure that the code written is correct. It also helps catch some mistakes that are otherwise difficult to capture. This is explained in more detail in Chapter 10. A Perl script consists of statements, and each statement is terminated with a semicolon (;). A statement is rather like a sentence in human languages which carries a certain meaning. For computer languages a statement is an instruction to be performed. The core of the program is on line 5. It is this statement that prints the text delimited by quotation marks to the screen (in a more accurate parlance, the text is sent to the standard output, which is the screen by default). print() is an example of a builtin function. A function usually accepts a number of parameters, or arguments. Parameters serve to provide a function with additional pieces of data that are necessary for its operation. In the example, print() takes the text as parameter in order to display it on screen. A set of basic functions is provided to you by Perl for performing different actions. Some of these functions will be introduced as you progress through this tutorial. Notice the strange \n at the end? It is one of the escape characters which will be described later in more detail. \n is used to insert a line break. Therefore, you see a blank line before returning to the command prompt.
Slide 27: 2.6 How A Perl Program Is Executed 15 Lines preceded by a # (sharp) sign are comments and are ignored by the perl interpreter. A comment does not need to be on its own line, it can be put at the end of a line as well. In that case, the remaining of the line (starting from # and until the end of the line) is regarded as a comment. Comments are helpful for you or other programmers who read your code to understand what it does. You can put anything you like as comments. Line 1 is also a comment as it is of interest to the shell only instead of the perl interpreter itself. The switch -w here is the same as that specified under the command line. This is read together with the path to perl to enable the display of warnings. GOOD PROGRAMMING PRACTICES Comments Comments are meant for human readers to understand the source code without the need of running the program once in your brain. This increases both the readability and maintainability of your source code. Many programmers are lazy to insert comments throughout the code. But it is very likely when you look at a piece of code wrote earlier you may not understand it anymore as it is very complicated without any comments in it. Therefore, you should include comments in appropriate places. Usually for a single block of code performing one particular function we will place a comment briefly describing what this code block does. You may also want to place a comment on a particular line if the meaning of the line is not immediately obvious. Of course, don’t deluge your source code with comments. For example, in the source code of the ‘Hello World’ program I placed a comment for the print statement. It is superfluous, in fact, as the meaning of this statement is pretty obvious. But I included it here because this is your first Perl function learnt. As you proceed, more constructive comments and less superfluous comments will be found in the examples. 2.6 How A Perl Program Is Executed Perl programs are distributed in source files. From the instant you invoke the perl interpreter to execute a script, a number of steps were involved before the program is executed. Preprocessing An optional preprocessing stage transforms the source file to the final form. This tutorial does not cover source preprocessing. For details please consult the perlfilter manpage. Tokenizing The source file is broken down into tokens. This process is called tokenization (or lexical analysis). Whitespaces are removed. Token is the basic unit that makes up a statement. By tokenizing the input parsing is becoming easier because all further processing are carried out on tokens, independent of whitespace. Parsing & Optimization (Compilation) Parsing involves checks to ensure the program being executed conforms to the language specification and builds a parse tree internally which describes the program in terms of microoperations internal to Perl (opcode). Some optimizations to the parse tree
Slide 28: 16 Chapter 2 Getting Started are performed afterwards. Finally, the parse tree built is used to execute the program. While in-depth understanding of any of these processes is not essential to practical Perl programming, the compilation phase will be mentioned in some later chapters that I believe it is a good idea to briefly introduce the phases involved beforehand. 2.7 Literals All computer programs have to handle data. In every program there are certain kinds of data that do not change with time. For example, consider a very much simplified CGI script that checks if the password input by the user matches the system. How would you implement it? It seems the simplest method would be to have the correct password specified in the script, and after the user has entered the password and hit the “Submit” button, compare it against the password input by the user. The standard password specified in this script does not change during the course of execution of the script. This is an example of a literal. Other terms that are also used are invariants and constants. In the previous Hello World example, the text “Hello, World!\n” on line 5 is also a literal (This piece of data cannot be changed during the time you are running the program). Literals can have a number of forms, just because we can have data of different forms. In Perl we can roughly differentiate numbers and strings. 2.7.1 Numbers There are several classes of numbers: integers and decimals (known as floating-point numbers in computer literature). In Perl, integers can be expressed in decimal (base 10), hexadecimal (base 16) or octal (base 8) notation. Octal numbers are preceded by a 0 (zero), e.g. 023 is equivalent to 19 10 (the subscript 10 denotes representation in base-10, i.e. decimal form); hexadecimal numbers are preceded by 0x (zero x), e.g. 0xfe is equivalent to 25410 . For hexadecimal digits A - F, it does not matter whether you specify them in lowercase or uppercase. That is, 0xfe is the same as 0xFE. Integers cannot be delimited by commas or spaces like 10,203,469 or 20 300. However, Perl provides a nice workaround as a substitute. An example is 4 976 297 305. This is just a facility to make large numbers easier to read by programmers, and writing 4976297305 is entirely correct in Perl. Decimals are those carrying decimal points. If the integral portion is 0, the integral portion is optional, i.e. -0.6 or -.6 work equally fine. Exponents (base 10) can also be specified by appending the letter “e” and the exponent to the real number portion. e.g. 2e3 is equivalent to 2 x 10 3 = 2000. 2.7.2 Strings A string is a sequence of characters enclosed (delimited) by either double quotes (”) or single quotes (’). They differ in variable substitution and in the way escape characters are handled. The text “Hello, World!\n” in the hello world example is a string literal, delimited by double quotes. We will defer variable substitution until we come to variables. Escape characters exist in many programming languages. An escape character consists of a backslash \ symbol followed by a letter.
Slide 29: 2.8 Introduction to Data Structures 17 Every escape character has a function associated with it. Escape characters are usually put inside double-quoted strings. Table 2.1 summarizes the most important escape characters used in Perl. These escape characters are predefined and can be used in double-quoted strings. There is another case where backslashes have to be used in a string. This is known as character escaping. What does that mean? Consider you would like to use double quotes in a double-quoted string. For example you would like to print this English sentence instead of the “Hello World” phrase in Example 2.1: Howdy says, ”Give me $500”. That is, you try to print this sentence by: print "Howdy says, "Give me $500"."; If you execute this statement, you will get into trouble, because " is used to mark the beginning and the end of the string literal itself. Perl locates the end of the string by searching forward until the second double quote is found. If the literal contains double quotes itself, Perl will not know where the string literal terminates. In the example above, Perl will think the string ends after “Howdy says, ”. Also, after you have learned variable substitution in the next chapter you will realize that the symbol $ is used for variable substitution. You have to tell Perl explicitly you would like to use the symbol as is instead of performing variable substitution. To get around this problem, just place the \ character before the two symbols concerned, and this is what we mean to “escape” a character. So, the correct way to print this sentence using double quotes is: print "Howdy says, \"Give me \$500\"."; However, wise Perl programmers will not do this, as the backslashes make the whole expression ugly. If we choose to use single quotes instead, we don’t even have to escape anything: print ’Howdy says, "Give me $500".’; Single-quoted strings do not support variable substitution, so the $ needs not be escaped. Also, because the symbol " does not carry any significance in the string, it does not need to be escaped as well. There are only two characters that need to be escaped in single-quoted strings, namely ’ and \. For double-quoted strings, a number of characters have to be escaped, and it would become clear as you work through the chapters in this tutorial. Empty strings are denoted by “” or ‘’, that is, two quotes with nothing in between. 2.8 Introduction to Data Structures Every programming language has certain kinds of data structures builtin. A data structure can be thought of as a virtual container residing in the memory in which data is stored. Each data structure is associated with a data type specifying the type of data permitted in the data structure. Data type is important in programming languages because data of different types are likely to be treated differently. For example, numbers are sorted by numerical value; while strings are sorted in alphabetical order. Many programming languages, like C++, Java and Visual Basic, have a large number of data types, e.g. integer, double, string, boolean ... just to name a few. They require declaration of a data type
Slide 30: 18 Chapter 2 Getting Started Escape Character \n \r \t Function Newline Starts a newline Carriage Return Returns to the starting point of the line Tab Analogous to striking the Tab key on your keyboard; However, using tab to make formatter output does not always generate the format expected. Backspace Analogous to the Backspace key; erases the last character Bell Creates a beep sound from the system buzzer (or sound card) ASCII character using hexadecimal notation Outputs the character which corresponds to the specified ASCII index (each n is a hexadecimal digit) ASCII character using octal notation Outputs the character which corresponds to the specified ASCII index (each n is an octal digit) Control Character For example, \cC is equivalent to pressing Ctrl-C on your keyboard Next letter uppercase The letter immediately following \u is converted to uppercase. For example, \uemail is equivalent to Email Next letter lowercase The letter immediately following \l is converted to lowercase. For example, \lEmail is equivalent to email All subsequent letters uppercase All the letters immediately following \U are converted to uppercase until \E is reached All subsequent letters lowercase All the letters immediately following \L are converted to lowercase until \E is reached Disables pattern matching until \E This would be covered in the “Regular Expressions” chapter. Ends \U, \L, \Q Terminates the effect of \U, \L or \Q. \b \a \xnn \0nn \cX \u \l \U \L \Q \E Table 2.1: The most commonly used escape characters in Perl
Slide 31: 2.8 Introduction to Data Structures 19 when a data structure is created, and this data type cannot be changed afterwards. There are both advantages and disadvantages to this approach. As different data types occupy different amount of storage in the memory, the underlying machine actually requires some type information to convert the high-level programming constructs into assembly instructions in the compilation stage. Also, by having the data type fixed there are less ambiguities as to how the data is to be handled. The most obvious disadvantage, of course, is that explicit data conversion is necessary in such programming languages. As of Perl 5, Perl officially differentiates only two types of data: scalar data and list data. Moreover, Perl does not enforce strict type-checking, instead, it is loosely-typed. This may change in Perl 6, but you will not see it in the near future. Scalar data represents a piece of data. All literals are scalar data. Variables are also scalar data. As the underlying machine requires explicit declaration of data types, Perl needs to convert the data between different data types as needed in the underlying implementation, while a Perl programmer can be oblivious to such data conversion. In the next chapter you would see example code in practice. Another type is list data. List data is an aggregation of scalar data. Arrays and hashes belong to this type. While you may not have a clear picture of how list data look like at this point, you would have a clear idea after reading the next chapter. Three basic data structures are provided by Perl, namely scalar variables, arrays and associative arrays (hashes). I am going to give an introduction to the three types of data structures at this point, and in the next chapter you would see the functions and operations associated with these data structures. A scalar variable, or simply a variable, is a named entity representing a piece of scalar data of which the content can be modified throughout its lifetime. What does this mean? A variable is conceptually like a virtual basket, and only one object is allowed in it at any one time. If at some time you would like to place something else in the basket, you have to replace the existing object with a new one, and the existing object is discarded. In Perl a variable can store a piece of string or number (or a reference, which we haven’t come to yet). Unlike other programming languages, Perl gives you the flexibility that at one time you may store a number and at other times you may store a string in the same variable, however, it is a good practice to always store data of a particular type at any time in the same variable to avoid confusion. Because the value of a variable can be modified at any point, and there can be many variables that are concurrently in use at a time, we have to specify which one to address. Therefore, variables are named, and on the other hand literals are unnamed. This name is known as an identifier. By default, all variables are global, that is, after the variable is first used in the script, it can be referred to at any time, anywhere until the script terminates. However, it is a good practice not to use global variables excessively. Instead, most variables are actually used for temporary storage only and can be restricted to be valid for a limited time. This concerns the lifetime of a variable, and in turn the idea of scope. This would be discussed in Chapter 5. Sometimes we are dealing with a set of related data. Instead of storing them separately in variables, we may store them as list data, which is a collection of scalar data sharing a single identifier (name). Arrays and hashes are two types of list data. An array is a named entity representing a list of scalar data, with each item assigned an index. In an array, the integral index (or subscript) uniquely identifies each item in the array. The first item has index 0, the one afterwards has index 1, and so on. Each item in an array is a piece of scalar data, and, therefore, (in Perl only) numbers as well as strings may coexist in the array. An array can be empty, that is, containing no elements (called a null array). A representation of an example ar-
Slide 32: 20 Chapter 2 Getting Started ray containing some data is shown below, the column on the left is the index and the data is on the right: Index 0 1 2 3 Data “Apple” 36 “Hello, World” “School” Table 2.2: Contents of a sample array in Perl A hash is a special data structure. It is similar to an array except that the index is not an integer, so the term “index” is not customarily used for hashes. Instead, a string is used for indexing, and is known as a key. The key is conceptually like a tag which is attached to the corresponding value. The key and the value forms a pair (key-value pair). Like an array, the keys in a hash have to be distinct to distinguish a key-value pair from another. Recall that ordering in arrays is determined by the indices of the items (because we can say the first item is the one which has subscript 0, the second item which has subscript 1, and so on). However, in a hash no such ordering is present. You will see in the next chapter that we may “sort” the hashes by keys or by values for presentation purposes. However, this does not physically reorder the keys or the values in the hash. It merely rearranges the key-value pairs for screen output or to be passed to another process for further processing. Hashes (or hash tables in Computer Science parlance) are especially useful in dictionary programs. Assume that the program works as follows. It requires a user to enter an English word into the text entry box that is to be searched in the dictionary database. Inside the dictionary is actually a long list of key-value pairs, where the key is the word entry and the value is an ID that the database uses internally to retrieve the corresponding record (containing the explanations, pronunciation etc.). The term entered by the user is queried in the dictionary. If the entry matches any key, the corresponding ID is obtained and is used to retrieve the record for the word specified; Otherwise, the term is not found and the program returns an error. Hash table is an efficient data structure for data storage. A well-implemented hash table requires only several comparisons to retrieve the value if the key is in the hash. More surprisingly, even if a given key does not exist in a hash, it is NOT necessary to search through all the keys in the hash before returning the key-not-found error. The reason for this concerns the principle behind hash tables. You may find more information in Appendix A or in any textbooks on data structures and algorithms. Several entries of a possible hash table for the above dictionary program is shown below: Key “Boy” “Apple” “Kite” ... Value 342 165 1053 ... Table 2.3: Contents of a sample hash in Perl In the next chapter, you will learn how to manipulate the data structures discussed. You will know how to construct an array, and remove items from it, etc.
Slide 33: 2.8 Introduction to Data Structures 21 Summary ” ” ” ” ” ” ” ” ” ” ” ” ” Perl is especially strong in text manipulation and extraction of information. To create a Perl program, you need only a text editor and the perl interpreter. The -w switch of the interpreter enables the output of warnings. The print() function can be used to output a string to the standard output (screen by default). Interpreter switches are either specified on the command line or in the script itself as a shebang line for Unix systems. Comments are preceded by the # symbol and can be placed anywhere in your program. A Perl program consists of statements, with each statement terminating with a semicolon. Integers can be expressed in hexadecimal, octal or decimal notation. Double-quoted string literals perform variable substitution, and recognize a number of escape characters. Single-quoted string literals do not perform variable substitution. Scalar data represents a single piece of data. It can be a literal or a scalar variable. List data represents a set of scalar data. Arrays and hashes belong to this type. An array uses a zero-based subscript to refer to the element being referred to, while hash elements are identified by the key.
Slide 34: 22 Chapter 2 Getting Started
Slide 35: Chapter 3 Manipulation of Data Structures 3.1 Scalar Variables I have qualitatively described how a scalar variable looks like in the previous chapter. Now we are going to look at how you can use it in your program. You refer to a variable by appending the identifier of the variable to the symbol $. For example, a variable named LuckyNumber is written as $LuckyNumber. 3.1.1 Assignment In Perl, you are not required to declare a variable before it is being used. However, before you ever use a variable in your program you should give it a default value. To assign a value to the variable we use the assignment operator (=). For example, we can assign the value 18 to the variable $LuckyNumber as follows: $LuckyNumber = 18; This statement is interpreted as follows: the value on the right hand side of the assignment operator is assigned to the data structure on the left. In this case, 18 is assigned to $LuckyNumber. However, the right hand side of the assignment operator is not confined to a literal only. The right hand side of an assignment operator is actually treated as an expression. An expression consists of a sequence of operations which evaluate to a value by means of operators. For example, (6 + 5) * 2 is an expression consisting of two operations (* denotes multiplication). Evaluating an expression means to deduce the result of the expression, by evaluating each operand, and applying the operators in a certain order (subject to operator precedence and associativity) to transform the expression into the value. The expression in this example evaluates to the scalar value 22. You will learn more about operators in the next chapter. Therefore, if we have another variable $Num which has the value of 8, executing the statement $LuckyNumber = $Num causes $Num to be evaluated on the right hand side, and thus its value, that is 8, is assigned to $LuckyNumber. So this is essentially $LuckyNumber = 8. Cascaded assignment is also allowed, e.g. $a = $b = 8; First, 8 is assigned to $b, and then the value of $b is assigned to $a. The net effect is that the two variables both have the value 8. In some other programming languages, to supply a default value before you first use a variable (initialization) is very important. For C/C++, you need to declare a variable before it is used. This 23
Slide 36: 24 Chapter 3 Manipulation of Data Structures reserves memory space for this variable. However, it is not required in C/C++ that a value needs to be assigned during variable declaration. A variable in this state is described as undefined. Getting a value of an undefined variable poses a very subtle source of error (some garbage values are returned — that is, arbitrary value without any meaning) that yields surprising results, and is very difficult to debug. In Perl, if you use a variable that is not initialized (for example, printing its value), the value undef (undefined) is returned. This is a special value that gives different values in different contexts. Contexts will be introduced in the last section of this chapter and, simply put, it is 0 in numeric context (i.e. if a number is expected), an empty string in string context (i.e. a string is expected) or FALSE in “Boolean” context (i.e. when either TRUE or FALSE is expected). However, if you have specified the -w switch, the interpreter should have warned you on using uninitialized variables. The use of uninitialized variables is not a good practice, and you have to ensure that all variables are given a value before being used. As already discussed, you can store scalar data of different types in a variable during the lifetime of the script, so you can now assign the string literal “eight” to $LuckyNumber. Perl will just happily accept it. Of course, you should try to avoid it, as described in the previous chapter. NOTES You may see the terms lvalue and rvalue in some other books or documentation. An lvalue refers to any valid entities that can be placed on the left hand side of the assignment operator, while an rvalue refers to any valid entities that can be placed on the right hand side of the assignment operator. A list, an array or scalar variable can be an lvalue. Literals (scalar in sense) are not lvalues. Different programming languages have different lvalues and rvalues. For example, these are lvalues (of course they can be rvalues as well): $var ($var, @lst) These are not: 46 ("mystuff", "apple") 3.1.2 Nomenclature Before you proceed, I will first talk about how to choose an identifier for a variable (and arrays or hashes alike). An identifier should start with a letter (A-Z, a-z) or underscore ( ). Subsequent letters may be alphanumeric characters (A-Z, a-z, 0-9) or an underscore ( ). No spaces are allowed in the middle of an identifier. There is one more important point. Perl is at all times case-sensitive. That means it differentiates lowercase and uppercase characters. The print() function you saw in the Hello World example cannot be replaced by Print, PRINT or anything else. Similarly, $var and $Var are two different variables. The last point to note is that identifiers cannot be longer than 255 characters (long identifiers are time-consuming to type and difficult to interpret — please avoid them). Another point to note is that the name of a variable, array or hash is formed by a symbol ($ for variable) and the identifier. Therefore, $Var, @Var and %Var can coexist. Although they have the same identifier, they are still unique names because the symbols are different. Also, you may use “reserved words” for the identifier, e.g. $print because the symbol before the identifier tells Perl that this is a variable. There
Slide 37: 3.1 Scalar Variables NOTES If you read on, you will discover that Perl has many builtin predefined variables which do not follow such a nomenclature scheme. For example, $1 - $9 are reserved for backreferencing in pattern matching (see Chapter 9 “Regular Expressions”), or other more awkward looking variables like $,, $", $ ... just to name a few. Because devoting a whole chapter just to introduce them would be too boring, I will introduce them as needed throughout the text. Alternatively, the perlvar manpage describes these Perl predefined variables in detail. 25 is thus no ambiguities. 3.1.3 Variable Substitution It’s time to talk about variable substitution. I told you that single-quoted strings do not allow variable substitution, while double-quoted strings can. Variable substitution means that variables embedded in a double-quoted string will be substituted by their values at the time the statement is evaluated. Consider this example: EXAMPLE 3.1 Celsius → Fahrenheit Converter 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #!/usr/bin/perl -w # c2f.pl # A Celsius->Fahrenheit Converter # Print the prompt print "Please enter a Celsius degree > "; # Chop off the trailing newline character chomp($cel = <STDIN>); $fah = ($cel * 1.8) + 32; # print value using variable interpolation print "The Fahrenheit equivalent of $cel degrees Celsius is $fah\n"; Line 9 looks awkward, but here we perform two operations, to accept user input and remove the trailing newline character. You don’t have to concern much about this statement yet, as this will be described in Chapter 8. Line 11 calculates the Fahrenheit temperature, and when execution of the script reaches line 14, it sees the two variables $cel and $fah. Then Perl replaces them with the values the two variables carry at that instant, and the resulting string is output to the screen. There is one problem arising from variable substitution. What if we have some words immediately following the variable, without even a space? Perl has provided a nice facility to get around this situation. You may put a pair of curly brackets around the identifier name to separate it from the surrounding text, e.g.
Slide 38: 26 Chapter 3 Manipulation of Data Structures print "${Num}th Edition"; 3.1.4 substr() — Extraction of Substrings Frequently you have to extract a sequence of characters from a string. In other words, you extract a substring from it. Perl provides you with the substr() function to extract a substring, provided that you already know in advance where to start extracting it. The syntax of the substr() function is as follows: substr STRING, OFFSET substr STRING, OFFSET, LENGTH substr STRING, OFFSET, LENGTH, REPLACEMENT As you can see, substr() can take 2-4 parameters depending on your needs. STRING is the string from which extraction is performed. OFFSET is a zero-based offset which indicates the position from which to start extraction. The first character of any string has an OFFSET 0, and 1 for the second character etc. In Perl, OFFSET can be negative, which counts from the end. For example, the last character of the string can be represented by the OFFSET -1. LENGTH is the number of characters to extract. If it is not specified, it extracts till the end of the string. The extracted substring is returned upon evaluation. Here are some examples: $string = "This is test."; print substr($string, 5); print substr($string, 5, 2); # is test. # is If REPLACEMENT is specified, the substring is replaced by the string obtained by evaluating REPLACEMENT, and the substring being replaced is returned. Alternatively, you may put substr() on the left hand side of an assignment operator, and REPLACEMENT on the right. Here is an example which replaces a substring of length 0 with the replacement string “not a ”, that implies inserting it at position 8: substr($string, 8, 0, "not a "); ## First method substr($string, 8, 0) = "not a "; ## Second method print $string; # This is not a test. 3.1.5 length() — Length of String You can find out the length of a string by using the length() function. The only parameter for the length() function is the string itself. It returns the number of characters in the string. Scalar variable is a very simple data structure. In the next section we are going to deal with lists, arrays and hashes that are a lot more interesting to play with. 3.2 Lists and Arrays Arrays are named entities, lists are not. The relationship between a list and an array is very much similar to that between a literal and a scalar variable. A list is merely an ordered set of elements. An array is just like a list but with a name thus can be referenced through an array variable. Studying the behaviour of lists allow us to progress naturally into arrays and hashes in subsequent sections.
Slide 39: 3.2 Lists and Arrays 27 3.2.1 Creating an Array Each item in the list is called an element. To create a list, simply delimit (separate) the elements with commas (,) and surround the list with a pair of parentheses. For example a list containing the names of some colours can be written as ("red", "orange", "green", "blue") An array can be created by assigning a list to an array variable. An array variable starts with the symbol @ (compare with the case of $ for scalar variables). Therefore, an array can be set up containing the list above, e.g. @colors = ("red", "orange", "green", "blue"); Alternatively, you may use the equivalent method of per-item assignment (to be discussed shortly): $colors[0] $colors[1] $colors[2] $colors[3] = = = = "red"; "orange"; "green"; "blue"; This array contains 4 elements. A null array is simply (). Lists can be nested, that is, a list may contain other lists or array variables as an element. However, the embedded lists will be expanded and merged with the container list. Any null lists or null arrays are removed. For example, if @unix = ("FreeBSD", "Linux"); @os = ("MacOS", ("Windows NT", "Windows ME"), @unix); @os is expanded into @os = ("MacOS", "Windows NT", "Windows ME", "FreeBSD", "Linux"); In the following example, @result will become a null array. Note that null lists are ignored. @nullarray = (); @result = ((), @nullarray); A useful operator that worths mentioning here is the range operator (..). If you would like to generate an array of consecutive integers this operator may come in handy, as you no longer have to use a loop to do it. Example: @hundrednums = (101 .. 200); However, the numbers must be in ascending order. If you would like to have an array of consecutive integers in descending order, you may construct it in ascending order using the range operator, and then reverse the position of the items using the reverse() function: @hundrednums = reverse (101 .. 200);
Slide 40: 28 Chapter 3 Manipulation of Data Structures 3.2.2 Adding Elements We know from the previous section that if we place an array variable or a sublist as an element of a list, the array variable or the sublist would be expanded and merged with the parent list, and in this operation the identity of the original array variables or sublists are lost. That means, you can no longer tell from the resulting list if a particular element originates from a sublist or an array variable. Therefore, it is natural to conclude that two arrays can be merged together by this operation: @CombinedArray = (@Array1, @Array2); The resulting array contains all the elements in @Array1, followed by that of @Array2. To append a scalar element to the end of an array, you can write, for example, @MyArray = (@MyArray, $NewElement); We can also append a list of scalar data to the end of an array by using the push() function. The syntax of the push() function is push ARRAY, LIST; where ARRAY is the array to which the list data are to be appended. LIST is a list specifying the elements to be appended to ARRAY. The mechanism of the push function is not much different from the interpolation of lists above, and I am more accustomed to the previous one than using the push function, because it is more intuitive to understand. push is a function. A function returns some values (not necessary scalar, can be list data as well for certain functions) after the operation is finished. For this function, the number of elements after element addition is returned. Consider the example below: $NumElements = push(@MyArray, @list); Note that I have added the parentheses around @MyArray and @list. The parentheses here are not actually necessary, but I added them here to make it obvious the parameters of the function. The return value is assigned to $NumElements. In the next chapter, you will learn operator precedence, which describes in detail when and where you should add parentheses. For the time being, stick to the way I have been doing and it would be fine. On the other hand, this operation inserts the element at the beginning of the array: @MyArray = ($NewElement, @MyArray); unshift is a function that inserts a list at the beginning of an array, and returns the number of elements after the operation. The syntax is unshift ARRAY, LIST;
Slide 41: 3.2 Lists and Arrays 29 where ARRAY is the array to which elements are added, and LIST is the list that is inserted at the beginning of ARRAY. Consider the following example: EXAMPLE 3.2 Demonstrating unshift() 1 2 3 4 5 6 7 8 #!/usr/bin/perl -w @alpha1 = ("a", "b", "c"); @alpha2 = ("d", "e", "f"); unshift @alpha2, @alpha1; $, = " "; # Prints a space in between elements print @alpha2; Again, disregard the line numbers that are for illustration only. Note that the ordering of the items of @alpha1 is preserved in @alpha2. What is worth noting is on line 7. We assign a space to an odd-looking variable, but what’s that? This is an example of Perl predefined variables, and $, is known as the output field separator. Without line 7, you would most likely get an output like “abcdef ” because by default this variable is an empty string. Note that the print() function can accept a list of parameters. The output field separator is the string that is added in between list elements in a print operation. You may assign any valid string literals to this variable. In this example, I put a space in between elements to make them easier to read. 3.2.3 Getting the number of Elements in an Array There are two ways in which you could obtain the number of elements stored in an array. The first method is to employ the concept of context. By evaluating an array in scalar context, we can obtain the number of elements in an array. You may not understand what this is for the time being, but we will return to this example when we come to contexts later in this chapter. In the following example, the number of elements in @colours is asssigned to $numElements: $numElements = @colours; The second method is a little bit clumsy to explain. In Perl, you can find out the subscript of the last item of an array by replacing the symbol @ with $#. For example, the subscript of the last item in @Array is given by $#Array. For historical reasons, Perl provides a facility for users to specify the subscript of the first element of an array. This is specified by assigning an integer to the predefined variable $[. This is 0 by default, and that’s why I said subscripts start from 0. However, some people may be accustomed to using 1 as the starting index (especially those who have used some “antique” programming languages). This value is not necessarily 0 or 1. You may set other values as well. However, you cannot assign values to this variable more than once. Although Perl provides this facility, you are advised not to use it because of potential confusions that may arise, especially if your project consists of a number of files. With both the start index and the index of the last element, we can get the number of elements by using the formula:
Slide 42: 30 Chapter 3 Manipulation of Data Structures Number of elements = Last Index - Starting Index + 1 = $#Array - $[ + 1 As the start index is 0 by default (that is, if you don’t specify otherwise), you may assume that the number of elements of @Array is given by $#Array + 1 3.2.4 Accessing Elements in an Array After we have added items to an array or list, we can access any of its elements by using the subscript operator ([]). For example, if we would like to retrieve the third element (remember that subscripts count from 0) from @colours and return the value to $col3, we can write $col3 = $colours[2]; Note that the symbol is $ instead of @ on the right hand side of the assignment operator. In Perl, because the value returned is a scalar value, the symbol $ is used. Though looking awkward, you can definitely use the subscript operator on a list, like this: $col3 = ("red", "orange", "green", "blue")[2]; Obviously, using the subscript operator in this way is not really useful. What about specifying a negative subscript? Negative subscripts count backwards, from the last element of the array. The last element has the subscript -1, and the next-to-last element has the subscript -2. Therefore, the first element of the array @array has the subscript -@colours (due to scalar context). Normally, negative subscripts are used as a convenience method to retrieve the last element of an array. At least, @array[-1] is definitely easier to understand than @array[$#array]. An array slice is a subset of elements from all the elements in an array. The subscript operator is not confined to one subscript only. You may specify a list of subscripts using the comma operator (,) and the range operator (..). You use the range operator to specify subscripts (must be integral) in a given range, and the comma operator to specify each subscript individually. Nothing will stop you from using both operators together, as in the following example: EXAMPLE 3.3 Array Slices 1 2 3 4 5 6 7 #!/usr/bin/perl -w @alpha = (’a’ .. ’z’); @slice = @alpha[4, 10 .. 15]; $, = " "; print @slice; In this example, the array slice @slice contains the 5 th element, and 6 elements starting from the 11 th element of @alpha. The resulting output is thus “e k l m n o p”. Note that because the array slice contains a list of values, the symbol of @alpha is @ on line 4.
Slide 43: 3.2 Lists and Arrays 31 We can, of course, modify the value of any element in an array. To do this, just assign a scalar value to the corresponding array element. Similar is the case for slice assignment, in which a list of scalar data is assigned as a list. For example: $colours[2] = "violet"; # The followings are identical @colours[2,4] = ("violet", "blue"); $colours[2] = "violet"; $colours[4] = "blue"; @colours[4,2] = ("blue", "violet"); If you specify a subscript that is larger than $#array, the size of the array shall grow in order to accommodate the newly added element. In this example, because @nums contains originally 3 elements, assigning a value to the 5th element leaves a “gap” at the 4th element, not assigned any values. The value of this element is undef. EXAMPLE 3.4 Array Expansion 1 2 3 4 5 6 7 #!/usr/bin/perl -w @nums = (3, 4, 5); $nums[4] = 7; # (3, 4, 5, undef, 7) expected $, = "\n"; print @nums; To show that there is an empty gap at the 4 th element, the output list separator is set to \n. So you would see a line like “Use of uninitialized value at eg0304.pl line 6” between 5 and 7 in the output if warnings are enabled. That’s because retrieving the value of an uninitialized value produces this Perl warning. 3.2.5 Removing Elements We can use the pop function to remove the last element of an array. It also returns the value of the item being removed. Syntax: pop ARRAY; In this example, the last item of @MyArray is removed and its value is assigned to $retval: $retval = pop(@MyArray); On the other hand, the shift function removes the first element of the array, so that the size of the array is reduced by 1 and the element immediately after the item being removed becomes the first element of the array. It also returns the value of the item being removed. Syntax: shift ARRAY; If ARRAY is empty, undef is returned.
Slide 44: 32 Chapter 3 Manipulation of Data Structures 3.2.6 splice(): the Versatile Function There is a generalized function for adding and removing elements from an array. The splice function is so general that it can do what push, pop, shift and unshift does. The syntax is: splice ARRAY, OFFSET [, LENGTH [, LIST]]; In this tutorial, the parts in slanted font denote optional parameters. These parameters are optional in the sense that in some situations they are optional; but not in other situations. I use [] to label that a parameter is optional. Note that [LIST] is placed within another optional parameter LENGTH. This means that if you have to specify the parameter LIST, you must also specify LENGTH; but not vice versa. If you specify the parameter LENGTH, you may or may not supply the parameter LIST. In general, this function removes LENGTH elements starting from the element of subscript OFFSET of ARRAY, and inserts LIST at OFFSET if any. Simply put, the list @ARRAY[OFFSET .. OFFSET + LENGTH - 1] is replaced by LIST. The syntax shows that this function takes three forms, and I am going to describe them one by one. splice ARRAY, OFFSET, LENGTH, LIST This performs exactly the action above. In the example below, an array containing the 26 lowercase alphabets was built, and 5 elements starting from the 5 th element (i.e. the letter “e”) is converted to uppercase. This is done on line 2. First @alpha[4 .. 8] contains the 5 letters that are to be converted to uppercase (remember that subscripts start at 0). The map function calls the uc function (uppercase) for every element in this list, thus converting ("e", "f", "g", "h", "i") to ("E", "F", "G", "H", "I"). The splice function, therefore, replaces the lowercase list with the uppercase one. uc and map would be covered later. EXAMPLE 3.5 splice 1 2 3 4 5 6 7 #!/usr/bin/perl -w # Example 3.5 - splice @alpha = (’a’ .. ’z’); splice @alpha, 4, 5, map(uc, @alpha[4 .. 8]); $, = ’ ’; print @alpha; What you will see on the screen should be all lowercase alphabets except E, F, G, H and I. splice ARRAY, OFFSET, LENGTH If you don’t specify the replacement list, the action is merely removing LENGTH elements from ARRAY starting from subscript OFFSET. splice ARRAY, OFFSET If you don’t specify the LENGTH, Perl assumes that all elements starting from OFFSET are removed. OFFSET is just a subscript and can be negative as well. A negative value specifies the OFFSET is counted
Slide 45: 3.2 Lists and Arrays 33 from the end of the array as mentioned earlier, e.g. -1 means the last item, -2 means the second last item etc. Therefore, pop @MyArray can be equivalently accomplished by splice @MyArray, -1. The following table summarizes how you can use splice in place of other functions discussed earlier (still remember that in Perl you always have a number of ways to do the same task?). You may find the equivalent method for push() strange, but after you have learnt the whole theory behind contexts you would understand it. We would use this illustration again when we come to contexts. Function push(@Array, $x, $y) pop(@Array) shift(@Array) unshift(@Array, @x) $Array[$x] = $y Equivalent Method splice(@Array, @Array, 0, $x, $y) splice(@Array, -1) splice(@Array, 0, 1) splice(@Array, 0, 0, @x) splice(@Array, $x, 1, $y) Table 3.1: Relationship of some array functions with splice() 3.2.7 Miscellaneous List-Related Functions There are a number of useful functions that allow you to manipulate list data in Perl. join STRING, LIST The join() function concatenates a list of scalars into a single string. It takes a STRING as its first argument which is the separator to be put in between the list elements LIST. For example, join ’+’, ’apple’, (’orange’, ’banana’) evaluates to apple+orange+banana reverse LIST In a list context, the reverse() function returns a list whose elements are identical to that of LIST except the order is reversed. For example: print join " ", reverse ’a’..’e’; #edcba The map() function takes on one of the following two forms: map BLOCK LIST map EXPR, LIST
Slide 46: 34 Chapter 3 Manipulation of Data Structures The map() function iterates over every item in the LIST, sets $ to the item concerned and executes BLOCK or EXPR on each iteration. The return value is a list consisting of the result of evaluation of all iterations. BLOCK is a code block enclosing a sequence of statements to be executed. EXPR can be any expression. Consider this example: @names = (’ALICE’, ’tOm’, ’JaSON’, ’peter’); print join(’, ’, map { ucfirst(lc($_)); } @names), "\n"; The output is Alice, Tom, Jason, Peter This example prints out each of the names in @names so that they all start with capital letters while the other characters are in lowercase. This is accomplished by first converting all characters to lowercase by the lc() function, and the first letter is capitalized using the ucfirst() function. This process is performed for each name in @names. The same expression may be rewritten as @names = (’ALICE’, ’tOm’, ’JaSON’, ’peter’); print join(’, ’, map (ucfirst(lc($_)), @names)), "\n"; The first form of map() is generally preferred to the second one because it is more flexible. The sort() function can be used to sort a list. By default, the sort() function sorts lexicographically. The items are ordered by comparing the items stringwise (using the cmp operator, the specifics of which will be introduced in the next chapter). This comparison is case-sensitive, because it is based on the ASCII values of each character. The sorted list is returned by the sort() function, while the original list remains intact. sort (’bear’, ’Post’, ’ant’); # (’Post’, ’ant’, ’bear’) Note that while comparing stringwise, capital letters are considered “smaller” than lowercase letters. To make the sort routine generic and allow sorting in any arbitary order, you may override the default sort criteria with your own rules. The method is to insert a code block before the list, similar to the case for map(). The value resulted from evaluation of the block determines how the items are sorted. The principle of constructing the contents of the block is too advanced at this stage. The rest of the details can be found in the next chapter. In the following I list several most commonly used sort criteria that you are likely to find useful: sort sort sort sort { { { { $a $b $a $b <=> <=> cmp cmp $b $a $b $a } } } } @list; @list; @list; @list; # # # # ascending numerical order descending numerical order ascending lexicographical order (default) descending lexicographical order # case-insensitive ascending lexicographical order sort { lc($a) cmp lc($b) } @list; # or use uc()
Slide 47: 3.2 Lists and Arrays 35 3.2.8 Check for Existence of Elements in an Array (Avoid!) Many new programmers, and programmers who have programmed in other languages tend to use arrays so extensively that are sometimes inappropriate from performance considerations. One classical problem is to deduce if a particular piece of scalar data matches any of the elements stored in an array (in alternative terminology, check for a hit or a miss). In some languages like BASIC where a hash is not a builtin type, there is still little excuse to solve this problem by searching an array, although you can still implement a hash yourself (For those who are interested in implementing a hash themselves, please read Appendix A). In general, you can check if a certain element exists in an array by linear search. That is, you search from the first element up to the end of the array. In the following code snippet, a random list containing 100 entries is generated randomly and you can enter a number to be searched. The program then searches for the number. When the search ends, it returns whether it is found, and how many elements the program has searched. EXAMPLE 3.6 Linear Search of an Array 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 #!/usr/bin/perl -w # Linear search of an array # Note that if you later on want to search for something from a # list of values, you shouldn’t have used an array in the first # place. # Generating 100 integers $NUM = 100; $MAXINT = 5000; # 1 + the maximum integer generated srand(); # initialize the randomize seed print "Numbers Generated:\n("; for $i (1 .. $NUM) { push @array, sprintf("%d", rand(1) * $MAXINT); print $array[$i-1]; print ", " unless ($i == $NUM); } print ")\n\n"; print "Please enter the number to search for >> "; chomp($toSearch = <STDIN>); # Linear search here $counter = 0; $hit = 0; foreach $num (@array) { $counter++; if ($num == $toSearch) { print "\"$toSearch\" found at subscript ", $counter - 1, "\n"; $hit = 1; last;
Slide 48: 36 34 35 36 37 Chapter 3 Manipulation of Data Structures } } if ($hit == 0) { print "\"$toSearch\" not found in array.\n"; } print "Number of comparisons: $counter/", scalar(@array), "\n"; The code itself is not very important here. In fact, many new constructs used in this program have not been discussed yet. The intent is for you to run the program instead of reading the source code (but you may do it). Try to look at the list of values printed, and try to enter a number that is not in the array, a number that appears early in the list, and a number that appears around the end of the list. A fraction is printed on the screen. The one on the left is the number of comparisons performed, while the one on the right is the number of elements in the array (which should be 100 in this example). You may also want to increase $NUM to increase the number of integers generated. Of course, in this case you will need to increase $MAXINT accordingly to minimize the chance of duplication of values. Note that it is possible to have duplicate values in the array owing to the random nature of the random number generator. In this program, the first occurrence (i.e. with the lowest subscript) will be returned. An alternative scheme is to use binary search, a rather classical topic in elementary computer science texts on algorithms. By using binary search, the whole set of values needn’t be searched in the worst case. In the worst case only log 2 n comparisons are required. However, there is an important requirement — the list needs to be already sorted. If not, sorting needs to be performed first. Here is the above example rewritten that employs the binary search algorithm: EXAMPLE 3.7 Binary Search of an Array 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 #!/usr/bin/perl -w # Binary search of an array # Note that if you later on want to search for something from a # list of values, you shouldn’t have used an array in the first # place. # Generating 100 integers $NUM = 100; $MAXINT = 5000; # 1 + the maximum integer generated srand(); # initialize the randomize seed print "Numbers Generated:\n("; for $i (1 .. $NUM) { push @array, sprintf("%d", rand(1) * $MAXINT); print $array[$i-1]; print ", " unless ($i == $NUM); } print ")\n\n"; print "Please enter the number to search for >> "; chomp($toSearch = <STDIN>); # First sort it in ascending numerical order
Slide 49: 3.2 Lists and Arrays 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 37 @sortedArray = sort {$a <=> $b} @array; # Binary search here $counter = 0; $start = 0; $end = $#sortedArray; $mid = 0; $hit = 0; $mid = sprintf("%d", ($start + $end)/2); while ($start <= $end) { $counter++; print "Searched: ", $sortedArray[$mid]; # which element is being ¶ searched print " in #[$start, $end] ["; # the subscript range print $sortedArray[$start], ", ", $sortedArray[$end], "] \n"; if ($sortedArray[$mid] == $toSearch) { # a hit! print "\n\"$toSearch\" found!\n"; $hit = 1; last; } elsif ($sortedArray[$mid] > $toSearch) { # decrease upper boundary -> mid value $end = $mid - 1; } else { # update lower boundary -> mid value $start = $mid + 1; } $mid = sprintf("%d", ($start + $end)/2); } if ($hit == 0) { print "\n\"$toSearch\" not found in array.\n"; } print "Number of comparisons: ", $counter, "/", scalar(@sortedArray), "\n"; This implementation is even more complicated than that of the linear search. What is worth noting is the very low number of comparisons required in any cases (For 100 integers the fraction printed in any case should be at most 7/100 1 ). That means among the 100 integers in the list, at most 7 elements searched is sufficient to deduce whether any specified number exists in the array. In Chapter 5, as an exercise, you would be asked to convert this program into the recursive form. Binary search works as follows. The list is sorted in either ascending or descending order (assume we have the list sorted in ascending order here). In this algorithm we always maintain two important variables — $start and $end. Initially they are set to 0 and 99 (the subscript of last element) respectively. Then the element in the middle (subscript 44) is examined and compared with the number to be searched. If the number examined is larger than the input number, that means the input number, if exists, would have a subscript less than 44, so we can shrink the range by reducing $end to 43. Otherwise it would have a subscript greater than 44, so we increase $start to 45. Such a bisection process is repeated with this new set of $start and $end, until the number is found or $end is less than $start. Note that on each iteration the range examined is halved, and that’s why the maximum number of comparisons is log2 (n). An example of binary sort is shown in Figure 3.1. 18 is to be searched and is found on the third iteration. The red boxes denote the element in the middle of the range being searched, and those in 1 The maximum number of comparisons can be calculated by ceiling(log (n)) where ceiling(y) is the minimum integer 2 that is greater than or equal to y. In this case ceiling(log2 (100)) = 7.
Slide 50: 38 Chapter 3 Manipulation of Data Structures grey are those that are bypassed. Figure 3.1: An illustration of binary sort Although it seems binary search performs a lot better compared with linear search, there is an important catch here — most arrays we encounter are not sorted, so the cost of sorting cannot be simply ignored in practice. As you can possibly see in both cases when $NUM is large, neither approach is performing in an efficient manner, especially when misses occur because in the linear search case, every element in the array has to be searched in the worst case (when a miss occurs); while in the binary search case, the array needs to be sorted first, and the average sorting time of an array grows with the array size 2 , so combining sorting and searching may be even slower than the linear search algorithm. Towards the end of the next section we shall redo this code with a hash, and you ought to find the code can be made cleaner and a hit or a miss can be determined faster in general, especially with large array sizes. 3.3 Hashes Hash is a special kind of data structure. There are several characteristics associated with it. It practically takes very short time to deduce whether any specified data exists. Also, the time it takes does not largely depend on the number of items stored. This is important because hashes are usually used for applications that handle a large amount of data. An array is simply a contiguous block of memory and is nothing more than that. In order to support the characteristic stated above, hashes require a slightly more complicated internal structure. This is outlined in Appendix A for your reference. It explains the general principles that further Perl knowledge is not necessary in order to understand it. However, in this section we are dealing with how we use hashes in Perl, and will not discuss the peculiarities of them here. As a quick review, each item in a hash has a key and a value. The key, which is a string, uniquely identifies an item in the hash. The value is any form of scalar data. Hash variables start with the symbol %. 3.3.1 Assignment We may assign a list to a hash variable. In this case, the list will be broken up two-by-two, the first one as the key and the second as the value: 2 In algorithmic analysis, the best sort algorithms can attain the time complexity of O(nlogn). That is, the sorting time t is in the form t = Knlog(n) where K is a constant and n is the size of the array. The function grows even faster than n, so performance degrades would be evident at relatively large values of n.
Slide 51: 3.3 Hashes %Age = (’Tom’, 26, ’Peter’, 51, ’Jones’, 23); 39 Because a hash contains multiple key-value pairs, this alternative syntax may seem more intuitive to look at: %Age = (’Tom’ => 26, ’Peter’ => 51, ’Jones’ => 23); The symbol => is defined as an almost equivalent symbol to the comma, so in general anywhere a comma is needed you can replace it with this symbol. However, the use of => is particularly intuitive for data that occur in pairs. In addition, Perl allows even less keystrokes by omitting the quotation marks: %Age = (Tom => 26, Peter => 51, Jones => 23); This is because Perl always interprets the word before the symbol => as a double-quoted string. However, if you omit the quotes, the key cannot have embedded whitespace (space, tabs and so on). To specify a key with embedded whitespace, the quotes must be specified. Also, because Perl uses the semicolon, not a newline, to mark the termination of a statement, you can make the hash assignment better to look at by writing it in multiline form: %Age = ( Tom => 26, Peter => 51, Jones => 23, ); NOTES One note about why we can omit the quotes here. According to the perldata manpage, a word that has no other interpretation in the grammar will be treated as if it were a quoted string. For example, if an unquoted word is not a reserved word, filehandles, labels etc. Perl will automatically treat it as a bareword. However, avoid barewords consisting entirely of lowercase letters, because all Perl reserved words are in lowercase. As Perl is case-sensitive, this eliminates the possibility of potential name clashes in future versions of Perl. Note that on the final line there seems to be a superfluous comma there. Yes, it is. The final comma before the closing parentheses can be omitted. But it is customary to have it there in the above multiline form because it is likely that you may append more key-value pairs later on. Then you will have to add the comma back, and it will result in an error if you have omitted to do so. Bear in mind that the four formats above are identical, and you may choose the form which looks best to you. An empty hash is an empty list assigned to a hash variable, similar to the case of arrays. Because in the assignment operation Perl expects a list as the rvalue, apart from lists you may as well assign an array, or even a subroutine returning a list to the hash variable. Note that the hash variable provides a list context here. If we assign a hash variable to an array, or anywhere a list is expected, the key-value pairs stored in the hash will be returned in list form. However, because of the way the key-value pairs are stored in a hash, they may be (actually most likely) returned in an order different from when they were put into the hash.
Slide 52: 40 Chapter 3 Manipulation of Data Structures 3.3.2 Accessing elements in the Hash Accessing an element in a hash is similar to that from an array, except we replace square brackets with curly ones and instead of an index, the key is used. Here’s an example: print $Age{Tom}; As you will learn in Chapter 10, a word appearing inside the curly braces not enclosed by quotes is evaluated as a string. Therefore, Tom is equivalent to ‘Tom’ in this case. To associate a scalar value to a key is as simple as: $Hash{’Key’} = $value; If Key already exists, it is assigned the supplied value; otherwise, a new key-value pair is added to the hash. 3.3.3 Removing Elements from a Hash Perl provides the delete function for removing a specified key-value pair from a hash. Here’s an example: delete $Age{Tom}; This function returns the deleted scalar value associated with the specified key(s). So if you print the return value in the above statement, the value(s) deleted would be displayed on screen. To delete all key-value pairs in a hash, you can of course use a loop to do it, but this is slow and it would be more efficient to use either method below: %Age = (); or undef %Age; Here is a special example for delete that is worth mentioning: 1 2 3 4 5 6 7 8 9 %Age = ( Tom => 26, Peter => 51, Jones => 23, ); @temp = delete @Age{’Tom’, ’Peter’}; $, = " "; print "Deleted values:", @temp, "\n"; print "Remaining keys:", keys @Age;
Slide 53: 3.3 Hashes 41 On line 6, we delete multiple key-value pairs from %Age. Note that the function returns a list of values associated with the deleted keys, so the symbol used should be @. Line 9 uses the keys() function. It returns a list of keys in the specified hash. There is a corresponding values() function that returns a list of values contained in the hash. Recall the sort() function mentioned earlier. Apart from arrays, you may use the sort() function together with the keys and values function to specify how to order the hash items. For example, %array = ( ’3’ => ’apple’, ’11’ => ’orange’, ’5’ => ’banana’, ); @key = sort { $a <=> $b } keys %array; @value = sort { $a cmp $b } values %array; # (’3’, ’5’, ’11’) # (’apple’, ’banana’, ’orange’) 3.3.4 Searching for an Element in a Hash You may test if a particular key exists in a hash by using the exists function. However, even if the key exists in the hash, the value associated may be undefined. Use the defined function to test if the value is defined. Recall earlier we had a program that generated 100 integers in random and you look for the existence of a particular number in the list. We now present the version using a hash. The number of comparisons taken is not shown, as you cannot get this information with a builtin hash. Of course if you build a hash yourself you would be able to see the efficiency of hashes. At least, you will find the implementation is a lot easier compared with the two previous approaches. EXAMPLE 3.8 Searching for an Element in a hash 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 #!/usr/bin/perl -w # Search for an element in a hash # Generating 100 integers $NUM = 100; $MAXINT = 5000; # 1 + the maximum integer generated srand(); # initialize the randomize seed print "Numbers Generated:\n("; for $i (1 .. $NUM) { $valueToInsert = sprintf("%d", rand(1) * $MAXINT); $hash{$valueToInsert} = 0; # in fact, any values can be assigned here print $valueToInsert; print ", " unless ($i == $NUM); } print ")\n\n"; print "Please enter the number to search for >> ";
Slide 54: 42 21 22 23 24 25 26 27 28 Chapter 3 Manipulation of Data Structures chomp($toSearch = <STDIN>); # Hash search here if (exists($hash{$toSearch})) { print "\"$toSearch\" found!\n"; } else { print "\"$toSearch\" not found!\n"; } Notice how clean it is to determine whether the number exists in the hash in this case (line 24). In this example, the numbers are stored as keys in the hash. We use the exists() function to check if the key exists in the hash. This function returns TRUE if the specified key exists. Note that you can as well use defined() in place of exists() in this example. This function returns true only if the key exists in the hash, and the value is not undefined (i.e. undef). Table 3.2 summarizes the difference between defined() and exists(): Function exists() Key Exists Yes Yes No Yes Yes No Value Undefined No Yes N/A No Yes N/A Return Value TRUE TRUE FALSE TRUE FALSE FALSE defined() Table 3.2: Differences between exists() and defined() In the example, because 0 (or anything except undef) is assigned as value to each key put into the hash, therefore, exists() and defined() yields the same results. Owing to the nature of a hash, duplicate keys are not allowed. However, as I have previously noted it is possible that by using such a random generation scheme duplicate elements may be generated, and in this case exactly only 1 instance is stored, so the number of elements in the hash in this example can be less than 100. 3.4 Contexts The idea of contexts in Perl may appear a bit odd at first glance, but you would soon discover that contexts are actually quite intuitive to understand, because there are many real life examples resembling contexts in Perl. First, look at these two phrases. Note that the same word “press” appears in both sentences, but their meanings in the phrases are entirely different. Press the button Freedom of press The word “press” plays different roles in the two sentences, one acting as a verb and the other as a noun. Therefore, to deduce the meaning of the word in the phrase, we have to look at the words surrounding
Slide 55: 3.4 Contexts it, in other words, the context. 43 I’m sure you would find many other examples that exemplify how contexts play their roles in our daily lives, so I am not going to spend too much time mentioning things you may have already known. Recall that earlier in this chapter I obtained the number of elements of an array by using an assignment like this: $numElements = @colours; You may find this assignment rather peculiar. It seems that I have been assigning list data to scalar data, and they actually don’t match! How can a scalar data store an array? Yes. That’s impossible (unless by using references - that we will see at a later part of this tutorial). Also recall that I mentioned earlier that I was “evaluating an array in scalar context”. Now I am telling you what this phrase means. Because you are now assigning something to a scalar variable, the data type expected on the right hand side of the assignment operator is naturally a scalar value. In this way, we have created a scalar context around the array @colours. Because a scalar value is requested instead of an array, Perl defines that by evaluating an array in scalar context, the number of elements stored in the array is returned and the value of which is assigned to $numElements. It is important to bear in mind that the rules as to how a data type is evaluated in another context depend on the definition, and there is no general rules of inference. Later on, when you are to write subroutines, you will be taught to use the wantarray() function to determine the contexts so that you can specify what to return from your subroutine in different contexts. Remember I have mentioned Boolean context earlier in this chapter? Actually there is not a “Boolean” context, because Perl does not have an intrinsic Boolean data type. Perl achieves this by using scalar context instead. Perl defines that the numeric zero (0), the empty string (“”) and the undefined value (undef) are interpreted as FALSE, and all other scalar values are interpreted as TRUE. Therefore, anywhere a Boolean test is expected you can place any scalar value there instead, and Perl shall evaluate it according to this rule. Sometimes you may want to evaluate a list in scalar context, but it is in list context instead. A useful function about contexts is scalar, which provides the necessary scalar context, as in the following example: print scalar(@array); Without the scalar function, the array is evaluated in list context and so the contents of the array will be printed. However, if we would like to print the number of elements in the array instead, the scalar function provides the necessary scalar context for this purpose. I do not aim at telling you everything about how different contexts give rise to different behaviours when we deal with operators at the moment. Instead, it is important to realize that operators can exhibit different behaviours depending on the context they are in. You are going to have a boring day probably reading the next chapter, as you are going to know in more details how each operator is affected by contexts.
Slide 56: 44 Chapter 3 Manipulation of Data Structures 3.5 Miscellaneous Issues with Lists Before we start a new chapter, let us pay attention to several issues concerning lists which have not yet been mentioned above. You have seen in the previous section that, because of evaluation of an array variable in scalar context, the number of elements in the array is returned: $a = @array; but things get entirely different when you write it this way: $a = (35, 48, 56); This is a special case, because in scalar context the list on the right of the assignment operator is actually a list of values delimited by the comma operator (,). This operator is borrowed from C/C++ and has nothing to do with lists or arrays. The behaviour of this operator is, evaluate each of the expressions (in this case, numbers) delimited by commas, and return the value resulting from the last expression. Therefore, in this example 56 is returned and assigned to $a. This applies to void context as well. While @array = (35, 48, 56); the commas there act as list argument separators. As a rule, always bear in mind that a list is only a list in list context. In other contexts the rules of comma operators apply. Thanks to the flexible syntax of Perl, we can assign multiple values concurrently, like this (this is called list assignment): ($a, $b) = (11, 22); Both sides are lists of the same size, so this is just a mapping: 11 is assigned to $a and 22 to $b. What about if the list on the right has more elements than the one on the left? Then the extra elements are simply ignored. What about if the list on the left has more elements? Then some variables in the left hand list shall receive the value undef, as you may expect. To swap the values of two variables, an operation that requires an additional temporary variable in most other programming languages, you can perform it in Perl simply by: ($x, $y) = ($y, $x); Consider this example and try to guess what happens: ($a, @b) = (11, 22, 33, 44); 11 is assigned to $a. The remaining three elements are assigned to @b. But what if you do this? (@a, $b) = (11, 22, 33, 44);
Slide 57: 3.5 Miscellaneous Issues with Lists 45 In this case, @a gobbles up all the values in the list, leaving the value undef to $b. We describe this behaviour as “greedy”. In this chapter, you have learnt how to use the three fundamental data structures in Perl, namely variables, arrays and hashes. I have also tried to give you some ideas on the concept of contexts, which is of fundamental importance in Perl. In the next chapter, I will introduce to you the operators available in Perl. Summary ” ” ” Scalar variables are prepended with the symbol $. For array and hash variables, the symbols are @ and % respectively. Scalar variables can be assigned a value by using the assignment operator =. A variable holds the value undef by default. Variable names are case-sensitive. They should start with an alphabet or an underscore character ( ). Subsequent characters may be decimal digits as well. Variables whose names not conforming to this nomenclature are predefined by Perl and serve special purposes. Variable substitution means variables embedded in a double-quoted string will be substituted by their respective values at the instant the string is evaluated. The substr() function extracts a sequence of characters from a given string. The length() function returns the number of characters in a given string. A list is an ordered set of scalar values. An array is a list associated with a name. An array may be created by assigning a list to an array variable. Nested lists are merged to form a single list when the list is evaluated, with null lists removed. The reverse() function returns a list whose items are identical to the input list except the items are arranged in reverse order. The push() function may be used to append a list of items to the end of an array. The unshift() function prepends a list of items to the beginning of an array. The subscript operator [] can be used to access a subset of elements in a list or an array. The pop() and shift() functions may be used to remove an item from an array. pop() removes the last item while shift() removes the first item. splice() is a general purpose function to add or remove array elements. join() concatenates a list of scalars into a string, inserting a string in between. map() executes a given code block for each list element, and the results evaluated are combined to form an array. Searching an array is not efficient. A hash should be used instead. A hash may be initialized by assigning it a list, whose elements are treated as a list of key-value pairs. To refer to a hash element, use curly braces with the key in between. The delete() function may be used to remove a key-value pair from a hash. You may use the exists() function to test if a given key exists in the hash. The defined() returns if a value is not undef. ” ” ” ” ” ” ” ” ” ” ” ” ” ” ” ” ” ” ”
Slide 58: 46 ” ” Chapter 3 Manipulation of Data Structures Behaviours of Perl operators and functions are influenced by context. The scalar() function forces a scalar context in an otherwise list context. The result of evaluating an array in scalar context is the number of items in the array. Evaluating a list in scalar or void context causes all list items to be evaluated, and the value of the last item is returned.
Slide 59: Chapter 4 Operators 4.1 Introduction Operators are important elements in any programming language. They are so called because they operate on data. For those who are new to computer programming the operators with which they may be most familiar are the arithmetic operators, like addition, subtraction, multiplication and division. Yet there are many more varieties of operators in programming languages. Perl provides more operators than most programming languages I could think of, but most of them fall within one of the several categories that we would go into detail in this chapter. Although devoting an entire chapter to discuss operators is rather boring, it is important for you to understand what operators are as they act as the glue to bind pieces of data into an expression. Arithmetic operators manipulate on numeric scalar data. Perl can evaluate an arithmetic expression, in a way similar to our daily-life mathematics. Assignment operators are used to assign scalar or list data to a data structure. Apart from = that you learned earlier, there are other operators, like +=, that perform additional operations at the same time. Comparison operators are used to compare two pieces of scalar data, e.g. alphabetically or numerically and returns a Boolean value. For example, you have two variables and you can use a comparison operator to deduce which one is numerically larger. Equality operators compares two pieces of scalar data and returns if their values are identical. They may be considered special cases of comparison operators. Bitwise operators provide programmers with the capability of performing bitwise calculations. Logical operators can be used to do some Boolean logic calculations. String manipulation operators manipulate on strings. There are some other operators that do not fall into the categories above. Some of them will be covered in this chapter, and the rest would be introduced as needed in subsequent chapters in this tutorial. Because Perl classifies all data into one of the two forms, namely scalar and list data, operators can be classified, similarly according to the number of operands, into two groups. Either the number of operands is fixed or variable. An operator that takes on one, two and three operands are referred to as unary, binary and ternary operators, respectively. List operators, on the other hand, can take a list of 47
Slide 60: 48 Chapter 4 Operators arguments as operands. 4.2 Description of some Operators In this section we shall study a number of operators. This is not intended to be a full coverage because it would be better for you to learn the rest later on in the tutorial as you accumulate more Perl knowledge. If you would like to have a detailed reference of the whole family of operators in Perl, please consult the perlop manpage. 4.2.1 Arithmetic Operators Arithmetic operators refer to the following operators: Operator + * / % + ++ -** Description Addition operator Subtraction operator Multiplication operator Division operator Modulus operator Positive sign Negative sign Autoincrement operator Autodecrement operator Exponentiation operator The operators +, -, *, / take two operands and return the sum, difference, product and quotient respectively. Note that the division operation is floating-point division. Unlike C, Perl does not offer builtin integral division. To get the integral quotient, you may use the int() function. For example, int(7 / 2) evaluates to 3. Perl won’t round a number to the nearest integer for you automatically. If you need this, the “correct” way to do this is $num = 7 / 2; print int($num+0.5), "\n"; The modulus operator is more problematic. The operands of this operand are both integers. If you feed any floating-point numbers (i.e. decimals) as operands they will be coerced to integers. Assume we are carrying out $a % $b. If $b is positive, the value returned is $a minus the largest integral multiple of $b such that the result is still positive. For example, 63 % 5 = 3 (63 = 12 × 5 + 3, the largest multiple in this case is 12) -63 % 5 = 2 (−63 = −13 × 5 + 2, the largest multiple in this case is −13)
Slide 61: 4.2 Description of some Operators 49 If $b is negative, the value returned is $a minus the largest integral multiple of $b such that the result is still negative. For example, 63 % -5 = -2 (63 = (−13) × (−5) + (−2), the largest multiple in this case is −13) -63 % -5 = -3 (−63 = 12 × (−5) + (−3), the largest multiple in this case is 12) Such behaviours are so tedious that most programmers simply use the modulus operator for the first case — where both operands are positive integers, which evaluates to the remainder of $a / $b. The positive/negative signs, just as in our usual mathematics, are unary operators that are affixed before a number to indicate whether it is positive or negative. Our usual convention is that the unary positive sign is not specified, because it is the default as expected. If you know C/C++, the autoincrement and the autodecrement operators are identical to what you have learned. For those who don’t know, it’s worth to spend a few minutes to repeat all the details here. These operators can be placed before or after a variable. There are four possible variations: Expression ++$var $var++ --$var $var-Description Prefix Increment Postfix Decrement Prefix Decrement Postfix Decrement The first two are autoincrement, while the remaining two are autodecrement. If autoincrement/autodecrement is performed as a statement on its own, the prefix or postfix configurations do not produce any difference. For example, both ++$a; and $a++; as standalone statements increase the value of $a by 1. However, they are different if the operators are used as part of a statement. Consider the following examples: A. B. $b = ++$a; $b = $a++; In statement A, $a is first incremented, and then the new value is returned. In statement B, however, the value is returned first, and then $a is incremented. Therefore, the value returned (and is thus assigned to $b) is the value before increment. The two forms differ in the order of increment/decrement and return of value. Autodecrement works in the same way, except the variable is decremented instead. In other words, statement A and statement B are identical in effect as the following respectively: ++$a; $b = $a; $b = $a; ++$a; # equivalent to statement A # equivalent to statement B The exponentiation operator calculates the n th power of a number. For example, 43 , i.e. 4*4*4 is expressed by 4**3, and the result is 64. Both operands can be floating point numbers. All the operands discussed in this section take on scalar values only. In other words, they create a scalar context for the operands.
Slide 62: 50 Chapter 4 Operators 4.2.2 String Manipulation Operators String manipulation operators include the following: Operator x . Description String repetition operator String concatenation operator The string concatenation operator is used to concatenate two strings. In other words, it glues two pieces of string together. For example, "hello " . "guy" results in the string “hello guy”. You can concatenate as many pieces of string as you wish by using a series of concatenation operators together, like this: $username . ", your disk quota is " . $quota . " Megabytes." In general, the concatenation operator dictates that both operands must be strings, and numeric operands would be converted to string form before concatenation (still remember that Perl does type conversions internally if necessary?). However, Perl can become highly confused about whether you are using the concatenation operator or the decimal point if both operands are numeric literals. For example: A. B. C. D. E. F. G. print print print print print print print "1"."1"; (return: “11”) "1".1; (return: “11”) 1."1"; (return: “11”) 1.1; (return: 1.1) 1 . 1; (return: “11”) 1. 1; (return: error!) 1 .1; (return: “11”) In case A, B and C, Perl thinks that the dot represents the concatenation operator because one or more operands is a string literal. Note the difference between cases D and E. Although whitespace does not influence how Perl interprets an expression in general, this is not the case as shown in this example. In case D, because the dot follows the first 1 immediately, Perl thinks that you would like to use the decimal point, and returns the numeric value 1.1. In case E, however, because there is a space before the dot, Perl thinks that you would like to use the string concatenation operator, and glues them up for you. So is case G. However, in case F, Perl thinks that you would like to supply a floating-point number like in case D, but you supply it with two numbers with no comma in between (note that 1. is identical to 1), so this is a syntax error! You get the same error if you replace case F with print 2 3;. However, because it is nonsense (although allowed) to concatenate two literals, just take this as some extra information and little attention can be paid to it. In scalar context, the string repetition operator returns the string specified by the left operand repeated the number of times specified by the right operand. For example,
Slide 63: 4.2 Description of some Operators $str = "ha" x 5; results in the string “hahahahaha” being assigned to $str. 51 In list context, if the left operand is a list in parentheses, it repeats the list the specified number of times. Examples: @array = ("a") x 3; # or even (a) x 3 will work, but not a x 3 results in the list ("a", "a", "a") being assigned to @array. @array = 3 x @array; replaces all the elements with the value 3. Note that the second operand always has a scalar numeric context. 4.2.3 Comparison Operators In Perl, there are two sets of comparison operators. The first set compares the operands numerically: Operator < > <= >= <=> Description less than greater than less than or equal to greater than or equal to general comparison The second set compares the operands stringwise: Operator lt gt le ge cmp Description less than greater than less than or equal to greater than or equal to general comparison At this point it is important to tell you one behaviour of Perl. As you know, Perl differentiates only scalar and list data. That implies that Perl is quite ignorant about whether the value of a given variable (or a scalar element of a list) is a string or a number. This is why we have two sets of comparison operators defined in Perl, in this way the programmer should choose the appropriate set of comparison operator for comparison. Because of the fact that Perl does not differentiate string and numbers much, it allows you to use a string wherever a number is required, and vice versa. However, this involves a conversion that I have to explain it here. This behaviour is not specific to comparison operators only, but a general rule that you have to bear in mind at all times when you are writing your own scripts. Consider the two statements below:
Slide 64: 52 A. B. Chapter 4 Operators print "23.1abc" + 4; print "23.1abc" . 4; The addition operator (+) requires a numeric context (i.e. it requires numeric values as operands). Therefore, Perl extracts the leading numeric portion of the double-quoted string until a non-numeric character is encountered, and converts this portion into a number (which yields 23.1 in this example). 4 is then added to this number, thus yielding 27.1 for the first statement. But what if the string started with non-numeric characters? Simply a 0 is returned. Also, if the string-to-number conversion involves a string that contains any non-numeric characters, Perl will display a warning message if you have warnings turned on. The concatenation operator (.) requires a string context. Therefore, Perl converts the second operand (4) into a string and is then appended to the end of the first operand (“23.1abc”), thus yielding the output “23.1abc4”. This is how Perl automatically converts between numbers and strings to and forth as needed. Comparing two numbers is easy, but what about two strings? How does the computer compare two strings, possibly with non-alphanumeric characters in it? Perl compares two strings by comparing the ASCII Code of each individual character in the string. As internally the computer only recognizes numbers, a way of representing characters with numbers have to be devised. ASCII representation is one of the several schemes available, and is well adopted. You can go to this website to consult the ASCII Table, which contains the characters with their associated ASCII code. ASCII codes are in the range 0 127, and there is an extended set in the range 128 - 255 which is not well supported on many systems. Don’t ask me why the characters are assigned in this way. The ASCII table was defined as such, and there’s little significance about its origin anyway. Let’s compare the two pieces of string in this example: “Urgent”, “agent” Perl compares character by character. First compare the first character of the two strings, ‘U’ and ‘a’. ‘U’ has the ASCII code of 85, while ‘a’ is 97. Because 97 is greater than 85, Perl decides that the latter character is greater, and thus “agent” is greater than “Urgent”. In the case of comparing “tooth” and “toothpicks”, the longer one prevails for obvious reasons. After you have learned how Perl handles string and numeric comparison in general, it’s time to look at how you specify the comparison with Perl. That’s easy, because you merely put the appropriate operator in between the two operands. For example, to test if “agent” is less than “Urgent” stringwise, simply specify "Urgent" lt "agent", and the result is true. Note that Perl does not have the intrinsic Boolean type, so if you pass the result directly to the print function, it is customary for Perl to output 1 for true, and “” (an empty string) for false. The best way to have the test result displayed properly is by using the conditional operator ?:. I will talk about this shortly later on in this chapter, but this is how you can do it: print "Urgent" lt "agent" ? "true" : "false"; causes the word “true” to be printed if the test is true, and “false” is printed if otherwise. Here is another example showing how numeric comparison may be used to decide on which block of code to be executed: if ($score >= 90) {
Slide 65: 4.2 Description of some Operators print "Well done. Your score is $score.\n"; } else { print "Work hard. Your score is $score.\n"; } #A #B 53 In the next chapter you will learn how to use the conditional statement like if as presented in this example. Basically, the concept is simple. If the value of $score is greater than or equals 90, statement A is printed; statement B is printed otherwise. Be aware that you may get different results if you compare two pieces of scalar data numerically or stringwise! The <=> and cmp operators can be regarded as general comparison operators. Because of the shape, they are sometimes referred to in Perl manpages as spaceship operators. <=> compares numerically while cmp compares stringwise. The action of these two operators are similar, and I shall take <=> as an illustration. The characteristics of <=> is as follows. Denote the left operand as $a and the right operand $b. If $a < $b, the result is -1. If $a > $b, the result is 1. If both operands are equal, that is, $a == $b (see below), the result is 0. This is a handy operator to quickly establish a trichotomy by determining in a single operation whether a number is greater than, equal to or less than another. Despite its power, it is seldom used in practice. It is mostly used with the sort() function to sort a list of scalars in a convenient manner. 4.2.4 Equality Operators Equality operators include the following: Operator == != eq ne Description equal (numeric comparison) not equal (numeric comparison) equal (stringwise comparison) not equal (stringwise comparison) Similar to the case for comparison operators, we have two sets of equality operators. One set for numeric comparison, the other set for strings. Equality operators can usually be regarded as part of the comparison operators, but some books may prefer to classify them into two categories. There’s actually little point to argue which approach is better, as different book authors take different views. Equality operators compares two pieces of scalar data and return a Boolean value (again, scalar value instead in Perl) that indicates if the two pieces of data are identical. The first two operators compare numerically, while the remaining two compare stringwise. For the equal operators (==, eq) they return true if the two operands are identical, false if otherwise. The inequality operators (!=, ne) have an opposite sense, they return false if the two operands are identical, true if otherwise. The equality and comparison operators we have covered so far are concluded by these four examples: A. B. ’true’ == ’false’ # true !! ’add’ gt ’Add’ # true
Slide 66: 54 C. D. Chapter 4 Operators ’adder’ gt ’add’ # true ’10’ lt ’9’ # true In example A, == requires a numeric context, thus both strings are converted into 0. Both sides are equal and evaluates to true (although you would receive a warning if -w is enabled). In example B, Perl will stop after checking the first character since ‘a’ is greater than ‘A’. Beware that in the ASCII table capital characters have smaller ASCII codes than the small letter counterparts! In example C, because the first three characters are the same and Perl cannot yet deduce whether ‘adder’ is greater than ‘add’ the longer string shall be considered greater. In example D, since we compare with lt, ‘1’ is less than ‘9’, therefore, the comparison evaluates to true. This example and example A illustrate why in Perl we need 2 sets of comparison operators. Because Perl is loosely typed and does type conversions automatically, there should be a method for Perl to know whether you would like to compare them as numbers or strings. 4.2.5 Logical Operators Logical operators include the following: Operator || or && and ! not xor Description Logical OR Logical AND Logical NOT, i.e. negation Logical XOR — Exclusive OR The logical operators performs Boolean logic arithmetic. We have seen how to do a test using the comparison and equality operators. But what if you would like to carry out two or more tests and check if all of them are true? Boolean algebra can do it rather easily. However, it is out of the scope of this tutorial somehow for me to teach you the specifics of Boolean algebra, and I would focus on how to use the Perl logical operators only. You may discover that there are two sets of OR, AND and NOT operators. ||, && and ! refer to the C-Style version. If you know C/C++, these operators would look familiar to you, and are continued to be supported in Perl. Perl also has its own set, consisting of or, and, not and xor. Note that Perl gives you an extra xor logical operator, that is not available in C/C++. The two sets differ only by precedence (which you will learn in the next section). The C-style operators have higher precedence, while the Perl operators have the lowest precedence among all the Perl operators. In the above example, we would like to see if the results of both tests are true. We can then use either logical AND operator to do it. The result would be true only if both tests are true, and false if otherwise. The logical OR operators return false only when both tests are false, and true if otherwise. The logical NOT operator toggles the truth value. For example, !(13 < 25) is false, because it inverts the truth value of 13 < 25, which is true. The exclusive or operator returns true if exactly one of the two operands is false. That is, one is true while the other is false. Here is the truth table of the logical operators we have covered so far: Here are several examples to conclude:
Slide 67: 4.2 Description of some Operators test1 true true false false test2 true false true false and && true false false false test true false not ! false true or || true true true false xor false true true false 55 Table 4.1: Truth table of various Perl logical operators (4<8) && (16<32) (return: true) (4<8) xor (16<32) (return: false) (4<8) || (16<10) (return: true) Another important behaviour I haven’t told you yet concerning logical operators is the short-circuiting property. Take and as an example. Given the expression (4 > 6) and (5 < 7) you can tell by merely looking at the first expression that the whole expression is false, regardless of the value of the second expression. Similar to the case for the or operator, if the first operand evaluates to true already, it is not necessary for Perl to examine the second expression. Therefore, Perl will just ignore that expression and do NOT even attempt to evaluate it. This behaviour is known as shortcircuiting. The logical AND as well as logical OR operators support short-circuiting. xor, for example, cannot short-circuit because its nature requires the value of both expressions be examined. Short-circuiting is significant because it eliminates several runtime errors, in particular in the following example we will not get the “division by zero” error because of short-circuiting: EXAMPLE 4.1 Safe Division 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 #!/usr/bin/perl -w # safediv.pl # Safe division which detects division-by-zero print "== Safe Division ==\n"; print "Please enter the dividend > "; chomp($x = <STDIN>); print "Please enter the divisor > "; chomp($y = <STDIN>); $quotient = ($y || undef) && $x/$y; if (!defined $quotient) { print "Division by zero!\n"; } else { print "$x / $y = $quotient\n";
Slide 68: 56 17 Chapter 4 Operators } The core of the program is on line 12, which appears to be a little bit complicated. This example makes use of short-circuiting property of && and ||. Let’s consider two cases, when $y is non-zero and zero. Consider the case if $y is non-zero. First $y is evaluated. Because it is not zero, the short-circuiting property of || causes the second operand undef not to be evaluated. Therefore, the first operand of && is $y itself. Because it is non-zero, the second operand has to be evaluated, which calculates the quotient, which is assigned to $quotient. If $y is zero, then the second operand of || has to be evaluated, which is undef. The operand of && is thus undef. Because this value is interpreted as a false value in Boolean context, the second operand of && is not evaluated and thus undef is assigned to $quotient. Subsequently, a check of whether $quotient is undef is sufficient to tell whether “division by zero” occurs. Depending on this result, either an error message or the quotient will be printed. 4.2.6 Bitwise Operators Bitwise operators refer to the following: Operator << >> & | ˆ ∼ Description Binary shift left Binary shift right Bitwise AND Bitwise OR Bitwise XOR Bitwise NOT The first two operators are the binary shift operators. The two operands of these operators must be integral. As you may know, numbers are represented in binary form internally. The left operand is the number to be operated on, while the right operand is the number of bits to be shifted. Let me explain this with the help of an example. Say you would like to perform 60 >> 2. First we convert 60 into binary notation, that is 111100 2 (subscript 2 means representation in base 2, that is, binary). In this example we intend to shift 2 bits to the right, that means the two least significant bits (the bits on the far right) are removed, and thus resulted in 11112 , which is 1510 in decimal notation, so 15 would be returned. We have used the binary shift right in the example. On the other hand, the binary shift left operator does the opposite. It shifts the integer specified left a specific number of bytes, filling the least significant bits with 0. Observant readers may notice that each binary bit shift to the left actually multiplies the number by 2. Try it. The remaining four operators compare the two integral operands bit by bit. As an example, to extract the 4 least significant bits of the number $num, we can use the expression $num & 15. The returned value is the integral expression of the last 4 bits of $num. The reason is quite simple. The binary representation of 15 is 11112 . Say $num takes the value of 37 (1001012 ). The operation is shown below:
Slide 69: 4.2 Description of some Operators 100101 & 001111 ---------------000101 57 The bitwise AND operator compares each bit. If both corresponding bits are 1, the resulting bit is 1; otherwise 0. The above example demonstrates a technique known as bit-masking, and 15 is the mask in this example. To extract specific bits, just set up a mask with the corresponding bits setting to 1, and 0 for other bits that we are not interested in. Note that although 15 is only 4-bit long, you may consider that it is extended automatically to 6 bits, with the two most significant bits set to 0. The other bitwise operators have the same semantics as their logical operator counterparts except the truth value is represented by 1 and 0 for true and false respectively. Therefore, the details of which are not repeated here. Bitwise operators are in general rarely used in most scripts. They are usually only used in applications such as cryptography or binary file access. 4.2.7 Assignment Operators Assignment operators refer to the following operators: Operator = += -= *= /= %= **= .= x= &&= ||= &= |= ˆ= <<= >>= Description Assignment operator Arithmetic manipulation with assignment String manipulation with assignment Logical manipulation with assignment Bitwise manipulation with assignment I have mentioned quite a lot about the ordinary assignment operator = in the previous chapters already. Now consider this example: $num = $num + 15; You should understand what this means, do you? The value stored in $num is added to 15, and the result is again assigned to $num. The net effect is to increment the value of the variable by 15. However, some people may prefer that manipulation and assignment be accomplished by one, instead of two operators. Therefore, Perl recognizes several shorthand notations as shown above. In general, if you can write a statement in this format: op1 = op1 operator op2; you can have a shorthand version like this: op1 operator= op2; For instance, the example above is identical to $num += 15; However, note that not all the operators mentioned above have such a shorthand version. Look at the list above for all the recognized shorthand
Slide 70: 58 Chapter 4 Operators operators. As an example, to invert all the bits in an integral scalar variable (that is, on a technical parlance, to assign the “1s complement” to the scalar variable), we have to write $num = ˜$num; 4.2.8 Other Operators Here I shall introduce to you some other operators that do not fit in any of the above main categories. The operators that would be covered here include the conditional operator and the range operator. The comma operator and => have already been described in detail in the previous chapter. Putting aside the builtin functions that may be considered operators in Perl, the remaining operators are ->, =∼ and !∼. I have chosen to defer mentioning the remaining operators because they are related to some later topics and I would cover them in those sections. The first one, the arrow operator is similar in some sense to the pointer-to-member operator in C. This would be introduced in the references chapter. The other two are used for pattern matching with regular expressions. The conditional operator ?: is a ternary operator. In other words, it has three operands. The syntax is as shown below: test-expr ? expr1 : expr2 The first operand (test-expr) is an expression. If this expression evaluates to true, expr1 will be returned by the conditional operator; otherwise, expr2 is returned. You are likely to use conditional operators quite often in the future because it is considered too “bulky” to use the if-else structure all the time. This is an example using if-else to compare the values of $a and $b, and assign the smaller value to $c: if ($a $c } else $c } < $b) { = $a; { = $b; By using the conditional operator, this 5-line structure (although no one will stop you from writing all this on one line) can be transformed into the simpler statement: $c = $a < $b ? $a : $b; The following example is extracted from the perlop manpage which demonstrates how the list context around the conditional operator propagates to expr1 and expr2. $a = $ok ? @b : @c; In this example, depending on the value of $ok, the number of elements of either @b or @c is returned and assigned to $a. First, the assignment to $a creates a scalar context around the conditional operator. Therefore, the conditional operator is expected to return a scalar value, and so the two possible values to be returned will be evaluated in scalar context.
Slide 71: 4.3 Operator Precedence and Associativity 59 Although the conditional operator is inherited from C, there is one important behaviour that is unique to Perl. The conditional operator can be assigned to if both expr1 and expr2 are legal lvalues. This is an example: ($whichvar ? $var1 : $var2) = $new_value; In the next section, “Operator Precedence and Associativity”, we would see an example on how you may obtain unexpected results involving the conditional operator because of operator precedence. The behaviour of the range operator (..) depends on the context around the operator. The range operator actually consists of two disparate operators in list context and scalar context. The range operator in list context is more frequently seen, so I will cover it first. I have introduced the range operator in list context briefly in the previous chapter, when I described how you can construct an array. The range operator in list context returns an array consisting of the values starting from the left value, with the value of each subsequent element incremented by one until the right value is reached. For example, @array[0..2] is synonymous with @array[0,1,2] to return an array slice. Note that both the left and right values would be converted to integer by chopping off the decimal portions in case they are not already integers. Not only integers can be used in the range operator, you can experiment with alphabets as well. For example, (’a’ .. ’z’) and (’A’ .. ’Z’) are two lists representing small letters and capital letters, respectively. 4.3 Operator Precedence and Associativity Now we have learned several operators, so it’s time for us to put them together. It is not uncommon that a given statement contains more than one operator. Operator precedence and associativity arise as a result. Recall that in elementary Maths class teachers teach us in a mathematical expression involving several arithmetic operators, multiplication and division shall be performed before addition and subtraction. For example, 3 + 6 × 7 is interpreted as 3 + (6 × 7), not (3 + 6) × 7. In this case, we say that multiplication and division has higher precedence than addition and subtraction operators. Because multiplication has a higher precedence than addition, the multiplication operation, involving the two operands 6 and 7, is performed first, and then the result 42 is added to 3, getting 45 as the result. In Perl, because there are many operators, the rules of precedence is far more complicated. In order to describe the relative precedence among the operators, most programming languages would use an operator precedence table to show the relative precedence of the operators, and Perl is of no exception. Table 4.2 shows the operator precedence and associativity table, which you can obtain from the perlop manpage. The operators are arranged in order of decreasing precedence. That is, the operators at the top (Terms and List Operators) have the highest precedence, while the operators at the bottom (or, xor) have the lowest precedence. Operators on the same line have the same precedence. The first column lists the associativity of the operators. Associativity is useful when there are several operators of the same precedence in a statement. In this situation, the order of evaluation of these operators depends on the associativity. If the associativity is right, then the rightmost one would be evaluated first, then the one on the left, and so on. That’s why cascaded assignment is possible. The idea is the same for left associative. You may also find that some operators are labelled “nonassoc”
Slide 72: 60 Chapter 4 Operators Associativity left left nonassoc right right left left left left nonassoc nonassoc nonassoc left left left left nonassoc right right left nonassoc right left left Operators Terms and list operators (leftward) -> ++ -** ! ∼ \ + - (unary) =∼ !∼ */%x +-. << >> named unary operators < > <= >= lt gt le ge == != <=> eq ne cmp & |ˆ && || .. ... ?: = += -= *= etc. (assignment operators) , => List operators (rightward) not and or xor Table 4.2: Operator Precedence and Associativity Table
Slide 73: 4.3 Operator Precedence and Associativity 61 (non-associative). The operators are classified as non-associative if the order of evaluation is not important, or not applicable for other reasons. In general, you don’t have to worry about the order of evaluation of the non-associative operators really much. To demonstrate the effect of operator precedence and associativity, let’s go over several scripts here. #!/usr/bin/perl -w $, = "\n"; $a = 13, $b = 25; $a += $b *= $c = 35 * 2; print "\$a == $a, \$b == $b, \$c == $c"; This one is practically easy. I would use it to illustrate how precedence and associativity works. The problematic statement is on line 5, and you shouldn’t have problems with the rest of the script, so I am focusing on this line only. To deal with this statement, first try to locate the operators concerned. The operators, by scanning from left to right, are: += *= = * Now locate the operators with the highest precedence, and it is * in this case. Therefore, multiplication would be performed first. The operands of this operator are 35 and 2, so this multiplication yields 70. The remaining are all assignment operators having the same precedence, so we look at their associativity. Because the associativity is right, the rightmost one is evaluated first, which is =. The operands are $c and 70 (the value of the expression 35 × 2), so 70 is assigned to $c. Then, $b *= 70 is evaluated, and $b is eventually assigned 1750 (70 × 25). At last, $a += 1750 causing $a to be assigned 1763. There is not much confusion here, but let’s consider another more tricky example, of which the result is not necessarily apparent at first glance. #!/usr/bin/perl -w $, = ", "; $a = 1, $b = 0; print $a >= $b ? $b : $a += 6, $a; I first saw a similar example from a C++ book, and I adapted it to become a Perl script. This is a really notorious example, because it includes ?:, which easily leads to unexpected result if you are not careful enough. I hope you could understand my explanation as this example, though seemingly short, is so notorious that it is rather difficult for me to explain it well. Now let’s have a quiz: without actually running it with your perl interpreter, try to deduce the printout of this example. Let’s give you two choices: A. B. 0, 1 6, 1
Slide 74: 62 Chapter 4 Operators (don’t look at the answer below before you have an answer in your mind) If you choose option A, I’m sorry, you’re wrong. But don’t despair, as many people share the same mistake as you do, and it is common and understandable for people to make mistakes. Although novices are more prone to make mistakes, nothing will stop veterans from making mistakes as well. Most computer programs we are using have bugs. Some bugs are obvious, but there are many more hidden ones which are not normally revealed unless your program gives unexpected output. Trust me, debugging is going to occupy most of your development time, and is the most tedious job for all programmers. If you choose option B, congratulations, you are correct. Hope that you did not get the correct answer by sheer guesswork. Anyway, let’s look at how we arrive at the answer. If you choose option A, you may have the impression that += has a high precedence, but this is not the case. In this example, ?: also exists, which has a higher precedence than +=, so ?: is evaluated before +=. Here comes the trouble. The operators, scanning from left to right, are: print >= ?: += , print is a list operator, and has the highest precedence. So Perl looks on the right hand side for its parameters. Then Perl sees >=, which has the second-highest precedence. So $a >= $b is then evaluated, yielding true. Among the remaining operators, ?: is the highest. The problem is, what are its operands? As you have learned, this operator has three operands, the so-called “test” part (that is $a >= $b, and have found to be true), the “true” part ($b), and the “false” part. But wait! What is the false part? $a or $a += 6? Its the former one, in the contrary to what you may have expected. As Perl scans from left to right to find the false part, it encounters the += operator, which has a lower precedence than ?:, and Perl stops and claims that the false part is $a. Then Perl knows what to do, so evaluates the conditional operator, returning the true part, that is $b (that holds the value 0). Anything left? Yes, we still have the dangling += 6 part! It is tricky that the conditional operator returns an lvalue if it can, and so $b += 6 is evaluated, causing $b to hold the value 6 instead. We have finished evaluating the first argument to the print operator, and the second operand is $a, which has been untouched. That’s how we arrive at the answer “6, 1”. Troublesome enough? Two long paragraphs just to explain one statement. If you are still scratching your head, try to read these two paragraphs repeatedly until you understand. So you see, operator precedence and associativity can trip you up if you are not careful. But what if this is not your intention? What if you’d like to have $a += 6 as your false part? Recall how you do this in arithmetic — adding a pair of parentheses. This applies to Perl as well. This is easily done as this: print $a >= $b ? $b : ($a += 6) , $a; In Perl, parentheses are treated as “terms”, which has the highest precedence in the precedence table. This ensures that all expressions in parentheses are evaluated before other operators do. Parentheses is the cure for those who enjoy writing complicated statements but wouldn’t like to memorize (or consult) the operator precedence table. Adding parentheses appropriately not only eliminates a great deal of effort when you or other programmers read your code at a later time, many unexpected errors could be eliminated as a result. But sometimes parentheses do not add much clarity to your code. Here’s an example:
Slide 75: 4.3 Operator Precedence and Associativity print(lc(shift(@MyArray))); 63 In this example, all but print are named unary operators. Because named unary operators have only one parameter, in general there is not much confusion if you just omit the parentheses. In conclusion, use parentheses wherever appropriate. Feel free to insert parentheses as you see fit to minimize confusion but don’t overdo it. There are several points to note arising from this example. First, although looking na¨ve to mention ı here, whitespace is NOT a determinant of the order of evaluation. You definitely cannot write the above statement in this way and expect it to work the same way the parenthesized version does: print $a >= $b ? $b :$a+=6 , $a; This is actually the same as line 5 in the example! So it returns “6, 1”, not “0, 1”. This is because when the Perl interpreter loads the script, it parses it and removes all intervening whitespace automatically. Second, you have to be careful about the use of parentheses. Consider these examples: A. B. C. D. E. print 1+2+3; print (1+2+3); print 1+(2+3); print (1+2)+3; print(1+2)+3; In the above examples, statements A–C would give the correct answer, 7, while the last two would give 3. Statements A and B have nothing special. In statement D, however, because the opening brace is placed immediately after the operator, Perl thinks that this pair of parentheses contain all the parameters to be sent to the operator. In this case, 1+2 is its only argument, so 3 is printed. Because the print operator returns nothing, thus leaving the dangling part “+3” for Perl to evaluate. Since this is a useless addition operation, a Perl warning would be displayed if you have warnings enabled. Statement E is the same as D (recall that whitespace does not matter?). For statement C, because you do not have an opening brace immediately following it, Perl does not think that there are parentheses to contain the parameters as in statement D. The parentheses in statement C is treated in the normal way, that is, evaluated first, and then the lower-precedence operators. Third, notice that in the operator precedence charts there are two entries for list operators. List operators (leftward) in this tutorial (and in the perlop manpage) refer to the name of the list operator concerned. Such a high precedence allows Perl to know where such operators occur, so that it knows where to look for their parameters. On the other hand, list operators (rightward) applies to the comma-separated expressions to pass to the list operator concerned as parameters. The extremely low precedence of this part implies that you do not normally need to put parentheses around the parameters. This part acts figuratively like a basket containing all the parameters. The only operators having lower precedence are the operators not, and, or and xor. For example, open HANDLE, "$path/.bash_history" or die "Can’t open file\n";
Slide 76: 64 Chapter 4 Operators Because or has a lower precedence than list operator (rightward), the or part is not treated as part of the operand for the open() operator. In the example, if the open operation fails, it returns undef, which is interpreted as false. Because short-circuit evaluation is not possible, the second part is executed, which outputs the error message and terminates the script. You would learn how to open files later on in this tutorial. 4.4 Constructing Your Own sort() Routine In the last chapter you have had an overview of using the sort() function to sort a list of values. I also provided you with a list of common search routines. By now you should have sufficient knowledge to understand the rest of the story. The principle is pretty simple but rather difficult to visualize. You override the default sort criteria by defining the ordering criteria in the code block of the sort() function. Two special variables $a and $b are defined in the block. In order to sort a list of values, the sort routine needs to establish the total ordering of any two arbitrary items in the list. To do so, it assigns the two items to $a and $b arbitrarily. The block should perform some comparison operations in the block based on these two values. If evaluation of the block yields a scalar value that is negative, the value held by $a comes before that of $b. If the result is positive, the value held by $b comes before that of $a. If the result is zero, then the two values should be considered equal. The <=> and cmp operators are usually used. The following is an example which illustrates numeric descending sorting. Assume the unsorted list is (3, 1, 4). To sort in descending numerical order, an appropriate expression which satisfies the above characteristics is $b <=> $a The following table illustrates how you can try to verify if the comparison works. 3 → 1 below means 3 comes earlier than 1 in the resulting list. $a 3 1 3 $b 1 4 4 $b <=> $a -1 1 1 Ordering 3→1 4→1 4→3 Table 4.3: An Example Illustrating sort() Therefore, the ordering 4 → 3 → 1 is established. Summary ” ” ” ” Operators carry out certain operations on data, known as operands. An expression consists of a combination of operators and operands. Unary, binary and ternary operators take on one, two and three operands respectively. List operators can take on a variable number of operands. In Perl, functions are simply treated as unary or list operators.
Slide 77: 4.4 Constructing Your Own sort() Routine ” ” ” ” ” ” ” ” ” Arithmetic operators carry out arithmetic calculations. Assignment operators are used to assign scalar or list value to a data structure. Comparison operators compare two pieces of scalar data and returns a Boolean value. 65 Equality operators are specializations of comparison operators which test if two pieces are considered equal. Bitwise operators can be used to perform binary arithmetic. Logical operators operate on Boolean values. String manipulation operators involve string concatenation operations. Operator precedence and associativity determines the order in which operators are evaluated in an expression. The sort() function sorts a list of scalar values. Ordering is determined by the return value of the code block provided in the sort() function.
Slide 78: 66 Chapter 4 Operators
Slide 79: Chapter 5 Conditionals, Loops & Subroutines 5.1 Breaking Up Your Code So long you have been writing your programs in one piece. You are totally allowed to carry on with this practice, however, lumping everything in one piece is often considered undesirable for the following reasons: Ease of maintain A single source file with tens of thousands of lines is for sure not easy to maintain. First, navigating around in the source file is messy — page up and page down keys are unlikely to be effective for files of this size and to locate a certain section of code much traversal is necessary. Variable Conflicts Variable management is not a trivial affair in practice when the program is a large one. It is easy for us to keep track of the variables in use for short programs we have been writing for now, but shortly when the code base grows in size, it becomes increasingly likely that variable name clash occurs at certain parts of the source code. That is, you use a variable that is defined somewhere else but you are unaware of it. This may give rise to some unexpected behaviour that are difficult to debug. Object-oriented programming is a desirable solution. We would discuss the concepts of scope and packages, which are fundamental to object-oriented programming that would be covered in the next chapter. 5.1.1 Sourcing External Files with require() The require() function can be used to source external files. It acts in this way — when execution reaches a require() statement, it first checks if the file has already been required. This is to avoid multiple inclusions and potential file inclusion loops (for example, A includes B, and B includes A etc.). The predefined variable @INC stores path prefixes the Perl file inclusion system use to find the file to be sourced. You may type the following command at the command prompt to output the content of @INC: perl -e ’print map {"$_\n"} @INC;’ On my Linux system, the list of paths is, in ascending order, cbkihong@cbkihong:∼/public_html> perl -e ’print map {"$_\n"} @INC;’ /usr/lib/perl5/5.8.0/i686-linux-thread-multi /usr/lib/perl5/5.8.0 67
Slide 80: 68 Chapter 5 Conditionals, Loops & Subroutines /usr/lib/perl5/site_perl/5.8.0/i686-linux-thread-multi /usr/lib/perl5/site_perl/5.8.0 /usr/lib/perl5/site_perl/5.6.1 /usr/lib/perl5/site_perl . There are 7 entries. The last entry is a single dot denoting the current directory. Note that the paths listed are likely to be different on each system. They are fixed in the Perl installation process. require() is an overloaded function serving three purposes: If the argument is a number, it checks if the perl interpreter version ($]) is greater than the version specified as the argument. This use generally appears in Perl modules to specify the earliest perl interpreter version with which the module can be executed; If it is a bareword, that is not enclosed in quotation marks, it is assumed to be the package name of a module with extension “.pm” (see Chapter 7) and any occurrences of ‘::’ are converted to ‘/’ on Unix-variant OSes or ‘\’ on Windows or MS-DOS. The result is a relative path to a Perl module; If the argument is a quoted string, it is treated as the relative path to the file to be included. For the last two cases, the relative path resulted is appended to each of the paths in @INC in turn to form a complete path and require() checks if a file exists at this location. The first file that is found is used. As you can see, the system modules directories are searched first, and at last the current directory is searched. If the search yields an existing file, the relative path is added to %INC as the key while the complete path formed as the value. The %INC hash variable is maintained by Perl which contains a list of files included. Whenever you invoke the require() function, it checks this hash to determine if the file specified already exists in %INC. If it does not, an entry is added to it; otherwise, the inclusion is silently bypassed. For example, say you have a source file which has the following require() statements: require ’myfile.pl’; require MyModule; The require() function will append the key-value pairs ( ’myfile.pl’ => ’/home/cbkihong/somedir/myfile.pl’, ’MyModule.pm’ => ’/home/cbkihong/somedir/MyModule.pm’, ) to %INC. Consider the situation if inside the module MyModule it also tries to require ’myfile.pl’, because an entry with the key ‘myfile.pl’ already exists in the hash, this inclusion is bypassed. Therefore, multiple file inclusions can be detected and avoided. If the file required does not return a true value (as a recapitulation — anything other than 0, an empty string or undef), require() fails with an error. Perl module authors may utilize this characteristic to abort the scripts using the module in case of errors. Therefore, you ought to see in later chapters that external files usually terminate with the statement 1; which returns the true value required.
Slide 81: 5.2 Scope and Code Blocks 69 The description above is a rewrite of the require() manpage. It is easy to understand and you are advised to read it for further information. In Chapter 7 you would learn the use() operator which is a wrapper of require() and is used to load Perl modules. 5.2 Scope and Code Blocks 5.2.1 Introduction to Associations Consider this simple assignment statement: $fruit = "apple"; How would you describe this statement? The string “apple” is assigned to the scalar variable named $fruit. Fine, this is how the statement is interpreted literally, but this description is not sufficient as you progress through the following chapters. A better and more in-depth way of expressing this statement is that a data object with the scalar value “apple” is created, and the name “fruit” is now associated with this data object. Why do we have to bother with this expression? Because up to now we have been writing programs with each data object associated with one and only one name, which is the name of the variable. However, in this chapter we would see that due to scoping rules the same name at different parts of a program can be associated with different data objects. This association is also known as a binding. In the next chapter we would introduce to you the possibility of establishing additional associations to a data object, which are references. Therefore, from now on you need to have a clear separation of the data object and its name associations. 5.2.2 Code Blocks The idea of confining the scope of associations arose from the fact that not all variables need to be valid throughout the lifetime of the program. This is particularly important in subroutines, which are self-contained reusable units containing a sequence of statements. The scope of an association refers to the region in a program in which the association is visible. Up to the present, we have been writing very simple scripts with scalar variables, arrays and hashes which, once created, could be used anywhere in the script. What we have been using in our examples so far are all global variables, in the sense that all associations are established during the compilation phase before execution, and are not destroyed until the program terminates because these variables are resolved from the symbol table, which exists as long as the program is in execution. Note that once you have created an association there is no way for you to manually destroy it. An association will only be destroyed when the scope in which it resides terminates. In this sense, a global variable has a global scope. Excessive use of global variables is considered a poor programming practice, because as your program becomes increasingly complicated, it is not impossible that your program may involve dozens to even hundreds of variables. Perhaps you may need some of them to be accessible throughout the whole script, but this is not generally the case. In particular, many variables are simply used to store data temporarily, for example, as a counter in a loop (as you will see later in this chapter). Introduction of limited scopes is seen as a cure to the problem. Scopes are defined in terms of environments. An environment is created by means of a code block, or simply, a block. A code block simply consists of a sequence of statements that makes up an environment, and is delimited by curly brackets (but the mere presence of { } doesn’t necessarily imply a code block — you will see do{} and eval{} later in this chapter that look like blocks, but they are not strictly one).
Slide 82: 70 Chapter 5 Conditionals, Loops & Subroutines Code blocks can be nested. In other words, one code block can appear in another code block. Like this: # Environment A { # Environment B { # Environment C } # Environment B resumes { # Environment D } # Environment B resumes } # Environment A resumes As you can see, after { a new environment is started. When } is reached, the current environment terminates, and the parent environment, that is, the environment that was previously in effect is reinstated. For example, when environment C terminates, environment B will resume. With my and local modifiers that we are going to explore in the next section, you can confine the scope of an association to within the extent of an environment — when the containing code block terminates, the association would be destroyed. NOTES Please interpret the above statement carefully. Although all associations are destroyed when the current environment terminates, this does not necessarily mean the objects themselves are destroyed. This is related to the garbage collection mechanism in Perl, which will be covered in the next chapter when we come to references. In particular, if additional references to the object exist, the object will not be destroyed. 5.3 Subroutines Breaking up your source code into multiple files is not the only way you can make your code more manageable. A complex program can be broken into smaller tasks, each of which carries out a well-defined function. Take an online book catalogue served in a typical library as an example, you can actually split the whole complicated program into smaller parts. For example, these are the functions that may be implemented in such a system: Search the catalogue by title, author, keywords, etc. Allows users to check their circulation record Allows users to reserve or renew items Perhaps to send a reminder to a borrower automatically if he forgets to return the borrowed items on time
Slide 83: 5.3 Subroutines 71 You should be able to think of many more, but this already exemplifies a program will be easier to write and manage if you break it up into smaller and simpler parts, with each part doing its intended task only. This shows how useful subroutines are. A subroutine consists of a sequence of statements defining an environment. Subroutines are just like user-defined functions in contrast to those provided by Perl. There are two types of subroutines — named subroutines and anonymous subroutines. Anonymous subroutines are not given a name while named subroutines are. We shall cover anonymous subroutines in Chapter 6 when we come to the topic of references. From now on, by means of subroutines we refer to named subroutines. 5.3.1 Creating and Using A Subroutine In general, before we can call a subroutine, we need to declare and define it. Declaring the subroutine makes Perl aware that a subroutine of a particular name exists. Defining means you explicitly describe what the subroutine does by listing the statements to be executed if the subroutine is being called. In general, subroutine declaration and definition go together. The syntax used to declare and define a subroutine is as follows: sub name [(prototype)] block block is the subroutine definition. It is a code block containing the statements to be executed when the subroutine is invoked. The rest is the subroutine declaration. It declares a subroutine with the name name. After the subroutine name you may insert an optional prototype part which contains a concise specification of the types of parameters to pass to the subroutine. Here is an example using a subroutine to calculate the sum of a list of scalars (presumably numeric): EXAMPLE 5.1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #!/usr/bin/perl -w sub sum (@) { # This subroutine takes a list of numbers as input # and returns the sum my $sum = 0; for my $tmp (@_) { $sum += $tmp; } return $sum; } # calculates 0 + 1 + 2 + ... + 100 and prints the value print sum 0 .. 100; # must be 5050. No doubt. Lines 3-11 contain the declaration as well as the definition of the subroutine sum(). Line 3 tells Perl that you are declaring a subroutine named sum, while lines 4-10 are the subroutine definition. Subroutines have to be declared and defined before being called in the script. If you place the print statement (line 14) before the subroutine declaration, your script simply won’t work as expected. This is because the perl interpreter reads your script sequentially. If it encounters the token sum before it is declared, Perl will not know sum is a user-defined subroutine as Perl cannot find it in the system libraries. That implies you should, in principle, always put subroutine declarations and definitions very early in source
Slide 84: 72 Chapter 5 Conditionals, Loops & Subroutines files. As you can see in the example, the subroutine is put before the source program that calls it. However, this may not be convenient sometimes. For example, when you have multiple subroutines which are inter-dependent on one another, it may not be convenient for you to find out a proper order of declaring the subroutines. Therefore, Perl allows you to make forward declarations, at which point the subroutines are declared but not defined. Forward declarations are put at the very top to declare the subroutines, and somewhere later on you give the definitions of the subroutines concerned, which can then be placed in any order. In the example, in order to make a forward declaration of the sum subroutine, it can be made like this: sub sum(@); This statement tells Perl in advance (before parsing the subroutine definition) that sum is the name of a subroutine accepting a variable number of scalars as arguments, and Perl will know to look for the definition of sum in the later part of the program. The forward declaration is the same as the declaration line except we replace the block containing the definition with a semicolon. Although you may not understand the my modifier and the for loop in the script at the moment, you may ask a question: why do I have to resort to subroutines if I can implement it directly with a for loop or a foreach loop? Well, note that the subroutine sum serves a general purpose. It adds up all the input values and return the result, regardless of the values of the input. Therefore, this subroutine is highly versatile and flexible — it can be used directly without modification on any occasion you would like to evaluate the sum of a list of scalars. If you write the script as follows, it can only be used to sum up all the integers between 0 and 100. Whenever you would like to evaluate a sum of something else, you have to rewrite the code: # calculates 0 + 1 + 2 + ... + 100 and prints the value my $sum = 0; for my $tmp (0 .. 100) { $sum += $tmp; } print $sum; By writing your code in subroutines, you are enforcing reusability of your code. Code reusability is important as we wouldn’t like to write similar pieces of code again and again. Later on you will learn how to build reusable modules where you can put your subroutines and save them as a file so that whenever you need to use the subroutines in another project, you just need to import the module, and the subroutines can be reused in your new project. This is very convenient. Here is how the sum subroutine can be readily applied to evaluate the sum in other situations: print sum values %score; $avg = sum(@nums)/@nums; # print the sum of the values of %score # evaluate the average of the values of @nums Apart from improved reusability, subroutines also help make debugging easier. Once we are certain that a subroutine is correct, we can safely apply it. It is likely that fewer number of errors would be committed when we combine the subroutines to form an entire program than writing it without using any subroutines. Unlike other languages like C or Java, Perl does not support named parameters. All incoming parameters are combined into one indistinguishable array @ . The first parameter is thus $ [0]. Therefore, if you have a mixture of arrays and scalars the elements of which would be combined into @ . You may
Slide 85: 5.3 Subroutines use pass-by-reference to be discussed in the next chapter to avoid it. 73 Also, note that the elements of @ are not copied by value. The elements of @ are “references” to the data values. They behave like references but do not look like references in terms of syntax. Basically, the idea is when you modify a value of an element in @ , the corresponding data object will be modified as well. Consider this example: sub test { $_[2] = 5; $_[3] = 6; } my @a = (1, 2, 3); my $b = 4; test(@a, $b); print join(’, ’, @a, $b), "\n"; # 1, 2, 5, 6 Here, when the third as well as the fourth element of @ are updated, the corresponding elements in @a and $b will be modified. What if we replace $b with the literal 4? Now because the data object cannot be modified, it will be an error. In principle, changing the value of parameters silently in a subroutine is a bad programming practice, although sometimes you cannot avoid it. You should document these cases clearly, for example, as comments in the source files. Because a parameter can be inadvertently modified in a subroutine, in general, you should not use the elements of @ directly in scripts. You can use the following technique: sub search { my ($myitem, @myarray) = @_; # use $myitem and @myarray thereafter # ... } my @array = (1, 2, 3); my $searchFor = 2; search($searchFor, @array); By using the my modifier, $myitem and @myarray are confined to within the search subroutine only. Here, the elements are copied by value to $myitem and @myarray. Therefore, the tie between them and the original data objects no longer exists. Even if you inadvertently modify the values or $myitem or @myarray, the changes won’t be made to the original data objects. Moreover, this emulates named parameter passing in other programming languages. A subroutine may return a list of values to the caller. As shown in Example 5.1, this is achieved by the return function. A subroutine may return a scalar or a list of scalars. Similar to the case for incoming parameters, multiple arrays or hashes are combined into one single list. Again, you may use references to circumvent this, though. 5.3.2 Prototypes Recall that while declaring a subroutine you may put an optional prototype specification. It describes the number of as well as the type of parameters in a compact form. The prototype gives Perl a clue as to
Slide 86: 74 Chapter 5 Conditionals, Loops & Subroutines how the arguments should be handled. Having an accurate grasp of the types of arguments expected is important, as illustrated in a mini case study below. The prototype specification comprises a sequence of symbols indicating the type of each argument. The symbols should look familiar to you, because they are the same symbols which are used to denote scalar variables, arrays, hashes etc. In front of each symbol you may prepend a backslash \ to indicate the element is to be passed by reference. This is covered in the next chapter. Symbol $ @ % & * Type Scalar variable Array Hash Anonymous subroutine Typeglob (Reference to Symbol Table entry) For example, if the prototype is ($$), that means the subroutine accepts two scalar variables as parameters. ($$@) implies the first two parameters to be evaluated in scalar context while the remaining parameters would be grabbed by an array variable. Note that you cannot have something like (@$) as the array variable (or hash variable alike) would take up all the input parameters. Always bear in mind that multiple parameters, after evaluating in their respective contexts, are combined together to become one indistinguishable array @ . A “programmer” who claimed to know Perl was asked by his boss to write a subroutine which inserts a list into an array at a certain position. There is already a splice() function which can do that for him, so he decided to write a wrapper which calls splice() to do the job. The boss, as a user, would like to use the subroutine in this format: insert @array, pos, list which is identical to the syntax of splice() except without the length parameter. The “programmer” wrote this: sub insert { # !!WARNING!! This does NOT work! my (@myarray, $pos, @list) = @_; return splice(@myarray, $pos, 0, @list); } Without even trying it, he handed it to his boss. The boss tried to use it in this way: @array = (1, 2, 3); insert(@array, 3, 4, 5, 6); It didn’t work, and he lost his job. Does that sound too stupid for you? Why doesn’t it work? As I have reiterated a number of times already, because all the parameters are combined into a single list when they are passed to the subroutine. You can’t really separate them back into the three parameters, because the first argument is an array which due to its “greediness” would take all the elements passed into the subroutine, leaving $pos and @list undefined. The proper way to do this is:
Slide 87: 5.3 Subroutines sub insert (\@$@) { my ($array, $pos, @list) = @_; return splice(@$array, $pos, 0, @list); } 75 It uses both pass-by-reference with a prototype added to make the types of parameters expected explicit. The use of @$array causes the original array to be modified, as we’ll cover in the next chapter. As for the prototype, we indicate the parameters are an array reference, a scalar and then a list. If a subroutine has a prototype, Perl will try to evaluate the parameters according to the prototype. Consider the case if the boss uses the subroutine in this way: @array = (1, 2, 3); insert(@array, @array, 4, 5, 6); This is identical to the previous case. Note that in the prototype the second parameter indicates a scalar is expected. Therefore, a scalar context is put around @array which causes the number of elements in @array to be passed as the second parameter, which causes the list specified to be appended to the end of the @array. You can also specify optional parameters. Compulsory parameters are separated from optional parameters by adding a semicolon in between. For example, say you have a subroutine whose declaration statement is sub mysub ($$;$$);, and you make the following subroutine call: mysub @array, "3", 9; Because @array is evaluated in scalar context, @ is the list (scalar(@array), "3", 9). The fourth parameter is empty. By using prototypes you can let Perl check parameter types and evaluate the parameters in the correct contexts. A sidenote about subroutine invocation. Traditionally, subroutines had to be prefixed with the & symbol when invoked, and the parentheses are compulsory in this case. For example, &mysub(@array, "3", 9); When a subroutine is invoked in this way, the prototype is ignored. Therefore, I recommend not to use the & form in general. A few situations where you need to use the & form will be covered in the next chapter. 5.3.3 Recursion Recursion is more of a technique rather than a feature of a programming language. It refers to the practice of tackling a problem through dividing it into smaller sub-problems and tackling them independently. Each of these sub-problems may also be subdivided if necessary. In programming languages, recursion is typically achieved through nested invocation of a subroutine, directly or indirectly. We’ll examine recursion with the help of an example. A palindrome is a sequence of characters that is identical regardless you read it in a forward or backward direction. For example, “dad” and “sees” are examples of palindromes. Here, we tackle the problem of determining whether a given string is a palindrome. For simplicity, we only consider strict palindromes that are symmetric character by character. Phrases like “Madam, I’m Adam” are generally considered palindromes, but we don’t classify them as such.
Slide 88: 76 Chapter 5 Conditionals, Loops & Subroutines There are (at least) two ways to tackle this problem, namely the iterative and recursive approach. First, we present the source program for the iterative approach: EXAMPLE 5.2 Palindrome 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 #!/usr/bin/perl -w # Determining whether a given string is a palindrome. # (Iterative approach) sub isPalindrome($); print "Enter a string > "; chomp(my $str = <STDIN>); if ($str ne ’’) { print "$str is ", isPalindrome($str)?"":"not ", "a palindrome.\n"; } else { print "The string should not be empty!\n"; } sub isPalindrome($) { my $string = $_[0]; my $ctr_l = 0; my $ctr_r = length($string)-1; while ($ctr_l <= $ctr_r) { # To do case-insensitive comparison, convert both to lowercase if ¶ applicable my $leftchar = lc substr($string, $ctr_l, 1); my $rightchar = lc substr($string, $ctr_r, 1); if ($leftchar ne $rightchar) { return 0; } else { $ctr_l++; $ctr_r--; } } return 1; } Then the recursive approach: 1 2 3 4 5 6 7 8 9 #!/usr/bin/perl -w # Determining whether a given string is a palindrome. # (Recursive approach) sub isPalindrome($); print "Enter a string > "; chomp(my $str = <STDIN>);
Slide 89: 5.3 Subroutines 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 77 if ($str ne ’’) { print "$str is ", isPalindrome($str)?"":"not ", "a palindrome.\n"; } else { print "The string should not be empty!\n"; } sub isPalindrome($) { my $string = $_[0]; # A standalone character or an empty string are by definition symmetric. # This signifies the deepest recursion stack possible. if (length($string) <= 1) { return 1; } # Here, what we need to do is to examine the first # and last character, and invoke a new isPalindrome # to deduce whether the string in the middle is a palindrome. my $leftchar = lc substr($string, 0, 1, ""); my $rightchar = lc substr($string, -1, 1, ""); return $leftchar eq $rightchar && isPalindrome($string); } The only part of concern is the subroutine definition of isPalindrome(). In the iterative approach, two pointers are maintained which initially point to the first and the last character, respectively. A loop is set up which iterates as the pointers move towards each other. When an unmatched character pair is found the value 0 is returned, which signifies the string is not a palindrome. Finally, when the positions of the left pointer and right pointer are swapped, that implies the entire string has already been scanned through and all character pairs matched (or 0 would have been returned), so we can then conclude the string is a palindrome. The logic behind the recursive scheme, however, seems to be cleaner and more intuitive. In the iterative approach, a single isPalindrome() invocation tackles the whole of the problem. However, the recursive approach suggests to break this problem into multiple levels. At each level, we merely compare the first and last character of the incoming string. The string in the middle is passed to a new invocation of isPalindrome() to deduce whether it is a palindrome. In other words, we define that a palindrome is one whose first and last character are identical and the substring in the middle is also a palindrome. If both conditions are satisfied we conclude the string is a palindrome; otherwise, it isn’t. Lines 21-23 in the recursive example handles the case when the incoming string is a single character or an empty string. In recursive schemes one always need to consider the case at which point recursion should stop. Recursion should not be allowed indefinitely (and in fact, you should avoid recursions of many levels, say possibly 500 levels deep because in practice the stack size is limited and you actually create an entry on the call stack (to be described shortly) as you recurse. Exceeding the limit results in a stack overflow error). Note that as recursion proceeds, the first and last character are being taken off of the string before passing to a new isPalindrome() invocation. Therefore, there must be a level at which the incoming string is either a single character or empty, depending on the number of characters in the original string specified by the user. Also note that the ordering of the two conditions on line 29 is significant. Because if we find that the border character pair doesn’t match, we can already claim the string is not a palindrome without having to test the string in the middle. I used the short-circuiting property of && to achieve this. If you swap the two conditions, then recursion must always have to proceed to the deepest level and the test is only carried out just before you exit from each level, which is
Slide 90: 78 Chapter 5 Conditionals, Loops & Subroutines just a waste of time. Depending on the problem nature, recursion may be a better solution compared with an iterative approach. For example, a program which searches through a directory structure (say, search for certain files on the hard disk) is nearly always implemented by a recursive scheme because directory structure is hierarchical, or in other words, nested by its very nature. Because the number of nested levels is not known in advance, an iterative scheme is unlikely to be appropriate. Recursion generally requires nested subroutine invocation. How is this possible? Subroutine invocation is internally kept track of by a series of stacks. A stack is a data structure. Think of a stack as a pile of books lying on your desk. Only two operations are allowed on a stack, either you place a book at the top of the stack, or you take away a book from the top of the stack. You are not allowed to insert into or remove from it any books in the middle. I will now briefly go into the process of subroutine invocation because certain concepts will be revisited in later chapters. The basic idea is that a call stack is maintained by your program. The top of the stack reflects your current environment. When a subroutine is invoked, a new entry is created on top of the stack to represent the new environment. Each environment holds a copy of the lexical variables (those declared with my, to be introduced later on) that are specific to its environment. When the current subroutine terminates, so is the environment and the entry is removed from the top of the call stack. The environment previously in effect is reinstated. Therefore, every invocation is independently represented by the corresponding entry on the stack. This is how subroutine invocations are handled in programming languages. Note that this mechanism does not impose any restrictions to nested subroutine invocation. Nested subroutine invocation is just performed in exactly the same way as invocation of another subroutine. 5.3.4 Creating Context-sensitive Subroutines As we have seen, several builtin functions are context-sensitive. That is, a function call exhibits different behaviours depending on the context around the function call. Subroutines may exhibit this behaviour too, with the wantarray() function. The wantarray() function determines the context around a specific subroutine invocation. Consider the following example: EXAMPLE 5.3 wantarray() 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #!/usr/bin/perl -w # wantarray.pl # Demonstrates how wantarray() may be used # Context-sensitive searching: # scalar: # search for first occurrence of substring and # return offset # list: # search for occurrence of substring in incoming list and # return matched strings sub search { my $substr = shift;
Slide 91: 5.3 Subroutines 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 79 if (wantarray()) { # remaining parameters are values being searched my @values = @_; my @retval = (); foreach (@values) { my $index = index($_, $substr); push @retval, (($index >= 0)?$_:()); } return @retval; } else { # search in incoming scalar string my $value = shift; my $index = index($value, $substr); return ($index >= 0)?$index:undef; } } my @items = (’systematic’, ’system’); my $scalar = ’delinquency’; my $search1 = search(’que’, $scalar); print "’que’ ", $search1?’’:’not ’, "found in $scalar\n"; my @search2 = search(’tic’, @items); $" = "\n"; print "’tic’ found in the following items:\n@search2\n"; In this example, search() is a context-sensitive subroutine which exhibits two different behaviours, depending on scalar or list context. On line 35, $search1 introduces a scalar context to the rvalue. Therefore, there is a scalar context around search(). Whereas on line 38, there is a list context instead. The wantarray() function returns a true value in a list context, while undef in a scalar context. Many constructs in this example are introduced at a later time in this chapter. However, the idea is pretty simple. In a list context, the subroutine accepts a variable number of parameters which are searched for the existence of the given substring. Strings with contain the given substring are returned as an array. On the other hand, in a scalar context, the subroutine accepts two parameters (you may pass additional parameters, but they are simply ignored) indicating the substring and the string being searched. The offset associated with the first occurrence of the string is returned using the index() function. I will not describe the index() function in detail in this tutorial, because its uses can be substituted by regular expressions which I will cover in Chapter 9. I would like to take this opportunity to introduce to you another predefined variable $" which appears on line 39. This is called the list separator. As illustrated in the example, substitution also applies to arrays in double-quoted strings. Therefore, you should escape all @ characters in string literals so that they are not interpreted as array variable substitution. The list separator is embedded in between list elements during array variable substitution of double-quoted strings.
Slide 92: 80 Chapter 5 Conditionals, Loops & Subroutines 5.4 Packages When you split your code into multiple files, Perl provides a nice mechanism to ensure that variables in different parts of your program do not clash. The mechanism is to divide your program into different namespaces. The idea is very simple — each namespace has a label which uniquely identifies the namespace and we prepend to variable names the label so that we can differentiate in case two variables in two namespaces happen to have the same name. C++ uses the notion of namespace, while in Perl terminology a namespace is called a package instead. Any variables not explicitly contained in any packages belong to the main package. Therefore, all variables we have been using in fact belong to the main package. By declaring additional packages we create shields so that variables in different packages would not interfere with each other. Packages are fundamental to object-oriented programming because each object is intended to be a self-contained unit. 5.4.1 Declaring a Package A package extends from the package declaration up to the end of the enclosing code block, the closing bracket of eval() (see Chapter 10) or the end-of-file, whichever comes first (see the perlmod manpage). To declare the start of a package, put package package name; Usually package declarations are placed at the beginning of source files to ensure that all variables in the file are protected. For example, #!/usr/bin/perl -w package Apple; # Package extends to the end of the file If a package declaration is placed inside a code block, the package extends to the end of the code block: #!/usr/bin/perl -w # ‘main’ package { package Apple; # package ’Apple’ extends to the end of the block $var = 3; } # ‘main’ package To avoid any misconceptions that may arise as you read on, I would like to remind you that packages appearing inside code blocks, such as in the example shown above, continue to exist after the block is closed. Always bear in mind that packages are only intended to prevent inadvertent clashing of namespace. It has nothing to do with scoping. In the above example, the variable $Apple::var still
Slide 93: 5.4 Packages 81 has the value of 3 after the containing block terminates. This is not the case for my or local variables, though, which we will come to shortly. Note that a package may be declared within the extent of another package. As illustrated in the above example, the ‘Apple’ package is declared within the extent of the ‘main’ package. Therefore, you can declare an ‘Orange’ package within the extent of the ‘Apple’ package, and each package protects the variables within its respective extent. By convention, the first letter of a package name is capitalized, except the ‘main’ package. 5.4.2 Package Variable Referencing If you omit the package name when referencing a variable, e.g. $somevar, it refers to the variable inside the current package. This also applies to variables in the main package. Therefore, by not explicitly declaring any packages in previous chapters we have been referring to variables in the current package, that is, the main package. If you need to refer to a package not contained in the current package, you need to qualify the variable with the package name prepended, with ‘::’ as the package separator. Therefore, to refer to a scalar variable $var in the apple namespace you need to write $apple::var. If the package name is empty but contains the package separator, e.g. $::var, the main package is assumed. Because you need to explicitly qualify a variable with the package name if you have to refer to it in another package, you will not modify it inadvertently unless that is your intention. That is exactly how namespaces work. There is also an old syntax of using a quote instead of double colon for referring to package variables, for example, $orange’var. This syntax may be deprecated in future versions of Perl. However, because it will be interpolated in double-quoted strings, you should beware of strings such as “$people’s pen”. You should disambiguate by putting a pair of curly braces around the variable name, such as “${people}’s pen”. 5.4.3 Package Variables and Symbol Tables Each package in Perl maintains its own symbol table. A symbol table keeps track of a list of symbols defined in the current package and their memory locations at which they can be found. For non-lexical variables (that is, those not declared with the my modifier) Perl needs to keep track of them because they are not confined to any environments. You may access the symbol table of a package through a hash whose name is the name of the package, followed by two colons. For example, the hash representing symbol table of the main package is %main::. The keys of the hash are the names of symbols defined in the package. The corresponding values is a scalar representing an internal data structure of Perl known as a typeglob which in turn holds the references to the actual symbols. To help you understand it, consider the @INC array that we described earlier in this chapter. This array is one of the predefined variables in Perl that is automatically listed in the symbol table of the main package. Therefore, there exists a key ‘INC’ in %main::, that is, $main::{’INC’}. The value is a typeglob which holds a list of references of symbols with the name INC, that is, @INC and %INC. The typeglob is represented by *main::INC. The typeglob has a number of slots, each of which stores the references of a different type such as scalar, array, hash, anonymous subroutine, filehandle and typeglob (which is just a reference to itself). If there isn’t a symbol of one type, the corresponding
Slide 94: 82 Chapter 5 Conditionals, Loops & Subroutines slot is simply null. You can access the reference of @INC and %INC through the symbol table with a so-called *FOO{THING} syntax. In this example, that are *main::INC{ARRAY} and *main::INC{HASH} respectively. Typeglobs were mainly used for parameter passing in earlier versions of Perl when references were not yet in the Perl language. In the next chapter you will be taught on how to use pass-by-reference for parameter passing. You may wonder why I need to mention typeglobs at all if it is no longer actively used. First, in order for you to understand how local or package variables work, you’ll need to know what a symbol table is. And, because the entries in a symbol table are represented by typeglobs, it is difficult for me not to mention typeglobs at all. 5.4.4 Package Constructors with BEGIN {} In each package you may define one or more subroutines with the name BEGIN, which are automatically executed when the package is being parsed. During parsing, when perl sees these subroutines, they are automatically executed in the order these subroutines are defined. That means, these subroutines are executed before the rest of the program does. Customarily, the sub keyword is omitted. That being said, these subroutines are not truly subroutines. After they have been executed, these subroutines are not registered in the symbol table. Therefore, you cannot manually invoke them during runtime. One of the major uses of BEGIN blocks is to modify @INC to include additional module search paths, especially because use is a compile-time statement (modules are loaded during compile-time as opposed to runtime — see Chapter 7). Therefore, you have to ensure that @INC gets modified before any use statements: BEGIN { @INC = (@INC, ’./libs’); } use MyLib; # .... # defined in ./libs 5.5 Lexical Binding and Dynamic Binding We have already learned how to define environments in a program by establishing code blocks in Section 5.2.2. Subroutine definition is also placed inside a code block so it also defines an environment in itself. However, I have not yet explained how you can restrict an association to within a certain environment. Recall that all variables we have been using are package variables. Once declared, package variables continue to exist in the symbol table as long as the program is running. Before we go into the details of the two types of bindings with respect to associations, let us first examine the general concept of referencing first. Let’s execute this program on your system: 1 2 3 4 5 for (keys %main::) { print $_, " => " , $main::{$_}, "\n"; } $abc = 3;
Slide 95: 5.5 Lexical Binding and Dynamic Binding The generated lines that are of interest at this point are shown below, with others omitted: ... stdin => *main::stdin ARGV => *main::ARGV INC => *main::INC ENV => *main::ENV abc => *main::abc ... 83 The above code dumps the content of the symbol table. On the left of the arrow are the names of the symbols, while on the right are the corresponding typeglobs. What appears to be interesting is that an entry for abc exists in the symbol table, regardless of the fact that the statement which assigns 3 to $abc has not yet been executed at the instant the symbol table dump is made. The reason is that a compilation step, despite invisible to users, was performed before the actual execution which scans the whole program for package variables which are then added to the symbol table. Therefore, before the program is actually executed the symbol table has already been constructed. As noted previously, a symbol table keeps track of the symbols that appear in the program. By doing so, the runtime environment (for example, the perl interpreter) prepares a list of symbols at an early point in time which facilitates it to arrange for storage space in the memory and, most importantly, to prepare for referencing operations that occur during execution of the program. Whenever a symbol appears in a program, a referencing operation is required to be carried out during execution to deduce which data object is associated with the given symbol. For example, when the statement $abc = 3; in the program above was executed, the runtime environment needs to find out which data object is associated with the scalar variable $abc. The goal of a referencing operation is to locate this association. Referencing has been made complicated because of the presence of scopes. Without scopes, referencing is easy because there can be only one variable of a certain name in each package. For example, throughout the duration of a program there can be one and only one scalar variable $Apple::var in the Apple package. $Orange::var is already a different scalar variable and do not interfere with $Apple::var at all. Therefore, throughout the program all references to $Apple::var always refer to the same data object. However, this may not apply to those variables which are confined by scoping rules. In Perl, two major types of scoping rules are supported, namely lexical scoping and dynamic scoping. Both scoping systems base on definition of environments such as code blocks, but the way referencing is performed is different. Most modern programming languages only support lexical scoping. An association that is lexically scoped is visible from the point in the environment in which it is defined, and all environments that appear inside the extents of that environment, until when another lexically scoped variable with the same name appears in those environments. In Perl, a variable declared with the my modifier is lexically scoped. Consider this example: 1 2 3 4 5 6 7 my $a = "Hello "; { $a .= "World\n"; print $a; # Hello World my $a = "Bye!\n"; print $a; # Bye! }
Slide 96: 84 8 Chapter 5 Conditionals, Loops & Subroutines # Hello World print $a; When execution proceeds to line 3, Perl needs to find out which data object $a refers to. At this instant, the current environment, that is the code block between line 2 and 5, is known as the local referencing environment. Perl first finds out that up to this point there is not any lexical variable with the name a in the local referencing environment. As local referencing fails, the referencing operation proceeds to search for one in the nonlocal referencing environments, by proceeding all the way up through parent environments. Here, we find a lexical $a in the parent environment, and the data object associated with that variable is used. Therefore, the string “World\n” is appended to the scalar value held by that data object. However, on line 5 a new lexical $a is declared at this point. Therefore, referencing operation performed at line 6 resolves to this lexical. Note that the lexical in the parent environment is untouched, but cannot be accessed anymore in the current environment. It is described as being hidden. When the code block terminates, all local associations are destroyed. Because of the reference-based garbage collection mechanism, the data object associated with the lexical on line 5 is also destroyed. The parent environment that is once invisible springs into existence again, and lexicals that were once hidden are visible again. Therefore, the last print() outputs “Hello World”. my expects a scalar or a list as its argument. We have seen a scalar used as the argument in the examples. Using a list as an argument with optional assignment looks like this: my ($a, $b) = (’Hello’, ’World’); If the variable list is not assigned, then they are given the values of undef. Lexical scoping models are recommended for several reasons. First, the use of lexical variables is faster compared with dynamically scoped ones. That is because lexical scoping can be solely determined from the nesting of code blocks, which is already fixed during the compilation phase. Therefore, referencing of lexical variables can be performed during compilation instead of at runtime. Dynamic scoping, on the other hand, also takes into account the dynamic factor of subroutine invocations. In Perl, local variables are dynamically scoped. As the use of dynamically scoped variables share similar problems as global variables in other programming languages, and the scope is dependent on the call stack which is determined by how the program calls subroutines at runtime, they make debugging more difficult, and are relatively slower. my variables are never listed in the symbol table. This fact is important as you go on and learn how to use typeglobs. Dynamic scoping, on the other hand, is based on the call stack instead of nesting of environments. In Perl, local variables are dynamically scoped. These variables are package variables and appear in the symbol table of the respective packages. The idea is, when a local variable is declared, the current value as can be accessed through the symbol table is saved temporarily in a hidden stack, and a new data object is created to hold the new value. When the current environment terminates, the current symbol table entry is removed, and the value that was previously saved is reinstated. Consider this example: 1 2 3 4 5 sub greeting { print $a; } $a = "Hello "; # Bye!
Slide 97: 5.5 Lexical Binding and Dynamic Binding 6 7 8 9 10 11 12 85 { $a .= "World\n"; print $a; # Hello World local $a = "Bye!\n"; &greeting; } print $a; # Hello World This is similar to the example I used above to introduce my variables in Perl, except now local is used and the second print is put in a subroutine. On line 9, the original symbol table entry for $a (with the scalar value “Hello World”) is replaced with a newly created data object whose value is “Bye!”. In the subroutine greeting(), because $a cannot be resolved in the local environment, the symbol table entry is used, and therefore “Bye!” is displayed. When the block terminates, the original symbol table entry saved is reinstated. Therefore, the value printed is “Hello World”. There are a few points to note here. Here is a slightly modified version of the above program for illustration: 1 2 3 4 5 6 7 8 9 10 11 12 13 my $a = "Hello "; sub greeting { print $a; } { $a .= "World\n"; print $a; local $::a = "Bye!\n"; &greeting; } print $a; When you run this program, you would find that all three print() result in the string “Hello World” being displayed. The reason is that during compilation phase all references to $a within the lexical scope have already been associated with the lexical variable. If you swap the positions of the lexical variable declaration and the subroutine, you will find that the localized package variable $a is printed in this case. Also, as shown on line 10, when you try to localize a variable when a lexical variable of the same name exists, you have to explicitly use the double-colon form to indicate the package symbol table entry. That is because, as explained, $a is tied to the lexical variable and trying to localize a lexical variable is a runtime error. Apart from a package variable, you can localize a member of composite type. For example, you can have # Hello World!!!! local $ENV{’PATH’} = ’/home/cbkihong/bin’; which causes the original value to be saved and temporarily replaced by the new given value. When the environment terminates the original value is restored.
Slide 98: 86 Chapter 5 Conditionals, Loops & Subroutines Because of potential confusions that may arise when you use local variables, one is generally not recommended to use local variables. 5.6 Conditionals A programming language is practically not useful if the statements are only allowed to run from the very first line to the last. Therefore, in this section we are going to talk about loops and conditionals. You have used the comparison and logical operators in the previous chapter. By using conditionals, you can specify a block of code to be executed if a particular condition (test) is satisfied. This is what conditionals exactly do. The if-elsif-else structure is the most basic conditional structure. The general form is: if (EXPR1) BLOCK 1 [elsif (EXPR2) BLOCK 2] ... [else BLOCK n] The parts in square brackets denote the optional parts. The if-elsif-else structure works as follows: if EXPR1 evaluates to true, statements in BLOCK 1 are executed, and the remaining elsif or else parts are bypassed. Otherwise, Perl jumps to the next elsif or else part, if any. Perl goes to the elsif part if the previous condition is not met (i.e. false). If EXPR2 evaluates to true, BLOCK 2 is executed and the remaining parts are bypassed. There can be as many elsif parts as you like, and Perl will test each condition successively until any test evaluates to true. The else part is placed at last, handling the situation when all the previous tests failed. The BLOCK n will be executed in this situation. Figure 5.1 presents the flowchart showing the sequence of actions performed inside an if-elsif-else conditional structure. The following is a simple program in which the user inputs a number, and the program deduces whether it is an even number, odd number or zero. EXAMPLE 5.4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 #!/usr/bin/perl -w $numtype = ""; print "Please enter an integer > "; chomp($num = <STDIN>); if ($num % 2 == 1) { $numtype = "an odd number."; } elsif ($num == 0) { $numtype = "zero."; } else { $numtype = "an even number."; }
Slide 99: 5.6 Conditionals 87 Figure 5.1: If-elsif-else Flowchart 15 16 print $num . " is " . $numtype . "\n"; This program has all the three parts, forming a complete if-elsif-else structure. Because 0 is customarily considered neither odd nor even, we have taken this special case into consideration. First it tests if the number is odd by checking if the modulus (remainder) is 1. If this is true, line 9 would be executed. Otherwise, it jumps to line 10 to test if the number is 0. If this test fails again, we know for sure that it should be an even number. Perl also has an unless conditional structure. The following example illustrates its use: 1 2 3 4 5 6 7 8 9 #!/usr/bin/perl -w print "Please enter your age > "; chomp($in = <STDIN>); unless ($in < 18) { print "You are an adult.\n"; } else { print "You are less than 18 years old.\n"; } If you use unless, the sense of the test is reversed. Line 6 is executed if the expression evaluates to false. If the expression evaluates to true, Perl executes the else part. In fact, the unless structure is somehow redundant. However, Perl gives you the flexibility to do your job in alternative ways. That is an exemplification of the Perl motto “There Is More Than One To Do It”. You can replace line 5 with if (!($in < 18)) { ... or even
Slide 100: 88 Chapter 5 Conditionals, Loops & Subroutines if ($in >= 18) { to achieve the same effect. 5.7 Loops Sometimes we would like to have a mechanism to execute a sequence of statements repeatedly for a specific number of times or under a particular condition. A loop is the answer. First, I will introduce the for loop. 5.7.1 for loop The for loop is inherited from C/C++. The general syntax is for ([init-expr]; [cond-expr]; [loop-expr]) BLOCK First, the initial expression init-expr is executed. In this part usually a variable would be defined that acts as a counter to keep track of the number of times executed. Then the conditional expression cond-expr is evaluated. If the expression evaluates to anything other than undef, empty string (“”) or the numeric 0 (i.e. the three scalar values that are defined as false), the BLOCK is executed. After the BLOCK has been executed, the loop-expr is evaluated. Then, a new cycle starts, and the cond-expr is evaluated again until the cond-expr evaluates to false, then the loop terminates. The process described above could best be visualized using a block diagram as shown in Figure 5.2. Figure 5.2: for-loop Flowchart We would now write a script that prints a special pattern on the screen. It consists of two isosceles triangles pointing vertically towards each other. EXAMPLE 5.5 Double Triangles 1 #!/usr/bin/perl -w
Slide 101: 5.7 Loops 2 3 4 5 6 7 8 9 10 11 12 13 14 89 print "Please input the width of the base (1-50) > "; chomp($input = <STDIN>); if ($input < 1 or $input > 50) { die "Input must be in the range (1..50)!\n"; } for ($trend = 0, $i = $input; $i <= $input; ($trend)?($i+=2):($i-=2)) { if ($i == 1 or $i == 2) { $trend = 1; } print " " x (($input - $i)/2) . "*" x $i . "\n"; } In the example, lines 5-7 handles the case if the user enters a number out of the range (1..50). In particular, the die() operator on line 6 outputs the error message specified to the standard error (STDERR), which is the screen by default, and then terminate the program. On lines 9-14 I used the for loop to print the asterisks line by line. The $i variable stores the number of asterisks to be printed at each cycle, while $trend keeps track of whether you decrease or increase the number of asterisks by 2 after each loop. Initially 0 is assigned to $trend, so that the conditional operator decrements $i by 2 after each loop. The upper triangle is completed when $i takes the value of 1 or 2, depending on whether the user has entered an odd or even number. At this point I assign 1 instead, so that in the next loop $i is incremented by 2. This loop is repeated until when $i is greater than the width of the base specified. In this case, the loop stops. Note that although line 9 looks a bit complicated and strange, this is grammatically correct. The init-expr part consists of two expressions separated with a comma operator. As you have learned from the previous chapter, both expressions would be evaluated while returning the value of the last expression. However, this return value is actually ignored in this case. 5.7.2 while loop A for loop is not the only type of loop structure available. Another form of the loop structure I would like to mention is the while loop. This structure is simpler compared with for loop, and the syntax of which is as follows: while (cond-expr) BLOCK How does it work? First, cond-expr is evaluated. If it evaluates to true, BLOCK is executed. After that cond-expr is tested again, and the loop just goes on indefinitely until cond-expr evaluates to false. 5.7.3 foreach loop Now I would introduce to you another loop structure that works closely with the list data structure. The general syntax of a foreach loop is as follows: foreach [[my] $loop var] ( list ) BLOCK
Slide 102: 90 Chapter 5 Conditionals, Loops & Subroutines In every cycle of a foreach loop, an element from the specified array or list (list) is retrieved and assigned to a temporary local variable $loop var, and BLOCK is executed. Looping continues until all the elements in list have been enumerated. For example, if we would like to check if a particular element exists in an array, we can use a foreach loop and iteratively checks if the returned element matches the data we are looking for, as in the example below: 1 2 3 4 5 6 7 8 9 10 11 12 13 #!/usr/bin/perl -w $searchfor = "Schubert"; @composers = ("Mozart", "Tchaikovsky", "Beethoven", "Dvorak", "Bach", "Handel", "Haydn", "Brahms", "Schubert", "Chopin"); $prompt = "$searchfor is not found!\n"; foreach $name (@composers) { if ($name eq $searchfor) { $prompt = "$searchfor is found!\n"; last; } } print $prompt; As mentioned previously, we should have used a hash in the first place, but we use a loop to demonstrate the use of foreach loop anyway. In this example, each of the names in @composers is compared with “Schubert” in turn, the name we are looking for. The loop keeps on going unless the name specified is found in the list, or all the elements have been exhausted without resulting in a match. It is apparent that “Schubert” is in the array, so we must always obtain a positive result. In each foreach cycle, an element from the specified list or array is assigned to the scalar variable specified. If the variable is omitted (note that $loop var is an optional argument), it defaults to $ . This is a special variable that Perl, in general, assigns temporary data to if no scalar variable is specified in certain operations. This special variable is used quite often in later chapters, like regular expressions, to shorten the length of the script. However, in my opinion, although you are allowed to make your scripts shorter by omitting specifying certain variables in Perl, it may cause your script to look more cryptic than necessary. However, many Perl programmers use such shorthands, and you should know how to interpret them, and this is the reason why I cover this here. The variable $loop var is in the form of a local variable. However, you may want to restrict it to be lexical scoped. In this case, put my after foreach, as shown in the syntax above. Some Perl programmers are lazy to type 7 characters for foreach so you may use the shorthand for instead. Perl can differentiate whether you use the for loop or foreach loop from the syntax. Also notice line 10. The last statement is one of the loop control statements. A loop control statement controls the execution of the loop. In the next subsection we would explore the loop control statements available in Perl. You may attach an optional continue block after a loop block, such as while or foreach, which will be executed before starting another iteration to evaluate the conditional expression. This is similar in purpose to loop-expr in the for loop, except this is a block instead of an expression. This is handy if you have to preform a number of operations before starting a new iteration that is otherwise clumsy or cannot be represented by an expression. Here is the double triangle example earlier rewritten with a continue block:
Slide 103: 5.8 Leftovers 91 EXAMPLE 5.6 Double Triangles (with continue block) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 #!/usr/bin/perl -w print "Please input the width of the base (1-50) > "; chomp(my $input = <STDIN>); if ($input < 1 or $input > 50) { die "Input must be in the range (1..50)!\n"; } { my ($trend, $i) = (0, $input); while ($i <= $input) { if ($i == 1 or $i == 2) { $trend = 1; } print " " x (($input - $i)/2) . "*" x $i . "\n"; } continue { $i += ($trend? 2 : -2); } } 5.7.4 Loop Control Statements Loop control statements can only be used inside loops to control the flow of execution. The next statement causes the rest of the code block to be bypassed and starts the next loop iteration. If a continue block is present, it is also executed before starting another iteration. For for loops, loop-expr is evaluated, and then cond-expr is evaluated; For while loops, cond-expr is evaluated; For foreach loops, the next element is taken from list. The last statement causes the rest of the code block to be bypassed and the loop then terminates. Execution starts at the statement immediately following the BLOCK. Any continue block is also bypassed. 5.8 Leftovers Before we end this chapter, we will look at some miscellaneous issues which have not yet been covered. A do block contains a sequence of statements which are sequentially executed, while returning the value resulting from the last statement. A do block is usually used with a modifier. Perl has defined a number of modifiers which can be attached to the end of a statement. As an example: $i = 0; do {
Slide 104: 92 Chapter 5 Conditionals, Loops & Subroutines print $i++, "\n"; } while ($i <= 30); which prints all the integers from 0 up to 30. If a modifier is attached to a do block, the block is executed once before evaluating the condition. If the condition is true, then the block is executed again. Otherwise, execution resumes on the next statement. The possible modifiers are: if EXPR unless EXPR while EXPR until EXPR foreach EXPR # same as "if (!EXPR)" # same as "while (!EXPR)" If a modifier is not attached to a do block, as in the following example, the expression in the modifier is evaluated before executing the rest of the statement: $i = 0; print $i++, "\n" while ($i <= 30); This is actually equivalent to # prints 0 to 30 $i = 0; while ($i <= 30) { print $i++, "\n"; } Summary ” ” ” ” ” ” The require() function is used to source in external files. The @INC array stores path prefixes Perl uses to find an external source file. An association exists between a variable name and the underlying data object. The scope of an association refers to the region in which the association is visible. Scopes are determined by nesting of environments. An environment is established inside a pair of curly braces. A subroutine consists of a sequence of statements defining an environment, which may accept a list of scalars as parameters and return a list of values to the caller. • • • • • ” Subroutines enforce reusability of code snippets and make debugging easier. Parameters are merged into a single list and passed into a subroutine which may be retrieved from the array @ . A subroutine may return a scalar value or a list of values to the caller by using the return function. Prototypes allow type checking of subroutine invocations at compile time. In a subroutine you may determine the context around the subroutine invocation by using the wantarray() function. Recursion refers to the technique of tackling a problem by defining the solution in terms of itself. In programming languages this is achieved by nested subroutine invocation.
Slide 105: 5.8 Leftovers ” Package, or namespace, is a mechanism to minimize variable name clashes. • • • 93 Variables of the same name may exist concurrently in each package. Variables not within the extents of any packages belong to the main package. Package variables are referenced by qualifying the variable name with the package name, followed by ::. * Defaults to the main package if :: is present but the package name is not specified. * Defaults to the current package in effect if both the package name and :: are absent. Package variables are resolved from the symbol table. Each package may have multiple BEGIN blocks which are executed at compile time. • • ” An association that is lexically scoped is visible from the point in the environment where it is established and all environments that exist within the extents of the current environment, until the end of the current environment. my variables are lexically scoped. The scope of an association that is dynamically scoped depends on the call stack. A dynamically scoped association exists as long as the current environment is not destroyed. local variables are dynamically scoped. ” Web Links 5.1 Coping with Scoping http://perl.plover.com/FAQs/Namespaces.html Questions A. In Example 3.7 I presented a binary search program to check if a user-specified integer exists in the list of 100 randomly-generated integers. Introduce the following changes to the program: convert it from iterative to recursive approach; only the search routine is sufficient. You are not required to write the parts which accept user input and generate integers in random; use lexical variables wherever appropriate; the search routine should be placed inside a package with name Search, in an external file “binarysearch recursive.pl”; There should be at least a binarySearch() subroutine which is in the Search package (namespace), which can be invoked as follows: require ’binarysearch_recursive.pl’; my $exists = Search::binarySearch(@array, $integer); where @array is any array containing a list of integers, $integer is the integer to search for, and $exists is 1 if the integer is found, undef otherwise. You may save the number of comparisons taken to reach a conclusion as a package variable $numSteps in the Search package that can be accessed directly or through a subroutine.
Slide 106: 94 Chapter 5 Conditionals, Loops & Subroutines
Slide 107: Chapter 6 References 6.1 Introduction We have covered the use of scalars in the first few chapters of this tutorial. However, we haven’t covered a very important type of scalar yet — references. Perl references are prevalently used nowadays, thanks to the official support of object-oriented programming starting from Perl 5. You need to have a good command of references before heading towards the topic of object-oriented programming. References look cryptic from the start. It resembles the concept of “pointers” in C/C++, and “hard links” in Unix filesystems (you may flip to Appendix D for a crash course on basic Unix). The use of references is in some sense a peculiar concept in programming. However, it turns out to be very useful in parameter passing and serves as the basis for constructing complex data structures in Perl (and therefore, it is central to object-oriented programming in Perl). Towards the end you will also learn to use typeglobs in your programs. Typeglobs were used to pass data structures to and from subroutines when references were not yet available in earlier versions of Perl. Today, they have been largely superseded by references, and the section on typeglobs is merely presented for your information only. A reference is a special form of scalar variable that stores the location (address) of a data structure. Fundamentally, when something needs to be stored in the memory, a memory address is required. It is only with the knowledge of the address of the object that we can access it. This is synonymous with your postal address in mail delivery, or your IP address for packet routing on the Internet. 6.2 Creating a Reference To create a reference, prefix the \ operator to the data object. For example, $a = \100; This creates a reference variable $a that points to a newly created data object which holds the literal 100. If the memory address of the data object is stored is at location 0x8101B8C, the reference $a would have an rvalue of 0x8101B8C. The following diagram is a pictorial representation of the situation. Note that the memory address varies from system to system, and because of the relocatable nature of processes, the address may probably vary on every execution. I just made it up here for the purpose of illustration. 95
Slide 108: 96 Chapter 6 References Figure 6.1: Scalar Reference In the diagram, the association is represented by a solid arrow. The real data object is represented by a cloud shape while the reference by a rectangle. You can further create a reference $b which points to $a, by $b = \$a; # scalar reference Figure 6.2: A Chain of Scalar References Apart from references of scalars, you can also create references of hashes, arrays, subroutines and typeglobs. Subroutine reference (or code reference/anonymous subroutines) and typeglob reference will be revisited afterwards. $arrayref = \@array; $hashref = \%hash; $coderef = \&subroutine; $globref = \*typeglob; # # # # array reference hash reference subroutine reference typeglob reference Though appearing awkward to do so, you may create a chain of references in one go without involving intermediate variables by prefixing multiple backslashes to the data object. For example, the above two statements are functionally equivalent to: $b = \\100; Be careful of enumerated lists as a special case! As indicated on the perlref manpage, taking a reference of an enumerated list evaluates to a list of references of the list elements: @list = \($a, $b, $c); # Actually (\$a, \$b, \$c) @list2 = \($a, @b); # Actually (\$a, \$b[0], \$b[1], ...) # Actually \$c. Remember list operator in scalar context? $scalarref = \($a, $b, $c); Recall that in an enumerated list all arrays or hashes are expanded to the constituent elements to form a single list. To create a reference to an enumerated list directly without creating an intermediate array first, which is called an anonymous array, enclose the list in [ ] instead:
Slide 109: 6.3 Using References $listref = [$a, $b, $c]; Similarly, to construct an anonymous hash, enclose the key-value pairs in { } instead: 97 $hashref = ’key1’ ’key2’ ’key3’ }; { => ’value1’, => ’value2’, => ’value3’, Because a reference is simply a scalar value, elements of an anonymous array (or values of anonymous hash) can also be a hash reference or array reference. In this way, we will be able to implement some complex data structures easily, which we shall explore later in this chapter as well as in the next chapter when I introduce object-oriented programming. For example, $hashref = { ’values’ => [ ’a’, ’b’, ’c’, ], ’device’ => ’screen’, ’options’ => { ’indent’ => TRUE, ’color’ => ’0xFFFF00’, }, }; which creates an anonymous hash containing three key-value pairs. The values key maps to an anonymous array, while the options key maps to an anonymous hash. The device key maps to a simple scalar string. References to subroutines behave like functors in C. They can be created in two ways. The first way is to prepend \& to the name of the subroutine. The & is required when you are referring to the name of a subroutine. For example, if you have a subroutine named somesub() you can take a reference to it by \&somesub. You can also create an anonymous subroutine directly, by using sub without specifying the name of the subroutine: $subref = sub { # This is a subroutine }; 6.3 Using References If you try to print() a reference variable, the type as well as the memory location will be displayed, for example CODE(0x814f3a0) for an anonymous subroutine.
Slide 110: 98 Chapter 6 References Once you have the reference variables, you may dereference them to access the underlying data objects. To dereference a reference variable, simply put a pair of curly braces around the reference variable and prepend it with the symbol which stands for the underlying type. For example, to dereference the reference variable $a in our earlier examples, we can write ${$a}. Other examples: @array = @{$arrayref}; $scalar = ${$arrayref}[0]; %hash = %{$hashref}; $scalar = ${$hashref}{’KEY’}; &{$coderef}(’a’, ’b’); # # # # # array Return the first element of array above hash Return the value whose key is ’KEY’ of hash above subroutine invocation In general, you can omit the curly braces around the reference variables in dereferencing operations. Situations that require them will be described at a later time. You can also dereference multiple levels deep. For example, consider the chain of references $b we saw at the beginning of this chapter, we can access the data object which holds the literal 100 by dereferencing it two times, by $a = \100; $b = \$a; print $$$b; # prints 100 However, creating a reference to a literal makes it read-only. For example, trying to modify its value in dereferencing is a runtime error: $a = \100; $b = \$a; $$$b = 90; # same as $$a = 90 Modification of a read-only value attempted at test.pl line 3. Because the value pointed to by $a is read-only, we can only change it by creating a new literal of the desired value, and point $a to it instead. $b would now reflect the new value when dereferenced: print "Before reference \$a changes: $$$b\n"; $a = \200; print "After reference \$a changes: $$$b\n"; The output is Before reference $a changes: 100 After reference $a changes: 200 This is different from the case below, which you may change the underlying value because because the reference is taken on a scalar variable, not a literal:
Slide 111: 6.3 Using References 99 Figure 6.3: Change in Reference Chain $a = 100; $b = \$a; $$b = 90; Note that subroutine prototypes are ignored when you invoke a subroutine through its reference, because prototypes are only used during compile time, at which point perl cannot yet resolve the subroutine to which the reference variable points. Recall that to dereference a reference variable we put a pair of curly braces around it and prepend a backslash to it. I then told you that the braces are customarily omitted. The fact is, the curly braces may contain anything, provided it returns a reference matching the expected type. For example, ${ $$ref{’KEY’} } dereferences the hash reference $ref, take the value associated with key KEY which is presumably a scalar reference, and dereference it. Note that this is not the same as $$$ref{’KEY’}, which is actually identical to ${${$ref}}{’KEY’} # $$ref->{’KEY’} (see below) which dereferences the scalar reference $ref to get the underlying hash reference, dereference it and then access the value associated with the key KEY. In other words, when the curly braces are omitted, access of array or hash is performed at last. As the reference chain gets more complicated the dereferencing expression can get very confusing. Therefore, Perl also supports a special syntax resembling the C pointer-to-member (arrow) operator. Table 6.1 summarizes the alternative forms. Operation $$arrayref[$index] $$hashref{$key} &$coderef(@args) Alternative Method $arrayref->[$index] $hashref->{$key} $coderef->(@args) Table 6.1: Alternative Syntax for Dereferencing The arrow operator can be cascaded, for example, $hashref->{’files’}->[0] = ’index.html’;
Slide 112: 100 Chapter 6 References presumably $hashref is a hash reference, and $hashref->{’files’} yields an array reference. However, Perl is smart in that by executing the above statement it will automatically create all the necessary data structures and references to fulfill this statement if they do not yet exist. This is known as autovivification. In other words, Perl essentially executes the following statement when $hashref doesn’t exist: $hashref = { ’files’ => [ ’index.html’, ], }; 6.4 Pass By Reference Recall that Perl combines the input parameters into a single list. Similarly is the case when a subroutine returns a list of scalars to the caller. Therefore, you cannot pass multiple lists to a subroutine as you cannot separate them into respective lists. References are useful in this regard because, no matter how complicated the underlying data object is, they are simply represented by a scalar, and you can pass scalars easily as usual. Moreover, passing by reference — even if you copy it to another variable — allows you to access and possibly modify the underlying data object. Pass by reference is efficient because only a scalar value is involved in parameter passing. When used with subroutine prototypes, subroutine invocation can never be more flexible than ever. Passing a reference is easy. Simply take the reference, and pass it as a parameter and get it back from @ in the subroutine. With prototypes (and it is not invoked using the & form) the caller does not even need to take the reference himself/herself if the corresponding symbol in the prototype is prefixed with the backslash \ symbol. The following example demonstrates both methods: sub passArgs (\%$) { my ($hashref1, $hashref2) = @_; # Both variables are now hash references } %myhash = (’key1’ => ’value1’, ’key2’ => ’value2’); passArgs(%myhash, \%myhash); However, because prototypes may not be observed, depending on the way the subroutine is invoked, you may wish to require the caller to take the reference and pass the reference instead of relying on prototypes, which is in my opinion the safest way to go at present. Perl does not support named parameter passing, but some programmers may also wish to emulate it as follows, which is recommended if you have to pass a number of parameters to a subroutine. sub somesub (%) { my %params = shift; # All parameters now in %params. my $filename = $params{’FILE’}; my @args = @{$params{’ARGS’}}; }
Slide 113: 6.5 How Everything Fits Together 101 somesub( ’FILE’ => ’/bin/ls’, ’ARGS’ => [ ’-l’, ’-R’, ’/home/cbkihong’, ], ); The advantage is you don’t need to remember and follow a predetermined ordering of parameters because there is no ordering in a hash. 6.5 How Everything Fits Together Here, let us consolidate the concepts you have learned in this and the previous chapter. This is also an appropriate opportunity to form a more complete picture as to how they all fit together. By now you should already have a clear separation of the names and their underlying data objects. Data objects exist independently from variable names as seen in programs. They are linked together by an association, or in other words, a binding. Bindings of lexical variables are determined and fixed in the compilation phase, before a program is even executed. Which data object a lexical variable is bound to is solely determined by the nesting of environments. Bindings of dynamic variables, including those declared with the local modifier are resolved at runtime from the symbol table, and are dependent on the call stack during program execution. Creating references to a data object actually implies establishing additional access paths to it. In general, you can access a data object in two ways. The first way is to access it through a variable which is visible and is bound to the data object. The second way is to access it through dereferencing a reference variable which points to the data object. Perl maintains the number of associations to every data object to determine when they should be freed. Simply put, Perl would destroy a data object when the number of associations to a data object drops to 0. This indicates it cannot be accessed from within the program anymore, and can therefore be safely destroyed. Consider this example: sub createArray () { my @array = (1, 2, 3); return \@array; } { my $arrayref = createArray(); print scalar @$arrayref, "\n"; } # $arrayref and underlying object destroyed In the createArray() subroutine, a data structure in the form of an array is created which holds the three elements, and a lexical array variable @array is created that associates with the array. Before the return function is executed, a reference to the array is created, which adds an association to the array. By the time the return function has been executed, the lexical association to the array has been
Slide 114: 102 Chapter 6 References destroyed. However, because a reference to the array still exists, the array is not deallocated. Therefore, when the number of elements of the array as accessible through the lexical array reference is printed on line 8 the value displayed is still 3. However, at the end of the block the lexical array reference is destroyed, so the reference count to the array is now 0, and the array object is also destroyed. 6.6 Typeglobs In the previous chapter we had a brief introduction to typeglobs. A typeglob represents an entry on the symbol table and stores a list of references to data objects with the same name. In this section, we describe how you may use typeglobs in your programs. In many cases, you seldom have to mess with typeglobs anymore as references are more convenient and flexible. Package variables, including those declared with local are resolved from symbol tables. You can use typeglobs to create symbol table aliases. For example, *array = *myarray; causes referencing operations on the symbol array to be resolved through the symbol table entry of myarray instead. Figure 6.4: Symbol Table Aliasing This causes $array to refer to $myarray, @array to @myarray and &array to &myarray etc. However, you rarely would like to create an alias for all types in practice. You can actually assign a reference of the desired type to a typeglob to replace the existing one. In Section 5.3.2 we wrote an insert() subroutine that allows you to insert a list into any given array at any arbitrary position. You may also rewrite it in this way to make use of typeglobs: sub insert (\@$@) { local *myarray = shift; # not my!! my ($pos, @list) = @_; return splice(@myarray, $pos, 0, @list); } Note that, unlike references, accessing through a typeglob does not require any dereferencing. With a typeglob entry in place you simply replace the * symbol with whatever type symbol that is required for
Slide 115: 6.6 Typeglobs the type. Some people may consider it convenient as a result. 103 Also note that you cannot use a lexical variable, that is one declared with my, to represent a typeglob because a typeglob is a symbol table entry which by nature cannot be lexical. However, this does not prevent you from assigning a lexical reference variable to a typeglob, though. Finally I would introduce to you the *foo{THING} notation, which you have already seen in the previous chapter in passing using *INC as an example. By using this notation you can get the individual references held in a given typeglob foo. Replace THING with the name of the type. The table below lists the possible types and their names: Type Scalar reference (\$foo) Array Reference (\@foo) Hash reference (\%foo) Subroutine reference (\&foo) Typeglob reference (\*foo) Format Variables (Not covered in this tutorial) Filehandle THING SCALAR ARRAY HASH CODE GLOB FORMAT IO The notations in parentheses represent an alternative way of accessing them, using references. Because of the absence of alternative ways to get the reference to a filehandle, *foo{IO} is most probably the only one that is more widely used. This is used when a file handle needs to be passed to and from a subroutine. We will talk about filehandles and the generic input/output mechanism at a later time. While typeglobs are still useful in modern Perl programming, you are generally advised to use them only if you have good reasons to do so, especially if that purpose can be fulfilled with other means such as references. First, typeglobs cannot be lexical variables while reference variables can. Also, typeglobs cannot be used to create complex data structures that are possible with references. Now you should have the fundamental knowledge to proceed to object-oriented programming. Summary ” ” ” ” ” ” ” Reference is a scalar value whose value is the address of a data structure. A reference is created by prefixing the \ symbol to a data object. An anonymous array is an array reference, which can be constructed by a pair of square brackets []. An anonymous hash is a hash reference, which can be constructed by a pair of curly braces {}. An anonymous subroutine is a subroutine reference, constructed by a subroutine definition without specifying a name. A data structure of arbitrary complexity can be easily implemented in Perl by using hash and array references as building blocks. You may access the underlying data object through a reference by dereferencing it.
Slide 116: 104 ” Chapter 6 References Pass by reference refers to the practice of passing a reference of a data structure as argument to a subroutine, which is the best way to pass an array or hash to a subroutine while not having its elements merged into @ . Named parameter passing may be emulated by passing all name-value pairs through a hash. Creating references to a data object actually implies establishment of additional access paths to it. Perl maintains a reference count for all data objects and they are not destroyed until the reference count drops to 0. A typeglob represents an entry of the symbol table and stores a list of references to data objects with the same name. References stored in a typeglob may be accessed using the *FOO{THING} syntax. ” ” ” ”
Slide 117: Chapter 7 Object-Oriented Programming 7.1 Introduction Object-oriented programming (OOP) is a popular term in the programming community. It represents an alternative approach of programming to cope with program development in the large. As I outlined in the previous chapter, maintenance of large scale programming projects become increasingly difficult as the size of code base increases. Object-oriented programming is seen by many in the Software Engineering community as a desirable solution to keep complex programming projects in order. In the text below we would explore the rationale behind this argument and point out how object-oriented model can help alleviate some of the deficiencies in the plain old procedural model. Perl started to support the notion of object-oriented programming in Perl 5. In fact, most of the necessary concepts that you need to understand object-oriented programming in Perl has already been covered in the previous chapters, and frankly, not much content are left for this chapter. However, because of the importance of object-oriented programming, and in order to avoid making the previous chapters too long if I lump them together, I have to dedicate a chapter to object-oriented programming. Compared with previous chapters, you will find longer and more complete examples in this chapter, so as to help you familiarize with OOP in a more practical context. You should flip back to the previous chapters if you have not read them carefully, because the Perl implementation of OOP is based on subjects studied in the previous chapters. In other words, knowledge of packages, scope, subroutines and references is a prerequisite to understanding OOP in Perl. If you have previously programmed in some other object-oriented programming languages like Java and C++, you may safely skip the section “Object-oriented Concepts” below. However, because the Perl implementation of OOP is very much different from other languages like PHP, Java and C++, you should not skim read the remaining sections as you would find the Perl approach to object-oriented programming alien to you (it happened to me as well when I learned Perl with some knowledge in C++), and there are some traps to which programmers who have written object-oriented programs in other languages are vulnerable. Please keep this in mind when you are reading this chapter. 7.2 Object-Oriented Concepts 7.2.1 Programming Paradigms The syntax of a programming language is largely influenced by the programming paradigm on which the language is based. A programming paradigm represents a framework which describes in a general, language-independent way how syntactic language elements are organized and processed. Without delving deep into a formal definition of programming paradigms, which is far beyond the scope of 105
Slide 118: 106 Chapter 7 Object-Oriented Programming this tutorial, I would rather state in a more casual way that a programming paradigm is supported by a school of thoughts to address a specific subset of programming tasks. Therefore, it is generally not appropriate to strictly claim that one paradigm is always better than the other. However, I personally believe if properly implemented, a properly-written object-oriented program is easier to maintain, and allows for a larger basis of extension by leveraging the power of object-oriented programming, for reasons that I would explain later in this section. There are various programming paradigms in use today. However, the majority of programmers adopt either one of two major paradigms, namely procedural programming and object-oriented programming. Our discussion so far has been solely based on the procedural programming paradigm. A complicated program is broken down into smaller pieces by delegating pieces of the source code to subroutines. and external source files. Variables are created and updated directly to maintain program state during execution. Object-oriented programming should not be considered a total revamp of procedural programming. Instead, it is best considered one that uses procedural programming as the basis with the emphasis on the way different logical components (i.e. classes) interact with each other. It enforces a more welldefined way of grouping related subroutines and data into a logical entity, and this is the foundation of object-oriented programming. For an executive summary of the various programming paradigms, please visit the Wikipedia.org entry on programming paradigm. Be prepared, they are conceptual computer science topics that cannot be easily understood. 7.2.2 Basic Ideas As I mentioned in the previous section, the idea of object-oriented programming is to group related subroutines and data into logical entities, each of which constituting its own domain. Such logical entities are known as classes. Each class defines a framework which describes the set of properties and behaviours of objects created (instantiated) from the class. Consider an airplane and a car. Because a car and an airplane have different characteristics and exhibit different behaviours (e.g. fly vs. move) we model them using two different classes. In other words, each class represents a certain type of object. In a class, behaviours are implemented as methods, which are subroutines associated with a class. For example, an airplane class would very likely include an ascend() method and a descend() method which contain the program needed for escalation and landing of an airplane. Also, different classes are likely to have different properties. For example, an airplane is likely to maintain an altitude property to track down the current height of the airplane above the ground. On the other hand, a car would not have this property. A class only defines the properties. The values of which are maintained independently in each class instance. After we have established a class, we need to create a class instance which is called an object. When a class instance (an object) is created, it possesses all the methods of the class and, as noted in the previous paragraph, each object holds an instance of the properties. This arrangement allows each object to carry its own set of property values. For example, consider a Car class which has only one property colour and one method move(). When we create several objects from the Car class, each of them “inherits” the method from the class and maintains a value representing the colour of the object.
Slide 119: 7.3 OOP Primer: Statistics 107 7.2.3 Fundamental Elements of Object-Oriented Programming An object-oriented programming language needs to qualify three fundamental properties, namely encapsulation, inheritance and polymorphism. These principles are fundamental to support the virtues of object-oriented programming. Adapted from Wikipedia.org, encapsulation refers to the practice of hiding data structures which represent the internal state of an object from access except through public methods of that object. Basically, that implies you should not change a property by directly modifying the internals of an object. Instead, you should modify it through the interface of the object. An interface is what is expected to be seen from outside of the object. In Perl, this includes all object methods. For example, while an airplane object maintains the altitude property, you should not modify its value directly. Instead, you invoke the methods ascend() and descend() to change its altitude because some actions need to be taken before climbing up or going down, which are accounted for by the two methods. By interacting with methods through the interface, the methods can check whether the operation is valid before committing any changes to the object. Inheritance allows a class (subclass) to inherit methods from another class (superclass). For example, a Helicopter class may inherit from the Airplane class the ascend() and descend() methods, while adding a stationary() method which an airplane doesn’t have and is a characteristic of a helicopter. By inheriting from another class, you don’t need to write the inherited code again (unless you would like to override them). Inheritance allows creation of code that can be easily reused. Polymorphism is a more abstract notion that cannot be easily explained without resorting to examples. The principle is that it allows programmers to use a consistent interface for method invocation on objects of different classes. These concepts would be revisited later on. However, before we go further, let us write a simple Perl program with object-oriented perspective so that you can appreciate how an object-oriented program looks like in Perl. 7.3 OOP Primer: Statistics In this section we write a simple Perl module Stats.pm which calculates the mean, variance and standard deviation of a set of input numbers. We then write a perl program stats.pl which uses the module written to output these statistics given the input of a set of numbers. The program listing is given first, and the theories are revisited afterwards. EXAMPLE 7.1 Statistics Calculator 1 2 3 4 5 6 7 8 9 10 # Stats.pm # The "Stats" Perl module package Stats; # Create a new class instance (object) # and return a reference of the object sub new { my $arg0 = $_[0]; my $cls = ref($arg0) || $arg0;
Slide 120: 108 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 Chapter 7 Object-Oriented Programming my $this = {}; bless $this, $cls; $this->clear(); return $this; } sub clear { my $this = $_[0]; $this->{’numlist’} = undef; $this->{’x_sum’} = 0; $this->{’x2_sum’} = 0; } # Append a value to the list sub addValue { my $this = $_[0]; my $num = $_[1]; if (defined $num) { push @{$this->{’numlist’}}, $num; $this->{’x_sum’} += $num; $this->{’x2_sum’} += $num**2; } } # Calculate total sub getTotal { my $this = $_[0]; return $this->{’x_sum’}; } # Calculate mean sub getMean { my $this = $_[0]; my @numlist = @{$this->{’numlist’}}; if (!@numlist) { return 0; } return $this->getTotal()/@numlist; } # Calculate variance sub getVariance { my $this = $_[0]; my @numlist = @{$this->{’numlist’}}; my $n = @numlist; my $sum_x2 = $this->{’x2_sum’}; my $sum_x = $this->{’x_sum’}; if (!$n) { return 0; } return ($n*$sum_x2 - $sum_x**2)/($n**2); } # Calculate standard deviation sub getStdDev {
Slide 121: 7.3 OOP Primer: Statistics 62 63 64 65 66 67 68 69 70 71 72 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 109 my $this = $_[0]; return $this->getVariance()**0.5; } # Get list of values sub getValueList { my $this = $_[0]; return @{$this->{’numlist’}}; } 1; #!/usr/bin/perl -w # stats.pl # This program uses Stats.pm to print out some assorted # statistics on the input numbers use Stats; # Catch Ctrl-C (SIGINT signal) $SIG{’INT’} = ’getResults’; my $obj = new Stats; sub getResults { print "\n\nResults ==============================\n"; print "Number of values: ", scalar($obj->getValueList()), "\n"; print "Total: ", $obj->getTotal(), "\n"; print "Mean: ", $obj->getMean(), "\n"; print "Standard Deviation: ", $obj->getStdDev(), "\n"; print "Variance: ", $obj->getVariance(), "\n"; exit (0); } print qq∼ Statistics Calculator Calculates several sets of statistics given a sequence of input numbers. Enter one value on each line. To exit, press Ctrl-C. ∼; while (1) { print ">> "; chomp(my $num = <STDIN>); $obj->addValue($num); } When the program is executed, the output looks like this: 1 cbkihong@cbkihong:∼/docs/perltut/src/oop$ perl -w stats.pl
Slide 122: 110 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Chapter 7 Object-Oriented Programming Statistics Calculator Calculates several sets of statistics given a sequence of input numbers. Enter one value on each line. To exit, press Ctrl-C. >> 13 >> 26 >> Results ============================== Number of values: 2 Total: 39 Mean: 19.5 Standard Deviation: 6.5 Variance: 42.25 In this transcript, 13 and 26 were input and then Ctrl-C was pressed to signal the end of input list. The results are then displayed. This program catches (intercepts) the SIGINT signal, which is generated when you press Ctrl-C. By default, if you press Ctrl-C when a program is running it promptly terminates the program. In this program, I demonstrated how to install a signal handler, which is a subroutine that is automatically invoked when the corresponding signal is caught. The signal handler is executed instead of terminating the program. Signals are messages sent by the operating system to a process (a program in execution). Because signal is a notion from Unix, and the message broadcasting mechanism is different on every operating system, behaviour of signal handling is platform-specific. Signals are readily supported on Unix platforms by using the signal handling system calls. Windows has limited support of signals. Support for common signals like SIGINT seems to be working on Windows, though. The example program works on both of my testing platforms, namely GNU/Linux and Activestate Perl 5.8.0 on Windows. I chose to use signal because catching of the SIGINT signal and executing the getResults() subroutine is handled automatically so we don’t need to place any extra code in the while loop to detect the end of input list. However, because an explanation of signals is out of the scope of this tutorial, and is not the main theme of this chapter, my discussion of signals will stop here. 7.3.1 Creating and Using A Perl Class Perl does not have any specialized syntax for classes as in many other object-oriented programming languages like C++, Java and PHP. That makes OOP in Perl looks more cryptic than it really is. A Perl class is contained in a file of extension ‘.pm’ in its own package. This is called a module. The filename of the module is the name of the class followed by ‘.pm’. The package name is also the name of the class. Recall from the previous chapter that if the package name contains ‘::’, it is changed to the directory separator as the pathname when the module is being sourced. The following table displays some example package names and the corresponding locations where they need to be saved. Recall that @INC contains a list of path prefixes that Perl uses to locate a Perl source file. A module contains all method definitions. Don’t forget to put the 1; at the end of the module. Omitting this results in a compile-time error. Before you use a module you should first import it into your program. You can use require that you were taught in Chapter 5. You may import a module in one of two ways. The first way is to pass
Slide 123: 7.3 OOP Primer: Statistics Package Name Stats Crypt::CBC File Path (relative to @INC) Stats.pm Crypt/CBC.pm 111 Table 7.1: Relationship of Package Names and File Placements the path to the module (relative to @INC) in quotes to the require function. For example, require "Crypt/CBC.pm"; Another way is to just specify the package name without quotes, for example require Crypt::CBC; However, the use function is preferred in general for Perl modules. The syntax of use is use MODULE [LIST]; It is semantically equivalent to BEGIN { require MODULE; import MODULE LIST; } # indirect object syntax which not only imports the module MODULE at compile time, it also imports the symbols specified by LIST into the current namespace (package) using the import class method. import is a special class method that may be defined by a class author. Many modules do not have any symbols to be exported, however, some of them do. If the method cannot be found, it is silently bypassed. For example, users of the CGI::Carp module may import from it the fatalsToBrowser symbol, which is actually the name of a subroutine that generates a page in HTML describing the details when an error occurs, which is then returned to the viewer’s browser. This is used by CGI scripts to easily trap runtime errors. The use statement required is use CGI::Carp ’fatalsToBrowser’; Documentation of a class should tell you how to use it, and whether any symbols need to be imported. Therefore, usually you don’t have to worry about this at all, unless you have to write a class that exports symbols. The use function is generally preferred to require because modules are imported at compile time. Therefore, missing modules are discovered at the time the program is compiled before execution, which saves you from an embarrassing situation where execution is on midway when Perl finds out some of the modules are missing and the program fails to continue. 7.3.2 How A Class Is Instantiated As mentioned previously, to instantiate a class is to create an object that belongs to the class. From the perlobj manpage, a Perl object created is simply a reference that happens to know which class it belongs to. How does everything fit together? Let us go back to the new() class method of the Stats class in our example. The most important statement is on line 12, that is, the bless() statement. The bless() statement makes the reference $this no longer just an ordinary hash reference. It becomes an object reference of the Stats class, and by doing so you can also access the methods from the reference. Therefore, this method is known as a
Slide 124: 112 Chapter 7 Object-Oriented Programming constructor, because it is where the object is created. A constructor is not necessarily called ‘new()’. It can be of any name, despite it is customarily called new. Object-specific data, or properties, are stored into the object reference. Let us explain each line one by one. my $arg0 = $_[0]; my $cls = ref($arg0) || $arg0; In Perl, methods are exactly identical in behaviour to subroutines. However, whether it acts like a method or a plain subroutine is determined solely by the way it is invoked. A method is a subroutine that expects an object reference as its first argument. Using the new() method as an example, the following ways of invoking it are identical: my $obj = new Stats; my $obj = Stats->new(); my $obj = Stats::new(Stats); # Indirect object syntax # C++-like class member invocation (recommended) # Resembling subroutine invocation With the indirect object syntax, the name of the method is placed first, followed by the object reference or name of the class. For example, getTotal $obj (’a’, ’b’) means that the method getTotal() is invoked on the object $obj. Any arguments that follow (’a’, ’b’) are arguments to the method. This form is generally only recommended for constructors, because this form of method invocation is not syntactically obvious. The second form is generally recommended as it’s less confusing. The class name or object reference is placed first, followed by ->, the method name and arguments. The last form is typical subroutine invocation. However, in this form you need to pass the class name or object reference as the first argument. Now go back to our example. We put the first argument to $arg0. The next line serves to obtain the class name from it. The ref() function tests if the parameter is a reference. If it is one, it returns a string indicating its type (see the ref() documentation for details). If it is an object reference, it returns the package name. By the short-circuiting behaviour of logical operators the package name is assigned to $cls. This caters for the fact that the new() method can be invoked on the Stats class or a class instance, although customarily, a constructor is only invoked on a class in most other programming languages. my $this = {}; bless $this, $cls; Here, we create an empty hash reference and bless() it to the Stats class. This operation is called bless, very likely because it works like a wizard playing with his magic wand, turning an ordinary reference into an object. Very magical indeed. To be an object, Perl only mandates a bless()ed reference. It doesn’t require to be a hash reference, although it is most likely chosen for its flexibility. Please read the perltoot manpage for its coverage on alternative forms of Perl objects. It accepts two parameters. If the second parameter is missing, it assumes the current package. The last two lines in the constructor resets the object data and return the object reference so that the caller of the constructor (in stats.pl as $obj) can save the object reference for use in later operations. The program reads in a number from each line, and invoke the addValue() method on $obj. This invokes the method with the object reference itself as the first parameter. The value, as the second
Slide 125: 7.4 Inheritance 113 parameter, is appended to a list of numbers that is stored with the object reference, and update the sum (x sum) and the sum of squares (x2 sum). These two pieces of statistics are used to calculate the various statistics, when the SIGINT signal is detected and the getResults() signal handler invoked. The variance and standard deviation are given by n n 1 (n ∑ x2 − ( ∑ xi )2 ) i n i=1 i=1 Variance = σ2 Standard deviation (σ) = The class provides a number of methods for users of the class to retrieve the list of values, total, mean, variance and standard deviation. Note that these methods are invoked on the blessed object returned by the constructor instead of on the class. Through the class interface, the class provides all the necessary methods to interact with without requiring the user to know anything about the implementation of the class. For example, you do not have to care about how data are stored and processed internally. Because an object is simply represented by a reference variable, unlike some other programming languages Perl does not have the notion of protected access specifiers to classify methods by different access permissions such as “public” or “private”. Users may also be possible to manipulate the underlying hash directly. However, by documenting the interface clearly class users will be encouraged to access the object through the interface instead of having to manipulate object data directly from the reference variable. This fulfills the initiative of encapsulation. 7.4 Inheritance Inheritance is important in any programming languages because it allows you to reuse portions of another source program in your program, so that you don’t have to rewrite them. You save development effort as a result. Perl implements inheritance in a very simple way. When a method is invoked on an object and the method cannot be found in the current package, the packages whose names are in the variable @ISA are searched in order for the missing method. In object-oriented terminology, we describe this situation as a derived class inheriting from a base class, because the derived class contains methods defined in itself as well as from the base classes. A base class is also called a superclass or a generic class, while a derived class is also called a subclass. When you put the name of a package into your package’s @ISA, all the methods defined in that package, together with any defined in their base classes as well as the base classes of base classes etc. are available in your package. You may wish to redefine some of the inherited methods because they may not suit your current class anymore. To redefine a method, which is called overriding a method, simply rewrite the method in the way you desire and put it in the current package. If the method is invoked on the current package or its subclasses, the overriding method will be invoked instead of the overridden ones (that is, those in the base classes). As an example, we will write a Stats2 class which inherits from Stats with the added functionality of deducing the maximum and minimum among the list of numbers input. Also attached below is the modified stats2.pl which uses Stats2 with the additional functionalities: EXAMPLE 7.2 Statistics Calculator (Extended) 1 2 3 # Stats2.pm # Derived "Stats" with some functionalities added
Slide 126: 114 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Chapter 7 Object-Oriented Programming package Stats2; @ISA = (’Stats’); use Stats; # inherits Stats # Note that by using the 2-argument form of # bless() the constructor can also be inherited # Override addValue() sub addValue { my ($this, $num) = @_; if (!defined $this->{’min’}) { $this->{’min’} = $num; $this->{’max’} = $num; } else { $this->{’min’} = ($num < $this->{’min’})?$num:$this->{’min’}; $this->{’max’} = ($num > $this->{’max’})?$num:$this->{’max’}; } # invoke the base class version of addValue() $this->SUPER::addValue($num); # OR $this->Stats::addValue($num) } # Find minimum sub getMinimum { my $this = $_[0]; return $this->{’min’}; } # Find maximum sub getMaximum { my $this = $_[0]; return $this->{’max’}; } 1; #!/usr/bin/perl -w # stats2.pl # This program uses Stats2.pm to print out some assorted # statistics on the input numbers use Stats2; # Catch Ctrl-C (SIGINT signal) $SIG{’INT’} = ’getResults’; my $obj = new Stats2; sub getResults { print "\n\nResults ==============================\n";
Slide 127: 7.4 Inheritance 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 115 print "Number of values: ", scalar($obj->getValueList()), "\n"; print "Minimum: ", $obj->getMinimum(), "\n"; print "Maximum: ", $obj->getMaximum(), "\n"; print "Total: ", $obj->getTotal(), "\n"; print "Mean: ", $obj->getMean(), "\n"; print "Standard Deviation: ", $obj->getStdDev(), "\n"; print "Variance: ", $obj->getVariance(), "\n"; exit (0); } print qq∼ Statistics Calculator Calculates several sets of statistics given a sequence of input numbers. Enter one value on each line. To exit, press Ctrl-C. ∼; while (1) { print ">> "; chomp(my $num = <STDIN>); $obj->addValue($num); } Note that by placing the name of a package in @ISA does not free you from the need of adding the use statement to import the contents of the package. Stats is the base class while Stats2 is the derived class. Here, the addValue() method is overridden to check whether the current input number is the minimum or maximum and update the counters internally if it is one to always keep track of the minimum and the maximum. Then, two new methods are defined that let users retrieve the minimum and maximum values. Occasionally an overriding method needs to invoke an inherited but overridden method. addValue() here is an example, because we overrode it to add new functionalities rather than to replace it in its entirety. You can prefix a package name before the name of the method, separated by two colons, to indicate the class from which to start searching for the method. The default is always to start searching from the package to which an object belongs. Therefore, you always have access to the current, overriding version. You can access the version defined in the Stats package, that is the base class with respect to Stats2, by $this->Stats::addValue($num); However, for convenience a pseudo class SUPER has been defined that always refer to the base classes. Essentially, the current, overriding version is bypassed, as indicated in the example program. The Stats2 class can actually make use of the getValueList() method to deduce the maximum and the minimum values on-the-fly instead of having to memorize two additional pieces of information, and the addValue() method does not need to be overridden as a result. This is left as an exercise for the readers.
Slide 128: 116 Chapter 7 Object-Oriented Programming Even if your class’s @ISA is empty, it still inherits from a class called UNIVERSAL, which is the base class of all classes in Perl. Therefore, methods defined in UNIVERSAL are available in any class. It has three methods, which are described below. isa(CLASS) This method allows you to check whether an object belongs to CLASS or a subclass of CLASS. It returns a true value if it is one, undef otherwise. You may use it in two ways. Object-oriented form: $obj->isa(’UNIVERSAL’); Procedural form: use UNIVERSAL ’isa’; isa($obj, ’UNIVERSAL’); # must be true, by definition can(METHOD) This method allows you to check whether an object has a method called METHOD. It returns the reference to the subroutine if there is one, undef otherwise. It also searches the upstream base classes. As an example, this is an indirect and silly way to invoke the isa method on the current package: sub isSubclass { my ($this, $cls) = @_; my $subref = $this->can(’isa’); $this->$subref($cls); } $obj->isSubclass(’UNIVERSAL’); Can you add methods to a package like UNIVERSAL without modifying the module file? The answer is you can. Here, I am going to demonstrate how to add to the UNIVERSAL class a constructor so that you don’t have to define any constructors in your modules. EXAMPLE 7.3 1 2 3 4 5 6 7 8 9 10 11 # universal.pl # Contains definitions to add to UNIVERSAL class package UNIVERSAL; sub new { my $cls = ref($_[0]) || $_[0]; my $this = bless {}, $cls; $this->initialize(); return $this; }
Slide 129: 7.4 Inheritance 12 13 14 15 16 17 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 117 sub initialize { # defaults to no-op # override to execute class-specific startup code } 1; # TestObject.pm # Test class module package TestObject; sub initialize { print "TestObject::initialize()\n"; $this->{’a’} = 6; } sub displayMessage { my ($this, $msg) = @_; print $msg; } 1; #!/usr/bin/perl -w # testuniversal.pl BEGIN { # Add additional methods to UNIVERSAL require "universal.pl"; } use TestObject; my $obj = new TestObject; $obj->displayMessage("Hello World!\n"); universal.pl contains the methods to add to the UNIVERSAL class. This file is sourced by a BEGIN block in testuniversal.pl which ensures that the module definitions are imported at compile time. Note that the TestObject module does not have to define the constructors as a result. Note that you may optionally specify an initialize() method in your class module, which is automatically invoked by the constructor (universal.pl::9) where you can place some initialization code. If you do not provide it, the version provided by universal.pl will be used instead, which is just an empty method that does not do anything. Although you may do it that does not necessarily mean you should do it. In general, I suggest converting universal.pl into a module which acts as the base class for all your other modules instead of adding new methods to the UNIVERSAL class. This example is given to demonstrate it works. However, if a user of TestObject forgets to import the universal.pl, then it will be an error as a constructor cannot be found.
Slide 130: 118 Chapter 7 Object-Oriented Programming Summary ” ” ” ” ” ” ” ” ” ” ” ” Object-oriented programming refers to the practice of grouping related subroutines and data into a self-contained logical entity, called a class. Each class defines a framework which describes the set of behaviours and properties of objects instantiated from the class. Methods are subroutines associated with the class. Properties are attributes possessed by objects instantiated from the class. Encapsulation refers to the practice of hiding internal class implementation from users. Class users are expected to only interact with it through public methods. Inheritance allows a subclass to inherit methods from a superclass, so that you may save development effort through reuse. A class module in Perl is simply a package, whose file extension is ‘.pm’. The use() function is usually used to load a Perl module at compile time. Every Perl class has a constructor which is responsible for creating objects belonging to the class. The bless() function associates a reference variable with a class in order for the reference to become an object. A method inherited from a superclass may be overridden by specifying the definition for the subclass. In class methods the SUPER pseudo class may be used to invoke an overridden method. All classes inherit a class named UNIVERSAL. Web Links 7.1 7.2 7.3 7.4 perlboot manpage — Object-Oriented Tutorial for Beginners http://www.perldoc.com/perl5.8.0/pod/perlboot.html perltoot manpage — Tom’s Object-Oriented Tutorial for Perl http://www.perldoc.com/perl5.8.0/pod/perltoot.html perltooc manpage — Tom’s Object-Oriented Tutorial for Class Data in Perl http://www.perldoc.com/perl5.8.0/pod/perltooc.html perlbot manpage — Object-Oriented Bag of Tricks http://www.perldoc.com/perl5.8.0/pod/perlbot.html
Slide 131: Chapter 8 Files and Filehandles 8.1 Introduction The ability to read from and write to files is nearly always essential to computer programs. Output is frequently generated in the course of execution of a program. However, if they are not stored in secondary storage media such as disks, they will disappear once the power is switched off. Therefore, file access is an important element of the input/output system. In this chapter, we will explore the general Perl input/output system and the functions we can use to access the filesystem. Similar to the C standard and to be in line with Unix concepts, Perl uses the concept of filehandles to represent an opened file. They are also known as file descriptors in programming languages like C. Although I did not mention explicitly, actually you have been working with filehandles throughout the tutorial. The statement print "Hello World!"; Appeared in the very first Perl program covered in this tutorial, it implicitly uses a filehandle to get the string printed on the screen. This statement is actually written as print STDOUT "Hello World!"; However, the filehandle STDOUT is assumed by default, so we have been omitting it all the way up to this chapter. Unix has a generalized view of input and output. Apart from files, hardware devices are also represented as files on a Unix filesystem. Therefore, input and output of hardware devices can be represented by input and output of files, through a filehandle. This generalized view helps abstract a user or programmer from the peculiarities of the implementation of each device, which are left to the device drivers that exist in the operating system kernel, allowing users to interact with each device in a uniform manner. With a strong Unix tradition, Perl adopts a similar notion as well. Even if you don’t use any flavours of Unix at all it does not mean you don’t have to read this chapter, because Perl uses the same concepts on every platform it runs on, so that a considerable degree of platform independence can be achieved. 8.2 Filehandles Perl has a number of predefined filehandles, namely STDIN, STDOUT and STDERR that you can use in command line applications for input/output on screen. STDIN, or standard input, is the filehandle from 119
Slide 132: 120 Chapter 8 Files and Filehandles which keyboard input can be read. STDOUT, or standard output, is where you should send the output of your program. STDERR, or standard error, is mostly used for outputting warning or error messages because sometimes people wouldn’t like error messages to be displayed. By dumping error messages to a separate filehandle the user may decide whether to display them or instead redirect them to, for example, a log file on disk for diagnosis at a later time. These three filehandles are always ready for you to use and you don’t need to create them manually. Filehandle is one of the data types available in Perl. As you have learnt in the chapter on references, filehandle has a separate slot in a symbol table entry. However, unlike other data types you do not have to prefix a filehandle with a type symbol. By convention, filehandles are in all capital characters to make them visually stand out from names of functions and subroutines etc. 8.2.1 open a File Unless you are working on one of the predefined filehandles, the first step you should take is to populate a filehandle. A populated filehandle represents an active stream which allows input or/and output of data. For the case of file access, a populated filehandle represents an opened file, which is then used by the input/output functions to read from or write to the file. Filehandle is also used in Perl socket programming which is used to represent a socket. However, due to time constraints network programming is not covered in the first edition of this tutorial. Unix supports two sets of file access functions. One set is provided by the operating system kernel, and the other set is provided by the standard C libraries installed on the system, which is implemented on top of the version provided by the kernel. The file access functions in Perl are actually interfaces to these functions. Perl allows access to both, through the open() function which invokes the version provided by the C libraries, and the sysopen() function invokes the operating system version. The use of open() is generally preferred to sysopen(), because it is simpler to use, but sysopen() is more powerful. sysopen() is not covered in this edition of the tutorial. You use the open() function to open a file. Usually, the open() function takes on one of the following forms: open FILEHANDLE, EXPR open FILEHANDLE, MODE, EXPR FILEHANDLE is either a filehandle or a lexical variable with the undef value, which is used by the open() function to store the reference to the filehandle created. EXPR is a scalar expression which contains the name of file to open(), and MODE describes the access mode to apply to the file, for example, whether read or write access are allowed on the file. If MODE is missing, it defaults to “<”, the read-only mode. Otherwise, MODE should be prepended to EXPR in the first form, if it is not given as a standalone argument. If open() is successful, it returns a nonzero value. Otherwise, undef is returned. You should always check the return value of file access functions and handle cases of failure to ensure your program is fault tolerant, especially if your program is to be used by other people instead of you, such as CGI scripts. Traditionally, a filehandle is usually used instead of a lexical filehandle reference. For example, my $retval = open(LOG, "<command.log");
Slide 133: 8.2 Filehandles 121 which creates LOG permanently on the symbol table of your current package. This may be acceptable to you, but you may wish to localize it to a certain subroutine, for example. As a recapitulation, this is the way to do it: local *LOG; my $retval = open(LOG, "<command.log"); However, because filehandles themselves cannot be lexical, and many Perl programmers are not familiar with the use of typeglobs, a better way would be to use a lexical filehandle reference, which you can easily pass around without needing any knowledge in typeglobs. An example is my $fhLOG; my $retval = open($fhLOG, "<command.log"); Here is a summary of the 6 access modes provided: MODE < +< > +> >> +>> Description Read-only access. Specified file must exist. Read-write access. Specified file must exist. Write-only access. File emptied if exists; created otherwise. Read-write access. File emptied if exists; created otherwise. Append-only access (file pointer at end-of-file). File created if not exist. Read-Append access (file pointer at end-of-file). File created if not exist. Table 8.1: File Access Modes for open() These access modes determine the operations that can be applied on a file. If you open() a file with read-only access but you try to write data to the corresponding filehandle, a runtime error will occur. Every open file has a file pointer, which determines the position of the next character read or write. The first four modes listed above position a file pointer at the beginning of a file, so that read/writes occur at the beginning of the file. < grants only reads access to the file. > grants only write access to the file, which is automatically created if it does not exist, and empties it before writing. Both +< and +> grants read-write access to the file, so you may read from as well as write to the filehandle. The difference between +< and +> is that for +<, the specified file must exist, while for +> the file is automatically created if it does not exist, and empties the content before writing. The last two modes place the file pointer at the end of a file. Therefore, data are written at the end of the file. A file opened with either of these two modes is created where necessary. Note that +< does in-place writing. If file write occurs in the middle of a file, it simply overwrites the exact number of characters from the file pointer required by the write, growing the file as necessary and other characters in the file are unaffected. The effect is similar to putting your text editor in “replace” or “overwrite” mode and typing characters at a cursor to overwrite the old text. 8.2.2 Output Redirection Output redirection allows you to redirect output sent to a certain filehandle to another filehandle. This is frequently used by shell script authors on Unix systems to redirect error messages to log files or
Slide 134: 122 Chapter 8 Files and Filehandles simply to throw them away as if they have not been output at all. To use I/O redirection, specify “>&” as the MODE, and EXPR is the name of the filehandle to which output to FILEHANDLE is redirected. An example is shown below, which redirects output that are sent to STDERR to a file that has been previously opened with the filehandle LOG instead. open (STDERR, ">&LOG"); Text that are sent to STDERR will be redirected to a file instead, so they are no longer output on screen. 8.3 File Input and Output Functions In this section, I will introduce to you various functions you can use to read from or write to a filehandle. Note that in functions expecting a filehandle as its argument you can use a lexical filehandle reference instead of a typeglob. Simply replace the filehandle with the lexical variable, for example, $fhLOG. 8.3.1 readline() — Read A Line from Filehandle The readline() function accepts a typeglob as parameter to read a line from the filehandle contained in the typeglob. In scalar context, each invocation of readline() reads up to the newline character. When no more lines can be read, undef is returned. An example is shown below, which copies a text file File1.txt to File2.txt. open FILE1, "<File1.txt" or die "Cannot open File1.txt!"; open FILE2, ">File2.txt" or die "Cannot open File2.txt!"; my $line; while ($line = readline(FILE1)) { print FILE2 $line; } close FILE1; close FILE2; However, customarily readline() is not frequently used because Perl provides an operator <FILEHANDLE> which is an interface of readline(). We can replace readline(FILE1) above with <FILE1>. In list context, both readline() and <FILEHANDLE> read all the way until end-of-file occurs, and split it into lines. The return value is an array with its elements being the lines extracted. This is seldom used in practice, because if the incoming file is very large, the generated array will also be very large, consuming a lot of memory space. Therefore, it is a lot safer to set up a loop to read in line by line as shown above. 8.3.2 binmode() — Binary Mode Declaration Not knowing if you are aware or not, text files actually are stored differently on Unix and MSDOS/Windows systems. The culprit is that the line termination characters used on these platforms are
Slide 135: 8.3 File Input and Output Functions 123 different. That’s why there is an ASCII/Binary transfer mode option in your FTP application. Binary files do not rely on line termination characters to denote the end-of-line. Therefore, a binary file is represented in an identical manner on these platforms. However, because text files uses line termination characters, and they vary from platform to platform, two files with identical textual content end up being represented differently on different systems. To ensure Perl programs have high levels of portability, in general Perl programmers simply have to treat \n as the line termination character. This is what we have readily assumed from the very beginning of this tutorial, and the underlying system carries out all necessary conversions for us automatically. However, this system does not work for MS-DOS/Windows systems because they distinguish between text files and binary files. Therefore, on these systems Perl needs to introduce a PerlIO layer on top of the native file access functions which converts between the underlying line termination characters and \n. This does not pose any problems for text files, as this conversion is actually intentional for text files. However, binary files should never be altered in any way. binmode() with just a single parameter of FILEHANDLE essentially instructs all read/write through the filehandle to bypass the conversion layer. Because binmode() is ignored on other systems, you should generally use it on all binary files for portability. It should be invoked right after a file is opened, i.e. open BMP, "<logo.bmp"; binmode BMP; If binmode() has two parameters, the second parameter indicates the PerlIO layers to apply which act as conversion filters. The “crlf ” layer is the layer we mentioned above for MS-DOS compatible systems which carries out line termination conversions. Usage of this form is generally not needed as the defaults are generally adequate for most programming purposes, and so are not described here. 8.3.3 read() — Read A Specified Number of Characters from Filehandle The syntax of read() is read FILEHANDLE, SCALAR, LENGTH[, OFFSET] which reads from FILEHANDLE LENGTH characters, usually equivalent to bytes and assign it to the scalar SCALAR. If OFFSET is given, it specifies the zero-based offset of SCALAR from which to start writing. This is usually used for binary files, but not necessarily. The file copying program shown above should generally not be used because binary files are not line oriented, and it does not use binmode() which causes file copying errors on MS-DOS compatible systems. Presumably the correct way is as follows: sub copy ($$) { my ($src, $dest) = @_; open FILE1, "<$src" or die "Cannot open $src!"; open FILE2, ">$dest" or die "Cannot open $dest!"; binmode FILE1; binmode FILE2; my ($buffer, $numChars); my $bufferSize = 1024; my $size = 0; while ($numChars = read(FILE1, $buffer, $bufferSize)) { $size += $numChars;
Slide 136: 124 Chapter 8 Files and Filehandles print FILE2 $buffer; } close FILE1; close FILE2; return $size; } 8.3.4 print()/printf() — Output To A FileHandle We have used print() quite a lot so it is not worthwhile repeating all the details here again. However, if a filehandle is specified, it outputs to the filehandle. Otherwise, the filehandle defaults to STDOUT, as I mentioned earlier in this chapter. printf() is similar to print(). However, it is an exceptionally powerful function that lets you perform varieties of type conversions on the fly. printf() is an artifact from the C standard I/O library. In C, generating a string is not a simple affair, because there is no variable interpolation as in Perl and there is not an easy and flexible way of string concatenation. Also, because C is strongly typed and you cannot concatenate a C-style string with other data types, for example, a number, you have to end up performing a lot of type conversions before the desired string can be successfully generated and eventually written to a file descriptor. Therefore, for convenience purpose C provides a set of functions collectively known as the printf() family of functions which allows generation of many common forms of string to be completed in one function call. While this function is very versatile it is also acclaimed as the most complicated function in C. This function works by constructing a concise but generally cryptic format string, which consists of the string with some placeholder fields inserted. These placeholder symbols describe the type conversion operation required for each of the fields to be inserted into the string, which are passed as additional arguments to the printf() function. printf() is one of the several few C builtin functions which accept a variable number of arguments. The complexity lies completely in constructing the proper format string. While Perl has flexible string interpolation and automatic type conversions, printf() is not as important as in C. Moreover, because printf() involves additional operations, it is less efficient compared with print(). Therefore, you should avoid it if print() suffices for the purpose. However, because it is indeed handy for certain kinds of conversion operations, I am going to describe it below. In Perl’s implementation, another function sprintf() exists. It returns the generated string instead of directing it to a filehandle. Otherwise, it is identical to printf(). In fact, printf() is internally implemented using sprintf(). This is intended to be an introductory description of most frequently used options only. For more information, please read the perlfunc/sprintf manpage. The following sprintf() example, however simple, gives you a taste of what it is like: # "Good Morning. The number is 6." sprintf("%s. The number is %d.", "Good Morning", 6); The format string is just a string with placeholders inserted. Placeholders start with the % character. Placeholders are replaced by the corresponding arguments given after the format string with the specified type conversion operations performed. Here, because we have 2 placeholders, we have two additional parameters given after the format string. %s indicates the placeholder value is a string, while %d indicates a signed integer. Perl includes an extension to printf() functions that you may specify the placeholder values in an arbitrary order. That is, the arguments may be given in a different order than the order the placeholders appear in the format string. However, because this only complicates matters, I shall not discuss this and mandate all arguments to be given in the same order that the placeholders
Slide 137: 8.3 File Input and Output Functions 125 appear in the format string. s and d here indicate the conversion type. Some other conversion types are defined as well. The most frequently used ones are: Conversion Type s d u x X f Here are some examples: Description String Signed integer Unsigned integer (decimal) Unsigned integer (hexadecimal, lowercase) Unsigned integer (hexadecimal, uppercase) Floating point number sprintf(’%X’, 12); # C sprintf(’Absolute value: %u’, -1); # 4294967295 (see below) When using the unsigned integer conversion type you have to ensure the value passed as parameter is not negative. This concerns the way an unsigned integer is represented in the underlying C library. All integers are represented with a fixed number of bits. On my system, this is 32 bits. Therefore, for unsigned integers 32 bits can represent all integers from 0 up to 2 32 − 1, that is 4294967295. Signed integers are also represented with 32 bits. The method of representation of negative integers is known as “2s complement”. The range of signed integers becomes -2147483648–2147483647. Negative integers are represented by inverting the bits of the corresponding positive magnitude and adding 1. Therefore, 1 and -1 are represented as follows: 1 -1 00000000 00000000 00000000 00000001 11111111 11111111 11111111 11111111 When you pass -1 as argument while the conversion type is u, the bit sequence of -1 is interpreted as if it is an unsigned integer. Note that this bit sequence is identical to 4294967295 in unsigned integer representation. Therefore, the string returned is 4294967295. Between % and the conversion type character additional options may be applied. We will dicuss some of them below. You may mix and match them, although a few of them are mutually exclusive as otherwise noted. You may stipulate a prepended space before a numeric conversion if the numeric value is not negative by inserting a space. For example, sprintf(’<% d>’, 3); sprintf(’<% d>’, -3); # < 3> # <-3> By replacing the space with a + symbol, the + sign is prepended instead of a space. Note that the space and + are mutually exclusive. If both exists, + is taken. For example, sprintf(’<%+d>’, 12); sprintf(’<%+f>’, 13.6); sprintf(’<% +f>’, 13.6); # <+12> # <+13.600000> # <+13.600000>
Slide 138: 126 Chapter 8 Files and Filehandles Once an argument has been converted to the string form, by default the corresponding placeholder is substituted by the string without introducing any padding, as we have seen in the above examples. However, we may introduce extra padding by specifying the minimum field width. It specifies the minimum length of the placeholder value string. If this number is larger than the length of the string, padding is added. By default the string is right-aligned so padding appears on the left. However, if the - symbol is prepended to the minimum field width, the string is left-aligned so padding appears on the right. The padding is in terms of spaces. If the minimum field width is smaller than the string length required, then the field width is ignored. For example, sprintf(’<%+12f>’, 13.6); sprintf(’<%+-12f>’, 13.6); sprintf(’<%+-5f>’, 13.6); # < +13.600000> # <+13.600000 > # <+13.600000> If the string is left-aligned (i.e. format string without the - symbol) you may place a 0 anywhere before the field width to instruct zero padding instead of spaces. For example, sprintf(’<%08d>’, 12345); sprintf(’<%0+12f>’, 13.6); sprintf(’<%+012f>’, 13.6); # <00012345> # <+0013.600000> # <+0013.600000> As you have seen, floating point numbers have a certain precision. The precision is influenced by the number of decimal places. Subject to fixed storage space of floating point numbers, the precision is not unlimited as in our usual mathematics. You may specify the precision by specifying the number of decimal places, together with a prepending ‘.’. For example, sprintf(’<%0+12.4f>’, 13.6); sprintf(’<%12.4f>’, 13.6); # <+000013.6000> #< 13.6000> If applied on a string, this specifies the maximum number of characters to fit, somehow like the Perl substr() function. For example, sprintf(’<%.4s>’, ’abcde’); sprintf(’<%10.4s>’, ’abcde’); sprintf(’<%10.4s>’, ’abc’); sprintf(’<%-10.4s>’, ’abcde’); # # # # <abcd> < abcd> < abc> <abcd > 8.3.5 seek() — Set File Pointer Position seek FILEHANDLE, POSITION, WHENCE You may use the seek() function to set the position of file pointer in a file specified by FILEHANDLE. POSITION is a signed integer indicating the new position, relative to the position indicated by WHENCE. WHENCE is an integer which is either 0, 1 or 2 representing the beginning-of-file, the current file pointer position and the end-of-file respectively. However, because hard-coding these integer values is not semantically obvious, we usually use the names of the corresponding constants available in the C standard I/O library instead. These constants are defined in the Perl Fcntl module. You can import the three constants by
Slide 139: 8.3 File Input and Output Functions use Fcntl ’:seek’; 127 The constants are SEEK SET, SEEK CUR and SEEK END respectively. After you have imported them, you can use them directly in your programs. seek() is frequently used with binary file access. Binary files usually have their data stored in a certain format that allows efficient access of data. To achieve this, some fields are encoded and saved at fixed positions in a binary file, which you may access directly with the seek() function. The new position is calculated as the sum of POSITION and the base position indicated by WHENCE. Here are some examples: seek(FILE, 0, SEEK_SET); seek(FILE, 5, SEEK_CUR); seek(FILE, -1, SEEK_END); # Jump to beginning of file # Jump 5 bytes forward # Jump to last byte of file 8.3.6 tell() — Return File Pointer Position The tell() function returns the position of the file pointer, in bytes. This function returns meaningful values only if used on an open()ed file. The position is a zero-based offset from the beginning of the file specified by the filehandle which is passed as the parameter. It is frequently used with seek() to move the file pointer about in a file for reading or writing. This is an example which deduces the size of a file by using seek() and tell(): use Fcntl ’:seek’; sub getFileSize ($) { my $filename = $_[0]; local *FILE; open FILE, "<$filename" or return undef; seek(FILE, 0, SEEK_END); my $size = tell(); # last filehandle read is FILE close FILE; return $size; } my $filename = $ARGV[0]; print "$filename\t\t", getFileSize($filename), " Bytes \n"; However, on a system that supports stat() this can be satisfactorily replaced by (stat($filename))[7] 8.3.7 close() — Close An opened File At the end, when you have finished working with a filehandle, you should close() it. Simply pass the filehandle as the parameter to close().
Slide 140: 128 Chapter 8 Files and Filehandles 8.4 Directory Traversal Functions In this section, I will introduce to you various functions you can use to traverse the directory structure of your system. Note that you ought to be using File::Find, which is easier to use and more flexible. In fact, it uses the functions that we cover below. However, some simple directory traversal operations may be more efficient if you implement them directly. Description of the functions is presented first. You will find a full example afterwards which uses these functions to build a class which performs file search. 8.4.1 opendir() — Open A Directory opendir DIRHANDLE, PATH This function prepares the directory PATH for subsequent directory traversal functions. If the directory exists, a true value is returned, and populates DIRHANDLE which is a filehandle. 8.4.2 readdir() — Read Directory Content readdir DIRHANDLE The readdir() function reads the content of the directory referred to by DIRHANDLE, which is populated by the opendir() function. In scalar context, each readdir() invocation returns an entry in the directory. When there is no more entry, undef is returned. This is similar to the behaviour of read() saw earlier. You may also get all the entries in one go by calling the function in list context, and an array containing all the entries inside will be returned. 8.4.3 closedir() — Close A Directory When you have finished traversal of the directory concerned, you should closedir() it. 8.4.4 Example: File Search In this example, we will write a class that allows users to search for a file recursively in a directory tree whose name matching a pattern specified by the user. The pattern is in the form of a regular expression for convenience purpose. It is by no means versatile as File::Find, but it will give you an idea of how to use the directory traversal functions. EXAMPLE 8.1 File Search 1 2 3 4 5 6 7 8 9 # FileSearch.pm # An Example File Search module package FileSearch; # Create a new class instance (object) sub new { my $arg0 = shift; my $cls = ref($arg0) || $arg0;
Slide 141: 8.4 Directory Traversal Functions 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 129 my $this = {}; bless $this, $cls; $this->initialize(@_); return $this; } sub initialize { my $this = shift; my %options = @_; foreach (keys %options) { $this->{$_} = $options{$_}; } } sub find { my ($this, $path) = @_; my @matches = (); my $pattern = $this->{’PATTERN’}; local *DIRENT; opendir DIRENT, $path or return (); # Get a list of entries in this directory, # and sort it lexicographically my @entries = sort { lc($a) cmp lc($b) } readdir(DIRENT); foreach my $fn (@entries) { my $entry = "$path/$fn"; # Recursive search if the entry is a directory, # We should ignore . and .. in our search if (-d $entry and $fn !∼ m/ˆ\.\.?$/) { push @matches, $this->find($entry); } $fn =∼ m/$pattern/ and push @matches, $entry; } closedir DIRENT; return @matches; } 1; The program below uses this module, getting the root of the search as well as the search pattern from user input, and uses the module to get a listing of files found. Finally, the list of files together with the corresponding file sizes are displayed: 1 2 3 4 #!/usr/bin/perl -w use FileSearch;
Slide 142: 130 5 6 7 8 9 10 11 12 13 14 15 16 17 Chapter 8 Files and Filehandles my ($pattern, $dirroot); print "Please specify the filename pattern by means of a regular expression:\n¶ >> "; chomp($pattern = <STDIN>); print "Please specify the root of directory tree to be searched:\n>> "; chomp($dirroot = <STDIN>); my $searchobj = new FileSearch(’PATTERN’ => $pattern); my @result = $searchobj->find($dirroot); foreach (@result) { print $_, " (", (stat($_))[7], " bytes)\n"; } The transcript of a sample session is shown below. It searches for files whose extension is .pm. Therefore, a list of Perl modules in the directory tree is shown. cbkihong@cbkihong:∼/docs/perltut/src/files$ perl search.pl Please specify the filename pattern by means of a regular expression: >> ˆ.*\.pm$ Please specify the root of directory tree to be searched: >> /home/cbkihong/docs/perltut /home/cbkihong/docs/perltut/scrap/Number.pm (344 bytes) /home/cbkihong/docs/perltut/src/ch06/Controller.pm (916 bytes) /home/cbkihong/docs/perltut/src/ch06/Stats.pm (1264 bytes) /home/cbkihong/docs/perltut/src/ch06/Stats2.pm (779 bytes) /home/cbkihong/docs/perltut/src/ch06/Stats3.pm (425 bytes) /home/cbkihong/docs/perltut/src/ch06/TestObject.pm (198 bytes) /home/cbkihong/docs/perltut/src/ch06/Timer.pm (1011 bytes) /home/cbkihong/docs/perltut/src/ch06/TrafficLight.pm (642 bytes) /home/cbkihong/docs/perltut/src/files/FileSearch.pm (796 bytes) The chomp() function removes the trailing newline character from the variable containing an input string if one is present. Note that the argument must be an lvalue because it performs in-place editing of the string instead of returning the modified string. The FileSearch::find() method is a recursive subroutine which takes a single parameter, that is the base directory in which to search for files matching the specified pattern, which is passed to the FileSearch object upon instantiation. The search itself is a depth-first search. That means at each level, for each directory entry the content of the directory is searched before the next directory is searched. Note that we have to avoid . and .. in each directory. In Appendix D in my discussion of hard links I explain what these two entries are. They refer to the current and the parent directory, respectively. Because they go up instead of go down the directory tree, we should not delve into them. This program looks fine. However, this search implementation can cause the search to go into an infinite loop on Unix systems in a certain case. That happens if two path segments on a path point to the same inode. First, let us agree on the terminology first. A path like /a/b/c/d consists of four path segments a, b, c and d. Now, let us assume b and d point to the same directory inode (for an exhaustive explanation of inodes, please read Appendix D). Now c is a directory inside b. However, c
Slide 143: 8.5 File Test Operators 131 is also a directory inside d because b and d are actually just two aliases representing the same inode. Therefore, /a/b/c/d/c is still a valid path, and we can find the directory entry d inside, so we have the path /a/b/c/d/c/d. It is obvious that this circularity will continue endlessly, and obviously you will go into an infinite loop as a result. This is exactly the reason why we have to avoid delving into . and .. in the program. Therefore, this problem normally should not occur. It only arises when a hard link is created manually that makes two directories point to the same inode. The core difficulty is that you cannot easily identify this circularity in the program because d appears exactly like any other directory entries. You are hereby asked to think of a way to patch this potential problem. (hint: use stat()) 8.5 File Test Operators Perl provides you with a set of file test operators that you can use to test a file against certain properties and return a truth value. Table 8.2 lists the most commonly used operators: Operator -r -w -x -o -R -W -X -O -e -z -s -f -d -l -u -g -k Description File is readable by effective user or group. File is writable by effective user or group. File is executble by effective user or group. File is owned by effective user. File is readable by real user or group. File is writable by real user or group. File is executble by real user or group. File is owned by real user. File exists. File is empty (zero size). File is not empty, size in bytes as return value. File is a regular file. File is a directory. File is a symbolic link. File has setuid bit set. File has setgid bit set. File has sticky bit set. Table 8.2: File Test Operators Note that many of these options are specific to Unix operating systems. -e, -f, -d, -z and -s can be used on most operating systems. The only parameter is either a filename or, in case the file has already been opened, its filehandle. For example, to ensure a certain regular file (abc.txt) exists, you can issue the following statement: (-e "abc.txt" && -f "abc.txt") or die "abc.txt not found or is not regular file!\n"; Notice there are two sets of operators to test whether a file is readable/writable or executable. In general, you should use the lowercase version instead of the uppercase version because the lowercase version uses the effective user or group identity, which always reflect the identity of the user or group under which the program is executed. You can get further information on effective and real user and group in
Slide 144: 132 Chapter 8 Files and Filehandles Appendix D with respect to setuid or setgid files. 8.6 File Locking File locking, or locking in general, is just one of the various solutions proposed to deal with problems associated with resource sharing. Sharing of resources frequently arises in our everyday life. Driving on a highway, boarding an elevator etc. are all manifestations of utilization of shared resources. In a computer program, you — the developer — have access to disks, peripheral devices etc. within a program that appears to be self-contained, which in turn leads you into thinking your program has exclusive access to these resources. No, you don’t. Resource sharing is equally ubiquitous as, if not more than, in your daily life. In your computer system there may be a few hundred programs in execution (processes). Many of them are hidden so you may not know they are running. However, with just one Central Processing Unit (CPU) only one of them can be executed at any instant. Therefore, the CPU actually performs a context switch at regular, but very short intervals when a currently-running process is suspended from execution by the CPU with its execution state saved, and another process is resumed. This context switch is performed so frequently that to a user there is a perception that all processes are executed concurrently, but they’re not. This is already one example of resource sharing, and the solution is to introduce fine-grained time slicing. Similarly, all programs that are executing have access to the disks. Therefore, it is not surprising that when a program is accessing a disk it is possible you can find another one that is also trying to access the disk at the same time, too. File locks are introduced to set temporary restrictions on certain files to limit how they can be shared among different processes. Depending on the nature of an operation, we come up with two types of locks. The first type is a shared lock, and another one is an exclusive lock. With respect to files, read access can be shared by multiple processes because read access does not result in any changes to the state of the shared resource. Therefore, a consistent view of the shared resource can be maintained. Write access, however, should by nature be carried out with exclusive access until the write operation is complete. Another write operation to the file which occurs before the current one is complete should be queued (certain cases may require it be cancelled instead, depending on the nature of the operation). A read access cannot be concurrent with a write access either because the write access will destroy the consistent view expected by the read access while it is still reading (for example, the write operation may be to delete the file altogether, while the read operation is still in its midway). Therefore, the solution is to introduce shared locks for read access, while exclusive locks for write access. Multiple concurrent shared locks are allowed. However, exclusive locks are mutually exclusive. On most Unix systems as well as platforms in the Microsoft Windows NT family (including Windows 2000, XP and Windows Server 2003 as of this writing) you may use a handy flock() function. Please see the “Function Implementations” section of the perlport manpage to see any platform-specific issues for your platform. Here is an example demonstrating file locking: 1 2 3 4 5 6 7 8 9 #!/usr/bin/perl -w # writer.pl # Writes into myfile.dat use Fcntl ’:flock’; open(FILE, ">>myfile.dat") or die "Cannot open myfile.dat!\n";
Slide 145: 8.6 File Locking 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 133 print "[$$] Requesting exclusive write lock for myfile.dat.\n"; flock(FILE, LOCK_EX); print "[$$] Requested exclusive write lock for myfile.dat.\n"; print "[$$] Writing (appending) to myfile.dat.\n"; # Any write operations should be put inside foreach ($ARGV[0]..$ARGV[1]) { print "[$$] Writing $_\n"; print FILE "$_ (process id $$)\n"; sleep 1; # sleep one second } flock(FILE, LOCK_UN); # release lock print "[$$] Released exclusive write lock for myfile.dat.\n"; close FILE; #!/usr/bin/perl -w # reader.pl # Reads from myfile.dat use Fcntl ’:flock’; open(FILE, "<myfile.dat") or die "Cannot open myfile.dat!\n"; print "[$$] Requesting shared read lock for myfile.dat.\n"; flock(FILE, LOCK_SH); print "[$$] Requested shared read lock for myfile.dat.\n"; print "[$$] Reading from myfile.dat.\n"; while (<FILE>) { sleep 1; print $_; } flock(FILE, LOCK_UN); # release lock print "[$$] Released shared read lock for myfile.dat.\n"; close FILE; This example consists of two programs, namely the writer and the reader. The writer program writes some sample data into a data file, while the reader program reads from the data file previously written data. The writer takes two command-line parameters which determine the values to be written to the data file. It first acquires an exclusive lock. When the lock is acquired, it starts writing the values to the file. When the write operation is complete, the lock is released. The reader works in a similar manner. It first acquires a shared lock, and when one is acquired it starts reading from the file. At the end, the shared lock is released. To try this example, open two or more commnad-line windows on your desktop environment and try different combinations, such as a writer in a window and a reader in another (writer-reader), reader-reader and writer-writer. Take the writer-reader as an example, when you execute the writer program in one window by
Slide 146: 134 Chapter 8 Files and Filehandles perl writer.pl 1 15 That means 15 lines of records will be written into the data file. You will see that there is a 1-second pause between each write, because of the sleep() function. It is inserted after each write or read so that you can have ample time to switch to another window to execute another writer or reader instance. Now switch to another window and start the reader before the writer has finished writing. No command-line parameters are needed for the reader. You will find that the reader stalls while trying to acquire a shared read lock because an exclusive lock is already in place. When the writer completes, the reader will be able to get the shared lock and proceed with reading. If you try the reader-reader combination, you ought to find that concurrent readers are allowed, which agrees to what I told you earlier. flock() accepts two parameters. The first one is the filehandle. The second argument indicates the locking operation required. Just like the case for seek(), it is just an integer whose mnemonic constants can be found by importing the Fcntl module, by importing the symbol ‘:flock’. The shared lock is represented by LOCK SH and the exclusive lock by LOCK EX. To release a lock you can specify LOCK UN. You may optionally bitwise-or LOCK NB with either LOCK EX or LOCK SH to indicate non-blocking. As demonstrated in the example, an flock() call will be blocked until the lock is acquired. With this optional flag, you may choose to skip if it fails and do something else, or you may retry it later on. The predefined variable $$ refers to the current process ID. It is displayed as well as saved to file to let you easily identify the various processes. Summary ” ” A filehandle is a high-level abstraction in order to provide a uniform interface to I/O involving different media such as file access and socket communication. There are three predefined file handles: STDIN, STDOUT and STDERR. • • • ” ” ” ” ” ” ” ” ” User input through the keyboard is read from STDIN. Output of a program are usually directed to STDOUT. Error messages are usually directed to STDERR. Before performing any operations on a file, you have to open() it. You should close() it when you are done with it. Output redirection allows you to redirect data sent to a certain filehandle to another filehandle. The readline() function reads a sequence of characters from a given filehandle until a newline is encountered. The <> operator in Perl serves the same purpose. binmode() ensures binary files are not corrupted on MS-DOS based operating systems. read() reads a given number of characters from a filehandle. Every open file maintains a file pointer, which you may manipulate with the seek() function. The file pointer position may also be obtained by the tell() function. The print() and printf() functions output a string to the given filehandle. printf() performs some transformations on the string beforehand. sprintf() is identical to printf() except the resulting string to be returned instead of written to any filehandles. The readdir() function traverses a given directory, whose filehandle as been populated by the opendir() function. Upon completion closedir() is invoked to close it.
Slide 147: 8.6 File Locking ” ” File test operator can be used to test if a file possesses a given property. 135 File locking serves to set temporary restrictions on how certain files may be shared among several processes. The flock() function is used.
Slide 148: 136 Chapter 8 Files and Filehandles
Slide 149: Chapter 9 Regular Expressions 9.1 Introduction Now we are marching into probably the most exciting chapter in this tutorial. Regular expressions (regexps, or even RE) are what makes Perl an ideal language for “practical extraction and reporting”, as the name implies. To give you an idea, let me first give you an overview on how to perform pattern matching in Perl. First, you construct the regular expression, which is essentially a sequence of characters describing the pattern you would like to match. The term “pattern” may seem a little bit foreign to you, but you may actually have had some experience of it already. For example, in MS-DOS if you would like to list all files with the extension .txt (presumably text files), you may issue a command like dir *.txt On Unix-like operating systems, similarly, you can specify ls *.txt The “*.txt” here can be described as a pattern as it is the specification used by the operating system (strictly speaking, the shell, i.e. the program in charge of reading commands from you and displaying output) to look for the file entries that have to be displayed. This is a very simple pattern, but Perl provides users with a very powerful set of regular expressions that can be used to specify the patterns, so you can make your search specification more specific. After constructing the regular expression, you can then bind the data to be searched (for example, text files or just a line of user input) to the pattern using the binding operators =∼ or !∼. In this process, you have provided the Perl regexp engine with both the data to search for and the data to be searched. The return value indicates if pattern matching is successful. If it is successful, you may want to store the data temporarily, or in a file, or export the results directly to the standard output. Regular expressions are used in Perl in a number of ways: Search for a string that matches a specified pattern, and optionally replacing the pattern found with some other strings Counting the number of occurrences of a pattern in a string 137
Slide 150: 138 Chapter 9 Regular Expressions Split a formatted string (e.g. a date like 02/06/2001) into respective components (i.e. into day, month and year) Validation of fields received from submitted HTML forms by verifying if a piece of data conforms to a particular format ... and much more Regular expressions are not exclusive to Perl only. It is a vital component in Unix and other Unix-like operating systems like Linux to provide users with powerful text search and replace capabilities. You may find that many software on Unix, like grep and awk, allows users to use regular expression to specify search specifications (although their implementations are slightly different from that of Perl). Although this is still rare in Windows, many new software with regexp capabilities are emerging because many famous Unix applications are now being ported to Windows and other system platforms. Moreover, a large set of Perl regexps are now being adopted in JavaScript and JScript implementations in Netscape and Microsoft Internet Explorer respectively. This is a piece of good news as sophisticated input validation can now be performed directly by the end users’ browsers. In this way, invalid data can be detected without sending anything to the server that causes the transmission delays. In C/C++ there are also regular expression libraries that programmers can easily use for adding regexp capabilities to their programs. That explains how useful regular expressions are in programming nowadays. As you may know there are books dedicating in their entirety to regular expressions in your local bookstores. Therefore, this chapter is by no means a complete coverage of regular expressions in fine detail. However, after you have finished this chapter you should appreciate how flexible and powerful regular expressions are in Perl and in Unix-like operating systems, and how they can accomplish tricky text manipulating tasks on the fly. 9.2 Building a Pattern 9.2.1 Getting your Foot Wet Now you will learn to build a pattern using the regular expressions offered by Perl. To search for a pattern match, simply construct the pattern and put it in between the two slashes of the m// operator. If you don’t need the bells and whistles, for example, you just need to know if the characters “able” appear in any given string, the pattern is as simple as: m/able/ Let’s put this to a test. Now, to see if this pattern occurs in the string “Capable”, we bind the twos together by using the binding operator =∼. Try this script: if ("Capable" =∼ m/able/) { print "match!\n"; } else { # This should NEVER happen print "no match!\n"; } There is not many things special here. Because the pattern “able” is in the string “Capable”, the words “match!” will be displayed. I intentionally use the literal “Capable” in the example to show that although the symbol looks like an assignment operator, it is not necessary for a valid lvalue on the left