SSH and SCP: Howto, tips & tricks

This tutorial is about SSH and SCP. You will learn how to connect to a remote host and how to copy between hosts. This tutorial also documents a few important differences between the commands.

Difficulty: Basic

Before we start: in this tutorial, you will come across both SSH and ssh. The difference is this: SSH is the general protocol, and ssh is the linux SSH client command.

SSH

SSH is some kind of an abbreviation of Secure SHell. It is a protocol that allows secure connections between computers. In this tutorial, we'll be dealing with the ssh command on Linux, the OpenSSH version. Most Linux distributions feature the OpenSSH client today, but if you want to be sure, have a look at the SSH manpage on your system. You can do this by typing:

[rechosen@localhost ~]$ man ssh

Note: this should be done in a terminal. This tutorial assumes that you have some basic terminal knowledge, like knowing how to start a terminal session on your system and being familiar with the basic commands and syntaxes.

If it displays something like this

NAME
ssh - OpenSSH SSH client (remote login program)

then you can be quite sure you're running the OpenSSH version. For more background information about SSH, see http://en.wikipedia.org/wiki/SSH.

The most simple case

In the most simple case, you can connect to a server that supports ssh with a syntax as short as this:

[rechosen@localhost ~]$ ssh yourserver

Note: If you do not have any ssh server nearby that you can access, you can also try this command with your own computer as a server. To do this, replace "yourserver" with "localhost".

Of course, yourserver should be replaced by a hostname or an ip address of the server you want to connect to. As you can see in the terminal snippet, I am logged in as rechosen. If you do not specify a username (I'll explain how to do that later in this tutorial), SSH will assume that you want to login with the username you're currently logged in with. So, in this case, SSH will try the username rechosen.

Of course, you need to be sure that the server supports ssh connections. The ssh client tries to connect to port 22 defaultly. This means that, if you want to connect to a remote host with the default settings, you should make sure that, if applicable, port 22 is forwarded to the server you're trying to connect to. You will find more regarding the SSH port further in this tutorial.

Now, back to the command we ran. If the server supports SSH connections and you can reach it by port 22, you should be prompted for a password (if this is the first time you try to connect to the server, ssh will first ask the question if you want to continue connecting, which can generally just be answered with a 'yes'). If you type a password here, you won't see asterisks appearing. Don't panic, this is ssh's normal behaviour. It makes connecting using ssh even more safe, because any accidental spectators won't be able to see the length of the password. After entering the password, if the username and the password were correct, you should be running a shell on the server. If not, make sure you are connecting to a server of which you know that you should be able to login with your username and the specified password. You could try connecting to your own computer (see the note beneath the terminal quote) or read on to learn how to specify an other username.

Once you're done trying the ssh shell, you can exit it by pressing Ctrl + D.

Specifying a username

It's actually quite simple to specify a different username. You might even already be familiar with it. See the following example:

[rechosen@localhost ~]$ ssh yourusername@yourserver

The above will make ssh try to connect with the username "yourusername" instead of (in my case) rechosen. This syntax is also used by a lot of other protocols, so it'll always come in handy to know it. By the way, you will still be asked for a password. For security reasons, it is not even possible to directly specify the password in the syntax. You will always be asked interactively, unless you start configuring the server in an advanced way (which is exactly why that topic is out of this tutorials scope: this tutorial documents how to use the clients, not how to configure the server).

Specifying a port

There are many reasons to move the ssh service to an other port. One of them is avoiding brute-force login attempts. Certain hackers try to get access to ssh servers by trying a lot of common usernames with common passwords (think of a user "john" with password "doe"). Although it is very unlikely that these hackers will ever get access to the system, there is an other aspect of the brute-force attacks that you'll generally want to avoid: the system and connection load. The brute-force attacks usually are done with dozens or even thousands of tries a second, and this unnecessarily slows down the server and takes some bandwidth which could've been used a lot better. By changing the port to a non-default one, the scripts of the hackers will just be refused and most of the bandwidth will be saved.

As the ssh command can't just guess the port, we will have to specify it if it's not the default 22 one. You can do that this way:

[rechosen@localhost ~]$ ssh -p yourport yourusername@yourserver

Of course, you will have to replace "yourport" with the port number. These is an important difference between ssh and scp on this point. I'll explain it further on.

Running a command on the remote server

Sometimes, especially in scripts, you'll want to connect to the remote server, run a single command and then exit again. The ssh command has a nice feature for this. You can just specify the command after the options, username and hostname. Have a look at this:

[rechosen@localhost ~]$ ssh yourusername@yourserver updatedb

This will make the server update its searching database. Of course, this is a very simple command without arguments. What if you'd want to tell someone about the latest news you read on the web? You might think that the following will give him/her that message:

[rechosen@localhost ~]$ ssh yourusername@yourserver wall "Hey, I just found out something great! Have a look at www.examplenewslink.com!"

However, bash will give an error if you run this command:

bash: !": event not found

What happened? Bash (the program behind your shell) tried to interpret the command you wanted to give ssh. This fails because there are exclamation marks in the command, which bash will interpret as special characters that should initiate a bash function. But we don't want this, we just want bash to give the command to ssh! Well, there's a very simple way to tell bash not to worry about the contents of the command but just pass it on to ssh already: wrapping it in single quotes. Have a look at this:

[rechosen@localhost ~]$ ssh yourusername@yourserver 'wall "Hey, I just found out something great! Have a look at www.examplenewslink.com!"'

The single quotes prevent bash from trying to interpret the command, so ssh receives it unmodified and can send it to the server as it should. Don't forget that the single quotes should be around the whole command, not anywhere else.

SCP

The scp command allows you to copy files over ssh connections. This is pretty useful if you want to transport files between computers, for example to backup something. The scp command uses the ssh command and they are very much alike. However, there are some important differences.

The scp command can be used in three* ways: to copy from a (remote) server to your computer, to copy from your computer to a (remote) server, and to copy from a (remote) server to another (remote) server. In the third case, the data is transferred directly between the servers; your own computer will only tell the servers what to do. These options are very useful for a lot of things that require files to be transferred, so let's have a look at the syntax of this command:

[rechosen@localhost ~]$ scp examplefile yourusername@yourserver:/home/yourusername/

Looks quite familiar, right? But there are differences. The command above will transfer the file "examplefile" to the directory "/home/yourusername/" at the server "yourserver", trying to get ssh acces with the username "yourusername". That's quite a lot information, but scp really needs it all. Well, almost all of it. You could leave out the "yourusername@" in front of "yourserver", but only if you want to login on the server with your current username on your own computer. Let's have a closer look at the end of the command. There's a colon over there, with a directory after it. Just like Linux's normal cp command, scp will need to know both the source file(s) and the target directory (or file). For remote hosts, the file(s)/directory are given to the scp command is this way.

You can also copy a file (or multiple files) from the (remote) server to your own computer. Let's have a look at an example of that:

[rechosen@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile .

Note: The dot at the end means the current local directory. This is a handy trick that can be used about everywhere in Linux. Besides a single dot, you can also type a double dot ( .. ), which is the parent directory of the current directory.

This will copy the file "/home/yourusername/examplefile" to the current directory on your own computer, provided that the username and password are correct and that the file actually exists.

You probably already guessed that the following command copies a file from a (remote) server to another (remote) server:

[rechosen@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile yourusername2@yourserver2:/home/yourusername2/

Please note that, to make the above command work, the servers must be able to reach each other, as the data will be transferred directly between them. If the servers somehow can't reach each other (for example, if port 22 is not open on one of the sides) you won't be able to copy anything. In that case, copy the files to your own computer first, then to the other host. Or make the servers able to reach each other (for example by opening the port).

Well, those are the main uses of scp. We'll now go a bit more in-depth about the differences between ssh and scp.

*: Actually you can also use it just like the normal cp command, withhout any ssh connections in it, but that's quite useless. It requires you to type an extra 's' =).

Specifying a port with scp

The scp command acts a little different when it comes to ports. You'd expect that specifying a port should be done this way:

[rechosen@localhost ~]$ scp -p yourport yourusername@yourserver:/home/yourusername/examplefile .

However, that will not work. You will get an error message like this one:

cp: cannot stat `yourport': No such file or directory

This is caused by the different architecture of scp. It aims to resemble cp, and cp also features the -p option. However, in cp terms it means 'preserve', and it causes the cp command to preserve things like ownership, permissions and creation dates. The scp command can also preserve things like that, and the -p option enables this feature. The port specification should be done with the -P option. Therefore, the following command will work:

[rechosen@localhost ~]$ scp -P yourport yourusername@yourserver:/home/yourusername/examplefile .

Also note that the -P option must be in front of the (remote) server. The ssh command will still work if you put -p yourport behind the host syntax, but scp won't. Why? Because scp also supports copying between two servers and therefore needs to know which server the -P option applies to.

Another difference between scp and ssh

Unlike ssh, scp cannot be used to run a command on a (remote) server, as it already uses that feature of ssh to start the scp server on the host. The scp command does have an option that accepts a program (the -S option), but this program will then be used instead of ssh to establish the encrypted connection, and it will not be executed on the remote host.

Tips & Tricks with ssh and scp

Quite a handy thing about scp is that it supports asterisks. You can copy all files in a remote directory in a way like this:

[rechosen@localhost ~]$ scp yourusername@yourserver:/home/yourusername/* .

And you can also just copy a whole directory by specifying the -r (recursive) option:

[rechosen@localhost ~]$ scp -r yourusername@yourserver:/home/yourusername/ .

Both of these also work when copying to a (remote) server or copying between a (remote) server and another (remote) server.

The ssh command can come in handy if you don't know the exact location of the file you want to copy with scp. First, ssh to the (remote) server:

[rechosen@localhost ~]$ ssh yourusername@yourserver

Then browse to the right directory with cd. This is essential Linux terminal knowledge, so I won't explain it here. When you're in the right directory, you can get the full path with this command:

[rechosen@localhost ~]$ pwd

Note: pwd is an abbreviation of Print Working Directory, which is a useful way to remember the command.

You can then copy this output, leave the ssh shell by pressing Ctrl + D, and then paste the full directory path in your scp command. This saves a lot of remembering and typing!

You can also limit the bandwidth scp may use when copying. This is very useful if you're wanting to copy a huge amount of data without suffering from slow internet for a long time. Limiting bandwidth is done this way:

scp -l bandwidthlimit yourusername@yourserver:/home/yourusername/* .

The bandwidth is specified in Kbit/sec. What does this mean? Eight bits is one byte. If you want to copy no faster than 10 Kbyte/sec, set the limit to 80. If you want to copy no faster than 80 Kbyte/sec, set the limit to 640. Get it? You should set the limit to eight times the maximum Kbyte/sec you want it to be. I'd recommend to set the -l option with all scp'ing you do on a connection that other people need to use, too. A big amount of copying can virtually block a whole 10 Mbit network if you're using hubs.

Final Words

Well, that was it! I hope you learned a lot. Of course, you can always have a quick look at this tutorial again if you forgot something. Please tell other people who might be interested about this tutorial, you'll help this blog to grow if you do =). Thank you for reading and have a lot of fun with your new knowledge!

Create a Compelling Resume Online With WordPress

Do you stand out?

Here’s a news flash – the economy is a little rough these days.  With the unemployment rate creeping up, it’s a time when you need to be clear about what you can bring to a prospective employer or client and, above all, to stand out from the competition.  Fortunately, it’s cheaper and easier than ever to make yourself “present with authority.”

One of my new year’s resolutions was to get my online identity sorted out, which means making the most of the available tools and delivering a consistent message. After taking inventory of the various professional sites to which I belong (e.g. LinkedIn, VisualCV), I decided that I needed an aggregator to take charge and deliver my message, my way, in my style. I had registered my own name as a domain many years ago but not done anything with it, and so I decided to use WordPress to tell my professional story.

What Are the Benefits?

If you think about your career as a product that you’re selling, wouldn’t it make perfect sense to have a web site? Of course it does, and you want your personal brand to have the same benefits:

• More and more, clients and employers are performing online searches to learn more about the people with which they are considering to engage. Having a search engine friendly web site makes it more likely they will find you.
• Indexing your experience through the use of keywords makes it easy for people to zero in on the skills and/or expertise in which they’re interested (more about that later).
• Multimedia capabilities (i.e. images, video, presentations, links) make it easier for you to tell your story in a vivid and interesting way.
• You can use specialized links to direct people to specific content areas of your profile.
• While this practice may soon be common place, for the moment at least it will help you to stand out from the crowd.

Laying the Groundwork

The first decision is your domain name. One option is to use Blogger or WordPress.org (e.g. jondipietro.wordpress.org), which is free. However, for the few dollars a year it costs you are far better off registering your own domain name; preferably your first and last names if available. This article will discuss building your online resume using WordPress on your own hosted site.
Once the basic WordPress installation is in place, the first thing you’ll want to do is to find a clean, professional theme and install it. There are a number of plugins that I install on every WordPress site right out of the gate:

• Add to Any Sharing and Subscribing buttons: Makes it easy for visitors to share, save, bookmark, and/or subscribe to your site.
• All in One SEO Pack : Provides a bunch of handy tools to make your site more search engine friendly.
• Contact Form 7 : Create customized forms for visitors to send you messages.
• Exclude Pages from Navigation : Allows you to easily build landing pages.
• FeedBurner FeedSmith : Synchs your blog with FeedBurner.
• Google Analyticator : Installs the Google Analytics script on your site so you can track traffic.
• Google XML Sitemaps : Creates the XML site map files that makes it easier for Google to discover pages on your site.
• Simple Tags : Makes it really easy to manage the tags that you’ll use to mark your areas of expertise.
• SlideShare : Allows you to embed SlideShare presentations.
• Social Links : Display buttons with links to your social networking sites in the sidebar.
• WordPress Database Backup : Ounce of prevention, and all that…
• WP-Mail-SMTP : Reconfigures the wp_mail() function to use SMTP instead of mail().

Turn the Page

Now it’s time to set up the pages.  I decided on the following site map:

• Home
• About Me
  • Work History
  • Skills
  • Volunteerism
• Experience
• Companies
• Social Networks
• Contact Me

The tricky thing about this is that, by default, WordPress publishes blog articles to the front page of the site.  In order to implement my strategy, it’s important to change that.  You can do this on the Reading Settings page by selecting “Home” for your front page and “Experience” as the posts page.

Change the default settings for the front and posts pages.

Next, you’ll need to populate the Home, About Me, Work Experience, and Skills pages.  The Home page functions as a sort of generic cover letter, while the other pages represent the customary sections of a resume.  However, you have the freedom to be a little more creative and verbose in this environment than on a paper resume.
What you don’t see in the site map (or in the menu) is my social networking landing page.  Another benefit of having your own web site is the ability to create landing pages from other sites that allow you to customize messages and, again, tell your story.  For example, Twitter provides very little space to customize your profile but they do allow you to enter a web site URL that you can point to a customized landing page.  This is where the “Exclude Pages from Navigation Menu” plugin comes in handy.

Rubber Meets Road

Skills, experiences, clients, etc... are displayed in the tag cloud widget.

The power of this approach now takes shape as you create blog posts to describe specific projects, publications, and experiences.  The key is carefully selected categories and liberal use of tags for the posts.  This will allow employers and/or clients to quickly zero in on the topics in which they are interested.  And since you’re making it easy to find the information they’re looking for, you can feel free to include lots of details, making them as interactive as possible.  You’ll want to include the Simple Tags tag could widget in the sidebar.
The categories are also important and you can use them to organize your pages and posts at a higher level than the tags.  How and where the categories are displayed is somewhat dependent upon your theme.  Some themes display them as menu itmes while others are displayed in the sidebar and is a matter of your personal preference.

Shout It Out Loud

Once your online resume is good to go, you can send out customized links to direct people straight to a particular area of interest.  For example, if I’m looking to secure a consulting contract for VB.NET development, I could send the following in an email:

“Please see the VB.NET projects listed on my online resume.”

The “VB.NET” link is http://www.jondipietro.com/tag/vbnet, which will automatically display a list of all pages or posts that were tagged with the VB.NET keyword.  You can begin to see how easy it is to send customized links to employers and clients that zoom right in to the areas on which you want them focused.

Conclusion

As I mentioned, I didn’t initially set out to create an online resume but once I started putting the idea together it became clear how compelling and useful this approach can be.  But I’m interested to hear about other creative ideas for leveraging this medium, so leave some comments.  Oh, and feel free to share a link to your own online resume.

How to fix the ‘The plugin does not have a valid header’ error when activating a WordPress plugin

You want to extend WordPress’ functionality and decided to download some plugins found in the Official WordPress Plugin repository. After searching the repo, you’ve found two plugins that accomplishes the tasks you needed to implement in your blog. Downloading the plugins gives you two ZIP files that when double-clicked, shows you the contents in the following format:

plugin-a.zip

plugin-a-folder
---- plugin-a-main-file.php
---- plugin-a-readme.txt

plugin-b.zip

plugin-b-folder
---- plugin-b-subfolder
-------- plugin-b-main-file.php
-------- plugin-b-readme.txt

There are two ways to install a plugin. Option 1 is to unzip the file and upload the contents to wordpress_root/wp-content/plugins/ directory. Option 2 is to use the Add New -> Upload option found in the left panel bar of the administrative page of WordPress. Let’s assume the two plugins went through the two installation options.

Installing ‘plugin-a.zip’ using Option 1

1. Unzip the file
2. Upload the contents to wordpress_root/wp-content/plugins/
3. Login as admin and go to Plugins -> Installed.
4. Locate plugin and click Activate link.
5. Result:
   Plugin activated.

Installing ‘plugin-a.zip’ using Option 2

1. Login as admin and go to Plugins -> Add New -> Upload
2. Locate the file plugin-a.zip by clicking the Browse button.
3. Click Install Now button.
4. Result:
   Unpacking the package?
   Installing the plugin?
   Plugin installed successfully.
5. Click Activate Plugin link.
6. Result:
   Plugin activated.

Installing ‘plugin-b.zip’ using Option 1

1. Unzip the file
2. Upload the contents to wordpress_root/wp-content/plugins/
3. Login as admin and go to Plugins -> Installed.
4. Locate plugin and click Activate link.
5. Error: Plugin does not exists in the list.

Installing ‘plugin-b.zip’ using Option 2

1. Login as admin and go to Plugins -> Add New -> Upload
2. Locate the file plugin-b.zip by clicking the Browse button.
3. Click Install Now button.
4. Result:
   Unpacking the package?
   Installing the plugin?
   Plugin installed successfully.
5. Click Activate Plugin link.
6. WordPress ? Error
   The plugin does not have a valid header.

Based on our testing, plugin-b.zip seems to fail on both methods of installation. To find out what is causing the error, we have to understand how the plugin installation works. Upon checking the core files I found this excerpt in get_plugins() function:

* WordPress only supports plugin files in the base plugins directory
* (wp-content/plugins) and in one directory above the plugins directory
* (wp-content/plugins/my-plugin). The file it looks for has the plugin data and
* must be found in those two locations. It is recommended that do keep your
* plugin files in directories.

The plugin data that the function is looking for can be found below:

01

02

/*

03	Plugin Name: Name Of The Plugin

04	Plugin URI: http://URI_Of_Page_Describing_Plugin_and_Updates

05	Description: A brief description of the Plugin.

06	Version: The Plugin's Version Number, e.g.: 1.0

07	Author: Name Of The Plugin Author

08	Author URI: http://URI_Of_The_Plugin_Author

09	License: A "Slug" license name e.g. GPL2

10

*/

11

?>

So in order to properly install and activate a plugin, the following two conditions must be satisfied:

• that a plugin main file (.php) must be placed in plugins root folder or in first-level subdirectory within plugins folder
• and it should contain the necessary plugin data for identification/validation purposes

In the case of plugin-b.zip, although the main PHP file contains the needed plugin data, the validate_plugin() function returned an error since neither the file can be found or check if it contains the plugin data because it is placed in the second level subdirectory, a directory location in which the function is not designed to scan the content.
To fix the plugin-b.zip file, it needs to have the following content structure:

plugin-b.zip

plugin-b-folder
---- plugin-b-main-file.php
---- plugin-b-readme.txt

I hope this post will enlighten WordPress users who wants to maximize their blog’s potential

What is new in Joomla 1.7 vs Joomla 1.6?

The Joomla team  recentlyannounced Joomla 1.7 is now stable for live sites.  This new version fixes a variety of issues with Joomla 1.6 as well as introduces the ability to upgrade to future versions from within the Administrator panel.

No more uploading via FTP, hooray!

This marks the first step towards a upgrade system that is similar to WordPress, thus making your life as an administrator easier.  The Joomla 1.7 release does not bring a huge amount of changes, but it is a vital step in the right direction for Joomla and it’s new release schedule.  In this article I will cover what is new, if you need to upgrade, and why I used such an ugly graphic for this post.

What are the newest and biggest features?

• One click upgrades – Starting from Joomla 1.6 you can upgrade your install by going to:  Extensions -> Extension Manager -> Update -> Find updates.
• Faster release cycles - Short term releases will be changing every six months and only supported for one month after the next release.
• Multi-Language improvements – Associate menu items to different languages.

There are a variety of other minor changes that you can review on the officialJoomla 1.7 page.

Thank you Joomla

Do I really need to upgrade?

Depending on what version of Joomla your site(s) are on now, this may be the last time you have to do a “scary upgrade”.   Scary meaning doing it all via FTP and hoping you didn’t miss any upgrade instructions or caveats.  Upgrading is really easy since J1.6, so this shouldn’t be a huge deal.

It’s also important to note that Joomla 1.6 will go end of life in August 2011.   That means no more security updates or patches or fixes for vulnerabilities.  If your site get’s hacked 6 months from now due to some kid developing some obscure technique, you will only have yourself to blame–not Joomla.  Upgrade them sites!

I am on Joomla 1.5.  How do I do it?

There is no upgrade path from J1.5 to J1.6 or J1.7.   You will have to migrate.  There are a variety of changes from J1.5 to newer versions including changes to tables and the ACL (access control list) so a migration is the only way.  If you are comfortable with migrating from different Joomla releases you probably have a system in place.   If not, you can try out this extension.  I personally have not tried it, but it seems pretty straightforward.  Take all the usual precautions and backup your databases and files before doing something like this.   Try it on your development site first and test the process before doing it on a live site.

I am on Joomla 1.6.  How do I do it?

It’s easy peasy if you already made the leap to Joomla 1.6.   Just go to: Extensions -> Extension Manager -> Update -> Find updates.

What’s up with that ugly Joomla image you used in the beginning of the article?

Kids refrigerator magnets are cute, cheap and fun.  They also stick to anything metal.  Slapping some fridge magnets on a whiteboard or something metal or *gasp*, your actual fridge is a good way to nag yourself that this upgrade is very important and needs to be done soon.

Do I have to worry about extensions breaking?

As with any upgrade there is always the potential for things to break.  You should never do a upgrade or migration without some tested and proven way to revert back to a working site if it all goes pear shaped.   The changes from J1.6 to J1.7 are pretty minor, but ther are so many extensions that do so many things, it’s hard to make a blanket statement.

How about you?

Do you have any tips or tricks I didn’t cover? Let us know in the comments, we would love to hear from you.