Cleaning up a directory – an example of chaining Unix commands

You might think that Unix is some old school way of using computers that is slowly fading away, but having come back to computing (after living in the highlands of Scotland with no particular reason to use a computer for nearly three years), I’d have to say Unix’s presence and utility is as strong as ever (and still isn’t overshadowed by its alternatives). It doesn’t appear to be going away.

What makes it so useful?

I think Unix’s strength now is probably what at some point was considered its weakness.  Unix goes back to when screens were just becoming common, a GUI was only added later when computers got fast enough to support it – so there’s a goldmine of tools that were created without the expectation of a GUI for solving a problem.
Unix seems over engineered sometimes, because it’s been so well engineered. You can do as much, or as little with it, as you want, it can be easily scaled up or down according to your needs – so it really lends itself to technologies like Docker, which are all about minimal installs (or at least installs made up of no more than you need for a particular task). Its maturity has made it very stable. Unix can be installed to live within the limited resources of whichever dusty old PC you can find (or VM that you can afford).  Possibly it’s still around because its security model works extremely well – it’s not unhackable, but it’s rare to hear about users accidentally installing malware (it’s still a risk though).

The underlying ethos

One critical idea behind Unix (it’s Command line or “shell”) is the idea that the output created from running one command should be able to be passed into another command. Therefore commands can be “chained” together to create complex operations. For instance you could create script that monitored for specific files or email attachments – you could have a script which retrieved the file, tested whether it contained a particular, string or name, process the file, such as filtering the lines of interest, then pass those lines to another command to sort them (say by a particular column), pass the results to an email utility as an encoded attachment and have it sent to your inbox. Not every command in Unix allows the chaining of input and output, but many do.
Another important ethos about using Unix is to appreciate that you should always aim to use the level of access you need to do your tasks and no more. I’d love to tell you about the colleague who took down the organisation’s critical database by accidentally deleting Oracle database files that he thought he couldn’t delete (that was the day we had the root password taken away, and we learned how we didn’t need to know it to do our jobs) – but it’s a long story and I’ll save it for another time.

What do you need to know about it?

You can achieve an awful lot with Unix without needing to be an engineer (or even a System Administrator). You really only need to know a handful of commands to begin with. You’ll definitely want some kind of text editor. And you’ll want at least an awareness of the kinds of tools (utilities) that are commonly found to help you automate or solve problems. At some point you’ll want to know about scripting tasks, and possibly scheduling them. Eventually you might want to know about how storage (or new devices) are installed and how the system boots up and can be shutdown (safely).

What’s the best way to learn Unix?

I’d say you need to have a problem to solve or at least a thing you want to achieve. Maybe think about setting up a local webserver for yourself. Sure you could do it quickly in Windows, but you’re trying to teach yourself something about Unix along the way. Of course, there’s YouTube, but I find you’ve really got to use new tech to learn it, watching will only get you so far.
I’ve never enrolled in a course, as such, to learn Unix; way way back in the dim distant past there were simply text only interactive tutorials that you could find “online” before there was a World Wide Web. Now, all you have to do is Google them. Here’s a couple that are certainly worth a gander:
I’d start looking at these two, a quick glance says they’re good:
http://www.ee.surrey.ac.uk/Teaching/Unix/
and
https://www.tutorialspoint.com/unix/
Based on how I feel I’ve come to know about Unix, I’ve compiled a kind of roadmap for learning, from the essentials to the progressively esoteric. Some will be familiar if you’ve ever used DOS, or possibly some will just be obvious.

Books

There are many, many books published on learning Unix; O’Reilly have a particularly rich selection of books on using Unix and Unix tools. Books about Unix are a nice exception to the general rule that you can throw out a technical book after a year – most flavours of Unix have not changed significantly over time, or at least they’ve settled down over time. (Having said that, this rule of thumb doesn’t seem to apply to OSX, because it regularly changes.) Two traps to be aware of – there are “families” of Unix which contain significant differences to each other (such as how the OS is configured for startup and shutdown and the location and naming of critical files) – so make sure the book you pick up is relevant to the Unix you need to learn about. And the are differences in the many “shells” (or command line environments) that you’ll encounter –  the differences are apparent as soon as you want to automate something. The choice of shell is more about personal taste; something you won’t necessarily ever care about – or may not care about for a very long time. If you’re going to learn  a Linux flavour of Unix – which is quite likely, then you’ll almost certainly want to learn about the Bash shell (but not necessarily).

Essential things to be able to do:

I’ve come up with my own list of things I (a Unix user) should know how to do, it’s not exhaustive and it’s not aimed at a person who wants to be a SysAdmin. It’s intended to just give a short roadmap of tasks that I believe are worth knowing, and a kind of progression to help you work out where to start.

Very Basic:

• Change directories
• List the contents of directories
• Know how to “filter” specific filenames
• View the contents of file
• Know how to retrieve the built in help found in most commands

Slightly less basic, but not too challenging:

• Know how to create a file from a command
• Be able to perform simple editing on files. (Choose an editor, probably vi or emacs, but don’t bother with both.)
• Become familiar with the “find” command
• Try the “grep” command
• Learn how the file permissions work – especially how umask maintains file permissions
• Be able to upload or download a file
• Know how to change your password

Slightly harder, but bordering on essential

• Know how to edit a file with commands like search and replace, how to add lines, and overwrite lines. Be confident with “bulk” commands like deleting ranges.
• Try more advanced options in “grep” like “OR” and “AND”.
• Know a little about AWK and/or SED – really powerful utilities for automation or repetitive tasks. You’ll want to learn about Regular Expressions (aka regex).
• Find out how to suppress STDERR messages so that files you create from command output is cluttered with messages you don’t want to see.
• Learn how to script – ultimately this is what it’s all about. How to free up your time with automation – or how to ensure something is done the same way every time. Deciding to learn about scripting rapidly get complicated because of the number of spin-off “flavours” of Unix and myriad of variant shells that work under them.
• Know how to capture the screen output and error messages from commands and scripts.
• Learn about CRON (scheduling).
• Understand how to identify a process (program) that needs to be “KILL”ed and how to kill a process. And also understand “top” so you monitor what’s happening on your system.

You may find that quite a few blogs and very often authors of books will want to tell you about the history of Unix. Knowing the history of Unix isn’t important for knowing how to use it (it can certainly help), but being aware of how it came to have so many spin-offs certainly makes it easier to accept that there can be a confusing number of ways to solve a problem – or that there may be times when things will work differently to what you’d expected. (Eg Shells, scripting languages, file organisation, and the startup and shutdown processes.)
Once you have this stuff under your belt, you could start to think about System Administration; how users are added, how groups are managed, how software is installed (and where it installed). How the kernel and device drivers are configured. How processes are monitored.
As with a lot of computing , learning Unix is pretty much endless. I’m still learning new stuff, but a really great thing about Unix is that the same philosophy applies to all the commands so learning new is always building on the foundation of what you can already do – it’s not like a lot of IT where the next new thing means you should throw away or forget what you already know. There’s always more than one way to solve a problem, whether there is a “best” solution is really only something you can know by gaining experience.
Then there’s also the huge wealth of utilities that most people will never need, but you can always learn . Beyond the everyday tools you’ll also find: command line tools for manipulating images – like resizing them or adding a filter, desktop publishing tools, PDF and PostScript manipulation tools, utilities for keeping files synced across machines (and across OSes). It’s pretty endless actually.
So, I’ve said a lot about Unix here, without actually showing you how to achieve anything. Rather, I’ve jotted down a list of things I think you should find out about – and overtime, I’ll flesh them out as posts. Eventually this page will be a really helpful starting place for learning about Unix, rather than just a list of things for you to go Google for yourself.
Your first task is to look at the links I included at the top and start learning.
Don’t know how to get your hands on Unix? Have a look at Docker – and install get a flavour of Unix to get started.
Slàinte mhath, Nick.