If you're not familiar with any of the following, here's where to go to
get started.
See also Chris Matzner's resources for the beginning astronomy researcher
Unix/Linux: an operating system and simple programming environment.
- Start with The Official Red Hat Linux Getting Started Guide at
this archive of the Redhat website.
(Redhat is one version of Linux.)
- To edit a text file from the command line in Linux, you can use either Emacs or Vi/Vim.
You can read about them both at
this archived site from the Redhat website.
(Prof. Hall can answer questions about Vi/Vim, but not about Emacs.)
- See the next two sections for an illustration of some basic linux commands.
Useful Linux & IRAF commands
- When you open a terminal window on a Mac, your working directory is your home directory: /Users/username
- type 'cd NAME' to change the working directory to the subdirectory NAME in the current directory. You can move through more than one directory at once; for example: 'cd NAME/SUBDIRECTORY'. You can move directly to a specific directory; for example: 'cd /Users/Shared/SDSS/'.
- If you don't have permission to move to a directory, or to open or edit a file, you can fix that if you have Administrator privileges. The 'sudo' command lets you override permissions by entering your password, and the 'chmod' command lets you change the privileges on a file or directory. To gain access to a directory or file called 'name', type: sudo chmod a+rwx name.
- type 'man' to get help for a Linux command; e.g., 'man cd' or 'man sort'
- type 'help' or 'phelp' to get help for an IRAF command; e.g., 'help sort'
- Note: Linux sort & join are similar to, but DIFFERENT in detail, than IRAF sort & join!
- Linux: cat, grep, join, sort. Try them all!
- IRAF: average, filecalc (in the CTIO package), join, sort.
- To view a text file without editing it, use 'more' or 'less' in Linux,
or 'page' in IRAF.
-
'tar' creates binary archives of groups of files - very useful for moving files
from computer to computer. Use 'tar cvf output.tar inputfile1 ... inputfileN'
to create the archive output.tar (c for create, v for verbose, f for filename);
'tar tvf output.tar' to list back the files (t for Table of Contents);
and 'tar xvf output.tar' to extract all the files (x for extract) or
'tar xvf output.tar inputfilename' to extract just the named file.
You can gzip (compress) a tar file, but you must gunzip (uncompress) it
before extracting anything, unless you use the z flag: 'tar xvzf output.tar.gz'.
- Redirection and Piping:
You can redirect the output of a task to a file, or pipe the output of a task
to another task. To redirect to a new file, use '>'. To append to an existing
file, use '>>'. To pipe, use '|'.
(These commands are identical in Linux and IRAF. Note: Linux will give you an
error if you're trying to append to a nonexistent file, while IRAF will just
treat '>>' as '>' in that case.)
- In IRAF, if you are getting strange errors, for example if you've exited a
task using ctrl-C, run the 'flpr' command ('flush processes') three times.
You can run multiple commands on the same line by separating them with ';'
(a semicolon): 'flpr ; flpr; flpr'.
- 'grep' is a powerful search tool for finding strings or 'regular expressions'
in files. To print out all lines which contain the string 999 with a space
before it in a file, do: grep ' 999' filename. To print out all lines which do
NOT contain the string 999 with a space before it, do: grep -v ' 999' filename.
Use '^' for the start of a line and '$' for the end of a line, so:
grep '^ 999$' filename will return all lines that consist solely
of a space followed by the string 999. Do 'man grep' or search the internet
to learn more about the awesome power of grep.
Be aware that using 'grep' from Linux and '!grep' from IRAF sometimes give
different results (for reasons I've never understood). The same applies to sed.
- 'sed' (Stream EDitor) is a powerful utility for formatting and
manipulating text & numbers.
For example, let's say you need to format a Right Ascension + Declination string
"123456+780900" into the format "hh:mm:ss +dd:mm:ss". You can do that by running:
sed 's/\(^..\)\(..\)\(..\)\(.\)\(..\)\(..\)\(..\)/\1:\2:\3 \4\5:\6:\7/' < oldfile > newfile
and "newfile" will contain the line "12:34:56 +78:09:00".
Yes, the syntax takes some getting used to, but you can get help by typing
'man sed' or 'info sed' or reading
this sed introduction & tutorial
at least through the "Is sed recursive?" section, including
this tutorial on regular expressions
(at least through the "Backreferences" section) when it is referenced in the sed tutorial.
Efficiency
- In Linux, you can use the up-arrow key to revisit earlier commands;
hit return on any one of them to repeat the command (you can even
edit them before running them, if you want to fix a typo for example).
- In IRAF, typing "e" for "ehistory" will bring up the previous command,
and then you can use the up-arrow key to revisit earlier commands.
You can even type "e squiggle" and it will bring up the most recent
command line starting with "squiggle".
- In Linux, typing '!' before a command will repeat the last incidence of
that command. Any trailing characters will be added to the command.
So if you did 'ls april*' to find all files starting with 'april',
you could then do '!ls may*' instead of 'ls april* may*' to find
all files whose names start with 'april' or 'may'.
- Tab completion: in Linux, whenever you hit the tab key while typing a
command, it will display the longest unique string fitting what
you've already typed, so you don't have to type the full filename
of a long file.
- In IRAF, you can set aliases for commonly used directories in your
'login.cl' file. If you look at that file, you'll see 'set home = ...'
near the start of the file; that line lets you type 'cd home' and
return to the home directory. So you can add a line that reads
"set work = /some/really/long/chain/of/directory/names/" and
then just type 'cd work' to get to that directory. Note that like all
updates to login.cl, it won't take effect until the next time you start
IRAF; however, you can type 'set work=...' on the command line to use
the alias in your current IRAF session.
Communication between computers
- Linux to Linux: ssh (secure shell) is used on one linux computer
to open a text window on another linux computer. For example, from a window
on ara I use 'ssh phall@fornax.phys.yorku.ca' to open a window on fornax.
(I could just use 'ssh phall@fornax' since they are both in the phys.yorku.ca
domain.) If I start IRAF on fornax, IRAF graphics appear on ara just fine,
but image display using ds9 is trickier to configure; see the IRAF website.
- Windows to Linux: ssh can be used on a windows machine to open
a window on a linux machine, although graphics display doesn't work. If
`putty' or `F-Secure Shell' doesn't already exist on your windows machine,
type 'putty download' into google, and download the putty ssh executable to
your Windows computer. Running the putty version of ssh is self-explanatory.
- scp (secure copy) is used to copy files from one linux computer to
another.
To copy a file from my current directory on my current linux machine to my home directory on felis, I would type 'scp filename phall@felis.phys.yorku.ca:'.
To copy files into a particular Shared directory on felis to the same directory on manis, I would open a terminal window on manis and type
'cd /Users/Shared/directory_name/' and
'scp phall@felis.phys.yorku.ca:/Users/Shared/directory_name/filename .'
(the '.' means the current directory).
You can find out your current working
directory on your current machine by typing 'pwd' [print working directory].
From Windows to Linux, use `F-Secure Shell' or 'pscp' [Putty scp] instead of scp.
- Windows to Linux: Xwin32 lets you bring up the full screen of a
linux computer on your windows machine.
- There are other communications options, but the above are the ones I'm
familiar with.
LaTeX
- Latex is a text formatting language: you write a mixture of text and
latex commands in a text file and then run the program 'pdflatex' to turn
that file into a document in PDF format.
- The following are two basic introductions to latex: you can
start at the beginning of this one, or
start with the Text Modes section of this one.
- Here's an overivew of Including graphics in a LaTeX document
- PDF files can be viewed and printed using the 'acroread' command.
- Presentation-format PDF files can be created with the "pdfscreen.sty"
style file. A very short introduction to this style file can be found in
this PDF file,
or you can recreate a sample presentation found on fornax as "/home/phall/sample.tar".
That "tar file" is a binary archive; see tar under UNIX commands above for how to extract files
from it. The 'sample.tar' archive contains pdfscreen.sty, sample.tex, and all the jpg files needed to
create sample.pdf by typing 'pdflatex sample' after you've extracted all the files. Edit sample.tex
to create your own presentation... you can create jpg files from other formats such as PostScript
by using 'convert -density 300 image.ps image.jpg'; type 'man convert' for details.
-
Writing & Speaking Guidelines for Engineering & Science
IRAF (Image Reduction and Analysis Facility):
a computer language for astronomical research.
- The IRAF home page is iraf.noao.edu, but be
warned that some parts of the website are truly ancient. By clicking on
'All Docs' under 'Documentation' in the left-hand panel, you'll go to a web
page with links to documentation on a variety of subjects... as you work on
different projects over time, you'll need to read more & more of these manuals.
And you can always search iraf.noao.edu and
IRAF.net for help if you have problems.
- To start, click on 'Recommended Documentation for every site'.
You'll want to read A Beginner's Guide to Using IRAF; if you want more after
that, read A User's Introduction to the IRAF Command Language.
- If you click on 'A Beginner's Guide to Using IRAF', most likely it will not
display because your internet browser doesn't recognize the format it's in.
Here's how to view it:
- Save the file with its default name of 'beguide.ps.Z'
- The '.Z' and '.gz' suffixes indicate compressed files.
To uncompress such a file, use 'gunzip': 'gunzip beguide.ps.Z'
(To create a .gz file, use 'gzip'. Don't worry about creating .Z files.)
- If you type 'ls' to list files in your current directory, you'll now
see the uncompressed file 'beguide.ps'. The '.ps' suffix indicates a
PostScript file.
- PostScript files can be viewed using 'gv' [or 'ggv']: 'gv beguide.ps &'.
The ampersand (&) tells the computer to run the gv program in the background,
so you can continue to run programs from the window while 'gv' is running.
[Try it without the ampersand to see the difference.]
- If the resolution of the text in the gv window is poor, hit 'a' in the
gv window. To quit the gv window, move the cursor to it and hit 'q'.
- To print a PostScript file, just type 'lpr beguide.ps' [lpr stands for
laserprinter] and it will be sent to the printer 'gutenberg', right in front
of the door to room 336. You CANNOT use lpr to print a compressed file
(ending in .Z or .gz or .zip); always uncompress the file first!
(Also be sure to print PDF files from a PDF viewer, since
'lpr file.pdf' will just print pages of junk on many printers.)
- Obscure but useful: IRAF usually disables the backspace key, so you have
to use the delete key to erase text. If you want to use the backspace key to
erase text, type (or put in your login.cl or loginuser.cl file) '!tset -e^H'.
(The '!' tells IRAF that what follows it is a Linux command, not an IRAF command.)
- Resetting some basic IRAF parameters. The default values of some
common IRAF parameters can be a bit dangerous, so I always reset them. Once
you've learned how to use the 'epar' command, go ahead and do the following
(you have to do it separately on each computer on which you run IRAF):
- 'epar dir' and set 'maxch=38' [so you can see long filenames in full]
- 'epar del' and set 'verify=yes' [won't delete files without asking you]
- 'epar imdel' and set 'verify=yes' [won't delete images without asking you]
- 'epar imren' and set 'verbose=yes' [tells you when images are renamed]
- 'epar movef' and set 'verbose=yes' [tells you when files are moved]
- 'epar copy' and set 'verbose=yes' [tells you when files are copied]
- 'epar imstat' and set 'fields=image,mean,mode,stddev,min,max' and 'binwidt=0.01'.
[for technical reasons I won't go into now]
- If you need to install IRAF on a linux machine, see see this website. Note that you should install external packages STSDAS/TABLES 3.13 instead of 3.12 (if you install them at all).
Making Plots
- The 'xgterm' window allows you to display graphics in IRAF. You can start
up a new xgterm by typing: 'xgterm -sb -bg black -fg yellow &' ... the '-sb'
gives you a scrollbar, and 'bg' and 'fg' control background & foreground colors.
- Use the 'fields' command to extract specific columns from a file, and then
pipe those columns to the 'graph' task to make a simple plot.
So 'fields filename 1,3 | graph' would plot column 1 of 'filename' on the
x-axis, vs. column 3 on the y-axis.
- Use the 'fields' command to extract a single column from a file and output
it to another file ('fields filename 3 > filename3'). Then run the 'phistogram'
task on that file ('phistogram filename3'). Alternatively, you can pipe
the output of fields to phistogram, but you must tell phistogram to expect
input from STDIN (standard input): 'fields filename 3 | phistogram STDIN'.
- SM is a powerful (and therefore complex) plotting program usable under
Linux. Type 'sm' to start a session. Start by reading the basic
tutorial
online, and then when using SM you can consult the online
reference manual,
which includes descriptions of all SM commands.
For example, using the 'points' and 'errorbar' commands will let you create a
plot showing both the X and Y uncertainties for every point, and using
'histogram' and 'shade' will allow you to overplot multiple histograms.
You'll need to create macro files to save your plotting routines.
Some example plots and macros can be found at the bottom of
this page.
-
IGI is a plotting language that can be called from within IRAF
(after loading the STSDAS and GRAPHICS packages); for details type "help igi"
in IRAF.
-
To grab a postscript snapshot of a plot in an IRAF graphics window, type
"=gcur" and return at the IRAF prompt. That puts you in the graphics window.
Then type ":.snap eps" and return to create a postscript file.
Then type "q" to quit out of the graphics window.
The postscript file you created will have a name starting with sgi
and ending with eps (type "dir sgi*eps" to find out its name);
it should be located in the directory where you typed "=gcur".
You'll probably want to rename it to an easier to remember name.
-
To trim postscript files, use the UNIX commands "eps2eps" or "ps2epsi".
Although they don't always do what you want, in general they trim unwanted
whitespace from around a plot.
Backups
- These are handled by SuperDuper!.
Statistics
- A very good reference is Data reduction and error analysis for the
physical sciences (2003) by Philip R. Bevington & D. Keith Robinson. The
York library has a copy, plus a copy of the 1969 version (by Bevington only).
For further info, when needed, consult any version of `Numerical Recipes',
or see Level 5 below.
Extragalactic Astronomy and Cosmology
- Caltech and Carnegie run
Level 5,
A Knowledgebase for Extragalactic Astronomy and Cosmology.
It's a good reference for introductions to many different topics.
Active Galactic Nuclei
- Start with Brad Peterson's "An Introduction to Active Galactic Nuclei".
I recommend skipping section 1.2, and all of chapter 9 except section 9.4.1.
Chapters 1-5 and 7 are the most important for a basic understanding, but
you're welcome to read the other chapters too (especially ones which I've
mentioned as important for your specific research project!).