Shell Scripting Tutorials (Pic-notes)

This post provides some pic notes for Shell Scripting Tutorials (a list of 62 short videos) .

No capital letters in bash shell command (All should be in lowercase)!

clear

who

pwd 

cal

date

touch — create empty (plain text) files

mkdir — create directory (directory is the same thing as folder, in terminal it is called directory, on GUI interface it is called folder)

cd  — change directory

cat

cat > test   — this will create a file called test and write text into the file. (Ctrl+ D to exit editing the file.)

cat < test (shortcut cat test  —  this will open an existing file called test and print its content in the terminal.

cat random test > sample  — this will merge two existing files called random and test, and write the content to a new file called sample

mv — rename (when the source and destination is in the same directory) or move files

rm

rm -r  (r means recursive)

rmdir   — rmdir (only for empty directory)

cp  — copy files and /or directories

ln file1 file 2 — (file 1 is a physical copy of file 2, when file 1 changes, file 2 will be updated automatically and in real time)

ln -s file1 file1_soft  — (file1_soft is a soft link  of file1, when file1 was removed, file1_soft will not work, because the physical copy it refers to has been gone.)

 

 

 

file * command

the out put of wc command:

number of line, word, and charactor in the input file.

wc -l           number of lines

wc -w         number of words

wc -c          number of characters

 

 

  • Shell Scripting Tutorial-12: Sort

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

-f   file

-d   direcotry

-s file size (greater than zero or not)

-c (character file, e.g.,  .dbf file)

-b (block file like video and image)

-r  (read permission)

-w (write permission)

-x   (executable or not)

cat >> $fname   — will not overwrite the existing content in the file and will append … cat > $fname will overwrite

-n (the length of the string is not zero)

-z (the length of the string is zero)

the output of $?  is 0 means the operation is true, if false, the output is 1.

 

-a  AND logic operator

because there is a new line symbol, so need to be -eq 2 for a character.

 

 

 

-0  or logic operator

|    pipe symbol

the output of echo $var   as the input of wc -c, use reverse quote symbol to group it as a whole for -eq  operator

|  — the pipe symbol tells the case not to end the case statement after checking [aeiou]*, need to check [AEIOU]* as well for the first case statement.

Normally 0 is for success, 1 is for failure of the result of $?

2 means invalid.

the default IFS is white space.

exec and tty command

exec — exec is a builtin command of the Bash shell. It allows you to execute a command that completely replaces the current process. The current shellprocess is destroyed, and entirely replaced by the command you specify. (more check here.)

tty — Print the file name of the terminal connected to standard input. ( check more at here.)

read — read is a builtin command of the Bash shell, which reads a line of text from standard input and splits it into words. These words can then be used as the input for other commands. (for more info check here.)

sleep command — used for delay (the unit of its parameter is in seconds)

(see also Shell Scripting Tutorial-11: Count Lines, Words & Characters Using ‘wc’)

In this tutorial you’ll learn to iterate over contents of a text file using for loop.

 

In this tutorial you’ll learn to use nested loops to print all possible combinations of digits 1, 2 and 3.

 

In this tutorial you’ll learn to use the break statement to transfer control from within the loop to the first statement after the loop.

In this tutorial you’ll learn to use the ‘continue’ statement to skip iterations in loops.

 

In this tutorial you’ll learn to execute multiple commands on a single line in the terminal without using a shell script.

In this tutorial you’ll learn to add or remove users on the system, also we’ll check out the ‘login’ command to login through a user account in a terminal window.

In this tutorial we’ll build a script that checks for a user every minute and then notifies us when it logs in and also displays the time the user was late in logging in.

In this tutorial we’ll improve the script we built in the previous lesson.

In this tutorial we’ll finish our work on the script we first met in the 56th tutorial. We’ll check every one second for the user and will also report time with greater precision.

In this tutorial you’ll learn to send messages to other users on your system using the ‘write’ command.

You’ll also learn to use ‘finger’ command to see which users who have disabled message reception and which have not.

one terminal (terminal 1):

Another terminal (terminal 2):

in terminal 1:

in Terminal 2:

In Terminal 1:

In terminal 2:

In terminal 1:

In Terminal 2:

EOF — End Of File

 

Finger command:

In terminal 1:

In Terminal 2:

 

In Terminal 1:

Note the * before pts/1 (that indicates newuser2 does not allow send msg to it)

 

In this tutorial you’ll learn to create your own commands using functions.

unset command to remove customized functions in terminal.

 

In this tutorial we’ll learn to execute a script from another script; not a particularly hard thing to do but I recorded a tutorial on it anyway.

Thank you so much for watching the tutorials in this course. I hope you guys had as much fun watching the tutorials as I had recording them for you.

The END:) happy shell scripting!

 

 

 

 

 

 

 

Use latexdiff to compare different versions of LaTeX tex files to visually mark differences

This post introduces how to use latexdiff to compare two versions of latex *.tex files, in order to visually mark changes to tex files.

(One colleague asked me about the usage of latexdiff, so I think it is a good time to write it up into a blog post to share with those who need some help with this.)

latexdiff is a Perl script and requires an installation of Perl 5.8 or higher.

Use latexdiff to mark differences and create a new LaTeX document with markup differences between two latex files.

  • Linux (Ubuntu) users

latexdiff can be found in Ubuntu repository and detailed description of usage can be found here.

In your terminal, issue the commands below to install latexdiff.

sudo apt-get update
sudo apt-get install latexdiff
# open your terminal, and cd to the tex files you need to compare
# issue the following command
latexdiff original_version.tex revised_version.tex > diff.tex

Run the diff.tex file like you run a normal tex file, you will see beautifully marked differences in the pdf file.

  • Mac OS X users

latexdiff is a Perl script and requires an installation of Perl 5.8 or higher.
Mac OS X includes an installation of Perl and no additional setup is required. latexdiff can be found in any CTAN repository
(If you are using MacTex on your mac,  latexdiff is already installed in MacTex under /usr/texbin/latexdiff. No need to install it separately. To use latexdiff in commend line, see below)
After you setting perl and latexdiff, just open your terminal, and cd to the path where you put you two versions of latex tex files, and then issue the command. (If the two tex files are located in different location, you can just give the relative path to that location for each version of tex file.)

 

$ latexdiff original_version.tex revised_version.tex > diff.tex
will compare original_version.tex to revised_version.tex and create a new file diff.tex showing the differences.
Run the diff.tex file like you run a normal tex file, you will see beautifully marked differences in the pdf file.
  • Online latexdiff 

when you get the diff.tex on the online latexdiff, you can paste into a new tex file and then you can run it and you will see the pdf which shows the differences.
  • Multiple tex files for each version

If each version of your text files contains multiple (sub) tex files or if you want to use latexdiff with Git, check the Reference lists for potential solutions.

References

LATEXDIFF: SUPERB DIFF TOOL FOR LATEX (pdf) — talks about solutions to multiple files, but this can only deal with sub-files on the same level, not nested cases.

Two LaTeX gems: ShareLaTeX and latexdiff (pdf)

Using Latexdiff For Marking Changes To Tex Documents (pdf)

Multiple-file LaTeX diff (pdf— this one is very good. – use python code to merge multiple tex files to one big tex file for each version and then use latexdiff to make comparision.

Latexdiff with subfiles (pdf) — this one talks about solutions to multiple tex files.

Using latexdiff with git (pdf) – Git Latexdiff  — this is about using git together with latexdiff

Git and latexdiff: compare versions of LaTeX documents (pdf) — this is also about using git together with latexdiff

 

 

 

Bash shell scripting resources

This post provides some useful (video) resources for bash shell scripting. (If you do not like video style tutorials, check here for a post that is not video based.)

The bash ( Bourne Again Shell) is the most common shell installed with Linux distributions and Mac OS.

  • Bash Basics (A video playlist with 9 videos) — The Basics of using Bash in Linux with a focus on Ubuntu.

Bash Basics Part 1 of 8 | Access and Navigation

       $ means you logged in as a normal user.

       # means logged in as a root user.

Bash Basics Part 3 of 8 | Privileges and Permissions

Bash Basics Part 8 of 8 | Bash Scripting

[LaTeX] Comment out some text in a bib file

This post introduces how to comment out some text in a .bib file.

Sometimes we need to keep some information associated with a paper, such as web link where we can find the file, but we do not want it appears in the references.

It said that “//” or “%” works for some cases, but apparently it does not work for me. My solution is use comment = {add the info you wanted to comment with this  pair of curly brackets},

See below for an example.

@article{author2015paper,
title={paper title here},
author={author name},
comment={https://example.pdf},
year={2015}
}

Free tools to merge and split PDF files on Ubuntu and Windows

This post provides some GUI and command line tools to merge and split PDF files on Ubuntu and Windows.

(For Mac users, check out my post here for solutions.)

  • PDF-Shuffler is a GUI package that allows us to merge, split and re-arrange pages from PDF documents

Install use the command in your terminal (I have tested, It works on Ubuntu 16.04 LTS.)

sudo apt-get update
sudo apt-get install pdfshuffler

It is pretty simple and straightforward to use.

Limitations:

If you try to merge pdf files that are encrypted, you will meet this error.

*.pdf is encrypted.
Support for encrypted files has not been implemented yet.
File export failed.
  • PDFtk is a simple and powerful command line tool for doing everyday things with PDF documents. It comes in three flavors: PDFtk Free, PDFtk Pro, and the original command-line tool PDFtk Server. There is a simple GUI free versio for Windows users, see below for the info.

Install use the command in your terminal (I have tested, It works on Ubuntu 16.04 LTS.)

sudo apt-get update
sudo apt-get install pdftk

If you are a windows user, you do not need to buy adobe pdf pro version to manipulate pdf files any more, see the link to download PDFtk free, it is a simple GUI package to use on Windows. See the picture below for a sense of it.

 

See below for examples of using PDFtk to manipulate PDF files via linux command line.

This tool lets you do all sorts of things with PDF documents, such as split, merge or rotate them, apply a background watermark, burst documents into single pages, etc. mostly used command: use pdftk to split PDF files, use the following command :

pdftk in.pdf cat 2-8 output out.pdf

This will extract pages 2 through 8 from in.pdf and save them as out.pdf. The input file in.pdf will not be altered. You can use any other page range in place of 2-8. Leave out the range to select the whole document.

And here is an example of how to merge PDF files:

pdftk in1.pdf in2.pdf cat output out.pdf

This will concatenate in1.pdf and in2.pdf, saving the result as out.pdf. If you want to merge individual page ranges, you have to assign handles to input files, like so:

pdftk A=in1.pdf B=in2.pdf cat A2-3 B4-5 \

output out.pdf

This will merge pages 2-3 from in1.pdf with pages 4-5 from in2.pdf into out.pdf. The backslash means that the input is continued on the next line. You can use it when you type in long commands, but you don’t have to.

 

pdftk in.pdf cat 1-12 output in_page1-12.pdf
pdftk in.pdf cat 15-end output in_page15-end.pdf

You can use this method to split a PDF in N ways, or to remove pages.

For example, to remove page 13:

pdftk in.pdf cat 1-12 14-end output out.pdf

Or use it to rotate pages and many other things, type man pdftk in your command to see the options.

Installation is also possible by downloading a binary (Windows, OS X, Linux) or using Homebrew.

 

  • cpdf, it is a powerful, free tools to manipulate PDF files using command line. The Coherent PDF Command Line Tools allow you to manipulate existing PDF files in a variety of ways. For example:
  • Merge PDF files together, or split them apart
  • Encrypt and decrypt
  • Scale, crop and rotate pages
  • Read and set document info and metadata
  • Copy, add or remove bookmarks
  • Stamp logos, text, dates, page numbers
  • Add or remove attachments
  • Losslessly compress PDF files

See below for some examples:

cpdf in.pdf 1-12 -o out.pdf

cpdf in.pdf 13-end -o out.pdf

Or, to split into 12-page-sized chunks:

cpdf in.pdf -split -chunk 12 -o out%%%.pdf

Note that: for a large pdf file (1000+ pages) the performance of cpdf was much better than PDFtk.

References:

Edit PDF files on Linux – the easiest way

Sometimes you run up in a situation when you need to edit a PDF file on Linux. For example, when you need to go through an old report which was in PDF format and you saw some typos. how to edit PDF files on Ubuntu?

I remembered that I used a program called pdfedit but it’s not available on Ubuntu or in any other Linux distributions anymore.

Things change now. In the past, we needed to use a dedicated application for this purpose. There are still plenty of tools to edit PDF files on Linux, but we don’t really need them, well not in most of the cases, because LibreOffice does the job for us.

Surprised? Yes, the popular Linux alternative to Microsoft Office can also edit PDF files. And perhaps this is the easiest way for editing PDF files.

 How to Edit PDF files on Linux
Most of the Linux distributions such as Ubuntu, Linux Mint, Fedora etc has LibreOffice as default office suite. If you do not have LibreOffice, you can install it.

Once you have installed LibreOffice, Use LibreOffice Draw to open the pdf file to edit. You will see that the file is in editable mode. You can use the select pointer to choose which bit of text to edit. Just click on the text which you want to edit. For those areas that are no text, you will need to use draw tool (e.g., pencil, circle etc.) to add things.

Once you are done with the edits, instead of saving the file (using Ctrl+S) option, click on Export to PDF button.

If you are not finising the edit yet, but need to save the editted file, save it and it will be saved as a .odg file. Next time you would like to continue the edit, just open the .odg file, you will be able to continue your edits.

It is pretty cool, right? We do not need to buy Adobe Pro version to edit pdf files any more.

Limitations of editing PDF files with Libreoffice.

This PDF editing does not work on scanned documents. The files which were originally created as text and saved as PDF can be edited very easily but it is not the case when you have scanned document because those pages are actually images. But you can use draw tools (e.g., pencil, circle) in the Libreoffice to add things like signature, tick sign etc, you just cannot edit the text part if it is a pdf file from scanning.

 

[LaTeX] Write partial differential equation (Ex. dQ/dt=ds/dt) with partial derivative signs

\documentclass{article}
\begin{document}

\begin{equation}
\frac{\partial Q}{\partial t} = \frac{\partial s}{\partial t}
\end{equation}

\end{document}

\begin{equation}
\frac{\partial h}{\partial t} + \frac{\partial uh}{\partial x} + \frac{\partial vh}{\partial y} = 0
\end{equation}

  • equation with numbering
\begin{equation}
 \frac{\partial h}{\partial t} + \frac{\partial uh}{\partial x} + \frac{\partial vh}{\partial y} = 0 
\end{equation}

  • equation with name no numbering
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
\newcommand{\eqname}[1]{\tag*{#1}}% Tag equation with name

\begin{document}

\begin{equation}
 \frac{\partial h}{\partial t} + \frac{\partial uh}{\partial x} + \frac{\partial vh}{\partial y} = 0 \eqname{Continuity Equation} \ 
\end{equation}

\end{document}

  • equation with both name and numbering
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
\newcommand{\eqname}[1]{\tag*{#1}}% Tag equation with name

\begin{document}

\begin{align}
 \frac{\partial h}{\partial t} + \frac{\partial uh}{\partial x} + \frac{\partial vh}{\partial y} = 0 \\ \eqname{Continuity Equation} \ 
\end{align}

\end{document}

 

  • equation with both name and numbering and ref equation in text body
\documentclass{article}
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
\newcommand{\eqname}[1]{\tag*{#1}}% Tag equation with name
\begin{document}

\begin{align}
 f(x) &= a \\ \eqname{Constant} \ g(x) &= ax \\ \eqname{Linear} \ h(x) &= ax^2+bx+c \label{abc} \\ \eqname{Quadratic}
\end{align}


See~\eqref{abc}.
\end{document}