Fixing Archive.org’s PDFs

Here’s the webpage for a very early edition of Huckleberry Finn. If you open the PDF using a modern PC or tablet, it will look fine though a little slow to load. If you open it on your Kindle, Nook Color, or some other older Ebook reader that displays PDFs, you’re in for a shock.

Each page in these PDFs are actually 3 images. When put together by a modern PDF reader, they make one nice scanned PDF page. If you’re not suing a modern reader, you see all 3 layers separately. This makes the book unreadable. Even if you are using a modern reader, these PDFs have a noticeable lag time compared to other documents because it is loading 3 images per page.

This guide which show you how to eliminate the first two images and reverse the third image to be white on black. Will this 100% fix the book?  No. However if you value text over presentation, it does make the book readable on any device including the good old E-ink Kindle.

Step 1. Install the applications (OpenSUSE)

sudo zypper in pdfmod imagemagick pandoc grename

Step 2. Convert the PDF to images. Create a directory for the files to go to first:

mkdir huck
pdfimages huckleberry.pdf huck/

Step 3. The files that are created are all -xxx.ppm and .pbm: Bash doesn’t like this. I use grename to rename every file so that they don’t begin with a hyphen

Step 4. cd to the directory and delete the extra image files:

cd huck
rm *.ppm

Step 5. Reverse the images of the .pbm files. This will create a new copy of the files with inverted colors.

for i in *; do convert -monochrome -colors 2 -depth 1 -negate $i in-$i; done

Step 6. Move the completed files to a next directory and delete the originals

mkdir finished
mv in* finished/
rm *.pbm

Step 7. cd to the finished directory and create a new pdf. This will take time and may freeze your computer. Be patient.

cd finished
convert `ls -v` huck_bw.pdf

Step 8. Shrink your newly created PDF because it is far too large right now.

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen \
-dNOPAUSE -dQUIET -dBATCH -sOutputFile=huck_bw_final.pdf huck_bw.pdf

Your new PDF is complete. It is not a pretty as the original but it is more handy.

I then use pdfmod to edit the metadata so the ebook is easier to work with in calibre.

I’m very interested if anyone has found a better way to do this with open source software that retains the color of the original but without the multiple layers.

Advertisement

Create a complete Tor Onion Service with Docker and OpenSUSE (EXPANDED)

I wrote this presentation on the weekends in April and May and it’s didn’t have quite the details that I wanted to put into it. Mostly I wanted it to be short and engaging. Putting in every detail that I wanted would have (I thought) been long and boring. I would like to take the time here to expand what went into the presentation and to make it a little more interesting.

htw1 (1)

I don’t really like these diagrams. They already existed on tor-project.org and were made by the EFF but it’s too high level and they don’t really tell the story that I wanted to tell. The actual description on how onion services with is here: https://www.torproject.org/docs/onion-services.html.en

I could have done a better job than what I did. The way that I prefer to describe it is like this:

After configuring your /etc/tor/torrc file, you run systemctl tor start and the local Tor daemon reaches out to the Tor networks and lets it know that you are running a local onion service. This creates a two-way link between your machine and the Tor network which is UDP traffic rather than TCP/IP as the Tor network never sees your actual local IP.

If I’m honest with myself, this is still pretty weak but it’s better than what I had. The best thing would be to take the information from Tor website almost verbatim and made slides but I didn’t do that.

Screenshot_20180527_142714

I glossed over this when I should have made more slides to help fill out the presentation. The brief anecdotes and really touch on a lot of reason why I think people should be using onion services such as:

Screenshot_20180527_143138

Being in the spotlight might just mean having a job where people know who you are. No matter who you are, there is a good chance that your political ideas will offend someone. Personally, I keep a strict no religion/no politics policy for myself at work. I just nod my head to everyone like I agree and/or understand. At home, it’s a different story. I am a political person and I care deeply about politics but I don’t want that interfering with my role at my company and I’m not alone. This is the point that would have had more punch than what I made and would have been a better case study on why onion sites are useful and needed.

Screenshot_20180527_144036

The first two should have been one topic and the last should have been a call back to a better description of how the onion routing and encryption works.

Silk_Road_Marketplace_Item_Screen

Nefarious websites such as Silk Road and the Playpen were better case studies on how onion services are misused.

Finally, I think more details on how the docker-compose files are built would have been more useful as well as some hands-on interaction. Those file are all at my Github but I ran through them so quickly I didn’t really give the audience time to see them during the presentation.

What is a CVE and How Can It Benefit Me?

Like a lot of the things that I write here, this is a question that came up in a ticket that I worked on recently. A customer recently received a message like this:

Samba is a freely available file- and printer-sharing application maintained and developed by the Samba Development Team. Samba allows users to share files and printers between operating systems on UNIX and Windows platforms. Samba is prone to a security-bypass vulnerability because it fails to properly enforce SMB signing when certain configuration options is enabled. Successfully exploiting this issue may allow attackers to bypass security restrictions and perform unauthorized actions by conducting a man-in-the-middle attack. This may lead to other attacks. The following versions are vulnerable: Samba 3.0.25 through 4.4.15 Samba 4.5.x versions prior to 4.5.14 Samba 4.6.x versions prior to 4.6.8.

This doesn’t actually tell us a lot. I could ping one of the Samba developers and ask them if they are aware of this vulnerability, if we’ve ever patched it, and if not what the status of it is. That’s could be a lot of time waiting for a reply and taking time out of the developer’s day to answer a fairly straightforward customer service question. However, there is an easier way.

When a software vulnerability is detected, it is reported as a CVE (Common Vulnerabilities and Exposures) number for that specific application. In this case, I found the CVE number that best matched the description that I was given and I was able to show the customer that we had patched it and which patch it was in.

One famous example was the “Heartbleed Vulnerability” from a few years ago which is CVE-2014-0160. SUSE retains a list of all CVE’s that we review and patch here: https://www.suse.com/security/cve/. As you can see here: https://www.suse.com/security/cve/CVE-2014-0160/ Heartbleed was patched in all versions of SLE 11 and 12 as well as OpenSUSE 12, 13, Leap, and Tumbleweed.

For those concerned about their system’s security, CVE’s are a great way to make sure that newly found vulnerabilities have been patched in their OS of choice.

More information:

About Patching: What is a Patch in SLE and OpenSUSE?

A while back I wrote a post on why you should patch your servers. I think it surprised some people. I got at least one comment from twitter saying, “I’m surprised you get so many tickets on this topic since security is so important in enterprise server environments.” And yet, we do. At any current time, we have multiple tickets asking for RCA (Root Cause Analysis) for a server crash or hang when the server has not been patched in month, years, or even ever. Sometimes they never register the server to receive patched and so never patch their server beyond what is in the base version that we ship in the beginning.

This post isn’t to complain. Its to help alleviate the problem. The first step is to discuss, what are patches and what do they do. Using a SUSE Customer Center (SCC) account, you can go to https://scc.suse.com/patches to view detailed information on all of our patches. I can get a list of them so far using this command:

jsevans@linux-rtf9:~> sudo zypper patches
Refreshing service 'Containers_Module_12_x86_64'.
Refreshing service 'SUSE_Linux_Enterprise_Server_12_SP2_x86_64'.
Refreshing service 'SUSE_Package_Hub_12_SP2_x86_64'.
Loading repository data...
Reading installed packages...
Repository | Name | Category | Severity | Interactive | Status | Summary
--------------------------------+-----------------------------------------+-------------+-----------+-------------+------------+----------------------------------------------------------------------------------
SLES12-SP2-Updates | SUSE-SLE-SERVER-12-SP2-2017-990 | security | important | --- | needed | Security update for glibc
SLES12-SP2-Updates | SUSE-SLE-SERVER-12-SP2-2017-994 | security | critical | reboot | needed | Security update for the Linux Kernel
SLES12-SP2-Updates | SUSE-SLE-SERVER-12-SP2-2017-998 | security | important | --- | not needed | Security update for openvp

As you can see, I need to apply three patches to this server.  Since patch, “SUSE-SLE-SERVER-12-SP2-2017-994” is listed as a critical update, we’ll review what makes this so important:

jsevans@linux-rtf9:~> zypper patch-info SUSE-SLE-SERVER-12-SP2-2017-994
Loading repository data...
Reading installed packages...



Information for patch SUSE-SLE-SERVER-12-SP2-2017-994:
------------------------------------------------------
Repository : SLES12-SP2-Updates
Name : SUSE-SLE-SERVER-12-SP2-2017-994
Version : 1
Arch : noarch
Vendor : maint-coord@suse.de
Status : applied
Category : security
Severity : critical
Created On : Mon 19 Jun 2017 05:28:39 PM CEST
Interactive : reboot
Summary : Security update for the Linux Kernel
Description :

The SUSE Linux Enterprise 12 SP2 kernel was updated to receive various security and bugfixes.



The following security bugs were fixed:

- CVE-2017-1000364: The default stack guard page was too small and could be "jumped over" by userland programs using
 more than one page of stack in functions and so lead to memory corruption. This update extends the stack guard page
 to 1 MB (for 4k pages) and 16 MB (for 64k pages) to reduce this attack vector. This is not a kernel bugfix, but a
 hardening measure against this kind of userland attack.(bsc#1039348)

The following non-security bugs were fixed:

- There was a load failure in the sha-mb encryption implementation (bsc#1037384).
Provides : patch:SUSE-SLE-SERVER-12-SP2-2017-994 = 1
Conflicts : [10]
 kernel-default.nosrc < 4.4.59-92.20.2
 kernel-default.x86_64 < 4.4.59-92.20.2
 kernel-default-base.x86_64 < 4.4.59-92.20.2
 kernel-default-devel.x86_64 < 4.4.59-92.20.2
 kernel-devel.noarch < 4.4.59-92.20.2
 kernel-macros.noarch < 4.4.59-92.20.2
 kernel-source.noarch < 4.4.59-92.20.2
 kernel-source.src < 4.4.59-92.20.2
 kernel-syms.src < 4.4.59-92.20.2
 kernel-syms.x86_64 < 4.4.59-92.20.2

In other words, this patch was written to avoid a possible security issue from a rogue application.

A quick and easy way to review what patches are needed for your system, simply run:

zypper patches | grep needed | grep -v "not "

This will allow you view the complete summary of all of your needed patches, you can run:

for i in `zypper lp | grep -i needed | awk '{ print $3 }'`; do zypper patch-info $i; done

If you haven’t patched in a while, this can be a lot of information. However, if you need to justify why you should patch, this is a great way to summarize the information. Another option is to visit https://www.suse.com/support/update/ which is a web-based repository for specific packages with much of the same information.

In my next post, I’ll discuss ways to intelligently apply patches to minimize downtime. In the meantime, here are some more resources.

SUSE Documentation

Did you know that SUSE provides free documentation for all of our products to the general public in multiple languages and in multiple formats? Not only that, it is released under the GNU Free Documentation or under a Creative Commons license. As a big fan of the work that the Creative Commons folks do, this makes me proud to work for a company that actually cares about giving back to the community. Not only is our documentation freely available, so are our knowledgebase articles. Unlike other companies, we don’t give just a few lines of the article and then prompt the user to buy our products in order to fix a problem. Also, now that SLE* and OpenSUSE share the same direct codebase, a problem solved in one product can easily be used to fix the other.

There are currently 20 manuals and white papers for SLES 12 SP 2 and 6 for OpenSUSE Leap 42.2. For the Linux newbie, spending $50+ for a paper book on one product at the local bookstore that will be out of date in two years is a huge investment. It makes sense for anyone wanting to get into Linux and Open Source to start with a distribution that makes it easy.

Our Free Documentation for SLE:

Installation and Administration

Deployment Guide
Administration Guide
Virtualization Guide
Storage Administration Guide
System Analysis and Tuning Guide
Security Guide

Additional Information

AutoYaST
Hardening Guide
Xen to KVM Migration Guide
Docker Quick Start
systemd in SUSE® Linux Enterprise 12 – White Paper
Virtualization Technologies
Networking with Wicked in SUSE® Linux Enterprise 12 – White Paper
Virtualization Best Practice
Subscription Management Tool (SMT)
Introduction to SUSE Linux Enterprise Server for the Raspberry Pi
Data Replication across Geo Clusters via DRBD
SUSE Linux Enterprise Server-Support for Intel Server Platforms

How Low Can You Go?

Every since my days on dial-up internet and telnet MUD’s, I’ve always been interested in low-bandwidth utilities. After all, not every user get’s to live in an area that provides unlimited high-speed broadband. Some, like those who depend on cellular or satellite internet, have to choose between work and watching a few YouTube videos.

This post will introduce some tools that will help you make the most out of a limited or high-latency Internet connection.

Mosh

https://mosh.org/

zypper in mosh

(SLES, Leap, Tumbleweed)

What does it do?

Mosh is a replacement for SSH. It uses the normal SSH protocol to initiate the connection to the remote server. However, after connecting, MOSH’s true magic begins. Rather than continuing using TCP port 22, MOSH changes to UDP, yet it is still encrypted. Due to its fault-tolerance nature, TCP tends to be quite slow and unresponsive when bandwidth is low and latency is high. UDP, on the other hand, isn’t worried about getting every byte correct. If a packet or two gets dropped, the connection stays up and continues working. Recently, I took a train from Prague to Nuremberg using the train’s complementary wifi. While traveling through rural areas, the connection often dropped to a crawl and yet my connection to my home computer never dropped.

How do I use it?

Install mosh on the computer that you are using and the computer that you want to connect to, the connect like you would using ssh.

mosh jsevans@myserver

Really, it’s that easy. There’s no extra daemon to start or configure. Mosh uses SSH to connect to the other server and activate Mosh on the receiving end and the two communicate.

Alpine

https://www.washington.edu/alpine/

zypper in alpine

(Leap, Tumbleweed)

What does it do?

Alpine is a text-based email client. Why would anyone use an antique like that? Easy, if email is important and your internet is limited, then Alpine might be your best option. With that said there is a learning curve, but it’s not as crazy as it seems at first.

How do I use it?

Below is a sample of what you will see at the main menu.

 ALPINE 2.20   MAIN MENU      Folder: INBOX      No Messages


    ?     HELP               -  Get help using Alpine

    C     COMPOSE MESSAGE    -  Compose and send a message

    I     MESSAGE INDEX      -  View messages in current folder

    L     FOLDER LIST        -  Select a folder to view

    A     ADDRESS BOOK       -  Update address book

    S     SETUP              -  Configure Alpine Options

    Q     QUIT               -  Leave the Alpine program


               For Copyright information press "?"
             [Folder "INBOX" opened with 0 messages]
? Help                              P PrevCmd              R RelNotes
O OTHER CMDS       > [ListFldrs]    N NextCmd              K KBLockH

Help, Compose, Setup, and Quit are self-explanatory. Message Index opens your Inbox and Folder List lists all of your email folders. The hardest part is setting it up for the first time. The University of Virginia has a fantastic tutorial for first-time installation using Gmail.

Screen

https://www.gnu.org/software/screen/

What does it do?

Many system administrators know about Screen, but it’s one of those tools that seems a little too mysterious for many new users. The main use of Screen is to create persistent shells. That means that if you start a new Screen session using the screen command, you can close that window and return to exactly where you left off.

How do I use it?

Try it for yourself. Run screen, then yast to get to console-mode yast, then close the window. If you were using yast on a remote server and the connection dropped, you might have lost all of your work. Open a new terminal window and type screen -x and you’ll see yast exactly how you left it. Screen has many more uses than that and the following tutorial will help you get the most out of it.

Using Screen in conjunction with Mosh can make even dial-up speeds bearable at the command line.

And now for something completely different.

There’s enough material out there to write a book or two on different text-based applications that can make life easier when bandwidth is limited, but I wanted to show something that could actually make system administration much easier for the admin who needs to work in the field. Did you know you can install a VM with KVM or Xen using only the console?

How do I do it?

There are basically three steps. First, install the virt-install application on your client machine.

sudo zypper in virt-install

Secondly, log into the remote machine and create an image to be used for storage for the new VM

qemu-img create -f qcow2 ./workspace/VM/opensuse42.2.qcow2 60G

Finally, the following command will use the ssh protocol to create the new VM and you will see everything in your console. Sadly, it doesn’t work with Mosh, but it does work quite well with Screen. Didn’t know that SUSE products had a text-only installation mode? It’s there and it works beautifully.

virt-install 
 --connect qemu+ssh://jsevans@10.0.1.35/system 
 --name leap42 
 --ram 2048 --disk path=/workspace/VM/opensuse42.2.qcow2 
 --vcpus 1 
 --os-type linux --os-variant generic 
 --network network=default 
 --graphics none 
 --console pty,target_type=serial 
 --location 'http://download.opensuse.org/distribution/leap/42.2/repo/oss/' 
 --extra-args 'console=ttyS0,115200n8 serial' 

Rather than giving one long command, it’s sometimes easier to break them down like this. You can still copy and paste the text here to test it for yourself. Just remember to replace my information with your own. On the second line of the command, I use the connect option to connect to the remote server with /system at the end of the command. Most of these options probably seem normal if you’re used to creating VM’s in virt-manager in KVM or even in VMWare for that matter. However, the location and extra-args options need some explanation. This installation technique only works with network installs so basically you need to install from a remote HTTP or FTP server and supply the URL or you’ll need to mount an ISO and serve the files via HTTP or FTP. You can see the url in the location option. The extra-args option tells KVM to emulate a serial connection similar to plugging in a serial cable to the back of a server and connecting directly. There is no virtual video card so there will be not be a remote VNC to view a GUI. There is only the serial connection. One difference in the final installation is that the kernel line will include directions for sending all text to the serial output. This makes booting your new VM easier to troubleshoot when booting. This technique isn’t limited to just SUSE products. Raymii.org has a fantastic guide on how to do remote installations with several other distributions.

LyX

Linux distributions are generally full of packages that have specific uses that most people don’t need or maybe they just don’t know that they need. In my previous post I wrote about QPhotoRec which I had never used before my little accident that actually saved me a huge headache. I didn’t know that this application existed until I started researching how to undelete applications in Linux and I was pleasantly surprised that it was already included in OpenSUSE. The application below is one of many that I’ve found that make life easier for me and maybe it will for you too.

In the words of the developers, LyX is a WYSIWYM (what you see is what you mean) document processor. This is opposed to WYSIWYG (what you see is what you get) word processors like LibreOffice Writer or Microsoft Word. What does that mean? It means that what you see on the screen is only an approximation of what will go into the document. Instead of giving you a 1 to 1 representation, LyX handles the typesetting elegantly to create beautiful professional documents that would require a lot of extra work to get right in a conventional word processor. Thre is an example of this blog post written in LyX with output as a PDF at the end of this article. I didn’t choose any special fonts or any special settings to impress you. I just chose the defaults and you can see the difference in quality.

LyX is based on LaTeX which was originally developed as a cross-platform language for publishing academic papers. With LyX it’s relatively easy to include a formula like:

However, anyone who has worked with writing papers on Microsoft Word or LibreOffice can attest to it being somewhat less friendly. It can be used to write papers of course, but also full books, screenplays, and scripts, in many different formats.

You can try out LyX by installing it with zypper using:

sudo zypper in lyx

Creating a simple beautiful document is actually quite easy. Input your text first, highlight the sections that need special attention such as title, author, section, chapter headings, etc., apply the format from the menu bar, and save and then preview your document by going to Document –> View [PDF (pdflatex)]. LyX will then save your file as a temporary PDF and open it in your local PDF Reader. When you do this, prepare to see a document that looks like it was professional typeset for a textbook.

Any application this powerful is undoubtedly complex. I won’t make you think that everything is very easy and there is no learning curve. There is, but it’s really not as steep as first appears. To get you started, here are a few resources to get started with LyX.

LyX Homepage: https://www.lyx.org/
LyX Tutorials: http://wiki.lyx.org/LyX/Tutorials

I hope to present you with more random yet useful applications in the future buried in the OpenSUSE repository.

This article via LyX.

I Deleted Everything

I goofed.

I’m an avid hobbyist photographer and I happen to live in one of the most beautiful cities in the world. Needless to say, I take a lot of pictures. Recently I upgraded the drive in my home desktop from a slow HDD to 256G SSD. My workflow is like this: I take pictures, I copy the RAW files from my SD Card to my 1.5TB external drive and then copy the ones that aren’t blurry or terrible to my local hard drive for editing and the best ones that are edited get promoted to Flickr. The SSD would make this a lot easier and faster because RAW image files tend to be relatively huge and time consuming to process.

I added the new SSD, installed Tumbleweed, copied my personal files from the old disk to the new one and then deleted the files on the old disk. I had plans to use it for another project. Except one thing; I was in the wrong directory when I deleted everything. All 400GB of pictures were gone. I immediately stopped everything because I knew that deleting files doesn’t actually remove them from the disk. It simply makes them able to be rewritten and I didn’t want to risk that happening.

How I recovered.

OpenSUSE Leap and Tumbleweed have an application called QPhotoRec that saved me.

I loaded the application with the gnomesu command as it needs root access to run:

gnomesu -c qphotorec

From here I was able to choose my disk, the kind of filesystem that I was using and where I wanted the files to be restored to and then I let it run. It’s not a fast process. It took around 8 hours to restore my 400GB and even then there was two thing that I wasn’t able to restore; the filenames and their directories. All of my files had their correct extensions but they were missing the filenames that they originally had and the directories they were in. Also, files inside of other files such as .iso or .tar files were also recovered including thumbnail photos that were stored in other files. QPhotoRec tries to make educated guesses about what is and is not a file and recovered everything but it’s not perfect. My job then was to reorganize my files into some semblance of how they were previously but at least they were there again.

Mistakes happen and hardware breaks. Files get deleted, sometimes important files. The best way to proceed is to always to keep backups (all of my most important files are encrypted and on a remote server) but when disaster happens through human error or otherwise, it’s good to know that there are options.