Issues with downloading and publishing books / Follow-up: Problems with books in the public domain

As I noted a few years ago ([1]):

We live in a world where great amounts of text, including by many great past authors, are in the public domain and also actually available on the Internet.

I still find myself constantly frustrated. Part of the benefit is removed by (often entirely unnecessary or arbitrary) artificial restrictions. Sometimes, all of it is removed.

A few additional words, both as a reader and as an author:

1. When possible, I strongly prefer to read e-books on my computer—not on e.g. a separate e-book reader or on a smartphone. For these purposes, I prefer PDF, as PDF (when done correctly!) preserves the original formatting of a printed book better than other formats and gives a more pleasant reading experience (less strain on the eyes, better readability, whatnot) than other popular formats.*

   *A secondary reason is that Linux is weak in support of other formats, which can lead to suboptimal display, the need to convert between formats, or, in extreme cases, files that are not readable at all. To avoid such issues, I stick to PDF, ePub, HTML, plain text, and, in rare exceptions, DjVu. (With reservations for the correct capitalization, here and elsewhere.)

   However, ever and ever again, I find that I have downloaded a PDF file that has none of the advantages of PDF through some crude conversion, effectively combining the disadvantages of two formats with the advantages of neither.* This typically in that someone has taken a plain-text file and run it through enscript (or some similar tool) to create something that looks like the original text file, fixed-width font included, or that someone has converted a web page into PDF through a print command (or some similar approach)—often with artificial headers indicating the file name, date of printing, or similar on each and every page.

   *All formats have advantages and disadvantages. For instance, plain text has (among others) the advantages of small files and of extreme flexibility, including that it can be viewed in, investigated with, and/or manipulated by tools such as less, grep, and vim. PDF, in contrast, shines with great formatting and the ability to print a hard-copy in a true-to-the-original manner.

   In both cases, I would have been much better off with the original file, keeping the advantages of the respective formats and foregoing the disadvantages of PDF. If, for some perverse reason, I needed a PDF, I could create it myself from the original file—and typically with a better result.

   To boot, despite a wide variety of free (both senses) software being available for local use, the conversion or editing has often been done with some type of online tool—which promptly adds further disadvantages through branding or advertising messages. In an extreme example, I once downloaded a PDF file where each and every page had a large and intrusive sun-like image in both margins. This rendered the file so unreadable, through the sheer annoyance, that I actually converted the PDF into plain text…

2. Many books in both PDF and ePub follow “bad practices” that are intended for a strict optimization for standalone e-readers, especially those sold by Amazon—and that, frankly, often are dubious even there. This includes artificial removal of margins, leaving the text immediately adjacent to the “physical” page borders (does not just look ugly, but hurts the eyes); artificial changes to interline distances, font size, or similar (ditto);* artificial removal of page numbers (due to front/back matter and similar, the indicator in the reader is not always enough); artificial removal of an original table of contents in favor of an automatically generated one (especially for non-fiction, the authors or editors have typically put in a lot of thought in the TOCs, which is now wasted—to the detriment of the readers); artificial removal of page numbers/references in TOCs (I often visit the TOC for purposes like finding out how long the current or the following chapter is, which is easy with page numbers, but not without them).

   *The exact manipulations vary, because different manipulators appear to have different goals. Notably, some appear to want to cram as much text as physically possible onto a single page, while some appear to want very large letters. In both cases, this likely reflects their personal habits, eye strength, whatnot on a standalone e-reader (maybe even the single, specific one that the individual manipulator uses)—and this is now forced onto the rest of the world, even on those who use computers.

   Note that, in doubt, content/formatting left in can always be removed later; content/formatting removed is usually gone for good. This is not the difference between, say, drinkers of red and white wine in a restaurant—it is the difference between drinkers of red wine and those who smash all the red-wine bottles to make room for more white wine.

3. Many books in both PDF and ePub have been shorn of images—without any warning to the prospective downloader. Now, sometimes the removal of images as an option is justifiable through the resulting size reduction; however, especially for non-fiction, the result can be highly detrimental and the choice should be left to the reader/downloader.
4. Some sites, notably Amazon, outright recommend or even demand “bad practices” like those mentioned above, with no consideration for other reading habits than standalone e-readers—not even with different versions for different formats, e.g. PDF for computers and ePub for standalone e-readers.
5. Format requirements for sale/upload are often too restrictive. For instance, a reason why my own first books are yet unpublished is that I went through the effort of giving them a nice formatting in LaTeX (from which PDF was generated), even doing some reading on topics like typography and book design in the process—only to find that sites likes Amazon screech like harpies when someone tries to deliver quality. At the time of my research,* Amazon did not even allow the upload of PDF, and instead presumed to take some other uploaded format** and convert that into PDF, should a customer wish to buy in PDF. Not only does the author lose in creative control, but he also has to take the potential hit from a poor conversion…

   *I have honestly lost track, but it was likely more than a year ago. I make no guarantees for the current situation (August 2022).

   **Likely, AZW; maybe, ePub or some other format, too.

   Worse, to my recollection, Amazon even presumes to include data like information about the author automatically and based on data stored with Amazon, reducing the author’s control further.

   Of course, all this fiddling, and the great risk that different sites use different rules, implies that the author will either be stuck on a single platform or be forced to adapt his book repeatedly for different platforms. (And woe to those who use a meta-platform, which distributes the same book, in the same version, to several different sales platforms.)

6. Of course, some sites have lost all contact with reality and demand, as sole upload, a Word-document… In other words, either the author has to write his books in Word to begin with, or he has to spend a horrendous amount of time (almost necessarily) manually converting from a more sensible format to Word.

   I am* a professional author. Products like Word should not be an option for a professional author.** I have more respect for someone who uses a pen, pencil, or typewriter, than for a Word user—pens and the like have a different set of advantages and disadvantages (a recurring theme) than LaTeX. Word is just bullshit.

   *Or was. Considering how little I have written since last summer, between construction noise, frustration with COVID countermeasures, demotivation from restrictive publishing options, and whatnot, my status might be under dispute.

   **That so many still limit themselves is scary. It is as if a professional carpenter would go to work using a kid’s toolbox. A central part of being a professional is to find and learn how to use a sufficiently powerful set of tools for the profession at hand. Those who do not, even should they earn a living in the field, scarcely deserve the title “professional”.

Excursion on how to do uploads better:
If Amazon was serious about both quality and genericness, it could and should provide a simple LaTeX template and/or LaTeX package (or some equivalent technology) with which the author could set up his book with a known-in-advance set of abilities and limitations. Afterwards, Amazon could simply generate the right formats from the corresponding LaTeX document.

Barring that, the best option would be to allow the authors to upload the formats that they want to support in the form that they want to support them, while the customers may either choose between the formats as uploaded or accept an automatic conversion with the explicit warning that the result might be poor.

Excursion on who-does-what:
A particular annoyance is that authors, both in modern “conventional” publishing* and in self-publishing are increasingly forced to do tasks that are unnatural matches with their likely skill profiles and interests, notably marketing, while those tasks that are more creative, short of the actual writing, are removed, including matters of book design and typography. If (!) the argument was that “authors know writing; we, the publishers, know typography, design, and marketing”, this might be acceptable.** In reality, the argument is “we, the publishers, make the creative decisions; you, the authors, do the boring leg-work”.

*One of several reasons why I ended up not even attempting the conventional route. (Other reasons include an apparently increasing shift in who earns what portion of the money, similar to the record industry, the need to be more “commercial” than I am, and the strong PC angle of the industry.)

**And, in my impression, this is how it used to be.

Advertisement

The struggling author: Amateurish Amazon and follow-up on construction noise

Another shitty day: It appears that the construction works are here again—and, again, without any notification or possibility to judge the size of the problem. Indeed, there is now scaffolding along the house wall, which could imply something very major and something not perpetrated by an individual apartment owner or resident but the actual building management.

Fortunately, the disturbances started in the mid-afternoon, and I could spend enough time walking to come back home after they had stopped. However, firstly, I have no idea how the future will look, and, secondly, the city is almost dead due to COVID-restrictions, meaning that there is very little to actually do, except walking (per se).

To conclude the day, I decided to finally open that Amazon KDP account that I will need in the mid- or long term. This was a frustrating and annoying experience. A partial summary (even at the risk of exceeding the policy for this closed-ish blog, but I need to unload the frustration):

1. The interface asks for an email address, sends a confirmation code to that address, and awaits entry of that code.

   OK.

2. The interface ADDITIONALLY asks for mobile phone number, sends a second confirmation code there, and awaits entry of that code.

   Not OK.

   Firstly, it must not be a prerequisite to have a cell phone to participate in various non–cell-phone activities. (Indeed, I have gone through quite long stretches without one and it is pure coincidence that I have a working cell-phone number at the time of writing.) Secondly, email confirmation should have been enough. Thirdly, Amazon claims that it would later be possible to opt out of cell phone verifications, but because it has to be activated the first time around, Amazon can now steal data that I would very much like to keep absent, e.g. to avoid abusive SMS/“text” spam. (Note that Amazon has no legitimate reason to know my telephone number, unlike e.g. my street address and email, for the current purposes. I have yet to investigate whether the opt-out claim holds true.)

   Moreover, the implementation was utterly incompetent, by repeatedly* resetting the country to the U.S. Here my explicit choice of Germany should have been kept; and the original default should have been Germany, too, as my address was German and I was clearly accessing the site from Germany. (In my recollection, but I might be wrong, Amazon even used German as the interface language.)

   *I tried to get past this step, as no mention of the reason had been made (itself a poor UI decision), by first entering a landline number, which is less susceptible to abuse. As the claim that a SMS had been sent was given after entry, I re-tried it with a cell-phone number, including (unthinkingly) a leading “0”. As no SMS arrived, I tried again, removing the “0”, for a total of three attempts.

3. A highly annoying, moving CAPTCHA needed to be answered.

   At best dubious, as there seems to be little reason to assume that someone goes to the immense effort of handling automatic confirmations per email and SMS for a purpose like creating an Amazon account. (Indeed, with this level of overall stringency, it might have been better to simply send a postal confirmation code and accept the temporary delay in exchange for one single confirmation.)

   Moreover, the implementation was awful, including crossing the border to where it becomes hard even for a human to complete the confirmation. (I needed two attempts, myself.)

4. I proceeded to enter the user account, an act apparently considered a separate log-in, despite following directly after the account-creation process, which required a second SMS confirmation.

   Not OK.

   Firstly, this particular type of two-factor authentication is very dubious in general,* increasing the efforts needed for trivial tasks disproportionately. (But note mentions of opt-out above.) Secondly, specifically in this situation, it was entirely redundant and my previous SMS confirmation should have been considered enough.

   *In fact, the two main scenarios where it is needed is (a) with idiot users who pick poor passwords (I use random and automatically generated ones) or have sloppy local security (I do not), (b) with idiot service providers who have too many flaws in their own systems or allow password hashes to get out (or, worse, have actually stored plain-text passwords). The risk of e.g. a snooper stealing a password exists, but is a lot smaller. Moreover, the (partially false) sense of security created by two-factor authentication can worsen the problem with (a); moreover, when more and more users access the Internet per cell phone, the value of this specific type of two-factor authentication is drastically reduced.

5. (My account was marked as incomplete (as expected), and I proceeded to complete the data. Note that the below items might be in the wrong order or be incomplete. It does, in particular, not include several amateurish oddities with the workflow and ambiguities concerning what-button-does-what.)
6. Address fields included an empty field for my telephone number, which was mandatory.

   Not OK.

   Firstly, my phone number is plainly and simply not Amazon’s business. Secondly, as a mobile number had already been entered, this should have been the pre-filled default.

7. For my bank information, separate entries of IBAN, BIC, and name-of-bank were needed.

   Not OK.

   This shows a fundamentally flawed approach, as the IBAN is intended to serve as the sole account identification. Requesting a separate BIC is amateur hour. (This unlike the “old” German system, where a BLZ identified the bank, and an account number the account within that bank.) The bank name might be acceptable as a safety check, but better systems fill it out based on the IBAN.* Moreover, it should be a near given that data like account numbers are copy-and-pasted, which would either make the check unnecessary (data is guaranteed to be correct) or pointless (if, highly unlikely, the original is faulty, repeated copy-and-paste procedures will not help).**

   *Here Amazon might be excused as an international operation.

   **However, other checks, like “is the IBAN of the right length” are still justified, to catch e.g. an incompletely copied IBAN.

8. I was led to the fill-out-the-U.S.-tax-excemption area.

   Not OK.

   A reasonable operation should have made sure that such nonsense is not necessary, e.g. through use of a non-U.S. subsidiary. A smaller company (or one, like Barnes & Nobles, highly U.S. centric) might have deserved a pass, but Amazon is one of the largest and most international companies in the world.

   (But I was already aware of the need to do this to avoid an absurd-for-any-European tax deduction of 30 % in favor of the U.S. (!) IRS, and had indeed even prepared by finding my German TIN in advance.)

9. Required further fields for the preceding item included address fields that had already been entered.

   Not OK.

   Already entered data should be taken as default values.

10. My German address contains an umlaut (a “ü”, to be specific). This was rejected when I tried to proceed.*

    *I am a little uncertain whether this was only with the tax fields or already with the main address fields. Below, I assume tax fields. If not, it is far worse.

    Not OK.

    Even assuming that this restriction was posed by the U.S. IRS, the check should have been performed during entry and a pre-filled value with a suggested correction provided and/or the data incompatibility should have been mentioned explicitly and up-front.

11. As I re-submitted, post-adaption, there was an apparent error text, which read merely that “This field has been corrected.” (or very similar), leaving me uncertain whether further action was necessary. I tried to save again, and was brought back to the same error message. (The page automatically centered on the “error”.) I checked the top and the bottom of the page, in vain, and tried a third time, just in case. I was returned to the same message. I now went through the page in detail and found, a little further down, but outside of the area displayed by the browser after Amazon’s deliberate focus, a request that I confirm the correctness of the correction.

    Not OK.

    The page should have made crystal clear that further action was needed and what action. (Note that the idiotic focus and choice of layout sabotaged this.) Moreover, as I had corrected the field, there should have been no further assumption of error than with any other data entry, making the inquiry/error/whatnot redundant.

Now let us see what future problems occur, including (I very strongly suspect) unsolicited and highly unwanted emails and/or text messages.

What an eBook is and is not

The topic of eBooks is common in the blogosphere—often as a discussion of whether eBooks are better or worse than regular books, which has the better future, or similar. (An example^e.)

This is all fine and dandy. What disturbs me, however, are the many incorrect assumptions made about eBooks. Typical mistakes include believing that eBooks are read on a Kindle (or a similar device), have a particular format, or are DRM infected.

If Amazon and its likes had their way, this might be the case; however, an eBook is simply a book in an electronic format—no more, no less. An HTML or plain-text file can also be an eBook, eBooks are regularly read on normal computers, and there are many, many eBooks that are free from DRM restrictions. Notably, a very sizable part of the classic literature is available free-of-charge on websites like Project Gutenberg^e.

My advice:

1. Make sure to not confuse eBooks in general with the heavily restricted and user-unfriendly eBooks that make out a sizable part of the commercial volume.

2. Take advantage of the many user-friendly, DRM-free, and free-of-charge eBooks that are available. Yes, if you want to (legally) read the latest Stephenie Meyer, you may have to shell out money; but, as a counter-weight, everything up to and including (most of) the Victorian era is in the public domain—as are many works of the 20th century and even a few of the 21st. (Including works dealing with vampires, fairies, and romance—and works that have stood the passage of time, where Meyer may be a mayfly.)

3. When you do buy eBooks try to stay away from those that are DRM-infested or in non-standard formats (safe alternatives: plain-text, HTML, PDF) to the degree possible. If sufficiently many do so, there is a chance that the industry will see the light.