ilya-sher.org Open in urlscan Pro
192.0.78.24 Public Scan

Back to summary

Submitted URL:
http://ilya-sher.org/
Effective URL:
https://ilya-sher.org/
Submission: On May 26 via api (May 26th 2022, 11:40:48 am UTC) from GB — Scanned from GB

Form analysis
3 forms found in the DOM

GET https://ilya-sher.org/

<form role="search" method="get" class="search-form" action="https://ilya-sher.org/">
  <label>
    <span class="screen-reader-text">Search for:</span>
    <input type="search" class="search-field" placeholder="Search …" value="" name="s">
  </label>
  <input type="submit" class="search-submit" value="Search">
</form>

POST https://subscribe.wordpress.com

<form method="post" action="https://subscribe.wordpress.com" accept-charset="utf-8" style="display: none;">
  <div class="actnbr-follow-count">Join 28 other followers</div>
  <div>
    <input type="email" name="email" placeholder="Enter your email address" class="actnbr-email-field" aria-label="Enter your email address">
  </div>
  <input type="hidden" name="action" value="subscribe">
  <input type="hidden" name="blog_id" value="110710407">
  <input type="hidden" name="source" value="https://ilya-sher.org/">
  <input type="hidden" name="sub-type" value="actionbar-follow">
  <input type="hidden" id="_wpnonce" name="_wpnonce" value="e93d20b270">
  <div class="actnbr-button-wrap">
    <button type="submit" value="Sign me up"> Sign me up </button>
  </div>
</form>

<form id="jp-carousel-comment-form">
  <label for="jp-carousel-comment-form-comment-field" class="screen-reader-text">Write a Comment...</label>
  <textarea name="comment" class="jp-carousel-comment-form-field jp-carousel-comment-form-textarea" id="jp-carousel-comment-form-comment-field" placeholder="Write a Comment..."></textarea>
  <div id="jp-carousel-comment-form-submit-and-info-wrapper">
    <div id="jp-carousel-comment-form-commenting-as">
      <fieldset>
        <label for="jp-carousel-comment-form-email-field">Email (Required)</label>
        <input type="text" name="email" class="jp-carousel-comment-form-field jp-carousel-comment-form-text-field" id="jp-carousel-comment-form-email-field">
      </fieldset>
      <fieldset>
        <label for="jp-carousel-comment-form-author-field">Name (Required)</label>
        <input type="text" name="author" class="jp-carousel-comment-form-field jp-carousel-comment-form-text-field" id="jp-carousel-comment-form-author-field">
      </fieldset>
      <fieldset>
        <label for="jp-carousel-comment-form-url-field">Website</label>
        <input type="text" name="url" class="jp-carousel-comment-form-field jp-carousel-comment-form-text-field" id="jp-carousel-comment-form-url-field">
      </fieldset>
    </div>
    <input type="submit" name="submit" class="jp-carousel-comment-form-button" id="jp-carousel-comment-form-button-submit" value="Post Comment">
  </div>
</form>

Text Content

Skip to content
Open Menu
* Home
* About Ilya Sher
* This Blog Objectives

Search
Search for:
Close

ILYA'S BLOG

SYSTEMS AND SOFTWARE ENGINEERING

THE ORIGINAL SIN IN IT

April 10, 2022April 10, 2022 Ilya SherLeave a comment

You have a program with human readable output. Now a second program needs that
information. There are two choices:

1. Do the technically right and challenging thing – create a protocol and
rewrite the first program and then write the second program (utilizing
something like libxo in the first program maybe).
2. Fuck everybody for the decades to come by deciding that parsing text is the
way to go.

I would like to thank everybody involved in choosing number 2!

jc is an effort to fix (more work around) that decision. Thanks to the author
and I hope it will make your DevOps life at least a bit more tolerable.

THE PSEUDO NARROW WAIST IN UNIX

March 6, 2022May 20, 2022 Ilya SherLeave a comment

BACKGROUND

This is a pain-driven response to post about Narrow Waist of Unix Architecture.
If you have the time, please read that post.

The (very simplified and rough) TL;DR of the above link:

1. The Internet has “Narrow Waist”, the IP protocol. Anything that is above
that layer (TCP, HTTP, etc), does not need to be concerned with lower level
protocols. Each piece of software therefore does not need to concern itself
with any specifics of how the data is transferred.
2. Unix has “Narrow Waist” which is text-based formats. You have a plethora of
tools that work with text. On one side of of Narrow Waist we have different
formats, on another side text manipulating tools.

I agree with both points. I disagree with implied greatness of the Unix “design”
in this regard. I got the impression that my thoughts in this post are likely to
be addressed by next oilshell blog posts but nevertheless…

FORMATS

Like hierarchy of types, we have hierarchy formats. Bytes is the lowest level.

BYTES

Everything in Unix is Bytes. Like in programming languages, if you know the base
type, you have a certain set of operations available to you. In case of Bytes in
Unix, that would be cp, zip, rsync, dd, xxd and quite a few others.

TEXT

A sub-type (a more specific type) of Bytes would be Text. Again, like in a
programming language, if you know that your are dealing with data of a more
specific type, you have more operations available to you. In case of Text in
Unix it would be: wc, tr, sed, grep, diff, patch, text editors, etc.

For the purposes of this discussion X is a sub-type of Text. CSV or JSON or a
program text, etc.

Is JSON a sub-type of Text? Yes, in the same sense that a cell phone is a
communication device, a cow is an animal, and a car is a transportation device.
Exercise to the reader: are this useful abstractions?

Cow is an animal

THE TEXT HELL

The typical Unix shell approach for working with X are the following steps:

1. Use Text tools (because they are there and you are proficient wielder)
2. One of:
1. Add a bunch of fragile code to bring Text tools to level where they
understand enough of X (in some cases despite existing command line tools
that deal specifically with X)
2. Write your own set of tools to deal with the relevant subset of X that
you have.
3. Optional but likely: suffer fixing and extending number 2 for each new
“corner case”.

The exception here are tools like jq and jc which continue gaining in popularity
(for a good reason in my opinion). Yes, I am happy to see declining number of
“use sed” recommendations when dealing with JSON or XML.

Interestingly enough, if a programmer would perform the above mentioned
atrocities in almost any programming language today, that person would be
pointed out that it’s not the way and libraries should be used and “stop using
square peg for round hole”. After few times of unjustified repetition of the
same offense, that person should be fired.

Square peg / round hole

Somehow this archaic “Unix is great, we love POSIX, we love Text” approach is
still acceptable…

PIPES TEXT HELL

1. Create a pipe between different programs (text output becomes text input of
the next program)
2. Use a bunch of fragile code to transform between what first program produces
and the second one consumes.

WHERE TEXT ABSTRACTION IS NOT USEFUL

Everywhere almost. In order to do some of the most meaningful/high-level
operations on the data, you can’t ignore it’s X and just work like it is Text.

EDITING

The original post says that since the format is Text, you can use vim to edit
it. Yes you can… but did you notice that any self respecting text editor comes
with plugins for various X’s? Why is that? Because even the amount of useful
“text editing” is limited when all you know you are dealing with Text. You need
plugins for semantic understanding of X in order to be more productive.

Wanna edit CSV in a text editor without CSV plugin? OK. I prefer spreadsheet
software though.

Have you noticed that most developers use IDEs that “understand” the code and
not Notepad?

LINES COUNT

Simple, right? wc -l my.csv. Do you know the embedded text in quotes does not
have newlines? Oops. Does it have header line? Oops.

TEXT REPLACEMENT

Want to try to rename a method in a Java program? sed -i
's/my_method/our_method/g' *.java, right? Well, depends on your luck. I would
highly recommend to do such kind of refactoring using an IDE that actually
understands Java so that you rename: only specific method in a specific class as
opposed to unfortunately named methods and variables, not to mention arbitrary
strings.

SEARCH / INDEXING

Yep… except that understanding of the semantics helps here quite a bit. That’s
why you have utilities which understand specific programming languages that do
the indexing.

CONCLUSION

I do not understand the fascination with text. Still waiting for any convincing
arguments why is it so “great” and why the interoperability that it provides is
not largely a myth. Having a set of tools enabling one to do subpar job each
time is better than not having them but is it the best we can?

My previous dream of eradicating text where it does not make sense (my blog post
from 2009) came true with HTTP/2. Apparently I’m not alone in this regard.

Sorry if anything here was harsh. It’s years of pain.

CLARIFICATION – LAYERING

Added: 2022-02-07 (answering, I hope,
https://www.reddit.com/r/ProgrammingLanguages/comments/t2bmf2/comment/hzm7n44/)

Layering in case of IP protocol works just fine. Implementer of HTTP server
really does not care about the low level transport details such as Ethernet.
Also the low level drivers don’t care which exactly data they deliver. Both
sides of the Waist don’t care about each other. This works great!

My claim is that in case of the Text Narrow Waist, where X is on one hand of and
the Text tools are on the other, there are two options:

1. Tools ignore X and you have very limited functionality you get out of the
tools.
2. Tools know about X but then it’s “leaky abstraction” and not exactly a
Narrow Waist.

That’s why I think that in case of Text, the Narrow Waist is more of an
illusion.

--------------------------------------------------------------------------------

Have a nice week!

ON ACCIDENTAL SERIALIZATION FORMATS

February 2, 2022February 2, 2022 Ilya SherLeave a comment

Let’s talk about the “just separate with comma and stick it into one field” type
of serialization.

You had two strings (abc and def) and you joined them with a separator. What do
you have now? One string with two elements, right? Right, abc,def. Well… two or
more actually, depending on how many times the chosen separator occurred in the
original strings: if they were a,bc and def, you’ve got a,bc,def, which is 3
elements according to our format. Oops. Leaving out the question whether leading
and trailing spaces are significant.

Wanna add escaping for the separator then? a,bc and def are now serialized as
a\,bc,def. Now the parsing became more complex. You can’t just split the string
by the separator (you would get 3 elements: a\ and bc and def. You need to scan
the serialized data, considering escaping when splitting. You also need to
remove the escaping character from the result. How about escaping the escape
character? If original data is a\bc, it is serialized as a\\bc). Yet another
something not to forget.

Don’t like escaping then? How about encoding like in URL? a,bc becomes a%2Cbc.
You can now once again split the string by the separator character… assuming it
was encoded. Which characters you encode anyway? If you encode all ASCII
characters, the result is 3 times the original and is completely unreadable. It
least you are “safe” with regards to separator now, it is encoded for sure so no
split problems. You have to add a decoding routine now though.

If your serialized thing goes into a database, consider how indexing would work.
It probably won’t. Maybe you should model your domain properly in the database
and not serialize at all. Hint: if the values ever need to be treated
differently/separately by the database, they go into different
cells/rows/columns/fields, not one. There are very rare exceptions. Notable
exception is the ability of databases to handle JSON fields (examples: MySQL,
PostgreSQL). Note that this capability can fit or not fit your use case.

Want to satisfy your artistic needs and do something clever about the
serialization? Do it at home then please. Don’t waste time that your colleagues
could use on something more productive than dealing with your custom format.

Strong advice: don’t do custom serialization format, use existing serialization
formats and libraries.

--------------------------------------------------------------------------------

Seen something to add to the above? Leave a comment!

API-LESS

November 26, 2021November 26, 2021 Ilya SherLeave a comment

I was always complaining about how AWS made APIs that are inconsistent in so
many dimensions. AWS teams can’t even (read “don’t care”) to agree on how to
name pagination related fields.

Because I am such who was that again? an important person, I hope at least
someone everybody in AWS have read my blog and they are checking by mistake it
at least twice a day for new stupid rants extreme wisdom.

One of these people said “Fook you, Ilya” and made a service without API. Not
kidding. Meet AWS Control Tower. No API, no CLI, no CloudFormation. “Take that,
Ilya!”

For some balance I will mention that other people in AWS are trying to do some
uniform API. Not stellar at the moment but hey, at least they are trying.

Have a nice $(curl https://api.timeofday.example.com/) !

AWS CLOUD CONTROL IS IN FOR REVIEW

October 4, 2021October 5, 2021 Ilya SherLeave a comment

Since 2016 I have worked on AWS library. On 2021-09-30, AWS released something
comparable, AWS Cloud Control API. If it’s comparable, let’s compare. I hope the
reader will find this unique perspective interesting.

PURPOSE

AWS Cloud Control provides uniform API to Create, Read, Update, Delete and List
different resources. In short, CRUDL.

My library aims to provide uniform simplified declarative interface where the
main functionality is to converge to desired state.

SUPPORTED RESOURCES

Cloud Control supports much more resources. My library supports very few, the
ones that I needed the most for my work. It’s just how experimental library by
one person in spare time compares with a product from AWS.

MODE OF OPERATION

CLOUD CONTROL

Operates on one resource at a time.

Cloud Control provides relatively low level API, CRUDL. The advantage of Cloud
Control over existing APIs is the uniformity, not achieving a desired state. You
either create or update, and you need to know which one you are doing. (Deletion
is a different story).

I am amazed that somebody at AWS was able to pull this off… where different
services don’t agree on naming conventions for tagging resources, how and what
can be tagged, and don’t have a convention for naming pagination fields. (Which
by the way is nightmare for the users but typically abstracted by Terraform or
CloudFormation). Sorry for the rant.

The operations are performed asynchronously. You have an API to query the
status: GetResourceRequestStatus. This approach is better for advanced use
cases, where multiple resources are being created simultaneously. (Like
CloudFormation?)

The operations are idempotent. This is achieved using client token, as in other
AWS services.

MY LIBRARY

Operates either on one resource or a set of resources of the same type at a
time. Edit: … from default, specified , or all regions.

The library looks up the resources and either creates or updates them, depending
on whether they exist or not.

The operations are performed synchronously. The library throws exception if
anything goes wrong. I find this approach much more user friendly for basic use
cases.

Idempotency is out of the picture. Should it be in? Probably. Looks like
omission on my side. How to best shield the user of the library from this issue?
Don’t know yet. Need to think.

DESCRIBING DESIRED STATE

CLOUD CONTROL

The “create” API has desired state parameter: DesiredState.

The “update” API doesn’t, which I find very strange. I thought that desired
state is something to be achieved no matter what the current state is. Desired
state is only used in the “create” API so from my perspective it could have been
called “properties” or whatever.

The “update” API has PatchDocument parameter, which I find not user friendly for
the following reasons:

1. Who actually works with JSON patch format? It’s the first time I see that I
need to use it.
2. I think it is less mentally challenging to describe … desired state and not
the delta. This is typically done by IaC tools, including CloudFormation:
calculate the diff between current and desired state for the user, so that
the user would not need to get into this.
3. It makes the update inconsistent with create.

MY LIBRARY

There is no separate create and update. The user specifies the desired state as
a parameter to the “converge” function. Converge then either creates or updates
the resource / resources. The (non existent) “create” and “update” are therefore
completely uniform in the library.

SEARCH / FILTERING

CLOUD CONTROL

Search is not supported. In practice it means listing all the resources and
filtering the result on the client side. Typically that can be done in AWS CLI
with the --query flag which is supported globally, for any AWS CLI command.
Unfortunately I don’t see a way to make it work in this situation. The returned
result has ResourceDescriptions field, an array where each item has Properties
field, the Properties field is a string (JSON). Apparently JMESPath does not
support parsing JSON in this situation. This means that the output of Cloud
Control AWS CLI will be piped to jq or maybe a programming language for
filtering and/or further processing.

MY LIBRARY

While the number of supported resources and filters is low, the library supports
filtering. The filtering is done on the server or on the client, completely
transparent to the users of the library. What’s done on the server and what’s on
the client? Simple – when filtering a given property is supported on the server
side, it’s done there.

DESIRED STATE FORMAT

CLOUD CONTROL

Cloud Control uses CloudFormation syntax. This makes sense.

MY LIBRARY

My library uses the same format as you would see when using AWS CLI to describe
the resource. It allows access to properties that CloudFormation does not have,
and is unlikely to have, so for example this works:

1. instance = AWS::Instance(...).converge(State = 'running') — which creates or
turns on the specified EC2 instance / instances. Turning on or off EC2
instances is not supported in CloudFormation.
2. vpc = AWS::Vpc(IsDefault=true).expect(1) — get a reference to default VPC
(to use in further operations).

CLOSING THOUGHTS

1. Cloud Control looks like a step in the right direction.
2. Having Properties as a string is a major ergonomic issue.
3. JSON patch for update is a huge ergonomic issue.
4. Search (filtering in List) functionality is missing.
5. “Desired state” naming is unjustified.

Cloud Control is a big effort, therefore let’s give the team some slack and see
how the API is improved with time? Hopefully soon

--------------------------------------------------------------------------------

Have a nice day, evening, or night! Mornings are just statistically less nice…

FAILED STEALING FROM PYTHON

September 29, 2021September 30, 2021 Ilya SherLeave a comment

I made a mistake. Hope you will learn something from it.

MENTAL SHORTCUTS

Heuristic, tl;dr for the purposes of this article – mental shortcut. The brain
chooses to do less thinking to save energy. It relies on simple rules to get the
result. The result is correct… some times.

I took a mental shortcut when working on my own programming language, Next
Generation Shell. It was a mistake.

Additionally, I have ignored the uneasy but very vague feeling that what I’m
doing is not 100% correct. From my experience I knew I shouldn’t ignore it but I
still did it. Another mistake.

I “THOUGHT”

Below are heuristics that led to the wrong decision.

Copying features from popular languages is pretty “safe”. After all, “everybody”
is using the language and it’s OK. Social proof. Wrong. Everybody does mistakes.
Popular languages have design issues too.

It’s OK to copy a feature because it’s very basic aspect of a language. Nope.
Python messed up arguments passing. And I copied that mess.

THE FIX

Python 3.8 has the fix. I have mixed feelings about it. Still not sure how I
should fix it in NGS.

TAKEAWAY

Beware of mental shortcuts. There are situations where these are not acceptable.
The tricky part is to detect that you are using a mental shortcut in a situation
where it’s not appropriate. I hope that with awareness and practice we can do
it.

Also note that your $job is most likely paying you to not take mental shortcuts.

THE WEB VS UNIX

September 26, 2021September 26, 2021 Ilya Sher8 Comments

I would like to share my perspective on what’s wrong with Unix. This time by
comparing and contrasting it to the Web.

USER AGENT

With some amount of stretch, one could say that the equivalent of the browser
(user agent) would be a shell with a terminal. You interact with it and it does
things for you, like the browser.

USER INTERFACE

The web browser is capable of rendering textual and graphical content. Unix
shell relies on the terminal (usually emulator mimicking decades old hardware)
for user interface and in most cases is limited to fixed width font text.

The main difference is how you can interact with the content. In the shell – you
can’t. The shell is out of the game. The content that you see on your screen
goes from a program and straight into the terminal, bypassing the shell.
Compared to a browser, that sounds insane: you are just unable to interact with
the content that you see. Ironically enough, this is happening in what’s called
“interactive shell“. Some terminals match the text with predefined set of
patterns and allow some minimal interaction such as ability to click on a link
to open it in a browser.

The browser is a strict superset when we look at interaction capabilities: you
can type in and you can interact with the objects on the screen by clicking
them. What an amazing concept! Maybe some day shells will be able to that too!
Meanwhile, in the shell, you type your commands and get a dump of text back,
with rare exceptions.

I would summarize that the shell is a shitty user agent.

I already can hear the coming “shell is not supposed to do it” argument. My
opinion: shell is supposed to do whatever is needed for me to be productive. If
it’s your “Unix Philosophy” vs me being productive then you can continue to use
Notepad (or ed, the standard text editor, for that matter) and I would be using
an IDE, OK?

LAYOUT ENGINE

They also call it browser engine. That’s because on the web it’s in the browser.
But where is it on Unix? Everywhere. Yes, the “Make each program do one thing
well” is out of the window long ago. Each program does (hopefully) one thing and
then it also does the layout of the output.

Each program has the following main options for handling the input/output:

1. Primitive output – the program dumps some text on standard output. Let’s
include colored text here. It’s just some additional color codes. This is
equivalent to not having a layout engine. Sample program in this category:
grep.
2. Interactive UI – the program uses ncurses or similar library. It’s
relatively small number of programs.
3. Layout engine – the program contains some form of a layout engine. This is
pretty common. Sample programs in this category: ls, ps, top, diff (columns
output), wc, …

Common issues with the “layout engines” causing unpleasant broken view in Unix
include:

1. Improper handling of data which is wider than some hard coded fixed value
2. Improper handling of Unicode
3. Failure to accommodate for “unexpected” terminal escape codes in the input
(which after processing find their way to the output in utilities like sed)

TCP/IP

Pipe

Let’s talk about pipes. Before everybody gets offended and says pipes is the
sacred cow best feature of Unix. Yes, it probably is.

Pipes would roughly correspond to the TCP/IP protocols. Pipes deliver data. For
now, let’s leave alone the fact that they are unidirectional as opposed to TCP,
which is bidirectional.

Since the web is a stack of protocols, the obvious question would be how other
parts of the stack correspond? Read on.

HTTP

HTTP would correspond to text. Well, mostly text. Sometimes null character
separated records. Sometimes something else. That’s the standard “format” to
communicate between Unix applications.

“Write programs to handle text streams, because that is a universal interface.”
– Basics of the Unix Philosophy.

The original claim is that text is the best for interoperability: large number
of utilities have text as input and also output text, manipulating it in many
different ways in between. Sounds like a dream. Except in reality this dream
turned out to be a nightmare.

Incompatible, ad-hoc formats

Text on Unix is not a single format. It’s a bunch of ad-hoc formats, typically
incompatible between different programs. That’s why we have a variety of tools
such as sed, cut, awk and alikes. Here is my hot take: these tools are not
solutions, they are workarounds. When you don’t have a protocol to communicate
between applications, you need a bunch of adapters. Like Sisyphus, one need to
write these adapters. All the time. Forever. Text parsing and manipulation feels
like core part of Unix. From my perspective it’s an accidental complexity.

On a philosophical note: the “universal interface” should have been a stream of
bytes or maybe even bits. I guess it was not found to be very useful. Apparently
if you add a line separator character it is good enough to become a
recommendation. But why stop there? Maybe add more structure? Maybe accommodate
the fact that most of the data is either records with named fields or tables
with named columns? Are you sure you counted the columns right for your awk
'{print $8}' ?

This is in contrast to HTTP which is spoken by everyone on the Web.

SOME HOPE

Newer CLIs do usually have an option to output JSON or (less prevalent) YAML.
They are forming a new ecosystem with different set of tools. From my
perspective, it is proving the point that the “universal interface” might not be
that universal and not as productive as envisioned (should I dare and say
“unacceptable”?) .

Should it be the half-way structured, aka semi-structured JSON? Is it the sweet
spot? I mean why stop here? Maybe we need something with schema? Let me know
what you think.

One of the notable projects, jc, is an adapter between the “universal interface”
and something that you would actually like to work with.

SHAMELESS PLUG

If only we could have a shell that could play a role in this new ecosystem… or
maybe even push it a bit in the direction of having semantically meaningful
objects on the screen so that interaction would be possible…

Yes, I am aware of other projects solving the same issue. While we mostly agree
on the problem, I haven’t yet seen a project which sees the solution the same
way as I do.

--------------------------------------------------------------------------------

Happy, DevOps-ing!

RUNNING ELEGANT BASH ON SIMPLE KUBERNETES (RANT)

March 19, 2021 Ilya SherLeave a comment

I was triggered by seeing “elegant” and “bash” in the same sentence. Here are
the titles I suggest the original author to consider for next blog posts:

1. Guide to Expressive Assembler
2. Removing Types from Scala
3. Adding Exceptions to Go
4. Introduction to Concise Java
5. Writing Synchronous JavaScript with Threads
6. Using Forth Without the Stack
7. Adding Curly Braces to Python
8. Making Guido Like Functional Programming
9. Using Uniform AWS APIs
10. Writing Safe big C Programs
11. Making C Higher Level Language than Portable Assembler
12. Making your Database Stateless
13. Making Eventual Consistency Immediate
14. How to Know that Backups are Working Without Doing Test Recovery
15. Finding Quality Code on Random Internet Sites
16. All Programming Languages are Beautiful (Illustrated)
17. Writing Bug-Free Code that does not Need Reviews
18. Learning Modern C++ in 3 Easy Steps in 2 Days
19. Stopping Hype Around Kubernetes – Practical Guide
20. Preventing Appearance of new JavaScript Frameworks
21. Why node_modules is not a Dumpster
22. Removing Most of the Syntax from Perl
23. Understanding Monads in 10 Minutes
24. How to Stop Debates and Fighting around OSS Licensing with 1 Month
25. Replacing bash in Next 20 Years

P.S. I’m still using bash at some places. Sadly, it’s still the most appropriate
solution in some cases, like we live in 90’s.

OPS TOOLS MARKETING BULLSHIT DICTIONARY

December 27, 2020December 27, 2020 Ilya SherLeave a comment

This article is aimed mostly at juniors. Lacking experience, you are a soft
target for marketing bullshit. I encourage critical thinking for evaluation of
products and services that are forced down your throat marketed to you.

BACKGROUND

The purpose of dishonest marketing is to mislead you into using a product and/or
a service that you don’t really need. This can waste your time and money.

Separate the products from marketing. The products themselves are not inherently
good or bad. The main question is whether to use product X or Y (and how) or
code the solution yourself given your specific situation: team, skill levels,
requirements, etc. This post will guide you how to see through marketing
bullshit when evaluating these products.

The list below is not comprehensive and I might add items later.

PRODUCTS MARKETING BULLSHIT

( in no particular order )

> Cool / coolness

If something is “cool” and you use it, you might write a blog post about it. It
has some PR value to you or your company. Other than that, “coolness” of a
product has nothing to do with its usefulness for your situation/use case.

> Our product is so much better than X

Note that often X will be the worst possible alternative. For example,
configuration management tools will most likely be compared to manual work
rather to other configuration management tools or any kind of automation. Ignore
and compare the suggested product to other viable alternatives.

> Use our product or become irrelevant / Our product is [becoming] industry
> standard

The message is sometimes direct but often it is hidden between the lines. This
is an attempt, many times successful, to exploit fear of missing out. Learn the
underlying principles then decide whether you want to use one of the products,
which come and go. Learning the underlying principles will take more time
upfront but you will become more professional. If you are junior Ops, learn
Linux and how to use it without any configuration management tools first; learn
using the cloud without CloudFormation or Terraform first; see the problems
which these tools trying to solve before using the tools.

> Our customers include big companies such as X, Y and Z

You have to understand what these companies are using the product/service for.
It can be a pilot for example. Check for common owners or investors. Remember
that big companies make mistakes too. Anyhow, this is irrelevant for your use
case and your situation until proven otherwise.

> Success story

The formula is simple: deep shit + our product = great success. Look closely.
Often deep shit + other sensible alternative solution would also be great
success. For example, configuration management tools show manual process and
then how it was automated. Chances are automating using scripts would be other
viable alternative.

> We abstract all the hard work away from you

You must understand what exactly is abstracted and how. After learning that, it
might happen that the perceived value of the product will drop. Skip the
learning step and start using the tool and you are in danger: you might need to
learn whatever was abstracted in hurry when facing a bug in the tool or be at
mercy of someone else to fix it. Remember: abstractions come at cost.

> Don’t reinvent the wheel, we have figured out everything already

This exploits your fear of looking stupid. Would you use a spaceship for
travelling from Moscow to Tokyo? I guess it would be cheaper to use a plane.
Even if tool X solves your problem it might cost you time and complexity and it
might be easier to code yourself or use simper tool. There are probably more
legitimate reasons not to use tool X.

COMPANIES MARKETING BULLSHIT

> Industry leading company

Who is not? Define your industry narrow enough and you are the leading company!
Simply ignore.

--------------------------------------------------------------------------------

Did I miss something? Let me know here in comments or on Reddit. Have a nice
day!

BASH OR PYTHON? THE SQUARE PEGS AND A ROUND HOLE SITUATION

October 31, 2020November 1, 2020 Ilya SherLeave a comment

The question “should I do it in bash or in Python?” is both frustrating and
common. Why one even needs to choose between two alternatives which are both
inadequate for the task at hand? Why try to pick one of the square pegs for the
round hole? I believe that you should not be in this annoying situation when
just trying to write a script and get back to your endless stream of other
todos.

Best illustration that I managed to find in 2 minutes

BOTH ARE INADEQUATE FOR OPS

bash does not meet any modern expectations for syntax, error handling nor has
ability to work with structured data (beyond arrays and associative arrays which
can not be nested). Let it go. You are not usually coding in assembly, FORTRAN,
C, or C++, do you? They just don’t match the typical Ops tasks. Don’t make your
life harder than it should be. Let it go. (Let’s not make it a blanket
statement. Use your own judgement when to make an exception).

Python along with many other languages are general purpose programming languages
which were not intended to solve specifically Ops problems. The consequence is
longer and less readable scripts when dealing with files or running external
programs, which are both pretty common for Ops. For example, try to check every
status code of every program you run, see how your code looks like. Sure you can
import 3rd party library for that. Is that as convenient as having automatic
checking by default + list of known programs which don’t return zero +
convenient syntax for specifying/overriding expected exit code? I guess not.

Your disorientation and frustration is completely legitimate.

ALTERNATIVES

Multitude of attempts to provide viable alternatives by different people are in
progress. As others authors, I would like to help my Ops colleagues to avoid
frustration and be productive. It just feels good.

--------------------------------------------------------------------------------

Informative section is over. Shameless plug about an alternative that I am
developing and how it is special follows.

NEXT GENERATION SHELL AS AN ALTERNATIVE

How Next Generation Shell differs from the alternatives? Reasonable question
that I would ask too before investing any more time if I was the reader.

Current shells as well as proposed alternatives treat UI as nothing has happened
since the 70-s: mostly typing commands and getting some text back.

How about real interaction with the objects on the screen? Oops. In a typical
shell there are no objects on the screen, it’s just a dumpster of combined text
(if you are lucky; could be binary) from stdout and from stderr from one or more
processes (unless steps were taken). WTF? It doesn’t help you to win. It helps
you to lose time. NGS does what is intended to make you productive. I have
organized my thoughts about how the UI should look and behave on the wiki page.

PROGRAMMING LANGUAGE

It looks like alternative solutions have the “let’s make the shell better”
approach and therefore are heavily based on the shell syntax and paradigms.

There is also “let’s make a library for existing language” approach which
doesn’t hit the target either. Can’t have a syntax for common ops tasks for
example.

And finally there is “let’s make a library and a syntax on top of existing
language”. Not sure about this one. Sounds good in theory. Looked some time ago
at something like this and the overall impression was … awkward (for the lack of
better word).

Approach in NGS: let’s make a good programming language for Ops, which fits the
use cases and has syntax and facilities for the most common tasks such as
running external programs.

The language follows the principle that the most common tasks should have their
own syntax or a library function (depending on usage frequency). Examples:

1. ``external program`` – runs external program and parses the output (JSON is
auto detected, easily extensible for anything else).
2. status(), log(), debug(), retry() – standard library functions. How many
times an Ops person should write his/her own retry()? It’s insane.
3. Argv() facility for constructing command line parameters (for calling
external program).
4. p=$(my_prog my_args &); ....; p.wait().

Small number of “big” core concepts in the language (types with inheritance,
multiple dispatch and exceptions).

DOGFOODING

Most of the standard library is in NGS.

The UI (only recently started working on it) is in NGS. It doesn’t make sense
that when a user of the shell wants to fix a bug in the UI and suddenly he/she
needs to learn Go, Rust, C or whatever other language.

We use NGS at work.

Most of the demo scripts come from either current or previous work.

HOW TO PROCEED?

1. Install NGS.
2. Consult documentation and look at sample scripts
3. Write scripts for non-production-critical tasks.
4. I am here to help. Do not hesitate to contact me with questions,
suggestions, or feedback. If there is anything Ops-y you are trying to do
seems to be easier in bash or Python – open an issue, because that’s a bug
from my perspective. Something is inconvenient? Yep, also a bug.

If you are like me, you will find at least some satisfaction in using the most
appropriate tool before continue to the myriad of other tasks that are in your
todo queue.

OR JUST LEARN MORE

1. Is NGS for you? Take a look at intended use cases.
2. Take a look at how NGS compares to other programming languages.
3. Browse sample scripts to get some impression about the language.

--------------------------------------------------------------------------------

Reddit:
https://www.reddit.com/r/devops/comments/jm3cwe/bash_or_python_the_square_pegs_and_a_round_hole/

POSTS NAVIGATION

Older Posts

Next Generation Shell is a modern programming language for DevOps. Join our
Discord or visit the official website.

FOLLOW ME ON TWITTER

RSS

RSS - Posts

ilya-sher.org Open in urlscan Pro 192.0.78.24 Public Scan

Form analysis 3 forms found in the DOM

GET https://ilya-sher.org/

POST https://subscribe.wordpress.com

Text Content

ilya-sher.org Open in urlscan Pro
192.0.78.24 Public Scan

Form analysis
3 forms found in the DOM