Avro – Unions

As a boolean:

{
	"fieldOne": "hello world",
	"fieldTwo": true,
	"foobar": {
		"boolean": false
	}
}

If value is null:

{
	"fieldOne": "hello world",
	"fieldTwo": true,
	"foobar": null
}

The above is what a field will look like as output where in Avro that field is defined as one of the Complex Types – Unions. Example below, it declares a schema which may be one of null or boolean.

{
  "type": [
    "null", 
    "boolean"
  ]
}

Convert a Map to POJO in Java

How do I convert a Map object to my other plain old Java object (POJO) without going into loops and having to write a class using Java reflection, or some other?

ANSWER

Well, yes, reflection is one but you didn’t want to do that yourself. For some good reason, I bet. It’s a good exercise if you have all the time in the world. But when faced with deadlines and having to write Unit tests for a whole class you wrote, there must be an easier way.

There is more than one way, but what I normally use is the Jackson ObjectMapper. Yes, the same one from the com.fasterxml.jackson library.

Anyway, with ObjectMapper it is pretty straightforward to do so. If I have a Person class like this:

    public class Person {
        private String firstName;
        private String lastName;
    }

My Map object will look like this:

        Map<String, Object> map = new HashMap<>();
        map.put("firstName", "Johnny");
        map.put("lastName", "Foo");

Then with ObjectMapper one can simply do this:

        ObjectMapper mapper = new ObjectMapper();
        Person person = mapper.convertValue(map, Person.class);

Alternatively, the keys in the Map may not align with the fields in the Person class. Well, I usually encounter this when working with JSON objects with lots of crazy looking field names. Something like this – NZT_Mor_First_Name__c – which I clearly don’t want my class field name to be like.

Well, we can use @JsonProperty annotation which is part of Jackson by the way and assign that our class field. The Person class will now look like:

    public class Person {

        @JsonProperty("NZT_Mor_First_Name__c")
        private String firstName;

        @JsonProperty("NZT_Mor_Last_Name__c")
        private String lastName;
    }

Again the Map will hold these values:

        Map<String, Object> map = new HashMap<>();
        map.put("NZT_Mor_First_Name__c", "Jose");
        map.put("NZT_Mor_Last_Name__c", "Yamut");

Now it will map out those weird looking key names to its corresponding class fields.

One thing to note is you might need to set ObjectMapper features such as ignoring unknown properties and make it case insensitive. Allow it a bit more room to wiggle, wiggle.

        ObjectMapper mapper = new ObjectMapper()
                .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
                .enable(MapperFeature.ACCEPT_CASE_INSENSITIVE_VALUES);

Should I quote Strings in YAML?

Should I or should I not? Is it required? Will it work even without quotes – single or double?

ANSWER

I’ve asked myself these questions several times. Often I end up double-quoting long strings, but not for one word ones or shorter strings. Which makes it rather inconsistent. Especially if that string contains characters like a forward slash, a colon or semi-colon. Like when you have a URL as a value. I suppose it’s out of habit.

Thing is, it is not required. Strings can be quoted or not. There is no difference. It will still work with or without it. Even with spaces in that string. YAML allows it. Below are 2 examples:

phrase: 'The quick brown fox jumps over the lazy dog.'
poem: "Hey, diddle, diddle. The cat and the fiddle. The cow jumped over the moon."

The key part of the key-value pair can also be quoted too. The YAML won’t error out. Again, it’s allowed. So this is valid:

'foo':
  "bar": foobar

Single quote or double quotes don’t matter. It can be mixed. Above examples are both acceptable.

However, not quoting is not true in all cases. There are certain instances when it is required such as when there is a colon in the key. In this case, the key needs to be quoted.

'key:': value

What about quotes in the value itself?

CORRECT - phrase1: 'The quick ''brown'' fox jumped over the lazy dog.'
WRONG- phrase2: "The quick ""brown"" fox jumped over the lazy dog."
CORRECT - phrase3: "The quick \"brown\" fox jumped over the lazy dog."
WRONG - phrase4: "The quick '"brown"' fox jumped over the lazy dog."
CORRECT - phrase6: The quick 'brown' fox jumped over the lazy dog.
CORRECT - phrase7: The quick "brown" fox jumped over the lazy dog.

Finally, what does YAML mean?

It is short for – YAML Ain’t Markup Language.

What’s my Windows version

You’ll probably get asked this once or twice. Especially when your Windows computer is not working right, and you are running all over the Internet trying to look for answers. Perhaps would-be good Samaritans might ask this off you as well.

And you reply, “Windows 10”. But rightfully so.

After all, isn’t that the Windows that is installed on your machine right now. Well, yes. Then again, no. No not in the sense that you are wrong. But sometimes the answer to the question of what Windows version is installed on your PC might be something else. Sometimes support may ask you this, and likely they will tell you how to give them the answer they are looking for. If you don’t already know this, then this can save time and misunderstanding. Below are the steps on how to get that information.

What version is my Windows?

  • Go to the Start Menu. Click it. Press the Windows key
  • Type “Run”. The shortcut key is Windows key + R.
  • Type “winver” in the field labeled with Open. You may also type winver in the search box on your taskbar if you have that one.
  • Click OK or press enter
  • Small dialog Windows will pop out title – About Windows.

The information will be found on this pop up window. On the first few lines you should see something like:

Microsoft Windows
Version 2004 (OS Build 19041.630)

Note: Don’t include quotes when typing in those words in the steps.

Is there a Clipboard for Windows 10?

What are the freely available Clipboard programs, if any, for Windows 10? Are there such? I don’t want ads in it and I prefer it simple as possible. No crazy or complex features. I’m only after a simple copy-paste of the short history of what I’ve copied on my desktop. I find this useful for my productivity. I somehow always ending up in open a Notepad and copying the items in there. It works but a clipboard would be better in my opinion.

ANSWER

Windows 10 already comes with a built-in Clipboard Manager. This was from an update some time ago. However it is not enabled by default, buried somewhere in the Settings and you will have to activate it. This feature is not very well known. But it is there and it works as expected.

To activate:

Open Settings via any of the following ways:

  • You can either search for it if you have that search bar on your Windows taskbar.
  • Start typing “Settings” when you have the Windows Start Menu opened.
  • Open the Start Menu then, then click on the Gear icon on the left hand side of the menu.
  • Windows + i shortcut key combo.

Once you are in Settings, go to System. That should be the first icon entry at the top.

You will be taken to another page where there are items at the left hand side. Scroll down and find Clipboard.

Under Clipboard history, toggle the button to On.

To get to your clipboard history, simply press Windows + V shortcut key combo. It will appear as a floating box over your last active application. Otherwise, it will show up on the right side of the desktop on top of the system tray.

The clipboard can contain a max of 25 entries. It is a FIFO (first in, first out) stack so the oldest entry will go out first once that entry limit is hit.

It supports plain text, HTML and images up to 4MB size.

Your clipboard history can be synced across your Windows devices. Be warned that if you enable this it means you are allowing Windows to save the history on Microsoft servers. Make sure you don’t have sensitive information in there when the Sync functionality is enabled.

Re-publish Kafka record not setting partition number?

Well, I plan to re-publish Kafka records that my app is consuming to another Kafka topic on another host (internal only). This sort of lets other groups consume data from this topic without having to create connections outside of the private network, since the original data source is a 3rd party vendor. They can then also do their own transformations or other business logic on it as if they were consuming from the original source. Delay from source to re-publish should be minimal. Will still be close to real time.

What I normally do when producing records is not to set a partition by default. Would this be okay when re-publishing records? Will there be conflict with original record partition number to the target topic partition?

ANSWER

How many partitions does your target topic have? Do you have freedom to create as many partitions as the source?

(1) If you can create 1:1 partitions then my suggestion is to retain the original partition number of that record when you publish to the destination topic.

(2) You can also assign the partition in a round-robin way when you have lesser partitions on your target topic. Save metadata info of that record in the headers, such as original partition assignment and etc. Perhaps that will be useful down the line for your consumers.

(3) Lastly, you can NOT assign a partition number. Instead, let Kafka do the magic for you. When no partitions are set, Kafka will determine the next partition in a round robin method. You can do this with a null value instead of an int for partition number.

Bottom line is that it depends on the project requirements, I suppose, or more so on the infrastructure of your destination Kafka.

Can’t copy text from webpage?

Sometimes I come upon a website that I cannot copy text from a HTML web page. Simply cannot click and highlight the lines I try to select for copying. I thought that my browser/mouse was not working. But I go to other sites, I am able to highlight/copy normally. They must’ve done something on that page probably to prevent content from being lifted of the page easily?

Why is this and how do they do it? Is this a javascript in the page that prevents me from copying?

ANSWER

Could be a script on the webpage. I’m certain that can be done or already been done through Javascript before.

I know that the same effect can be achieved via CSS alone. It is easier, cleaner and does not add to another Javascript messing up or slowing down a site. I prefer this method over the other.

Below are the CSS lines that does what you want. Different ones for different browsers. It is vendor specific for some, but does the same exact thing.

   user-select: none; /* Supported by Chrome, Firefox & Opera */
  -moz-user-select: none; /* Firefox (older) */
  -ms-user-select: none; /* Edge */
  -khtml-user-select: none; /* Konqueror */
  -webkit-touch-callout: none; /* Safari for iOS */
  -webkit-user-select: none; /* Safari for Mac */

Try it out. You can make it for entire page/site.

* {
     /* CSS lines go here */
}

Or you can make it so only targeted content will have it using Class or ID selectors.

#text-no-select {
     /* ID selector -  CSS lines go here */
}

.text-no-select {
     /* Class selector -  CSS lines go here */
}

Be aware this does not totally prevent anyone from copying content on a website. There are many other ways to go around this or even with a Javascript impementation. It is a minor roadblock.

How to make IntelliJ check and create SerialVersionUID

When the class implements Serializable I want the serialVersionUID field to be auto-created by IntelliJ IDE. This feature comes in automatically in Eclipse. But the former does not.

How do I make IntelliJ create the field or warn me that it is not defined in the class yet?

ANSWER

It is not turned on by default in IntelliJ. Can be easily done by going to the following:

File > Settings > Editor > Inspections > Java > Serialization Issues

Then find and select – Serializable class without ‘serialVersionUID’ – by checking on the box to the right of it.

Similarly, you can search for it once in File > Settings. Keyword for search can be – ‘serialVersionUID’ – without the quotes.

The path or location of this feature on IntelliJ may vary. It should be the same on Linux and Windows for versions 2019 and 2020.

This will only throw a warning. It does not magically create the field in all those classes that implements Serializable straight away. But then IntelliJ will help you create it, like any other decent IDE out there. Go to the class name declaration, you may find that it is subtly highlighted in yellow to hint that there is a user action that can be done. Hover then click on the option – Add ‘serialVersionUID’ field – and you will get the desired result like this one:

private static final long serialVersionUID = 6265605531073886420L;

Database tables map One To Many without foreign key in @JoinColumn

Possible to not have foreign key relationship in child table to its parent in the database, while in JPA it will still be able to map out fine? How will this be done?

Can I still use @JoinColumn to define the field for reference back to the parent table?

This is a one-to-many relationship. One parent record, many child records.

I only care about writes to the database tables right now.

ANSWER

Yes, no FKs defined in the database is fine. JPA will still be able to map out there relationships as defined in your entity classes.

In fact, enabling – spring.jpa.generate-ddl – in your application properties file will create the necessary Foreign Key and/or Unique Key constraints in the database. This is vendor dependent however.

For a One To Many relationship, it is straightforward to do. Example below definition below:

    @OneToMany(cascade = CascadeType.ALL)
    @JoinColumn(name = "recommendation_id")
    private List<CommentEntity> commentList;

This will map out the entity object values to the Comment table, assuming that is the name of the target table. Each one will be persisted to the child table along with the ID of the parent Recomendation under the recommendation_id column.

How to ignore unknown fields when parsing JSON using Jackson

In Java, there is Jackson 2 library that is very popularly used when reading JSON objects and mapping those values out to a POJO.

Happy path, as long a I have all the fields from the JSON defined in the POJO then okay. In a perfect world it will work 100%.

But there is no perfect world. The JSON object in other scenarios can contain unknown fields. These are not in my POJO. Application fails from reading that JSON to POJO.

Getting this error:

Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "status" (class com.example.dto.Evaluation), not marked as ignorable (6 known properties: "Book",

ANSWER

Use this Jackson annotation at a class level. This has to be configured per class. That should do it. Will ignore any unknown properties silently including nested objects.

@JsonIgnoreProperties(ignoreUnknown = true)

When you are using Jackson ObjectMapper directly, that can be configured to ignore unknown properties globally as well.

ObjectMapper objectMapper = new ObjectMapper();
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

How Do I Decode An OpenShift Secret In A Mounted Volume?

Mounting an OpenShift Secret to its own volume is straightforward. There are plenty of examples on how to do it. The Web is littered with examples. Next, the most common thing any manual, guide or tutorial would say is to encode the secret in Base64.

For example I have a SSL certificate stored in a Java KeyStore file format or JKS. The recommended way is to store it in a Secret instead of ConfigMap since it is sensitive information. Of course, that goes without saying the JKS file is password-protected.

Second, it must be as a Base64 string before I save it as OpenShift Secret.

How do I get the Base64 string to be decoded in the mounted volume? This one does not seem to get many answers.

Do I need to include shell commands to decode it and write it to a folder?

Which folder should I write it to as best practice?

ANSWER

#1 The encoded JKS Secret as an environment variable

This is an option but I’m not a fan of it. The JKS file when encoded can become very long, especially when the file size is large. I don’t think environment variables were meant to be used like this – hold very long text values. That said, this is an easier implementation.

Map out the secret as an env var in OpenShift or Kubernetes. Then you can use that env var to echo the value and write it out to a file.

echo -n $SECRET_JKS_VAR | base64 --decode > /file/path/to/decoded-secret.jks

#2 Mount for read, mount for write

You have the secret mounted to a file path. That is for reading. Now you need to decode, which means writing to a file so your application can read it back unencoded.

First, define a mount point as and empty directory. It must be writable. Then make it memory only.

Next, read the JKS file from its mount point, write it out to the empty dir mount point.

cat /mount/file/path/for-reading/encoded-secret.jks | base 64 --decode > /mount/file/path/for-writing/decoded-secret.jks

I like this approach better. It makes more sense rather than mapping a very long base64-encoded text value as an environment variable. The caveat is here is slightly more configurations to be made. Also if I’m not mistaken the in-memory volume count against your app memory quota. Should be negligible unless you write thousands of files into it.

#3 Lastly, write where you can

Underneath that container is a file system. Where you have permission to write, then do so there. If it’s Linux, then this decision is pretty much arbitrary IMHO. Put it in /tmp or /home or /mnt. Security wise, others might have access to your app’s pod, that means they can get to the Secret as well. But that is another topic.

WorPress Editor Won’t Load Content Anymore

Got this weird WordPress behaviour recently. While most pages load successfully, editing something – like a post or a page – will draw a blank browser tab. Zero content whatsoever. It has the same result for both Firefox and Chrome. Not tried on other browsers.

ANSWER

Uncaught EvalError: Refused to evaluate a string as JavaScript because 'unsafe-eval' is not an allowed source of script in the following Content Security Policy directive: "default-src 'unsafe-inline'

This was the error that I got when I went into the page, opened developer tools, then reloaded the page.

Caused by additional custom Header policy on the web server of the website. Looks something like this:

Header set Content-Security-Policy "default-src 'unsafe-inline' 

Add ‘unsafe-eval’ to go around it. Use of eval() is usually considered bad practice and not secure hence it’s getting blocked until it is allowed by adding it into the policy. Now the Header will look like this after modifying it. Don’t forget to restart the web server.

Header set Content-Security-Policy "default-src 'unsafe-inline' 

More detailed explanations can be had here: https://scotthelme.co.uk/content-security-policy-an-introduction/

How to Fix Gradle Wrapper Permission Denied Error

I am getting this “gradlew permission denied” error after pushing my code to my GitLab repository. The build is not able to continue because the gradle wrapper is not able to run.

What is causing this and how to fix it?

ANSWER

From a local terminal/command line, use the Git command that follows to fix this issue:

git update-index --chmod=+x gradlew

Continue to commit the modifications to the gradlew file.

Push the changes to your Git repository.

You can read more at this page: https://www.joseyamut.xyz/2020/08/15/fix-gradlew-permission-denied-on-openshift-deploy/

How to show Snap app icon properly on Ubuntu Unity desktop

Had this issue where after I logged into my Ubuntu desktop all Snap apps icons do not show properly. Instead I see a plain icon in place of what the app icon should be. Have you the same experience?

This might have been fixed in later Ubuntu LTS releases, I really don’t know. But I am still on 16.04 Xenial Xerus until its EOL. Also, the reason is that I love the Unity Desktop. The Gnome Shell replacement just doesn’t cut it for me.

I have been looking for a solution on this issue for some time. The 2 main things people point out are:

– Add /snap/bin to the /etc/environment file, and;

– Something to do with the profile.

Neither of those solutions worked for me. But it did work for others according to what I have read so far.

ANSWER

To fix this icon issue, create a symlink of the Snap’s .desktop shortcut file to your user’s .local/share/applications directory.

It is as simple as that. Once done the problem will go away, and you will be left with a nice-looking icon that came with the Snap installation. This won’t go away on following restarts.

For more details on how to do this fix exactly, refer to this article – https://www.joseyamut.xyz/2020/08/13/getting-that-ubuntu-snap-icon-right-on-unity-shell/

How Do I See the List of Users Allowed on a MS SQL Server Database

I want to know the users that are allowed access to a MS SQL Server database. I only want a specific database. Is there a user list hidden somewhere?

What are the ways to go about doing this? Is there an app or through a SQL query?

ANSWER

Use Microsoft SQL Server Management Studio. That is the fastest way to do it. Find the database from the explorer menu on the left-hand side. Expand the folder icon. Go to the Security folder, then click on Users next. That’s it!

If you’re using a different SQL client and you can’t see Security > Users from the GUI. Then your other option is through a SQL query.

This is the query below. More details at this page here: https://www.joseyamut.xyz/2020/08/06/get-user-list-in-a-microsoft-sql-server-database/.

USE <database_name>;
SELECT name AS username,
       type_desc AS type,
       authentication_type_desc AS auth_type
FROM sys.database_principals
ORDER BY type;

Happy user hunting! =)